How Do We Assess the Trustworthiness of AI? Introducing the Trustworthiness Assessment Model (TrAM)

May combine two claims.

Accurate trustworthiness assessment enables trustors to base their trust on adequate expectations about the system and to make informed decisions about trusting behavior.

“When someone's trustworthiness assessment is accurate, they can base their trust on adequate expectations about the system's capabilities and limitations [82] and make informed decisions about their trusting behavior.”

• cites [82]Not checked yet

  Not checked in the free run.

May combine two claims.

Inaccurate trustworthiness assessment can lead to trust based on false expectations and inadequate trust, with risks of overtrust and undertrust.

“If the assessment is inaccurate, the person's trust in the system may be based on false expectations, and consequently, may be inadequate.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Propensity to trust develops across the lifespan and may be particularly important in unfamiliar situations with unknown trustees.

“Propensity to trust develops across the lifespan and may be particular important in unfamiliar situations with unknown trustees [5].”

• cites [5]Not checked yet

  Not checked in the free run.

May combine two claims.

Propensity to trust is a relatively stable human disposition, which limits the ability of external factors to influence trust development processes through propensity to trust.

“Research suggests that propensity to trust is a relatively stable human disposition [5], which limits the ability of external factors to influence trust development processes through propensity to trust.”

• cites [5]Not checked yet

  Not checked in the free run.

May combine two claims.

Although the disposition to trust humans cannot be directly extrapolated to machines, the literature on trust in automation recognizes propensity to trust as a stable disposition.

“While the disposition to trust humans cannot be directly extrapolated to the disposition to trust machines, the literature on trust in automation also recognizes the notion of propensity to trust as a stable disposition [52,57,82].”

• cites [52]Not checked yet

  Not checked in the free run.

• cites [57]Not checked yet

  Not checked in the free run.

• cites [82]Not checked yet

  Not checked in the free run.

May combine two claims.

Perceptions of trustworthiness are situational and subjective.

“Perceptions of trustworthiness are situational and subjective [5,52,91].”

• cites [5]Not checked yet

  Not checked in the free run.

• cites [52]Not checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

At the macro level, individual micro-level trustworthiness assessments propagate across different trustors, such that one stakeholder's assessment affects other stakeholders' assessments of the same system.

“At the macro level, we propose that individual micro-level trustworthiness assessments propagate across different trustors -one stakeholder's trustworthiness assessment of a system affects other stakeholder's trustworthiness assessments of the same system.”

Classified as probably not a substantial claim. May combine two claims.

Trustworthiness perceptions reflect a trustor's assessment of the trustee's abilities, principles, and intentions, as well as beliefs about whether the trustee could help the trustor achieve their goals.

“In this understanding, trustworthiness perceptions reflect a trustor's assessment of the trustee's abilities, principles, and intentions, as well as a trustor's beliefs about whether the trustee could help the trustor to achieve their goals [5,22,81,82,91].”

• cites [5]Not checked yet

  Not checked in the free run.

• cites [22]Not checked yet

  Not checked in the free run.

• cites [81]Not checked yet

  Not checked in the free run.

• cites [82]Not checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

To understand how people arrive at their perceived trustworthiness of a system, it is necessary to understand both individual assessments of system trustworthiness at the micro level and trustworthiness propagation processes among trustors at the societal level.

“In conclusion, we propose that in order to understand how people arrive at their perceived trustworthiness of a system, it is necessary to better understand both individual assessments of system trustworthiness (i.e., the micro level) and trustworthiness propagation processes among trustors at the societal level (i.e., the macro level).”

Probably not a substantial claim: may not pass S4 (based).

Individual standards of a few stakeholders can propagate to the trustworthiness assessments of all other stakeholders along the system lifecycle.

“because the individual standards of a few stakeholders can propagate to the trustworthiness assessments of all other stakeholders along the system lifecycle.”

Probably not a substantial claim: may not pass S4 (based).

An accurate assessment of a system's actual trustworthiness is the basis for adequate trust and adequate trusting behavior.

“Second, we propose that an accurate assessment of a system's actual trustworthiness is the basis for adequate trust, and adequate trusting behavior.”

Classified as probably not a substantial claim.

In interpersonal trust, ability, benevolence, and integrity subsume all facets of perceived trustworthiness.

“Mayer et al. [91] propose that ability (a “group of skills, competencies, and characteristics that enable a party to have influence within some specific domain”, p. 717), benevolence (“the extent to which a trustee is believed to want to do good to the trustor, aside from an egocentric profit motive”, p. 718), and integrity (“the trustor's perception that the trustee adheres to a set of principles”

• cites [91]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Lee and See translated ability, benevolence, and integrity to trust in automation as performance, purpose, and process.

“Lee and See [82] translated these facets to trust in automation referring to performance (“the competency or expertise as demonstrated by its ability to achieve the operator's goals”, p. 59), purpose (“corresponds to faith and benevolence and reflects the perception that the trustee has a positive orientation towards the trustor”, p. 59) and process (“the degree to which the automation's algorithm”

• cites [82]Not checked yet

  Not checked in the free run.

May combine two claims.

Although trust and trusting behavior can be positively related, the stakes associated with the actual decision to engage in trusting behavior prevent trust from translating directly into trusting behavior.

“Although trust and trusting behavior can be positively related [73], the stakes associated with the actual decision to engage in trusting behavior toward a trustee prevent trust from translating directly into trusting behavior.”

• cites [73]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

If perceptions of trustworthiness and intentions of trust influence behavior, then trustworthiness, trust, and trusting behavior are expected to be positively related.

“Nevertheless, if we assume that perceptions (i.e., of trustworthiness) and intentions (i.e., trust) influence behavior, then trustworthiness, trust, and trusting behavior are expected to be positively related [135].”

• cites [135]Not checked yet

  Not checked in the free run.

• AssumptionThe authors state this as an assumption: Perceptions of trustworthiness and intentions of trust influence behavior.

May combine two claims.

Inaccurate trustworthiness assessments are a cause of miscalibrated trust, as evident in automation bias and algorithm aversion.

“Moreover, inaccurate trustworthiness assessments as a cause of miscalibrated trust [75,88,109] are evident in research on the automation bias phenomenon (i.e., that people initially tend to overestimate the trustworthiness of systems for certain tasks; [98,108]) and in research on algorithm aversion (i.e., that people seem to underestimate the trustworthiness of systems after seeing the system”

• cites [75]Not checked yet

  Not checked in the free run.

• cites [88]Not checked yet

  Not checked in the free run.

• cites [109]Not checked yet

  Not checked in the free run.

May combine two claims.

Previous research has distinguished between actual and perceived trustworthiness and has emphasized the importance of accurately assessing actual trustworthiness.

“We acknowledge that previous research has also distinguished between AT and PT, and has also emphasized the importance of accurately assessing AT [82,103,138].”

• cites [82]Not checked yet

  Not checked in the free run.

• cites [103]Not checked yet

  Not checked in the free run.

• cites [138]Not checked yet

  Not checked in the free run.

May combine two claims.

In the TrAM, automation bias and algorithm aversion boil down to the system's actual trustworthiness being either lower (automation bias) or higher (algorithm aversion) than the trustor's perceived trustworthiness.

“In the TrAM, these examples boil down to the fact that the system's AT was either lower (automation bias) or higher (algorithm aversion) than the trustor's PT of that system [64,71].”

• cites [64]Not checked yet

  Not checked in the free run.

• cites [71]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Characteristics of other people that are to be assessed are often latent constructs.

“Characteristics of other people that are to be assessed are often latent constructs.”

May combine two claims.

There is typically no way to perfectly measure individual facets of actual trustworthiness, nor to accurately assess how they combine to actual trustworthiness.

“However, there is typically no way to perfectly measure these individual facets (e.g., because only a subset of all available data can be accessed to assess system performance), nor to accurately assess how they combine to AT [52,82].”

• cites [52]Not checked yet

  Not checked in the free run.

• cites [82]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

Human trustors need to detect and utilize certain, possibly many, cues to arrive at their perceived trustworthiness.

“Human trustors then need to detect and utilize certain (and possibly many) cues to arrive at their PT.”

Probably not a substantial claim: may not pass S4 (based).

The relationship between actual trustworthiness and perceived trustworthiness sets the stage for everything that follows in the trust development process, including adequate trust and trusting behavior.

“We propose that this relationship between AT and PT sets the stage for everything that follows in the trust development process (e.g., in terms of adequate trust and trusting behavior).”

Classified as probably not a substantial claim.

Benevolence, integrity, and ability are treated as facets of actual trustworthiness.

“For example, benevolence, integrity, and ability are treated as facets of AT [50,82,91].”

• cites [50]Not checked yet

  Not checked in the free run.

• cites [82]Not checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

May combine two claims.

In the micro-level trustworthiness assessment process, trustors cannot directly assess a system's actual trustworthiness; instead, they assess it on the basis of available cues.

“Specifically, in the micro-level trustworthiness assessment process (Figure 2), we propose that trustors cannot directly assess a system's AT.Instead, they assess a system's AT on the basis of available cues.”

Classified as probably not a substantial claim. May combine two claims.

A system's actual trustworthiness depends on the system characteristics and on the individual standards of trustworthiness.

“To summarize, a system's AT depends on the system characteristics and on the individual standards of trustworthiness.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

System characteristics in the context-free space are objective in the sense that they do not depend on the trustor.

“It is important to distinguish that system characteristics in this (only theoretically existing) context-free space are also objective in the sense that they do not depend on the trustor.”

Classified as probably not a substantial claim.

The specification of a system's ability depends on users' goals, interests, and personal alternatives to the system.

“For instance, the specification of a system's ability depends on the users' goals (e.g., screening vs. decision making), interests (e.g., saving money vs. data privacy), and their personal alternatives to the system (e.g., one's own ability to perform the task at hand).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

System accuracy is a facet of a system's ability.

“For instance, system accuracy will be a facet of a system's ability.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Actual trustworthiness is always subjective in the sense that it is relative to the trustor's individual standards of trustworthiness.

“In contrast, AT is always subjective in the sense that it is relative to the trustor's individual standards of trustworthiness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Differences in individual standards may result from the granularity and specification of the trustor's standards.

“Differences in the individual standards may also result from the granularity (from general to detailed) and specification (from explicit to implicit) of the trustor's standards.”

May combine two claims.

Individual standards are not arbitrary; they refer to a common understanding of the concept of trustworthiness of systems, influenced by public discussions about trustworthy AI.

“Individual standards are not arbitrary: they refer to a common understanding of the concept of trustworthiness of systems, which is also influenced by public discussions about trustworthy AI [29,50,151].”

• cites [29]Not checked yet

  Not checked in the free run.

• cites [50]Not checked yet

  Not checked in the free run.

• cites [151]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Actual trustworthiness reflects the degree to which the system characteristics match a trustor's individual standards for trustworthiness with respect to a particular task and point in time.

“Specifically, AT reflects the degree to which the system characteristics match a trustor's individual standards for trustworthiness with respect to a particular task and point in time.”

Classified as probably not a substantial claim.

What constitutes a trustworthy system differs between trustors and depends on their individual standards for system trustworthiness.

“What constitutes a trustworthy system differs between trustors and depends on their individual standards for system trustworthiness [70,103].”

• cites [70]Not checked yet

  Not checked in the free run.

• cites [103]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

What is high enough for a system's accuracy may be different for different trustors.

“What is high enough may be different for different trustors.”

May combine two claims.

A system's actual trustworthiness can be influenced by changes in the system characteristics and by changes in the individual standards.

“The AT of a system can be influenced by changes in the system characteristics and by changes in the individual standards.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Perceived trustworthiness may consist of the same general facets and subfacets as actual trustworthiness, and since these are not observable, they must be inferred from manifest variables, i.e., observable cues.

“PT may consist of the same general facets (e.g., ability, integrity, benevolence), same subfacets (e.g., accuracy for ability), and since these are not observable, they must be inferred from manifest variables (i.e., observable cues).”

Probably not a substantial claim: may not pass S2 (checkable).

Cues are pieces of evidence that presumably provide insight into the actual trustworthiness of a system.

“Cues are pieces of evidence that presumably provide insight into the AT of a system [18,85,136].”

• cites [18]Not checked yet

  Not checked in the free run.

• cites [85]Not checked yet

  Not checked in the free run.

• cites [136]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

The facets that contribute to overall perceived trustworthiness are similar to those that contribute to overall actual trustworthiness.

“The facets that contribute to overall PT are similar to those that contribute to overall AT (see Figure 3).”

May combine two claims.

Single cues may provide only narrow or even misleading insights into the actual trustworthiness of a system, and each cue is related to the actual trustworthiness to some degree.

“Single cues may provide only narrow or even misleading insights into the AT of a system, and each cue is related to the AT to some degree.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The facets of perceived trustworthiness represent the external evaluation, e.g., through researchers, and must not necessarily reflect the trustors' internal representation of their perceived trustworthiness.

“Note, that these facets rather represent the external evaluation (e.g., through researchers) and must not necessarily reflect the trustors' internal representation of their PT.”

May combine two claims.

Trustors could lower their individual standards, which would increase the system's actual trustworthiness.

“Trustors could also lower their individual standards, which would increase the system's AT [134].”

• cites [134]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Trustors may not be able to mark some boxes after the assessment, or may be unsure about whether they should mark certain boxes.

“This means that trustors may not be able to mark (i.e., tick or cross out) some of the boxes after the assessment, or they may be unsure about whether they should mark certain boxes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Research has investigated how different cues are weighted and associated with trust in decision aids.

“In line with this, research has investigated how different cues are weighted and associated with trust in decision aids [23].”

• cites [23]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In most cases, after the trustworthiness assessment, there is still uncertainty about the trustee's actual trustworthiness.

“In most cases, after the trustworthiness assessment, there is still uncertainty about the trustee's AT.”

Classified as probably not a substantial claim.

Perceived trustworthiness reflects the result of a trustor's assessment of the trustee's actual trustworthiness.

“PT reflects the result of a trustor's assessment of the trustee's AT [5,91].”

• cites [5]Not checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

May combine two claims.

Factors influencing cue detection include trustor's mood and emotions, attentional capacity, situation awareness, time pressure, and experience with a system.

“Possible factors influencing cue detection are trustor's mood and emotions [12,15,38,95], attentional capacity [49], situation awareness [34], time pressure [115], or their experience with a system [132].”

• cites [12]Not checked yet

  Not checked in the free run.

• cites [15]Not checked yet

  Not checked in the free run.

• cites [38]Not checked yet

  Not checked in the free run.

• cites [95]Not checked yet

  Not checked in the free run.

• cites [49]Not checked yet

  Not checked in the free run.

• cites [34]Not checked yet

  Not checked in the free run.

May combine two claims.

Perceptions regarding system accuracy vary with the perceived difficulty of the system's task, implying that contextual information may influence cue utilization.

“For instance, perceptions regarding system accuracy vary with the perceived difficulty of the system's task, implying that contextual information may influence cue utilization [107].”

• cites [107]Not checked yet

  Not checked in the free run.

May combine two claims.

On the trustor's side, cue detection and cue utilization determine how strongly cues relate to trustor's PT and thus affect the accuracy of the assessment of AT.

“On the trustor's side, cue detection and cue utilization determine how strongly cues relate to trustor's PT and thus affect the accuracy of the assessment of AT.”

May combine two claims.

Cues may be uncorrelated with AT, although trustors may use them to form their PT.

“In some cases, cues may also be uncorrelated with AT, although trustors may use them to form their PT.”

May combine two claims.

Anthropomorphism may be a cue that is not related to the AT of a system, but it may be a cue that is often utilized by trustors and strongly influences the PT of the system.

“Thus, anthropomorphism may be a cue that is not related to the AT of a system, but it may be a cue that is often utilized by trustors and strongly influences the PT of the system.”

Classified as probably not a substantial claim.

In order to make an accurate and informed assessment, relevant cues must be available.

“In order to make an accurate and informed assessment, relevant cues must be available.”

May combine two claims.

Anthropomorphizing a system may increase trust, but trust may also become (more) inadequate due to an inaccurate trustworthiness assessment.

“As a consequence, anthropomorphizing a system may increase trust, but trust may also become (more) inadequate due to an inaccurate trustworthiness assessment”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

User interface properties such as low contrast could make cue detection more difficult.

“Beyond that, user interface properties such as low contrast could make cue detection more difficult.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The macro-level trustworthiness assessment of a system can be viewed as a net of micro-level trustworthiness assessments of different trustors, each of which provides new trustworthiness cues to other trustors.

“To understand how the accuracy of a stakeholder's trustworthiness assessment of a system is influenced by the trustworthiness assessment of other stakeholders, we propose that the macro-level trustworthiness assessment of a system can be viewed as a net of microlevel trustworthiness assessments of different trustors, each of which provides new trustworthiness cues to other trustors (see Figure 5).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Trustors can, but do not have to, exchange cues with each other.

“Trustors can, but do not have to, exchange cues with each other.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Automation bias is a phenomenon associated with overtrust in systems.

“For instance, such effects have been shown in research on the automation bias as a phenomenon associated with overtrust in systems [108]”

• cites [108]Not checked yet

  Not checked in the free run.

May combine two claims.

Little domain-, task-, and system knowledge may lead to incorrect utilization of cues, which may lead to inadequate assessments of system's AT.

“In contrast, little domain-, task-, and system knowledge may lead to incorrect utilization of cues, which may lead to inadequate assessments of system's AT (e.g., assuming that a high-quality user interface or the logo of a popular company indicates high system accuracy [8,54,72]).”

• cites [8]Not checked yet

  Not checked in the free run.

• cites [54]Not checked yet

  Not checked in the free run.

• cites [72]Not checked yet

  Not checked in the free run.

May combine two claims.

AT is likely to differ between stakeholders assessing the same system, and the system characteristics contain as many ATs as there are trustors assessing the system.

“Since AT is likely to differ between stakeholders assessing the same system, the system characteristics contain as many ATs as there are trustors assessing the system.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Algorithm aversion is a phenomenon associated with undertrust after seeing systems fail.

“and in research on algorithm aversion as a phenomenon associated with undertrust after seeing systems fail [27].”

• cites [27]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

ATs of different trustors may overlap to varying degrees.

“ATs of different trustors may overlap to varying degrees (see Figure 5).”

May combine two claims.

Stakeholders utilize cues, arrive at their PT based on these cues, and as a result may produce new cues for other stakeholders.

“These stakeholders utilize cues, arrive at their PT based on these cues, and as a result may produce new cues (e.g., certificates, system manuals, testimonials) for other stakeholders that assess system trustworthiness (see Figure 5).”

May combine two claims.

The trustworthiness propagation process is not a pipeline, nor does it involve a strict hierarchy; rather, it takes place in a complex social network of stakeholders.

“It is not a pipeline, nor does it involve a strict hierarchy; rather, it takes place in a complex social network of stakeholders.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

End user's individual standards may often be implicit, but may become more explicit and specific as they interact with the systems.

“End user's individual standards may often be implicit, but may become more explicit and specific as they interact with the systems.”

May combine two claims.

End users play a special role in the trustworthiness propagation process because they repeatedly assess a system's AT by interacting with the system and have to decide whether a system is trustworthy in general and whether to rely on a single output.

“End users play a special role in the trustworthiness propagation process, because they a) repeatedly assess a system's AT by interacting with the system, and b) have to decide whether a system is trustworthy in general, but also whether they want to rely on a single output (i.e., end users have to decide whether a single system recommendation belongs to the 5% incorrect cases reported in the”

• cites [18]Not checked yet

  Not checked in the free run.

May combine two claims.

Stakeholders' individual standards shape their assessment of system trustworthiness and, consequently, their provision of secondary cues.

“Stakeholders' individual standards shape their assessment of system trustworthiness and, consequently, their provision of secondary cues.”

May combine two claims.

If Bob heavily utilizes secondary cues provided by Alice, and if Alice's individual standards do not match Bob's, he might mistakenly believe that the system is trustworthy for him.

“If Bob heavily utilizes secondary cues provided by Alice, and if Alice's individual standards do not match Bob's, he might mistakenly believe that the system is trustworthy for him.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Secondary cues are important in increasing the efficiency of the micro-level trustworthiness assessment process.

“Secondary cues are important in increasing the efficiency of the micro-level trustworthiness assessment process.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Deployers' individual standards for trustworthiness will be tied to their business goal.

“Their individual standards for trustworthiness will be tied to their business goal.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

System designers will provide secondary cues such as documentation, a summary of training, validation and test data, accuracy metrics, limitations and capabilities of the system, information presented in the user interface, and model cards.

“Following their assessment and with reference to regulatory requirements, system designers will provide secondary cues such as a documentation [70], a summary of their training, validation and test data, accuracy metrics, limitations and capabilities of the system, information presented in the user interface, and model cards [100].”

• cites [70]Not checked yet

  Not checked in the free run.

• cites [100]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Continuous daily interactions allow end users to quickly adjust their weighting and utilization of cues.

“Presumably, continuous daily interactions allow end users to quickly adjust their weighting and utilization of cues.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

System designers' individual standards for a trustworthy system might include stable performance, specific requirements in the labeling process, higher performance than the current state-of-the-art ADM system on a benchmark data set, and robustness.

“Designers' individual standards for a trustworthy system might include stable performance [39,48,110], combined with specific requirements in the labeling process, a higher performance than the current state-of-the-art ADM system on a benchmark data set [105,122], and robustness.”

• cites [39]Not checked yet

  Not checked in the free run.

• cites [48]Not checked yet

  Not checked in the free run.

• cites [110]Not checked yet

  Not checked in the free run.

• cites [105]Not checked yet

  Not checked in the free run.

• cites [122]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Users may get better or worse at detecting and utilizing cues.

“Users may get better (e.g., due to more knowledge and experience) or worse (e.g., due to habituation and less awareness) at detecting and utilizing cues.”

Probably not a substantial claim: may not pass S4 (based).

End users may provide secondary cues to other end users.

“End users may provide secondary cues to other end users.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Market approval for ADM systems, especially of those used in high-risk contexts, is likely to be granted only if the system meets a list of requirements consistent with standards agreed upon within a particular institutionalized community of values.

“At least for the foreseeable future, market approval for ADM systems, especially of those used in high-risk contexts, is likely to be granted only if the system meets a list of requirements consistent with standards agreed upon within a particular institutionalized community of values.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The areas in the circle reflect the result of the utilization of the detected cues.

“Note that the areas in the circle reflect the result of the utilization of the detected cues.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The order of cues may matter; one cue may undermine or defeat other cues, and cues may mutually reinforce or enhance each other's relevance.

“Furthermore, the order of cues may matter, one cue may undermine or defeat other cues, and cues may mutually reinforce or enhance each other's relevance.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The trustor's propensity to trust can affect the position of the cone.

“The trustor's propensity to trust can affect the position of the cone.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If the likelihood or strength of expected negative outcomes increases, the dashed line moves clockwise, making it less likely that the trustor will actually engage in trusting behavior.

“If the likelihood or strength of expected negative outcomes increases, this could be represented by moving the dashed line clockwise, making it less likely that the trustor will actually engage in trusting behavior”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

As the system is assessed, the area indicating the presence of trustworthiness begins to fill the circle clockwise at 12 o'clock, followed by the area indicating the absence of trustworthiness.

“As the system is assessed, the area indicating the presence of trustworthiness (light grey; checked boxes) begins to fill the circle clockwise at 12 o'clock, followed by the area indicating the absence of trustworthiness (dark grey; crossed out boxes).”

• AssumptionThe authors state this as an assumption: Prior to all assessments, the circle is completely filled with uncertainty.

May combine two claims.

If perceptions indicating the presence of a system's trustworthiness do not fill enough space to reach the lower bound of the cone, the trustor will be unwilling to be vulnerable to the trustee, and thus will not trust.

“If perceptions indicating the presence of a system's trustworthiness do not fill enough space to reach the lower bound of the cone, the trustor will be unwilling to be vulnerable to the trustee, and thus will not trust (Figure 6C).”

Classified as probably not a substantial claim.

The weights of the cues reflect the subjective importance with which a cue influences PT.

“The weights of the cues reflect the subjective importance with which a cue influences PT.”

Classified as probably not a substantial claim.

Calibrated trust has been defined as a match of a system's trustworthiness and a trustor's trust in the system.

“Calibrated trust has been defined as a match of a system's trustworthiness and a trustor's trust in the system (e.g., [80,82]).”

• cites [80]Not checked yet

  Not checked in the free run.

• cites [82]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

As the areas indicating the presence or absence of trustworthiness become larger, uncertainty about the trustee decreases.

“As the areas indicating the presence or absence of trustworthiness become larger, uncertainty about the trustee decreases.”

Classified as probably not a substantial claim.

As the uncertainty and perceived absence of trustworthiness about the trustee increase, it becomes less likely that trustor will be willing to make themselves vulnerable to the trustee.

“As the uncertainty and perceived absence of trustworthiness about the trustee increase, it becomes less likely that trustor will be willing to make themselves vulnerable to the trustee.”

Classified as probably not a substantial claim.

During utilization, the trustor weights the detected cues, which then influence perceptions regarding the presence or absence of trustworthiness.

“During utilization, the trustor weights the detected cues, which then influence perceptions regarding the presence or absence of trustworthiness.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The use of cues is a nonlinear, complex matter.

“The details of this function (e.g., which cues are more important than others, how cues are combined) are ultimately empirical questions, but we suggest that the use of cues is a nonlinear, complex matter.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Red flag cues could strongly increase the perceived absence of trustworthiness while decreasing the proportion of perceived presence of trustworthiness.

“For example, there may be red flag cues that could strongly increase the perceived absence of trustworthiness while decreasing the proportion of perceived presence of trustworthiness.”

Probably not a substantial claim: may not pass S4 (based).

As the area indicating the presence of trustworthiness becomes larger, it becomes more likely that the trustor is willing to accept the remaining uncertainty and any perceived absence of trustworthiness.

“As the area indicating the presence of trustworthiness becomes larger, it becomes more likely that the trustor is willing to accept the remaining uncertainty and any perceived absence of trustworthiness.”

Probably not a substantial claim: may not pass S2 (checkable).

The accuracy of the trustworthiness assessments has strong implications for whether trust and potential trusting behavior are well-informed and based on adequate expectations given the trustor's individual standards.

“Given the relationships described in the previous paragraphs, the accuracy of the trustworthiness assessments has strong implications for whether trust and potential trusting behavior are well-informed and based on adequate expectations given the trustor's individual standards.”

Probably not a substantial claim: may not pass S4 (based).

The degree of accuracy is the result of the comparison between a trustor's PT and a system's AT.

“The degree of accuracy is the result of the comparison between a trustor's PT and a system's AT.”

Probably not a substantial claim: may not pass S4 (based).

The proportions of accurate and inaccurate assessments reflect how accurately a trustor's PT reflects the system's AT.

“The proportions of accurate and inaccurate assessments reflect how accurately a trustor's PT reflects the system's AT.”

Classified as probably not a substantial claim. May combine two claims.

If the area representing the perceived presence of trustworthiness passes the lower bound of the cone, the trustor is willing to make themselves vulnerable and is in a position to trust.

“That means if the area of the circle that is filled with perceptions indicating the presence of trustworthiness passes the lower bound of the cone, the trustor is willing to make themselves vulnerable -the trustor is in a position to trust.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Calibrated trust has received much attention, but also increasing criticism, due to the lack of a clear definition and a clear way to measure it.

“This concept has received much attention, but also increasing criticism, due to the lack of a clear definition and a clear way to measure it (for a recent review, see [144]).”

• cites [144]Not checked yet

  Not checked in the free run.

May combine two claims.

The TrAM complements existing trust models by adding the trustee and its characteristics, specifying the concepts relevant to the trustworthiness assessment, describing their relations, and emphasizing the role of PT in the overall trust development process.

“In this paper, we describe the TrAM, which complements existing trust models (e.g., [82,91]) by adding the trustee (and its characteristics), specifying the concepts relevant to the trustworthiness assessment (system characteristics, individual standards, AT, and PT), describing their relations, and emphasizing the role of PT in the overall trust development process.”

• cites [82]Not checked yet

  Not checked in the free run.

• cites [91]Not checked yet

  Not checked in the free run.

May combine two claims.

If a single cue in the trustworthiness propagation process is irrelevant, based on an inaccurate assessment, reflects intentional inaccuracy, or if the stakeholder who produced it has individual standards inconsistent with other stakeholders, the trustworthiness assessment of a large number of stakeholders may become inaccurate.

“If this cue is irrelevant, based on an inaccurate assessment, reflects intentional inaccuracy (i.e., misinformation), or if the stakeholder who has produced this cue has individual standards that are not consistent with those of the rest of the stakeholders, the trustworthiness assessment of a large number of stakeholders may become inaccurate.”

May combine two claims.

Environmental changes may affect the actual trustworthiness of a system, and this may also affect the relevance of a cue.

“For instance, considering that environmental changes may affect the AT of a system, this may also affect the relevance of a cue [55].”

• cites [55]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

An inaccurate trustworthiness assessment could erroneously increase or decrease perceived trustworthiness and thereby increase the likelihood of expectation violation or perceived betrayal of the trustee.

“An inaccurate trustworthiness assessment could erroneously increase or decrease the perceived trustworthiness and thereby increase the likelihood of expectation violation or even perceived betrayal of the trustee.”

Probably not a substantial claim: may not pass S4 (based).

Factors that influence the accuracy of the trustworthiness assessment and the uncertainty about the trustee are relevance and availability of cues on the system side, and detection and utilization of cues on the trustor's side.

“Specifically, factors that influence the accuracy of the trustworthiness assessment and the uncertainty about the trustee are relevance and availability of cues on the system side, and detection and utilization of cues on the trustor's side.”

Classified as probably not a substantial claim.

Three factors could be calibrated to achieve an optimal human-system interaction: the accuracy of the trustworthiness assessment, the uncertainty about the trustee, and the trustor's threshold for their willingness to be vulnerable.

“In Figure 6, we have identified three factors that could be calibrated to achieve an”

Probably not a substantial claim: may not pass S2 (checkable).

A highly accurate trustworthiness assessment provides a solid basis for well-informed and appropriate decisions in further stages of the trust development process.

“A highly accurate trustworthiness assessment then provides a solid basis for well-informed and appropriate decisions in further stages of the trust development process (e.g., well-informed decisions about whether to engage in trusting behavior).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Making relevant cues may determine the accuracy of trustors' trustworthiness assessment.

“With respect to the trust development process, making relevant cues may determine the accuracy of their trustworthiness assessment.”

Probably not a substantial claim: may not pass S2 (checkable).

The accuracy of the trustworthiness assessment is a prerequisite to whatever calibrated trust is.

“We propose that the accuracy of the trustworthiness assessment is a prerequisite to whatever calibrated trust is.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

A strong match between AT and PT can more simply be described as a high accuracy of the trustworthiness assessment.

“We propose that a strong match between AT and PT can more simply be described as a high accuracy of the trustworthiness assessment.”

May combine two claims.

Trustors who have clear and explicit individual standards may be better able to compare different systems, as well as compare their own standards to a system's actual trustworthiness.

“Trustors who have clear and explicit individual standards, may be better able to compare different systems, as well as compare their own standards to system's AT.”

May combine two claims.

It may be more appropriate to assume that AT is assessed and leads to PT, and that PT then affects whether people will be in a position to trust and to engage in trusting behavior.

“It may be more appropriate to assume that AT is assessed and leads to PT, and that PT then affects whether people will be in a position to trust and to engage in trusting behavior.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Knowing which cues are relevant would enable the design of systems tailored to the trustor's individual standards for trustworthiness.

“Knowing which cues are relevant would enable the design of systems that are tailored to the trustor's individual standards for trustworthiness.”

Probably not a substantial claim: may not pass S4 (based).

The micro-level trustworthiness assessment process implies that, in order to achieve an accurate assessment of a system's actual trustworthiness, relevant cues must be made available on the system's side.

“The micro-level trustworthiness assessment process implies that, in order to achieve an accurate assessment of a system's AT, relevant cues must be made available on the system's side.”

May combine two claims.

Individual standards for trustworthiness may differ between stakeholders, and if these differences are not explicit, incorrect assumptions about cues may lead to inadequate utilization and thus low accuracy in the assessment of actual trustworthiness.

“Individual standards for trustworthiness may differ between the stakeholders (see also [70]), and if these differences are not explicit, incorrect assumptions about cues may lead to an inadequate utilization and thus low accuracy in the assessment of AT.”

• cites [70]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Power imbalance, both interinstitutional and interpersonal, may be an important factor in anticipating the trustworthiness assessment process.

“What might be an important factor in anticipating the trustworthiness assessment process is power imbalance -interinstitutional and interpersonal.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

An interinstitutional power imbalance might occur between deployers and users of the system, or between earlier trustors and later trustors within the trustworthiness propagation process.

“An interinstitutional power imbalance might occur between deployers and users of the system, or between earlier trustors and later trustors within the trustworthiness propagation process.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The accuracy of the trustworthiness assessment may increase when different trustors providing secondary cues make their individual standards explicit and available for scrutiny by others.

“So, it may be possible that the accuracy of the trustworthiness assessment increases when different trustors providing secondary cues make their individual standards explicit and available for scrutiny by others.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Knowing which cues are frequently utilized and heavily weighted by users to assess system trustworthiness can influence system design by adapting cues to make them more adequately utilized.

“Knowing which cues are frequently utilized and heavily weighted by users to assess system trustworthiness can further influence system design by adapting cues to make them more adequately utilized (e.g., decreasing saliency of cues that are frequently utilized incorrectly) [30].”

• cites [30]Not checked yet

  Not checked in the free run.

May combine two claims.

Given the relative nature of trustworthiness, trustors who see a trustworthy AI label may mistakenly assume that their individual standards are met, and may not examine the system in detail, which could lead to an inaccurate trustworthiness assessment.

“Given the relative nature of trustworthiness, trustors who see this label may mistakenly assume that their individual standards are met, and may not examine the system in detail [118], which could lead to an inaccurate trustworthiness assessment.”

• cites [118]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

In hierarchical work environments, supervisors may give less weight to, or not even detect, cues provided by employees using new systems.

“Conversely, in hierarchical work environments, supervisors may give less weight to, or not even detect, cues provided by employees using new systems.”

Probably not a substantial claim: may not pass S2 (checkable).

Communication about trustworthy systems may need to emphasize the relative and individual nature of trustworthiness.

“A final implication of our model is that communication about”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Cues provided by supervisors or higher-level management may be weighted more heavily by employees.

“Cues provided by supervisors or higher-level management may be weighted more heavily by employees.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Interpersonal power imbalances may arise in hierarchical work environments.

“Interpersonal power imbalances may arise in hierarchical work environments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Individual standards may affect trustors' detection of cues, as they may influence what trustors are looking for in a system.

“Individual standards may also affect trustors' detection of cues, as they may influence what trustors are looking for in a system.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Trustors may initially not be able to communicate their individual standards, or may not be aware of the cues they have used in assessing a system's trustworthiness.

“For instance, trustors may initially not be able to communicate their individual standards, or may not be aware of the cues they have used in assessing a system's trustworthiness.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Previously proposed trustworthiness facets may reflect issues that are important to all trustors' individual standards.

“For example, previously proposed trustworthiness facets may reflect issues that are important to all trustors' individual standards.”

Classified as probably not a substantial claim. May combine two claims.

Explainable AI research develops transparent models where rules provide relevant cues, or posthoc explainability approaches that try to generate relevant cues by analyzing the decision processes of a black-box model.

“For example, XAI research develops transparent models, e.g., models based on simple rules, where these rules provide relevant cues, or posthoc explainability approaches that try to generate relevant cues by analyzing the decision processes of a black-box model [11,67,86,99,145].”

• cites [11]Not checked yet

  Not checked in the free run.

• cites [67]Not checked yet

  Not checked in the free run.

• cites [86]Not checked yet

  Not checked in the free run.

• cites [99]Not checked yet

  Not checked in the free run.

• cites [145]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Individual standards are not arbitrary but relate to a common understanding of trustworthiness.

“It is important to emphasize that we do not expect these standards to be arbitrary, but rather expect them to relate to a common understanding of trustworthiness.”

Probably not a substantial claim: may not pass S2 (checkable).

In cases where accurate trustworthiness assessments are critical, it will be necessary to simultaneously optimize for the availability of relevant cues and for system performance.

“This discussion also implies that in cases where accurate trustworthiness assessments are critical, it will be necessary to simultaneously optimize for the availability of relevant cues and for system performance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Labels such as trustworthy AI labels can help people quickly assess whether a product meets their individual standards, a process that would otherwise require a resource-intensive trustworthiness assessment process.

“We are not saying that such labels and labels are always misleading -they can help people to quickly assess whether a product meets their individual standards, a process that would otherwise require a resource-intensive (trustworthiness) assessment process.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Calling a label trustworthy AI may lead to different expectations among different trustors and should therefore be accompanied by a summary of the individual standards that were relevant to a particular institution that assessed the trustworthiness of a system.

“However, we do say that calling a label”

Classified as probably not a substantial claim.

Explainable AI research aims to make relevant cues available.

“In the terminology of our model, this research aims to make relevant cues available [78].”

• cites [78]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

There is overlap in individual standards across trustors, but the exact combination of features, as well as their weight in forming perceived trustworthiness, may be idiosyncratic.

“Accordingly, we expect overlap in the individual standards across trustors, but also that the exact combination of features, as well as their weight in forming PT, may be idiosyncratic.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The suggestion that relevance and availability of cues are central to the accurate assessment of system trustworthiness provides an argument for the importance of research in explainable AI.

“The suggestion that relevance and availability of cues are central to the accurate assessment of system trustworthiness also provides an argument for the importance of research in the area of explainable AI (XAI).”

May combine two claims.

Individual standards of system trustworthiness differ across trustors and are task- and context-specific.

“Second, we claim that individual standards of system trustworthiness differ across trustors and are task-and context-specific.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

An accurate assessment of system trustworthiness is a prerequisite for adequate and informed trust and trusting behavior.

“An accurate assessment of system trustworthiness is therefore a prerequisite for adequate and informed trust and trusting behavior.”

Classified as probably not a substantial claim.

The Trustworthiness Assessment Model extends existing trust models by adding the trustee and its characteristics.

“In doing so, we explicate a process that focuses on trustworthiness as the basis for trust, and a model that extends existing trust models by adding the trustee and its characteristics.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The model makes assumptions about theoretical processes and concepts that are currently difficult to capture and that may not be readily accessible to the stakeholders involved.

“First, the model makes assumptions about theoretical processes and concepts that are currently difficult to capture and that may not be readily accessible to the stakeholders involved.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Each concept in the model requires empirical evaluation and interdisciplinary theoretical refinement.

“Each concept in our model requires empirical evaluation and interdisciplinary theoretical refinement -something we hope will happen in response to the ideas we present in this paper.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The safe and effective use of ADM systems requires an accurate assessment of the system, its strengths and weaknesses against a trustor's goals and standards.

“The safe and effective use of ADM systems requires an accurate assessment of the system, its strengths and weaknesses against a trustor's goals and standards.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The authors do not expect trustors to always explicitly engage in cue detection and utilization, nor may they be motivated to engage in such a process.

“We do not expect trustors to always explicitly engage in cue detection and utilization, nor may they be motivated to engage in such a process.”

May combine two claims.

On the trustor's side, factors that influence the accuracy of trustworthiness assessments include the trustors' individual standards and the detection and utilization of cues.

“On the trustor's side, factors that influence the accuracy of trustworthiness assessments include the trustors' individual standards and the detection and utilization of cues.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Effective human oversight requires an accurate trustworthiness assessment.

“Thus, effective human oversight requires an accurate trustworthiness assessment.”

Probably not a substantial claim: may not pass S2 (checkable).

If we want effective human oversight, we need to focus on making relevant cues available.

“In line with other research [43], our model suggests that if we want effective human oversight, we need to focus on making relevant cues available.”

• cites [43]Not checked yet

  Not checked in the free run.