Decision Provenance: Harnessing Data Flow for Accountable Systems

May combine two claims.

Decision provenance can help improve accountability by assisting oversight, empowerment, investigation, audit, and compliance.

“As a result, decision provenance can help improve accountability by assisting oversight, empowerment, investigation, audit, and compliance.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Transparency is important for holding those responsible for systems to account, as it enables identification, audit, and oversight.

“Transparency is important for holding those responsible for such systems to account, as it enables identification, audit, and oversight.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Decision provenance will complement and support ongoing work focusing on accountability as it regards algorithmic decision-making.

“We further outline the role of decision provenance in relation to ML as an illustrative case study, arguing that it will complement and support the ongoing work focusing on accountability as it regards algorithmic decision-making.”

Probably not a substantial claim: may not pass S2 (checkable).

Technology will not itself solve accountability issues.

“Technology will not itself 'solve' accountability issues”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

More information regarding the nature of systems could allow those building and deploying systems to improve their quality and identify points of failure, while empowering others by offering more insight into systems and data beyond their direct control.

“Generally speaking, more information regarding the nature of systems could allow those building and deploying systems to improve their quality and to identify points of failure, while empowering others by offering more insight into the nature of the systems and data that is beyond their direct control [4], [22], [23].”

• cites [4]Not checked yet

  Not checked in the free run.

• cites [22] sourceNot checked yet

  Not checked in the free run.

• cites [23] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Technical measures can assist in making systems more transparent, improving the accountability of responsible organisations and individuals, and assisting them in meeting their obligations.

“Technical measures can assist in making systems more transparent, improving the accountability of the organisations and individuals responsible for them, and assisting them in meeting their obligations.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Data provenance methods show much promise as a technical means for increasing the transparency of interconnected systems.

“This paper argues that data provenance methods show much promise as a technical means for increasing the transparency of these interconnected systems.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Decision provenance can help facilitate oversight, audit, compliance, risk mitigation, and user empowerment in algorithmic systems.

“We argue that decision provenance can help facilitate oversight, audit, compliance, risk mitigation, and user empowerment, and we also indicate the implementation considerations and areas for research necessary for realizing its vision.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Considerations of data flow and systems more broadly are important to discussions of accountability and complement attention to algorithmic specifics.

“More generally, we make the case that considerations of data flow, and systems more broadly, are important to discussions of accountability, and complement the considerable attention already given to algorithmic specifics.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Decision provenance can aid accountability by assisting regulatory oversight and technical investigation, facilitating compliance and recourse, increasing user agency, and contributing to better system design, operational management, and risk mitigation.

“In this paper, we explore how this aids accountability through assisting in regulatory oversight and technical investigation, facilitating compliance and recourse, increasing user agency, and generally contributing to better system design, operational (run-time) management, and risk mitigation.”

Probably not a substantial claim: may not pass S2 (checkable).

Greater visibility over interconnections and assemblages of systems is important for increasing accountability for them.

“From the above, we can see that greater visibility over the interrconnections and assemblages of systems is important for increasing accountability for them.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A highly interconnected context poses particular accountability challenges which to date have been underexplored.

“However, a highly interconnected context poses particular accountability challenges which to date have been underexplored.”

Probably not a substantial claim: may not pass S2 (checkable).

Decision provenance helps expose decision pipelines to make visible the nature of inputs to and cascading consequences of any decision or action, alongside entities involved, systems-wide.

“In this way, decision provenance helps expose the decision pipelines in order to make visible the nature of the inputs to and cascading consequences of any decision or action (at design/ or run-time), alongside the entities involved, systems-wide (e.g.see Fig. 2).”

• AssumptionRests on an assumption: The use of data provenance methods generally for systems-related accountability concerns has yet to be considered in depth.

Probably not a substantial claim: may not pass S4 (based).

The lack of visibility over data flows in interconnected systems poses various challenges from a legal point of view.

“Likewise, the lack of visibility over data flows in interconnected systems poses various challenges from a legal point of view.”

• AssumptionRests on an assumption: Visibility over the nature and flow of data is lost once it moves beyond a technical or administrative boundary.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The complexity of interconnected environments poses significant challenges for accountability.

“The complexity of such interconnected environments poses significant challenges for accountability.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

At a technical level, accountability is grounded in transparency and control.

“It is said that at a technical level, accountability is grounded in transparency and control [4], [13].”

• cites [4]Not checked yet

  Not checked in the free run.

• cites [13] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

There is a clear need for provenance methods for accountability concerns in interconnected systems.

“Towards this, there appears a clear need for provenance methods, which concern recording information about the nature and flow of data and the contexts in which it is processed; however, their use for accountability concerns in such contexts have had little consideration.”

• AssumptionRests on an assumption: Data flows that drive interconnected systems are often invisible or opaque.

Probably not a substantial claim: may not pass S4 (based).

Technical transparency might not itself address legal accountability concerns.

“technical transparency might not itself address legal accountability concerns”

Probably not a substantial claim: may not pass S2 (checkable).

There is real potential in applying data provenance methods as a means of working towards greater visibility over interconnections and assemblages of systems.

“Given that data drives systems, there appears to be real potential in applying data provenance methods as a means of working towards this.”

• AssumptionRests on an assumption: The use of data provenance methods generally for systems-related accountability concerns has yet to be considered in depth.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

In interconnected systems, data flow drives everything, integrating systems and bringing their functionality.

“Importantly, in these interconnected systems data flow drives everything, in that the flow of data between, through, and across systems works to integrate them and brings their functionality [22].”

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Transparency providing knowledge of the nature of systems can help in evaluating correctness, bias, fairness, and whether systems operate within expected parameters.

“transparency providing knowledge of the nature of systems can help in evaluating those aspects for correctness, bias, fairness, and whether they are operating within the parameters that are expected or desired by their designers, operators, or end-users”

Classified as probably not a substantial claim.

Technology can provide tooling and information to assist and facilitate accountability.

“however, technology can provide the tooling and information to assist and facilitate accountability, for example, by providing 'evidence' indicating what is happening, or by enabling constraints, responses, and rectifications.”

Probably not a substantial claim: may not pass S3 (load-bearing).

An interconnected system-of-systems poses practical implementation challenges given the requirement for the provenance regime to operate consistently throughout.

“An interconnected system-of-systems -with its many moving parts -poses practical implementation challenges, given the requirement for the provenance regime to operate consistently throughout, as we now explore.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Proactive measures could allow users to set policy that constrains particular information flows or keep users informed of changes in circumstance.

“That said, proactive measures could also be employed to allow users to set policy that constrain particular information flows [22], or keep users informed of any change in circumstance.”

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Measures that help users make better-informed decisions about which systems they engage with may incentivise technology producers to improve their practices and align offerings with user expectations.

“Measures that help users make better-informed decisions about which systems they engage with may also work to incentivise those producing and operating technology to improve their practices at all steps, and to help better align their offerings with user expectations and preferences.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

If data being passed into a live model comes from a recognisably different source than the data used to train the model, actions could be taken to prevent such processing.

“For example, if data being passed into a live model comes from a recognisably different source (sampling distribution) than the data that was used to train the model, actions could be taken to prevent such processing.”

May combine two claims.

Provenance will not solve all accountability challenges, but it assists by providing visibility in complex environments by exposing relations and dependencies where such visibility may not otherwise exist.

“Naturally, provenance will not solve all accountability challenges, but it assists by providing visibility in complex environments by exposing the relations and dependencies where such visibility may not otherwise exist.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Knowledge of the nature of data flowing in, out and within a system can help systems designers and operators monitor, maintain and improve the quality of their systems on an ongoing basis.

“Generally, knowledge of the nature of the data flowing in, out and within a system, in terms of where it has come from, where it goes, how it has and is being used, as well as the relevant environmental and computational contexts, can help systems designers and operators (manually or through automatic means) monitor, maintain and improve the quality of their systems on an ongoing basis.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The increasing popularity of ML risks non-domain experts unintentionally misusing datasets that are unsuitable in particular contexts.

“the increasing popularity of ML risks those who are not domain experts unintentionally misusing datasets which are unsuitable in particular contexts [38]”

• cites [38] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

For provenance to enable accountability, the consistency, visibility, and veracity of provenance information across a federated environment are crucial.

“For provenance to enable accountability, crucial is the consistency, visibility, and veracity of provenance information across this federated environment.”

Classified as probably not a substantial claim.

Decision provenance can offer preemptive advantages by allowing organisations greater control over how their data and services are subsequently used.

“In this way, decision provenance can offer preemptive advantages, by allowing organisations to have greater control over how their data (that for which they have responsibilities) and services are subsequently used.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In an interconnected context, a training dataset may comprise data from a range of sources, including sensors, human input, system logs, data brokers, and outputs from calculations, analytics, or other ML models.

“In an interconnected context, a training dataset may comprise data from a range of sources, including sensors, human input, system logs, data brokers, and so on, and can also include outputs from calculations, analytics, or indeed, other ML models ( §III).”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Evidence of prior behaviour does not necessarily imply future behaviour will remain the same, but it can be indicative.

“Of course, evidence of prior behaviour does not necessarily imply future behaviour will remain the same; however it can be indicative.”

May combine two claims.

Exposing the decision pipeline can help identify steps behind leakage of personal or sensitive data and track cascading consequences of a particular event.

“Exposing the decision pipeline can help in identifying aspects such as the steps behind the leakage of personal or otherwise sensitive data, and in tracking the cascading consequences of a particular (perhaps erroneous) event.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One categorisation of provenance mechanisms entails two categories: disclosed and observed provenance.

“One categorisation of provenance mechanisms entails two categories [24]: disclosed and observed provenance.”

• cites [24] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

Enhanced means for governance could encourage collaboration and help tackle data silos.

“Indeed, enhanced means for governance could encourage collaboration, and help in tackling data silos.”

Probably not a substantial claim: may not pass S2 (checkable).

In practice, exposing the decision pipeline will require several complementary capture regimes.

“In practice, it is likely that exposing the decision pipeline will require several complementary capture regimes.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Decision provenance methods are useful for organisations to help manage their own data supply chains.

“Beyond regulators and oversight bodies, the same methods are useful for organisations to help in managing their own data 'supply chains'.”

May combine two claims.

Where harm is caused by a failure in complex systems, decision provenance can help identify which system caused the harm, thereby helping identify the responsible and potentially liable organisation.

“Where harm is caused by a failure, particularly in complex systems environments, decision provenance can help identify which system caused that harm, thereby helping to identify the organisation responsible and potentially liable for that harm and assisting in holding them to account.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Provenance information could inform model selection, where the lineage of data determines at run-time the appropriate model on which the data is applied.

“As an example, provenance information could inform model selection: where the lineage of data determines (at runtime) the appropriate model on which the data is applied.”

Probably not a substantial claim: may not pass S2 (checkable).

The use of decision provenance techniques has much potential to aid the auditing and oversight activities of regulators.

“The use of decision provenance techniques has much potential to aid the auditing and oversight activities of regulators, as provenance involves generating detailed information for assisting oversight that was previously unavailable.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The importance of technical audit and investigation concerns will only increase as IoT and smart cities become a reality.

“The importance of such concerns will only increase as the visions of the IoT and smart cities become a reality, due to their complex and highly-interconnected nature and also their potential for emergent functionality and real-time reconfiguration [4], [22].”

• cites [4]Not checked yet

  Not checked in the free run.

• cites [22] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Decision provenance complements ML-oriented work on algorithmic accountability by providing auditors with information of the wider practical context in which automated decision making processes were defined, built and operate, as a means for indicating where deeper investigations can take place.

“As such, decision provenance complements the ML-oriented work on 'algorithmic accountability' (i.e. that focused on the decision-making elements), by providing auditors with information of the wider, practical context in which automated decision making processes were defined, built and operate, as a means for indicating where deeper (model-centric or other) investigations can then take place.”

Probably not a substantial claim: may not pass S2 (checkable).

Measures driven by provenance information could allow users to take action, such as exercise rights, complain to a regulator, or stop using the service.

“Such measures, driven by provenance information, could allow users to take action, e.g.exercise their rights, complain to a regulator, stop using the service.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The potential for provenance methods to allow one to see across systems and their interconnections can enable proactive steps to address problems before they arise.

“The potential for provenance methods to allow one to 'see' across systems and their interconnections can also enable proactive steps to be taken in order to address problems before they arise.”

May combine two claims.

Provenance can show the nature of training datasets; revealing the pipeline can allow greater knowledge of the sources of data and assist in assessing their veracity.

“Provenance has a clear role to play in showing the nature of the training datasets, in that revealing the pipeline can allow for greater knowledge of the sources of data, assisting in assessing their veracity.”

Classified as probably not a substantial claim.

Decision provenance helps investigate complex and interconnected systems by exposing the data flow and interconnections leading up to a decision or action, and the entities involved.

“As §IV describes, decision provenance helps investigate complex and interconnected systems by exposing the data flow and interconnections leading up to a decision or action, and the entities involved.”

Classified as probably not a substantial claim.

Once the decision pipeline has been used to identify the source or sources potentially related to an issue, auditors can narrow their focus and go into detail.

“Once the decision pipeline has been used to identify the particular source(s) (technical components and/ or legal entities) potentially related to an issue, auditors can then narrow their focus and go into detail.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Observed provenance captures information by observing what occurs, for example by embedding the capture mechanism into the platform or infrastructure to capture data flows regarding applications running within that environment.

“Observed provenance captures information by observing what occurs; for example, by embedding the capture mechanism into the platform/infrastructure (e.g. an operating system (OS)), to capture the data flows regarding the applications running within that environment [41]- [43].”

• cites [41]- [43]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

If a user could see in advance that giving data to a system could result in undesired decisions or flow to an undesired entity, this helps the user make a more informed choice about whether to use that system.

“If a user could see in advance that giving data to a particular system could, for example, (a) result in certain decisions being made that have particular undesired consequences; or (b) flow to an undesired entity (such as a certain advertisement network), then this helps that user to make a more informed choice about whether to use (allow their data to flow into) that system.”

Classified as probably not a substantial claim.

Knowledge of the nature of data flow between parties could make it possible to ensure and validate that data is processed compatibly with the contract governing that data flow or processing relationship.

“For instance, where contractual obligations exist between entities, knowledge of the nature of data flow between parties could make it possible to ensure and validate that data is processed in a way compatible with the contract governing that data flow or processing relationship.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Many open challenges for decision provenance operating between systems and across boundaries stem from the scale, federation, and complexity of wide-scale distributed systems that encompass a range of technologies and organisations with different and possibly competing incentives.

“In terms of decision provenance operating between systems and across boundaries, many of the open challenges predominately stem from the scale, federation, and complexity of what are effectively wide-scale distributed systems that can encompass a range of technologies, and also a number of organisations with different and possibly competing incentives.”

Probably not a substantial claim: may not pass S2 (checkable).

Decision provenance information could help users make better-informed decisions about the services they use.

“Decision provenance information could help users make better-informed decisions about the services that they use.”

Probably not a substantial claim: may not pass S3 (load-bearing).

It is increasingly recognised that ML practitioners should record information relating to datasets, such as motivation for creation, composition, collection, preprocessing, distribution, maintenance, and legal and ethical considerations.

“It is increasingly recognised the importance for the ML practitioners, themselves, to record information relating to the datasets, such as their motivation for creation, composition, process of collection, preprocessing, distribution, maintenance, and legal and ethical considerations [38].”

• cites [38] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Rules could prevent a data subject's personal data being presented to an ML model at run-time unless they have chosen to allow solely automated decision making.

“Further, rules could, for example, prevent a data subject's personal data being presented to a ML model -at run-time -unless they have chosen to allow solely automated decision making 20”

Probably not a substantial claim: may not pass S2 (checkable).

Decision provenance can provide much information, perhaps complementing other records, for data and model assessment.

“In this way, decision provenance can provide much information, perhaps complementing other records, for data and model assessment.”

May combine two claims.

A general observed capture mechanism is comparatively less-targeted and raises serious overhead considerations given the propensity for the volume of recorded information to be extremely large and complex.

“However, a general observed capture mechanism is comparatively less-targeted, and raises serious overhead considerations given the propensity for the volume of the recorded information to be extremely large and complex [44].”

• cites [44]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Decision provenance involves technical means for collecting information relating to data and models, potentially including the computational contexts in which models were constructed, throughout a data supply chain.

“decision provenance involves technical means for collecting information relating to data and models, potentially including the computational contexts in which models were constructed, right throughout a data supply chain.”

Classified as probably not a substantial claim. May combine two claims.

Although provenance has been considered for some specific compliance aspects, the broader accountability considerations have yet to be explored in depth.

“Though, as mentioned, provenance has been considered for some specific compliance aspects (see for e.g.[22], [23], [27]- [29]), the broader accountability considerations have yet to be explored in depth.”

• cites [22] sourceNot checked yet

  Not checked in the free run.

• cites [23] sourceNot checked yet

  Not checked in the free run.

• cites [27]- [29]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

By capturing the provenance and nature of model inputs and the decision itself, decision provenance could be used to detect conflicts and uphold compliance at run-time.

“By capturing the provenance and nature of the model inputs and the decision itself (i.e. the model outputs), decision provenance could be used to detect conflicts and uphold compliance at run-time.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Given that provenance information may be relevant for different audiences, there may be a need for various tools and extensions to assist.

“Given that provenance information may be relevant for different audiences, there may be a need for various tools and extensions to assist.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Decision provenance will not itself solve all the complexities of increasingly interconnected systems; rather it provides visibility over the relations and dependencies of such systems where such visibility may not otherwise exist.

“Decision provenance won't itself solve all the complexities of increasingly interconnected systems, rather it provides visibility over the relations and dependencies of such systems where such visibility may not otherwise exist.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Means for increasing accountability in a systems-of-systems context will only increase in importance as visions of pervasive computing such as smart cities, the Internet of Things, and autonomous transport systems become a reality.

“Indeed, means for increasing accountability in a systems-ofsystems context will only increase in importance as visions of pervasive computing: smart cities, the Internet of Things, autonomous transport systems, etc., become a reality.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Exposing information of the contexts and nature of data flow, and the broader decision pipeline, is important, relevant and complements other accountability work that focuses on the decision-making elements.

“In all, our wider goal is to highlight and raise awareness that exposing information of the contexts and nature of data flow, and the broader decision pipeline, is important, relevant and complements other accountability work that focuses on the decision-making elements (e.g.ML models).”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In a systems context, it is important that there is a common representation of provenance data so that it can be widely understood and interoperate system-wide.

“In a systems context, it is important that there is a common representation of provenance data so that it can be widely understood and interoperate system-wide.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Since data-driven assemblages tend to be opaque, there is a pressing need to expose the nature and flow of data leading to, and resulting from, a decision or action to help raise accountability in interconnected environments.

“Since these data-driven assemblages tend to be opaque, there is a pressing need to expose the nature and flow of data leading to, and resulting from, a decision or action to help raise levels accountability in interconnected environments.”

• AssumptionRests on an assumption: Data-driven assemblages tend to be opaque.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The W3C PROV standard provides an extensible mechanism for modelling provenance data, while ontologies provide the means for describing what has been captured.

“The W3C PROV 25 standard provides an extensible 26 mechanism for modelling provenance data, while ontologies (description vocabularies) provide the means for describing what has been captured.”

May combine two claims.

The security of and access to provenance data raises accountability questions, given that it may itself be sensitive or come with various legal obligations, particularly in a cross-organisational context.

“Further, the security of, and access to the provenance data itself raises accountability questions, given that, as information about data, systems and processing, it may itself be sensitive [23], or come with various legal obligations -particularly in a cross-organisational context.”

• cites [23] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Data retention considerations are particularly important if provenance data serves as evidence supporting investigations or other accountability processes.

“Data retention considerations are particularly important if serving as 'evidence' supporting investigations or other accountability processes.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Provenance data captured may be difficult to interpret for some, for instance non-technical users, and those seeking to interact with such data may not be familiar with the nature of the systems involved.

“This means that the provenance data captured may be difficult to interpret for some -for instance, making sense of data representing different levels of the technical stack/abstraction may be beyond the expertise of non-technical users -and, further, those seeking to interact with such data may not be familiar with the nature of the systems involved.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

There is a clear requirement for means that assure and ensure the integrity of the capture regime, recorded data, query mechanism, and other compliance mechanisms, and maintain that integrity throughout any pipeline.

“In all, there is a clear requirement for means that assure and ensure the integrity of the capture regime, recorded data, the query mechanism, and any other compliance mechanisms (such as policy enforcement regimes) that leverage provenance data, and also to ensure that this integrity is maintained throughout any pipeline [22], [23], [48].”

• cites [22] sourceNot checked yet

  Not checked in the free run.

• cites [23] sourceNot checked yet

  Not checked in the free run.

• cites [48]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

For provenance information to be useful, it must be reliable.

“For provenance information to be useful, it must be reliable.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Standards, guarantees and best practices regarding capture mechanisms will be important for increasing completeness in provenance records, for reconciliation and interoperability, and for enabling provenance at scale.

“These will be important for increasing levels of completeness in provenance records, including the reconciliation and interoperability just mentioned, and more generally, to enable provenance at scale.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Concerns about interpreting provenance data are exacerbated in a large-scale systems context and are problematic where accountability is the aim.

“Such concerns are exacerbated in a large-scale systems context, and are problematic where accountability is the aim.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

There is a clear opportunity for human computer interaction research to assist in ensuring that decision provenance approaches enable a representation that assists end-users regarding their accountability concerns.

“There is a clear opportunity for human computer interaction (HCI) research to assist in ensuring that decision provenance approaches enable a representation that assists end-users regarding their accountability concerns.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Decision provenance offers much potential for assisting with issues of responsibility, technical compliance, and improved user agency in systems.

“We therefore propose decision provenance as an important piece of this accountability 'jigsaw', offering much potential for assisting with issues of responsibility, technical compliance, and improved user agency in systems.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The storage requirements and possible sensitivity of provenance data tend towards such information being federated across organisations.

“The storage requirements and possible sensitivity of provenance data tends towards such information being federated across organisations.”