From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Open accessAbstract
Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context. Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces. We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture: deterministic behavior moves into code, manifests, schemas, and validation artifacts around a replaceable composition boundary, while source-backed claims remain the authority for runtime answers. We instantiate it on a public-data slice of five Korean corporate groups (25 listed companies) and evaluate three research questions. (1) The harness preserves its source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts across the fixed validation scenarios; a fault-injection control confirms the validators flag deliberately broken contracts. (2) The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded. (3) The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely. A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility. The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper presents a harness-engineering method that moves enterprise LLM-agent behavior from prompts into code-owned manifests, source-backed claims, answer contracts, and validation traces, instantiated on a public-data slice of five Korean corporate groups (25 listed companies, 113 runtime claims). Across fixed scenarios, fault injection, model substitution, and an enforcement-layer ablation, the code-owned harness preserved source-grounding, routing, trace, hygiene, and recommendation-language contracts, while prompt-only enforcement admitted violations and a bolt-on guardrail over-refused.
Key findings
- Fixed validation: 30/30 scenarios passed all required checks; 109/109 expected source-claim references resolved; 30/30 trace, 30/30 answer, 60/60 hygiene, and 0 failed runs. Fault injection detected 7/7 deliberately broken contract dimensions, each flagged by the validator owning that dimension.
- Model substitution: across Claude Sonnet 4, GPT-4.1 mini, and Gemini 2.5 Flash over 30 scenarios × 3 repeats (270 runs), code-owned checks passed 270/270; live output was valid on first pass in 234/270, carried selected source-claim references in 252/270, and used required answer structure in 250/270, with final harness pass 198/270. Final per-model pass rates were Claude Sonnet 4 68.9% [58.7, 77.5], GPT-4.1 mini 85.6% [76.8, 91.4], and Gemini 2.5 Flash 65.6% [55.3, 74.6] (χ² = 10.57, df = 2, p = 0.0051).
- Enforcement-layer ablation: holding the model fixed (anthropic/claude-sonnet-4), prompt-only admitted 30 violations on 30 adversarial runs (15 recommendation-language, 15 internal-trace/identifier leaks), all blocked by the harness; the external guardrail blocked 28/30 adversarial runs but falsely refused 4 benign runs, lowering utility to 88/120 versus harness 120/120.
- Runtime/latency: 5/5 live-API corporate-group connectivity tests and 15/15 live answer-hygiene tests passed; enabling process-level caching and prewarming reduced mean latency from 1648.23 ms to 69.40 ms and raised runs within the 1500 ms budget from 16/30 to 30/30.
Methods
- Harness-engineering reconstruction: source manifests, evidence records, promoted source-backed claims, maintained wiki context, code-owned product rules, validation gates, and audit traces; a deterministic schema-checked composer provides the replayable baseline, with a replaceable live-LLM composition boundary and deterministic fallback.
- Data: bounded public-data slice of five Korean corporate groups (Samsung, SK, Hyundai Motor, LG, Hanwha), 25 listed companies, 113 source-backed runtime claims, and a 25-claim priority reporting layer; sources include OpenDART, issuer IR materials, KRX OPEN API, and NAVER News Search.
- Evaluation: 30 fixed validation scenarios (six per group) plus fault injection; live-LLM composition-boundary check across three hosted models, 30 scenarios × 3 repeats = 270 runs at temperature 0.2; enforcement-layer ablation over 40 scenarios × 3 repeats = 120 runs per condition (harness, prompt-only, external guardrail), paired by scenario and repeat with McNemar tests.
Limitations
- Stated: the contribution is engineering-focused and evaluates preservation of code-checkable contracts, not correctness of promoted claims or investment quality; the reference slice has uneven source and claim depth, with one group as the deepest reference case and four group extensions; the live-LLM composition-boundary check is a dated snapshot rather than bit-for-bit reproducible because hosted model identifiers and nondeterministic generation may change exact outputs.
- Stated/evident: the system is a pre-commercial research prototype on public data before customer logs, production credentials, or operational logs; expert review and deployment evidence are left to later domain-value evaluation; live-LLM provider composition latency is out of scope; the adoption ladder is a proposed planning heuristic, not derived from the experiments.
- Evident: the bounded slice (five groups, 25 companies, 113 claims) limits generalizability; adversarial scenarios use two templates per group; model substitution covers only three hosted model identifiers.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 29 cited sources checked · 0 supported · 0 flagged
- Findingown data
A fault-injection control confirms the validators flag deliberately broken contracts.
“a fault-injection control confirms the validators flag deliberately broken contracts”
Quote in the paperSubstantial claim - Findingown data
Within the bounded public-data slice, all configured source, trace, answer, and output-hygiene contracts passed.
“Within the bounded public-data slice, all configured source, trace, answer, and output-hygiene contracts passed.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
- Findingown data
The external-guardrail baseline blocks 28 of the 30 adversarial runs.
“blocks 28 of the 30 adversarial runs”
Quote in the paperSubstantial claim - Findingown data
There were 30 violations on 30 distinct runs in the prompt-only condition.
“30 violations on 30 distinct runs”
Quote in the paperSubstantial claim - Findingown data
The code-owned gate blocked every one of those violations.
“the code-owned gate blocked every one”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Findingown data
The contrast between prompt-only and harness conditions is significant under McNemar's test (p < 0.001).
“this contrast is significant under McNemar's test [69] (𝑝 < 0.001; full pairwise tests in Table 8).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [69]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Prompt-only leaked internal identifiers or trace records on the 15 leak-bait runs.
“leaked internal identifiers or trace records on the 15 leak-bait runs”
Quote in the paperSubstantial claim - Findingown data
Each mutation was flagged only by the validator that owns the broken dimension.
“each mutation was flagged only by the validator that owns the broken dimension.”
Quote in the paperSubstantial claim - Findingown data
Enabling process-level caching and prewarming raised the runs within the configured 1500ms budget from 16/30 to 30/30.
“raised the runs within the configured 1500ms budget from 16/30 to 30/30”
Quote in the paperNot a substantial claim - Findingown data
All seven mutations were detected (7/7) in the fault-injection sensitivity check.
“all seven mutations were detected (7/7)”
Quote in the paperSubstantial claim - Findingown data
The code-owned checks (trace, internal-trace leakage, source links, follow-up quality, and recommendation-language) passed on all 270 runs.
“The code-owned checks—those the harness enforces regardless of the model: trace, internal-trace leakage, source links, follow-up quality, and recommendation-language—passed on all 270 runs”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
- Findingown data
Every expected source-claim reference across the 30 fixed scenarios resolved to a promoted claim (109/109).
“The 109/109 source-claim result means that every expected claim reference resolved to a promoted claim.”
- Also stated in: § 5.2 Scenario Contract Outcomes · plain
“summed across the 30 scenarios, these expected references total 109, all of which resolved.”
Quote in the paperSubstantial claim- Rests on an assumption: The expected source-claim references listed for each fixed scenario are the correct required references for that scenario.
- Also stated in: § 5.2 Scenario Contract Outcomes · plain
- Findingown data
No scenario was stopped by a configured blocker (0 failed runs).
“Finally, the 0 failed runs indicate that no scenario was stopped by a configured blocker.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
- Findingown data
The bolt-on external-guardrail baseline blocks every admitted violation.
“The bolt-on external-guardrail baseline also blocks every admitted violation”
Quote in the paperSubstantial claim - Findingown data
A chi-square homogeneity test rejects equal pass rates across the three model identifiers (chi2 = 10.57, df = 2, p = 0.0051).
“A 𝜒 2 homogeneity test [68] on the model × final-pass/fail table then asks whether these rates differ by more than that sampling uncertainty, and rejects equal pass rates across the three model identifiers (𝜒 2 = 10.57,df = 2, 𝑝 = 0.0051).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [68]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
The fault-injection runs act as a negative control, showing that the validators detect deliberately broken source, routing, trace, answer, leakage, link, and latency conditions rather than passing everything by construction.
“the fault-injection runs act as a negative control, showing that the validators detect deliberately broken source, routing, trace, answer, leakage, link, and latency conditions rather than passing everything by construction”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Findingown data
Claude Sonnet 4's final harness-contract pass rate was 68.9% (95% Wilson CI [58.7, 77.5]).
“Claude Sonnet 4 68.9% [58.7, 77.5]”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Gemini 2.5 Flash's final harness-contract pass rate was 65.6% (95% Wilson CI [55.3, 74.6]).
“Gemini 2.5 Flash 65.6% [55.3, 74.6]”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
The external-guardrail baseline lowers utility to 88/120.
“lowering utility to 88/120”
Quote in the paperSubstantial claim - Findingown data
The code-owned checks (trace contract, development-leak absence, source-link package, follow-up quality, and recommendation-language absence) pass on all 270 runs.
“the code-owned checks (trace contract, development-leak absence, source-link package, follow-up quality, and recommendation-language absence) pass on all 270 runs”
Quote in the paperSubstantial claim - Findingown data
Enabling process-level caching and prewarming of live source retrieval reduced mean latency from 1648.23ms to 69.40ms on the 30 fixed validation scenarios.
“enabling it reduced mean latency from 1648.23ms to 69.40ms”
Quote in the paperSubstantial claim - Findingown data
The live output carried the source-claim references the harness had selected in 252/270 runs.
“carried the source-claim references the harness had selected in 252/270”
Quote in the paperNot a substantial claim - Findingown data
Every one of the 30 prompt-only violations was blocked by the harness.
“every one blocked by the harness.”
Quote in the paperSubstantial claim - Findingown data
The 72 final failures fall entirely on the model-composed checks (output contract, source-claim references, and visible answer structure).
“the 72 final failures fall entirely on the model-composed checks (output contract, source-claim references, and visible answer structure)”
Quote in the paperSubstantial claim - Findingown data
The harness preserved full utility by falling back to the deterministic composer rather than refusing.
“the harness preserved full utility by falling back to the deterministic composer rather than refusing”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Findingown data
The 6/6 outcomes per corporate group indicate that the differentiated validation roles were preserved across the reference slice.
“the 6/6 outcomes per corporate group indicate that these differentiated validation roles were preserved across the reference slice.”
Quote in the paperSubstantial claim - Findingown data
The fixed validation-scenario runs show that the expected source, routing, trace, answer-structure, and recommendation-language contracts held across the slice.
“The fixed validation-scenario runs show that the expected source, routing, trace, answer-structure, and recommendation-language contracts held across the slice”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Findingown data
The live output used the required answer structure in 250/270 runs.
“used the required answer structure in 250/270”
Quote in the paperNot a substantial claim - Findingown data
Where live output was invalid, the run recorded a fallback to the deterministic composer rather than silently accepting it.
“Where live output was invalid, the run recorded a fallback to the deterministic composer rather than silently accepting it”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Rests on an assumption: The deterministic composer produces correct or valid answers.
- Findingown data
The live output was valid on first pass in 234/270 runs.
“the live output was valid on first pass in 234/270 runs”
Quote in the paperSubstantial claim - Findingown data
The external-guardrail baseline falsely refuses 4 benign fixed-validation runs.
“it falsely refuses 4 benign fixed-validation runs”
Quote in the paperSubstantial claim - Findingown data
The answer contract held in the 15 runtime samples when live sources replaced fixtures.
“showing that the answer contract held in these samples when live sources replaced fixtures”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Rests on an assumption: The deterministic composer produces correct or valid answers.
- Rests on an assumption: The live filing, market, and news sources are accurate and authoritative.
- Findingown data
The three repeats returned the same final verdict in 82.2% of the 90 model-scenario cells.
“For repeat stability, the three repeats returned the same final verdict in 82.2% of the 90 model-scenario cells.”
Quote in the paperSubstantial claim - Findingown data
The 35 output-contract validation failures include the 2 JSON-parse failures and 33 parsed-but-invalid structured outputs.
“its 35 failures include the 2 JSON-parse failures and 33 parsed-but-invalid structured outputs”
Quote in the paperNot a substantial claim - Findingown data
Final required harness contracts passed in 198 of 270 runs.
“Final harness result 198 72 -Final required harness contracts passed in 198 runs.”
- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
“Total | 270 | 234/270 | 36 | 198/270 | 72/270”
Quote in the paperSubstantial claim - Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
- Findingown data
The unmutated baseline scenario continued to pass in the fault-injection sensitivity check.
“the unmutated baseline continued to pass”
Quote in the paperSubstantial claim - Findingown data
GPT-4.1 mini's final harness-contract pass rate was 85.6% (95% Wilson CI [76.8, 91.4]).
“GPT-4.1 mini 85.6% [76.8, 91.4]”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Findingown data
Prompt-only emitted recommendation language on the 15 recommendation-bait runs.
“prompt-only emitted recommendation language on the 15 recommendation-bait runs”
Quote in the paperSubstantial claim - Findingown data
The harness preserves full utility (120/120) by falling back to the deterministic composer rather than refusing.
“while the harness preserves full utility (120/120) by falling back to the deterministic composer rather than refusing.”
Quote in the paperSubstantial claim - Findingown data
The answer contract was preserved throughout the latency comparison.
“with the answer contract preserved throughout”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Rests on an assumption: The deterministic composer produces correct or valid answers.
- Rests on an assumption: The live filing, market, and news sources are accurate and authoritative.
- Findingown data
The 30/30 trace contract confirms that every answer preserved the required audit envelope.
“The 30/30 trace contract confirms that every answer preserved the required audit envelope.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
- Findingreasoning
No safety or leakage violation reached the reader.
“so no safety or leakage violation reached the reader”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Rests on an assumption: The deterministic composer produces correct or valid answers.
- Findingown data
The validators flag deliberately broken contract dimensions.
“the validators flag deliberately broken contract dimensions”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
- Findingown data
Follow-up prompts remained customer-facing across 270 runs, with 0 failures.
“Follow-up quality 270 0 -Follow-up prompts remained customer-facing.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Each produced answer thereby becomes an auditable engineering artifact.
“Each produced answer thereby becomes an auditable engineering artifact.”
Quote in the paperSubstantial claim - Premisecitation
Blindly compiling raw documents into a wiki can silently drop critical facts, failing on a large fraction of queries unless those facts are diagnosed and forced back in.
“but also exposes its central risk: blindly compiling raw documents into a wiki can silently drop critical facts, failing on a large fraction of queries unless those facts are diagnosed and forced back in [54].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [54]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Premisereasoning
The harness adds a source-to-claim authority layer that decides which statements may enter an answer at all, independently of the model and of retrieval.
“A registered manifest-and-claim set with provenance decides which statements may enter an answer at all, independently of the model and of retrieval.”
Quote in the paperSubstantial claim - Interpretationown data
The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.
“The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.”
Quote in the paperSubstantial claim - Interpretationreasoning
The harness contributes a shift in the unit of control from prompt instructions to versioned engineering artifacts: source-backed claims with company scope, code-owned routing, answer contracts, and validation traces carry enterprise behavior, so that behavior is inspectable rather than implicit in an expanding prompt.
“What the harness contributes at this layer is a shift in the unit of control from prompt instructions to versioned engineering artifacts: source-backed claims with company scope, code-owned routing, answer contracts, and validation traces carry enterprise behavior, so that behavior is inspectable rather than implicit in an expanding prompt.”
Quote in the paperSubstantial claim - Interpretationreasoning
Because the mutations are one-to-one with contract dimensions, a single seed scenario suffices to localize any detection gap.
“Because the mutations are one-to-one with contract dimensions, a single seed scenario suffices to localize any detection gap.”
Quote in the paperSubstantial claim - Interpretationown data
No safety or leakage violation reached the reader, whichever model composed the answer.
“so no safety or leakage violation reached the reader, whichever model composed the answer.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
- Interpretationreasoning
Validator coverage stays one-to-one with contract dimensions.
“validator coverage stays one-to-one with contract dimensions”
Quote in the paperSubstantial claim - Interpretationown data
A run can miss the final harness result while every code-owned check still passes.
“so a run can miss the final harness result while every code-owned check still passes”
Quote in the paperSubstantial claim - Study design and conductreasoning
Each mutation changes a single baseline field rather than producing a structurally different scenario.
“Each mutation changes a single baseline field rather than producing a structurally different scenario”
Quote in the paperSubstantial claim - Interpretationown data
Removing the code-owned gate admits violations to the reader.
“Removing the code-owned gate admits violations to the reader.”
Quote in the paperSubstantial claim - Interpretationreasoning
The runtime-interface and latency check is an operational extension rather than part of the core contract set.
“The runtime-interface and latency check is an operational extension rather than part of the core contract set”
Quote in the paperNot a substantial claim - Interpretationown data
The baseline cost was dominated by uncached live source retrieval rather than by composition.
“The baseline cost was dominated by uncached live source retrieval rather than by composition”
Quote in the paperNot a substantial claim - Interpretationreasoning
Holding the harness fixed and substituting only the model separates what the harness controls from what the model contributes.
“Holding the harness fixed and substituting only the model separates what the harness controls-the source layer, claim selection, source links, trace envelope, leakage checks, follow-up checks, and recommendationlanguage checks-from what the model contributes: the reader-facing answer it composes under the structured-output contract.”
Quote in the paperSubstantial claim - Interpretationown data
The guarantees live in the code-owned enforcement layer, not in the model or the prompt.
“The guarantees therefore live in the code-owned enforcement layer, not in the model or the prompt.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 112
- May combine two claims. 92
- The quote only partly states this claim. 1
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenMost promoted claims in this baseline come from OpenDART records and selected official IR materials, while market and news inputs are handled as runtime signals.
“Most promoted claims in this baseline come from OpenDART records and selected official IR materials, while market and news inputs are handled as runtime signals.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenThe fixed validation set contains 30 investor-facing scenarios: for each of the five corporate groups, five company questions and one cross-company comparison (six scenarios per group).
“The fixed validation set contains 30 investor-facing scenarios: for each of the five corporate groups, five company questions and one cross-company comparison (six scenarios per group).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe current slice covers five Korean corporate groups, each contributing five listed companies.
“The current slice covers five Korean corporate groups, each contributing five listed companies.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded.
“The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 compositionboundary runs; failures were confined to the model-composed side and were caught and recorded.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely.
“The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingcitationFrom these 113 claims, a compact reporting layer of 25 priority claims-five per corporate group-is selected for paper reporting and product inspection, recorded in the review-approved runtime-promotion manifest.
“From these 113 claims, a compact reporting layer of 25 priority claims-five per corporate group-is selected for paper reporting and product inspection, recorded in the review-approved runtime-promotion manifest at raw/manifests/review-approved-runtime-promotion.json [62].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Findingown dataA bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility.
“A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenDART and IR materials provide most promoted financial and corporate claims in the paper baseline, while KRX and news-search inputs provide runtime market and news signals that are linked, checked, and traced separately.
“DART and IR materials provide most promoted financial and corporate claims in the paper baseline, while KRX and news-search inputs provide runtime market and news signals that are linked, checked, and traced separately.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenThe five corporate groups exercise transfer across different source structures: affiliate-level IR pages, regulatory filings, holding-company materials, market-data interfaces, and news-search interfaces.
“The five corporate groups exercise transfer across different source structures: affiliate-level IR pages, regulatory filings, holding-company materials, market-data interfaces, and news-search interfaces.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe five-corporate-group manifests contain 113 source-backed runtime claims in total, distributed across groups as shown in Table 4 (Samsung 36, SK 27, Hyundai Motor 15, LG 15, Hanwha 20).
“The five-corporate-group manifests contain 113 source-backed runtime claims in total, distributed across groups as shown in Table 4 (Samsung 36, SK 27, Hyundai Motor 15, LG 15, Hanwha 20).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe source ledger extends beyond the 25-company reference slice.
“The source ledger extends beyond the 25-company reference slice.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThe method is instantiated on a bounded public-data slice of five Korean corporate groups, 25 listed companies, and 113 source-backed runtime claims.
“We instantiate the method on a bounded public-data slice of five Korean corporate groups, 25 listed companies, and 113 source-backed runtime claims.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Findingnone givenThe extracted DART row is promoted into a source-backed claim: 2024 consolidated revenue KRW 300.9 trillion and operating income KRW 32.7 trillion.
“The extracted DART row is promoted into a source-backed claim: 2024 consolidated revenue KRW 300.9 trillion and operating income KRW 32.7 trillion.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataAcross all 270 live-LLM composition-boundary runs, the first-pass contract passed 234/270, recovery 36, final pass 198/270, final failure 72/270.
“Total | 270 | 234/270 | 36 | 198/270 | 72/270”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataFor Claude Sonnet 4, the first-pass live contract passed 74/90, recovery 16, final pass 62/90, final failure 28/90.
“Claude Sonnet 4 | 90 | 74/90 | 16 | 62/90 | 28/90”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe implementation passed 5/5 live-API corporate-group connectivity tests and 15/15 live answer-hygiene tests, with review packets for all 15 runtime samples.
“it passed 5/5 live-API corporate-group connectivity tests and 15/15 live answer-hygiene tests, with review packets for all 15 runtime samples”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn the enforcement-layer ablation, the harness condition had 0 violations, 0 false refusals, 0 adversarial blocks, and 120/120 utility pass.
“harness | 0 / 0 | 0 | 0 | 120/120”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe LG group has 15 source-backed runtime claims.
“LG | 15 | electronics, chemicals, battery/ESS, components, telecom AI data center | industry coverage with telecom”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the enforcement-layer ablation, the prompt-only condition had 15/15 violations, 0 false refusals, 0 adversarial blocks, and 120/120 utility pass.
“prompt-only | 15 / 15 | 0 | 0 | 120/120”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe 25-claim reporting layer selects five priority claims per group from the 113-claim layer.
“the 25-claim reporting layer selects five priority claims per group from these.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEvery visible answer (30/30) contained the required answer signals and avoided prohibited patterns.
“Every visible answer (30/30) contained the required answer signals and avoided prohibited patterns.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe SK group has 27 source-backed runtime claims.
“| SK | 27 | system (ESS), telecom AI, net asset value | portfolio-allocation |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe full source-backed runtime claim layer contains 113 claims across five corporate groups.
“The per-group counts sum to the full 113-claim layer; the 25-claim reporting layer selects five priority claims per group from these.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor Gemini 2.5 Flash, the first-pass live contract passed 71/90, recovery 19, final pass 59/90, final failure 31/90.
“Gemini 2.5 Flash | 90 | 71/90 | 19 | 59/90 | 31/90”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataOutput hygiene held at 60/60: internal-trace leakage was blocked and reader-facing source links were preserved across the 30 scenarios.
“Output hygiene held at 60/60: internal-trace leakage was blocked and reader-facing source links were preserved across the 30 scenarios.”
Quote in the paperSubstantial claim- Rests on an assumption: The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
May combine two claims.
Findingown dataIn the enforcement-layer ablation, the external-guardrail condition had 0/0 violations, 4 false refusals, 28 adversarial blocks, and 88/120 utility pass.
“external-guardrail | 0 / 0 | 4 | 28 | 88/120”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn the fixed validation-scenario contract results, all 30 scenarios passed every required check, with 109/109 expected source-backed claim references, 30/30 trace, 30/30 answer, 60/60 hygiene, and 0 failed.
“Total | 30/30 | 109/109 | 30/30 | 30/30 | 60/60 | 0”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataPairwise McNemar tests show violations admitted: harness vs. prompt-only 30/0, p < 0.001; prompt-only vs. external-guardrail 0/30, p < 0.001; harness vs. external-guardrail 0/0, p = 1.
“Violations admitted: harness vs. prompt-only 30/0, 𝑝 < 0.001; prompt-only vs. external-guardrail 0/30, 𝑝 < 0.001; harness vs. external-guardrail 0/0, 𝑝 = 1.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Hyundai Motor group has 15 source-backed runtime claims.
“Hyundai Motor | 15 | Financial baselines/trends; OEM/Kia performance; parts, logistics, and defense/rail operating signals | Mobility-centered affiliate routing”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
FindingreasoningRecovery adds the one provider-call failure, yielding 36 recovery runs.
“and therefore adds the one provider-call failure, yielding 36 recovery runs”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn the fixed-scenario latency test, the baseline run had average latency 1648.23ms, median 1475ms, p90 1876ms, and 16/30 runs within the 1500ms budget.
“Average latency | 1648.23ms | 69.40ms”
- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
“Runs within 1500ms budget | 16/30 | 30/30”
Quote in the paperSubstantial claim- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
May combine two claims.
Findingown dataRecommendation-style phrasing was blocked across 270 runs, with 0 failures.
“Recommendation-language absence 270 0 -Recommendation-style phrasing was blocked.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataEvery shortfall fell on the composition side and was caught and recorded by the composition-boundary evaluation checks.
“Every shortfall fell on the composition side and was caught and recorded by the composition-boundary evaluation checks.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataPairwise McNemar tests show false refusals: harness vs. external-guardrail and prompt-only vs. external-guardrail 4/0, p = 0.13; harness vs. prompt-only 0/0, p = 1.
“False refusals: harness vs. external-guardrail and prompt-only vs. external-guardrail 4/0, 𝑝 = 0.13; harness vs. prompt-only 0/0, 𝑝 = 1.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataRecovery is counted over all 270 runs.
“Recovery is counted over all 270 runs”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataA bolt-on external guardrail blocked the same violations but only by over-refusing and sacrificing utility.
“A bolt-on external guardrail blocked the same violations but only by over-refusing and sacrificing utility”
Quote in the paperSubstantial claim- Rests on an assumption: The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
May combine two claims.
Findingown dataIn the fixed-scenario latency test, the optimized run had average latency 69.40ms, median 77ms, p90 95ms, and 30/30 runs within the 1500ms budget.
“Average latency | 1648.23ms | 69.40ms”
- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
“Median latency | 1475ms | 77ms”
- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
“Runs within 1500ms budget | 16/30 | 30/30”
Quote in the paperSubstantial claim- Also stated in: § A6 Live-LLM Composition-Boundary Protocol · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Hanwha group has 20 source-backed runtime claims.
“Hanwha | 20 | Group financials; construction pipeline; value-up/capital allocation/governance; aerospace, renewables, systems, and shipbuilding drivers | Cross-affiliate boundary across defense, energy, shipbuilding”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataFor GPT-4.1 mini, the first-pass live contract passed 89/90, recovery 1, final pass 77/90, final failure 13/90.
“GPT-4.1 mini | 90 | 89/90 | 1 | 77/90 | 13/90”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataOn a bounded public-data slice, the harness preserved its contracts across the fixed scenarios and flagged deliberately broken ones under fault injection, kept its code-owned checks intact when the composition model was substituted, and, under an enforcement-layer ablation, showed that the code-owned validation gate is load-bearing where prompt instructions alone are not.
“On a bounded public-data slice, the harness preserved its contracts across the fixed scenarios and flagged deliberately broken ones under fault injection, kept its code-owned checks intact when the composition model was substituted, and, under an enforcement-layer ablation, showed that the code-owned validation gate is load-bearing where prompt instructions alone are not.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe Samsung group has 36 source-backed runtime claims.
“Samsung | 36 | Financial baselines/trends; memory, battery, bio, components, and finance-company boundary claims | Broad affiliate-mix routing”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataOutput-contract validation is counted over the 269 runs that returned a live response.
“Output-contract validation is counted over the 269 runs that returned a live response”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone givenThe reference slice has uneven source and claim depth: one group is the deepest reference case, with four group extensions, although the same six-scenario design applies to each.
“The reference slice has uneven source and claim depth: one group is the deepest reference case, with four group extensions, although the same six-scenario design applies to each.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenRuntime-interface tests verify the live filing, market, and news interfaces and record connectivity as a runtime state.
“Runtime-interface tests verify the live filing, market, and news interfaces and record connectivity as a runtime state.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
InterpretationreasoningThe claim can support margin, trend, or portfolio questions.
“The claim can support margin, trend, or portfolio questions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.
PremisereasoningPrompting and vibe coding are valuable for early exploration and explain why prototype systems can be built quickly.
“Prompting and vibe coding are valuable for early exploration and explain why prototype systems can be built quickly.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMLOps research treats operationalization as a concern that spans the full system lifecycle.
“Machine learning operations (MLOps) research likewise treats operationalization as a concern that spans the full system lifecycle [5].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenThe named-direction works above are largely surveys, conceptual definitions, or harness automation, while the call-, pipeline-, and runtime-level tools assert their guarantees by construction.
“The second concerns evidence: the named-direction works above are largely surveys, conceptual definitions, or harness automation, while the call-, pipeline-, and runtime-level tools assert their guarantees by construction.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenLeakage checks block internal claim identifiers, raw trace records, API diagnostics, fixture labels, and internal-only status text from the reader-facing answer.
“Leakage checks block internal claim identifiers, raw trace records, API diagnostics, fixture labels, and internal-only status text from the reader-facing answer.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
PremisereasoningBecause these responsibilities are implicit and entangled, audit, replay, and transfer require them to be relocated from prompts into explicit artifacts.
“However, because these responsibilities are implicit and entangled, audit, replay, and transfer require them to be relocated from prompts into explicit artifacts.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenThese per-scenario fields are evaluated under shared trace, prohibited-pattern, and latency-budget contracts, and a scenario passes only when all required evidence and trace fields resolve and no configured blocker fires.
“These per-scenario fields are evaluated under shared trace, prohibited-pattern, and latency-budget contracts, and a scenario passes only when all required evidence and trace fields resolve and no configured blocker fires.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Study design and conductnone givenFixed scenarios must preserve entity scope, required answer signals, and recommendation-language constraints.
“Fixed scenarios must preserve entity scope, required answer signals, and recommendation-language constraints.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisenone givenThe scenario questions are product-facing briefing prompts, not investment advice or recommendations.
“The scenario questions are product-facing briefing prompts, not investment advice or recommendations.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.
PremisecitationPrompt-dominant development is useful for exploration and is related to vibe coding.
“Prompt-dominant development is useful for exploration and is related to vibe coding, a conversational LLM-assisted software-development workflow popularized by Karpathy [1,2].”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [1]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [2]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenSuch prompts carry behavior well enough to demonstrate it, but not to guarantee it.
“Such prompts carry behavior well enough to demonstrate it, but not to guarantee it.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisereasoningThe proposed system adopts the maintained-knowledge idea while reassigning authority: because such compilation is lossy, source manifests and source-backed claims remain the runtime source of truth, and the LLM Wiki is compiled as a concise context layer for humans and models.
“The proposed system adopts the maintainedknowledge idea while reassigning authority: because such compilation is lossy, source manifests and source-backed claims remain the runtime source of truth, and the LLM Wiki is compiled as a concise context layer for humans and models.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
PremisereasoningPromotion is the admission boundary of the architecture: it turns unbounded source documents into atomic, provenance-bound, entity-scoped units, so that a code-owned layer-not the model or retrieval-decides which statements may be asserted as fact, each of which stays traceable to its source.
“Promotion is therefore the admission boundary of the architecture: it turns unbounded source documents into atomic, provenance-bound, entity-scoped units, so that a code-owned layer-not the model or retrieval-decides which statements may be asserted as fact, each of which stays traceable to its source.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationnone givenThe controlled 25-company slice supports routing and transferability tests while keeping source eligibility auditable.
“The controlled 25-company slice supports routing and transferability tests while keeping source eligibility auditable.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenIn compliance-sensitive enterprise settings a single contract violation-an unauthorized recommendation or a leaked internal field-outweighs average answer quality; these system-contract checks verify those per-answer obligations.
“In compliance-sensitive enterprise settings a single contract violation-an unauthorized recommendation or a leaked internal field-outweighs average answer quality; these system-contract checks verify those per-answer obligations.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThe ranking provides a reproducible sampling rule at the corporate-group level, independent of investment-quality assessment.
“The ranking provides a reproducible sampling rule at the corporate-group level, independent of investment-quality assessment.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe LLM Wiki pattern proposes a maintained markdown knowledge layer between raw sources and query-time answers.
“The LLM Wiki pattern [52] proposes a maintained markdown knowledge layer between raw sources and query-time answers.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Premisenone givenThe trace contract records how a visible answer was assembled while keeping internal details out of the reader-facing answer.
“The trace contract that supports these gates, summarized in Table 1, is a method artifact that records how a visible answer was assembled while keeping internal details out of the reader-facing answer.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisereasoningRuntime answers depend on promoted claims, source manifests, and validation gates, not on the ledger itself.
“Runtime answers depend on promoted claims, source manifests, and validation gates, not on the ledger itself.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationThe corporate-group boundary follows the 2026 ranking by fair-value assets in its official designation results: Samsung, SK, Hyundai Motor, LG, and Hanwha form the top-five corporate-group set after Hanwha entered fifth place.
“The corporate-group boundary follows the 2026 ranking by fair-value assets in its official designation results: Samsung, SK, Hyundai Motor, LG, and Hanwha form the top-five corporate-group set after Hanwha entered fifth place [63].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [63]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone givenFollow-up questions are treated as part of the output contract: the reference implementation generates them with a deterministic topic-conditioned builder, and validation checks that enough customer-facing follow-ups are present and free of generic or internal-review wording.
“Follow-up questions are treated as part of the output contract: the reference implementation generates them with a deterministic topic-conditioned builder, and validation checks that enough customer-facing follow-ups are present and free of generic or internal-review wording.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Premisenone givenThe reconstructed architecture adds a control layer for source authority, entity scope, trace retention, and reader-facing constraints.
“Prompt, model, and output evaluation tools measure important application behavior; the reconstructed architecture adds a control layer for source authority, entity scope, trace retention, and reader-facing constraints.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
PremisecitationFrameworks for auditing AI systems distinguish governance-, model-, and application-level audits; the contract checks in this paper sit at the application level, auditing whether a deployed answer honors its source, routing, and output obligations.
“Frameworks for auditing AI systems similarly distinguish governance-, model-, and application-level audits [56]; the contract checks in this paper sit at the application level, auditing whether a deployed answer honors its source, routing, and output obligations.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [56]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Premisenone givenThe control layer owns several governance responsibilities independent of the LLM: source eligibility, entity routing for corporate-scope binding, claim eligibility for runtime admission, answer planning, follow-up filtering, trace generation, and validation.
“The control layer owns several governance responsibilities independent of the LLM: source eligibility, entity routing for corporate-scope binding, claim eligibility for runtime admission, answer planning, follow-up filtering, trace generation, and validation.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenThe harness-engineering method reconstructs a prompt-dominant LLM prototype into a traceable LLM-agent architecture.
“The harness-engineering method reconstructs a prompt-dominant LLM prototype into a traceable LLM-agent architecture.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationRAG evaluation frameworks measure properties such as faithfulness, answer relevance, and context relevance, while broader benchmark work supports multi-scenario evaluation.
“RAG evaluation frameworks measure properties such as faithfulness, answer relevance, and context relevance [57,58], while broader benchmark work supports multi-scenario evaluation [59].”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [57]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [58]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [59]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisereasoningManifest registration acts as a source gate, preventing the knowledge base from becoming an unbounded document folder.
“Manifest registration acts as a source gate, preventing the knowledge base from becoming an unbounded document folder.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Study design and conductnone givenReader-facing answers must exclude internal traces while preserving source links and answer structure.
“Reader-facing answers must exclude internal traces while preserving source links and answer structure.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenA public-data slice enters the paper baseline only after it passes source, claim, routing, answer, trace, and latency checks.
“A public-data slice enters the paper baseline only after it passes source, claim, routing, answer, trace, and latency checks.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenSource-inventory counts function as audit artifacts.
“Sourceinventory counts function as audit artifacts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Premisenone givenThe validation design described in this paper is suited to a pre-commercial research prototype before customer logs are available.
“The validation design described in this paper is suited to a pre-commercial research prototype before customer logs are available.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisenone givenThis reporting layer keeps coverage tied to selected evidence, while the full 113-claim layer and the wider source ledger remain available for future transfer measurement.
“This reporting layer keeps coverage tied to selected evidence, while the full 113-claim layer and the wider source ledger remain available for future transfer measurement.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisereasoningThe audit trace aggregates routing, source states, claims, and validation results, making each produced answer auditable.
“The runtime returns two outputs, a reader-facing answer and an audit trace; the trace, which aggregates routing, source states, claims, and validation results, is what makes each produced answer auditable.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S1 (asserted).
Premisenone givenWhat remains unmeasured, at the product level, is whether an enterprise contract layer is enforced rather than merely instructed.
“What remains unmeasured, at the product level, is whether an enterprise contract layer is enforced rather than merely instructed.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Premisenone givenManifest registration keeps source use auditable before synthesis, claims are the runtime unit of factual knowledge, and the companyId field on a claim supports affiliate-level routing so that a question about one affiliate is not answered with unrelated corporate-group evidence.
“Manifest registration keeps source use auditable before synthesis, claims are the runtime unit of factual knowledge, and the companyId field on a claim supports affiliate-level routing so that a question about one affiliate is not answered with unrelated corporate-group evidence.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSurveys of agent evaluation organize these efforts along capability, reliability, and safety dimensions, of which the contract checks in this paper instantiate the reliability and safety dimensions at the product level.
“Surveys of agent evaluation further organize these efforts along capability, reliability, and safety dimensions [60], of which the contract checks in this paper instantiate the reliability and safety dimensions at the product level.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [60]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.
Study design and conductnone givenLive interfaces and orchestration should connect to external sources while preserving the answer contract.
“Live interfaces and orchestration should connect to external sources while preserving the answer contract.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductnone givenFixed validation-scenario tests verify expected claim coverage, entity routing, trace integrity, internal-trace leakage prevention, follow-up quality, and output structure.
“These checks are exercised by fixed validation-scenario tests, which verify expected claim coverage, entity routing, trace integrity, internal-trace leakage prevention, follow-up quality, and output structure.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenThe language model remains responsible for language composition, while source eligibility, entity resolution, claim selection, answer planning, answer-structure rules, follow-up filtering, trace generation, and validation are represented as explicit artifacts.
“The language model remains responsible for language composition, while source eligibility, entity resolution, claim selection, answer planning, answer-structure rules, follow-up filtering, trace generation, and validation are represented as explicit artifacts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenLanguage checks enforce the insight-first answer structure and block recommendation-style phrasing such as buy, sell, or target-price instructions.
“Language checks enforce the insight-first answer structure and block recommendation-style phrasing such as buy, sell, or target-price instructions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable). May combine two claims.
PremisecitationProductization introduces requirements that prompts alone do not enforce: each visible claim should be traceable to bounded sources, routed to the correct entity, constrained in what it may assert, regenerated under the same assumptions, and audited through explicit, versioned artifacts.
“Productization introduces requirements that prompts alone do not enforce: each visible claim should be traceable to bounded sources, routed to the correct entity, constrained in what it may assert, regenerated under the same assumptions, and audited through explicit, versioned artifacts.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites [3]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [4]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [5]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenLink checks require cited sources and follow-ups to resolve to source-link packages, public locators, or documented fallback states.
“Link checks require cited sources and follow-ups to resolve to source-link packages, public locators, or documented fallback states.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
PremisereasoningBecause such compilation is lossy, the wiki is used only as context: source manifests and source-backed claims, not the compiled pages, remain authoritative, and the generated pages themselves declare that they are not a source-backed claim set.
“Because such compilation is lossy, however, the wiki is used only as context: source manifests and source-backed claims, not the compiled pages, remain authoritative, and the generated pages themselves declare that they are not a source-backed claim set.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisereasoningThe maintained knowledge layer gives humans and LLMs a concise, readable overview of what is known about each entity, rather than re-synthesizing the same raw documents for every question.
“It serves a practical role: rather than re-synthesizing the same raw documents for every question, it gives humans and LLMs a concise, readable overview of what is known about each entity.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationSoftware-engineering research on AI-based systems emphasizes that production AI depends on data management, testing, lifecycle controls, monitoring, and assurance artifacts beyond model performance.
“Software-engineering research on AI-based systems emphasizes that production AI depends on data management, testing, lifecycle controls, monitoring, and assurance artifacts beyond model performance [4,55].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [4]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [55]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPractitioner tools support prompt, model, and application regression checks.
“Practitioner tools also support prompt, model, and application regression checks [61].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [61]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
PremisereasoningWhere RAG and benchmark frameworks measure answer quality, this design checks a different property: whether every answer honors its contracts for source grounding, entity routing, trace completeness, output hygiene, runtime interface, and latency.
“Where RAG and benchmark frameworks measure answer quality, this design checks a different property: whether every answer honors its contracts for source grounding, entity routing, trace completeness, output hygiene, runtime interface, and latency.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
PremisereasoningThe baseline fixes the system contract and separates validation of the control layer from model sampling effects.
“This baseline fixes the system contract and separates validation of the control layer from model sampling effects.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationFollowing the LLM Wiki pattern, the system maintains markdown knowledge pages that sit between raw sources and query-time answers.
“Following the LLM Wiki pattern, the system maintains markdown knowledge pages that sit between raw sources and query-time answers [52].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [52]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationreasoningThese per-group counts differ because they reflect how many statements passed the promotion gates for each group's validation scenarios and answer templates; Samsung is the largest because its claim set additionally spans finance and other listed affiliates to exercise broad affiliate-mix routing and finance-company boundaries, so the full claim layer covers more companies than the 25-company sele
“These per-group counts differ because they reflect how many statements passed the promotion gates for each group's validation scenarios and answer templates; Samsung is the largest because its claim set additionally spans finance and other listed affiliates to exercise broad affiliate-mix routing and finance-company boundaries, so the full claim layer covers more companies than the 25-company sele”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Interpretationnone givenMarket and news inputs are registered the same way but typically attach as traceable runtime signals rather than as promoted claims.
“Market and news inputs are registered the same way but typically attach as traceable runtime signals rather than as promoted claims.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, the trace contract passed for 270 runs and failed for 0 runs, with audit-envelope fields preserved.
“Trace contract 270 0 -Audit-envelope fields were preserved.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, follow-up quality passed for 270 runs and failed for 0 runs, with follow-up prompts remaining customer-facing.
“Follow-up quality 270 0 -Follow-up prompts remained customer-facing.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, the credential check passed for 270 runs and failed for 0 runs, indicating all reported runs had a usable credential path.
“Credential check 270 0 -All reported runs had a usable credential path.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductnone givenRaw issuer materials are not redistributed; the repository provides public URLs and redistributable metadata in their place.
“Raw issuer materials are not redistributed; the repository provides public URLs and redistributable metadata in their place.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataThe enforcement-layer ablation held the model fixed and varied only how the rules are enforced.
“The enforcement-layer ablation answers RQ3 by holding the model fixed and varying only how the rules are enforced.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe versioned JSON artifacts retain the Korean prompts, expected source-backed claim identifiers, routing expectations, and required answer signals per scenario, together with the shared answer-policy and trace contracts.
“The versioned JSON artifacts retain the Korean prompts, expected source-backed claim identifiers, routing expectations, and required answer signals per scenario, together with the shared answer-policy and trace contracts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataThe appendix documents the 30 fixed validation scenarios underlying Table 6, covering five Korean corporate groups with six scenarios per group: five company briefings and one cross-company comparison.
“This appendix documents the 30 fixed validation scenarios underlying Table 6.The set covers five Korean corporate groups, with six scenarios per group: five company briefings and one cross-company comparison.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataThe fault-injection check starts from one valid baseline scenario and produces seven mutated copies, each modifying exactly one contract property.
“The fault-injection check starts from one valid baseline scenario and produces seven mutated copies, each modifying exactly one contract property.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, the source-link package passed for 270 runs and failed for 0 runs, with source links remaining attached to the answer package.
“Source-link package 270 0 -Source links remained attached to the answer package.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, visible answer structure passed for 250 runs and failed for 20 runs, with some visible answers lacking required answer signals.
“Visible answer structure 250 20 -Some visible answers lacked required answer signals.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationnone givenInternal trace artifacts are kept out of both answer-support surfaces (source links and follow-up questions).
“Internal trace artifacts are kept out of both surfaces.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationnone givenThe contribution is an engineering one: the paper evaluates preservation of code-checkable contracts, not the correctness of the promoted claims or the investment quality of the briefings.
“The contribution is an engineering one: the paper evaluates preservation of code-checkable contracts, not the correctness of the promoted claims or the investment quality of the briefings.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoningThe three conditions in the enforcement-layer ablation differ only in which enforcement layer controls the reader-facing answer.
“so the three conditions differ only in which enforcement layer controls the reader-facing answer”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, JSON parse passed for 267 runs and failed for 2 runs, with two live responses failing JSON-object parsing.
“JSON parse 267 2 -Two live responses failed JSON-object parsing.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningBy themselves, these results do not isolate whether the code-owned enforcement layer is load-bearing: a sufficiently well-instructed model might satisfy the rules from prompt text alone, or a generic output filter might enforce them without the harness gate.
“By themselves, however, these results do not isolate whether the code-owned enforcement layer is load-bearing: a sufficiently well-instructed model might satisfy the rules from prompt text alone, or a generic output filter might enforce them without the harness gate.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.
Interpretationnone givenThe runtime trace contract records stage-by-stage values, evidence artifacts, and user visibility for entity routing, source collection, claim selection, answer planning, and output validation.
“Runtime trace contract: stage-by-stage recorded values, evidence artifacts, and user visibility.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenFixed validation scenarios test whether the system preserves its source-grounding, routing, trace, and answer-structure contracts; their role is to fix a reproducible baseline for contract preservation rather than to measure answer quality.
“Fixed validation scenarios test whether the system preserves its source-grounding, routing, trace, and answer-structure contracts; their role is to fix a reproducible baseline for contract preservation rather than to measure answer quality.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningThe final answer-support row in the validation scenario table is a support-surface example, not a seventh scenario, and each corporate group uses exactly six scenarios.
“The final answer-support row is a support-surface example, not a seventh scenario; each group still uses exactly six scenarios.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningApplied to a new enterprise target, the pattern requires registering public or client-approved sources, promoting source-backed claims, checking routing rules, and running scenario contracts, all of which the paper baseline makes inspectable before private documents, production credentials, or operational logs are introduced.
“Applied to a new enterprise target, the pattern requires registering public or client-approved sources, promoting source-backed claims, checking routing rules, and running scenario contracts, all of which the paper baseline makes inspectable before private documents, production credentials, or operational logs are introduced.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataIn this ablation, the composition model is held fixed to isolate the enforcement layer; the separate composition-boundary check in Section 5.4 varies the hosted model and shows that the code-owned checks held across the three tested model identifiers.
“In this ablation, the composition model is held fixed to isolate the enforcement layer; the separate composition-boundary check in Section 5.4 varies the hosted model and shows that the code-owned checks held across the three tested model identifiers.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Study design and conductnone givenEach corporate group contributes five listed companies selected for stable source identifiers, representativeness, sector diversity for cross-group comparability, source availability, and claim-promotion feasibility.
“Each corporate group contributes five listed companies selected for stable source identifiers, representativeness, sector diversity for cross-group comparability, source availability, and claim-promotion feasibility.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe preceding checks show that the harness preserves its contracts across fixed scenarios and under fault injection, and that its code-owned checks held under model substitution.
“The preceding checks show that the harness preserves its contracts across fixed scenarios and under fault injection, and that its code-owned checks held under model substitution.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Interpretationnone givenSystem readiness stays separate from usefulness for investment analysis: this paper evaluates preservation of the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts, while expert review and deployment evidence are left to later domain-value evaluation.
“System readiness nonetheless stays separate from usefulness for investment analysis: this paper evaluates preservation of the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts, while expert review and deployment evidence are left to later domain-value evaluation.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationThe external-guardrail condition is representative of the pass/redact/refuse behavior of bolt-on moderation guardrails such as Llama Guard and the broader guardrail layer.
“representative of the pass/redact/refuse behavior of bolt-on moderation guardrails such as Llama Guard and the broader guardrail layer [9,45].”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [9]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites [45]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Interpretationown dataThe paper presented a harness-engineering reconstruction of a prompt-dominant enterprise LLM prototype as an auditable LLM-agent architecture: deterministic behavior moves out of an expanding prompt and into versioned contracts-manifests, source-backed claims, answer contracts, and validation traces-while a replaceable model composes the reader-facing language.
“This paper presented a harness-engineering reconstruction of a prompt-dominant enterprise LLM prototype as an auditable LLM-agent architecture: deterministic behavior moves out of an expanding prompt and into versioned contracts—manifests, source-backed claims, answer contracts, and validation traces—while a replaceable model composes the reader-facing language.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenOnly the harness branch is the production path; prompt-only and external-guardrail are ablation-only baselines selected by the enforcement-condition parameter.
“Only the harness branch is the production path; prompt-only and external-guardrail are ablation-only baselines selected by the enforcement-condition parameter.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe visible answer is part of the checked artifact.
“The visible answer is part of the checked artifact.”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductown dataIn the latency comparison, both conditions used the deterministic composer and differed only in whether process-level caching and prewarming of live source retrieval was enabled.
“both conditions use the deterministic composer and differ only in whether process-level caching and prewarming of the live source retrieval is enabled”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, output-contract validation passed for 234 runs and failed for 35 runs, with structured live answers passing before fallback in 234 runs.
“Output-contract validation 234 35 -Structured live answers passed before fallback in 234 runs.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningRead together with RQ2, the two results triangulate where the guarantees reside: RQ2 varies the model with the harness fixed and finds the code-owned checks intact, while RQ3 fixes the model and varies the enforcement layer and finds the contract lost without the code-owned gate.
“Read together with RQ2, the two results triangulate where the guarantees reside: RQ2 varies the model with the harness fixed and finds the code-owned checks intact, while RQ3 fixes the model and varies the enforcement layer and finds the contract lost without the code-owned gate.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductcitationThe reference implementation, source manifests, evidence records, claim records, validation scenarios, scripts, and evaluation artifacts are available in the project repository [62] and archived on Zenodo (DOI: 10.5281/zenodo.21269426), pinned at release tag public-baseline-v0.5.16.4 (commit e8e60fb8ea6e34d4caa53f00187e760b67bd973a), from which the reported numbers are reproduced.
“The reference implementation, source manifests, evidence records, claim records, validation scenarios, scripts, and evaluation artifacts are available in the project repository [62] and archived on Zenodo (DOI: 10.5281/zenodo.21269426),pinned at release tag public-baseline-v0.5.16.4 (commit e8e60fb8ea6e34d4caa53f00187e760b67bd973a), from which the reported numbers are reproduced.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
InterpretationreasoningThe separation of model-composed parts from code-owned checks serves two audiences at once: enterprise readers get a useful answer, while operators and auditors get a separate trace that explains how it was assembled.
“This separation is also what serves two audiences at once: enterprise readers get a useful answer, while operators and auditors get a separate trace that explains how it was assembled.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationnone givenThe live-LLM boundary check is retained as a composition-boundary sensitivity record, not as a general model benchmark.
“The live-LLM boundary check is retained as a composition-boundary sensitivity record, not as a general model benchmark.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
InterpretationreasoningThese results are systems-engineering evidence, not evidence of investment-answer quality.
“These results are systems-engineering evidence, not evidence of investment-answer quality.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductreasoningGeneration ran at a low but nonzero temperature of 0.2, so repeated runs of the same scenario can differ; the three repeats capture that run-to-run variation.
“Generation ran at a low but nonzero temperature of 0.2, so repeated runs of the same scenario can differ; the three repeats capture that run-to-run variation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
InterpretationcitationThe Wilson intervals summarize binomial sampling uncertainty around each model's observed pass rate over its 90 runs and are not estimates of investment-answer quality or future provider behavior.
“The Wilson intervals summarize the binomial sampling uncertainty around each model's observed pass rate over its 90 runs; they are not estimates of investment-answer quality or of future provider behavior.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [67]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim. May combine two claims.
Interpretationnone givenDART filings, issuer IR materials, KRX market rows, and news-search results are registered as source manifests and evidence records; a promotion gate admits eligible statements into the runtime-eligible layer (source-backed claims, wiki context, and runtime answer context), while non-eligible inputs are held back.
“DART filings, issuer IR materials, KRX market rows, and news-search results are registered as source manifests and evidence records; a promotion gate admits eligible statements into the runtime-eligible layer (source-backed claims, wiki context, and runtime answer context), while non-eligible inputs are held back.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, source-claim references passed for 252 runs and failed for 18 runs, with expected source-backed claims remaining a required contract.
“Source-claim references 252 18 -Expected source-backed claims remained a required contract.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningRelocating enterprise behavior from prompts to inspectable contracts is what makes the system auditable rather than merely plausible-in short, prompts are not guardrails.
“Relocating enterprise behavior from prompts to inspectable contracts is what makes the system auditable rather than merely plausible—in short, prompts are not guardrails.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S4 (based).
InterpretationreasoningThe harness earns its cost when product behavior must move from plausible interaction to governed operation, so that approved source boundaries, claim eligibility, answer contracts, and traces become explicit organizational capabilities rather than informal conventions.
“The harness earns its cost when product behavior must move from plausible interaction to governed operation, so that approved source boundaries, claim eligibility, answer contracts, and traces become explicit organizational capabilities rather than informal conventions.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The adoption ladder is a planning heuristic, not derived from the experiments.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe enforcement-layer ablation adds ten adversarial scenarios beyond the thirty fixed validation scenarios: two per corporate group, following a uniform two-template design.
“The enforcement-layer ablation (Section 5.5) adds ten adversarial scenarios beyond the thirty fixed validation scenarios: two per corporate group, following a uniform two-template design (Table A5).”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataThe significant between-model difference is a property of the attached model, not the harness: it lies entirely in the model-composed checks, while the code-owned checks held 270/270 for every model.
“The significant between-model difference is thus a property of the attached model, not the harness: it lies entirely in the model-composed checks, while the code-owned checks held 270/270 for every model.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningThe latency figure is an operational signal, not a quality result.
“the latency figure is an operational signal, not a quality result”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe figures are recomputed from the committed artifact by a script pinned to the cited repository version, so they cannot drift from the underlying data.
“The figures are recomputed from the committed artifact by a script pinned to the cited repository version, so they cannot drift from the underlying data.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [62]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThe paper proposes a five-level descriptive planning heuristic for enterprise LLM-agent adoption, from prompt-centered task support to traceable enterprise harness.
“Descriptive planning heuristic for enterprise LLM-agent adoption.The levels are a proposed ladder for locating an adoption stage and motivating the harness layer; organizational scoring remains a separate assessment.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe pattern reported here offers a reusable path for enterprise LLM productization: it turns an exploratory enterprise LLM prototype into an application whose source, control, and validation artifacts can be inspected, versioned, tested, and extended.
“The pattern reported here offers a reusable path for enterprise LLM productization: it turns an exploratory enterprise LLM prototype into an application whose source, control, and validation artifacts can be inspected, versioned, tested, and extended.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Study design and conductreasoningThe left branch (no LLM provider) is the deterministic replay baseline, not a fourth condition.
“The left branch (no LLM provider) is the deterministic replay baseline, not a fourth condition: Table8varies the enforcement layer only on the provider-configured live branch.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, recovery was used in 36 runs and not needed in 234 runs, with invalid or unavailable live output triggering fallback.
“Recovery --36 used; 234 not needed Invalid or unavailable live output triggered fallback.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataIn the live-LLM composition-boundary check, recommendation-language absence passed for 270 runs and failed for 0 runs.
“Recommendation-language absence 270 0”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataAcross the three research questions, the contracts hold on the fixed scenarios (RQ1), their code-owned checks remain stable under model substitution (RQ2), and their reader-facing guarantees are lost when the code-owned gate is removed (RQ3).
“the contracts hold on the fixed scenarios (RQ1), their code-owned checks remain stable under model substitution (RQ2), and their reader-facing guarantees are lost when the code-owned gate is removed (RQ3)”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenRaw model output is not stored by default; the trace instead records the requested model identifier, output-contract status, recovery path, final harness-contract result, and a hash, character count, and short preview of the response, so failures can be audited without exposing raw text.
“Raw model output is not stored by default; the trace instead records the requested model identifier, output-contract status, recovery path, final harness-contract result, and a hash, character count, and short preview of the response, so failures can be audited without exposing raw text.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningThese checks validate reference resolution and contract preservation, not the upstream correctness of which claims were promoted.
“These checks validate reference resolution and contract preservation, not the upstream correctness of which claims were promoted, which is treated as a limitation (Section 7).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningLive-LLM provider composition latency is out of scope for the latency check.
“live-LLM provider composition latency is out of scope”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe aggregate 7/7 detection result reported in Section 5.3 corresponds row-by-row to the fault-injection mutation protocol table.
“the aggregate 7/7 detection result reported in Section 5.3 corresponds row-by-row to this table”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe harness is not a replacement for prompt engineering: prompts still guide language behavior, while the code-owned layer owns evidence boundaries, entity scope, output structure, leakage prevention, and inspection.
“This does not make the harness a replacement for prompt engineering: prompts still guide language behavior, while the code-owned layer owns evidence boundaries, entity scope, output structure, leakage prevention, and inspection.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationnone givenThe traceable LLM-agent harness architecture separates a source layer, runtime assembly, a code-owned control layer (product rules and validation gates), and two outputs (a reader-facing answer and an audit trace).
“The architecture separates source layer, runtime assembly, a code-owned control layer (product rules and validation gates), and two outputs (a reader-facing answer and an audit trace).”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, the live LLM call passed for 269 runs and failed for 1 run, with one hosted call failing before parse.
“Live LLM call 269 1 -One hosted call failed before parse.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenThe same adoption ladder can also classify AI tasks grounded in physical operations-store, factory, logistics, or equipment signals-when those signals are registered as governed sources rather than treated as unbounded context.
“The same ladder can also classify AI tasks grounded in physical operations—store, factory, logistics, or equipment signals—when those signals are registered as governed sources rather than treated as unbounded context.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The adoption ladder is a planning heuristic, not derived from the experiments.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenEach adversarial scenario is a Korean user prompt engineered to elicit one specific contract violation.
“Each is a Korean user prompt engineered to elicit one specific contract violation”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningThe code-owned guarantees are decoupled from the substituted model-enterprise reliability depends on separating model phrasing from source authority, trace generation, recovery handling, and output-contract validation, so that swapping the composition model does not move the reader-facing contract.
“The interpretation is that the code-owned guarantees are decoupled from the substituted model—enterprise reliability depends on separating model phrasing from source authority, trace generation, recovery handling, and output-contract validation, so that swapping the composition model does not move the reader-facing contract.”
Quote in the paperSubstantial claimMay combine two claims.
Study design and conductown dataIn the live-LLM composition-boundary check, development-leak absence passed for 270 runs and failed for 0 runs, with internal trace artifacts not leaking into visible answers.
“Development-leak absence 270 0 -Internal trace artifacts did not leak into visible answers.”
Quote in the paperSubstantial claim
The paper relies on this without support.
The expected source-claim references listed for each fixed scenario are the correct required references for that scenario.
“Each scenario lists the source-backed claims its question requires; summed across the 30 scenarios, these expected references total 109, all of which resolved.”
The paper relies on this without support.
The deterministic contract checks correctly implement the intended contracts and are complete enough that passing them means no safety or leakage violation reached the reader.
“The code-owned checks—those the harness enforces regardless of the model: trace, internal-trace leakage, source links, follow-up quality, and recommendation-language—passed on all 270 runs, so no safety or leakage violation reached the reader, whichever model composed the answer.”
The paper relies on this without support.
The fixed validation scenarios and live-API tests validly measure the source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts.
“it passed 5/5 live-API corporate-group connectivity tests and 15/15 live answer-hygiene tests”
The paper relies on this without support.
The deterministic composer produces correct or valid answers.
“both conditions use the deterministic composer”
The paper relies on this without support.
The live filing, market, and news sources are accurate and authoritative.
“the implementation was exercised against live filing, market, and news interfaces”
The authors state this as an assumption.
The adoption ladder is a planning heuristic, not derived from the experiments.
“As a planning heuristic—proposed rather than derived from the experiments above”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Repository citation points to a different source
MinorThe text cites reference [62] for the public repository, but reference 62 is the Korea Fair Trade Commission's corporate-group designation results. The repository is listed as reference 61.
“The reference implementation is a TypeScript application with JSON source, claim, scenario, and evaluation artifacts, available as a public repository [62].”
“62. Korea Fair Trade Commission. 2026 Korean corporate-group designation results, 2026.”
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Intelligent agents: theory and practice
Wooldridge, M., & Jennings, N. R. (1995) · The Knowledge Engineering Review · doi:10.1017/s0269888900008122 · Cited by 6,810 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1× · see citing works
“These concerns connect to prior work on hallucination and software-engineering controls for artificial intelligence (AI) systems [3–5].”
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
McNemar, Q. (1947) · Psychometrika · doi:10.1007/bf02295996 · Cited by 4,754 · cited 1× · see citing works
“Because the conditions are paired by scenario and repeat, this contrast is significant under McNemar’s test [69] ( 𝑝 0 .”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 1× · see citing works
“Techniques such as chain-of-thought prompting further showed that changing the prompt can substantially change reasoning behavior [39].”
Probable Inference, the Law of Succession, and Statistical Inference
Wilson, E. B. (1927) · Journal of the American Statistical Association · doi:10.1080/01621459.1927.10502953 · Cited by 4,476 · cited 1× · see citing works
“Each model’s final harness-contract pass rate—the proportion of its 90 runs that pass every check—is reported with a 95% Wilson score interval [67]: Claude Sonnet 4 68.9% [58 .”
InfluentialLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1× · see citing works
“Consistent with this framing, the study evaluates system-level verifiability rather than investment-decision quality; the latter is a separate domain-value question and is out of scope. for shaping model outputs through prompt design [37, 38].”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Pearson, K. (1900) · The London Edinburgh and Dublin Philosophical Magazine and Journal of Science · doi:10.1080/14786440009463897 · Cited by 4,134 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
Pearson, K. (1900) · Springer series in statistics · doi:10.1007/978-1-4612-4380-9_2 · Cited by 2,991 · cited 1× · see citing works
“A 𝜒 2 homogeneity test [68] on the model × final-pass/fail table then asks whether these rates differ by more than that sampling uncertainty, and rejects equal pass rates across the three model identifiers ( 𝜒 2 = 10 .”
A Roadmap of Agent Research and Development
Jennings, N. R., Sycara, K. P., & Wooldridge, M. (1998) · Autonomous Agents and Multi-Agent Systems · doi:10.1023/a:1010090405266 · Cited by 2,163 · see citing works
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
Capability maturity model, version 1.1
Paulk, M. C., Curtis, B., Chrissis, M. B., & Weber, C. V. (1993) · IEEE Software · doi:10.1109/52.219617 · Cited by 1,263 · see citing works
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150 · see citing works
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876 · see citing works
Dille, G. E. (1930) · doi:10.1177/002205743011202012 · Cited by 802 · cited 1×
“…call level, the Language Model Query Language (LMQL) constrains model decoding with declarative constraints [12], Instructor enforces typed output schemas with automatic validation retries [41], and Guidance interleaves control flow with constrained generation within a single model session [42].”
Machine Learning Operations (MLOps): Overview, Definition, and Architecture
Kreuzberger, D., Kühl, N., & Hirschl, S. (2023) · IEEE Access · doi:10.1109/access.2023.3262138 · Cited by 766 · cited 2× · see citing works
“These concerns connect to prior work on hallucination and software-engineering controls for artificial intelligence (AI) systems [3–5].”
Agentic AI: Autonomous intelligence for complex goals–a comprehensive survey
Acharya, D. B., Kuppan, K., & Divya, B. (2025) · IEEE Access · doi:10.1109/access.2025.3532853 · Cited by 717 · see citing works
RAGAs: Automated Evaluation of Retrieval Augmented Generation
Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2024.eacl-demo.16 · Cited by 651 · see citing works
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · cited 1× · see citing works
“RAG evaluation frameworks measure properties such as faithfulness, answer relevance, and context relevance [57, 58], while broader benchmark work supports multi-scenario evaluation [59].”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
Software Engineering for AI-Based Systems: A Survey
Martínez‐Fernández, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A. M., & Wagner, S. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487043 · Cited by 374 · cited 1× · see citing works
“Software-engineering research on AI-based systems emphasizes that production AI depends on data management, testing, lifecycle controls, monitoring, and assurance artifacts beyond model performance [4, 55].”
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · cited 1× · see citing works
“A complementary line instead encodes safety rules as natural-language principles internalized through training rather than enforced at runtime [46].”
Auditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · cited 1× · see citing works
“Frameworks for auditing AI systems similarly distinguish governance, model, and application-level audits [56]; the contract checks in this paper sit at the application level, auditing whether a deployed answer honors its source, routing, and output obligations.”
A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., & Chadha, A. (2024) · arXiv · doi:10.48550/arxiv.2402.07927 · Cited by 288 · cited 1× · see citing works
“Consistent with this framing, the study evaluates system-level verifiability rather than investment-decision quality; the latter is a separate domain-value question and is out of scope. for shaping model outputs through prompt design [37, 38].”
Assuring the Machine Learning Lifecycle: Desiderata, Methods, and Challenges
Ashmore, R., Calinescu, R., & Paterson, C. (2022) · ACM Computing Surveys · doi:10.1145/3453444 · Cited by 256 · see citing works
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · arXiv · doi:10.48550/arxiv.2305.14627 · Cited by 223 · see citing works
Enabling Large Language Models to Generate Text with Citations
Gao, T., Yen, H., Yu, J., & Chen, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.398 · Cited by 216 · see citing works
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08155 · Cited by 193 · see citing works
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.20 · Cited by 148 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N. F., Zhang, T., & Liang, P. (2023) · arXiv · Cited by 143 · see citing works
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143 · see citing works
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Rebedea, T., Dinu, R., Sreedhar, M. N., Parisien, C., & Cohen, J. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2023.emnlp-demo.40 · Cited by 130 · see citing works
Prompting Is Programming: A Query Language for Large Language Models
Beurer-Kellner, L., Fischer, M., & Vechev, M. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3591300 · Cited by 117 · cited 2× · see citing works
“At the call level, the Language Model Query Language (LMQL) constrains model decoding with declarative constraints [12], Instructor enforces typed output schemas with automatic validation retries [41], and Guidance interleaves control flow with constrained generation within a single model session…”
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · see citing works
RAGAs: Automated Evaluation of Retrieval Augmented Generation
Es, S., James, J., Espinosa-Anke, L., & Schockaert, S. (2023) · arXiv · doi:10.48550/arxiv.2309.15217 · Cited by 79 · cited 1× · see citing works
“RAG evaluation frameworks measure properties such as faithfulness, answer relevance, and context relevance [57, 58], while broader benchmark work supports multi-scenario evaluation [59].”
Evaluation and Benchmarking of LLM Agents: A Survey
Mohammadi, M., Li, Y., Lo, J., & Yip, W. (2025) · Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 · doi:10.1145/3711896.3736570 · Cited by 74 · cited 1× · see citing works
“Surveys of agent evaluation further organize these efforts along capability, reliability, and safety dimensions [60], of which the contract checks in this paper instantiate the reliability and safety dimensions at the product level.”
Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16789 · Cited by 71 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71 · see citing works
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 60 · cited 2× · see citing works
“At the pipeline level, Demonstrate-Search-Predict and its successor DSPy represent multi-step pipelines as declarative programs that can be tested and optimized as software [11, 43, 44].”
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testuggine, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.06674 · Cited by 56 · cited 2× · see citing works
“In the external-guardrail condition, the gate is replaced by a deterministic bolt-on filter that may pass, redact, or refuse, with no deterministic fallback—representative of the pass/redact/refuse behavior of bolt-on moderation guardrails such as Llama Guard and the broader guardrail layer [9, 45].”
Meske, C., Hermanns, T., der Weiden, E. V., et al. (2025) · IEEE Access · doi:10.1109/access.2025.3645466 · Cited by 55 · cited 2×
“Prompt-dominant development is useful for exploration and is related to vibe coding, a conversational LLM-assisted software-development workflow popularized by Karpathy [1, 2].”
Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., & Zaharia, M. (2022) · arXiv · doi:10.48550/arxiv.2212.14024 · Cited by 52 · cited 1× · see citing works
“At the pipeline level, Demonstrate-Search-Predict and its successor DSPy represent multi-step pipelines as declarative programs that can be tested and optimized as software [11, 43, 44].”
PersonaRAG: Enhancing Retrieval-Augmented Generation Systems with User-Centric Agents
Zerhoudi, S., & Granitzer, M. (2024) · IR-RAG@SIGIR · doi:10.48550/arxiv.2407.09394 · Cited by 43
AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents
Zhong, H., & Zhu, S.-X. (2026) · arXiv.org · doi:10.48550/arxiv.2605.13357 · Cited by 22 · cited 1×
“Related efforts define the harness as a runtime substrate, treat code itself as the agent’s harness, or argue that reported agent gains are partly harness-driven rather than model-driven [49–51].”
Building Guardrails for Large Language ModelsPreprint
Dong, Y., Mu, R., Jin, G., Qi, Y., Hu, J., Zhao, X., Meng, J., Ruan, W., & Huang, X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01822 · Cited by 22 · see citing works
Eliza: A Web3 friendly AI Agent Operating System
Walters, S., Gao, S., Nerd, S., et al. (2025) · arXiv.org · doi:10.48550/arxiv.2501.06781 · Cited by 20
Why Do Multi-Agent LLM Systems Fail?
Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.13657 · Cited by 15 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
Retrieval as Reasoning: Self-Evolving Agent-Native Retrieval via LLM-Wiki
Ming, H., Li, F., Wu, X., et al. (2026) · arXiv.org · doi:10.48550/arxiv.2605.25480 · Cited by 9 · cited 1×
“Recent work develops this pattern further—compiling documents into linked wiki pages exposed through tool-calling for agent-native retrieval [53]—but also exposes its central risk: blindly compiling raw documents into a wiki can silently drop critical facts, failing on a large fraction of queries…”
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., & Cohen, J. (2023) · arXiv · doi:10.48550/arxiv.2310.10501 · Cited by 8 · cited 2×
“We take up harness engineering —an emerging practice of wrapping LLM agents in a code-owned control layer [8–10]—and develop it into a measurable, contract-based method that relocates product behavior from prompts into explicit, versioned contracts [11, 12].”
Toward efficient vibe coding: An LLM-based agent for low-code software development
Malamas, N., Tsardoulias, E., Panayiotou, K., & Symeonidis, A. L. (2025) · Journal of Computer Languages · doi:10.1016/j.cola.2025.101367 · Cited by 8 · cited 1× · see citing works
“Vibe coding extends prompt design from isolated task instructions to an iterative development workflow in which software behavior is co-created through natural-language dialogue with AI tools [1, 2, 40].”
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2023) · arXiv · doi:10.48550/arxiv.2311.09476 · Cited by 7 · cited 1× · see citing works
“RAG evaluation frameworks measure properties such as faithfulness, answer relevance, and context relevance [57, 58], while broader benchmark work supports multi-scenario evaluation [59].”
Autonomous AI Agents for Multi-Platform Social Media Marketing: A Simultaneous Deployment Study
Ahn, J., & Kim, M. (2025) · Electronics · doi:10.3390/electronics14214161 · Cited by 7 · see citing works
Self-citationWiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systems
Huerta, J. M. (2026) · arXiv.org · doi:10.48550/arxiv.2605.07068 · Cited by 4 · cited 1×
“…into linked wiki pages exposed through tool-calling for agent-native retrieval [53]—but also exposes its central risk: blindly compiling raw documents into a wiki can silently drop critical facts, failing on a large fraction of queries unless those facts are diagnosed and forced back in [54].”
Meta-Harness: End-to-End Optimization of Model Harnesses
Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026) · arXiv · doi:10.48550/arxiv.2603.28052 · Cited by 3 · cited 1× · see citing works
“…the harness as the layer that coordinates externalized memory, skills, and protocols into governed execution [10], while other work automatically evolves coding-agent harnesses under observability or searches over task-specific harness code using source code, scores, and execution traces [47, 48].”
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
Lin, J., Liu, S., Pan, C., Lin, L., Dou, S., Xi, Z., Huang, X., Yan, H., Han, Z., Gui, T., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25850 · Cited by 1 · cited 1× · see citing works
“…the harness as the layer that coordinates externalized memory, skills, and protocols into governed execution [10], while other work automatically evolves coding-agent harnesses under observability or searches over task-specific harness code using source code, scores, and execution traces [47, 48].”
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
Zhou, C., Chai, H., Chen, W., Guo, Z., Shan, R., Song, Y., Xu, T., Yang, Y., Yu, A., Zhang, W., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.08224 · Cited by 1 · cited 2× · see citing works
“…surveys the emerging area, positioning the harness as the layer that coordinates externalized memory, skills, and protocols into governed execution [10], while other work automatically evolves coding-agent harnesses under observability or searches over task-specific harness code using source code,…”
Ning, X., Tieu, K., Fu, D., Wei, T., Li, Z., Bei, Y., Zou, J., Ai, M., Liu, Z., Li, T.-W., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.18747
Harness Engineering for Language Agents: The Harness Layer as Control, Agency, and RuntimePreprint
He, C., Zhou, X., Wang, D., et al. (2026) · Preprints.org · doi:10.20944/preprints202603.1756.v2
Flagged first, then by citations
Research works printed but not linked: 2
- ISO/IEC/IEEE. Systems and software engineering -vocabulary. Technical Report ISO/IEC/IEEE 24765:2017, International Organization for Standardization, International Electrotechnical Commission, and Institute of Electrical and Electronics Engineers, 2017. URL https://www.iso.org/standard/71952.html. Accessed 2026-07-09.
- Korea Fair Trade Commission. 2026 Korean corporate-group designation results, 2026. URL https://www.ftc.go.kr/www/selectBbsNttView.do?bordCd=3&key=12&nttSn=47410& searchCtgry=01%2C02. Press release with attached PDF; accessed 2026-07-09.
Cited by
Sorted by citations
Ranking Is Not Authority: Executable Context-Routing Contracts for Coding Agents
Mahesh, N. (2026) · Proceedings of the 1st International Workshop on PromptOps and Vibe Coding · doi:10.1145/3843779.3844639
- Authors
- Ahn, J., & Kim, M.
- Affiliations
- Institute for Educational Leadership (US)
- ORCID
- Ahn, Joongho
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2026
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2607.08028
- OpenAlex
- W7167935309
- Semantic Scholar
- 279e4d4fdb168820476c0a19348e5af93ff90291
- Cited by
- 1
- References
- 62
- Citations by year
- 2026: 1
- Type
- posted content
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Keywords
- LLM agents, source grounding, traceability, guardrails, fault injection, productization, over-refusal
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review