The DOI matches its Crossref record; 18 of the 36 references it cites were found in a registry or an index, and 2 are not checked yet; 17 of the 31 sources its claims cite were checked for support.
How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction
Open accessAbstract
This study examines the quality of generative and collaborative AI for fictional prose in Japanese. We rolled out an experiment followed by focus groups to elicit responses to 25 texts with the second half randomly presenting the original text, zero-shot-AI-generated text, AI-generated with prompt specification, and human-AI collaboration using multiple-shot and prompt specification. We found that human-AI collaboration had the highest perceived quality, followed by the original text, AI-generated with prompt specification, and finally zero-shot AI-generated content. While participants mentioned the relative absence of originality in human-AI-collaboration, this content was deemed more human-like than the original text due to the polished editing and selection resulting from inputting drafts, adding or replacing nouns, and rearranging sentences and dialogues. The results presented in this study offer a positive outlook on the use of Generative AI for creative writing, but they also highlight the limitations of these tools for developing highly original literary work.
Verified 2026-10-12 · 2 references not checked yet
Overview
CoChat Summary
TL;DR
This study examines Japanese readers’ rankings of 25 stories whose second halves varied across four conditions: original human writing, zero-shot AI generation, detailed-prompt AI generation, and human-AI collaboration. Human-AI collaboration received the highest mean quality rating but was rated lowest in originality, and the only robust quality contrast was that zero-shot AI generation was rated lower than collaboration.
Key findings
- Human-AI collaboration (D) had the highest mean score at 145.0, followed by original human writing (A) at 139.6, detailed prompt specification (C) at 134.6, and zero-shot generation (B) at 130.7; Table 1 reports A = 139.9, B = 130.4, C = 134.7, D = 145.0.
- The only statistically significant pairwise quality contrast was D vs. B (Cohen’s d = .56); D vs. A (d = .20) and D vs. C (d = .39) were small and not significant, while B vs. A was marginally significant.
- Between-condition variance was .061 versus average within-condition variance of .643, and original human writing showed the highest variance (SD = 32.8) while human-AI collaboration showed the lowest (SD = 20.2), suggesting AI’s main effect is reduced variance rather than higher mean quality.
- Human-AI collaboration was rated highest in perceived quality but lowest in originality; focus groups described it as polished, natural, and “safe” but lacking distinctive originality, while original human writing produced polarized responses.
Methods
- The corpus comprised 25 Japanese stories, each with a shared human-authored first half and four alternative second halves: A original human-written text, B zero-shot ChatGPT-4 generation, C detailed prompt specification, and D human-AI collaboration with human editing; this yielded 100 story versions and over 73,000 Japanese syllables.
- Sixty participants were recruited, with 58 valid responses analyzed; participants had backgrounds in creative writing, Japanese literature, or related fields, and 86.2% were native Japanese speakers. They ranked all four endings per story in randomized order using scores of 10, 7, 4, and 1, between July and August 2024.
- Two focus groups followed in mid-August 2024 with 4 and 3 participants, respectively; discussions were transcribed, anonymized, and translated. Quantitative analysis used average scores, t-tests, and mixed-effects models with lme4 and ordinal packages in R.
Limitations
- The paper states that some Version A excerpts were drawn from early in their source narratives, which may have introduced apparent incoherence where scenes skipped key plot developments; it recommends future studies use longer, more self-contained text samples.
- The paper states that focus groups were small and drawn from the experimental sample; the two sessions also diverged on whether AI-generated content was distinguishable.
- Evident from the text: the sample is not representative of general readers because participants were selected for creative writing or literature backgrounds, and the study is limited to Japanese fiction, short passages, and ChatGPT-4 at a specific time.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 7 Not valid · 9 Not checked · 54 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“These observations are consistent with Gholami et al.'s (2024) findings on AI's restricted memory capacity and its tendency to produce narrative incoherence in longer texts.”
What the sources say
- Partly supportedcites Gholami et al.'s (2024) sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) describes limited memory capacity on a single accelerator when training or serving large language models. That is a hardware constraint, not a finding about how much earlier text an AI can retain; the paper does not report narrative incoherence in longer texts. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“the relative weakness of zero-shot generation is consistent with research on AI's limitations in sustaining narrative coherence (Gholami et al., 2024)”
What the sources say
- Could not verifycites Gholami et al., 2024 sourceExperimental — accuracy not yet measured
The Profiling and Conclusion sections discuss AI models’ training, deployment, and memory bottlenecks, not narrative coherence. Searches for zero-shot generation and coherent story generation found no passage in our copy addressing the claimed limitation. (LLM, conf=0.76)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Peng et al. (2023) found comparable effects among software developers using GitHub Copilot, who completed entry-level tasks 55% faster than the control group.”
What the sources say
- Partly supportedcites Peng et al. (2023) sourceExperimental — accuracy not yet measured
The Introduction reports that the GitHub Copilot group completed a JavaScript HTTP-server task 55.8% faster than the control group, supporting the approximate 55% figure. The Study Design describes one specific task, not a set of tasks characterized as entry-level, so that part of the claim is not established by our copy. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Köbis and Mossink (2021) found a mild aversion to AI-generated poetry even when participants were not explicitly told about its provenance.”
What the sources say
- Supportedcites Köbis and Mossink (2021) source
The Study 1 Discussion (section 6.3) says judges slightly preferred human-written over algorithm-generated poems whether they knew the poems’ origins or not. This directly supports mild aversion in the condition where provenance was not disclosed. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“the absence of a clear quality advantage for human-AI collaboration over original human writing complicates more optimistic accounts of AI's creative potential (Pretsch, 2023; Hutson and Schnellmann, 2023; Çelik, 2023).”
What the sources say
- Not matchedcites Pretsch, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites Hutson and Schnellmann, 2023 source
In “AI and Creative Writing” (page unavailable), the paper reports case studies comparing AI-assisted and unassisted student writing. It says the tool did not necessarily produce better submissions, supporting the claimed absence of a clear quality advantage. The citing sentence’s broader interpretation about AI’s creative potential is its own framing. (LLM, conf=0.86)
- Not matchedcites Çelik, 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
14 more claims
Create a free account to see all 80 claims and every source verdict.
- Probably not a substantial claim. 25
- It may state two separate claims as one. 50
- The quote only partly states this claim. 4
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Experiment
“The t-test shows that the sample mean of 137.5 is significantly greater than the expected average score per respondent (Figures 2 and 3).”
§ Experiment
“We transform this ordinal vector into a numeric vector by assigning a score of 10 to top-ranked stories, 7 to second best, 4 to third best, and 1 point for the version ranked worst.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Conclusion
“First, the more robust experimental finding is that zero-shot AI generation is reliably rated lower than all other conditions;”
§ INSERT FIGURE 1 HERE
“Looking at the pairwise comparisons between versions, we find that D vs B is statistically significant, and that B vs A is marginally significant, but that D vs A and C vs A are not.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Experiment
“Table 1 shows that the average scores favored version D: Human-AI collaboration, with an average score of 145.0 (scores statistically significant at p < .01), followed by A: Original human-written text, which scored an average of 139.6; then C: Detailed prompt specification, which averaged 134.6, and finally B: Zero-shot prompt with 130.7”
§ References
“| A: Original | 139.9 | 32.8 | 81 | 115.0 | 133.0 | 161.0 | 211 | | B: Zero-shot | 130.4** | 29.2 | 69 | 111.0 | 130.0 | 151.0 | 203 | | C: Detailed prompt | 134.7 | 22.7 | 76 | 121.0 | 133.0 | 148.0 | 187 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ INSERT FIGURE 1 HERE
“the total possible range in the scoring spans from a minimum of roughly 106 to a maximum of around 169 per respondent.”
§ References
“| A: Original | 139.9 | 32.8 | 81 | 115.0 | 133.0 | 161.0 | 211 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Recruitment
“The following session took place at 8 PM on August 16, 2024, with three participants.”
§ Focus groups
“participants in the second session were evenly divided between males and females.”
Citations
Cites
Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
Noy, S., & Zhang, W. (2023) · doi:10.2139/ssrn.4375283 · Cited by 2,157
Experimental evidence on the productivity effects of generative artificial intelligence
Noy, S., & Zhang, W. (2023) · Science · doi:10.1126/science.adh2586 · Cited by 2,150
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406
Gholami, A., Yao, Z., Kim, S., Hooper, C., Mahoney, M. W., & Keutzer, K. (2024) · IEEE Micro · doi:10.1109/mm.2024.3373763 · Cited by 369
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Hitsuwari, J., Ueda, Y., Yun, W., & Nomura, M. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107502 · Cited by 267
Is Chatgpt a menace for creative writing ability? An experiment
Niloy, A. C., Akter, S., Sultana, N., Sultana, J., & Rahman, S. I. U. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12929 · Cited by 198
Rethinking creativity: creative industries, AI and everyday creativity
Lee, H.-K. (2022) · Media, Culture & Society · doi:10.1177/01634437221077009 · Cited by 194
AI-generated vs human-authored texts: A multidimensional comparison
Berber Sardinha, T. (2024) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2023.100083 · Cited by 139
AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably
Porter, B., & Machery, E. (2024) · Scientific Reports · doi:10.1038/s41598-024-76900-1 · Cited by 133
Metadata
- Authors
- Asano, Y., & Bastos, M.
- Affiliations
- City St George's, University of London (GB)
- Department of Media, Culture and Creative Industries, City St George’s, University of London, UK
- ORCID
- Bastos, Marco
- Venue
- Journal of Creative Communications
- Publisher
- SAGE Publications
- Published
- 2026-08-30
- ISSN
- 0973-2586 (ISSN-L)
- 0973-2594
- OpenAlex
- W7133554842
- Semantic Scholar
- e79c0d9fd4b096686294e8667cf3a732e606ec1b
- Cited by
- 0
- References
- 34
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- creativecommons.org/licenses/by-nc/4.0/ (vor)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Journal of Creative Communications (publishedVersion, cc-by-nc)
- AoIR Selected Papers of Internet Research (publishedVersion)
- SSRN Electronic Journal (acceptedVersion)
- City Research Online (City University London) (submittedVersion)
- Funders
- University College Dublin (R20825, R21650)
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (406504/2022-9, /2022-9)
- Topics
- Social Sciences › Law › Literary and Philosophical Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence in Games
- Social Sciences › Arts and Humanities › Visual Arts and Performing Arts › Artistic and Creative Research
- Keywords
- generative AI, human-AI collaboration, AI-generated text, reader perception, perceived quality, originality, creative writing, AI-assisted editing, stylistic expression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction. https://app.cochat.ai/papers/10.1177/09732586261463934
- The paper
- Asano, Y., & Bastos, M. (2026). How Creative Is AI Writing? Generative and Collaborative AI in Japanese Fiction. Journal of Creative Communications. https://doi.org/10.1177/09732586261463934
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.