CTRL: A Conditional Transformer Language Model for Controllable Generation
Open accessAbstract
Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model-based source attribution. We have released multiple full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl.
AI snapshot
Written from the paper’s full text.
TL;DR
CTRL is a 1.63 billion-parameter conditional Transformer language model trained to generate text conditioned on control codes derived from naturally co-occurring structure such as domains, URLs, subreddits, ratings, and task templates. It is released with pretrained models and supports controllable generation plus model-based source attribution over training domains.
Key findings
- CTRL conditions on control codes to govern style, content, and task behavior; for identical prompts, different domain codes produce predictable variation, and generation can proceed with no prompt when a code is supplied.
- Control codes can be combined for finer control (e.g., `Reviews`, `Rating:`, and a value) and mixed zero-shot across domains/tasks (e.g., diet subreddit with English-German translation codes; politics subreddit with a French prompt) even when that combination was not in training.
- Domain control codes partition the training data and allow ranking which subsets are most likely given a sequence; the paper uses a uniform prior over domain codes because the empirical prior overweights large-data domains.
- The paper proposes penalized sampling that discounts previously generated tokens; greedy sampling with θ ≈ 1.2 is reported as a good balance between truthful generation and lack of repetition.
Methods
- Model: conditional language model \(p(x|c)\) trained on raw text prepended with control codes; Transformer with \(d=1280\), inner dimension \(f=8192\), 48 layers, 16 heads per layer, dropout 0.1 after residual connections, tied token/output embeddings, BPE vocabulary of roughly 250K tokens, and sequence lengths of 256 or 512.
- Data and training: 140 GB of text from Wikipedia (En, De, Es, Fr), Project Gutenberg, 45 subreddits, OpenWebText, news data, Amazon Reviews, Europarl/UN WMT data, and QA pairs from ELI5 and MRQA; trained in TensorFlow with global batch size 1024 across 256 Cloud TPU v3 cores for 800k iterations (~2 weeks), using Adagrad with linear warmup from 0 to 0.05 over 25k steps and gradient norm clipping to 0.25.
- Analysis: qualitative generation examples across control codes; source attribution by ranking domains using \(p(c|x)\) with a uniform prior over domain control codes; sampling discussed via temperature/top-k/nucleus and the proposed penalized sampling scheme.
Limitations
- Stated: training data does not have universal coverage and contains cultural associations from original sources; the model is biased toward patterns in its training data, and source attribution is descriptive only, not a measure of veracity or a basis for normative claims.
- Stated: source attribution is sensitive to small changes in the prompt, and sequence lengths were limited to 256 or 512 due to memory and optimization constraints, with a sliding-window approach used for longer generation.
- Evident from the text: evaluation is largely qualitative through generation examples and attribution tables; no quantitative benchmark, perplexity, or human-evaluation results are reported.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
4 of 10 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ 4.1 SAMPLING · p. 4
Greedy sampling is known to create repetitions of phrases or sentences even for large well-trained models.
“However, this is known to create repetitions of phrases or sentences even for large well-trained models (Radford et al., 2019; Holtzman et al., 2019).”
Quote in the paperSubstantial claim1 of 2 sources checked- cites Radford et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Holtzman et al., 2019 source
- Findingnone given§ ABSTRACT · p. 1
CTRL is a 1.63 billion-parameter conditional transformer language model trained to condition on control codes that govern style, content, and task-specific behavior.
“We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior.”
Quote in the paperSubstantial claim - Findingnone given§ 4.2 CONTROL CODES · p. 5
During training, CTRL learns relationships between the structure of URLs and the text that follows.
“During training, CTRL learns relationships between the structure of these URLs and the text that follows.”
Quote in the paperSubstantial claim- Rests on an assumption: In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
- Findingreasoning§ 4.2 CONTROL CODES · p. 5
At inference, novel URLs can be used to specify a variety of features: domain, subdomain, entities, entity relations, and even dates.
“At inference, novel URLs can be used to specify a variety of features: domain, subdomain, entities, entity relations, and even dates.”
Quote in the paperSubstantial claim- Rests on an assumption: In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
- Findingown data§ 4.2 CONTROL CODES · p. 5
When the politics subreddit code is mixed with a prompt that starts in French, CTRL generates text despite no examples of this kind being found in the training data.
“In the second example we mix the politics subreddit (r/politics) with a prompt that starts in French though no examples of this kind were found in the training data.”
Quote in the paperNot a substantial claim
- Findingown data§ 4.1 SAMPLING · p. 4
For the prompt 'Q: What is the capital of Australia?', a well-trained model assigns higher probability mass to the correct answer Canberra but non-zero probability mass to other cities such as Melbourne, Sydney, Brisbane, Darwin, and Perth.
“Given a prompt: Q: What is the capital of Australia? , a well-trained model assigns higher probability mass to the correct answer, Canberra, but a non-zero probability mass to other cities such as Melbourne, Sydney, Brisbane, Darwin, and Perth, see Figure 1.”
Quote in the paperSubstantial claim - Findingown data§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 9
Mixing the Politics domain with a French prompt generates text despite the model never seeing this combination in training.
“In the second example, the Politics domain is mixed with a French prompt despite never seeing this combination in training.”
Quote in the paperSubstantial claim - Findingown data§ 4.2 CONTROL CODES · p. 5
Even for identical prompts, control codes allow for predictable variation in generation.
“Examples in Table 1 demonstrate that even for identical prompts, control codes allow for predictable variation in generation.”
- Also stated in: § 4.2 CONTROL CODES · p. 6 · plain
“Even for identical prompts , control codes allow for predictable variation in generation.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 CONTROL CODES · p. 6 · plain
- Findingnone given§ 5 SOURCE ATTRIBUTION · p. 9
Domain control codes can be used to partition the training data into mutually exclusive sets.
“The domain control codes can be used to partition the training data into mutually exclusive sets.”
Quote in the paperSubstantial claim - Findingnone given§ 4.2 CONTROL CODES · p. 5
Task-specific control codes constrain the generation process the most by triggering task-specific generation.
“These codes constrain the generation process the most, by triggering task-specific generation.”
Quote in the paperSubstantial claim - Findingnone given§ 4.2 CONTROL CODES · p. 5
Most control codes for CTRL specify the overall style of generated text by indicating a particular domain of training data.
“Most control codes for our model specify the overall style of generated text by indicating a particular domain of training data.”
Quote in the paperNot a substantial claim - Findingnone given§ 9 CONCLUSION · p. 12
CTRL has 1.63 billion parameters.
“With 1.63 billion parameters, CTRL is the largest publicly released language model to date.”
Quote in the paperSubstantial claim - Findingown data§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 9
Some control codes can be mixed to generate text with novel cross-over behavior.
“Some codes can be mixed to generate text with novel cross-over behavior.”
Quote in the paperSubstantial claim - Findingown data§ 4.2 CONTROL CODES · p. 5
When a diet subreddit code is mixed with machine translation control codes for English and German, the generated text is coherent across multiple translated lines.
“In the first example we mix a diet subreddit (r/keto) with machine translation control codes for English and German. In contrast to using Translation in 2, the generated text with mixed codes is coherent across multiple translated lines.”
Quote in the paperNot a substantial claim - Findingown data§ 4.2 CONTROL CODES · p. 5
Additional control codes can be added to the domain code to increasingly constrain generation.
“Additional control codes can be added to the domain code in order to increasingly constrain generation.”
Quote in the paperSubstantial claim - Findingown data§ 5 SOURCE ATTRIBUTION · p. 9
The empirical prior of the training data weights domains with large amounts of data too heavily.
“We found that the empirical prior of the training data weights domains with large amounts of data too heavily.”
Quote in the paperSubstantial claim - Findingnone given§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 8
The token 'us-president' is interpreted differently by CTRL depending on the date used (2007, 2014, 2018).
“Note that us-president is interpreted differently by the model depending on the date used ( 2007 , 2014 , 2018) .”
Quote in the paperNot a substantial claim- Rests on an assumption: In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
- Findingown data§ 4.2 CONTROL CODES · p. 5
CTRL can generate domain-specific text without any prompt.
“The examples in Table 2 show how CTRL can generate domain-specific text without any prompt.”
- Also stated in: § 4.2 CONTROL CODES · p. 7 · plain
“With CTRL, no prompt is necessary as long as a control code is provided.”
Quote in the paperSubstantial claim - Also stated in: § 4.2 CONTROL CODES · p. 7 · plain
- Findingown data§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 8
CTRL is trained with links as control codes.
“CTRL is trained with links as control codes .”
Quote in the paperSubstantial claim - Findingreasoning§ ABSTRACT · p. 1
CTRL's control codes allow it to predict which parts of the training data are most likely given a sequence.
“These codes also allow CTRL to predict which parts of the training data are most likely given a sequence.”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“Because all control codes can be traced back to a particular subset of the training data, CTRL can be used to predict the subset of training data that is most likely given a sequence.”
Quote in the paperSubstantial claim - Also stated in: § 1 INTRODUCTION · p. 2 · plain
- Findingnone given§ 4.1 SAMPLING · p. 5
Using greedy sampling and θ ≈ 1.2 yields a good balance between truthful generation and lack of repetition.
“We find that using a greedy sampling and θ ≈ 1 . 2 yields a good balance between truthful generation and lack of repetition.”
Quote in the paperNot a substantial claim - Findingnone given§ 1 INTRODUCTION · p. 1
With 1.63 billion parameters, CTRL can generate text conditioned on control codes that specify domain, style, topics, dates, entities, relationships between entities, plot points, and task-related behavior.
“With 1.63 billion parameters, our Conditional Transformer Language (CTRL) model can generate text conditioned on control codes that specify domain, style, topics, dates, entities, relationships between entities, plot points, and task-related behavior.”
Quote in the paperSubstantial claim - Findingnone given§ 4.2 CONTROL CODES · p. 5
A small number of control codes are related to specific tasks like question answering and translation.
“A small number of control codes are related to specific tasks like question answering and translation.”
Quote in the paperNot a substantial claim - Findingnone given§ 9 CONCLUSION · p. 12
CTRL is trained with control codes.
“It is trained with control codes so that text generation can be more easily controlled by human users.”
Quote in the paperSubstantial claim - Findingreasoning§ 5 SOURCE ATTRIBUTION · p. 9
CTRL provides model-based evidence that certain domains are more likely to contain language similar to given statements.
“CTRL provides model-based evidence that certain domains are more likely to contain language similar to given statements, but it should not be used to make normative or prescriptive claims.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
By specifying a prior over domain control codes for p(c), it is straightforward to compute a ranking of domains.
“By specifying a prior over domain control codes for p ( c ) , it is straightforward to compute a ranking of domains:”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
The partition by domain control codes supports a simple method for determining which subsets of the training data the language model considers most likely given a sequence.
“This supports a simple method for determining which subsets of the training data the language model considers most likely given a sequence.”
Quote in the paperSubstantial claim - Interpretationreasoning§ ABSTRACT · p. 1
CTRL's control codes provide a potential method for analyzing large amounts of data via model-based source attribution.
“This provides a potential method for analyzing large amounts of data via model-based source attribution.”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“This explicit relationship between CTRL and its training data can be exploited to analyze the correlations that the language model has learned from each domain, and it provides a means of studying large amounts of text through the language model.”
Quote in the paperSubstantial claim - Also stated in: § 1 INTRODUCTION · p. 2 · plain
- Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
All applications of CTRL inherently depend on those original associations for prediction.
“All applications of the model inherently depend on those original associations for prediction.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
The source attribution method relies on exploiting the original associations to establish relationships between the language model and its training data.
“In fact, this method of source attribution relies on exploiting the original associations to establish relationships between the language model and its training data.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 7 FUTURE DIRECTIONS · p. 11
CTRL is biased towards the patterns of language used in the training data.
“Therefore the model is biased towards the patterns of language used in the training data.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 8
Users can mimic the structure of URLs that appear during training to create novel content during generation.
“The links in these examples do not actually link to text; users can mimic the structure of the URLs that appear during training to create novel content during generation.”
Quote in the paperSubstantial claim- Rests on an assumption: In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
- Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
CTRL does not have a notion of whether any particular cultural association is good or bad, right or wrong, true or false.
“The model does not have a notion of whether any particular cultural association is good or bad, right or wrong, true or false.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 3 LANGUAGE MODELING WITH CTRL · p. 2
The control code c provides a point of control over the generation process, including when sampling the first token x0.
“The control code c provides a point of control over the generation process. This is true even when sampling x 0 , in contrast to the traditional language modeling framework described in Sec. 2.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9
CTRL only learns correlations between cultural associations and domains.
“It only learns correlations between cultural associations and domains.”
Quote in the paperSubstantial claim - Study design and conductreasoning§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 11
The authors released all trained CTRL models.
“We reify these principles by releasing all trained CTRL models.”
Quote in the paperNot a substantial claim
- Classified as probably not a substantial claim. 32
- May combine two claims. 15
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 7 FUTURE DIRECTIONS · p. 11For important skills, inclusion of supervised data or task-specific data generated through unsupervised means can lead to obvious improvements.
“For important skills, the inclusion of supervised data or task-specific data generated through unsupervised means (Artetxe et al., 2017; Lewis et al., 2019) can lead to obvious improvements.”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Artetxe et al., 2017 source
- cites Lewis et al., 2019
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingcitation§ 6 RELATED WORK · p. 10Relying on inference-time methods that are closer in behavior to context-based losses provides a reasonable solution to the trade-off between novelty and consistency when prioritizing control, as long as the language model's distribution is sufficiently confident in its decisions.
“this work found that relying on inference-time methods (Fan et al., 2018; Holtzman et al., 2019) that are closer in behavior to context-based losses (See et al., 2017; Welleck et al., 2019) provides a reasonable solution as long as the distribution of the language model is sufficiently confident in its decisions.”
Quote in the paperNot a substantial claim2 of 3 sources checked- cites Fan et al., 2018
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Holtzman et al., 2019 source
Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured - cites See et al., 2017 source
Probably not a substantial claim: may not pass S4 (based).
Findingreasoning§ 1 INTRODUCTION · p. 2Control codes for question answering and machine translation make these skills easily accessible with CTRL.
“Control codes for question answering and machine translation make these skills easily accessible with CTRL.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone given§ ABSTRACT · p. 1Multiple full-sized, pretrained versions of CTRL have been released at https://github.com/salesforce/ctrl.
“We have released multiple full-sized, pretrained versions of CTRL at https://github.com/salesforce/ctrl .”
- Also stated in: § 1 INTRODUCTION · p. 2 · plain
“we have released multiple full-sized, pretrained versions of CTRL at https://github.com/ salesforce/ctrl .”
Quote in the paperNot a substantial claim- Also stated in: § 1 INTRODUCTION · p. 2 · plain
May combine two claims.
Findingown data§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 8The token 'star' is interpreted differently based on the domain (cnn vs. etonline), and topic (style vs. politics) can be varied even for identical entities (george-clooney).
“Similarly, star is interpreted differently based on the domain ( cnn vs. etonline ) and topic ( style vs. politics ) can be varied even for identical entities ( george-clooney ).”
Quote in the paperSubstantial claim- Rests on an assumption: In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
Classified as probably not a substantial claim. May combine two claims.
Findingnone given§ 5 SOURCE ATTRIBUTION · p. 9The data used to train CTRL does not have universal coverage and contains the cultural associations present in the original sources.
“We note that the data used to train this model does not have universal coverage and contains the cultural associations present in the original sources.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingown data§ 9 CONCLUSION · p. 12CTRL is the largest publicly released language model to date.
“With 1.63 billion parameters, CTRL is the largest publicly released language model to date.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone given§ 1 INTRODUCTION · p. 2Control codes allow for the straightforward inclusion of task-specific data in a way that improves important skills without harming the generality of the model.
“These control codes also allow for the straightforward inclusion of task-specific data in a way that improves important skills without harming the generality of the model.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4.2 CONTROL CODES · p. 5Control codes can specify that the model should start with a title and specify a particular rating for reviews.
“These examples demonstrate constraints specifying that the model should start with a title and by specifying a particular rating for reviews.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ 5 SOURCE ATTRIBUTION · p. 10The source attribution procedure is sensitive to small changes in the prompt.
“Note that this procedure is sensitive to small changes in the prompt.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 9Mixing translation codes into the Diet domain causes the model to alternately generate English and German sentences while respecting the Diet domain and remaining coherent across translations.
“In the first example, we mix translation codes into the Diet domain. By doing so, the model continues alternatively generates English and German sentences while respecting the Diet domain and remains coherent across translations.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingreasoning§ 9 CONCLUSION · p. 12CTRL's control codes allow text generation to be more easily controlled by human users.
“It is trained with control codes so that text generation can be more easily controlled by human users.”
Quote in the paperSubstantial claim- Rests on an assumption: The chosen control codes represent a reasonably large variety in control over domain, topic, entities, entity relations, and dates.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Findingnone given§ 7 FUTURE DIRECTIONS · p. 11CTRL is designed to make the interface between humans and language models more intuitive.
“CTRL is designed to make the interface between humans and language models more intuitive.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 3.2 EXPERIMENTAL SETTINGS · p. 3Training CTRL with sequence lengths of 256 and 512 showed that a sliding-window approach allows generation beyond these windows, and there was little difference in quality between the two models within the first 256 tokens.
“We experimented with sequence lengths of 256 and 512 due to memory and optimization constraints. Despite training on relatively short sequences compared to other approaches, we found that a sliding-window approach allows for generation beyond these windows, and we also found little difference in quality between the two models within the first 256 tokens.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone given§ 5 SOURCE ATTRIBUTION · p. 10Attribution experiments use the model trained on sequences of length 256, which was trained longer and provided better estimation of source.
“Attribution experiments use the model trained on sequences of length 256 ; it was trained longer and provided better estimation of source.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone given§ 1 INTRODUCTION · p. 1Text can be generated in more predictable ways by controlling for content or changing the domain even when the initial prompt remains fixed.
“Text can be generated in more predictable ways by controlling for content or changing the domain even when the initial prompt remains fixed.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone given§ 9 CONCLUSION · p. 12CTRL's control codes allow users to explicitly specify domain, subdomain, entities, relationships between entities, dates, and task-specific behavior.
“These codes allow users to explicitly specify domain, subdomain, entities, relationships between entities, dates, and task-specific behavior.”
Quote in the paperSubstantial claim- Rests on an assumption: The chosen control codes represent a reasonably large variety in control over domain, topic, entities, entity relations, and dates.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 4.2 CONTROL CODES · p. 5Complex control codes for question answering and machine translation act as a template mixed with a natural language prompt.
“In Table 4, we demonstrate relatively complex control codes for question answering and machine translation that act as a template mixed with a natural language prompt.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ 7 FUTURE DIRECTIONS · p. 11CTRL is trained on a small subset of the possible data available.
“CTRL is trained on a small subset of the possible data available.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given§ 7 FUTURE DIRECTIONS · p. 11CTRL offers an explicit method for analyzing the relationship between the model and its current training data.
“CTRL offers an explicit method for analyzing the relationship between the model and its current training data.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Findingnone given§ Links https://www.cnn.com/style/09/20/2018/george-clooney-interview · p. 8Links provide a way to specify domain, subdomain, entities, entity relations, and even date.
“Links provide a way to specify domain, subdomain, entities, entity relations, and even date.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Findingnone given§ 3 LANGUAGE MODELING WITH CTRL · p. 2CTRL is a conditional language model that is always conditioned on a control code c and learns the distribution p(x|c).
“CTRL is a conditional language model that is always conditioned on a control code c and learns the distribution p ( x | c ) .”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone given§ 1 INTRODUCTION · p. 2Control codes can be combined during generation to create novel cross-over between task-specific behavior and domain/content codes.
“These codes can be combined with codes during generation to create novel cross-over between control codes that are task-specific behavior and those that are related to domain and content.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingreasoning§ 5 SOURCE ATTRIBUTION · p. 9Contradictory statements are often attributed to the same sources, and competing claims often appear in the same contexts.
“This is evidenced by the fact that contradictory statements are often attributed to the same sources: competing claims often appear in the same contexts.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone given§ 1 INTRODUCTION · p. 1Humans can use CTRL's control codes to trigger generation of text from different linguistic communities without having to understand how to prompt with particular linguistic patterns.
“Humans can use these codes to trigger generation of text from different linguistic communities without having to understand how to prompt with particular linguistic patterns.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 12Prior to release, the research team conducted a technology foresight exercise to anticipate possible malicious use cases.
“Prior to release, the research team conducted a technology foresight exercise to anticipate possible malicious use cases.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Premisecitation§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 12An anticipatory focus on possibilities rather than probabilities lessens several shortcomings of formal risk assessment in the face of contested assumptions, which has proven ineffective in identifying the most profound future impacts of innovation.
“This anticipatory focus on possibilities rather than probabilities lessens several shortcomings of formal risk assessment in the face of contested assumptions, which has proven ineffective in identifying the most profound future impacts of innovation (Stilgoe et al., 2013).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Stilgoe et al., 2013 source
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 7 FUTURE DIRECTIONS · p. 11Including data for specific tasks need not harm the general nature of an unsupervised learning process.
“This work suggests that including data for specific tasks need not harm the general nature of an unsupervised learning process.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 11The authors' actions reflect principles from a recent sociology-based AI governance framework that aims to expand responsible innovation to consider networks of users, dynamics, and feedback.
“Our actions reflect principles from a recent sociology-based AI governance framework that aims to expand responsible innovation to consider networks of users, dynamics, and feedback (Varshney et al., 2019).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Varshney et al., 2019 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 5 SOURCE ATTRIBUTION · p. 9CTRL is a descriptive tool for analyzing correlations in large amounts of text.
“It is a descriptive tool for analyzing correlations in large amounts of text.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.
Premisecitation§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 11Much scientific research and innovation can benefit the public but may also be diverted to harmful uses or have unintended negative impacts.
“Although much scientific research and innovation can benefit the public, it may also be diverted to harmful uses or have unintended negative impacts (without animus).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Brundage et al. (2019)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationreasoning§ 3.2 EXPERIMENTAL SETTINGS · p. 3CTRL's vocabulary is approximately 4 times larger than similar approaches, making its effective sequence length in characters comparable.
“Further, we note that our vocabulary is approximately 4 times larger than similar approaches, hence the effective sequence length in characters is comparable.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given§ 8 CTRL-ALT-DEL: THE ETHICS OF LARGE LANGUAGE MODELS · p. 12Rather than self-governance, the authors sought to diversify inputs to governance through prerelease review from experts at the Partnership on AI.
“Rather than self-governance, we sought to diversify inputs to governance through prerelease review from experts at the Partnership on AI (PAI).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone given§ 4.1 SAMPLING · p. 5The penalized sampling approach succeeds only if the model has learned a sufficiently reliable distribution.
“We note in passing that this approach succeeds only if the model has learned a sufficiently reliable distribution.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 7 FUTURE DIRECTIONS · p. 11The training data is likely not representative of many linguistic communities.
“The data is likely not representative of many linguistic communities”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone given§ 5 SOURCE ATTRIBUTION · p. 10Source attribution cannot be considered a measure of veracity, but only a measure of how much each domain token influences a given sequence.
“Source attribution cannot be considered a measure of veracity, but only a measure of how much each domain token influences a given sequence.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone given§ ABSTRACT · p. 1Control codes for CTRL were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation.
“Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 4.2 CONTROL CODES · p. 5The coherence of the mixed diet/translation generation is an influence of the Diet domain because it had multiline examples in the training data, whereas the translation data consisted of shuffled single lines.
“This structure is an influence of Diet because it had multiline examples in the training data, whereas the translation data consisted of shuffled single lines.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text.
“Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text.”
The paper relies on this without support.
In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.
“In the vast majority of cases, text collected for training is associated with a URL, which often contains information pertinent to the text it represents.”
The paper relies on this without support.
The chosen control codes represent a reasonably large variety in control over domain, topic, entities, entity relations, and dates.
“The particular choice of control codes in this work is intended to represent a reasonably large variety in control over domain, topic, entities, entity relations, and dates.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The repeating problem has been observed in nearly all text-generation models.
“The repeating problem has been observed in nearly all text-generation models17,18”
From AI models collapse when trained on recursively generated data (2024) · cites this paper as [17]
The Related Work section (p. 10) identifies repetition as a problem in text generation and discusses methods for reducing it. It does not say the problem has been observed in nearly all text-generation models; that prevalence claim is unsupported. (LLM, conf=0.86)
Experimental — accuracy not yet measuredTraining on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models.
“Similar to multitask learning, training on heterogenous data annotated with control codes has been shown to improve controlled generation in autoregressive language models (Keskar et al., 2019; ElSahar et al., 2020; Sudhakar et al., 2019; Li et al., 2020).”
From TLDR: Extreme Summarization of Scientific Documents (2020) · cites this paper as Keskar et al., 2019
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
(2010) · International Conference on Machine Learning · Cited by 13,995 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
(2010) · Cited by 8,261 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., & Janvin, C. (2003) · Journal of machine learning research · doi:10.1162/153244303322533223 · Cited by 8,257 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
(2017) · International Conference on Machine Learning · Cited by 6,382 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775 · see citing works
InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial NetsPreprint
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.03657 · Cited by 3,379 · see citing works
Developing a framework for responsible innovation
Stilgoe, J., Owen, R., & Macnaghten, P. (2013) · Research Policy · doi:10.1016/j.respol.2013.05.008 · Cited by 3,184 · see citing works
A Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1044 · Cited by 2,879 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
Poplack, S. (1980) · Linguistics · doi:10.1515/ling.1980.18.7-8.581 · Cited by 2,579 · see citing works
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · see citing works
Image-Based Recommendations on Styles and Substitutes
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2766462.2767755 · Cited by 2,348 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F. B., WATTENBERG, M. P., Corrado, G. S., et al. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00065 · Cited by 1,842 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586 · see citing works
Adafactor: Adaptive Learning Rates with Sublinear Memory CostPreprint
Shazeer, N., & Stern, M. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1804.04235 · Cited by 1,410 · see citing works
The Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · see citing works
Hierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · see citing works
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
Transformer-XL: Attentive Language Models Beyond a Fixed-Length ContextPreprint
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019) · arXiv (Cornell University) · doi:10.4230/lipics.ecrts.2023.7 · Cited by 860 · see citing works
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.07228 · Cited by 849 · see citing works
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. (2018) · Apollo - University of Cambridge Repository · doi:10.17863/cam.22520 · Cited by 849 · see citing works
Constraints on Language Mixing: Intrasentential Code-Switching and Borrowing in Spanish/English
Pfaff, C. W. (1979) · Language · doi:10.2307/412586 · Cited by 779 · see citing works
Regularizing and Optimizing LSTM Language Models
Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697 · see citing works
Self-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · see citing works
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 642 · see citing works
Unsupervised Neural Machine Translation
Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1710.11041 · Cited by 632 · see citing works
New York Times Annotated Corpus
Sandhaus, E., & Linguistic Data Consortium (2012) · TIB Data Manager · doi:10.57702/qbrk4fuh · Cited by 624 · see citing works
Multi-task Sequence to Sequence LearningPreprint
Luong, M.-T., Le, Q. V., Sutskever, I., Vinyals, O., & Kaiser, Ł. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06114 · Cited by 606 · see citing works
Large Language Models in Machine Translation
Brants, T., Popat, A. C., Xu, P., Och, F. J., & Dean, J. B. (2007) · Cited by 574 · see citing works
Defending Against Neural Fake News
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1905.12616 · Cited by 569 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562 · see citing works
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
Hashimoto, K., xiong, C., Tsuruoka, Y., & Socher, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1206 · Cited by 535 · see citing works
Self-citationNewsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies
Grusky, M., Naaman, M., & Artzi, Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1065 · Cited by 509 · see citing works
Findings of the 2019 Conference on Machine Translation (WMT19)
Barrault, L., Bojar, O., Costa‐jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., et al. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5301 · Cited by 461 · see citing works
The Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448 · see citing works
Self-citationFactSheets: Increasing trust in AI services through supplier's declarations of conformity
Arnold, M., Bellamy, R. K. E., Hind, M., Houde, S., Mehta, S., Mojsilović, A., Nair, R., Ramamurthy, K. N., Olteanu, A., Piorkowski, D., et al. (2019) · IBM Journal of Research and Development · doi:10.1147/jrd.2019.2942288 · Cited by 434 · see citing works
Explain Yourself! Leveraging Language Models for Commonsense Reasoning
Rajani, N. F., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1487 · Cited by 421 · see citing works
Self-citationAbstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
Nallapati, R. M., Zhou, B., dos Santos, C. N., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.06023 · Cited by 371 · see citing works
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359 · see citing works
Fine-tuned Language Models for Text Classification.Preprint
Howard, J., & Ruder, S. (2018) · arXiv (Cornell University) · Cited by 357 · see citing works
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2021068 · Cited by 346 · see citing works
Neural Text Summarization: A Critical Evaluation
Kryscinski, W., Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1051 · Cited by 335 · see citing works
Self-citationELI5: Long Form Question Answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · arXiv · doi:10.48550/arxiv.1907.09190 · Cited by 325 · see citing works
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
Tying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · see citing works
Self-citationA Simple Method for Commonsense Reasoning
Trinh, T. H., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1806.02847 · Cited by 293 · see citing works
Fast Decoding in Sequence Models using Discrete Latent VariablesPreprint
Kaiser, Ł., Roy, A., Vaswani, A., Parmar, N., Bengio, S., Uszkoreit, J., & Shazeer, N. (2018) · International Conference on Machine Learning · doi:10.48550/arxiv.1803.03382 · Cited by 260 · see citing works
Neural Text Generation with Unlikelihood Training
Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., & Weston, J. (2019) · arXiv · doi:10.48550/arxiv.1908.04319 · Cited by 254 · see citing works
Kaiser, L., Gomez, A. N., Shazeer, N., Vaswani, A., Parmar, N., Jones, L., & Uszkoreit, J. (2017) · arXiv · doi:10.48550/arxiv.1706.05137 · Cited by 253 · see citing works
Caswell, I., Chelba, C., & Grangier, D. (2019) · Conference on Machine Translation · doi:10.18653/v1/w19-5206 · Cited by 241 · see citing works
Unsupervised Question Answering by Cloze Translation
Lewis, P., Denoyer, L., & Riedel, S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1484 · Cited by 146 · see citing works
Increasing Trust in AI Services through Supplier's Declarations of Conformity
Hind, M., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv.org · doi:10.48550/arxiv.1808.07261 · Cited by 134
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., & Manning, C. D. (2018) · arXiv · doi:10.48550/arxiv.1809.09600 · Cited by 97 · see citing works
Artificial Intelligence and Responsible Innovation
Brundage, M. (2016) · Synthese Library/Synthese library · doi:10.1007/978-3-319-26485-1_32 · Cited by 94 · see citing works
Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep NetworksPreprint
Ginsburg, B., Castonguay, P., Hrinchuk, O., Kuchaiev, O., Lavrukhin, V., Leary, R., Li, J., Nguyen, H., Zhang, Y., & Cohen, J. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.11286 · Cited by 83 · see citing works
Large Memory Layers with Product Keys
Lample, G., Sablayrolles, A., Ranzato, M., Denoyer, L., & Jégou, H. (2019) · arXiv · doi:10.48550/arxiv.1907.05242 · Cited by 72 · see citing works
NewsQA: A Machine Comprehension DatasetPreprint
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.09830 · Cited by 65 · see citing works
Unifying Question Answering, Text Classification, and Regression via Span ExtractionPreprint
Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09286 · Cited by 36 · see citing works
Self-citationFilling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection
Moryossef, A., Aharoni, R., & Goldberg, Y. (2019) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/w19-3807 · Cited by 35 · see citing works
Memory-Efficient Adaptive OptimizationPreprint
Anil, R., Gupta, V., Koren, T., & Singer, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.11150 · Cited by 33 · see citing works
Answers Unite! Unsupervised Metrics for Reinforced Summarization ModelsPreprint
Scialom, T., Lamprier, S., Piwowarski, B., & Staiano, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01610 · Cited by 24 · see citing works
Pretrained AI Models: Performativity, Mobility, and Change
Varshney, L. R., Keskar, N. S., & Socher, R. (2019) · arXiv · doi:10.48550/arxiv.1909.03290 · Cited by 20 · see citing works
Self-citationUnifying Question Answering and Text Classification via Span Extraction.Preprint
Keskar, N. S., McCann, B., Xiong, C., & Socher, R. (2019) · arXiv (Cornell University) · Cited by 20 · see citing works
Self-citationDomain Control for Neural Machine TranslationPreprint
Kobus, C., Crego, J., & Sénellart, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06140 · Cited by 16 · see citing works
Kaiser, D. I., & Moreno, J. D. (2012) · Nature · doi:10.1038/492345a · Cited by 11 · see citing works
Memory-Efficient Adaptive Optimization for Large-Scale Learning.
Anil, R., Gupta, V., Koren, T., et al. (2019) · Cited by 9 · see citing works
SumQE: a BERT-based Summary Quality Estimation ModelPreprint
Xenouleas, S., Malakasiotis, P., Apidianaki, M., & Androutsopoulos, I. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00578 · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Deep contextualized word representations
Peters, M. E., Neumann, M., Iyyer, M., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.05365
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · arXiv · doi:10.48550/arxiv.1801.06146
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2016) · arXiv · doi:10.48550/arxiv.1608.05859
Explain Yourself! Leveraging Language Models for Commonsense Reasoning
Rajani, N. F., McCann, B., Xiong, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1906.02361
Self-citationA Neural Attention Model for Abstractive Sentence Summarization
Rush, A. M., Chopra, S., & Weston, J. (2015) · arXiv · doi:10.48550/arxiv.1509.00685
Caswell, I., Chelba, C., & Grangier, D. (2019) · arXiv · doi:10.48550/arxiv.1906.06442
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
Hashimoto, K., Xiong, C., Tsuruoka, Y., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.01587
Self-citationNeural Text Summarization: A Critical Evaluation
Kryściński, W., Keskar, N. S., McCann, B., et al. (2019) · arXiv · doi:10.48550/arxiv.1908.08960
Self-citationUnsupervised Question Answering by Cloze Translation
Lewis, P., Denoyer, L., & Riedel, S. (2019) · arXiv · doi:10.48550/arxiv.1906.04980
Filling Gender & Number Gaps in Neural Machine Translation with Black-box Context Injection
Moryossef, A., Aharoni, R., & Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1903.03467
Flagged first, then by citations
Research works printed but not linked: 5
- Mart´ ın Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for largescale machine learning. In 12th { USENIX } Symposium on Operating Systems Design and Implementation ( { OSDI } 16) , pp. 265-283, 2016.
- Ruth Schwartz Cowan. The consumption junction: A proposal for research strategies in the sociology of technology. In Wiebe E. Bijker, Thomas P. Hughes, and Trevor J. Pinch (eds.), The Social Construction of Technological Systems , pp. 261-280. MIT Press, Cambridge, MA, USA, 1987.
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
- Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems , pp. 62946305, 2017.
- Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointergenerator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , volume 1, pp. 1073-1083, 2017.
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · see citing works
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078 · see citing works
Diffusion Models: A Comprehensive Survey of Methods and Applications
Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., & Yang, M.-H. (2024) · ACM Computing Surveys · doi:10.1145/3626235 · Cited by 1,704 · see citing works
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
“Another possibility is to label the pre-training data based on code quality, then condition the model on the ’high quality’ label at deployment time (Keskar et al., 2019).”
BackgroundLarge language models generate functional protein sequences across diverse families
Madani, A., Krause, B., Greene, E. R., Subramanian, S., Mohr, B. P., Holton, J. M., Olmos, J. L., Xiong, C., Sun, Z. Z., Socher, R., et al. (2023) · Nature Biotechnology · doi:10.1038/s41587-022-01618-2 · Cited by 1,078 · see citing works
Self-citationClimbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082 · see citing works
“Similarly, control codes, or tokens like tl;dr, have been used to prompt large LMs to perform summarization and other tasks (Radford et al., 2019; Keskar et al., 2019).”
MethodPrompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · see citing works
“Recent work in the literature has focused on controlling natural language generation using traditional approaches from machine learning like novel architectures which condition outputs [15, 16], more advanced sampling techniques [6, 11], gradient-based”
BackgroundRecipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
AI models collapse when trained on recursively generated data
Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R., & Gal, Y. (2024) · Nature · doi:10.1038/s41586-024-07566-y · Cited by 793 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · see citing works
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474 · see citing works
“Language specific control codes , or prefixes, is an approach introduced in [29, 30] that facilitates constrained language modeling.”
BackgroundWordcraft: Story Writing With Large Language Models
Yuan, A., Coenen, A., Reif, E., & Ippolito, D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511105 · Cited by 374 · see citing works
Siriwardhana, S., Weerasekera, R., Wen, E., Kaluarachchi, T., Rana, R., & Nanayakkara, S. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00530 · Cited by 337 · see citing works
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581225 · Cited by 240 · see citing works
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Schick, T., Udupa, S., & Schütze, H. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00434 · Cited by 227 · see citing works
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation
Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445924 · Cited by 244 · see citing works
Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3434237 · Cited by 194 · see citing works
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · see citing works
“We include some metadata on the code files and StackOverflow questions/answers directly in our training data to allow attribute-conditioned generation [40, 75] and attribute prediction.”
MethodTweepFake: about detecting deepfake tweets
Fagni, T., Falchi, F., Gambini, M., Martella, A., & Tesconi, M. (2021) · PLOS ONE · doi:10.1371/journal.pone.0251415 · Cited by 185 · see citing works
TLDR: Extreme Summarization of Scientific Documents
Cachola, I., Lo, K., Cohan, A., & Weld, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.428 · Cited by 169 · see citing works
“Following previous success utilizing control codes (Keskar et al., 2019), we append control codes specifying the style of the output to the source documents and perform training in a multitask manner by shuffling paper-title pairs with paper-TLDR pairs.”
MethodSparks: Inspiration for Science Writing using Language Models
Gero, K. I., Liu, V., & Chilton, L. (2022) · Designing Interactive Systems Conference · doi:10.1145/3532106.3533533 · Cited by 161 · see citing works
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121 · see citing works
SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114 · see citing works
Few-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107 · see citing works
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · see citing works
“While we draw inspiration from studies that use control tokens to start and guide text generation (Lu et al., 2022; Keskar et al., 2019), our trained LM uses critique tokens to assess its own predictions after each generated segment as an integral part of the generation output.”
Evaluation of large language models for discovery of gene set function
Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D. (2025) · Nature Methods · doi:10.1038/s41592-024-02525-x · Cited by 104 · see citing works
Language Models as Agent Models
Andreas, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.423 · Cited by 92 · see citing works
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91 · see citing works
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · see citing works
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features
Rashkin, H., Reitter, D., Tomar, G. S., & Das, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.58 · Cited by 74 · see citing works
Reducing Conversational Agents’ Overconfidence Through Linguistic Calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00494 · Cited by 72 · see citing works
Reducing conversational agents' overconfidence through linguistic calibration
Mielke, S. J., Szlam, A., Dinan, E., & Boureau, Y.-L. (2020) · arXiv · doi:10.48550/arxiv.2012.14983 · Cited by 72 · see citing works
Controllable Natural Language Generation with Contrastive Prefixes
Qian, J., Dong, L., Shen, Y., Wei, F., & Chen, W. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.229 · Cited by 74 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00567 · Cited by 70 · see citing works
McCoy, R. T., Smolensky, P., Linzen, T., Gao, J., & Celikyilmaz, A. (2021) · arXiv · doi:10.48550/arxiv.2111.09509 · Cited by 70 · see citing works
MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores
Kao, S.-C., & Krishna, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00065 · Cited by 58 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49 · see citing works
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47 · see citing works
ConDA: Contrastive Domain Adaptation for AI-generated Text Detection
Bhattacharjee, A., Kumarage, T., Moraffah, R., & Liu, H. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.40 · Cited by 43 · see citing works
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey
Kumar, S., Balachandran, V., Njoo, L., Anastasopoulos, A., & Tsvetkov, Y. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.241 · Cited by 38 · see citing works
“Math word problem generation with mathematical consistency and problem context constraints,”
Wang, Z., Lan, A., & Baraniuk, R. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.484 · Cited by 39 · see citing works
Design in the DARK: Learning Deep Generative Models for De Novo Protein Design
Moffat, L., Kandathil, S. M., & Jones, D. T. (2022) · doi:10.1101/2022.01.27.478087 · Cited by 25 · see citing works
Wingate, D., Shoeybi, M., & Sorensen, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.412 · Cited by 26 · see citing works
Automatic Product Copywriting for E-commerce
Zhang, X., Zou, Y., Zhang, H., Zhou, J., Diao, S., Chen, J., Ding, Z., He, Z., He, X., Xiao, Y., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21508 · Cited by 24 · see citing works
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29 · see citing works
“We set adaptive plausibility constraint ( α ) to 0.1 and repetition penalty ( θ ) to 1.2 as per prior studies(Li et al., 2022; Keskar et al., 2019).”
MethodInfluentialPlug-and-Blend: A Framework for Plug-and-Play Controllable Story Generation with Sketches
Lin, Z., & Riedl, M. O. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · doi:10.1609/aiide.v17i1.18891 · Cited by 16 · see citing works
Universality of representation in biological and artificial neural networks
Hosseini, E., Casto, C., Zaslavsky, N., Conwell, C., Richardson, M., & Fedorenko, E. (2024) · doi:10.1101/2024.12.26.629294 · Cited by 10 · see citing works
Mirowski, P., Mathewson, K. W., Pittman, J., & Evans, R. (2022) · arXiv · doi:10.48550/arxiv.2209.14958 · Cited by 10 · see citing works
Nano: Nested Human-in-the-Loop Reward Learning for Few-shot Language Model Control
Fan, X., Lyu, Y., Liang, P. P., Salakhutdinov, R., & Morency, L.-P. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.758 · Cited by 3 · see citing works
Joint Audio and Speech Understanding
Gong, Y., Liu, A. H., Luo, H., Karlinsky, L., & Glass, J. (2023) · arXiv · doi:10.48550/arxiv.2309.14405
“Generation Setting: We use a plain generation setting of Temperature = 0.1, Top K = 500, and Top P = 0.95 with a repetition penalty of 1.1 [21, 22] for all tasks.”
MethodInfluentialProtecting Language Models Against Unauthorized Distillation through Trace Rewriting
Ma, X., Yeoh, W., Zhang, N., & Vorobeychik, Y. (2026) · arXiv · doi:10.48550/arxiv.2602.15143
“Early approaches trained conditional LMs with explicit control codes that govern style and content (Keskar et al., 2019).”
Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods
Crothers, E., Japkowicz, N., & Viktor, H. (2022) · arXiv · doi:10.48550/arxiv.2210.07321 · Cited by 196 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · arXiv · doi:10.48550/arxiv.2101.00190 · Cited by 289 · see citing works
Improving Code Autocompletion with Transfer Learning
Zhou, W., Kim, S., Murali, V., & Aye, G. A. (2021) · arXiv · doi:10.48550/arxiv.2105.05991
“Code sequences fed to our multilingual model are modified by prepending a control code [15] to indicate the source language.”
SciRepEval: A Multi-Format Benchmark for Scientific Document Representations
Singh, A., D'Arcy, M., Cohan, A., Downey, D., & Feldman, S. (2022) · arXiv · doi:10.48550/arxiv.2211.13308 · Cited by 76
“We explore two state-of-the-art methods to generate format-specific representations: control codes (Keskar et al., 2019; Raffel et al., 2020) as input signal; and parameter-efficient adapter methods (Pfeiffer et al., 2021; Stickland and Murray, 2019), where a separate network module is introduced…”
Method
- Authors
- Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., & Socher, R.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.05858
- OpenAlex
- W2973049837
- Semantic Scholar
- 75acc731bdd2b626edc74672a30da3bc51010ae8
- Cited by
- 751
- References
- 103
- Citations by year
- 2026: 8 · 2025: 29 · 2024: 75 · 2023: 164 · 2022: 135 · 2021: 186 · 2020: 145 · 2019: 9
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- controllable text generation, control codes, unsupervised learning, source attribution, large language models, pre-trained language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
