Language Models and Cognitive Automation for Economic Research

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.

The LLM answered other concept questions superbly, including the invisible hand theorem, Monte Carlo simulation, Cournot versus Bertrand equilibrium, leading theories of exchange rate determination, and several more.

“Other concept questions that the LLM answered superbly included the following, for which I am leaving out the responses for space reasons:”

Probably not a substantial claim: may not pass S4 (based).

When asked for citations, LLMs may hallucinate and make up authoritative-sounding papers that do not actually exist.

“When asked for citations, there is a risk that they may hallucinate and make up authoritative-sounding papers that do not actually exist.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can evaluate different ideas by providing pros and cons of different research plans.

“They can also evaluate dierent ideas, in particular by providing pros and cons of different research plan.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The LLM confuses the two fundamental theorems of welfare economics.

“The two fundamental theorems of welfare economics are cornerstones of modern economics, but for some reason the LLM confuses the two.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can translate rough bullet points into well-structured sentences that are clear and easy to read.

“One of the most useful capabilities of LLMs may be to translate rough bullet points into well-structured sentences that are clear and easy to read.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

The channels listed by the model for AI and inequality are on-topic, largely reasonable, and span a wide range of what the author could think of plus some more.

“The channels listed by the model are not path-breaking, but they are on-topic, largely reasonable, and span a wide range of what I could think of plus some more.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can evaluate text for its style, its clarity, or similar.

“LLMs can also evaluate text for its style, its clarity, or similar.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

LLMs are very capable at summarizing text and usually adept at extracting the most relevant content from a passage.

“First, LLMs are very capable at summarizing.They are usually adept at extracting the most relevant content from a passage of text”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can write, edit, modify, translate, or debug snippets of code based on instructions in plain English or other common languages.

“They can write, edit, modify, translate, or debug snippets of code based on instructions in plain English (or other common languages).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can convert text to be comprehensible to readers of different levels.

“Another useful application is that the system can convert text to be comprehensible to readers of dierent levels”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

LLMs can act as a tutor and explain many common economic concepts very clearly at a level that is quite useful for students who are trying to learn new, and even for more advanced researchers who are branching out beyond their main area of expertise.

“LLMs can act as a tutor and explain many common economic concepts very clearly at a level that is quite useful for students who are trying to learn new, and even for more advanced researchers who are branching out beyond their main area of expertise.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

LLMs can be very useful for writing code, in particular for standard programming tasks, data manipulation and repetitive tasks.

“LLMs can be very useful for writing code, in particular for standard programming tasks, data manipulation and repetitive tasks.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

LLMs are quite capable and useful for many tasks related to writing, from synthesizing text based on bullet points, changing the style of text, editing text, evaluating styles, and generating titles, headlines, and tweets.

“This implies that they are quite capable and useful for many tasks related to writing, from synthesizing text based on bullet points, changing the style of text, editing text, evaluating styles, and generating titles, headlines, and tweets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can explain their edits afterwards so that students can learn from corrections.

“As illustrated by the follow-up question, the system can even even explain its edits afterwards so that students can learn from any corrections it has made.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs are of limited usefulness for searching the literature.

“LLMs are of limited usefulness for searching the literature.”

Probably not a substantial claim: may not pass S4 (based).

LLMs are just as good at providing arguments in favor of a given point as they are for the counterarguments.

“LLMs do not care about which side of an argument they are on they are just as good at providing arguments in favor of a given point as they are for the counterarguments.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

LLMs can serve both as assistants for micro tasks and as tutors for coding.

“LLMs can serve both as assistants for micro tasks and as tutors for coding.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

LLMs are quite useful in brainstorming ideas and examples related to a defined theme.

“Since LLMs are trained on a vast amount of data that represents a cross-section of all human knowledge, they are quite useful in brainstorming (or, perhaps more aptly, net -storming) ideas and examples related to a dened theme.”

Probably not a substantial claim: may not pass S4 (based).

Current LLMs can produce text in a style that sounds highly authoritative even when they hallucinate.

“Current LLMs can product text in a style that sounds highly authoritative even when they hallucinate, i.e., when the content is completely wrong.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The results generated by an LLM change depending on the prompt; even small changes in prompts can lead to completely different outputs.

“Furthermore, note that the results generated by an LLM change depending on the prompt even small changes in prompts, such as dierent spacing or punctuation, can lead to completely dierent outputs.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can come up with catchy headlines or paper titles.

“Another useful application is to come up with catchy headlines or paper titles”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Once references are found and verified not hallucinated, LLMs are very capable of formatting them in the desired manner.

“Once references are found and it is veried that they are not hallucinated, LLMs are very capable of formatting them in the desired manner”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

LLMs are frequently able to provide standard references that are frequently cited in the literature, but all references they provide should always be double-checked.

“They are frequently able to provide standard references that are frequently cited in the literature, but even so, all references they provide should always be double-checked.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

LLMs are increasingly useful for ideation, i.e., generating, selecting, and developing ideas.

“One part of research for which LLMs are increasingly useful is the process of ideation, i.e., generating, selecting, and developing ideas.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can generate tweets that summarize a text.

“The following exercise is similar but converts the abstract I pasted into ve tweetable chunks that would work well for Twitter”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can edit text for grammatical or spelling mistakes, style, clarity, or simplicity.

“LLMs can edit text for grammatical or spelling mistakes, style, clarity, or simplicity.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

LLMs can write text in LaTeX format and draft beamer slides in LaTeX code based on bullet points.

“Of perhaps greater relevance for our daily writing, LLMs can also write text in LaTeX format, which automatically surrounds variables by the $ signs required for LaTeX, and it can draft beamer slides in LaTeX code based on bullet points.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The system is highly useful for catching typos and violations of basic syntax.

“As illustrated, here, the system is highly useful for catching typos and violations of basic syntax.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The remainder of the derivations in the first mathematical example is largely gibberish.

“The remainder of the derivations is largely gibberish.”

May combine two claims.

The mathematical abilities of the system are still quite limited: after obtaining the correct solution, when the sign mistake in the Lagrangian was corrected and the remainder of the derivations regenerated, the system produced gibberish.

“However, the mathematical abilities of the system are still quite limited: after obtaining the correct solution, I corrected the sign mistake in the Lagrangian and attempted to regenerate the remainder of the derivations, but the system produced gibberish.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The system derives two of the three first-order conditions correctly but stumbles on the derivative of the iso-elastic utility function.

“It derives two of the three rst-order conditions correctly as if the sign on the budget constraint in the Lagrangian had just been a typo but stumbles on the derivative of the iso-elastic utility function.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The system proceeded to correctly derive the solution of the maximization problem, i.e., the optimal levels of x and y as a function of the exogenous parameters.

“The system proceeded to correctly deriving the solution of the maximization problem, i.e., the optimal levels of x and y as a function of the exogenous parameters.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the Lagrangian, the system includes the budget constraint with an unusual sign.

“In the Lagrangian, the system includes the budget constraint with an unusual sign.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

LLMs have become useful research tools for tasks ranging from ideation, writing and background research to data analysis, coding, and mathematical derivations.

“LLMs have become useful research tools for tasks ranging from ideation, writing and background research to data analysis, coding, and mathematical derivations.”

May combine two claims.

The ability of LLMs to perform mathematical derivations is emerging but still limited at this point.

“The ability of LLMs to perform mathematical derivations is emerging but still limited at this point.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The LLM correctly fills in a suitable budget constraint and sets up the associated maximization problem.

“It can be seen that the LLM correctly lls in a suitable budget constraint and sets up the associated maximization problem.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs can act as tutors that help users to find the packages and functions that they need for specific use cases.

“Moreover, they can act as tutors that help users to nd the packages and functions that they need for specic use cases.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Current LLMs are quite reliable in translating short pieces of code across most common languages.

“Current LLMs are quite reliable in translating short pieces of code in this fashion across most common languages.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLMs can format data, extract data from plain text, classify and score text, extract sentiment, and even simulate human test subjects.

“LLMs can format data, extract data from plain text, classify and score text, extract sentiment, and even simulate human test subjects.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

These LLM capabilities can be accessed not only through a web interface but also via an API that allows large amounts of data to be formatted, extracted, classified, etc.

“Moreover, what is perhaps most useful is that these capabilities can be accessed not only through a web interface, like what is shown in the demonstrations below, but also via an API (Application Programming Interface) that allows large amounts of data to be formatted, extracted, classied etc.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.

The system's assessment of whether the December 2022 FOMC statement is hawkish or dovish is correct and well-argued.

“The system's assessment is correct and well-argued.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs can explain what a given piece of code does in plain English.

“LLMs can also do the reverse and explain what a given piece of code does in plain English, akin to a tutor:”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

For longer sequences of code translation, human assistance is still required.

“For longer sequences, human assistance is still required.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Summaries of longer texts are limited by the transformer technology on which the systems are based.

“Summaries of longer texts are limited by the transformer technology on which the systems are based.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

LLMs increasingly have comparative advantage in generating content; humans currently have comparative advantage in evaluating and discriminating content.

“LLMs increasingly have comparative advantage in generating content; humans currently have comparative advantage in evaluating and discriminating content.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

LLMs are a category of foundation models, which can be regarded as the new paradigm in artificial intelligence of the 2020s.

“LLMs are a category of foundation models, which can be regarded as the new paradigm in articial intelligence of the 2020s (Bommasani et al., 2021).”

• cites Bommasani et al., 2021Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The paper's own exploration attempted to refrain from cherry-picking and illustrate both capabilities and failures to provide a balanced version of LLM capabilities as of early 2023.

“I have attempted to refrain from cherry-picking and illustrate both the capabilities and failures of the LLMs I explored to provide a balanced version of their capabilities as of early 2023.”

Probably not a substantial claim: may not pass S4 (based).

LLMs are useful for micro tasks because of their high speed and low transaction cost.

“LLMs are useful for such tasks because of their high speed and the low transaction cost.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Once a certain threshold of training compute is crossed, LLMs almost predictably develop the ability to perform arithmetic computations, to unscramble words, or to perform Q&A.

“For example, Wei et al. (2022a) report that once a certain threshold of training compute is crossed, LLMs almost predictably develop the ability to perform arithmetic computations, to unscramble words, or to perform Q&A.”

• cites Wei et al. (2022a)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

LLMs have crossed the threshold to become useful across a wide range of cognitive tasks.

“LLMs have just crossed the threshold to become useful across a wide range of cognitive tasks as was illustrated by the viral reception of ChatGPT”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Other significant capabilities that have emerged from language models include coding, translation, and rhyming.

“Other signicant capabilities that have emerged from language models include coding, translation, and rhyming.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Many LLMs actually exhibit greater capabilities than what is known (capabilities overhang).

“This suggests that there may in fact be signicant capabilities overhang, i.e., that many LLMs actually exhibit greater capabilities than what is known.”

Probably not a substantial claim: may not pass S1 (asserted), S4 (based).

Economic researchers who take advantage of LLMs to automate micro tasks will become significantly more productive.

“I hypothesize that ongoing advances will improve the performance of LLMs across all of these domains, and that economic researchers who take advantage of LLMs to automate micro tasks will become significantly more productive.”

• Also stated in: § 1 Introduction · plain

  “I posit that researchers can signicantly increase their productivity by incorporating LLMs into their workow.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

In human-written texts, there is a strong correlation between authoritative style and insightful content, but LLMs have learned the former without being reliable on the latter.

“In human-written texts, there is a strong correlation between authoritative style and insightful content, but LLMs have learned the former without being reliable on the latter.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

LLMs are advancing rapidly, becoming more accurate and powerful with every new iteration.

“Moreover, whereas the level of human intelligence is relatively static, LLMs are advancing rapidly, becoming more accurate and powerful with every new iteration.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

At present, LLMs are most useful as assistants that can automate small micro tasks that researchers engage in numerous times during the day but that are too small to be assigned to human research assistants.

“At present, I view LLMs to be most useful as assistants that can automate small micro tasks that researchers engage in numerous times during the day but that are too small to be assigned to human research assistants.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

LLMs are useful as tutors in coding and data analysis tasks as well as in ideation and writing.

“Moreover, LLMs are also useful as tutors in coding and data analysis tasks as well as in ideation and writing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ChatGPT has a tested IQ of 147 (99.9th percentile) on a verbal-linguistic IQ test.

“However, a former Chairman for Mensa International reports that ChatGPT has a tested IQ of 147 (99.9th percentile) on a verbal-linguistic IQ test (Thompson, 2023).”

• cites Thompson, 2023Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

To maximize reproducibility, setting the Temperature parameter of the model to 0 makes the responses provided by the system deterministic.

“To maximize reproducibility, I set the Temperature parameter of the model to 0, which makes the responses provided by the system deterministic.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Large language models such as ChatGPT have the potential to revolutionize research in economics and other disciplines.

“Large language models (LLMs) such as ChatGPT have the potential to revolutionize research in economics and other disciplines.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

LLMs have super-human capabilities in processing large amounts of text.

“LLMs also have super-human capabilities in processing large amounts of text.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

LLMs display a form of creativity that had long been reserved for humans.

“This is illustrates vividly how LLMs dier from earlier applications of deep learning in economics they display a form of creativity that had long been reserved for humans.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Frieder et al. (2023) develop a dataset of graduate-level mathematical questions and conclude that ChatGPT's mathematical abilities are significantly below those of an average mathematics graduate student.

“Frieder et al. (2023) develop a dataset of graduate-level mathematical questions and conclude that ChatGPT's mathematical abilities are signicantly below those of an average mathematics graduate student”

• cites Frieder et al. (2023)Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Argyle et al. (2022) propose the use of LLMs to simulate human subjects, based on the observation that the training data of LLMs contains a large amount of information about humanity.

“Argyle et al. (2022) propose the use of LLMs to simulate human subjects, based on the observation that the training data of LLMs contains a large amount of information about humanity.”

• cites Argyle et al. (2022)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Simulated results from LLMs contain valuable information and, if used correctly, can provide useful insights about our society.

“However, they also contain valuable information.If used correctly, they can provide useful insights about our society, from which all the data used to train the LLMs ultimately originate.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

In the longer term, a similar bitter lesson may apply to economics: with enough compute, sufficiently advanced AI systems may be able to produce and articulate superior economic models, and the cognitive work of human economists may eventually become redundant.

“Yet a similar bitter lesson may apply to economics: with enough compute, suciently advanced AI systems may be able to produce and articulate superior economic models, and the cognitive work of human economists like that of all other researchers may eventually become redundant.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

In Kasparov's terminology, LLMs have entered the period of weak competition with cognitive workers, including economic researchers.

“In Kasparov's terminology, LLMs have entered the period of weak competition with cognitive workers, including economic researchers.”

• cites Kasparov (2017)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

We are currently at the competition dot, and LLMs are garnering a lot of attention.

“We are currently at the competition dot, and LLMs are garnering a lot of attention.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

There is a significant risk that simulated results from LLMs simply propagate false stereotypes, and they must hence be used with great care.

“There is a signicant risk that the simulated results simply propagate false stereotypes, and they must hence be used with great care.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In the short term, cognitive automation via LLMs will allow researchers to become significantly more productive.

“In the short term, cognitive automation via LLMs will allow researchers to become signicantly more productive.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Human researchers will focus on their comparative advantage by posing the questions, suggesting directions for obtaining answers, discriminating which parts of the produced content are useful, editing, and providing feedback, akin to an advisor.

“Human researchers will focus on their comparative advantage by posing the questions, suggesting directions for obtaining answers, discriminating which parts of the produced content are useful, editing, and providing feedback, akin to an advisor.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

At this point, human researchers, especially when AI-assisted, are still the best technology around for generating economic research.

“At this point, human researchers, especially when AI-assisted, are still the best technology around for generating economic research!”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

In the medium term, LLM-based assistants and tutors will become increasingly useful for generating the content that makes up research papers.

“In the medium term, I anticipate that LLM-based assistants and tutors will become increasingly useful for generating the content that makes up research papers.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Over time, further advances will imply that LLMs are performing their tasks better and better so that the need for humans to provide inputs, edits, and feedback will diminish.

“Over time, further advances will imply that LLMs are performing their tasks better and better so that the need for humans to provide inputs, edits, and feedback will diminish.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

A growing number of researchers will incorporate LLMs into their workflow.

“I expect that a growing number of researchers will incorporate LLMs into their workow.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Horton (2022) showcases applications to economics, using simulated test subjects to replicate and extend upon several behavioral experiments.

“Horton (2022) showcases applications to economics, using simulated test subjects to replicate and extend upon several behavioral experiments.”

• cites Horton (2022)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The cognitive automation ushered in by the rapid rise of LLMs poses important and urgent new research questions to economists.

“I also believe that the cognitive automation ushered in by the rapid rise of LLMs poses important and urgent new research questions to economists, of which I will brainstorm a few:”

Probably not a substantial claim: may not pass S3 (load-bearing).

Noorbakhsh et al. (2021) show that LLMs can be fine-tuned for mathematical tasks.

“Noorbakhsh et al. (2021) show that LLMs can be ne-tuned for mathematical tasks.”

• cites Noorbakhsh et al. (2021)Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

This could help to increase the overall speed of progress in economics, although it risks leaving behind those who do not take advantage of LLMs.

“This could help to increase the overall speed of progress in economics, although it risks leaving behind those who do not take advantage of LLMs.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Our work will affect the concepts and representations through which future AI systems will view economic questions and, ultimately, through which they will view the world.

“Second, our work will aect the concepts and representations through which future AI systems will view economic questions and, ultimately, through which they will view the world”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Economists have the tools to translate concepts from the social sciences and humanities, such as human objectives, into analytic concepts like preferences that are more easily accessible to machines.

“Continuing on the last question, economists have the tools to translate concepts from the social sciences and humanities, such as human objectives, into analytic concepts like preferences that are more easily accessible to machines.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Economists have experience analyzing agency and control problems and their solutions.

“And we have experience analyzing agency and control problems and their solutions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There are two channels through which economists can make important contributions to AI alignment work.

“In fact, there are two channels through which economists can make important contributions to this line of work:”