In conversation with Artificial Intelligence: aligning language models with human values

“At the same time, what modality of conversation is deemed to be ‘civil’ tends to vary widely according to cultural and historical context, and to be heavily indexed toward existing social conventions Dı́az et al. (2022). Given this variation across time and place, the social standards that define civil speech may or m…”

“Large-scale language technologies are increasingly used to enable various forms of linguistic communication in contexts ranging from biomedical research to education to machine translation [17, 19, 73].”

“2For GPT-3, see [19]; for BERT, see [28]; for Turing-NLG-A-17, see [85]; for CLIP, see [81]; for Gopher, see [82].”

“6See, for example, Mehrabi et al. (2021), Mitchell et al.”

“35See, for example, Habermas (2016) and Held (2006). 36For a stronger interpretation of these deliberative norms we can turn to Habermas’s theory of deliberative democracy and in his conception of ideal speech (Habermas, 1984; 1987). Habermas focuses on the possibility of a critical discussion that is inclusive and fr…”

“We suggest that Gricean maxims can help map out the path ahead.”

“15For a detailed discussion about Syntax, see Van Valin and LaPolla [99].”

“38For a recent survey of attempts to reduce ‘toxic speech’, see Fortuna and Nunes [34]. consideration.”

“Scientists pursue different goals in advancing scientific knowledge: they use science to explain or understand things about the world [87, 97], such as the formation of clouds; to predict phenomena [88], such as the structure of proteins; or to control the world around us [68] via, for example, the development of medi…”

“20For excellent introductions to the topic of pragmatics, see Grice (1968, 1989), Recanati (1989), Horn & Ward (2008), Stalnaker (2014), Thomas (1995), Goodman & Lassiter (2015), and Bergen et al.”

“28For critical discussion of Gricean maxims, questions about their universality, and alternative proposals, see Frederking [35], Keenan [56], Sperber and Wilson [92], and Wierzbicka [106].”

“See, for example, Dale and Reiter [27] and Asher et al. [7].”

“4For an in-depth examination of value alignment, see Gabriel [36].”

“These decisions are informed, in turn, by how seriously scientists account for the risk of making a mistake in accepting or rejecting a hypothesis, understood against a backdrop of human interests and human needs [30].”

“23 This common ground is the body of information that is presupposed or shared by the parties in the discourse, about the discourse itself, and about the situation of the participants in that discourse - and it sets the boundaries of the situation relevant to the linguistic conversation [41, 3, 93].”

“These technologies are also known as ‘dialogue systems’ [105].”

“For an excellent discussion, see Chierchia and McConnell-Ginet (2000). 18There have been several attempts to impute meaning to this sentence, such as Stanford’s competition to show that it is not in fact meaningless; see https://linguistlist.”

“40See Simonton [91],Boden [16], and Sternberg and Lubart [94].”

“To date, efforts to create aligned conversational agents have centred on the identification and mitigation of harms, such as the proliferation of inappropriate stereotypes or of hateful speech [12, 50, 103].”

“31 Consider, for example, the fact that no scientific hypothesis is ever completely verified (Rudner, 1953).”

“35See, for example, Habermas (2016) and Held (2006). 36For a stronger interpretation of these deliberative norms we can turn to Habermas’s theory of deliberative democracy and in his conception of ideal speech Habermas (1984, 1987). Habermas focuses on the possibility of a critical discussion that is inclusive and fre…”

“28 One application of conversational agents to the scientific domain is Galactica, which was launched by Meta to assist researchers and students by responding to their science questions Taylor et al. (2022). Galactica is trained on 48 million examples of scientific articles, textbooks, encyclopedias and other sources,…”

“One area in which this plays out is data annotation, and the need to have a diverse set of samplers who are able to adequately capture the implied meaning of terms (Waseem, 2016; Dı́az et al., 2022).”

“We suggest that Gricean maxims can help map out the path ahead.”

“and about the situation of the participants in that discourse — and it sets the boundaries of the situation relevant to the linguistic conversation (Grice, 1981; Allan, 2013; Stalnaker, 2014).”

“32Heather Douglas [31] has explored how judgements about the seriousness of social consequences im- pact the amount of evidence scientists demand before declaring a chemical toxic.”

“This includes relational considerations such as the intended audience, their level of familiarity with a given topic, the specific role of the language technology, and the underlying power relationships between interlocutors (see, for example, Androutsopoulos [6], Bell [26], and Clark and Murphy [26]).”

“Does the same notion of truth apply across a wide range of conversational contexts, or might standards of truthfulness vary according to the subject under consideration and to relevant conversational norms? Equally, there is widespread concern that the output of large-scale language models is biased [1, 15].”

“28For critical discussion of Gricean maxims, questions about their universality, and alternative proposals, see Frederking [35], Keenan [56], Sperber and Wilson [92], and Wierzbicka [106].”

“With further research, it may be 42See, for example, Binns [14], Lee et al. [64], Gabriel [37], and Bondi et al. [18]. possible to use our framework to refine both human and automatic evaluation of the performance of conversational agents.”

“41See Simonton (2000), Boden (1998), and Sternberg & Lubart (1999). 42Evans et al. (2021) discuss creative uses of expressivess and commissives.”

“22For contemporary philosophical examinations of linguistic context, see Kaplan (1979) and Stalnaker (2014).”

“31For example Elizabeth Anderson [5] has shown that researchers’ conceptualisation of divorce as something negative, rather than as a positive phenomenon, impacts the way they collect and analyse data.”

“However, there are other options for making sense of ‘truth’ (Kirkham, 1992).”

“This discursive domain is particularly relevant 30The clear-cut distinction between epistemic and non-epistemic value judgements is challenged by Rooney [84] and Longino [67].”

“matters (Bakker et al., 2022), and the government of Taiwan has pioneered the use of digital platforms as a mechanism to enhance democratic decision-making.”

“Nonetheless, these virtues may still need to be balanced against one another in order for a scientific claim to be acceptable [61, 70].”

“[4] for an introduction to a top-down approach.”

“It could be enriched further through analysis of other sociological and philosophical traditions such as Luhmann’s (1995) system theory, Latour’s (2007) actor-network theory, or Cameron’s (1992) feminist analysis of linguistic theory.”

“Scientists pursue different goals in advancing scientific knowledge: they use science to explain or understand things about the world [87, 97], such as the formation of clouds; to predict phenomena [88], such as the structure of proteins; or to control the world around us [68] via, for example, the development of medi…”

“16For discussion of the connections between syntax and semantics, see Heim and Kratzer (1185). 17There are many theories about how semantic analysis should be approached.”

“2For GPT-3, see [19]; for BERT, see [28]; for Turing-NLG-A-17, see [85]; for CLIP, see [81]; for Gopher, see [82].”

“29 Given that the focus is on scientific discourse, the cooperative goal of the discourse can be understood in relation to the diverse aims of the sciences [32, 78, 79].”

“See, for example, Leech [65] and Carston [23] for discussion.”

“This includes relational considerations such as the intended audience, their level of familiarity with a given topic, the specific role of the language technology, and the underlying power relationships between interlocutors (see, for example, Androutsopoulos [6], Bell [26], and Clark and Murphy [26]).”

“3 However, early instances of these models also present a number of risks and possible failure modes, including the production of false, offensive, or irrelevant information that could lead to a range of harms [15, 50, 104].”

“However, it has been shown that, in practice, scientists inevitably make choices on the basis of ethical beliefs, their perception of the social good derived from those beliefs, and other value judgements [31, 86, 66].”

“20 For excellent introductions to the topic of pragmatics, see Grice (1968), Grice (1989), Recanati (1989), Horn and Ward (2008), Stalnaker (2014), Thomas (1995), Goodman and Lassiter (2015), and Bergen et al. (2016). 21Schools of pragmatics differ with respect to how they draw the boundary between semantics and pragm…”

“3 However, early instances of these models also present a number of risks and possible failure modes, including the production of false, offensive, or irrelevant information that could lead to a range of harms [15, 50, 104].”

“[9] propose three ideals – helpfulness, honesty, and harmlessness – that can help align conversational agents with human values.”

“See, for example, Leech [65] and Carston [23] for discussion.”

“Conversational agents have been shown to perform well on a variety of computational metrics, supporting the emergence of new kinds of capability and opportunity [17, 95].”

“Scientists pursue different goals in advancing scientific knowledge: they use science to explain or understand things about the world [87, 97], such as the formation of clouds; to predict phenomena [88], such as the structure of proteins; or to control the world around us [68] via, for example, the development of medi…”

“Consider, for example, ostentatious or selfablating demonstrations of respect that may be demanded in hierarchical, patriarchal, racist, or caste-based societies [21].”

“This discursive domain is particularly relevant 30The clear-cut distinction between epistemic and non-epistemic value judgements is challenged by Rooney [84] and Longino [67].”

“must themselves be subjected to reflective evaluation and critique (Ackerly, 2000; Walzer, 1993).”

“Scientists pursue different goals in advancing scientific knowledge: they use science to explain or understand things about the world [87, 97], such as the formation of clouds; to predict phenomena [88], such as the structure of proteins; or to control the world around us [68] via, for example, the development of medi…”

“21For excellent introductions to the topic of pragmatics, see Grice [40, 42], Recanati [83], Horn and Ward [51], Stalnaker [93], Thomas [96], Goodman and Lassiter [39], and Bergen et al. [13].”

“12There are a variety of non-English language models [107].”

“29 Given that the focus is on scientific discourse, the cooperative goal of the discourse can be understood in relation to the diverse aims of the sciences [32, 78, 79].”

“Finally, the pragmatic theory ties truth, in one way or another, to human needs and the resolution of problematic situations (for a great discussion, see Haack [43]).”

“[57] explore the ways in which human designers may misspecify objectives for conversational agents and propose mitigation strategies to avoid the creation of accidentally misaligned conversational agents.”

“[33] discuss creative uses of expressivess and commissives.”

“It could be enriched further through analysis of other sociological and philosophical traditions such as Luhmann’s (1995) system theory, Latour’s (2007) actor-network theory, or Cameron’s (1992) feminist analysis of linguistic theory.”

“18 Semantic norms, therefore, provide a template for the generation of comprehensible sentences and comprise a second important area of research for the evaluation of language agents (see Kapetanios et al. (2013) and Maulud et al. (2021) for some examples of such research efforts).”

“14See, for example, Morris [76], Montague [75], and Silverstein [90].”

“Roughly speaking, a correspondence theory of truth holds that what one believes or says is true if it corresponds to the way things actually are in the world (for a contemporary exposition, see Bunge [20]).”

“(2021), and Kasirzadeh Kasirzadeh (2022) for some discussion. 7There is also a second functional meaning of ‘good speech’ which is defined at a higher-level of abstraction (See Kasirzadeh & Klein (2021) for levels of abstraction in machine learning systems).”

“8There is also a second functional meaning of ‘good speech’ which is defined at a higher-level of abstraction (See Kasirzadeh and Klein [55] for levels of abstraction in machine learning systems).”

“This can be explained since the objective of many LLMs, including GPT-3, is to predict the next element in a text, based on a large corpus of text data from the internet, (15) thus LLMs learn to replicate biases and stereotypes present in that data (16).”

“Simply put, language modeling is associated with predicting what word should come next considering what word preceded it [100].”

“Nonetheless, GPT-3 may replicate the biases in web texts, leading to negative behaviors such as fabricating facts, perpetuating stereotypes, and producing logical fallacies [30, 31].”