(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice

Classified as probably not a substantial claim.

Experts' recommendations for LLM response strategies center around helping users identify 'right questions to ask' and relevant information rather than providing definitive legal judgments.

“We share experts' recommendations for LLM response strategies, which center around helping users identify 'right questions to ask' and relevant information rather than providing definitive legal judgments.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some experts suggested leveraging LLMs' multi-turn dialogue capabilities to polish users' questions and distill relevant facts through follow-up questions instead of seeking definitive legal judgement.

“Instead of seeking definitive legal judgement, some suggested leveraging LLMs' multi-turn dialogue capabilities to polish users' questions and distill relevant facts through follow-up questions.”

May combine two claims.

The authors identified 25 key dimensions that impacted experts' preferences and should inform potential LLM responses.

“For RQ1 , we identified 25 key dimensions that should inform potential LLM responses (Figure 3).”

• Also stated in: § 4.1 Dimensions · p. 6 · plain

  “We identified 25 key dimensions that impacted experts' preferences for appropriate LLM responses.”

• AssumptionRests on an assumption: The 20 recruited legal professionals are representative enough of legal expertise that their judgments can inform responsible LLM policies for legal advice.

Classified as probably not a substantial claim.

The paper presents a 4-dimension framework: (1) User attributes and behaviors, (2) Nature of queries, (3) AI capabilities, and (4) Social impacts.

“By synthesizing the factors that impacted LLM response appropriateness, we present a 4-dimension framework: (1) User attributes and behaviors, (2) Nature of queries, (3) AI capabilities, and (4) Social impacts.”

• Also stated in: § 1 INTRODUCTION · p. 2 · plain

  “We classified dimensions into four categories-(1) User attributes and behaviors, (2) Nature of queries, (3) AI capabilities, and (4) Social impacts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experts proposed additional layers of ethical guidelines such as 'Don't pretend to be a human,' or 'Respect the justice system.'

“Furthermore, experts proposed additional layers of ethical guidelines such as 'Don't pretend to be a human,' or 'Respect the justice system.'”

Classified as probably not a substantial claim.

Experts found that LLM systems could potentially weigh the possible downsides including what the user may not have considered that could harm them, such as emotional effects or potential consequences in workplace or relationships.

“Experts found that LLM systems could potentially weigh the possible downsides including what the user may not have considered that could harm them, such as emotional effects or potential consequences in workplace or relationships (P4, P13, P20).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Experts considered 2 dimensions of possible ways that LLM-generated responses could have on users and society.

“Experts considered 2 dimensions of possible ways that LLM-generated responses could have on users and society.”

Classified as probably not a substantial claim.

The workshop's structured, case-based deliberations yielded nuanced insights into the multi-layered tensions that arise when using LLMs for legal advice.

“Our workshop's structured, case-based deliberations yielded nuanced insights into the multi-layered tensions that arise when using LLMs for legal advice.”

May combine two claims.

Experts extensively discussed confidentiality risks, which can be differentiated in a practical and normative sense.

“Experts extensively discussed confidentiality risks (P4, P7-9, P12, P14, P16), which can be differentiated in a practical and normative sense.”

Classified as probably not a substantial claim. May combine two claims.

These consequences include risks to indirectly affected third parties, such as explicit bias and stereotypes in advice, ensuing impacts of how advice is interpreted and acted upon, and the long-term assimilation of values.

“These consequences include risks to indirectly affected third parties, such as explicit bias and stereotypes in advice, ensuing impacts of how advice is interpreted and acted upon, and the long-term assimilation of values.”

Classified as probably not a substantial claim. May combine two claims.

LLM systems evade responsibility either through intermediary immunity laws or non-negotiable disclaimer clauses committing users to bear potential damages.

“LLM systems evade responsibility either through intermediary immunity laws or non-negotiable disclaimer clauses committing users to bear potential damages (P8, P16).”

May combine two claims.

The authors identified 8 user-related dimensions that AI systems should consider, breaking down into user attributes and user behavior.

“Our participants identified 8 user-related dimensions that AI systems should consider that broadly break down into dimensions related to (1) User attributes and (2) User behavior.”

May combine two claims.

Our analysis uncovered 25 distinct dimensions to ensure safe and effective LLM legal advice, spanning four key categories: (1) User attributes and behaviors, (2) Query characteristics, (3) AI capabilities, and (4) Social impacts.

“Our analysis uncovered 25 distinct dimensions to ensure safe and effective LLM legal advice, spanning four key categories: (1) User attributes and behaviors, (2) Query characteristics, (3) AI capabilities, and (4) Social impacts.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

User attributes include identity and background, geographic location, legal sophistication, and access to resources.

“User attributes include identity and background, geographic location, legal sophistication, and access to resources.”

Classified as probably not a substantial claim.

Experts expressed concerns that LLM systems might reproduce structural stereotypes and discrimination.

“Experts expressed concerns that LLM systems might reproduce structural stereotypes and discrimination (P5, P10, P13, P17, P20).”

Classified as probably not a substantial claim. May combine two claims.

Legal contexts are inherently complex, and lawyers often spend considerable time eliciting relevant facts and identifying the 'right questions to ask.'

“Participants emphasized that legal contexts are inherently complex (P11), and lawyers often spend considerable time eliciting relevant facts and identifying the 'right questions to ask' (P12).”

Probably not a substantial claim: may not pass S3 (load-bearing).

From a practical perspective, experts warned against an LLM system's accidental leak of sensitive information, highlighting the potential for unintended breaches of confidentiality.

“From a practical perspective, experts warned against an LLM system's accidental leak of sensitive information (P4), highlighting the potential for unintended breaches of confidentiality.”

Classified as probably not a substantial claim.

Lawyers' strict code of conduct and negligence liability apply even to informal suggestions.

“While lawyers' strict code of conduct and negligence liability apply even to informal suggestions (P17)”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

User behavior refers to aspects such as reliability, intent, agency, and ambiguity, which can be deduced from the user's inputs and interactions with the AI system but are likely not explicitly stated.

“On the other hand, user behavior refers to aspects such as reliability, intent, agency, and ambiguity, which can be deduced from the user's inputs and interactions with the AI system but are likely not explicitly stated.”

Classified as probably not a substantial claim. May combine two claims.

Participants expressed frustration with missing case facts, noting that depending on the details that come out, it could change the analysis and outcomes.

“This idea emerged as participants expressed frustration with missing case facts: 'I don't think there's enough information to go off of, and that depending on the details that come out, it could change the analysis, therefore the outcomes.' (P13)”

Classified as probably not a substantial claim.

Applying relevant laws to specific user situations constitutes legal opinion.

“Providing relevant laws is fine (e.g., driving under influence (DUI) is illegal in Washington) but applying it to specific user situations constitutes opinion (e.g., falling asleep in the driver's seat in the parking lot after drinking alcohol could be a DUI) (P2, P13, P17, P19).”

May combine two claims.

Most experts condoned offering pertinent legal information, while expressing reservations at LLMs providing a legal opinion due to reasons such as insufficient AI capabilities or user protection.

“As Figure 4 shows, most experts condoned offering pertinent legal information, while expressing reservations at LLMs providing a legal opinion due to reasons such as insufficient AI capabilities or user protection.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Experts disagreed at times: some were more optimistic about future development, while others believed that issues like hallucinations might persist.

“Throughout the discussion, experts disagreed at times: some were more optimistic about future development, while others believed that issues like hallucinations might persist.”

May combine two claims.

Users lack confidentiality and accountability protections governing attorney advice: Conversations with AI systems risk disclosure in legal proceedings and inaccurate guidance evades professional negligence liability.

“Section 4.1.3 reveals that users lack confidentiality and accountability protections governing attorney advice: Conversations with AI systems risk disclosure in legal proceedings and inaccurate guidance evades professional negligence liability.”

May combine two claims.

The collective deliberation itself revealed critical hidden dimensions and elicited justifications that shed new light on existing dimensions.

“Finally, the collective deliberation itself revealed critical hidden dimensions and elicited justifications that shed new light on existing dimensions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some warned that LLM developers should exercise caution when eliciting extensive personal information from users, given confidentiality concerns.

“However, some warned that LLM developers should exercise caution when eliciting extensive personal information from users, given confidentiality concerns (P13, P16).”

Probably not a substantial claim: may not pass S3 (load-bearing).

What feels morally neutral in one culture may feel problematic in another, especially for minority groups.

“P20 noted that what feels morally neutral in one culture may feel problematic in another, especially for minority groups.”

Classified as probably not a substantial claim.

The most promising and heavily-discussed possibility during the workshops is leveraging multi-turn interactions, allowing LLMs to ask follow-up questions and clarify users' legally meaningful inquiries.

“The most promising and heavily-discussed possibility during the workshops is leveraging multi-turn interactions , allowing LLMs to ask follow-up questions and clarify users' legally meaningful inquiries.”

Classified as probably not a substantial claim.

Experts' recommendations for LLM response strategies and guiding principles centered around helping users identify and prepare salient information for legal proceedings rather than recommending specific legal actions.

“Second, we shared experts' recommendations for LLM response strategies and guiding principles for generating appropriate responses-these centered around helping users identify and prepare salient information for legal proceedings rather than recommending specific legal actions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two authors initially coded 2 transcripts respectively and developed a codebook of dimensions and responses, informed by Kim et al.'s human-AI-context framework.

“Two authors initially coded 2 transcripts respectively and developed a codebook of dimensions and responses, informed by the Kim et al.'s human-AI-context framework [37].”

• cites [37] [37] Sunnie S. Y. Kim, Elizabeth Anne Watkins, Olga Russakovsky, Ruth Fong, and Andrés Monroy-Hernández. 2023. Humans, AI, and Context: Understanding End-Users' Trust in a Real-World Computer Vision Application. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (Chicago, IL, USA) (FAccT '23) . Association for Computing Machinery, New York, NY, USA, 77-88. https://doi.org/10.1145/ sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Stockdale & Mitchell finds that legal advice privilege may still apply between users and AI chatbots in some jurisdictions.

“while Stockdale & Mitchell finds that legal advice privilege may still apply between users and AI chatbots in some jurisdictions [80].”

• cites [80] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Most prior research in this field speculates on high-level concerns such as inaccuracy and real-world impacts, but rarely articulates concrete criteria for when and why LLMs should or should not provide professional advice to users.

“Most prior research in this field speculates on high-level concerns such as inaccuracy and real-world impacts [43, 44, 54, 86, 93]. However, these studies rarely articulate concrete criteria for when and why LLMs should or should not provide professional advice to users.”

• cites [43]Not checked yet

  Not checked in the free run.

• cites [44]Not checked yet

  Not checked in the free run.

• cites [54] sourceNot checked yet

  Not checked in the free run.

• cites [86] sourceNot checked yet

  Not checked in the free run.

• cites [93]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

Responsible AI legal advice requires a cross-disciplinary synthesis that spans technology, law, and ethics, learning from accumulated knowledge in professional communities.

“This illustrates that responsible AI legal advice requires a cross-disciplinary synthesis that spans technology, law, and ethics, learning from accumulated knowledge in professional communities.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

California law allows paralegals to do fact-gathering and retrieving 'information,' but not to provide 'legal advice.'

“For instance, California law allows paralegals to do fact-gathering and retrieving 'information,' but not to provide 'legal advice.' [3]”

• cites [3] [3] 2007. Cal. Bus. & Prof. Code § 6450. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Spahn argues non-lawyers using AI to provide legal advice or prepare documents for third parties could violate the UPL.

“Applying this rule to AI systems, Spahn argues non-lawyers using AI to provide legal advice or prepare documents for third parties could violate the UPL [79]”

• cites [79] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

Haupt stresses AI's professional advice must demonstrate competence, trust, responsibility, and ethics.

“Reflecting on professional ethics, Haupt stresses AI's professional advice must demonstrate competence, trust, responsibility, and ethics [32].”

• cites [32] [32] Claudia E. Haupt. 2019. Artificial professional advice. Yale Journal of Law and Technology 21 (2019), 55-77. -Not checked yet

  Not checked in the free run.

May combine two claims.

Incorporating perspectives from domain experts is an emerging approach for responsible LLM policies, but has not been applied to the legal domain.

“Incorporating perspectives from domain experts is an emerging approach for responsible LLM policies [7], but has not been applied to the legal domain.”

• cites [7] [7] Maria Antoniak, Aakanksha Naik, Carla S. Alvarado, Lucy Lu Wang, and Irene Y. Chen. 2023. Designing Guiding Principles for NLP for Healthcare: A Case Study of Maternal Health. (2023). arXiv:2312.11803 -Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

Relying on LLMs for legal queries raises concerns due to the significant expertise required and the potential real-world consequences of the advice.

“However, relying on LLMs for legal queries raises concerns due to the significant expertise required and the potential real-world consequences of the advice.”

Probably not a substantial claim: may not pass S1 (asserted).

Narrowing down factual patterns and applying rules engages deep judgement.

“P16 argued that narrowing down factual patterns and applying rules engages deep judgement, stating 'You're starting to make the AI become your lawyer.'”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Two coders independently analyzed the data and cross-checked each other's work.

“Following this, two coders independently analyzed the data and cross-checked each other's work.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Influencing users' emotional states is highly problematic absent oversight, given risks of uncontrolled bias and manipulation.

“P13 cautioned that influencing users' emotional states is highly problematic absent oversight, given risks of uncontrolled bias and manipulation.”

Classified as probably not a substantial claim. May combine two claims.

IRAC entails (1) identifying the legal issue, (2) stating the rule that applies, (3) analyzing how the particular facts interact with the stipulations of the rule, and (4) finally deducing the conclusion.

“IRAC entails (1) identifying the legal issue, (2) stating the rule that applies, (3) analyzing how the particular facts interact with the stipulations of the rule, and (4) finally deducing the conclusion [14].”

• cites [51]Not checked yet

  Not checked in the free run.

• cites [14] [14] Columbia Law School Writing Center. 2001. Organizing a Legal Discussion (IRAC, CRAC, etc.) . Retrieved May 1, 2024 from https://www.law.columbia. edu/sites/default/files/2021-07/organizing\_a\_legal\_discussion.pdf -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Given that English-written data predominantly represented in training datasets, LLM responses may disproportionately reflect the values and perspectives of English-speaking populations.

“Given that English-written data predominantly represented in training datasets, experts noted that LLM responses may disproportionately reflect the values and perspectives of English-speaking populations (P8).”

• AssumptionRests on an assumption: English-written data is predominantly represented in training datasets.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both coders examined all documents and reached consensus on codes, rendering inter-rater reliability metrics unnecessary.

“In this process, both coders examined all documents and reached consensus on codes, rendering inter-rater reliability metrics unnecessary [48].”

• cites [48] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Wendel stated that the 'core lawyering functions' such as recommending the course of action or drafting contracts cannot be delegated to AI agents due to technical limitations and accountability deficits.

“Wendel stated that the 'core lawyering functions' such as recommending the course of action or drafting contracts cannot be delegated to AI agents due to technical limitations and accountability deficits [89].”

• cites [89]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Broader ethical considerations beyond just technically accurate guidance include assessing scenarios that create harm despite the good intentions of the advice.

“P17 highlighted broader ethical considerations beyond just technically accurate guidance, including assessing scenarios that create harm despite the good intentions of the advice.”

May combine two claims.

LLM systems focusing on issue and rule identification provide fact-finding 'information,' while analysis and conclusions may cross into tailored 'opinion.'

“Our findings suggest that LLM systems focusing on issue and rule identification provide fact-finding 'information,' while analysis and conclusions may cross into tailored 'opinion,' as illustrated in Figure 5.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The EU AI Act's categorization of AI legal assistance tools as 'high-risk' subjects them to heightened responsibilities instead of banning them outright.

“The EU AI Act's categorization of AI legal assistance tools as 'high-risk,' which subjects them to heightened responsibilities instead of banning them outright, may speak to this potential shift [22].”

• cites [22] [22] European Commission. 2021. Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts . Retrieved May 1, 2024 from https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=celex:52021PC0206 COM(2021) 206 final 2021/0106(COD). -Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

UPL regulations prohibit non-lawyers from advising in many states, carrying criminal penalties.

“Moreover, as Section 4.2.2 explains, UPL regulations prohibit non-lawyers from advising in many states, carrying criminal penalties.”

Probably not a substantial claim: may not pass S3 (load-bearing).

This study was reviewed and approved by the Institutional Review Board.

“This study was reviewed and approved by our Institutional Review Board.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The codebook was finalized through multiple all-author meetings.

“The codebook was finalized through multiple all-author meetings.”

Classified as probably not a substantial claim. May combine two claims.

By engaging in multi-turn interactions, LLMs could help users refine their questions, focus on key aspects of their cases, and seek relevant expertise.

“By engaging in multi-turn interactions, LLMs could help users refine their questions, focus on key aspects of their cases, and seek relevant expertise.”

Probably not a substantial claim: may not pass S2 (checkable).

Our 4-dimension framework provides a useful analytical lens that can be applied to future exploring LLM policies in the legal advice and other professional contexts.

“Our 4-dimension framework we have developed provides a useful analytical lens that can be applied to future exploring LLM policies in the legal advice and other professional contexts.”

Probably not a substantial claim: may not pass S3 (load-bearing).

LLM-mediated dialogues could streamline time-consuming processes such as 'screening interviews,' 'first calls,' or 'intake meetings.'

“They felt that LLM-mediated dialogues could streamline time-consuming processes such as 'screening interviews' (P12), 'first calls' (P14), or 'intake meetings' (P15).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants were fully debriefed on the nature and purpose of the study during the workshop.

“Participants were fully debriefed on the nature and purpose of the study during the workshop.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Uncontrolled LLM advice effectively constitutes illegally unauthorized practice of law (UPL).

“Given this gap, P18 argued that uncontrolled LLM advice effectively constitutes illegally unauthorized practice of law (UPL).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The Texas Court provides guidelines for court staff and illustrative examples like in Table 2 that distinguish between permissible and forbidden questions.

“For example, the Texas Court provides guidelines for court staff and illustrative examples like in Table 2.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Policymakers can derive valuable insights to inform LLM policies grounded in the centuries-old wisdom and experience of the legal profession, while also accounting for the challenges presented by LLM technologies.

“Through this approach, we argue that policymakers can derive valuable insights to inform LLM policies grounded in the centuries-old wisdom and experience of the legal profession, while also accounting for the challenges presented by LLM technologies.”

May combine two claims.

Experts prioritize providing users with relevant information while refraining from offering explicit recommendations or opinions, striking a balance between assisting users and maintaining the LLM system's role as an informative tool rather than a decision-maker.

“The concentration of votes in the middle of the distribution suggests that experts prioritize providing users with relevant information while refraining from offering explicit recommendations or opinions, striking a balance between assisting users and maintaining the LLM system's role as an informative tool rather than a decision-maker.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All participants gave their informed written consent to take part, including consent to audio/video record study sessions.

“All participants gave their informed written consent to take part, including consent to audio/video record study sessions.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Advice in harassment cases could further victimize previously affected individuals.

“P6 emphasized the potential for unintended consequences on vulnerable groups, using the example of how advice in harassment cases could further victimize previously affected individuals.”

Classified as probably not a substantial claim.

One of our contributions is to shed light on existing legal and ethical barriers to LLMs' legal advice which have been overlooked in the literature.

“One of our contributions is to shed light on existing legal and ethical barriers to LLMs' legal advice which have been overlooked in the literature.”

May combine two claims.

The distinction between legal information and opinion is not always clear-cut, as exemplified by Grievance Comm. of Bar v. Dacey, where publishing a booklet providing trust and tax information crossed the line into unauthorized legal opinion.

“However, it is important to acknowledge the complexity of distinguishing between legal information and opinion, as the line between the two can often be blurred in practice, as exemplified by cases such as Grievance Comm. of Bar v. Dacey [1], where the court found that publishing a booklet providing trust and tax information crossed the line into unauthorized legal opinion.”

• cites [1] [1] 1966. Grievance Comm. of Bar v. Dacey, 222 A.2d 339 (Conn.), appeal dismissed, 386 U.S. 683. -Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

The aggregated data used to train these systems could gradually skew the LLM's performance to favor majority demographics unless measures are taken to actively protect minority views.

“They cautioned that the aggregated data used to train these systems could gradually skew the LLM's performance to favor majority demographics unless measures are taken to actively protect minority views (P5).”

Probably not a substantial claim: may not pass S3 (load-bearing).

While our taxonomy conceptualizes a concrete set of dimensions, how these dimensions could change the appropriateness of LLM responses remains unexplained.

“Finally, while our taxonomy conceptualizes a concrete set of dimensions, how these dimensions could change the appropriateness of LLM responses remains unexplained.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Our case-based method has utility in engaging expert perspectives on LLM response appropriateness in professional domains beyond the legal sphere.

“Finally, we posit that our case-based method has utility in engaging expert perspectives on LLM response appropriateness in professional domains beyond the legal sphere.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not engage end-users like clients of legal services.

“Third, we did not engage end-users like clients of legal services.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Our participants' responses are conditioned by their prior experience with state-of-the-art LLM technology, such as ChatGPT empowered by GPT-4.

“Second, our participants' responses are conditioned by their prior experience with state-of-the-art LLM technology, such as ChatGPT empowered by GPT-4.”