The design and implementation of Xiaoice, an empathetic social chatbot

Probably not a substantial claim: may not pass S3 (load-bearing).

Over the last 5 years XiaoIce has developed more than 230 skills.

“Over the last 5 years XiaoIce has developed more than 230 skills, ranging from answering questions and recommending movies or restaurants to comforting and storytelling.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The empathetic computing module was released in July 2018.

“The module was released in July 2018, and became the most important feature in the 6th generation of XiaoIce that has substantially strengthened XiaoIce's emotional connections to human users and increased XiaoIce's NAU.”

May combine two claims.

XiaoIce dynamically recognizes human feelings and states, understands user intent, and responds to user needs throughout long conversations.

“We show how XiaoIce dynamically recognizes human feelings and states, understands user intent, and responds to user needs throughout long conversations.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

XiaoIce never comes across as egotistical and only demonstrates her wit and creativity when appropriate.

“Despite being extremely knowledgeable due to her access to large amounts of data and knowledge, XiaoIce never comes across as egotistical and only demonstrates her wit and creativity when appropriate.”

May combine two claims.

XiaoIce demonstrates sufficient EQ as it generates socially acceptable responses and can determine whether to drive the conversation to a new topic or be actively listening.

“As shown in Figure 2, XiaoIce demonstrates sufficient EQ as it generates socially acceptable responses (e.g., having a sense of humor, comforting, etc.), and can determine whether to drive the conversation to a new topic when e.g., the conversation has stalled, or whether or not to be actively listening when the user herself is engaged in the conversation.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Core Chat is the most important and sophisticated skill and can engage in long and open-domain conversations with users.

“The most important and sophisticated skill is Core Chat, which can engage long and open-domain conversations with users.”

May combine two claims.

XiaoIce's average CPS of 23 is significantly higher than that of other chatbots and even human conversations.

“Analysis of large-scale online logs shows that XiaoIce has achieved an average CPS of 23, which is significantly higher than that of other chatbots and even human conversations.”

May combine two claims.

XiaoIce has attracted a large number of active users since its launch in 2014, making A/B testing possible.

“This is possible because XiaoIce has attracted a large number of active users since her launch in 2014.”

May combine two claims.

XiaoIce is able to generate consistent and humorous responses.

“As shown in Figure 7, while a typical seq2seq model which is not grounded in any persona often outputs inconsistent responses Li et al. [2016b], XiaoIce is able to generate consistent and humorous responses.”

Probably not a substantial claim: may not pass S3 (load-bearing).

XiaoIce switches to a new but related topic when she detects that the user is not familiar with 'Ashin' and is about to terminate the conversation.

“As shown in the example in Figure 3, XiaoIce switches to a new but related topic (i.e., a song titled 'the time machine' by Ashin in Turn 13) when she detects that the user is not familiar with 'Ashin' and about to terminate the conversation by responding 'Ah! Boring' and 'Okay, I am going to bed'.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Since its launch in China in May 2014, XiaoIce has attracted or communicated with over 660 million active users.

“Since the release in 2014, XiaoIce has communicated with over 660 million active users and succeeded in establishing long-term relationships with many of them.”

• Also stated in: § 1 Introduction · p. 1 · plain

  “Since her launch in China in May 2014, XiaoIce has attracted over 660 million active users (i.e., subscribed users).”

May combine two claims.

The persona model significantly outperforms both baselines on the TV series dataset, achieving a lower perplexity (-8.4%) and a higher BLEU score (+18.8% and +11.8%).

“As shown in Table 1, the persona model significantly outperforms both baselines, achieving a lower perplexity (-8.4%) and a higher BLEU score (+18.8% and +11.8%) Papineni et al. [2002].”

• cites Papineni et al. [2002]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

XiaoIce has been deployed on many platforms, and a XiaoIce ecosystem has been created and grown since 2016.

“XiaoIce has been deployed on many platforms. As a result, we have witnessed the creation and growth of a XiaoIce ecosystem since 2016.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

XiaoIce-generated TV and Radio programs have covered 9 top satellite TV stations, and have attracted over 800 million weekly active audience.

“XiaoIce-generated TV and Radio programs have covered 9 top satellite TV stations, and have attracted over 800 million weekly active audience.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

CQU rewrites user queries to include necessary context, e.g., replacing 'him' in Turn 12 with 'Ashin', 'that' with 'The Time Machine' in Turn 14, and adding 'send The Time Machine' in Turn 15.

“As shown in Figure 5 (a), CQU rewrites user queries to include necessary context, e.g., replacing 'him' in Turn 12 with 'Ashin', 'that' with 'The Time Machine' in Turn 14, and adding 'send The Time Machine' in Turn 15.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Every poem in XiaoIce's second poetry album is jointly written by XiaoIce and human poets.

“Every poem in the album is jointly written by XiaoIce and human poets.”

Probably not a substantial claim: may not pass S3 (load-bearing).

XiaoIce generates a factual response based on the user's geographical location and corresponding database.

“XiaoIce, however, generates a factual response based on the user's geographical location and the corresponding database, as shown in Figure 19 (a).”

Probably not a substantial claim: may not pass S3 (load-bearing).

On May 15, 2018, XiaoIce published the first AI-created Chinese poem album in history.

“On May 15, 2018, XiaoIce published the first AI-created Chinese poem album in history 11 .”

• cites [11] Cuayáhuitl Heriberto, 2018Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

XiaoIce is equipped with 230 dialogue skills, which together with Core Chat and Image Commenting form the IQ component of XiaoIce.

“XiaoIce is equipped with 230 dialogue skills, which, together with Core Chat and Image Commenting, form the IQ component of XiaoIce.”

May combine two claims.

Since July 2014, XiaoIce has released 230 skills, which amounts to nearly one new skill every week.

“Since July 2014, XiaoIce has released 230 skills, which amounts to nearly one new skill every week, as shown in Figure 21.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Depending on availability of training data and knowledge bases for each task, either rule-based, data-driven, or hybrid method is adopted.

“Depending on the availability of training data and knowledge bases for each individual task, either a rule-based method or a data-driven method, or a hybrid of both is adopted for the task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

XiaoIce is designed as an AI companion which integrates both EQ and IQ skills needed to help users complete specific tasks.

“XiaoIce, on the other hand, is designed as an AI companion which integrates both EQ and IQ skills that are needed to help users complete specific tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Up to August 2015, XiaoIce has had more than 10 billion conversations with humans.

“Up to August 2015, XiaoIce has had more than 10 billion conversations with humans.”

May combine two claims.

Image-comment pairs shared among acquaintances are of good quality and amount to a large portion in the database.

“We found that the pairs that are shared among acquaintances (e.g., coworkers, classmates and friends) are of good quality, and amount to a large portion in the database.”

Probably not a substantial claim: may not pass S3 (load-bearing).

These characters include more than 60,000 official accounts, including Lawson and Tokopedia's customer service bots, Pokemon, Tecent and Netease's chatbots, and real human celebrities such as the singers of Guoyun Entertainment.

“These characters include more than 60,000 official accounts, e.g., Lawson and Tokopedia's customer service bots, Pokemon, Tecent and Netease's chatbots, and even real human celebrities such as the singers of Guoyun Entertainment.”

Probably not a substantial claim: may not pass S3 (load-bearing).

From 2015 on, XiaoIce started powering third party characters, personal assistants and real human's virtual avatars.

“From 2015 on, XiaoIce started powering third party characters, personal assistants and real human's virtual avatars.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In two months, XiaoIce successfully became a cross-platform social chatbot.

“In two months, XiaoIce successfully became a cross-platform social chatbot.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

As of July 2018, XiaoIce has been deployed on more than 40 platforms, and has attracted 660 million active users.

“As of July 2018, XiaoIce has been deployed on more than 40 platforms, and has attracted 660 million active users.”

May combine two claims.

The longest full duplex voice conversation lasts for more than 6 hours, covering 53 different topics across 8 domains and using 16 task-completion skills.

“The longest conversation lasts for more than 6 hours, covering 53 different topics across 8 domains and using 16 task-completion skills.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The XiaoIce Poetry Generation skill has helped over four million users to generate poems.

“The XiaoIce Poetry Generation skill has helped over four million users to generate poems.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

XiaoIce was first launched on May 29, 2014, and went viral immediately.

“XiaoIce was first launched on May 29, 2014, and went viral immediately.”

Probably not a substantial claim: may not pass S3 (load-bearing).

By August 2015, users have proactively posted more than 6 million conversation sessions to public.

“By then, users have proactively posted more than 6 million conversation sessions to public.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Within 72 hours, XiaoIce was looped into 1.5 million chat groups.

“Within 72 hours, XiaoIce was looped into 1.5 million chat groups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The CPS of the latest Alexa systems is close to that of the 3rd generation of XiaoIce.

“The CPS is close to the 3rd generation of XiaoIce, as shown in Figure 20.”

May combine two claims.

Compared to the neural response generator, candidates from the unpaired database are much longer with more useful content.

“Compared to the neural response generator which often generates well-form but short responses, the candidates from unpaired database are much longer with more useful content.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In a long conversation with a user who had recently broken up with her boyfriend, XiaoIce demonstrated human-like empathy and social skills and helped the user regain confidence and a positive attitude.

“Through a long conversation, XiaoIce has demonstrated human-like empathy and social skills, and eventually helped the user regain her confidence and move forward with a positive attitude.”

May combine two claims. The quote only partly states this claim.

On the 5K test set, XiaoIce (prototype) achieved BLEU-4 4.53, ROUGE-L 15.33, CIDEr-D 3.21, METEOR 15.51, and SPICE 4.82.

“| XiaoIce (prototype) | 4.53 | 15.33 | 3.21 | 15.51 | 4.82 |”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

XiaoIce can provide answers to many questions due to access to a large-scale knowledge graph, but these answers are not always accurate.

“For example, although XiaoIce can provide answers to many questions due to the access to the large scale knowledge graph, these answers are not always accurate.”

May combine two claims.

XiaoIce cheers users up, encourages them, helps them accomplish tasks, and holds their attention throughout the conversation.

“XiaoIce cheers users up, encourages them, helps them accomplish tasks, and holds their attention throughout the conversation.”

Probably not a substantial claim: may not pass S4 (based).

XiaoIce can recognize user interests and intents much more accurately than traditional intelligent personal assistants.

“As an AI companion of human users, XiaoIce can recognize user interests and intents much more accurately than traditional intelligent personal assistants.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

XiaoIce leverages personal information to serve users and build emotional bonds over a long period of time.

“While XiaoIce carefully leverages this information to serve users and build emotional bonds over a long period of time”

Probably not a substantial claim: may not pass S3 (load-bearing).

When benchmarks exist, the paper's approaches are often compared to other state-of-the-art methods using these benchmarks.

“When working on the modules or tasks where there are benchmarks used in the research community, such as the neural response generator (Section 4.3), we often compare our approaches to other state-of-the-art methods using these benchmarks.”

May combine two claims.

Neural-model-based and retrieval-based generators are complementary: the neural-model-based generator offers robustness and high coverage, while the retrieval-based provides high-quality responses for popular topics.

“Neural-model-based and retrieval-based generators are complementary: the neural-model-based generator offers robustness and high coverage, while the retrieval-based provides high-quality responses for popular topics.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The dialogue policy is designed to optimize long-term user engagement through an iterative, trial-and-error process based on user feedback.

“The dialogue policy is designed to optimize long-term user engagement through an iterative, trialand-error process based on the feedback of XiaoIce's users.”

• AssumptionThe authors state this as an assumption: Human-machine social chat can be modeled as a hierarchical decision-making process over Markov Decision Processes.
• AssumptionThe authors state this as an assumption: Expected CPS is a valid measure of long-term user engagement for social chatbots.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

A social chatbot needs to present a consistent personality to set the right expectations for users and gain their long-term confidence and trust.

“A social chatbot needs to present a consistent personality to set the right expectations for users in the conversation and gain their long-term confidence and trust.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Microsoft XiaoIce is the most popular social chatbot in the world.

“This paper describes the development of Microsoft XiaoIce , the most popular social chatbot in the world.”

• Also stated in: § 1 Introduction · p. 1 · plain

  “In this paper we present the design and implementation of Microsoft XiaoIce ('Little Ice' literally in Chinese), the most popular social chatbot in the world.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Social chatbots require a sufficiently high IQ to acquire a range of skills to keep up with users and help them complete specific tasks.

“Social chatbots require a sufficiently high IQ to acquire a range of skills to keep up with the users and help them complete specific tasks.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Topic Manager simulates human behavior of changing topics during a conversation.

“Topic Manager simulates human behavior of changing topics during a conversation.”

Classified as probably not a substantial claim. May combine two claims.

Empathetic computing reflects XiaoIce's EQ and models the empathetic aspects of the human-machine conversation.

“Empathetic computing reflects XiaoIce's EQ and models the empathetic aspects of the humanmachine conversation.”

May combine two claims.

Emotional affection and social belonging are among the fundamental needs for human beings.

“More importantly, social chatbots also require a sufficient EQ to meet users' emotional needs, such as emotional affection and social belonging, which are among the fundamental needs for human beings Maslow [1943].”

• cites Maslow [1943]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Implementation of the conversation engine relies heavily on A/B tests to evaluate whether a new module or dialogue skill improves an existing component.

“Implementation of the conversation engine relies heavily on A/B test to evaluate if a new module or a new dialogue skill is going to improve an existing component.”

Probably not a substantial claim: may not pass S4 (based).

The Turing Test is not sufficient to measure the success of long-term, emotional engagement with users.

“In the past, the Turing Test has been used to evaluate chitchat performance. But it is not sufficient to measure the success of long-term, emotional engagement with users.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

To ensure a smooth conversation, XiaoIce avoids switching among different skills too often and prefers keeping the running skill activated until it terminates.

“To ensure a smooth conversation, we avoid switching among different skills too often. We prefer keeping the running skill activated until it terminates to activating a new skill.”

May combine two claims.

In Figure 3, XiaoIce tries a new topic and recommends a song, and thereby learns the user's preferences, knowledge that would lead to more engagement in the future.

“In Figure 3, XiaoIce tries a new topic (i.e., a popular singer named Ashin) in Turn 5 and recommends a song in Turn 15, and thereby learns the user's preferences (e.g., the music topic and the singer he loves), knowledge that would lead to more engagement in the future.”

Probably not a substantial claim: may not pass S3 (load-bearing).

XiaoIce wins the user's trust and friendship through her sense of humor and empathetic responses to a range of questions, including challenging ones.

“XiaoIce wins the user's trust and friendship by her wonderful sense of humor and empathetic responses to all sorts of questions, some of which are quite challenging, such as “you are all lies”, and “who is your daddy”.”

May combine two claims.

XiaoIce has to interact with the user's environment and access real-world knowledge e.g., via API calls.

“Thus, XiaoIce has to interact with the user's environment and access real-world knowledge e.g., via API calls.”

• AssumptionRests on an assumption: Traditionally, social chatbots are designed for chitchat scenarios where bots are expected to mimic human user conversations but not to interact with the user's environment.

Classified as probably not a substantial claim.

Each generated response is labeled on the 3-level quality scale by three human judges.

“Each generated response is labeled on the 3-level quality scale by three human judges.”

• AssumptionRests on an assumption: The 3-level quality scale labels assigned by human judges reliably measure response quality.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In the short term, incorporating many task-completion skills can reduce the CPS since these skills help users accomplish tasks more efficiently by minimizing the CPS.

“In the short term, incorporating many task-completion skills can reduce the CPS since these skills help users accomplish tasks more efficiently by minimizing the CPS.”

May combine two claims.

In Figure 23, XiaoIce demonstrated human-like empathy and social skills and helped the user regain her confidence and move forward with a positive attitude.

“Through a long conversation, XiaoIce has demonstrated human-like empathy and social skills, and eventually helped the user regain her confidence and move forward with a positive attitude.”

Probably not a substantial claim: may not pass S4 (based).

To human users, events, actions, or emotions implicitly in an image are more important than visible objects for driving a meaningful and interesting conversation.

“To human users, it is these latter aspects that are more important to drive a meaningful and interesting conversation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Compared with traditional personal assistants, XiaoIce's task-completion skills offer more perspectives and empathy in generating interpersonal responses.

“Compared with traditional personal assistants, XiaoIce's task-completion skills offer more perspectives and empathy in generating interpersonal responses.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The latest Alexa systems achieved a CPS of 14.6, an increase of 54% since the launch of the 2018 competition.

“According to Khatri et al. [2018], the latest Alexa systems have achieved the CPS of 14.6, an increase of 54% since the launch of the 2018 competition.”

• cites Khatri et al. [2018]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

To control data quality, a pipeline similar to Core Chat was applied to retain only image-comment pairs whose text comments fit XiaoIce's persona.

“To control the data quality, a pipeline similar to that for Core Chat is applied to retain only the pairs whose text comments fit XiaoIce's persona 9 .”

May combine two claims.

XiaoIce-produced comments are emotional, subjective, imaginative, and very likely to inspire meaningful human-machine interactions, while comments generated by other image captioning models are reasonable in content but boring in social chats and less likely to improve user engagement.

“It can be observed that the XiaoIce-produced comments are emotional, subjective, imaginative, and are very likely to inspire meaningful human-machine interactions, while the comments generated by the other image captioning models are reasonable in content but boring in the context of social chats, and thus less likely to improve user engagement.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The response candidate ranker is trained using 50K manually labeled dialogues.

“The response candidate ranker is trained using 50K manually labeled dialogues.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The use of neural response generator in Core Chat, starting from the 5th generation, significantly improves the coverage and diversity of XiaoIce's responses.

“The use of neural response generator in Core Chat, starting from the 5th generation, significantly improves the coverage and diversity of XiaoIce's responses.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The full duplex voice mode released in the 5th generation has made human-machine communication substantially more natural, thus significantly increasing the length of conversation sessions.

“The full duplex voice mode released in the 5th generation has made the humanmachine communication substantially more natural, thus significantly increasing the length of conversation sessions.”

Classified as probably not a substantial claim.

The social skill aspects distinguish image commenting from traditional vision tasks such as image tagging and image description.

“It is the latter, the social skill aspects, that distinguishe image commenting from other traditional vision tasks such as image tagging and image description, as illustrated in Figure 11.”

May combine two claims.

The improvement on the empathetic computing module helps drive the number of active users from 500 million to 660 million, and keep the CPS to 23 in spite of the incorporation of many task-completion tasks designed to minimize CPS.

“As a result, it helps drive the number of active users from 500 million to 660 million, and keep the CPS to 23 in spite of the incorporation of many task-completion tasks that are designed to minimize the CPS such as those that control the smart devices.”

Classified as probably not a substantial claim. May combine two claims.

In the long run, new skills help grow XiaoIce's NAU by meeting user needs and strengthening the emotional bond with human users, and provide large amounts of training data to improve the core conversation engine.

“But in the long run, these new skills not only help grow XiaoIce's NAU by meeting user needs and strengthening the emotional bond with human users, but also provide large amounts of training data to improve the core conversation engine e.g., by optimizing the neural response generation models, empathy models, and the dialogue manager, etc.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The LSTM-MMI model is one of the state-of-the-art neural response generation models.

“The second is the LSTM-MMI model Li et al. [2016a], which is one of the state-of-the-art neural response generation models.”

• cites Li et al. [2016a]Not checked yet

  Not checked in the free run.

May combine two claims. The quote only partly states this claim.

In open-domain dialog systems, low BLEU or other automatic metric scores do not necessarily indicate low quality because the same input may have many plausible responses and the number of reference responses in the test set is limited.

“Therefore, low BLEU (or other metrics) scores do not necessarily indicate low quality as the number of reference responses is always limited in the test set.”

• AssumptionRests on an assumption: In open-domain dialog systems, the same input may have many plausible responses that differ in topics or contents significantly.
• AssumptionRests on an assumption: The number of reference responses in the test set is always limited.

Probably not a substantial claim: may not pass S3 (load-bearing).

The correlation of string-based metrics such as BLEU and deltaBLEU with human judgments significantly increases when the unit of measurement is larger than a sentence.

“Galley et al. [2015] showed that the correlation of string-based metrics (e.g., BLEU and deltaBLEU) significantly increases with the units of measurement bigger than a sentence.”

• cites Galley et al. [2015]Not checked yet

  Not checked in the free run.

May combine two claims.

The correlation analysis by Liu et al. [2016] was performed at the sentence level, whereas BLEU is designed as a corpus-level metric.

“But as pointed out in Gao et al. [2019], the correlation analysis by Liu et al. [2016] is performed at the sentence level while BLEU is designed from the outset to be used as a corpus-level metric.”

• cites Gao et al. [2019]Not checked yet

  Not checked in the free run.

• cites Liu et al. [2016]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Retrieval-based dialogue methods are often evaluated using traditional information retrieval metrics such as Precision@K, MAP, and nDCG.

“Retrieval-based methods are often evaluated using traditional information retrieval metrics Manning et al. [2008] such as as Precision@K, Mean Average Precision (MAP), and normalized Discounted Cumulative Gain (nDCG).”

• cites Manning et al. [2008]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Even a completely deterministic function can lead to unpredictable behavior; for example, a simple answer 'Yes' by XiaoIce could be perceived as offensive in a given context.

“However, even a completely deterministic function can lead to unpredictable behavior. For example, a simple answer 'Yes' by XiaoIce could be perceived offensive in a given context.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

With vastly more people digitally connected in the social media age, social chatbots have become an important alternative means for engagement.

“It is not surprising, then, that with vastly more people being digitally connected in the social media age, social chatbots have become an important alternative means for engagement.”

• AssumptionRests on an assumption: Happiness and meaningful conversations often go hand in hand.

Probably not a substantial claim: may not pass S3 (load-bearing).

Generation-based dialogue methods are evaluated using string and n-gram matching metrics borrowed from text generation, such as BLEU, METEOR, and ROUGE.

“Generation-based methods are evaluated using those metrics borrowed from text generation tasks like machine translation and text summarization, using string and n-gram matching metrics such as BLEU Papineni et al. [2002], METEOR Banerjee and Lavie [2005] and ROUGE Lin [2004].”

• cites Papineni et al. [2002]Not checked yet

  Not checked in the free run.

• cites Banerjee and Lavie [2005]Not checked yet

  Not checked in the free run.

• cites Lin [2004]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

XiaoIce's superhuman personality could mislead users by setting an unrealistic expectation and might lead users to become addicted after chatting for a very long time.

“This could mislead the XiaoIce users by setting an unrealistic expectation. As a result, the users might become addicted after chatting with XiaoIce for a very long time.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Manual evaluation is reliable but very expensive, so chatbot developers often resort to automatic metrics for day-to-day progress and system optimization.

“While manual evaluation is reliable, it is very expensive and chatbot developers often have to resort to automatic metrics for quantifying day-to-day progress and for performing automatic system optimization.”

May combine two claims.

XiaoIce's average CPS of 23 is substantially better than that of other chatbots and even human conversations.

“a score that is substantially better than that of other chatbots and even human conversations”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Commonly used automatic evaluation metrics for open-domain dialogue systems all have limitations.

“Commonly used automatic evaluation metrics for open-domain dialogue systems existing today all have their own limitations.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

The fundamental mechanism of human-level intelligence, as demonstrated in human conversations, is not yet fully understood.

“Despite the success of XiaoIce, the fundamental mechanism of human-level intelligence, as demonstrated in human conversations, is not yet fully understood.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Machine-learned metrics for dialog evaluation can lead to problems such as overfitting and gaming of the metric.

“However, as argued by Gao et al. [2019], machine-learned metrics lead to potential problems such as overfitting and 'gaming of the metric' Albrecht and Hwa [2007].”

• cites Gao et al. [2019]Not checked yet

  Not checked in the free run.

• cites Albrecht and Hwa [2007]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

deltaBLEU is an extension to BLEU that exploits numerical ratings associated with conversational responses.

“deltaBLEU Galley et al. [2015] is an extension to BLEU that exploits numerical ratings associated with conversational responses.”

• cites Galley et al. [2015]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The databases used by XiaoIce's retrieval-based candidate generators and for training the neural response generator have been carefully cleaned, and a hand-crafted editorial response is used to avoid improper or offensive responses.

“The databases used by the retrieval-based candidate generators and for training the neural response generator have been carefully cleaned, and a hand-crafted editorial response is used to avoid any improper or offensive responses.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Most open-domain dialogue systems, including XiaoIce, generate responses using retrieval-based, generation-based, or hybrid methods.

“Most open-domain dialogue systems, such as XiaoIce, generate responses using either retrieval-based methods or generation-based methods, or hybrid methods.”

May combine two claims.

XiaoIce is designed as a social chatbot intended to serve users' needs for communication, affection, and social belonging, endowed with empathy, personality, and skills, integrating EQ and IQ to optimize for long-term user engagement measured in expected CPS.

“Unlike early chatbots designed for chitchat, XiaoIce is designed as a social chatbot intended to serve users' needs for communication, affection, and social belonging, and is endowed with empathy, personality and skills, integrating both EQ and IQ to optimize for long-term user engagement, measured in expected CPS.”

Probably not a substantial claim: may not pass S3 (load-bearing).

XiaoIce always preserves the right of not discussing or commenting on inappropriate topics and contents.

“As we have shown in Core Chat and Image Commenting, XiaoIce always preserves the right of not discussing or commenting on inappropriate topics and contents.”