M. D. Anderson Breaks With IBM Watson, Raising Questions About Artificial Intelligence in Oncology

Abstract

In 2012, the University of Texas M. D. Anderson Cancer Center in Houston partnered with IBM to develop the artificial intelligence program, called IBM Watson, as a clinical decision tool in oncology. Five years and $62 million later, M. D. Anderson let its contract with IBM expire before anyone used Watson on actual patients. Last February, a university audit of the project exposed many procurement problems, cost overruns, and delays. Although the audit took no position on Watson’s scientific basis or functional capabilities, it did describe challenges with assimilating Watson into the hospital setting. Experts familiar with Watson’s applications in oncology describe problems with the system’s ability to digest written case reports, doctors’ notes, and other text-heavy information generated in medical care. That type of unstructured data differs from the structured data entered into drop-down boxes and other point-and click fields in an electronic medical record that Watson can more readily interpret. “The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated,” said Amy Abernethy, M.D., chief medical officer at Flatiron Health, a New York–based health care technology company, and former director of cancer research at the Duke Cancer Institute in Durham, N.C. Watson is famous for beating human contestants on Jeopardy in 2011. It can read 800 million pages per second, and it draws on the combined contents of PubMed, the National Cancer Institute’s Drug Dictionary, the Sanger Institute’s Catalogue of Somatic Mutations in Cancer database, every study registered on clinicaltrials.gov, and many other resources. IBM Watson Health, a specialized division commercializing the system in genomics and drug discovery, as well as oncology, has relationships with roughly 50 other institutions. “The M. D. Anderson experience is telling us that solving data quality problems in unstructured data is a much bigger challenge for artificial intelligence than was first anticipated.” Watson’s selling point is that it helps doctors stay current with the volume of new findings published every day. IBM has been working on natural language processing capabilities that allow Watson to increasingly understand human speech. Oncologists have been trying to teach the computer system to think like a cancer doctor by training it with real and made-up cases. At M. D. Anderson, those training exercises started with a pilot focused on leukemia. But medical diagnosis is more complicated than Jeopardy questions. Institutions use medical terms in different ways, and despite the best efforts of software engineers, Watson still can’t interpret medical language as well as humans can. At M. D. Anderson, for instance, Watson couldn’t reliably distinguish the acronym for acute lymphoblastic leukemia, ALL, from the shorthand for allergy, which is often also written ALL. “To do that correctly, Watson would have to interpret those written characters in the appropriate context,” Abernethy said. “It would have to know that ALL in the context of high white blood cell counts or a bone marrow transplant means leukemia and not something else.” M. D. Anderson suspended its leukemia pilot midstream before switching to lung cancer “because project leaders thought that area would provide greater opportunity for a timely completion,” the UT audit stated. Andrew Seidman, M.D., a medical oncologist at the Memorial Sloan Kettering Cancer Center in New York, has been trying to teach Watson how to treat breast cancer. He said that to bring its accumulated knowledge to bear on treating patients at the hospital, Watson needs to be better integrated with electronic medical records. “The more efficient way to do that would be for it to extract medical attributes automatically,” Seidman said. “But many of the attributes we want Watson to digest aren’t found in categorical structured data—they’re buried in narrative form in doctors’ consultation notes.” And though natural language processing is central to Watson’s functionality, Seidman said that “it remains a work in progress.” Andrew Norden, M.D., deputy chief health officer at IBM Watson Health, acknowledged the need to clarify abbreviations in unstructured medical notes and “other things Watson hasn’t seen before.” He added, “But this isn’t rocket science—the trick is getting the right digital content to Watson so that Watson can read it.” In M. D. Anderson’s case, a third party—the London-based consulting firm PricewaterhouseCoopers—was contracted to develop a bioinformatics tool to integrate Watson with the hospital’s record system. According to M. D. Anderson’s audit, Watson’s treatment recommendations during the lung cancer pilot agreed with those of its human teachers nearly 90% of the time. “This is a very high level of accuracy,” Norden said.Abernethy , however, was more skeptical. “What does 90%