Homework answers / question archive / IBM Watson—Not Yet Ready for Prime Time? The University of Texas MD Anderson Cancer Center is one of the world’s most respected medical centers—devoted exclusively to cancer patient care, research, education, and preven- tion

IBM Watson—Not Yet Ready for Prime Time? The University of Texas MD Anderson Cancer Center is one of the world’s most respected medical centers—devoted exclusively to cancer patient care, research, education, and preven- tion

Writing

Share With

IBM Watson—Not Yet Ready for Prime Time?

The University of Texas MD Anderson Cancer Center is one of the world’s most respected medical centers—devoted exclusively to cancer patient care, research, education, and preven- tion. Its 21,000 employees are focused on one mission: to end cancer. Each year, the center treats over 100,000 people, and it is currently running more than 1,200 clinical trials designed to test various types of cancer treatments.
In 2012, the center launched an ambitious project in collaboration with IBM to create the Oncology Expert Adviser (OEA), a learning system employing the IBM Watson cognitive com- puting program. OEA was designed to read and learn from MD Anderson’s vast database of electronic medical records, academic literature, research data, and treatment options. In theory, OEA would glean patterns from all this data and use that knowledge to make suggestions to improve individual patient cancer care. In addition, OEA was intended to match cancer patients to appropriate clinical trials to offer patients an opportunity to fight their cancers by participating in trials  of new therapies.

Unfortunately, after five years of effort (at a cost of $62 million), the center has not achieved its desired results, according to an assessment performed by the University of Texas System Audit Office. Incorporating machine learning software into complex healthcare settings is extremely challenging, and according to the audit report, the system never reached a develop- ment phase in which it could be used in a clinical setting. The audit uncovered project manage- ment failings as well as limitations in the program’s ability to integrate with other hospital systems.
Initially, the OEA pilot project was focused on leukemia, but that effort was “suspended mid- project” because of lack of progress, and the project was refocused on lung cancer. The lung- cancer system was tested as a pilot project in 2015 and was able to suggest the same treatment plan as MD Anderson physicians in 90 percent of select cases. However, in order for the system to be useful in making complex healthcare-related recommendations, medical personal need to know how OEA reached a conclusion and what data and logic was used. A visualization tool called WatsonPaths attempted to fill this need; however, that tool was also complex, requiring a high level of technical sophistication to interpret how Watson arrived at its conclusion.
Another issue noted in the audit of the system is that the lung-cancer pilot was conducted using data from the hospital’s old electronic records system, which was replaced in 2016, and the project team has been unable to integrate the pilot program with the hospital’s current elec- tronic health records. In addition, clinical trial and drug-protocol data in the OEA system are outdated. Before the program could be tested further, major rework would be required to first convert data from the old system to make it compatible with the new system, and then retrain OEA on all the data in the new records system.
Another goal of the project was to make OEA widely available to physicians at partner hospitals outside MD Anderson. However, the audit found that the program was never piloted with partner hospitals. The audit cited cybersecurity concerns and “lack of engagement or inter- est” by partner hospitals as factors that prevented the testing of the technology outside MD Anderson.
Defining a successful treatment plan for lung cancer is certainly a challenging task, more so than many other problems to which the computing power of Watson has been applied. OEA did

ot prove that vast amounts of patient data and knowledge of various courses of cancer treatment are as significant as advocates of artificial intelligence make them out to be. MD Anderson and the OEA project still needs to provide evidence that the technology could be developed to a level at which it would reliably improve patient outcomes, lower costs, or provide some other benefit.

Critical Thinking Questions
1. As noted in the case, the OEA pilot system was tested in 2015 and was able to suggest the same treatment plan as MD Anderson physicians on 90 percent of select cases. Should MD Anderson accept this degree of accuracy? How should it determine an acceptable level of accuracy for such a system?
2. What key learnings did MD Anderson gain from this effort that may influence future efforts?
3. Should MD Anderson personnel have known that in choosing a new health-records system they would need to restart the entire OEA project? If so, what factors may have caused them to make this choice?