By Anant Madabhushi and Kaustav Bera

How Man and Machine Can Work Together to Diagnose Diseases in Medical Scans

What does AI see in this picture? (Credit: NIH Image Gallery, CC BY-NC)

With artificial intelligence, machines can now examine thousands of medical images – and billions of pixels within these images – to identify patterns too subtle for a radiologist or pathologist to identify.

The machine then uses this information to identify the presence of a disease or estimate its aggressiveness, likelihood of survival or potential response to treatment.

We are engineers at the Center for Computational Imaging and Personalized Diagnostics. Our team works with physicians and statisticians to develop and validate these kinds of tools.

Many worry that this technology aims to replace doctors. But we believe the technology will work in tandem with humans, making them more efficient and helping with decisions on complicated cases.

Artificial Intelligence: Friend or Foe?

Machine Learning and Medical Images

In one study, researchers at Stanford showed that machines were just as accurate as trained dermatologists in distinguishing skin cancers from benign lesions in 100 test images.

In another, computer scientists at Google used an AI approach called deep learning to accurately identify which patients had diabetic retinopathy – a constellation of changes in the retina due to diabetes – from high-resolution photographs of the retina. Another deep learning project at Google successfully predicted cardiovascular disease risk from retina images.

Our group has been developing new ways to identify disease in scans like MRI and CT, as well as digitized tissue slide images.

Artificial Intelligence Continues to Reach New Heights

In biopsied images of heart tissue from 105 patients with heart disease, our algorithms predicted with high accuracy which patients would go on to have heart failure.

In another study involving MRI scans from prostate cancer patients, our computer algorithms identified clinically significant disease in more than 70% of cases where radiologists missed it. In half of the cases where radiologists mistakenly thought that the patient had aggressive prostate cancer, the machine was able to correctly identify that no clinically significant disease was present.

How Man and Machine Can Work Together to Diagnose Diseases in Medical Scans

A CT scan showing cancer in the eye. (Credit: Wellcome Collection, CC BY)


Predicting Outcomes and Treatment Response

Our team has also been developing approaches to predict a patient’s response to specific therapies and monitor early treatments.

Take the case of immunotherapy. Immunotherapy drugs boost the body’s own immune defences to fight against the cancer. They have shown tremendous promise in comparison to traditional chemotherapy, but certain caveats limit their widespread use. Only about 1 in 5 patients with lung cancer might actually respond to immunotherapy regimens. Additionally, these therapies cost upwards of US$200,000 per patient per year and can have toxic effects on the immune system. Physicians need a way to determine accurately which patients might benefit.

The Ethical Implications of Using Artificial Intelligence in Medicine

To identify the likelihood of a successful response before therapy begins, our lab is building software to examine routine diagnostic CT scans of lung tumours. The software looks at tumour texture, intensity and shape, as well as the shape of vessels feeding the nodules. This information might help oncologists optimize the therapeutic dosage or alter a patient’s treatment plan.

There are many cancers and other diseases where computational tools to predict disease aggressiveness or treatment response could aid physicians. For example, a study published this June compared women with breast cancer who had been treated with adjuvant chemotherapy or standard endocrine therapy. For about 70% of the patients, the chemotherapy had no demonstrated benefit compared to the standard approach. Preventing unnecessary and often deleterious chemotherapy thus becomes a key issue for doctors. However, currently, the only way to predict outcome depends upon expensive genomic tests that destroy tissue.

Artificial Intelligence Requires Attention Despite Progress

We are currently working on a new way to interrogate digitized tissue images from biopsies of the breast. Our new project, starting this year, will test the technology on women with breast cancers at Tata Memorial Hospital in Mumbai, India.


Making It Possible

There is a massive opportunity for clinicians, radiologists and pathologists to enrich their decisions with artificial intelligence. That’s particularly true when it comes to building treatment plans tailored to the individual patient.

Before such technology can be used in hospitals, researchers like ourselves need to do further tests to ensure it’s reliable and valid. This can be done by carrying out tests at multiple medical institutions.

Using Artificial Intelligence to Revolutionise Precision Medicine

It’s also important for physicians to be able to interpret the technology. They’re unlikely to adopt technology that cannot be explained by existing biology research. For example, our lung tumour software looks at vessel shape because studies show that the degree of convolutedness of the vessels feeding the tumour can negatively affect drug delivery.

That’s why it’s crucial artificial intelligence researchers like ourselves engage clinicians early in the development process as equal collaborators.


Anant Madabhushi 
Professor of Biomedical Engineering, Case Western Reserve University

Kaustav Bera
Research Associate, Case Western Reserve University 

The Conversation →