The Critical Need for Human Oversight in Healthcare AI

While AI holds great promise for automating and improving many healthcare processes and tasks - including clinical decision support - there are risks that some users will become overly dependent on these systems in ways that could be potentially detrimental to patients, says attorney Lee Kim, senior principal of cybersecurity and privacy at the Healthcare Information and Management Systems Society.

Some applications of AI in healthcare today, such as clinical support tools that help radiologists identify cancer on mammograms, are lifesaving, Kim said.

"Automation and AI, if left unchecked - it could be a real problem," she said. "It's human nature for people to be a little bit lazy or complacent about things - to assume these AI systems are inherently intelligent and that they would make the right decisions," she said in an interview with Information Security Media Group conducted during the recent HIMSS cyber forum in Boston.

"Right now, clinicians are comfortable with things like ensuring that what comes out of AI-based clinical decision support is likely legitimate and in line with clinical protocols," Kim said. "But laziness is creeping in already. I've heard from clinicians that may no longer keep in their heads drug-drug or drug allergy interactions because the decision support system pops it up automatically. That's a slippery slope.

"I would plead with people that we should always have some kind of human oversight in review, otherwise the patient safety, values, judgements and the intimate connections healthcare professionals have with their patients - the human element - will be gone."

In the interview (see audio link below photo) Kim also discusses:

• Challenges for publicly traded healthcare companies involving compliance with the U.S. Securities and Exchange Commission's new cybersecurity reporting requirements;
• Emerging regulatory issues for healthcare sector entities to closely monitor involving the Federal Trade Commission and the Department of Health and Human Services;
• Deepfakes and other growing threats facing healthcare sector entities in the months ahead.

Kim has served as a team leader of the U.S. Department of Homeland Security's analytic exchange program and as a member of the National Cybersecurity Training and Education Center National Visiting Committee. Before joining HIMSS, she practiced law in the areas of IT, healthcare technology, intellectual property and privacy and security. She also previously worked in the healthcare technology field.