E5174. Building IQ for AI: Common Causes for Inaccurate Machine Learning Algorithm Image Analysis Results and Best Practice Use: A Pictorial Review
Authors
Vivian Ho;
University of Washington
Zachary Miller;
University of Washington
Brian Bresnahan;
University of Washington
Nathan Cross;
University of Washington
Matt Cham;
University of Washington
Jonathan Medverd;
University of Washington
Background
Image analysis-based, machine learning algorithms are being rapidly developed and commercialized. Many such artificial intelligence (AI) tools are becoming adopted into clinical practice. Radiologists need to understand the roles of AI tools in radiology, their performance characteristics and limitations, and how to avoid potential pitfalls to define and develop best practice use. This pictorial review will illustrate causes of AI image analysis tool false positive and false negative results, provide a framework for an AI quality assurance (QA) program, and propose guidelines for the use of AI tools as they pertain to resident education. Radiologist awareness of AI tool challenges and “blind spots” will benefit utilization of these tools in their practice, and residents can learn about pros/cons of AI tools during their training.
Educational Goals / Teaching Points
After this education session, learners will be able to: describe benefits, limitations, and potential pitfalls of using AI algorithms in radiology; outline characteristics of best practice-focused AI implementation guidelines, using a practical example developed in an academic medical center that is applicable to many radiology practice settings; and consider factors associated with AI tool implementation in clinical practice as they pertain to resident education.
Key Anatomic/Physiologic Issues and Imaging Findings/Techniques
Pulmonary embolism algorithm – false positive causes: flow artifact, contrast timing, pulmonary nodules, beam hardening artifact, mucus plugging, intravascular supportive equipment, motion artifact, and partial volume averaging. Pulmonary embolism algorithm – false negative causes: total or near total occlusive lesions, parenchymal consolidation/atelectasis, diffuse lung disease. Aortic dissection algorithm – false positive causes: atherosclerosis, aneurysm or ectasia, ulcerated plaques, penetrating atherosclerotic ulcer. Aortic dissection algorithm – false negative causes: unclear; algorithm “black box.” Nonalgorithmic failure modes: examination not sent to or processed by AI engine, AI algorithm result not available at the time of examination interpretation, true positive AI algorithm result not recognized by radiologist, natural language processing errors during QA process.
Conclusion
Radiologist and resident awareness of machine learning, image analysis-based algorithm benefits and limitations is needed to support optimal use of AI tools in daily practice. More studies are needed to determine appropriate utilization and conduct ongoing quality monitoring after AI tools are implemented in radiology practices.
