A study published in Nature earlier this year centres around the AI-Rad Companion Chest X-ray from Siemens Healthineers, an artificial-intelligence based application used to analyse chest X-rays. The study examines the effectiveness of the AI-Rad, its key benefits and features as well as any potential drawbacks and concerns from a diagnostic perspective.
The researchers explain that the AI-Rad is designed to detect five specific radiographic findings: pulmonary lesions, consolidation, atelectasis, pneumothorax and pleural effusion. In terms of how it works, the AI-Rad “solely analyses the posterior-anterior view of chest X-ray images and creates secondary capture DICOM (Digital Imaging and Communications in Medicine) objects reporting on the results of the analysis”. In addition, the AI-Rad provides a ‘confidence score’ on a scale of one t0 10 for each finding to illustrate the algorithm’s certainty for the presence of that particular finding. It is “considered a diagnostic aid to support radiologists in their clinical routine”.
A total of 499 patients were enrolled in the study at random. Their radiographs were evaluated in a consensus decision between a junior radiologist and a senior radiologist with over 20 years of experience, with both radiologists unaware of the study. The AI-Rad performed its own evaluation of the radiographs for comparison.
Findings indicated by both the AI-Rad and the written report were compared to the findings of a ground truth reading, formed via consensus decision from two radiologists after assessing additional radiographs and CT scans.
In terms of performance metrics, the study notes that “the AI-Rad offers a good sensitivity for the detection of pneumothoraces” and “the AI-Rad detected seven out of 10 pneumothoraces correctly.” The study highlights that the AI-Rad “might offer higher sensitivity for certain findings” and the false discovery rate (FDR) of the AI-Rad in certain cases was also “considerably higher.”
Looking at the detection of lung lesions, the AI-Rad’s sensitivity was “superior in compassion to the sensitivity of the written report” however it should be noted that, unlike the AI-Rad, “radiologists immediately evaluate the findings they detect and whether it is worth mentioning in the written report.” The study adds that is is “conceivable that a small, calcified granuloma” that has been present for a long time “may not be mentioned in the written report, but it indicated by the AI-Rad.”
In terms of detecting consolidations, it was noted that the AI-Rad can provide “slightly higher sensitivity for the detection of atelectasis compared to the written report” – however it is still unclear whether small atelectasis were detected by radiologists, but considered not worth mentioning in the report.
The journal goes on to highlight the limitations of the study itself; as the research evaluated the performance of the AI-Rad in isolation against the performance of radiologists, rather than its intended purpose as a clinical support tool.
Overall, the results indicate that the AI-Rad can offer an increased sensitivity for the detection of certain findings compared to the written report, however it is important to note that “this advantage is partially offset by the disadvantage of a higher FDR of the AI-Rad.”
At this current developmental stage, the study notes that “the high NPVs (negative-predictive-values) for the detection of the pre-defined findings” are potentially the greatest benefit of the AI-Rad.
To read the full study, click here.
Citations: Niehoff, J.H., Kalaitzidis, J., Kroeger, J.R. et al. Evaluation of the clinical performance of an AI-based application for the automated analysis of chest X-rays. Sci Rep 13, 3680 (2023). https://doi.org/10.1038/s41598-023-30521-2
