The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

Samuel G Armato; Michael F McNitt-Gray; Anthony P Reeves; Charles R Meyer; Geoffrey McLennan; Denise R Aberle; Ella A Kazerooni; Heber MacMahon; Edwin J.R van Beek; David Yankelevitz; Eric A Hoffman; Claudia I Henschke; Rachael Y Roberts; Matthew S Brown; Roger M Engelmann; Richard C Pais; Christopher W Piker; David Qing; Masha Kocherginsky; Barbara Y Croft; Laurence P Clarke

doi:10.1016/j.acra.2007.07.008

Back

The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

Journal article

Peer reviewed

The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans

Samuel G Armato, Michael F McNitt-Gray, Anthony P Reeves, Charles R Meyer, Geoffrey McLennan, Denise R Aberle, Ella A Kazerooni, Heber MacMahon, Edwin J.R van Beek, David Yankelevitz, …

Academic radiology, Vol.14(11), pp.1409-1421

2007

DOI: 10.1016/j.acra.2007.07.008

PMCID: PMC2290739

PMID: 17964464

Files and links (1)

url

https://www.research.ed.ac.uk/en/publications/bd15a3b1-f8cc-4d7d-857a-2aaab017490aView

Open Access

Abstract

The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the “truth” against which nodule-based studies are measured. Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial “blinded read” phase, radiologists independently marked lesions they identified as “nodule ≥3 mm (diameter),” “nodule <3 mm,” or “non-nodule ≥3 mm.” During the subsequent “unblinded read” phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist’s own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus. After the initial blinded read phase, 71 lesions received “nodule ≥3 mm” marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as “nodule ≥3 mm” by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist. The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules ≥3 mm. Nevertheless, substantial variabilty remains across radiologists in the task of lung nodule identification.

thoracic imaging

interobserver variability

computed tomography (CT)

computer-aided diagnosis (CAD)

Lung nodule

Details

Title: Subtitle: The Lung Image Database Consortium (LIDC): An Evaluation of Radiologist Variability in the Identification of Lung Nodules on CT Scans
Creators: Samuel G Armato - The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637
Michael F McNitt-Gray - University of California, Los Angeles
Anthony P Reeves - Cornell University
Charles R Meyer - University of Michigan
Geoffrey McLennan - University of Iowa
Denise R Aberle - University of California, Los Angeles
Ella A Kazerooni - University of Michigan
Heber MacMahon - The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637
Edwin J.R van Beek - University of Iowa
David Yankelevitz - Cornell University
Eric A Hoffman - University of Iowa
Claudia I Henschke - Cornell University
Rachael Y Roberts - The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637
Matthew S Brown - University of California, Los Angeles
Roger M Engelmann - The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637
Richard C Pais - University of California, Los Angeles
Christopher W Piker - University of Iowa
David Qing - University of California, Los Angeles
Masha Kocherginsky - The University of Chicago, Department of Radiology, MC 2026, The University of Chicago, 5841 S. Maryland Avenue, Chicago, IL 60637
Barbara Y Croft - National Cancer Institute
Laurence P Clarke - National Cancer Institute
Resource Type: Journal article
Publication Details: Academic radiology, Vol.14(11), pp.1409-1421
DOI: 10.1016/j.acra.2007.07.008
PMID: 17964464
PMCID: PMC2290739
NLM abbreviation: Acad Radiol
ISSN: 1076-6332
eISSN: 1878-4046
Publisher: Elsevier Inc
Language: English
Date published: 2007
Academic Unit: Roy J. Carver Department of Biomedical Engineering; Radiology; Internal Medicine
Record Identifier: 9984051533702771

Metrics

38 Record Views

87 Times Cited - Web of Science