Truthmarker: a tablet-based approach for rapid image annotation

The development of automated techniques for the analysis of image data is an important and active area of research. To make progress, this research requires annotations of image data to build and validate models used for analysis. Given this requirement, the development of software tools that streamline the collection of annotations would be of great benefit to image analysis researchers. Such tools should meet the following requirements: rapid generation of annotations for large data sets, annotation and data management that is straightforward for users, flexibility for application to many diverse image datasets, configurability to allow the collection of annotations to be tuned for a specific research goal, and generation of annotation data in a standardized format so that it can be easily parsed and analyzed. Truthmarker was designed as a tablet computer based image annotation tool to meet these requirements. Researchers can configure Truthmarker to fit the needs of a particular study by specifying an annotation model that fine tunes the user interface and resulting data to fit the annotation task. The quality of annotations generated using Truthmarker was evaluated by recruiting medical experts to annotate ophthalmic images for severity of diabetic retinopathy, a leading cause of blindness. These annotations were compared to annotations of the same images assigned using standard desktop computer based tools. The results, as measured by κ statistics and accuracy, indicate that Truthmarker annotations were of equivalent quality compared to those that were created using desktop-based tools.