Development of real-time multimodal optical imaging systems with augmented reality

Advancements in the development of imaging software and hardware has led to growing research interest in integration of multiple optical modalities. Current multimodal optical imaging systems rely either on a hardware or a software approach for image registration. Registration through hardware implementation often results in heavy and bulky structures with relatively high costs in mechanical design and optical componentry. Feature-based registration methods on the other hand, rely on mutual features between source and target. Once applied across different regions of optical spectrum, these methods lack accuracy and consistency.

Aimed at the improving multimodal optical imaging, my doctoral thesis embodies hardware and software development, system characterization, and experimental investigation of novel optical imaging systems in the medical field. I designed a broadly applicable cyber-physical framework to perform real-time multimodal distance-aware image registration. Based on this framework, a prototype was designed and built for real-time distance-aware image registration of thermal and color imageries. In another work, a multimodal intraoperative imaging system was built for computer-aided surgery to register fluorescence and color imaging modalities, and was successfully applied to image-guided surgery in biological models. In exploring the applications of our framework, a novel method was developed to assist with high-throughput temperature screening, applicable to fever monitoring. To improve the user experiment with wearable imaging systems in clinical applications, an augmented reality system with dynamic 3-dimensional image magnification capability was developed, and partially offset displays were used to increase situational awareness during surgical procedures. The framework and systems developed in this work have shown great potential to facilitate various medical and industrial real-time imaging applications.