On June 13, National Taiwan University Hospital (NTUH) and long-time partner Microsoft Taiwan jointly announced a mixed reality (MR) application project. The NTU Hospital team will utilize Microsoft's HoloLens 2 head-mounted display and related systems to convert 10 years' worth of CT images into 3D stereoscopic hologram images that can be projected in mid-air. Application scenarios include doctor-patient communication, preoperative planning, gross anatomy teaching, and more.
NTUH vice superintendent I-Rue Lai pointed out that Taiwan's advanced ICT sector is an important driving force behind improvements in healthcare quality. This joint project with Microsoft will utilize the HoloLens 2 and systems like the Microsoft Teams video conferencing platform to improve the efficiency of internal teaching, doctor-patient communication, and remote consultations. A remote consulting session with NTUH's Yunlin branch was also demonstrated live.
Chien-chang Lee, VP of NTUH's Center of Intelligent Healthcare, cited fracture diagnosis as an example. He pointed out that with only 2D images, there's a higher probability of misdiagnosis. With 3D images, there will be a model that can be rotated, allowing doctors to observe the fracture from different angles. This can result in a more comprehensive assessment of the severity of the injury.
The team used the open-source software 3D Slicer to create stereoscopic images and mark the focus of infection. The Microsoft team will then use the Apoqlar software to convert the images into holographic images. This process can be completed in around 30 minutes. All the operator need is to wear the HoloLens 2 to manipulate the images via hand gestures. There's no need for them to hold any other devices.
Lee stated that holographic imaging wasn't able to be widely adopted in the medical field in the past because it not only requires professional engineers to process the images but also needs to be completed on HPC computers. It was a time-consuming and labor-intensive process that was also very limited in terms of productivity.
Once the processing time for the 3D images is reduced to 30 minutes, this technology can be used to assist in preoperative planning. In addition, cadavers for anatomy studies are typically quite scarce. With MR technology, this issue can be alleviated as well.
Lee admitted that they used HTC's headset devices in previous anatomy teaching scenarios. However, they experienced dizziness during the sessions and the images were more cartoonish. They also tried Meta's products before switching to HoloLens 2 upon seeing the maturity of Microsoft's technology.
When asked about their procurement plan, Lee stated that Microsoft has already donated two sets to NTUH. Since image processing still requires some time and operators need to undergo certain training, the technology is mainly used for education and doctor-patient communication right now. It cannot assist in emergency medical procedures yet.
Shi-Yi Yang, a senior manager at Microsoft, pointed out that this project combines end, network, and cloud technologies. HoloLens serves as the end-user device, the network transmission smoothness affects the user experience, and the cloud supports real-time and massive computing demands. For remote consultations, it is also crucial to ensure smooth switching between multiple windows.
Microsoft Taiwan COO Flora Chen mentioned that there is a lot of external interest in Microsoft's activities regarding AI. Therefore, the hope is that Microsoft can have more collaborations with NTUH in terms of generative AI. For example, they can improve patient experience and data insights through the Azure OpenAI services and ChatGPT.
