Explore scan-derived anatomy and CT/MRI slices in tabletop AR. A Ctrl+Space finalist.
Medical scans are usually viewed as flat slices. I built Medimagen to let people explore the relationship between those slices and a three-dimensional model placed on a real surface. I was very excited to do this in AR because a 3D model on a 2D screen only goes so far. I'm eager to bring this into The experience includes skull and brain meshes, anatomical labels, registered CT/MRI slices, and a stacked-slice preview.
A substantial part of the work happened before the AR scene: preparing anatomy in 3D Slicer, cleaning geometry, aligning data, and exporting assets that could stay registered in the app. Medimagen was a finalist in the Expo × ReactVision Ctrl+Space hackathon.
Medimagen was first started during the Expo × ReactVision Ctrl+Space hackathon, where it was incubated alongside Neo Rock ’Em Sock ’Em Robots and Guess the Qubit within a shared AR prototyping workspace. While they originated together under a shared experimental codebase during the hackathon, each was built as its own distinct concept with plans to polish and publish each individually.
Prepare anatomy for a phone
Inspecting the inner skull surface during cleanup in 3D Slicer.
I used 3D Slicer to work with scan-derived skull and brain geometry, inspect surfaces, and prepare the models for export. The goal was to retain useful anatomical structure while producing assets that could be positioned and explored in a mobile AR scene.
Keep models and slices in alignment
CT and aligned MRI overlaid in a sagittal slice in 3D Slicer.
The skull, brain, and slice views need to describe the same space. I worked through CT/MRI registration and used a consistent coordinate transform when exporting the model assets. In the app, the model pair moves together rather than giving each child a different placement transform.
Explore in AR, and show the limits
Educational visualization prototype; the experimental volume renderer is still under investigation.
A frame from the Medimagen hackathon demo showing scan-slice controls.
Users can place, scale, rotate, and inspect the anatomy, then switch to scan slices and stacked-slice visualization. An experimental GPU volume-rendering path is also part of the investigation, but it remains unverified on iPhone. The case study distinguishes that work from the implemented anatomy and slice views.
The challenges
First attempt at a volume render in the app, which was not successful.
The biggest limitation of the current app is the time that has to be spent in 3D Slicer preparing the scans. I converted the scans to a 3D model and the cleaned them up. I also had to align the CT and MRI scans so that they could be viewed together. The volume renderer is still under investigation, but it is a promising path for future work. This 3D volume render was not successfully implemented and a lesser-than fallback version is in the app now. The use of AI could potentially do the cleaning up of the scans and the alignment of the CT and MRI scans, but that is not implemented yet. The app is a prototype and is not intended for medical use but I'd love to continue working on this project if I could get funding to do so.