VIRTOSHA: Advancing Surgical Training with VR and Haptic Technologies
The aim of the VIRTOSHA project is to create a virtual reality training environment with haptic feedback for surgical operations on hard and soft tissues. Authoring software will enable the creation of various scenarios, where tissue and bone properties can be parameterized to simulate different patients. By analyzing real operations on patients and specimens, the steps and options of the procedure are captured, including the behavior of tissue, implants, and tools, as well as the haptic feedback. Using the authoring tool, different processes and avatars can be linked together to create new training sequences. Correct simulation of the tissues and the precise integration of hand movements are crucial for immersion. The Krüger group’s main focus is on improving hand tracking, based on VR tracking information, sensor readings, and potentially cameras as additional data sources. This effort aims to enhance the accuracy and responsiveness of hand motion capture within the virtual environment, thereby increasing the realism and effectiveness of the training. The haptic interaction with tools is based on a drilling and screwing simulation to be developed, which will be implemented with haptic arms and allow for realistic manipulations. Further features for an educational experience include the representation of tissue characteristics during manual and tool-guided interaction, as well as realistic rendering. All components will be connected via a data hub that manages all processes and structures and serves as the system’s interface. The entire training environment is to be developed at the UKB (University Hospital Bonn) to create a meaningful teaching tool in close collaboration with medical users.
The role of the Krüger group at UKB
Prof. Björn Krüger’s team plays a pivotal role in advancing hand tracking and haptic interaction technologies within the VIRTOSHA project. Their work focuses on the following key areas:
- Enhanced Hand Tracking Technologies
The group is developing and integrating sensor-equipped surgical tools with optical hand tracking systems to achieve highly accurate motion capture for virtual reality (VR) surgical simulations. This includes synchronizing data from multiple modalities to improve precision and ensure real-time compatibility with VR environments. - Haptic Interaction Development
The research group is contributing to the design and implementation of haptic interaction systems. This involves integrating a haptic arm into VR simulations to replicate realistic tactile feedback during surgical procedures. Their work includes calibrating tools for dynamic interactions, simulating tissue resistance, and refining the immersive experience for users. - User-Centered Studies
Through comprehensive user studies, the group evaluates the usability and realism of the VR systems. These studies involve surgeons, trainees, and other medical professionals to ensure the simulations accurately reflect real-world surgical procedures. Feedback from these studies is instrumental in refining the hand tracking and haptic interaction components to meet the needs of medical practitioners.
Collaborations
Here’s a list of collaboration partners for the VIRTOSHA project:
- MindPort
Expertise in VR development and interface design. - Haption
Specialists in haptic interaction systems and hardware development. - TH Köln (Cologne University of Applied Sciences)
Leading the development of simulations for soft and hard tissue behavior and contributing to the Digital Twin architecture. - UKB (University Hospital Bonn)
Contributing medical expertise, user studies, and evaluation of hand tracking and haptic systems. Includes collaboration between Prof. Krüger’s and Dr. Welle’s groups.
These partners bring together expertise in VR, haptics, simulation, and medical practice, creating a multidisciplinary approach to advancing surgical training technologies.
Funded by:
Publications
2025
Optimized Sensor Position Detection: Improving Visual Sensor Setups for Hand Tracking in VR Conference
2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), 2025.