Vision-Based Surgical Tool Tracking

Tracking of surgical tools have attracted attention in theliterature due to its essential role in a number of applications,ranging from surgical skill assessment to task automationin robotic surgery. The long-term goal of the surgical tool tacking is to enable visual servo control of the surgical tools to perform precise visually guided manipulation tasks. For instance, given the 3D pose of a needle in the camera frame, let the robot arms pick up the needle and place the needle in a desired pose.

The method proposed in this project for robotic surgical tool tracking is a vision-based Bayesian state-estimation approach. The proposed method uses forward kinematics of the robotic surgical manipulator for state evolution, based on an approximate calibration of the robotic manipulator and the endoscopic surgical camera. Image streams acquired from the stereo endoscopic cameras are used as the sensing modality for the measurement updates in the Bayesian state-estimation. Specifically, as part of the method, an on-line virtual rendering algorithm is employed to create virtual images of the surgical tool by generating the silhouette of the defined tool geometry under the endoscopic camera view. The observation likelihoods used for Bayesian measurement updates are then estimated from the similarity of the virtual images of the tool pose hypotheses generated by the virtual rendering algorithms and the real images that are captured by the stereo vision system. A particle filter is employed as the underlying Bayesian estimator, as the system is non-linear and non-Gaussian.

The tracking performance of the proposed method was evaluated on a simulation of the da Vinci robotic surgery system (the Gazebo simulation environment of the Robot Operating System), and an actual physical da Vinci robotic surgical system.