|
Interest
Points Tracking for Soft Tissues
|
We combined the region based
and feature based methods. A confidence based matching strategy was
proposed. Given two images with spatial or temporal differences and Telestration markers in one image, we first applied
region-based matching method to find correspondences in the other image. A
confidence score was used to measure the matching. If the confidence score
is higher than a threshold, then the matching was considered good enough
and the feature based matching is not applied in this case. If the
confidence score is lower than another threshold, this correspondence is
considered as an outlier. Otherwise we will apply feature-based method to
find correspondences. Again, the result is evaluated by a confidence score.
Outliers will be removed by applying global geometric constraints. In
3D Telestration, we need to apply both spatial
and temporal matching. The only difference between spatial and temporal
matching is the similarity measurement used in feature based matching.
|
|
ABSTRACT
Compared
with bones, more texture features can be detected from the soft tissues with
a high resolution stereo endoscope. Since the shape of the soft tissues
changes, the 3D reconstruction becomes much difficult and time consuming.
According to literatures, instead of obtaining the 3D shape of the soft
tissues/organs from two still stereo images, surgeons are more interested in
tracking important features/markers in the 3D view. In other words, tracking
soft tissues has more applications and obtains more attentions in recent
years. Among general soft tissue tracking problems, there exists an
important application called Telestration. With its
difficult learning curve, minimally invasive surgery has an especially strong
need for mentors. Researchers have developed ”2D Telestration”,
the technique for a remote surgeon mentor to draw on the operating surgeon’s
video display, to make surgical mentoring available over long distances to
increase the availability of mentors. 2D Telestration
has been in use for many years for onsite/remote education and surgical
training. However, with the development of the
high-definition stereo endoscopes and 3D display of the surgical control console,
2D Telestration is not sufficient or even hinders
the mentoring. Due to the existence of the depth difference, 2D Telestration can not provide the 3D vision of the real
surgical environment. On the other hand, if we project the 2D markers to the
3D view of the surgical robotics system, the markers will be floating above
the anatomy which can be distracting. Thus a 3D Telestration
technique becomes a natural solution and strong need. 3D Telestration
is to locate and track the 2D markers in both the space (between a pair of
the stereo images) and time domain, and then fuse the correspondences on the
surgical robotics system console to achieve the 3D view in real time. Since
the Telestration markers selected by surgeons might
not be the salient image features, an image based descriptor for Telestration markers is very important. To handle the
deformation and noise, we combine local and global, geometrical and texture
information to achieve the robust tracking of 3D Telestration
markers1.
1This
work is done when the author interned in Intuitive Surgical, Inc. from Sept.
to Dec. 2007. The author has collaborated with Wenyi
Zhao, David Hirvonen and Tao Zhao to complete the
research prototype development of 3D Telestration.
The author also wants to thank all members from R&D group for their
instructive discussion and technical support.
|
PATENTS
“Efficien 3-D Telestration for
Local and Remote Robotic Proctoring”
Wenyi Zhao, Chenyu Wu, David Hirvonen,
Christopher J. Hasser
[Pending]
“Robust Sparse Image
Matching for Robotic Surgery”
Wenyi Zhao, Chenyu Wu, David Hirvonen,
Tao Zhao, Brian D. hoffman, Simon Dimaio
[Pending]
|
PICTURES (click on thumbnails to enlarge images)
|
Hierarchical feature descriptor:
the blue cross indicates the Telestration marker
selected by the surgeon. The pink crosses are interest points detected by
DOG corner detector around the marker, within the searching window. After
each pink interest point is located, a sift feature is assigned to it. The
entire set of interest points with their geometric distribution consist of
our hierarchical feature descriptor.
|
|
Confidence based
matching scheme. Region based method, feature based method and global geometric
constraints are three important components of the matching scheme.
|
|
Region based matching.
Left column shows the Laplacian pyramid in
different levels. Middle column shows the corresponding
correlation images. The color regions are segmented by Mean Shift. Each
color region is assigned a value based on the correlation. The resulted
values are fitted to a Gaussian surface as the right image shows. The
virtual peak of the Gaussian surface is assigned as the confidence score to
this level. By the same means each level will obtain a confidence score.
The final confidence score for this matching is the maximal score.
|
|
Refine matching: Q is the Telestration
marker. P1, P2 and P3 are interest points detected
around Q. Q’ is the correspondence located by region-based matching. P’1, P’2 and P’3 are interest points detected around Q’. Moreover, based on feature based matching, P1 corresponds to P’1, P2 corresponds to P’2 and P3 corresponds to P’3. The position of Q’ will be refined by matched interest points.
|
|
Global geometric constraint: Q1, Q2
and Q3 are Telestration markers
in the reference image. Q’1, Q’2 and Q’3
are correspondences detected by the previous steps. Q’1 and
Q’2 have high confidence score but Q’3 is
an outlier. The actual position of Q’3 can be predicted
by the global geometric constraint.
|
|
