errors the movie industry lost one of
their greatest director Abbas Kiarostami
that I was raised by his amazing movies
during during the last 20 30 years and
we lost him quite young to medical
errors on the other side many of us have
seen fruit of excellence medicine
excellence medical practice I personally
tried you know experience it my dad had
in 86 a bypass than an hour tikka stent
in 98 the carotid artery repair in 2008
and pacemaker in 2012 so basically
science and medicine and good
performances extended his life and he is
blessing to us and to his big family for
over 25 years
surgical performance affects our life
and the lives of our you know loved one
but what is this performance how can we
improve this performance so basically
this performance is complicated I mean I
usually tell my students you know a
surgeon who cuts well and sutures well
is not really the definition of surgery
you don’t need 30 years of acquiring
knowledge and education and many years
of hands-on experience to cut and suture
the performance is about information
analysis of tests information making
decisions and then doing the action and
this is in surgery this is a very
dynamic process now about 100 years ago
or about 120 years ago doctors had no
imaging so what would how do they how
did they decide about their patient
so as we say but I do know the engineers
they would look at patients as a black
box like a device that you get so the
only way they could analyze that was to
looking at all the input/output so
that’s where they could look what is the
input output of this system looked
through this input output eventually
make observation of that that black box
that they wanted to know what’s going
wrong inside it taking looking at the
symptoms measuring signals coming from
that closed system eventually taking
some samples blood sample urine or
saliva or maybe going more invasively
doing a biopsy and trying to get samples
of this system in the you know in the
extreme case even opened the patient to
see what’s wrong which was happening in
end of 19th century in the last century
thanks to a lot of development in
imaging starting from home can
Hounsfield all the way to mansfield we
went from not having imaging to have
x-ray CT ultrasound M or Pet PET CT so
we started to have a lot of imaging and
a lot of information and also to have
computational power to visualize that
however the majority of this information
was mostly displayed on you know on
monitors on this place so the best
doctor of medicine experts radiologists
or even surgeons they were looking at
this display to analyze this data and
make their mental model of what’s going
wrong in inside the surgery room this
was even more complicated because they
had to interact with this data and if
the data is far away on a screen even
interacting with it becomes very complex
so about ten years ago we took our team
with our head mounted display
with our surgical partners and we went
to Berlin in an orthopedic conference
and we showed to hundreds of surgeons
that they in fact tacks to this
augmented reality or medical augmented
reality they could see the CT data the
data of the patient directly on top of
the Anatoly they could take a wireless
mouse interact with that data instead of
seeing an extra shot or a CT on a screen
they could actually directly see it on
the patient and analyze it directly on
the patient they could they could have
volume rendering thanks Nick
they were very very excited we we wanted
to bring this further to bring it into
procedures into surgeries so how do we
bring this augmented reality into the
operating room the first augmented
reality system we went which went on
many many operating room was thanks to a
startup company of two of my PhD
students and the surgical company which
allowed many processes such as sentinel
lymph node biopsies to be able to see
actually the nodules in augmented
reality and also in virtual reality but
why did we need this visual neonatal
window on the side because that
materiality was very nice was showing
them where these nodules are but to get
a 3d distance 3d perception doing
interaction with that data they still
needed this additional visual reality
display the reason is augmented reality
is not only about 3d image overlay 3d
image overlay is a very very little part
of the augmented reality if we want to
be augmented humans we want to see
beyond the wavelengths that our eyes can
see we want to hear
beyond the frequencies that are
system can absorb or analyze but still
do you know if I take the X er if I take
the city is still I have to bring it to
this limited bandwidth that humans can
see and this transfer is not easy if I
only argument the 3d even in the stereo
on top of the feet you see it floating
your brain says this is floating this is
not integrated so what we know at that
point tried to do is to use human
perception and the cues we are using
every day to actually fool the system
and improve our perception to use the
human cues of perception and make the
human believe that the human is really
seeing 3d inside that patient so now
this little tricks of computational
salesmen mathematics the same data now
you feel much more integrated as we are
looking we are using a few cues one of
them is parallax when I move I see more
I can see different positions I can’t
see more information that allows me to
believe that that Anatomy is behind the
surface but I still keep the ghosts of
the high curvature information on the
surface because if I remove that I lose
the shape of the of the Anatomy we also
use the fact that if I look directly
into the window I have to see through it
if I look on the side I see less so we
start to go you know instead of looking
at the cadaver and sovereign pose the
city that I see completely floating we
can go to a more complex visualization
this is the same CT data that now we are
looking through meta cues of perception
of human and now you start getting
confused what is real what is virtual
because now we are using if you see the
lines on the surface and continuing
while
feel that you’re saying inside the
anatomy how could we enforce this
coexistence of real and virtual so how
can I work with real tools in a virtual
environment and can now you know go
towards a direction where I cannot
perceive them or separate them from each
other so this is Felix one of our master
student a few years ago working with
Christopher Kristoff
Vishal Meyer he is looking with a tool
and now what you see here is the shadow
of the tool it’s following both on real
and virtual so as he’s moving this
shadow doesn’t exist this is a virtual
shadow but you see it both in real and
virtual so your brain starts to feel
that all this belongs to one environment
so your brain starts to put everything
into one shape now if I have
argumentation how do I explore the 3d
usually when you have a 3d data you move
it around and get all kinds of
perception about full 3d shape of the
object you have in front of you now if I
am in an argumentative situation when I
have the patient in front in front of me
and I see the anatomy of the patient I
can’t ask the doctor if I move it I lose
the alignment and of course I can’t ask
the surgeon to go under the table to see
the other side so we got inspired
but what dentists do we took a virtual
handheld tangible mirror and we brought
it inside this augmented reality view
which means now I can have my
augmentation but I can create any
impossible view and therefore I can
interact with that data so now I have an
augmentation but I can look at any of
you I am interested inside that patient
now one of the challenging point real
challenging point when you want to bring
the solution such complex solutions
operating room is how to integrate it
into the workflow and how to make sure
that you are showing the relevant
information if I show everything suffer
imposed it may not be useful you have to
show the relevant information at the
right time so many years ago we
developed this system where the x-ray
system and the camera system were
attached thanks to mirror system x-ray
and optics always had the same view so
the advantage was that when we were
going to the surgery and actually we
went to 40 trauma surgery in the center
of the Munich thanks to incredible
surgeons too so you could you could
position the patient for the first time
you could take an x-ray stop the x-ray I
want to use minimum x-ray
so you take one x-ray you stop it and
now you can start working under the
optic but you can see through the
patient and about 40 patient used for 40
patients this system was used quite
successfully now this is nice
I see the x-ray I see my hand I can go
through it and I can reduce the number
of x-ray from 40 to about 2 or 3 and
improve the surgery but now the question
is that this material is seen is very
crowded so we started to bring in
machine learning
how does machine learning play a role
here we have to learn we have to
understand what the doctor needs to see
at each moment from each modality if I
have x-ray optics in this case or if I
have PET CT M or etc during the surgery
the surgeon needs to see the relevant
information only so what we started was
a very simple machine learning system we
sell the system that I want to come
combined x-ray on optics but I want you
the system to automatically
recognize which element is in the scene
in optics and x-ray view and to
automatically decide how the blending
should happen to give me the optimized
view so instead of what we were showing
on the left-hand side now the system
automatically shows what is on the
right-hand side so it says if I have a
knife I want to see it I don’t want the
x-ray on my visualization on my hand I
am interested in the x-ray of internal
anatomy but I don’t want it to reduce my
performance by making it overloaded so
now at each pixel of the optics and
x-ray the system is deciding
automatically what that pixel is
information is and how it should be
merged into virtual and real the last
part is how to create this for the next
generation how to prepare the next
generation you know we have this
acceptance issue but we also now we want
this to come to our everyday life so we
started to work on augmented reality
also for education and for patient and
public information for our last few
years and actually I brought a system
here that after my talk you can try it
so basically the system allows you to to
get in front of the system and then the
system sees to rule you so basically the
system is going to recognize me and then
it starts to augment my body so with the
left hand I can show different part of
anatomy and focus on different anatomy
of my body with the right hand I can
increase the amount of information go
from simple balls to vasculature to
arteries and vein to internal anatomy
and finally to muscles and then I can
see the system with and then I can
analyze it we brought this to many
places to many schools
can turn it off if you want it was very
intuitive so we brought it to Essen and
about 2000 people played with the system
they were children they were adult and
all of them well right away started to
play around with their own body look at
different part of their body play and
try to understand what’s going on and
the signal was so simple the system was
so easy to handle that not only children
and adult but also politicians could
vote with it so you can imagine that the
system was extremely simple now the main
idea the main message I have here is if
we go for you know in the last century
imaging come in and radiation therapy
departments nuclear medicine Department
radiology departments they had to bring
physicists into their departments to
help them take advantage of radiation
therapy take advantage of you know pet
em or CT etc and our generation or
community or society now needs to have
computational scientists at the heart of
hospital it’s not acceptable that we can
on my mobile phone we can get all the
information that we need in a matter at
our fingertip and our surgeon the
surgeon who is doing the surgery on us
cannot get the information needs to
improve its performance right in front
of him thank you [Applause]