in crime TV shows you may remember
seeing forensic investigators use
computers to reconstruct the face from
the skull and identify the victim in
many forensic cases or even massive
disasters when the body remains are
badly damaged and no other information
is available facial reconstruction
becomes a powerful tool to help identify
the body and it has been successfully
applied in many cases but it’s what we
saw on TV real can the investigators
already just use computers to get the
face from the skeletal remains
unfortunately this is not possible yet
in real cases the restoration of the
skeletal remains and the reconstruction
of the face still need to be processed
manually by forensic specialists this
procedure could take up to several days
to finish which is very time-consuming
labor-intensive also reconstructions
done by different persons could be
different because of their subjectivity
the issues of time labor and
subjectivity are inhibiting
investigators from solving crimes
bringing about justice and giving the
deceased the closure they deserve these
issues are ones we are trying to solve
and we are developing computer programs
to do it with support from the National
Science Foundation and in collaboration
with forensic anthropologists at
Louisiana State University our team is
building the first digital scale
restoration and facial reconstruction
pipeline and developing novel computer
graphics and machine learning programs
to make this whole procedure more
efficient and more objective in this
talk I will show you our most recent
progress in the development of this
pipeline which has three main components
fragmented skull reassembly damage
restoration and facial
construction the first thing we are
looking at at the bones themselves when
they come to us the skulls are usually
fragmented with part of the missing
often damaged by animal or environmental
activities so the first step is to
restore these pieces back to their
original geometry conventionally
fragmented skulls need to be reassembled
and held together with wax or other
glues manually this is again slow and
dependent on the modelers skill so in
our pipeline we first 3d scan these
fragments to get their digital models
then have computer programs reassemble
them for us but the challenge is how do
we teach computers to do the reassembly
automatically let’s first consider the
reassembly problems in two dimensional
space and this is equivalent to teaching
computers to solve jigsaw puzzles
although it’s been studied in computer
vision fields for more than two decades
we found that most existing algorithms
only work on toy examples and couldn’t
even handle moderately complex puzzles
so recently we developed a new algorithm
to do the reassembly we trained the
machine to recognize textures of images
so that it could identify and group
together potential neighboring pieces
which often have similar or related
textures we also develop a new
hierarchical model to search for the
optimal composition between fragments
that maximizes the mutual consistency of
local alignments in other words it picks
the correct stitching between fragments
by analyzing how well they make them fit
with all their neighbors from the big
pictures with this new design our
algorithms can greatly outperform
assisting approaches and could now solve
puzzles there are four or five times
more complex than those that could be
handled by existing algorithms after
explaining our idea in 2d let’s see how
this works with 3d items suppose we have
a ceramic norm model and we break it
into fragments
then we digitize these fragmented pieces
and our program can automatically
reassemble them back to the normal model
again on real skull this digital
reassembly could also be applied this
shows an example of fragmented skull
that got reassembled by our program
after fragment reassembly we shift from
this to the damaged restoration which
repairs the still missing or damaged
region on the skull so we need a program
to help us guess which should be filled
in this missing part we have developed
our restoration system by combining
three effective strategies we can repeal
a skull using its self symmetry as human
skulls are generally symmetric we can
also repair the missing region by
transplanting its counterpart from a
template skull we can also predict the
missing parts from existing regions
using a statistical shake model built
upon a skull database by combining all
these three effective strategies we have
developed a reliable restoration system
for damaged skulls after skull
restoration the last step is the facial
reconstruction this is where the skull
meets the flesh performing an accurate
facial reconstruction is actually
challenging even for forensic
specialists this is because although the
general geometry of the face can be
mostly determined by the skull certain
features such as lip shape eyebrows and
hair style cannot be inferred from the
bones so a realistic facial
reconstruction often relies on certain
artistic interpretation from the modeler
to avoid these ambiguities in our
pipeline instead of directly
reconstructing the face from the skull
we first reconstruct many faces from
portrait photos such photos will come
from a database of missing persons then
we perform a superimposition to match
the skull with each reconstructed face
and see how likely this scalp belongs to
that person
finally this match
can be used to revise the reconstructed
face to better match the skull to train
computers to do the facial
reconstruction from a photo
we built a deep neural network and used
about 50,000 pairs of 3d face scans and
portrait photos and trained the feature
extraction and face modeling so that the
program can reconstruct the face from
any new given photo to do the
superimposition the program will check
if this phase corresponds to this skull
how well the tissue depth will match
with the facial geometry for example
here the green points indicates regions
where tissue deaths have small deviation
from the standard while the red point
indicate regions where tissue deaths are
abnormal the matching probability can
then be calculated based on the ratio of
green and red points now with this
program when given a skull we could
calculate its matching probability with
each face in a database and return a
list of best matched faces and on a
specific reconstructed face the program
could identify then refine the not well
matched region according to the skull to
get to resynthesize a new face that
better matches the skull in terms of
speed our program after training can
finish the reconstruction and
superimposition within just a few
seconds and its accuracy and reliability
will continuously improve with the
collection of more and more data we are
excited what we have developed here
could potentially save forensic
specialists days and weeks from their
work time and this may lead to a faster
path to answers for people in the wake
of disasters or tragedies besides their
forensic applications these technologies
also open the door to more general
applications for example instead of just
reassembling jigsaw puzzles we could
train computers to reassemble shredded
enriched documents and become unshredded
for important files instead of just
repairing damaged skulls we could train
computers repair other things such as
valuable ancient
Kings or relics and instead of just
reconstructing faces from portraits we
could train computers to reconstruct any
3d scenes from our available photos to
get our physical world digitized all in
all this research is laying the
foundation to make the fully
computerized facial reconstruction a
reality it is also making many more
computer assisted data restorations
possible in a very near future
what are we going to restore or
reconstruct next I don’t know
but now anything is possible thank you [Applause]
seeing forensic investigators use
computers to reconstruct the face from
the skull and identify the victim in
many forensic cases or even massive
disasters when the body remains are
badly damaged and no other information
is available facial reconstruction
becomes a powerful tool to help identify
the body and it has been successfully
applied in many cases but it’s what we
saw on TV real can the investigators
already just use computers to get the
face from the skeletal remains
unfortunately this is not possible yet
in real cases the restoration of the
skeletal remains and the reconstruction
of the face still need to be processed
manually by forensic specialists this
procedure could take up to several days
to finish which is very time-consuming
labor-intensive also reconstructions
done by different persons could be
different because of their subjectivity
the issues of time labor and
subjectivity are inhibiting
investigators from solving crimes
bringing about justice and giving the
deceased the closure they deserve these
issues are ones we are trying to solve
and we are developing computer programs
to do it with support from the National
Science Foundation and in collaboration
with forensic anthropologists at
Louisiana State University our team is
building the first digital scale
restoration and facial reconstruction
pipeline and developing novel computer
graphics and machine learning programs
to make this whole procedure more
efficient and more objective in this
talk I will show you our most recent
progress in the development of this
pipeline which has three main components
fragmented skull reassembly damage
restoration and facial
construction the first thing we are
looking at at the bones themselves when
they come to us the skulls are usually
fragmented with part of the missing
often damaged by animal or environmental
activities so the first step is to
restore these pieces back to their
original geometry conventionally
fragmented skulls need to be reassembled
and held together with wax or other
glues manually this is again slow and
dependent on the modelers skill so in
our pipeline we first 3d scan these
fragments to get their digital models
then have computer programs reassemble
them for us but the challenge is how do
we teach computers to do the reassembly
automatically let’s first consider the
reassembly problems in two dimensional
space and this is equivalent to teaching
computers to solve jigsaw puzzles
although it’s been studied in computer
vision fields for more than two decades
we found that most existing algorithms
only work on toy examples and couldn’t
even handle moderately complex puzzles
so recently we developed a new algorithm
to do the reassembly we trained the
machine to recognize textures of images
so that it could identify and group
together potential neighboring pieces
which often have similar or related
textures we also develop a new
hierarchical model to search for the
optimal composition between fragments
that maximizes the mutual consistency of
local alignments in other words it picks
the correct stitching between fragments
by analyzing how well they make them fit
with all their neighbors from the big
pictures with this new design our
algorithms can greatly outperform
assisting approaches and could now solve
puzzles there are four or five times
more complex than those that could be
handled by existing algorithms after
explaining our idea in 2d let’s see how
this works with 3d items suppose we have
a ceramic norm model and we break it
into fragments
then we digitize these fragmented pieces
and our program can automatically
reassemble them back to the normal model
again on real skull this digital
reassembly could also be applied this
shows an example of fragmented skull
that got reassembled by our program
after fragment reassembly we shift from
this to the damaged restoration which
repairs the still missing or damaged
region on the skull so we need a program
to help us guess which should be filled
in this missing part we have developed
our restoration system by combining
three effective strategies we can repeal
a skull using its self symmetry as human
skulls are generally symmetric we can
also repair the missing region by
transplanting its counterpart from a
template skull we can also predict the
missing parts from existing regions
using a statistical shake model built
upon a skull database by combining all
these three effective strategies we have
developed a reliable restoration system
for damaged skulls after skull
restoration the last step is the facial
reconstruction this is where the skull
meets the flesh performing an accurate
facial reconstruction is actually
challenging even for forensic
specialists this is because although the
general geometry of the face can be
mostly determined by the skull certain
features such as lip shape eyebrows and
hair style cannot be inferred from the
bones so a realistic facial
reconstruction often relies on certain
artistic interpretation from the modeler
to avoid these ambiguities in our
pipeline instead of directly
reconstructing the face from the skull
we first reconstruct many faces from
portrait photos such photos will come
from a database of missing persons then
we perform a superimposition to match
the skull with each reconstructed face
and see how likely this scalp belongs to
that person
finally this match
can be used to revise the reconstructed
face to better match the skull to train
computers to do the facial
reconstruction from a photo
we built a deep neural network and used
about 50,000 pairs of 3d face scans and
portrait photos and trained the feature
extraction and face modeling so that the
program can reconstruct the face from
any new given photo to do the
superimposition the program will check
if this phase corresponds to this skull
how well the tissue depth will match
with the facial geometry for example
here the green points indicates regions
where tissue deaths have small deviation
from the standard while the red point
indicate regions where tissue deaths are
abnormal the matching probability can
then be calculated based on the ratio of
green and red points now with this
program when given a skull we could
calculate its matching probability with
each face in a database and return a
list of best matched faces and on a
specific reconstructed face the program
could identify then refine the not well
matched region according to the skull to
get to resynthesize a new face that
better matches the skull in terms of
speed our program after training can
finish the reconstruction and
superimposition within just a few
seconds and its accuracy and reliability
will continuously improve with the
collection of more and more data we are
excited what we have developed here
could potentially save forensic
specialists days and weeks from their
work time and this may lead to a faster
path to answers for people in the wake
of disasters or tragedies besides their
forensic applications these technologies
also open the door to more general
applications for example instead of just
reassembling jigsaw puzzles we could
train computers to reassemble shredded
enriched documents and become unshredded
for important files instead of just
repairing damaged skulls we could train
computers repair other things such as
valuable ancient
Kings or relics and instead of just
reconstructing faces from portraits we
could train computers to reconstruct any
3d scenes from our available photos to
get our physical world digitized all in
all this research is laying the
foundation to make the fully
computerized facial reconstruction a
reality it is also making many more
computer assisted data restorations
possible in a very near future
what are we going to restore or
reconstruct next I don’t know
but now anything is possible thank you [Applause]
Please follow and like us: