[Applause]
this is the Ghent Altarpiece by Anthony
egg which is an incredibly impressive
picture it was incredibly influential
when it was first painted at the
beginning of the 15th century if you
ever get to againt
in Belgium you should go see it it’s a
politic and when you close it you see on
the middle on the right-hand side
this scene of the Annunciation we’re
behind the Virgin Mary there’s a book a
book that you see here while there’s not
a book there’s a painting of a book
we’re talking about an area of the
painting it’s about 3 inches high and
art historians were wondering whether
Van Eyck who is known to be a great
perfectionist had pushed his
perfectionism so far as to actually
really paint a text I mean a real text
rather than the little vertical and
horizontal marks that might give that
illusion we have the slightest okay so
if you show me a book like that in a
museum I can’t read it because these
medieval give a lettering the literary
format are very hard to read for a non
specialist but specialists couldn’t read
it on this painting either and that’s
because you see all these cracks the
calculator on this painting that’s over
600 years old makes it very hard to
distinguish where you have real letters
which are worn a bit and where the
cracks and so most art historians said
well it’s not a real text come on but I
one of the fields in which I work is
using building mathematical tools for
image analysis and so when I heard of
this I said well maybe we can try to
find algorithms that find these cracks
automatically and then once we’ve marked
them can in paint them and if I had
known how hard a task that was I might
have kept my math but
it turned out it was very challenging
and but we did solve it and became part
of a PhD thesis of a student in
electrical engineering in Brussels and
together with researchers at Kent
we published a paper in which the
solution was published and we went from
the crack Allu on on the right here to
the text on the on the left to the text
on the right and well when I first saw
the results I said well fat lot of good
that has done to us I mean I couldn’t I
couldn’t read anything but specialists
said oh my god look at this and they
could very clearly recognize 12 groups
of words which not only established
beyond a doubt that Vanek really had
painted the text but they found that in
fact they could identify what text it
was a page of a manuscript that had been
copied a few decades earlier in Rouge
and that was a text by Thomas Aquinas on
the Annunciation in an unseen painting
scene so they were absolutely delighted
and there was this was was this made a
very nice article in art history I
wanted to start my story today with that
to tell you that even prior to my main
topic today we had learned how to
recognize and find and in paint cracks
in paintings so my story is about an art
exhibit that is going on right now in
the North Carolina Museum of Art in
Raleigh if you haven’t seen it yet I
recommend it highly
it’s last day’s tomorrow so you don’t
have a lot of time so this was really
something in which we sailed uncharted
waters and let me show you how so it all
started because an altarpiece that dates
back to the 14th century from a small
Church in the maquis region in Italy had
been broken up at in the early 20th
century and sold to collectors three of
those panels are in the North Carolina
Museum of Art collection one is in
Portland three are
in the Metropolitan Museum of Art one is
in Chicago and the very last one well
the very last one is missing art
conservators after they realized and our
curators that these panels belong to the
same altarpiece have looked everywhere
but they haven’t found this panel
they had this dream of bringing all the
panels together here in North Carolina
the other museum said yes yes but you
won’t still have it complete and I
fortunately I happen to know a very
gifted artist but also an art
conservator and art reconstruction
expert shall of the Gospels who they
then commissioned to paint a
reconstruction of this missing painting
she does this mostly for museums for
educational purposes here’s an example
of something she made for a museum in
hotted um so it’s a little bit a portion
of a painting they have that she
reconstructed and for this educational
piece she stopped a quarter in short
with every layer of preparation and so
you see she’s already four months into
working on this panel before she even
starts painting all the different layers
of sizing of gesso and drawing and
gilding and so on so they commissioned a
panel for her for this missing painting
and what you could ask yourself how do I
do that on a painting I mean I don’t
have the panel how do i reconstruct well
the seven panels the seven small panels
that existed reconstructed the followed
episodes mark important episodes of the
life of John the Evangelist as it was
described literally in this medieval
Beth bestseller the golden legend in
which lives of saints are recounted and
we know it’s a best-seller because of
many medieval books we have only may be
part of a copy or something and this of
this we have hundreds so everybody
wanted one so they knew what scene had
to be depicted and then they made a
composition of a scene that borrowed
lots of stylistic elements of the other
panels so as not to invent something
that would not have been appropriate now
if they ever find the ninth panel
it’s not going to look identical to this
but it’s highly likely in this style and
then here you see the buildup of the
panel just like you saw the layers the
you see it being gilded and polished and
then paint is going to come in if my
clicker keeps working and the verdad
show on the faces the different paint
layers and as they get modeled more and
the final result is absolutely fantastic
and you can see that final new panel
with a documentary of how the panel was
made at a museum and you see differences
I mean on old panels where you have this
punch mark that have been knocked into
the gilding they have over all did the
six hundred years since the painting was
made being filled with death they’ve
gradually filled with dirt and they now
show like a dark relief but originally
unless polished on this polished gold
they gave you the reflection since when
and with the candles in the church they
would have given you a sparkling halo
around the head of the saint so you can
all see that on the new panel and so
people they’re the curators and
conservatives were so happy with this
whole experience and how I’d
occasionally it would be and there’s
this glorious new painting but then they
realized that they couldn’t put it next
to the other panels because everybody
would have eyes only for this new one
this new bright shiny one and it would
be the only one that was not authentic
so I then said well I always say that
math can help because what we could do
is we could study the panels and the old
panels and the new panels and we could
on the one hand find Correspondence of
color between the two different types of
panels and change color we could use
that to virtually age the new painting
and in fact when you go to do Z we see
this virtually aged one a printout of
that exposed neck to the other panels
but we could also if we learned all
about that we could rejuvenate the old
panels and that’s what we did
so the crackle ooh well you’ve seen we
can find calculate and we can paint it
in so we did that here you see how we do
indeed find all the red markings is the
automatically found cracks and we once
we did that we can in paint you see an
example here we did that very gradually
with an iterative algorithm that you see
here gradually progressing it was very
important to us to not produce just a
chromel I mean we wanted to use
scholarly grounded methods in order to
make this transition so we never took it
to places where you didn’t have evidence
for how it should have been and if we
didn’t do a perfect job well so be it I
mean but we didn’t we didn’t impose our
vision so here you have the background
of one of those paintings and then the
cracks removed then we could color remap
so here you have the eight authentic
panel the old family and the background
of the ninth new panel and because the
compositions were borrowed from each
other we knew exactly how the pigments
that were new now looked after all their
aging and we could use that in order to
color remapping for these paintings so
we here you see the color remapping for
the old fourteenth century painting on
the left and remapped two new colors on
the right here you see the next one the
new one on the right
coloring mapped to the old version on
the left and then of course to make a
virtual aged version we had to put in
cracks and then we had to add the gold
and that is now what is in in the museum
and I hope some of you get a chance to
see it tomorrow it’s not expensive by
uber to get an Oscar to to SEMA so it’s
absolutely gorgeous to see all those
paintings together you so this whole
effort of the rejuvenation and the aging
was done with the bus connections group
with Under
it’s here Duke and we had a ball and of
course we had a lot of help also from
Charlotte Augustus who gave us advice on
the colors but you see the
reconstruction of that ninth panel was
uncharted waters for the art
conservators and acuities they had never
done something like that and they really
enjoyed experience and then the whole
remapping and and crack removal and so
on was an exhilarating experience also
for us it was no longer a research
project because we had learned to do
that research earlier on the outer piece
of hint but it turned out they were
uncharted waters for the mathematicians
as well because we came up to a new
problem we got to work with platypus
platypus is a software package that we
put together and that saw it’s that’s
that that came that came to life because
of the GC panel because in preparing for
the work on the GC panel I asked to see
the x-rays and when they showed me the
x-rays here you see a detail and the
x-ray I said what’s this nonsense it
looks like it’s behind prison bars I
mean isn’t that annoying to you and the
conservator said no cone says yes it is
annoying and it’s the cradle to cradle
well cradling is something that you see
on many paintings on wood panel here you
see a detail of the other piece of Ghent
it’s also cradled and the cradling is a
hardwood structure of a lava structure
that’s attached to the back of these
panels he spends fortunately about an
inch thick and they would kind of work
and warp with temperature changes and
when collectors bought them from the
churches they preferred something that
was flattered they also in some cases
wanted to remove wood worm damage or
other damage but in many cases it was
just to flatten it and so they made them
much thinner and then gave them
structural in credit in integrity with
this cradle and so this cradle would be
attached to the
back of it with with big fixed numbers
and then sliding members the sliding
members slid probably in the first few
years and then after that the wood had
worked so much that it they were rock
fast but when you x-ray something like
that of course all you see is cradle I
mean and so all the subtle brushwork
that you use the x-rays for or the
subtle details of conservation are
swamped out by that big difference and
so when I heard about this I said well
maybe math can help so and but it was a
very challenging problem because not
only it wasn’t so hard to find the
location of those cradle members and the
first part of our algorithm does that
and it can be assisted if it’s a little
bit difficult by actually by the by the
art conservator but the more interesting
thing is that after you’ve removed that
wood structure you actually still have a
lot of wood grain from that cradle
sitting there as you can see here on the
right and they have been by then we have
become heroes for they are conservators
so they said but you could you could you
please take the wood grain away of the
cradle but we want to keep the wood
grain of the panel I said she really and
I said but maybe we can and it turned
out it was very challenging I mean so
here you have on the left a piece of
Aggie see and on the right the the we
have removed the intensity but the wood
grain is still there and on the Left
wood grain is gone so how did we do that
well we knew of course where the
different cradle members have been and
so we use machine learning techniques to
isolate first of all what wood grain was
and then to find where the different
wood grain what nature was of wood grain
while you only had panel of panel Plus
cradle and so on and then we could
remove it and that was a part of the the
graduate work of one of my graduate
students and we were very proud and so
it’s now used this is for a piece of an
egg here you see a piece of the paddle
in
we actually can compare with because
that was a case where they removed the
the the cradle structure by hand because
it was causing damage to the painting
and you can see that we are actually
much closer to the ground truth with
platypus the name of the software then
with what what what people could do
before so this was done with
collaboration of art conservators art
curators and students and postdocs and
Platypus has learned to leave its cradle
and if everything goes well with the
presentation then we now can see the
movie that is the rejuvenated panel of
the legacy so thank you very much [Applause]