Can artificial intelligence really give us a glimpse of lost masterpieces?

In 1945 fire claimed three of Gustav Klimt’s most controversial paintings. The “Faculty Paintings” were commissioned in 1894 for the University of Vienna and were unlike any of the Austrian symbolist’s earlier works – as they became known. As soon as he presented them, the critics were in revolt over their dramatic departure from the aesthetics of the time. Professors at the university rejected them immediately, and Klimt withdrew from the project. Shortly afterwards, the works found their way to other collections. During World War II, they were placed in a castle north of Vienna for storage, but the castle burned down, and the paintings presumably followed. The only thing left today are some black and white photographs and writings from the time. Still, I just stare at them.

Well, not the paintings themselves. Franz Smola, a Klimt expert, and Emil Wallner, a machine learning researcher, spent six months combining their expertise to revive Klimt’s lost work. It has been a laborious process, one that started with the black and white photos and then incorporated artificial intelligence and dozens of knowledge of the painter’s art in an attempt to recreate what the lost paintings might have looked like. The results are what Smola and Wallner show me – and even they are amazed at the captivating technicolor images that AI produced.

Let’s make one thing clear: No one is saying that this AI is bringing Klimt’s original works back. “It’s not a process of recreating the actual colors, it’s re-coloring the photographs,” Smola is quick to remark. “Media photography is already an abstraction from the real works.” What machine learning does is give a glimpse of something that was thought to have been lost for decades.

Smola and Wallner find this nice, but not everyone supports AI that fills these gaps. The idea that machine learning recreates lost or destroyed works is, like the Faculty’s paintings themselves, controversial. “My biggest concern is about the ethical dimension of using machine learning in the context of conservation,” says art conservator Ben Fino-Radin, “just because of the vast amount of ethical and moral issues that have plagued the field of machine learning.”

It is quite certain that the use of technology to revive works of human art is fraught with difficult questions. Even if there was a perfect AI that could figure out what colors or brushstrokes Klimt could have used, no algorithm can generate authorized intentions. Debates about this have raged for centuries. Back in 1936, before Klimt’s paintings were destroyed, essayist Walter Benjamin argued against mechanical replication, even in photographs, saying that “even the most perfect reproduction of a work of art is lacking in one element: its presence in time and space, its unique existence. in the place where it happens to be. ” This was written by Benjamin in The work of art in the age of mechanical reproduction, is what he called the “aura” of a work. For many art lovers, the notion of a computer reproducing the intangible element is absurd, if not downright impossible.

And yet, there is still much to learn from what AI can do. The faculty’s paintings were crucial to Klimt’s development as an artist, a crucial bridge between his more traditional earlier paintings and later, more radical works. But how they looked in full color has remained shrouded in mystery. This is the puzzle that Smola and Wellner tried to solve. Their project, organized through Google Arts and Culture, was not about perfect reproductions; it was about giving a glimpse of what is missing.

To do this, Wallner developed and trained a three-part algorithm. First, the algorithm was fed with a few hundred thousand images of art from the Google Arts and Culture database. This helped it understand objects, art and composition. Next, it was trained specifically in Klimt’s paintings. “This creates a bias towards his colors and his motifs over time,” Wallner explains. And finally, the AI ​​got color traces for specific parts of the paintings. But without color references to the paintings, where did these traces come from? Even Klimt expert Smola was surprised at how many details the writings of the time revealed. Because the paintings had been considered so cumbersome and strange, critics tended to describe them at length, right down to the artist’s color choices, he says. “You could call it the irony of history,” says Simon Rein, project manager. “The fact that the paintings created a scandal and were rejected enables us to restore them better because there was so much documentation. And those kind of data points, if introduced into the algorithm, create a more accurate version of what those paintings probably looked like at the time.

The key to this accuracy lies in pairing the algorithm with Smola’s expertise. His research revealed that Klimt’s work during this period tends to have strong patterns and consistency. Studying existing paintings from before and after the Faculty’s paintings gave clues as to the colors and motifs that went back in his work at that time. Even the surprises Smola and Wallner encountered are confirmed with historical evidence. When Klimt first showed his paintings, critics noticed his use of a red, which at the time was rare in the artist’s palette. But The woman’s three ages, painted shortly after the Faculty’s paintings, boldly uses a red that Smola believes is the same color that attracted attention when it was first seen in the Faculty’s paintings. Writings from the time also raise a hue about the shocking green sky in another faculty painting. Pairing these writings with Smola’s knowledge of Klimt’s particular palette of vegetables when they were introduced into the algorithm is what produced one of the first surprising images out of AI.


Leave a Comment