If you could chat with a famous artist from the past, who would it be and what would you ask her or him? Would you ask her about her inspiration for a specific sculpture or painting? Or would you ask him a question about his personal life, his thoughts or the philosophy behind his art?
Such a scenario that brings an artist and his or her work to life even years or centuries after his or her death sounds appealing to anyone who is committed to the mediation of art. One way to get there could be through artificial intelligence (AI), for example via chatbots that simulate a conversation between the artist and a visitor. These kinds of attempts and projects already exist. But how developed is this technology? Which approaches are being pursued? And, can art also contribute to the development of AI?
The boom of AI and chatbots in our daily lives
In recent years AI has made great progress. Chatbots and other applications that rely on AI are booming for various reasons:
- commercialization (big players like Google, Apple, Microsoft, IBM, Amazon recognized the potential)
- development of the technology & methodology
- greater computing power, better computer chips, cheap but good sensors
- big amounts of data (keyword: big data)
- improved / implementation of (new) AI processes (deep learning, neural networks)
Further, user acceptance is growing. Voice assistants like Alexa or Siri have become indispensable. Chris Messina, the inventor of the hashtag, product designer and technologist who has previously worked for Google, Uber and Molly, interprets this as:
And that is exactly what chatbots and voice assistants based on AI can do for the user. Instead of opening another app or typing something into a search engine, the chatbot can help with the request. It is convenient and improves the user experience.
Can AI help us to understand Art?
In museums as well the usage of AI is trending in a wide variety of areas, ranging from enriching collections information to visitor analysis or voice assistance. Especially interesting are those approaches that use AI to generate additional value to the mediation of art, or the ones that use gamification to create deeper understanding or emotional engagement with the art works in the observer.
Pioneering work combining art mediation and AI was done by the Museum of Science and Technology “Leonardo da Vinci” in Milan in 2002. The concept was to build a virtual character based on Leonardo da Vinci. Users were able to chat online on the museum’s website with “Leonardo” (note: The first iPhone was released in 2007, so the common way of chatting via mobile phone was writing SMS). The knowledge base for the chatbot was built up together with an agency specialized in natural language interfaces as well as university students. Stefania Boiano and Guiliano Gaia themselves, back then the persons leading the project, evaluate this project to have failed.
So, what happened? The project team focused majorly on the conversation and marketed it that way. Secondly, they built their virtual character on an actual person and if this wasn’t challenging enough, they chose a universal talent and outstanding inventor by picking Leonardo da Vinci. Therefore user expectations were very high but couldn’t be fulfilled since the chat with the virtual character felt nothing like a conversation with its iconic model. The chatbot could only interact in basic introductory chats which led to quick frustration on the user side and they stopped using it.
15 years later, Stefania and Guilano could put their lessons learned into practice in another chatbot project in Milan. By that time, already co-founders of their own cultural innovation studio, InvisibleStudio, they were responsible for the development and implementation of the “Di Casa in Casa” (“from House to House”) project for “The House Museums of Milan”, a network of four Milanese historic house museums. As the name suggests the aim of the project was to build an interactive and engaging digital project that guides its users through all four museums. The solution applied was a combination of gamification and chatbot guidance. Users could find hidden clues, fight against a mysterious magician inspired by a real historical figure of the Renaissance and get to a final discovery by following an actual path through the museums and looking at the artworks, all accompanied by their chatbot guide. That way users would focus on the game and tasks as well as the museum’s displays and less on the quality of the conversation they were having with the chatbot. As a result the users were satisfied with the chatbot and the game. In a pilot with teenage students, 66% found it educational – and 72% even highly entertaining.
A similar approach was also followed for example by the Carnegie Museum’s chatbot in Pittsburgh. Andy, so the name of the virtual character, guides its users through different sites of the institute by involving them into a game where they can collect digital stamps. Andy as well as the chatbot of The House Museums of Milan are based on Facebook Messenger. Many people already have the app on their phones and can start with the experience right away.
Another strategy in developing attractive-to-talk-to chatbots is for example to let the artwork explain itself in a conversation with the visitor. Projects like “dialog with the artwork” of the Museo de Arte Moderno de Buenos Aires is a great example. Every artwork got its own virtual character based on its theme and art movement. Questions that the chatbot answered were for example: How old are you? What is your meaning? Why did the painter use these specific colours?
This did not only attract three times more people, but users also seem to like the conversations since the chats had an average duration of one minute and 20 seconds.
A recent project from Germany is CHIM. CHIM stands for “chatbots in museums” and is a project by the German Research Center for Artificial Intelligence, Linon Medien and the Federal Ministry of Education and Research. It aims to create a learning museum’s guide based on AI that is also economical. The foundation for the chatbot is science based knowledge on visitor research and digital knowledge transfer on the one hand and human machine interaction, natural language processing and AI on the other hand. Users will be able to ask questions, in a chat or via voice assistant, and will receive answers in multimedia formats like text, audio, pictures or videos. Additionally the chatbot will create personalised museum tours that meet the needs and interests of the visitor.
To be able to realize this, CHIM will work as a hybrid. Very specific questions can be answered with facts that the chatbot can request from the existing museum database. It gets trickier with open and more general questions though. To answer those, the chatbot works with language models. These models are trained with great amounts of data. To support this process, the project team additionally started a campaign to collect relevant questions from people.
If you want to participate in the learning journey of CHIM, you can do so by asking questions on artworks here. The collected questions will be assigned to existing content or if there is no suitable material to answer single questions, experts will create pertinent content for them. It will also be very transparent to the user that he or she interacts with a computer and not with a person since transparency will also be a key element when it comes to understanding. If the chatbot is unsure about the question, it will make a guess and ask the user for clarification. For example the chatbot could write: “I am not totally sure, but did you mean ‘XY’?”
Can Art help us to improve AI?
In the quest for creating an AI that we truly love to interact with, TED speaker Raphael Arar, shows us an unconventional way where art fuels research by creating experiences that give in depth insights into human emotions, interaction and intuition. These are the key pieces that AI lacks to become more tangible and user friendly in the long run by reducing the cold and unpleasant feeling one gets when communicating with AI.
“Through art, we’re tackling some of the hardest questions, like what does it really mean to feel? Or how do we engage and know how to be present with each other? And how does intuition affect the way that we interact?”
To go a bit more into detail: When Raphael talks about emotions, he doesn’t “simply” mean joy or anger, emotions that AI is already capable of identifying. He talks about more complex feelings that can sometimes even be difficult to describe for ourselves. For instance, in one of his art installations he had a closer look into the feeling of nostalgia. To translate this specific and complex emotion into something an AI can process, Raphael and his interdisciplinary team had to split it up. The first step was to ask someone to share a memory. The computer then analized elements that it was already familiar with, like basic emotions, or the usage of certain wordings like “childhood” or “home” as well as past tense. With this information it created a nostalgia score. In the last step, the computer showed how nostalgic it judged the story by displaying it to the observer in the artwork. The artwork was a light-based-sculpture. The rosier the light shined, the more nostalgic the memory was according to the computer. Manifested in the art one could now decide for him- or herself how well the computer understood his or her emotion.
Are you curious and want to know more about this idea? Enjoy watching Raphael Arar’s talk.
On a journey
Visions like the one I drew at the beginning of the article stir up expectations that cannot be fulfilled (to its full extent) by now. Therefore it is important to keep those realistic so that promising projects can be accepted and used.
Plus, chatbots and voice assistants need a certain amount of resources for the set-up and development at the beginning, which implies costs and effort for the institutions. Helpful are thereby open APIs or cooperations. Further technological leaps, large scale networking and new approaches might also lead (again) to do the yet impossible.
The developments and research approaches of the last few years show that developers, user experience designers, museum professionals and other professions contributing are on a challenging but promising journey to create additional insights and experience for people who are interested in art by using AI. This is, however, not a one-way-street as Raphael Arar proves with his art-led contribution to developing a more user-friendly AI. So stay curious and watch out for conversational AI on your next museum visit.
Header image by Wladislaw Peljuchno on Unsplash
