Date
November 2024
Type
RAG model, database
Made with
JavaScript, OpenAI API, RAG, vector embeddings
What it is
Artifact is a web platform that allows users to upload an image of any artwork, and then displays similar paintings from the (for now) Art Institute of Chicago’s collection. It provides an easy way to explore related artworks based on style, composition, or subject.
how it works
Artifact is a RAG model. The museum’s entire collection was scraped and analyzed using the museum’s own data and a combination of the Artsy and Wiki APIs. The data is then stored in a vector database and used for querying.
conservative use of AI
The model could be easily scaled up using more museum APIs. The strings used for the embeddings are long enough for an accurate analysis, but short enough to be cost-effective. Each artwork analysis costs an average of 0.0012 cents to be embedded, meaning 100k artworks could be analyzed for a little over one dollar.
Building the database
List all artworks in the Art Institute of Chicago’s collection, collect data about artworks using multiple APIs, create an embedding, and store in vector database (Pinecone).
Querying the database
In the main application, an image uploaded by the user is analyzed with OpenAI Vision to build an efficient query to fetch the 10 most similar artworks.
Performance Analysis
The data from the artwork on the left was created with OpenAI Vision, while the artwork on the right is on the vector database, and its data was created by analyzing several APIs.
*Subjects*: portrait, woman, seated figure, interior, chair, clothing, expression, human figure, muted colors, blue (color), green (color), red (color) *Classifications*: painting, oil on canvas, european painting *Terms*: oil painting, portrait, seated figure, interior scene, 19th century, muted colors, expression, human figure, blue, green, red *Medium*: Oil on canvas *Artist Genes*: Post-Impressionism, Figurative Art, Individual Portrait, Interior, Human Figure, Muted Palette *Genre*: portrait painting
*Subjects*: Century of Progress, world's fairs, Chicago World's Fairs, portraits, women; *Classifications*: painting, modern and contemporary art; *Terms*: painting, Century of Progress, modern and contemporary art, world's fairs, Chicago World's Fairs, portraits, women; *Medium*: Oil on canvas; *Artist Genes*: Modern, 1900 - 1917 Art, East European Art, Figure Studies, French Art, Gestural, Group Portrait, Human Figure, Individual Portrait, Nude, Oil Painting, Painting, Portrait, Post-Impressionism, Pre-World War II School of Paris, Rural Life, Scenes of Everyday Life ; *Movement*: Expressionism; *Genre*: portrait
Performance Analysis
When inputting an artwork that exists in the database, the match percentage is not perfect but it will always be one of the top matches.
