Multimodal text and image search

With vectors, we're able to search not just text but images. Unlock features like:

  • search for images with text
  • image to image search

This is because in vector form, we're able to compare the meaning of these two data formats.

Start by creating vector fields

Vectors are an AI data format which you can create based on your image URL fields. You will want to use our image-text model.

Learn about how to vectorize your data.

Search your vector fields

When you vectorize a field, such as image_url, documents in your dataset will have a new field appended with the AI model used to create the vector ([email protected]) and _vector_. So in our example: [email protected]_vector_.

These new image-text vector fields can be searched with text or with an image URL. Our API will automatically vectorize your search query with the same model so we can compare.

Here's an example of using our SDK to search a dataset of images with text:

// import client and query builder
import { DiscoveryClient, QueryBuilder } from "@relevanceai/sdk";

// initialize client
const client = new DiscoveryClient({
  project: process.env.RELEVANCE_PROJECT,
  api_key: process.env.RELEVANCE_API_KEY,
});

const dataset = client.dataset("my-dataset");

// search "image_url" vector field with a text query
const query = QueryBuilder().query("dogs in a park").vector("[email protected]_vector_");
const results = await dataset.search(query);