IEmbeddingStorageService

Overview

The IEmbeddingStorageService provides a standardized interface to interact with vector databases. It is utilized by Wisej.AI components and tools that depend on embeddings and vector queries to perform semantic searches across documents.

See Vector Databases for the list of supported storage providers.

Wisej.AI utilizes the IEmbeddingStorageService through the DocumentSearchTools and the SmartHub.IngestDocumentAsync() method. In addition to these capabilities, the service also offers functionality to delete and list collections and documents, providing a comprehensive interface for managing vector data.

Implementing the IEmbeddingStorageService is particularly complex due to the diversity, limitations, and intricacies involved in managing vectors, metadata, and text chunks across heterogeneous services.

For example, vector databases typically treat text chunks as individual "documents," even though these are merely segments of a larger document. They generally do not offer built-in mechanisms to link these chunks back into a cohesive document. However, they do provide unstructured metadata fields where custom values can be stored, allowing for some level of organization and context.

In Wisej.AI's implementation, the metadata field is utilized to store both document-specific metadata—such as the document name, size, file path, and other relevant details—and internal system metadata. This includes information about the embedding model used to generate the vectors and an indicator of whether the chunk is the first (master) chunk. The master chunk contains all the document metadata to prevent redundancy across chunks.

Metadata in the Pinecone database

It's important to note that embedding models are not interchangeable. If you switch the embedding model, all vectors stored in the database that were generated using a different model become incompatible and cannot be used. This is because each model produces vectors with unique characteristics and dimensions, making them specific to the model that generated them.

Default Implementation

Wisej.AI provides several implementations for the IEmbeddingStorageService :

1. AzureAISearchEmbeddingStorageService: This service leverages Azure AI Search capabilities to store and retrieve embeddings, offering seamless integration with Microsoft Azure's robust cloud infrastructure.

2. ChromaEmbeddingStorageService: This service is designed to work with Chroma's storage solutions, allowing for efficient embedding management and retrieval with Chroma's specialized features.

3. FileSystemEmbeddingStorageService: As the name suggests, this service utilizes the local or network file system for storing embeddings. It is ideal for scenarios where simplicity and local storage are preferred.

4. MemoryEmbeddingStorageService: This implementation stores embeddings in memory, making it suitable for temporary storage needs during application runtime. It offers fast retrieval but lacks persistence.

5. PineconeEmbeddingStorageService: Leveraging the Pinecone vector database, this service provides scalable and efficient storage solutions for embeddings with high-performance retrieval capabilities.

6. QdrantEmbeddingStorageService: This service is based on Qdrant's vector similarity search engine, which is designed for fast and accurate retrieval of embeddings stored in complex datasets.

Each of these services offers unique benefits and can be chosen based on the specific requirements of the application in development.

For more information, including the source code for some of the built-in implementations, please refer to the Vector Databases page.