DocumentSearchTools
Last updated
Last updated
DocumentSearchTools
module is capable of listing, querying, and summarizing documents stored within a vector database. It utilizes the to generate Retrieval-Augmented Generation (RAG) content for the AI, ensuring efficient and intelligent information processing.
The preconfigured prompt is this:
The DocumentSearchTools class provides multiple methods that the AI can use to search a virtually unlimited storage of unstructured data to accomplish a variety of objectives, enhancing its flexibility and adaptability in processing and analyzing data.
When creating an instance of the DocumentSearchTools
, you have the option to specify a collection name, which functions similarly to a folder. Additionally, you can provide a filter callback function. This callback will be used to filter or screen the documents processed by the tool, allowing you to control which documents are considered for further operations.
Additionally, you have the ability to override or set several properties that define the operational limits for the DocumentSearchTools
when retrieving content from a vector storage. These properties allow you to tailor the behavior of the tool, ensuring that it meets your specific requirements for the scope and scale of document retrieval.
To incorporate reranking functionality for re-ordering chunks retrieved by the vector search, you can override the RerankAsync
method. Within your implementation, you have the flexibility to utilize any reranking approach of your choice. This allows you to tailor the reranking process to best suit your specific requirements and improve the relevance of the search results.
CollectionName
Read-write and overridable. Default is null.
The CollectionName
parameter is utilized to organize documents within the vector storage database into distinct groups, analogous to virtual folders. This allows for structured management and retrieval of documents. When performing a vector query, it is restricted to operate within a specified collection. Should the CollectionName
be left null, the system defaults to using the default collection space. This ensures that documents are always associated with a particular collection space, even if one is not explicitly defined.
TopN
Read-write and overridable. Default is 10.
TopN
specifies the maximum number of qualifying chunks that are utilized to construct the RAG (Retrieval-Augmented Generation) context. A chunk is deemed qualified if its similarity score exceeds the threshold defined by MinSimilarity
. By setting the TopN
value, you determine how many of these high-similarity chunks will be included in the context for further processing, allowing you to balance between performance and accuracy according to your application's requirements.
MaxClusters
MinSimilarity
Read-write and overridable. Default is 0.25f. This setting defines the minimum similarity threshold used to filter and select qualified chunks from documents. By establishing this threshold, Wisej.AI can efficiently determine which segments of the document are closely aligned with the desired criteria, enhancing the accuracy and relevance of the selection process.
MaxContextTokens
Read-write and overridable. Default is 4096. This setting specifies the maximum number of tokens that can be returned to the AI within the Retrieval-Augmented Generation (RAG) context string. By limiting the token count, you ensure that the context provided to the AI remains concise and manageable.
MaxDocumentsSearch
Read-write and overridable. Default is 100. The MaxDocumentSearch setting restricts the number of documents returned when conducting a generic search query. For example, if you search for documents related to quantum computing research, and the system identifies 10,000 potential matches, this setting will limit the retrieval to a manageable subset of those documents. By controlling the document count, you can ensure more efficient handling and analysis of search results, avoiding overwhelming the system with excessive data.
RerankingEnabled
This tool relies on several services, many of which are pre-configured by default:
ITokenizerService
IEmbeddingStorageService
IEmbeddingGenerationService
IRerankingService
We recommend registering your own IDocumentConversionService
and utilizing a professional library such as Aspose for document-to-text conversion. The built-in converter currently utilizes PdfPig and OpenXML, which are the same tools employed by Semantic Kernel. However, these tools have limitations and may not be capable of accurately processing complex documents containing tables and images.
To enable the use of the simply add it to a SmartHub, SmartAdapter, SmartSession or SmartPrompt.
Read-write and overridable. Default is 5.
MaxClusters
determines the upper limit on the number of clusters that can be generated by the summarization function, which utilizes the algorithm.
Read-write and overridable. Default is false. Enables reranking of the vector search results through the .
Please ote that the default is set to , which is intended solely for testing purposes. Before deploying this tool in an application, it is recommended to choose an appropriate vector storage solution and register the correct service. The process is demonstrated below: