Architecture

Detailed description of the Wisej.AI architecture and all its components.

Overview

Wisej.AI is specifically designed with developers in mind to seamlessly integrate AI systems into business applications.

High Level Wisej.AI Architecture

The system fundamentally operates by utilizing the application's controls and code to construct sophisticated prompts. These prompts are then sent to the AI provider, and the response is parsed in accordance with the requirements of the controls and code.

Wisej.AI is composed of these key components:

• SmartEndpoint

• SmartPrompt

• SmartAdapter

• SmartTool

• SmartSession

• SmartHub

In developing Wisej.AI, several fundamental principles were adopted: maintaining simplicity, avoiding over-engineering, delivering real value to developers, and ensuring the system is self-contained with minimal dependencies.

Wisej.AI supports .NET Core, .NET Framework, C#, and VB.NET.

To maintain simplicity and minimize dependencies, Wisej.AI does not utilize libraries such as Newtonsoft, Semantic Kernel, or any REST clients.

SmartEndpoint

As the name implies, the SmartEndpoint components are responsible for managing all communications between an endpoint and the rest of the system. An endpoint refers to the URL of the AI engine where the Large Language Model (LLM) is running. This can be hosted on a local server on-premises, a private server in a data center, at any public AI providers, or even in your browser as a JavaScript worker.

SmartEndpoint components are designed to support any AI provider and handle various payload types, including binary data and images. They define numerous properties and methods specifically tailored to handle inference and embedding requests efficiently.

SmartPrompt

The SmartPrompt component manages both system and user prompts. It resolves prompt text from INI files, replaces placeholders with parameter values, and offers a straightforward method for interacting with the AI.

For instance, using Wisej.AI in its simplest form requires just an endpoint. Here's an example:

C#

var prompt = new SmartPrompt();
var endpoint = new OpenAIEndpoint() { ApiKey = "..." };
var response = await prompt.AskAsync(
    endpoint, 
    "Tell me a fun short story about a bee in less than 50 words.");

Console.WriteLine(response.Text);

VB.NET

Dim prompt As New SmartPrompt()
Dim endpoint As New OpenAIEndpoint() With {.ApiKey = "..."}
Dim response = Await prompt.AskAsync( _
    endpoint, _
    "Tell me a fun short story about a bee in less than 50 words.")

Console.WriteLine(response.Text)

An even simpler approach, requiring fewer lines of code, is available by using the SmartHub. However, regardless of the method chosen, an endpoint is always necessary.

Prompts can either be written directly in code or referenced using a key string enclosed in square brackets. SmartPrompt will locate the key and read the corresponding prompt from any .ini file stored in the /AI folder. See SmartPrompt INI for more information on INI files.

Prompts can also include placeholders, or parameters, which are replaced at runtime. These placeholders must be enclosed within {{ }}. However, when setting a parameter value, you should specify the parameter name without the curly brackets.

Parameters are always replaced before submitting a question but only within the initial System Prompt and the current user prompt. The earlier example using parameters might look like this:

C#

var prompt = new SmartPrompt();
var endpoint = new OpenAIEndpoint() { ApiKey = "..." };

prompt.Parameters.Add("word-count", 50);
var response = await prompt.AskAsync(
   endpoint, 
   "Tell me a fun short story about a bee in less than {{word-count}} words.");

Console.WriteLine(response.Text);

VB.NET

Dim prompt As New SmartPrompt()
Dim endpoint As New OpenAIEndpoint() With {.ApiKey = "..."}

prompt.Parameters.Add("word-count", 50)
Dim response = Await prompt.AskAsync( _
    endpoint, _
    "Tell me a fun short story about a bee in less than {{word-count}} words.")

Console.WriteLine(response.Text)

SmartAdapter

SmartAdapters act as mediators between the application and the AI, adapting data and processes to fit specific needs. Each adapter has a distinct role and set of features. For instance, the SmartDataEntryAdapter extracts structured data from unstructured sources and updates Wisej.NET controls with the relevant information. Similarly, the SmartObjectAdapter performs this function by setting properties of .NET objects instead of updating controls. Another example is the SmartChartAdapter, which creates charts from unstructured data.

A SmartAdapter operates by leveraging an instance of the SmartHub, which in turn requires an instance of the SmartEndpoint. When using a SmartAdapter, you're essentially working within the structure outlined in the diagram at the top of this page.

Once the adapter is properly configured, the following example demonstrates the powerful capabilities of the SmartObjectAdapter:

C#

public class Person
{
  [Description("First name")]
  public string FirstName { get; set; }

  [Description("Surname")]
  public string Surname { get; set; }

  [Description("Job title")]
  public string Title { get; set; }

  [Description("Extract list of personal interests from the text, summarize in 1-3 words maximum and change to title case")]
  public string[] Interests { get; set; }

  [Description("Person Address")]
  public string Address { get; set; }
}

var adapter = new SmartObjectAdapter()
{
  Hub = new SmartHub()
  {
    Endpoint = new OpenAIEndpoint()
    {
      ApiKey = "..."
    }
  }
};

// Extract the field values described above from resume.pdf.
var person = await adapter.FromStreamAsync<Person>(File.OpenRead("resume.pdf"));

// Or
var person = await adapter.FromImageAsync<Person>(businessCardImage);

VB.NET

Public Class Person
    <Description("First name")> _
    Public Property FirstName As String

    <Description("Surname")> _
    Public Property Surname As String

    <Description("Job title")> _
    Public Property Title As String

    <Description("Extract list of personal interests from the text, summarize in 1-3 words maximum and change to title case")> _
    Public Property Interests As String()

    <Description("Person Address")> _
    Public Property Address As String
End Class

Dim adapter = New SmartObjectAdapter() With { _
    .Hub = New SmartHub() With { _
        .Endpoint = New OpenAIEndpoint() With { _
            .ApiKey = "..." _
        } _
    } _
}

' Extract the field values described above from resume.pdf.
Dim person = Await adapter.FromStreamAsync(Of Person)(File.OpenRead("resume.pdf"))

' Or
Dim person = Await adapter.FromImageAsync(Of Person)(businessCardImage)

With just one line of code, we accomplish a task that would be nearly impossible to achieve using standard programming methods.

SmartTool

SmartTools are integral to the Wisej.AI architecture. By utilizing SmartTools, you can "compose" powerful AI systems that deliver significant value to your applications. A SmartTool, at its core, is a function—code that executes, accepts parameters, and returns a result.

In Wisej.AI, having an exceptionally powerful tools system is a core feature. SmartTools in Wisej.AI go beyond simple method callbacks or the plugin systems found in Semantic Kernel. They are designed to offer rich, extensible functionalities, providing developers with advanced capabilities to enhance and augment their applications seamlessly.

These functions (tools) operate within the context in which they were created, allowing them to:

• Keep state information

• Update controls on the screen

• Work within the user session

• Access any resource available to the application

• Asynchronously interact with external systems, i.e. databases, search engines, vector database

• Provide parameters that alter their own prompt

• Interact with the calling context and callers

• Interact with the user through the Wisej.NET UI

• Return synchronous and asynchronous values

Wisej.AI offers a unique capability to interact seamlessly with the user interface (UI) without limitations. Unlike Wisej.AI, Microsoft Semantic Kernel plugins are not integrated with any specific UI framework, which restricts their ability to perform such interactions. In any event, achieving this level of UI interaction would be extremely challenging with templating frameworks like Blazor, Angular, React, or other templating frameworks.

Since code often speaks louder than words, here is a very simple example to illustrate how SmartTools function:

C#

var prompt = new SmartPrompt();
var endpoint = new OpenAIEndpoint();
var response = await prompt.AskAsync(endpoint, "What time is it?");
Console.WriteLine(response.Text);

// output:
// I'm sorry, but I don't have the capability to 
// provide real-time information such as the current time.

prompt.UseTool(GetCurrentTime);
response = await prompt.AskAsync(endpoint, "What time is it?");
Console.WriteLine(response.Text);

// output:
// The current time is 4:51:38 PM.

...
private static string GetCurrentTime()
    => DateTime.Now.ToLongTimeString()

VB.NET

Dim prompt As New SmartPrompt()
Dim endpoint As New OpenAIEndpoint()
Dim response = Await prompt.AskAsync(endpoint, "What time is it?")
Console.WriteLine(response.Text)

' output:
' I'm sorry, but I don't have the capability to
' provide real-time information such as the current time.

prompt.UseTool(AddressOf GetCurrentTime)
response = Await prompt.AskAsync(endpoint, "What time is it?")
Console.WriteLine(response.Text)

' output:
' The current time is 4:51:38 PM.

...

Private Shared Function GetCurrentTime() As String
    Return DateTime.Now.ToLongTimeString()
End Function

Initially, the AI has responded that it cannot provide the current time. This is accurate, as AI models do not have the capability to gather real-time information outside of what is encoded in their neural network weights.

After incorporating prompt.UseTool(GetCurrentTime), the AI became capable of returning the current time. Essentially, we provided the AI with a tool, enabling it with a new capability. By programming such tools—be they small functions or more complex ones—we extend the AI's functionality.

In Wisej.AI, the interaction between a "new feature" (the tools) and the AI is handled effectively by the system. Essentially, every interaction is managed by an agent, ensuring seamless integration and functionality.

Wisej.AI employs its own tool definition prompting, rather than using the default defined by the AI model in use. This approach provides greater control over tool usage, reduces token consumption, and enables the use of tools even with models that were not originally trained with such capabilities.

Note that the AddTool() and AddTools() methods support call chaining:

this.smartObjectAdapter
    .UseTools(new WebSearchTools())
    .UseTools(new MathTools())
    .UseTool(TurnLightsOnOff);

Keep in mind that tools can also be implemented as asynchronous methods. This allows the code within these tools to interact with the user through the Wisej.NET user interface. Additionally, it enables the invocation of other services. Through the use of asynchronous methods, your application can handle long-running operations without blocking the user interface.

In addition to async tools, Wisej.AI also supports anonymous tools:

prompt.UseTool(
    [Description("Returns the current date/time")]
    () => DateTime.Now);

Being a lamba method, it runs in context allowing the code in the lamba to reference local variables declared in the containing code.

SmartSession

As explained on the General Concepts page, LLMs do not possess any memory, do not remember past interactions, and are inherently stateless. Each time you ask a question, it is akin to talking to Dory the fish from Finding Nemo.

Dory is a fictional blue tang fish and a major character of Pixar's animated film series Finding Nemo. She suffers from short-term memory loss.

The Wisej.AI agent system operates within the SmartSession. This is where Wisej.AI constructs the prompts, invokes the tools, and repeats the process until the LLM has completed the requested task. Additionally, the SmartSession in Wisej.AI is capable of managing message length when the context window is exceeded by either trimming or summarizing previous messages.

Every interaction with the LLM involves an instance of the SmartSession component. Even for a single prompt, Wisej.AI always creates and disposes of a session internally. This ensures that every request can utilize tools and operate within the boundaries set by Wisej.AI.

You can use the SmartSession, or a custom-derived class, to maintain the state of the interaction with the LLM and preserve the context effectively.

C#

using (var session = new SmartSession(new OpenAIEndpoint()))
{
  var response = await session.AskAsync(
    "Extract the emails in the image.",
    screenshot);
    
  Console.WriteLine(response.Text);
  
  response = await session.AskAsync("Extract all the dates.");
  
  Console.WriteLine(response.Text);
}

VB.NET

 Using session As New SmartSession(New OpenAIEndpoint())
   Dim response = Await session.AskAsync( _
     "Extract the emails in the image.", _
     screenshot)
     
   Console.WriteLine(response.Text)

   response = Await session.AskAsync("Extract all the dates.")
   
   Console.WriteLine(response.Text)
End Using

The example above demonstrates a session with two interactions with the LLM. In the first interaction, an image and a prompt are submitted. In the second interaction, you can refer to the image without resubmitting it, as it is part of the session and will be automatically included.

Interactions that include images are only compatible with models that have vision capabilities. Adapters that work with images have the UseOCR property allowing the usage the IOCRService and extract text from the image (without relying on the model's vision) before sending it to the AI.

SmartHub

The final component is the SmartHub. As the name suggests, it functions as a central hub, connecting an endpoint, multiple adapters, custom tools, and your Wisej.NET application.

With the SmartHub, you can manage messages, errors, and parameters related to all the adapters connected to both the SmartHub and the endpoint. A SmartHub always requires one endpoint, but it can be associated with multiple adapters.

In addition to centralizing events and connecting adapters to an endpoint, the SmartHub offers a variety of higher-level functions that leverage all the built-in services in Wisej.AI. Essentially, it provides developers with a high-level interface to interact with the AI.

C#

var hub = new SmartHub();
hub.Endpoint = new OpenAIEndpoint();

// Splits the pdf into chunks using IDocumentConversionService and ITextSplitterService
// Generates the embeddings for each chunk using IEmbeddingGenerationService
// Stores the chunks and the embeddings in a vector DB using IEmbeddingStorageService
await hub.IngestDocumentAsync("JP-CV.pdf", "jp-cv-00012.pdf");

// Generates the embedding using IEmbeddingGenerationService
var query = await hub.EmbedAsync("Experience");

// Finds the 1 chunk that is most relevant to "Wisej.NET" and
// has a similarity of at least 0.2.
var topChunks = await hub.SimilarityQueryAsync(
	"Wisej.NET",
	new[] { "Wisej.NET is...", "Bananas are not apples..." },
	1,
	0.2f);

VB.NET

Dim hub As New SmartHub()
hub.Endpoint = New OpenAIEndpoint()

' Splits the PDF into chunks using IDocumentConversionService and ITextSplitterService
' Generates the embeddings for each chunk using IEmbeddingGenerationService
' Stores the chunks and the embeddings in a vector DB using IEmbeddingStorageService
Await hub.IngestDocumentAsync("JP-CV.pdf", "jp-cv-00012.pdf")

' Generates the embedding using IEmbeddingGenerationService
Dim query = Await hub.EmbedAsync("Experience")

' Finds the 1 chunk that is most relevant to "Wisej.NET" and
' has a similarity of at least 0.2.
Dim topChunks = Await hub.SimilarityQueryAsync(
    "Wisej.NET",
    New String() { "Wisej.NET is...", "Bananas are not apples..." },
    1,
    0.2F)

When the SmartHub is placed onto a designer surface of a component, it automatically binds to the top-level component and registers all methods marked with the [SmartTool.Tool] attribute as tools.

In all code snippets when an endpoint is created without an api key, the assumption is that you have the ApiKeys.json file in /AI, see ApiKeys.json File.

Events

In Wisej.AI, the majority of objects are designed to trigger various events, enabling applications to interact with AI processes at multiple stages. While individual SmartAdapters may offer their own unique events, three key components—SmartPrompt, SmartSession, and SmartHub—uniformly raise a standardized set of events. This consistent event pattern facilitates seamless integration and enhances the interactivity between the application and Wisej.AI functionalities.

Name	Description
Start	Occurs when the session starts processing a question.
Done	Occurs when the session is done processing a question.
Error	Occurs when an error is encountered.
BeforeSendRequest	Occurs before a request is sent.
AfterResponseReceived	Occurs after a response is received.
ConvertParameter	Occurs when a parameter needs to be converted.
BeforeInvokeTool	Occurs before a tool is invoked.
AfterInvokeTool	Occurs after a tool is invoked.
PrepareMessages	Occurs after the messages have been prepared and before they are send to the AI.

Events are typically triggered in the following sequence:

1. SmartSession

2. SmartPrompt

3. SmartHub

When event arguments are derived from HandledEventArgs, listeners have the option to set the Handled property to true. This signals that the event has already been addressed by the listener. For instance, when handling the BeforeInvokeTool event, if you set e.Handled to true, the system will not proceed to invoke the tool, as it will presume that your code has already taken care of the necessary actions.

This feature allows you to intercept a tool call and carry out additional operations either before, after, or instead of executing the tool. This capability provides flexibility in customizing the behavior of your application, enabling you to implement complex logic or alternative workflows as needed.