LogoLogo
HomeNewsSupportVideos
  • Welcome
  • Wisej.NET
  • Concepts
    • Getting Started
    • General Concepts
    • Architecture
    • Extensibility
    • AI Providers
    • Vector Databases
    • Usage Metrics
    • Logging & Error Handling
  • Markup
  • Components
    • API
      • SmartAdapter
        • SmartAudioTTSAdapter
        • SmartAudioWhisperAdapter
        • SmartCalendarAdapter
        • SmartChartAdapter
        • SmartChartJS3Adapter
        • SmartChatBoxAdapter
        • SmartComboBoxAdapter
        • SmartCopilotAdapter
        • SmartDataEntryAdapter
        • SmartDocumentAdapter
        • SmartFullCalendarAdapter
        • SmartObjectAdapter
        • SmartPictureBoxAdapter
        • SmartQueryAdapter
        • SmartRealtimeAdapter
        • SmartReportAdapter
        • SmartTextBoxAdapter
        • SmartAdapter.ExtendsAttribute
        • SmartAdapter.FieldNameAttribute
        • SmartAdapter.FieldPromptAttribute
        • SmartAdapter.FieldRectangleAttribute
        • SmartAdapter.WorksWithAttribute
      • SmartEndpoint
        • AmazonBedrockEndpoint
        • AnthropicEndpoint
        • AzureAIEndpoint
        • CerebrasEndpoint
        • DeepSeekEndpoint
        • GoogleAIEndpoint
        • GroqCloudEndpoint
        • GroqCloudEndpointWhisper
        • HuggingFaceEndpoint
        • HuggingFaceJavaScriptEndpoint
        • LocalAIEndpoint
        • LocalAIEndpointImageGen
        • LocalAIEndpointTTS
        • LocalAIEndpointWhisper
        • NvidiaAIEndpoint
        • OllamaEndpoint
        • OpenAIEndpoint
        • OpenAIEndpointDallE
        • OpenAIEndpointRealtime
        • OpenAIEndpointTTS
        • OpenAIEndpointWhisper
        • SambaNovaEndpoint
        • SmartHttpEndpoint
        • TogetherAIEndpoint
        • XAIEndpoint
        • SmartEndpoint.Metrics
        • SmartEndpoint.Response
      • SmartExtensions
      • SmartHub
        • SmartSession.ConvertParameterEventArgs
        • SmartSession.ConvertParameterEventHandler
        • SmartSession.ErrorEventArgs
        • SmartSession.ErrorEventHandler
        • SmartSession.InvokeToolEventArgs
        • SmartSession.InvokeToolEventHandler
        • SmartSession.MessagesEventArgs
        • SmartSession.MessagesEventHandler
      • SmartObject
      • SmartPrompt
        • SmartAgentPrompt
        • SmartParallelPrompt
        • SmartPrompt.Parameter
        • SmartSession.ConvertParameterEventArgs
        • SmartSession.ConvertParameterEventHandler
        • SmartSession.ErrorEventArgs
        • SmartSession.ErrorEventHandler
        • SmartSession.InvokeToolEventArgs
        • SmartSession.InvokeToolEventHandler
        • SmartSession.MessagesEventArgs
        • SmartSession.MessagesEventHandler
      • SmartRealtimeSession
      • SmartSession
        • SmartSession.ConvertParameterEventArgs
        • SmartSession.ConvertParameterEventHandler
        • SmartSession.ErrorEventArgs
        • SmartSession.ErrorEventHandler
        • SmartSession.InvokeToolEventArgs
        • SmartSession.InvokeToolEventHandler
        • SmartSession.Message
        • SmartSession.MessageCollection
        • SmartSession.MessageRole
        • SmartSession.MessagesEventArgs
        • SmartSession.MessagesEventHandler
        • SmartSession.TrimmingStrategy
      • SmartTool
        • SmartTool.IToolProvider
        • SmartTool.ToolAttribute
        • SmartTool.ToolContext
      • Markup
        • MarkupExtensions
      • Controls
        • UVLightOverlay
      • Embeddings
        • EmbeddedDocument
        • Embedding
        • Matches
        • Metadata
      • Helpers
        • ApiKeys
        • Markdown
        • TextTokenizer
      • Services
        • DefaultSessionTrimmingService
        • IDocumentConversionService
          • DefaultDocumentConversionService
        • IEmbeddingGenerationService
          • DefaultEmbeddingGenerationService
          • HuggingFaceEmbeddingGenerationService
        • IEmbeddingStorageService
          • AzureAISearchEmbeddingStorageService
          • ChromaEmbeddingStorageService
          • FileSystemEmbeddingStorageService
          • MemoryEmbeddingStorageService
          • PineconeEmbeddingStorageService
          • QdrantEmbeddingStorageService
        • IHttpClientService
          • DefaultHttpClientService
        • ILoggerService
          • DefaultLoggerService
        • IOCRService
          • DefaultOCRService
        • IRerankingService
          • DefaultRerankingService
          • LocalAIRerankingService
          • PineconeRerankingService
        • ISessionTrimmingService
          • DefaultSessionTrimmingService
        • ITextSplitterService
          • RecursiveCharacterTextSplitterService
          • TextSplitterServiceBase
        • ITokenizerService
          • DefaultTokenizerService
        • IWebSearchService
          • BingWebSearchService
          • BraveWebSearchService
          • GoogleWebSearchService
      • Tools
        • ArxivTools
        • ChartJS3Tools
        • DatabaseTools
        • DataTableFilterTools
        • DocumentSearchTools
        • DocumentTools
        • FullCalendarTools
        • IToolsContainer
        • MathTools
        • ToolsContainer
        • UtilityTools
        • WebSearchTools
    • Built-in Services
      • IOCRService
      • ILoggerService
      • ITextSplitterService
      • ITokenizerService
      • IHttpClientService
      • IWebSearchService
      • IRerankingService
      • ISessionTrimmingService
      • IDocumentConversionService
      • IEmbeddingStorageService
      • IEmbeddingGenerationService
    • Built-in SmartTools
      • ToolsContainer
      • MathTools
      • UtilityTools
      • DatabaseTools
      • DocumentTools
      • DocumentSearchTools
      • WebSearchTools
      • ChartJS3Tools
      • FullCalendarTools
    • Built-in SmartAdapters
      • SmartAdapter
      • SmartAudioTTSAdapter
      • SmartAudioWhisperAdapter
      • SmartCalendarAdapter
      • SmartChartAdapter
      • SmartChartJS3Adapter
      • SmartChatBoxAdapter
      • SmartComboBoxAdapter
      • SmartCopilotAdapter
      • SmartDataEntryAdapter
      • SmartDocumentAdapter
      • SmartFullCalendarAdapter
      • SmartObjectAdapter
      • SmartPictureBoxAdapter
      • SmartQueryAdapter
      • SmartRealtimeAdapter
      • SmartReportAdapter
      • SmartTextBoxAdapter
    • Configure Services
    • Using SmartHub
    • Using SmartTools
    • Using SmartPrompt
    • Using SmartSession
    • Using SmartRealTimeAdapter
    • UVLightOverlay Control
Powered by GitBook
On this page
  • Constructors
  • OpenAIEndpointRealtime()
  • Properties
  • Model
  • TranscriptionModel
  • URL
  • UseNativeTools
  • Methods
  • AddOptions(message, session)
  • BuildInstructions(session)
  • BuildToolsPayload(session)
  • CreateConfiguration(session)
  • Disable()
  • Dispose(disposing)
  • Enable()
  • GetApiURL()
  • GetEphemeralTokenAsync(session)
  • Mute()
  • OnFunctionCall(e)
  • OnResponseCreated(e)
  • OnResponseDone(e)
  • OnTranscriptionCompleted(e)
  • ReadUsage(message, reply)
  • SendMessages(messages)
  • StartListening(session)
  • StopListening(session)
  • Unmute()
  • Events
  • FunctionCall
  • ResponseCreated
  • ResponseDone
  • TranscriptionCompleted
Export as PDF
  1. Components
  2. API
  3. SmartEndpoint

OpenAIEndpointRealtime

Wisej.AI.Endpoints.OpenAIEndpointRealtime

PreviousOpenAIEndpointDallENextOpenAIEndpointTTS

Last updated 5 days ago

Namespace: Wisej.AI.Endpoints

Assembly: Wisej.AI (3.5.0.0)

Represents a component that manages communication with OpenAI's real-time API endpoints.

public class OpenAIEndpointRealtime : SmartHttpEndpoint
Public Class OpenAIEndpointRealtime
    Inherits SmartHttpEndpoint

provides a convenient way to connect to and interact with OpenAI's real-time endpoints within a Wisej application. It extends , inheriting HTTP communication capabilities and adding specialized logic for real-time data exchange with OpenAI services. This component can be used to send prompts, receive streaming responses, and manage session state with OpenAI's real-time APIs.

Constructors

OpenAIEndpointRealtime()

Initializes a new instance of the class with default settings.

Properties

Model

: Gets or sets the model used for chat completions. (Default: "gpt-4o-realtime-preview")

Methods

Adds options to the message payload.

Parameter
Type
Description

message

The message payload to modify.

session

The current session context.

Sets default values for temperature and max tokens in the message payload.

Constructs the instruction string for the OpenAI real-time API endpoint based on the current session context.

Parameter
Type
Description

session

This method generates the instruction prompt by extracting the system message from the session, incorporating tool prompts, and replacing template placeholders such as {{tools}}, {{today}}, and {{language}} with their corresponding values. The resulting instruction string is then processed to replace any additional session parameters. Override this method to customize how instructions are built for the OpenAI endpoint.

Builds the tools payload based on the session context.

Parameter
Type
Description

session

The session context.

Throws:

Creates the configuration payload for the OpenAI real-time API session.

Parameter
Type
Description

session

This method builds a dynamic configuration object that includes options, tools, instructions, and modalities required by the OpenAI realtime API. If audio transcription is enabled, the configuration will also include the transcription model. The transcription_enabled property is removed from the final configuration payload.

Disables the OpenAIEndpointRealtime component, stopping it from processing or sending further requests.

Releases the unmanaged resources used by the component and optionally releases the managed resources.

Parameter
Type
Description

disposing

true to release both managed and unmanaged resources; false to release only unmanaged resources.

Enables the OpenAIEndpointRealtime component, allowing it to process and send requests.

This method activates the component, allowing it to resume communication with the OpenAI Realtime API endpoint after being disabled.

Constructs the API URL for the real time api.

Asynchronously retrieves an ephemeral token for the specified session from the OpenAI realtime API endpoint.

Parameter
Type
Description

session

This method constructs a request to the OpenAI real-time API endpoint to obtain a temporary (ephemeral) token associated with the provided session. The token is typically used for short-lived authentication or authorization purposes during real-time communication. The method sends a POST request with the session configuration and parses the response as a dynamic object.

Mutes the OpenAIEndpointRealtime component, suppressing any audio or notification output.

Parameter
Type
Description

e

This method is called to notify subscribers that a function call has been detected in the AI's response. Override this method to provide custom handling for function calls.

Parameter
Type
Description

e

Parameter
Type
Description

e

Parameter
Type
Description

e

Reads and updates the usage statistics from the OpenAI real-time API reply.

Parameter
Type
Description

message

The message object to update with usage data.

reply

The dynamic object containing the API response, expected to include input_tokens and output_tokens properties.

Sends an array of messages to the OpenAI Realtime API endpoint for processing.

Parameter
Type
Description

messages

An array of dynamic message objects to be sent to the OpenAI endpoint.

Use this method to transmit one or more messages to the OpenAI Realtime API endpoint. Each message should conform to the expected structure required by the endpoint.

Initiates listening for real-time events from the OpenAI endpoint using the specified session.

Parameter
Type
Description

session

Stops listening for real-time events from the OpenAI endpoint for the specified session.

Parameter
Type
Description

session

Unmutes the OpenAIEndpointRealtime component, restoring audio or notification output.

Events

This event is triggered when the SmartRealtimeAdapter identifies a function call in the AI's response, allowing the application to handle or execute the function as needed.

TranscriptionModel

: Gets or sets the model used for the audio transcription. (Default: "gpt-4o-transcribe")

URL

: Gets or sets the base URL for the OpenAI API. (Default: "https://api.openai.com/v1")

UseNativeTools

: (Default: True)

AddOptions(message, session)

BuildInstructions(session)

The instance containing the session context and parameters used to build the instruction string.

Returns: . A string containing the fully constructed instructions for the OpenAI real-time API endpoint.

BuildToolsPayload(session)

Returns: . A JSON object representing the tools payload.

Thrown when the session is null.

CreateConfiguration(session)

The instance containing session-specific options and settings.

Returns: . A dynamic object representing the configuration payload for the session.

Disable()

This method deactivates the component, preventing any further communication with the OpenAI Realtime API endpoint until is called.

Dispose(disposing)

Enable()

GetApiURL()

Returns: . The full API URL for the real time api..

GetEphemeralTokenAsync(session)

The instance representing the current user session for which the ephemeral token is requested.

Returns: . A task that represents the asynchronous operation. The task result contains a dynamic object representing the ephemeral token response from the API.

Mute()

Use this method to temporarily silence the component without disabling its core functionality. To restore audio or notifications, call .

OnFunctionCall(e)

Raises the event.

A instance containing the event data.

OnResponseCreated(e)

Raises the event.

An instance containing the event data.

OnResponseDone(e)

Raises the event.

A instance containing the event data for the completed response.

OnTranscriptionCompleted(e)

Raises the event.

A instance containing the event data for the completed transcription.

ReadUsage(message, reply)

SendMessages(messages)

StartListening(session)

The instance representing the current user session. This parameter cannot be null.

This method establishes a connection to the OpenAI real-time API endpoint by obtaining an ephemeral token for the provided session and starting the listening process. It also triggers the OnStart event for the session and updates the associated component in the Wisej application. If session is null, an is thrown.

StopListening(session)

The instance representing the current user session. This parameter cannot be null.

This method terminates the connection to the OpenAI real-time API endpoint for the given session. It also triggers the OnDone event for the session to indicate that listening has stopped. If session is null, an is thrown.

Unmute()

This method re-enables audio or notifications that were previously suppressed by .

FunctionCall

Occurs when a function call is detected in the response from the OpenAI Realtime API.

ResponseCreated

Occurs when a new response is created by the OpenAI real-time endpoint.

ResponseDone

Occurs when the response from the OpenAI real-time endpoint has been fully received and processing is complete.

TranscriptionCompleted

Occurs when a transcription operation has completed.

String
String
Boolean
String
Object[]
ArgumentNullException
Object
String
Task<Object>
ArgumentNullException
ArgumentNullException
FunctionCallEventHandler
EventHandler
ResponseDoneEventHandler
TranscriptionCompletedEventHandler
Enable
Unmute
FunctionCall
ResponseCreated
ResponseDone
TranscriptionCompleted
Mute
Object
SmartSession
SmartSession
SmartSession
SmartSession
SmartSession
SmartSession
Boolean
SmartSession
SmartSession
FunctionCallEventArgs
FunctionCallEventArgs
EventArgs
EventArgs
ResponseDoneEventArgs
ResponseDoneEventArgs
TranscriptionCompletedEventArgs
TranscriptionCompletedEventArgs
Message
Object
Object[]
SmartSession
SmartSession
SmartSession
SmartSession
SmartEndpoint
SmartHttpEndpoint
OpenAIEndpointRealtime
OpenAIEndpointRealtime
SmartHttpEndpoint
OpenAIEndpoint
String