SmartRealtimeAdapter

Overview

The SmartRealTimeAdapter enables real-time data processing when working with OpenAI's realtime endpoint. By using this adapter, users can interact with the AI model using natural voice commands and receive spoken responses.

The real-time API operates via the browser's Real-Time Communication (RTC) interface. In Wisej.AI, the OpenAIEndpointRealtime implementation leverages a JavaScript component to establish a WebSocket connection with OpenAI’s endpoint. This enables seamless bi-directional streaming: the user’s voice is captured and streamed to the AI server, while the AI’s voice responses are streamed back to the browser in real-time. This architecture ensures low-latency, interactive voice communication between the user and the AI model, fully integrated within Wisej.NET applications.

Configuration

[WorksWith(typeof(OpenAIEndpointRealtime))]

Properties

In addition to the properties inherited from the SmartAdapter, the SmartRealtimeAdapter exposes the following additional properties.

Voice

The “voice” of the model can be customized to suit your application’s needs. Supported voice options include: Alloy (default), Ash, Ballad, Coral, Echo, Sage, Shimmer, and Verse. These voices provide a variety of speaking styles and tones, allowing you to select the most appropriate voice for your application's context and user experience.

TranscriptionEnabled

This setting specifies whether transcription functionality is enabled. When transcription is enabled, the adapter raises the TranscriptionReceived event each time the user's spoken input is successfully transcribed into text. The event provides the transcribed text, allowing your application to process or display the user's speech in real time.

Listening

This value indicates whether the component is currently listening for user voice input. Its state changes when the StartListening() or StopListening() methods are invoked.

Muted

This value indicates whether the component is currently muted. When muted, the component will not play back any audio responses from the AI model.

Enabled

This value specifies whether the component is configured to accept voice input from the user. When enabled, the component will listen for and process voice commands. Disabling this option prevents the component from capturing or reacting to any user speech input.

Events

AnswerReceived

The AnswerReceived event is triggered immediately after the AI model returns a response. This event may occur before or during the browser's playback of the audio response. By handling this event, your application can access the AI’s textual answer as soon as it is available.

TranscriptionReceived

The TranscriptionReceived event is triggered when TranscriptionEnabled is set to true and the AI model has successfully transcribed the user's voice input. This event provides the transcribed text immediately after processing, allowing your application to access and utilize the user's spoken input in real time.

There is no guarantee that the AnswerReceived and TranscriptionReceived events will occur in a specific order. Either event may be triggered before or after the other, depending on processing times and network conditions