DefaultTokenizerService
Wisej.AI.Services.DefaultTokenizerService
Namespace: Wisej.AI.Services
Assembly: Wisej.AI (3.5.0.0)
Provides services for tokenizing text, including counting tokens, tokenizing, and truncating content based on a token limit.
public class DefaultTokenizerService : ITokenizerService
Constructors
DefaultTokenizerService()

Initializes a new instance of DefaultTokenizerService.
Methods
CountTokens(text, encoder)

Counts the number of tokens in the specified text using an optional encoder.
Returns: Int32. The number of tokens in the specified text.
This method uses a tokenizer to count the number of discrete tokens in the provided input text. It can optionally accept an encoder parameter that influences how the tokenization is performed. Example usage:
var tokenizerService = new DefaultTokenizerService();
int tokenCount = tokenizerService.CountTokens("This is a sample text.");
ArgumentNullException Thrown if text is null.
Tokenize(text, encoder)

Tokenizes the specified text into an array of tokens using an optional encoder.
Returns: String[]. An array of tokens extracted from the specified text.
This method splits the input text into its constituent tokens, which can then be processed individually. The optional encoder parameter can alter the tokenization process. Example usage:
var tokenizerService = new DefaultTokenizerService();
string[] tokens = tokenizerService.Tokenize("This is a sample text.");
ArgumentNullException Thrown if text is null.
TruncateContent(text, maxTokens, encoder)

Truncates the specified text to a maximum number of tokens, using an optional encoder.
Returns: String. A string representing the truncated text based on the specified maximum tokens.
This method reduces the length of the input text by truncating it to a specified number of tokens. This is useful for scenarios where the input text needs to fit within a token limit. Example usage:
var tokenizerService = new DefaultTokenizerService();
string truncatedText = tokenizerService.TruncateContent("This is a sample text.", 5);
ArgumentNullException Thrown if text is null.
ArgumentOutOfRangeException Thrown if maxTokens is less than zero.
Implements
Represents a service for tokenizing text, counting tokens, and truncating content based on token limits.
Last updated