# DefaultTokenizerService

Namespace: **Wisej.AI.Services**

Assembly: **Wisej.AI** (3.5.0.0)

Provides services for tokenizing text, including counting tokens, tokenizing, and truncating content based on a token limit.

{% tabs %}
{% tab title="C#" %}

```csharp
public class DefaultTokenizerService : ITokenizerService
```

{% endtab %}

{% tab title="VB.NET" %}

```visual-basic
Public Class DefaultTokenizerService
    Inherits ITokenizerService
```

{% endtab %}
{% endtabs %}

## Constructors

### ![](/files/ptrKjmmRoQB76pvrIqh0) DefaultTokenizerService()

Initializes a new instance of [DefaultTokenizerService](https://docs.wisej.com/api?q=wisej.ai.services.defaulttokenizerservice).

## Methods

### ![](/files/ptrKjmmRoQB76pvrIqh0) CountTokens(text, encoder)

Counts the number of tokens in the specified text using an optional encoder.

| Parameter                                    | Type                                                          | Description                                                                 |
| -------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **text**                                     | [String](https://docs.microsoft.com/dotnet/api/system.string) | The text to be tokenized.                                                   |
| **encoder** ![](/files/52BqKf9SltWQ97FbQSBF) | [String](https://docs.microsoft.com/dotnet/api/system.string) | The name of the encoder to use for tokenization. Defaults to "o200K\_base". |

**Returns:** [Int32](https://docs.microsoft.com/dotnet/api/system.int32). The number of tokens in the specified text.

This method uses a tokenizer to count the number of discrete tokens in the provided input text. It can optionally accept an encoder parameter that influences how the tokenization is performed.\
Example usage:

```csharp

var tokenizerService = new DefaultTokenizerService();
int tokenCount = tokenizerService.CountTokens("This is a sample text.");

```

* [ArgumentNullException](https://docs.microsoft.com/dotnet/api/system.argumentnullexception)\
  Thrown if *text* is null.

### ![](/files/ptrKjmmRoQB76pvrIqh0) Tokenize(text, encoder)

Tokenizes the specified text into an array of tokens using an optional encoder.

| Parameter                                    | Type                                                          | Description                                                                 |
| -------------------------------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------------- |
| **text**                                     | [String](https://docs.microsoft.com/dotnet/api/system.string) | The text to be tokenized.                                                   |
| **encoder** ![](/files/52BqKf9SltWQ97FbQSBF) | [String](https://docs.microsoft.com/dotnet/api/system.string) | The name of the encoder to use for tokenization. Defaults to "o200K\_base". |

**Returns:** [String\[\]](https://docs.microsoft.com/dotnet/api/system.string). An array of tokens extracted from the specified text.

This method splits the input text into its constituent tokens, which can then be processed individually. The optional encoder parameter can alter the tokenization process.\
Example usage:

```csharp

var tokenizerService = new DefaultTokenizerService();
string[] tokens = tokenizerService.Tokenize("This is a sample text.");

```

* [ArgumentNullException](https://docs.microsoft.com/dotnet/api/system.argumentnullexception)\
  Thrown if *text* is null.

### ![](/files/ptrKjmmRoQB76pvrIqh0) TruncateContent(text, maxTokens, encoder)

Truncates the specified text to a maximum number of tokens, using an optional encoder.

| Parameter                                    | Type                                                          | Description                                                                  |
| -------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **text**                                     | [String](https://docs.microsoft.com/dotnet/api/system.string) | The text to be truncated.                                                    |
| **maxTokens**                                | [Int32](https://docs.microsoft.com/dotnet/api/system.int32)   | The maximum number of tokens to retain in the truncated text.                |
| **encoder** ![](/files/52BqKf9SltWQ97FbQSBF) | [String](https://docs.microsoft.com/dotnet/api/system.string) | An optional encoder name to use for tokenization. Defaults to "o200K\_base". |

**Returns:** [String](https://docs.microsoft.com/dotnet/api/system.string). A string representing the truncated text based on the specified maximum tokens.

This method reduces the length of the input text by truncating it to a specified number of tokens. This is useful for scenarios where the input text needs to fit within a token limit.\
Example usage:

```csharp

var tokenizerService = new DefaultTokenizerService();
string truncatedText = tokenizerService.TruncateContent("This is a sample text.", 5);

```

* [ArgumentNullException](https://docs.microsoft.com/dotnet/api/system.argumentnullexception)\
  Thrown if *text* is null.
* [ArgumentOutOfRangeException](https://docs.microsoft.com/dotnet/api/system.argumentoutofrangeexception)\
  Thrown if *maxTokens* is less than zero.

## Implements

| Name                                                                  | Description                                                                                              |
| --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| [ITokenizerService](/ai/components/api/services/itokenizerservice.md) | Represents a service for tokenizing text, counting tokens, and truncating content based on token limits. |


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.wisej.com/ai/components/api/services/itokenizerservice/wisej.ai.services.defaulttokenizerservice.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.