> For the complete documentation index, see [llms.txt](https://docs.wisej.com/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.wisej.com/ai/components/api/helpers/wisej.ai.helpers.texttokenizer.md).

# TextTokenizer

Namespace: **Wisej.AI.Helpers**

Assembly: **Wisej.AI** (3.5.0.0)

Provides methods for tokenizing text content using a specified encoder.

{% tabs %}
{% tab title="C#" %}

```csharp
public class TextTokenizer
```

{% endtab %}

{% tab title="VB.NET" %}

```visual-basic
Public Class TextTokenizer
```

{% endtab %}
{% endtabs %}

Supports the following models:

* o200K\_base Used by gpt-4o and gpt-4o-mini.
* cl100k\_base Used by gpt-4-turbo, gpt-4, gpt-3.5-turbo, text-embedding-ada-002, text-embedding-3-small, text-embedding-3-large
* p50k\_base Used by text-davinci-002, text-davinci-003.

## Methods

### ![](/files/6srAlWaUVvAWWpvmGKFp) CountTokens(content, name)

Counts the number of tokens in the specified text content using the given encoding model.

| Parameter                                 | Type                                                          | Description                                                                   |
| ----------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| **content**                               | [String](https://docs.microsoft.com/dotnet/api/system.string) | The text content to analyze.                                                  |
| **name** ![](/files/52BqKf9SltWQ97FbQSBF) | [String](https://docs.microsoft.com/dotnet/api/system.string) | The name of the encoder to use for counting tokens. Default is "o200K\_base". |

**Returns:** [Int32](https://docs.microsoft.com/dotnet/api/system.int32). The number of tokens in the content.

This method calculates the total number of tokens present in the content based on the specified encoding model.

```csharp

int tokenCount = TextTokenizer.CountTokens("Sample text");
Console.WriteLine($"Number of tokens: {tokenCount}");

```

**Throws:**

* [ArgumentNullException](https://docs.microsoft.com/dotnet/api/system.argumentnullexception)\
  Thrown when the content is null.

### ![](/files/6srAlWaUVvAWWpvmGKFp) Tokenize(content, name)

Tokenizes the specified text content using the given encoder.

| Parameter                                 | Type                                                          | Description                                                                |
| ----------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------- |
| **content**                               | [String](https://docs.microsoft.com/dotnet/api/system.string) | The text content to tokenize.                                              |
| **name** ![](/files/52BqKf9SltWQ97FbQSBF) | [String](https://docs.microsoft.com/dotnet/api/system.string) | The name of the encoder to use for tokenization. Default is "o200K\_base". |

**Returns:** [IReadOnlyCollection\<String>](https://docs.microsoft.com/dotnet/api/system.collections.generic.ireadonlycollection-1). A read-only collection of tokens extracted from the content.

This method uses the specified encoding model to break down the content into individual tokens.

```csharp

var tokens = TextTokenizer.Tokenize("Sample text");
foreach (var token in tokens)
{
Console.WriteLine(token);
}

```

**Throws:**

* [ArgumentNullException](https://docs.microsoft.com/dotnet/api/system.argumentnullexception)\
  Thrown when the content is null.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.wisej.com/ai/components/api/helpers/wisej.ai.helpers.texttokenizer.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.