google-genai/com.google.genai/LocalTokenizer

LocalTokenizer

public final class LocalTokenizer

[Experimental] Text Only Local Tokenizer.

This class provides a local tokenizer for text only token counting.

LIMITATIONS:

  • Only supports text based tokenization and no multimodal tokenization.
  • Forward compatibility depends on the open-source tokenizer models for future Gemini versions.

NOTE: The SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.

Constructors

Link copied to clipboard
public void LocalTokenizer(String modelName)
Creates a new LocalTokenizer for the specified model.

Functions

Link copied to clipboard
public ComputeTokensResult computeTokens(Content content)
Computes the token ids and string pieces for a single content object.
public ComputeTokensResult computeTokens(String content)
Computes the token ids and string pieces for a text string.
public ComputeTokensResult computeTokens(List<Content> contents)
Computes the tokens ids and string pieces in the input.
Link copied to clipboard
public CountTokensResult countTokens(Content content)
Counts the number of tokens in a single content object using default configuration.
public CountTokensResult countTokens(String content)
Counts the number of tokens in a text string using default configuration.
public CountTokensResult countTokens(List<Content> contents)
Counts the number of tokens in a list of content objects using default configuration.
public CountTokensResult countTokens(Content content, CountTokensConfig config)
Counts the number of tokens in a single content object.
public CountTokensResult countTokens(String content, CountTokensConfig config)
Counts the number of tokens in a text string.
public CountTokensResult countTokens(List<Content> contents, CountTokensConfig config)
Counts the number of tokens in a given text.