Class LocalTokenizer
-
- All Implemented Interfaces:
public final class LocalTokenizer[Experimental] Text Only Local Tokenizer.
This class provides a local tokenizer for text only token counting.
LIMITATIONS:
- Only supports text based tokenization and no multimodal tokenization.
- Forward compatibility depends on the open-source tokenizer models for future Gemini versions.
NOTE: The SDK's local tokenizer implementation is experimental and may change in the future. It only supports text based tokenization.
-
-
Constructor Summary
Constructors Constructor Description LocalTokenizer(String modelName)Creates a new LocalTokenizer for the specified model.
-
Method Summary
Modifier and Type Method Description CountTokensResultcountTokens(List<Content> contents, CountTokensConfig config)Counts the number of tokens in a given text. CountTokensResultcountTokens(List<Content> contents)Counts the number of tokens in a list of content objects using default configuration. CountTokensResultcountTokens(Content content, CountTokensConfig config)Counts the number of tokens in a single content object. CountTokensResultcountTokens(Content content)Counts the number of tokens in a single content object using default configuration. CountTokensResultcountTokens(String content, CountTokensConfig config)Counts the number of tokens in a text string. CountTokensResultcountTokens(String content)Counts the number of tokens in a text string using default configuration. ComputeTokensResultcomputeTokens(List<Content> contents)Computes the tokens ids and string pieces in the input. ComputeTokensResultcomputeTokens(Content content)Computes the token ids and string pieces for a single content object. ComputeTokensResultcomputeTokens(String content)Computes the token ids and string pieces for a text string. -
-
Constructor Detail
-
LocalTokenizer
LocalTokenizer(String modelName)
Creates a new LocalTokenizer for the specified model.- Parameters:
modelName- the name of the model to load (e.g., "gemini-2.
-
-
Method Detail
-
countTokens
CountTokensResult countTokens(List<Content> contents, CountTokensConfig config)
Counts the number of tokens in a given text.
- Parameters:
contents- The contents to tokenize.config- The configuration for counting tokens.- Returns:
A CountTokensResult containing the total number of tokens.
-
countTokens
CountTokensResult countTokens(List<Content> contents)
Counts the number of tokens in a list of content objects using default configuration.
- Parameters:
contents- The contents to tokenize.- Returns:
A CountTokensResult containing the total number of tokens.
-
countTokens
CountTokensResult countTokens(Content content, CountTokensConfig config)
Counts the number of tokens in a single content object.
- Parameters:
content- The content to tokenize.config- The configuration for counting tokens.- Returns:
A CountTokensResult containing the total number of tokens.
-
countTokens
CountTokensResult countTokens(Content content)
Counts the number of tokens in a single content object using default configuration.
- Parameters:
content- The content to tokenize.- Returns:
A CountTokensResult containing the total number of tokens.
-
countTokens
CountTokensResult countTokens(String content, CountTokensConfig config)
Counts the number of tokens in a text string.
- Parameters:
content- The text content to tokenize.config- The configuration for counting tokens.- Returns:
A CountTokensResult containing the total number of tokens.
-
countTokens
CountTokensResult countTokens(String content)
Counts the number of tokens in a text string using default configuration.
- Parameters:
content- The text content to tokenize.- Returns:
A CountTokensResult containing the total number of tokens.
-
computeTokens
ComputeTokensResult computeTokens(List<Content> contents)
Computes the tokens ids and string pieces in the input.
- Parameters:
contents- The contents to tokenize.- Returns:
A ComputeTokensResult containing the token information.
-
computeTokens
ComputeTokensResult computeTokens(Content content)
Computes the token ids and string pieces for a single content object.
- Parameters:
content- The content to tokenize.- Returns:
A ComputeTokensResult containing the token information.
-
computeTokens
ComputeTokensResult computeTokens(String content)
Computes the token ids and string pieces for a text string.
- Parameters:
content- The text content to tokenize.- Returns:
A ComputeTokensResult containing the token information.
-
-
-
-