google-genai/com.google.genai/LocalTokenizer/computeTokens

computeTokens

public ComputeTokensResult computeTokens(List<Content> contents)

Computes the tokens ids and string pieces in the input.

Return

A ComputeTokensResult containing the token information.

Parameters

contents

The contents to tokenize.

public ComputeTokensResult computeTokens(Content content)

Computes the token ids and string pieces for a single content object.

Return

A ComputeTokensResult containing the token information.

Parameters

content

The content to tokenize.

public ComputeTokensResult computeTokens(String content)

Computes the token ids and string pieces for a text string.

Return

A ComputeTokensResult containing the token information.

Parameters

content

The text content to tokenize.