Interface ComputeTokensResult

Local tokenizer compute tokens result.

interface ComputeTokensResult {
    tokensInfo?: TokensInfo[];
}

Properties

Properties

tokensInfo?: TokensInfo[]

Lists of tokens info from the input.