agentscope.token

The token module in agentscope

class TokenCounterBase

基类：object

The base class for token counting.

abstract async count(messages, **kwargs)

Count the number of tokens by the given model and messages.

参数:

• messages (list[dict])

• kwargs (Any)

返回类型:

int

class GeminiTokenCounter

基类：TokenCounterBase

The Gemini token counter class.

__init__(model_name, api_key, **kwargs)

Initialize the Gemini token counter.

参数:

• model_name (str) -- The name of the Gemini model to use, e.g. "gemini-2.5-flash".

• api_key (str) -- The API key for Google Gemini.

• **kwargs -- Additional keyword arguments that will be passed to the Gemini client.

返回类型:

None

async count(messages, tools=None, **config_kwargs)

Count the number of tokens of gemini models.

参数:

• messages (list[dict])

• tools (list[dict] | None)

• config_kwargs (Any)

返回类型:

int

class OpenAITokenCounter

基类：TokenCounterBase

The OpenAI token counting class.

__init__(model_name)

Initialize the OpenAI token counter.

参数:

model_name (str) -- The name of the OpenAI model to use for token counting.

返回类型:

None

async count(messages, tools=None, **kwargs)

Count the token numbers of the given messages.

备注

OpenAI hasn't provided an official guide for counting tokens with tools. If you have any ideas, please open an issue on our GitHub repository.

参数:

• messages (list[dict[str, Any]]) -- A list of dictionaries, where role and content fields are required.

• tools (list[dict], defaults to None)

• kwargs (Any)

返回类型:

int

class AnthropicTokenCounter

基类：object

The Anthropic token counter class.

__init__(model_name, api_key, **kwargs)

Initialize the Anthropic token counter.

参数:

• model_name (str) -- The name of the Anthropic model to use, e.g. "claude-2".

• api_key (str) -- The API key for Anthropic.

• kwargs (Any)

返回类型:

None

async count(messages, tools=None, **kwargs)

Count the number of tokens for the given messages

备注

The Anthropic token counting API requires the multimodal data to be in base64 format,

参数:

• messages (list[dict]) -- A list of dictionaries, where role and content fields are required.

• tools (list[dict] | None, defaults to None) -- The tools JSON schemas that the model can use.

• **kwargs (Any) -- Additional keyword arguments for the token counting API.

返回类型:

int

class HuggingFaceTokenCounter

基类：TokenCounterBase

The token counter for Huggingface models.

__init__(pretrained_model_name_or_path, use_mirror=False, use_fast=False, trust_remote_code=False, **kwargs)

Initialize the huggingface token counter.

参数:

• pretrained_model_name_or_path (str) -- The name or path of the pretrained model, which will be used to download the tokenizer from Huggingface Hub.

• use_mirror (bool, defaults to False) -- Whether to enable the HuggingFace mirror, which is useful for users in China.

• use_fast (bool, defaults to False) -- The argument that will be passed to the tokenizer.

• trust_remote_code (bool, defaults to False) -- The argument that will be passed to the tokenizer.

• **kwargs -- Additional keyword arguments that will be passed to the tokenizer.

返回类型:

None

async count(messages, tools=None, **kwargs)

Count the number of tokens with the tokenizer download from HuggingFace hub.

参数:

• messages (list[dict]) -- A list of message dictionaries

• tools (list[dict] | None, defaults to None) -- The JSON schema of the tools, which will also be involved in the token counting.

• **kwargs (Any) -- The additional keyword arguments that will be passed to the tokenizer, e.g. chat_template, padding, etc.

返回类型:

int