agentscope.token

The token module in agentscope

class TokenCounterBase[source]

Bases: object

The base class for token counting.

abstract async count(messages, **kwargs)[source]

Count the number of tokens by the given model and messages.

Parameters:

  • messages (list[dict])

  • kwargs (Any)

Return type:

int

class GeminiTokenCounter[source]

Bases: TokenCounterBase

The Gemini token counter class.

__init__(model_name, api_key, **kwargs)[source]

Initialize the Gemini token counter.

Parameters:

  • model_name (str) – The name of the Gemini model to use, e.g. “gemini-2.5-flash”.

  • api_key (str) – The API key for Google Gemini.

  • **kwargs – Additional keyword arguments that will be passed to the Gemini client.

Return type:

None

async count(messages, tools=None, **config_kwargs)[source]

Count the number of tokens of gemini models.

Parameters:

  • messages (list[dict])

  • tools (list[dict] | None)

  • config_kwargs (Any)

Return type:

int

class OpenAITokenCounter[source]

Bases: TokenCounterBase

The OpenAI token counting class.

__init__(model_name)[source]

Initialize the OpenAI token counter.

Parameters:

model_name (str) – The name of the OpenAI model to use for token counting.

Return type:

None

async count(messages, tools=None, **kwargs)[source]

Count the token numbers of the given messages.

Note

OpenAI hasn’t provided an official guide for counting tokens with tools. If you have any ideas, please open an issue on our GitHub repository.

Parameters:

  • messages (list[dict[str, Any]]) – A list of dictionaries, where role and content fields are required.

  • tools (list[dict], defaults to None)

  • kwargs (Any)

Return type:

int

class AnthropicTokenCounter[source]

Bases: object

The Anthropic token counter class.

__init__(model_name, api_key, **kwargs)[source]

Initialize the Anthropic token counter.

Parameters:

  • model_name (str) – The name of the Anthropic model to use, e.g. “claude-2”.

  • api_key (str) – The API key for Anthropic.

  • kwargs (Any)

Return type:

None

async count(messages, tools=None, **kwargs)[source]

Count the number of tokens for the given messages

Note

The Anthropic token counting API requires the multimodal data to be in base64 format,

Parameters:

  • messages (list[dict]) – A list of dictionaries, where role and content fields are required.

  • tools (list[dict] | None, defaults to None) – The tools JSON schemas that the model can use.

  • **kwargs (Any) – Additional keyword arguments for the token counting API.

Return type:

int

class HuggingFaceTokenCounter[source]

Bases: TokenCounterBase

The token counter for Huggingface models.

__init__(pretrained_model_name_or_path, use_mirror=False, use_fast=False, trust_remote_code=False, **kwargs)[source]

Initialize the huggingface token counter.

Parameters:

  • pretrained_model_name_or_path (str) – The name or path of the pretrained model, which will be used to download the tokenizer from Huggingface Hub.

  • use_mirror (bool, defaults to False) – Whether to enable the HuggingFace mirror, which is useful for users in China.

  • use_fast (bool, defaults to False) – The argument that will be passed to the tokenizer.

  • trust_remote_code (bool, defaults to False) – The argument that will be passed to the tokenizer.

  • **kwargs – Additional keyword arguments that will be passed to the tokenizer.

Return type:

None

async count(messages, tools=None, **kwargs)[source]

Count the number of tokens with the tokenizer download from HuggingFace hub.

Parameters:

  • messages (list[dict]) – A list of message dictionaries

  • tools (list[dict] | None, defaults to None) – The JSON schema of the tools, which will also be involved in the token counting.

  • **kwargs (Any) – The additional keyword arguments that will be passed to the tokenizer, e.g. chat_template, padding, etc.

Return type:

int