.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "build_tutorial/model.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_build_tutorial_model.py: .. _model_api: Model APIs ==================== AgentScope has integrated many popular model API libraries with different modalities. .. note:: 1. The text-to-speech (TTS) and speech-to-text (STT) APIs are not included in this list. You can refer to the section :ref:`tools`. 2. The section only introduces how to use or integrate different model APIs in AgentScope. The prompt requirements and prompt engineering strategies are left in the section :ref:`prompt-engineering`. .. list-table:: :header-rows: 1 * - API - Chat - Text Generation - Vision - Embedding * - OpenAI - ✓ - ✗ - ✓ - ✓ * - DashScope - ✓ - ✗ - ✓ - ✓ * - Gemini - ✓ - ✗ - ✗ - ✓ * - Ollama - ✓ - ✓ - ✓ - ✓ * - Yi - ✓ - ✗ - ✗ - ✗ * - LiteLLM - ✓ - ✗ - ✗ - ✗ * - Zhipu - ✓ - ✗ - ✗ - ✓ * - Anthropic - ✓ - ✗ - ✗ - ✗ There are two ways to use the model APIs in AgentScope. You can choose the one that suits you best. - **Use Configuration**: This is the recommended way to build model API-agnostic applications. You can change model API by modifying the configuration, without changing the code. - **Initialize Model Explicitly**: If you only want to use a specific model API, initialize model explicitly is much more convenient and transparent to the developer. The API docstrings provide detailed information on the parameters and usage. .. tip:: Actually, using configuration and initializing model explicitly are equivalent. When you use the configuration, AgentScope just passes the key-value pairs in the configuration to initialize the model automatically. .. GENERATED FROM PYTHON SOURCE LINES 71-80 .. code-block:: Python import os from agentscope.models import ( DashScopeChatWrapper, ModelWrapperBase, ModelResponse, ) import agentscope .. GENERATED FROM PYTHON SOURCE LINES 81-90 Using Configuration ------------------------------ In a model configuration, the following three fields are required: - config_name: The name of the configuration. - model_type: The type of the model API, e.g. "dashscope_chat", "openai_chat", etc. - model_name: The name of the model, e.g. "qwen-max", "gpt-4o", etc. You should load the configurations before using the model APIs by calling `agentscope.init()` as follows: .. GENERATED FROM PYTHON SOURCE LINES 90-108 .. code-block:: Python agentscope.init( model_configs=[ { "config_name": "gpt-4o_temperature-0.5", "model_type": "openai_chat", "model_name": "gpt-4o", "api_key": "xxx", "temperature": 0.5, }, { "config_name": "my-qwen-max", "model_type": "dashscope_chat", "model_name": "qwen-max", }, ], ) .. rst-class:: sphx-glr-script-out .. code-block:: none [] .. GENERATED FROM PYTHON SOURCE LINES 109-110 For the other parameters, you .. GENERATED FROM PYTHON SOURCE LINES 112-116 Initializing Model Explicitly -------------------------------- The available model APIs are modularized in the `agentscope.models` module. You can initialize a model explicitly by calling the corresponding model class. .. GENERATED FROM PYTHON SOURCE LINES 116-122 .. code-block:: Python # print the modules under agentscope.models for module_name in agentscope.models.__all__: if module_name.endswith("Wrapper"): print(module_name) .. rst-class:: sphx-glr-script-out .. code-block:: none PostAPIChatWrapper OpenAIChatWrapper OpenAIDALLEWrapper OpenAIEmbeddingWrapper DashScopeChatWrapper DashScopeImageSynthesisWrapper DashScopeTextEmbeddingWrapper DashScopeMultiModalWrapper OllamaChatWrapper OllamaEmbeddingWrapper OllamaGenerationWrapper GeminiChatWrapper GeminiEmbeddingWrapper ZhipuAIChatWrapper ZhipuAIEmbeddingWrapper LiteLLMChatWrapper YiChatWrapper AnthropicChatWrapper .. GENERATED FROM PYTHON SOURCE LINES 123-124 Taking DashScope Chat API as an example: .. GENERATED FROM PYTHON SOURCE LINES 124-138 .. code-block:: Python model = DashScopeChatWrapper( config_name="_", model_name="qwen-max", api_key=os.environ["DASHSCOPE_API_KEY"], stream=False, ) response = model( messages=[ {"role": "user", "content": "Hi!"}, ], ) .. GENERATED FROM PYTHON SOURCE LINES 139-147 The `response` is an object of `agentscope.models.ModelResponse`, which contains the following fields: - text: The generated text - embedding: The generated embeddings - image_urls: Refer to generated images - raw: The raw response from the API - parsed: The parsed response, e.g. load the text into a JSON object - stream: A generator that yields the response text chunk by chunk, refer to section :ref: `streaming` for more details. .. GENERATED FROM PYTHON SOURCE LINES 147-155 .. code-block:: Python print(f"Text: {response.text}") print(f"Embedding: {response.embedding}") print(f"Image URLs: {response.image_urls}") print(f"Raw: {response.raw}") print(f"Parsed: {response.parsed}") print(f"Stream: {response.stream}") .. rst-class:: sphx-glr-script-out .. code-block:: none Text: Hello! How can I assist you today? Embedding: None Image URLs: None Raw: {"status_code": 200, "request_id": "caf63707-b397-9551-8ce7-d73e28f68973", "code": "", "message": "", "output": {"text": null, "finish_reason": null, "choices": [{"finish_reason": "stop", "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}}]}, "usage": {"input_tokens": 10, "output_tokens": 9, "prompt_tokens_details": {"cached_tokens": 0}, "total_tokens": 19}} Parsed: None Stream: None .. GENERATED FROM PYTHON SOURCE LINES 156-196 .. _integrating_new_api: Integrating New LLM API ---------------------------- There are two ways to integrate a new LLM API into AgentScope. OpenAI-Compatible APIs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your model is compatible with OpenAI Python API, reusing the `OpenAIChatWrapper` class with specific parameters is recommended. .. note:: You should take care of the messages format manually. Taking `vLLM `_, an OpenAI-compatible LLM inference engine, as an example, its `official doc `_ provides the following example for OpenAI Python client library: .. code-block:: python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="token-abc123", ) completion = client.chat.completions.create( model="NousResearch/Meta-Llama-3-8B-Instruct", messages=[ {"role": "user", "content": "Hello!"} ], temperature=0.5, ) print(completion.choices[0].message) It's very easy to integrate vLLM into AgentScope as follows: - put the parameters for initializing OpenAI client (except `api_key`) into `client_args`, and - the parameters for generating completions (expect `model`) into `generate_args`. .. GENERATED FROM PYTHON SOURCE LINES 196-210 .. code-block:: Python vllm_model_config = { "model_type": "openai_chat", "config_name": "vllm_llama2-7b-chat-hf", "model_name": "meta-llama/Llama-2-7b-chat-hf", "api_key": "token-abc123", # The API key "client_args": { "base_url": "http://localhost:8000/v1/", # Used to specify the base URL of the API }, "generate_args": { "temperature": 0.5, # The generation parameters, e.g. temperature, seed }, } .. GENERATED FROM PYTHON SOURCE LINES 211-212 Or, directly initialize the OpenAI Chat model wrapper with the parameters: .. GENERATED FROM PYTHON SOURCE LINES 212-223 .. code-block:: Python from agentscope.models import OpenAIChatWrapper model = OpenAIChatWrapper( config_name="", model_name="meta-llama/Llama-2-7b-chat-hf", api_key="token-abc123", client_args={"base_url": "http://localhost:8000/v1/"}, generate_args={"temperature": 0.5}, ) .. GENERATED FROM PYTHON SOURCE LINES 224-246 RESTful APIs ^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your model is accessed via RESTful post API, and OpenAI-compatible in response format, consider to use the `PostAPIChatWrapper`. Taking the following curl command as an example, just extract the **header**, **API URL**, and **data** (except `messages`, which will be passed automatically) as the parameters for initializing the model wrapper. For an example post request: .. code-block:: bash curl https://api.openai.com/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer $OPENAI_API_KEY" -d '{ "model": "gpt-4o", "messages": [ {"role": "user", "content": "write a haiku about ai"} ] }' The corresponding model wrapper initialization is as follows: .. GENERATED FROM PYTHON SOURCE LINES 246-261 .. code-block:: Python from agentscope.models import PostAPIChatWrapper post_api_model = PostAPIChatWrapper( config_name="", api_url="https://api.openai.com/v1/chat/completions", # The target URL headers={ "Content-Type": "application/json", # From the header "Authorization": "Bearer $OPENAI_API_KEY", }, json_args={ "model": "gpt-4o", # From the data }, ) .. GENERATED FROM PYTHON SOURCE LINES 262-263 Its model configuration is as follows: .. GENERATED FROM PYTHON SOURCE LINES 263-276 .. code-block:: Python post_api_config = { "config_name": "{my_post_model_config_name}", "model_type": "post_api_chat", "api_url": "https://api.openai.com/v1/chat/completions", "headers": { "Authorization": "Bearer {YOUR_API_TOKEN}", }, "json_args": { "model": "gpt-4o", }, } .. GENERATED FROM PYTHON SOURCE LINES 277-282 If your model API response format is different from OpenAI API, you can inherit from `PostAPIChatWrapper` and override the `_parse_response` method to adapt to your API response format. .. note:: You need to define a new `model_type` field in the subclass to distinguish it from the existing model wrappers. .. GENERATED FROM PYTHON SOURCE LINES 282-306 .. code-block:: Python class MyNewModelWrapper(PostAPIChatWrapper): model_type: str = "{my_new_model_type}" def _parse_response(self, response: dict) -> ModelResponse: """Parse the response from the API server. Args: response (`dict`): The response obtained from API server and parsed by `response.json()` to unify the format. Return (`ModelResponse`): The parsed response. """ # TODO: Replace by your own parsing logic return ModelResponse( text=response["data"]["response"]["choices"][0]["message"][ "content" ], ) .. GENERATED FROM PYTHON SOURCE LINES 307-315 From Scratch ^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you decide to implement a new model API from scratch, you need to know the following concepts in AgentScope: - **model_type**: When using model configurations, AgentScope uses the `model_type` field to distinguish different model APIs. So ensure your new model wrapper class has a unique `model_type`. - **__init__**: When initializing from configuration, AgentScope passes all the key-value pairs in the configuration to the `__init__` method of the model wrapper. So ensure your `__init__` method can handle all the parameters in the configuration. - **__call__**: The core method of the model wrapper is `__call__`, which takes the input messages and returns the response. Its return value should be an object of `ModelResponse`. .. GENERATED FROM PYTHON SOURCE LINES 315-333 .. code-block:: Python class MyNewModelWrapper(ModelWrapperBase): model_type: str = "{my_new_model_type}" def __init__(self, config_name, model_name, **kwargs) -> None: super().__init__(config_name, model_name=model_name) # TODO: Initialize your model here def __call__(self, *args, **kwargs) -> ModelResponse: # TODO: Implement the core logic of your model here return ModelResponse( text="Hello, World!", ) .. GENERATED FROM PYTHON SOURCE LINES 334-342 .. tip:: Optionally, you can implement a format method to format the prompt before sending it to the model API. Refer to :ref:`prompt-engineering` for more details. Further Reading --------------------- - :ref:`prompt-engineering` - :ref:`streaming` - :ref:`structured-output` .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 2.444 seconds) .. _sphx_glr_download_build_tutorial_model.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: model.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: model.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: model.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_