agentscope.service.web.web_digest module

parsing and digesting the web pages

agentscope.service.web.web_digest.is_valid_url(url: str) → bool

Use urlparse to check if a URL is valid :param url: string to be checked :type url: str

返回:

True if url is valid, False otherwise

返回类型:

bool

agentscope.service.web.web_digest.load_web(url: str, keep_raw: bool = True, html_selected_tags: Sequence[str] | None = None, self_parse_func: Callable[[Response], Any] | None = None, timeout: int = 5) → ServiceResponse

Function for parsing and digesting the web page.

参数:

• url (str) – the url of the web page

• keep_raw (bool) – Whether to keep raw HTML. If True, the content is stored with key “raw”.

• html_selected_tags (Optional[Sequence[str]]) – the text in elements of html_selected_tags will be extracted and stored with “html_to_text” key in return.

• self_parse_func (Optional[Callable]) – if “self_parse_func” is not None, then the function will be invoked with the requests.Response as input. The result is stored with self_define_func key

• timeout (int) – timeout parameter for requests.

返回:

If successful, ServiceResponse object is returned with content field is a dict, where keys are subset of:

”raw”: exists if keep_raw is True, store raw HTML content`;

”self_define_func”: exists if self_parse_func is provided, store the return of self_define_func;

”html_to_text”: exists if html_selected_tags is provided and not empty;

”json”: exists if url links to a json webpage, then it is parsed as json.

For example, ServiceResponse.content field is

{
    "raw": xxxxx,
    "selected_tags_text": xxxxx
}

返回类型:

ServiceResponse

agentscope.service.web.web_digest.parse_html_to_text(html_text: str, html_selected_tags: Sequence[str] | None = None) → str

Parse the obtained HTML file.

参数:

• html_text (str) – HTML source code

• html_selected_tags (Optional[Sequence[str]]) – the text in elements of html_selected_tags will be extracted and returned.

返回:

If successful, ServiceResponse object is returned with content field is processed text content of the selected tags,

返回类型:

ServiceResponse

agentscope.service.web.web_digest.digest_webpage(web_text_or_url: str, model: ModelWrapperBase | None = None, html_selected_tags: Sequence[str] = ('h', 'p', 'li', 'div', 'a'), digest_prompt: str = "You're a web page analyser. You job is to extract importantand useful information from html or webpage description.\n") → ServiceResponse

Digest the given webpage.

参数:

• web_text_or_url (str) – preprocessed web text or url to the web page

• model (ModelWrapperBase) – the model to digest the web content

• html_selected_tags (Sequence[str]) – the text in elements of html_selected_tags will be extracted and feed to the model

• digest_prompt (str) – system prompt for the model to digest the web content

返回:

If successful, ServiceResponse object is returned with content field filled with the model output.

返回类型:

ServiceResponse