​

Overview

The Web Crawler integration allows you to extract information from websites. While the integration offers both Live Search and Indexed Search, we highly recommend indexing a website first before you query it: larger websites can take several minutes to index, and the live search will be limited to the front page.

​

Authentication

The Web Crawler integration does not require any authentication and is available to all users.

​

Search Options

Following options are available for the options parameter of web_crawler:

​

Resources

The Web Crawler integration returns Website resources:

Show Website Schema

​

resource_id

string

required

The unique identifier of the website, typically the URL.

​

source

string

required

The provider that fetched the website, which is web_crawler if this integration is used to query or index it.

​

url

string

required

The URL of the website

​

title

string

The title of the website, extracted from the HTML title tag.

​

description

string

The description of the website, extracted from the HTML meta description tag or Open Graph descriptions.

​

image_url

string

The image URL of the website, extracted from the HTML meta description tag or Open Graph images.

​

language

string

The language of the website, extracted from the HTML meta language tag.

​

favicon

string

The favicon URL of the website, extracted from the HTML head.

​

summary

text

A summary of the document that can be fed directly into LLMs. When retrieved from the /query endpoint, this may summarize only the sections of the document returned as highlights (ie. parts relevant to your query). Otherwise, it will summarize the entire document.

​

data

elements[]

A structured representation of the website’s content. This field is only returned if the resource is returned from the /documents/get endpoint. If the website is returned from the /query endpoint, this field will be empty and the highlights field will contain the relevant sections of the website.

​

highlights

highlight[]

A list of highlights from the website relevant to the query. This field is only returned if the resource is returned from the /query endpoint.

​

Additional Endpoints

​

Index a website before querying it

GET /integrations/web_crawler/index Call this endpoint to index a website for indexed search. The website will be crawled recursively and added to the search index.

Show parameters

​

url

string

required

​

max_depth

number

default:2

The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.

​

max_pages

number

default:50

The maximum number of pages to crawl.