Overview

The Web Crawler integration allows you to extract information from websites. While the integration offers both and , we highly recommend indexing a website first before you query it: larger websites can take several minutes to index, and the live search will be limited to the front page.

Authentication

The Web Crawler integration does not require any authentication and is available to all users.

Search Options

Following options are available for the options parameter of web_crawler:

url

string

required

The URL of the website to crawl. Trailing slashes are ignored.

max_depth

number

default:0

The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.

Resources

The Web Crawler integration returns Website resources:

Show Website Schema

resource_id

string

required

The unique identifier of the website, typically the URL.

source

string

required

The provider that fetched the website, which is web_crawler if this integration is used to query or index it.

url

string

required

The URL of the website

title

string

The title of the website, extracted from the HTML title tag.

description

string

The description of the website, extracted from the HTML meta description tag or Open Graph descriptions.

image_url

string

The image URL of the website, extracted from the HTML meta description tag or Open Graph images.

language

string

The language of the website, extracted from the HTML meta language tag.

favicon

string

The favicon URL of the website, extracted from the HTML head.

summary

text

A summary of the document that can be fed directly into LLMs. When retrieved from the /query endpoint, this may summarize only the sections of the document returned as highlights (ie. parts relevant to your query). Otherwise, it will summarize the entire document.

data

elements[]

A structured representation of the website’s content. This field is only returned if the resource is returned from the /documents/get endpoint. If the website is returned from the /query endpoint, this field will be empty and the highlights field will contain the relevant sections of the website.

highlights

highlight[]

A list of highlights from the website relevant to the query. This field is only returned if the resource is returned from the /query endpoint.

Additional Endpoints

Index a website before querying it

GET /integrations/web_crawler/index Call this endpoint to index a website for indexed search. The website will be crawled recursively and added to the search index.

Show parameters

url

string

required

max_depth

number

default:2

The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.

max_pages

number

default:50

The maximum number of pages to crawl.

Integrations

​Overview

​Authentication

​Search Options

​Resources

​Additional Endpoints

​Index a website before querying it

Overview

Authentication

Search Options

Resources

Additional Endpoints

Index a website before querying it