The Web Crawler integration allows you to extract information from websites. While the integration offers both and , we highly recommend indexing a website first before you query it: larger websites can take several minutes to index, and the live search will be limited to the front page.
Authentication
The Web Crawler integration does not require any authentication and is available to all users.
Search Options
Following options are available for the options parameter of web_crawler:
string
required
The URL of the website to crawl. Trailing slashes are ignored.
number
default:0
The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.
Resources
The Web Crawler integration returns Website resources:
Show Website Schema
string
required
The unique identifier of the website, typically the URL.
string
required
The provider that fetched the website, which is web_crawler if this integration is used to query or index it.
string
required
The URL of the website
string
The title of the website, extracted from the HTML title tag.
string
The description of the website, extracted from the HTML meta description tag or Open Graph descriptions.
string
The image URL of the website, extracted from the HTML meta description tag or Open Graph images.
string
The language of the website, extracted from the HTML meta language tag.
string
The favicon URL of the website, extracted from the HTML head.
text
A summary of the document that can be fed directly into LLMs. When retrieved from the /query endpoint, this may summarize only the sections of the document returned as highlights (i.e., parts relevant to your query). Otherwise, it will summarize the entire document.
elements[]
A structured representation of the website’s content. This field is only returned if the resource is returned from the /documents/get endpoint. If the website is returned from the /query endpoint, this field will be empty and the highlights field will contain the relevant sections of the website.
highlight[]
A list of highlights from the website relevant to the query. This field is only returned if the resource is returned from the /query endpoint.
Additional Endpoints
Index a website before querying it
GET /integrations/web_crawler/indexCall this endpoint to index a website for indexed search. The website will be crawled recursively and added to the search index.
Show parameters
string
required
number
default:2
The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.
number
default:50
The maximum number of pages to crawl.
Was this page helpful?
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Hyperspell","colors":{"primary":"#eF7E75","light":"#eF7E75","dark":"#FF7E75"},"logo":{"light":"https://mintcdn.com/hyperspell/v3b7lBADAA2jgsr8/logo/light.svg?fit=max\u0026auto=format\u0026n=v3b7lBADAA2jgsr8\u0026q=85\u0026s=daa1192bf1d8015701d50d2c12b5e843","dark":"https://mintcdn.com/hyperspell/v3b7lBADAA2jgsr8/logo/dark.svg?fit=max\u0026auto=format\u0026n=v3b7lBADAA2jgsr8\u0026q=85\u0026s=f902d22b6c72a38457450091a65af5b1","href":"https://hyperspell.com"},"favicon":"https://mintcdn.com/hyperspell/v3b7lBADAA2jgsr8/favicon.svg?fit=max\u0026auto=format\u0026n=v3b7lBADAA2jgsr8\u0026q=85\u0026s=e3868fcf92af7815460b8dc73c41e908","appearance":{"default":"light"},"background":{"decoration":"gradient","color":{"light":"#f9f9f9","dark":"#151313"}},"navbar":{"links":[{"href":"mailto:hello@hyperspell.com","label":"Support"},{"href":"https://hyperspell.com/blog","label":"Blog"}],"primary":{"type":"button","label":"Open Dashboard","href":"https://app.hyperspell.com"}},"footer":{"socials":{"x":"https://x.com/hyperspell","github":"https://github.com/hyperspell","linkedin":"https://linkedin.com/company/hyperspell"}},"fonts":{"heading":{"family":"Exposure","weight":400,"source":"fonts/exposure.woff2","format":"woff2"},"body":{"family":"Geist"}},"integrations":{"plausible":{"domain":"docs.hyperspell.com"}},"contextual":{"options":["copy","chatgpt","claude"],"display":"header"},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"hyperspell","hasAudioNarration":false,"actualSubdomain":"hyperspell","customDomains":["docs.hyperspell.com"],"gitSource":{"type":"github","owner":"hyperspell","repo":"hyperspell","deployBranch":"main","contentDirectory":"apps/docs","isPrivate":true,"mountPath":"","label":""},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"REMOVE_BRANDING":{"status":"DISABLED"},"CONTEXTUAL_FEEDBACK":{"status":"DISABLED"},"CODE_SNIPPET_FEEDBACK":{"status":"DISABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6afa6d2848b4ff5c0ecae2ad","clientVersion":"0.0.4143","searchFilterCounts":[{"language":"default","tag":"API Reference","count":69},{"language":"default","tag":"Integrations","count":68},{"language":"default","tag":"Documentation","count":18},{"language":"default","tag":"Memories","count":18},{"language":"default","tag":"Connecting Agents","count":9},{"language":"default","tag":"Resources","count":4},{"language":"default","tag":"Emotional State","count":4},{"language":"default","tag":"Evaluation","count":4},{"language":"default","tag":"Folder Policies","count":4},{"language":"default","tag":"Live","count":4}]},"assistantConfig":{},"basePath":"","hasChatPermissions":true,"hasWebMcpPermissions":true,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":true},"customCss":[],"page":{"pagePath":"integrations/all/web_crawler","slug":"/integrations/all/web_crawler","pageMetadata":{"title":"Web Crawler","description":"Get answers from websites","href":"/integrations/all/web_crawler"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Get answers from websites\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];const IndexedSearch=()=>{const{Icon,Tooltip}=_provideComponents()||{};if(!Icon)_missingMdxReference(`Icon`,true);if(!Tooltip)_missingMdxReference(`Tooltip`,true);return _jsx(Tooltip,{tip:`Indexed search uses advanced semantic, hybrid, and graph search methods to find the most relevant results. It's very fast and accurate, but requires upfront indexing of data.`,children:_jsxs(`span`,{className:`mint-text-sky-600`,children:[_jsx(Icon,{icon:`database`,size:14,color:`#0084d1`,iconType:`solid`}),` Indexed Search`]})})};const LiveSearch=()=>{const{Icon,Tooltip}=_provideComponents()||{};if(!Icon)_missingMdxReference(`Icon`,true);if(!Tooltip)_missingMdxReference(`Tooltip`,true);return _jsx(Tooltip,{tip:`Live search uses AI to search your integrations native APIs in real-time. This is slower than indexed search and often not as accurate, but does not store any data on Hyperspell's servers and is always up to date.`,children:_jsxs(`span`,{className:`mint-text-emerald-600`,children:[_jsx(Icon,{icon:`bolt`,size:14,color:`#009966`,iconType:`solid`}),` Live Search`]})})};function _createMdxContent(props){const _components={code:`code`,p:`p`,..._provideComponents(),...props.components},{Expandable,Heading,ParamField,ResponseField,_MdxComponentBoundary}=_components;if(!Expandable)_missingMdxReference(`Expandable`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);if(!_MdxComponentBoundary)_missingMdxReference(`_MdxComponentBoundary`,true);return _jsxs(_Fragment,{children:[_jsx(Heading,{level:`1`,id:`overview`,children:`Overview`}),`\n`,_jsxs(_components.p,{children:[`The Web Crawler integration allows you to extract information from websites. While the integration offers both `,_jsx(_MdxComponentBoundary,{name:`LiveSearch`,children:_jsx(LiveSearch,{})}),` and `,_jsx(_MdxComponentBoundary,{name:`IndexedSearch`,children:_jsx(IndexedSearch,{})}),`, we highly recommend indexing a website first before you query it: larger websites can take several minutes to index, and the live search will be limited to the front page.`]}),`\n`,_jsx(Heading,{level:`2`,id:`authentication`,children:`Authentication`}),`\n`,_jsx(_components.p,{children:`The Web Crawler integration does not require any authentication and is available to all users.`}),`\n`,_jsx(Heading,{level:`2`,id:`search-options`,children:`Search Options`}),`\n`,_jsxs(_components.p,{children:[`Following options are available for the `,_jsx(_components.code,{children:`options`}),` parameter of `,_jsx(_components.code,{children:`web_crawler`}),`:`]}),`\n`,_jsx(ParamField,{body:`url`,type:`string`,required:true,id:`param-url`,children:_jsx(_components.p,{children:`The URL of the website to crawl. Trailing slashes are ignored.`})}),`\n`,_jsx(ParamField,{body:`max_depth`,type:`number`,required:false,default:0,id:`param-max-depth`,children:_jsx(_components.p,{children:`The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.`})}),`\n`,_jsx(Heading,{level:`2`,id:`resources`,children:`Resources`}),`\n`,_jsxs(_components.p,{children:[`The Web Crawler integration returns `,_jsx(_components.code,{children:`Website`}),` resources:`]}),`\n`,_jsxs(Expandable,{title:`Website Schema`,children:[_jsx(ResponseField,{name:`resource_id`,type:`string`,required:true,id:`param-resource-id`,children:_jsx(_components.p,{children:`The unique identifier of the website, typically the URL.`})}),_jsx(ResponseField,{name:`source`,type:`string`,required:true,id:`param-source`,children:_jsxs(_components.p,{children:[`The provider that fetched the website, which is `,_jsx(_components.code,{children:`web_crawler`}),` if this integration is used to query or index it.`]})}),_jsx(ResponseField,{name:`url`,type:`string`,required:true,id:`param-url-1`,children:_jsx(_components.p,{children:`The URL of the website`})}),_jsx(ResponseField,{name:`title`,type:`string`,id:`param-title`,children:_jsx(_components.p,{children:`The title of the website, extracted from the HTML title tag.`})}),_jsx(ResponseField,{name:`description`,type:`string`,id:`param-description`,children:_jsx(_components.p,{children:`The description of the website, extracted from the HTML meta description tag or Open Graph descriptions.`})}),_jsx(ResponseField,{name:`image_url`,type:`string`,required:false,id:`param-image-url`,children:_jsx(_components.p,{children:`The image URL of the website, extracted from the HTML meta description tag or Open Graph images.`})}),_jsx(ResponseField,{name:`language`,type:`string`,required:false,id:`param-language`,children:_jsx(_components.p,{children:`The language of the website, extracted from the HTML meta language tag.`})}),_jsx(ResponseField,{name:`favicon`,type:`string`,required:false,id:`param-favicon`,children:_jsx(_components.p,{children:`The favicon URL of the website, extracted from the HTML head.`})}),_jsx(ResponseField,{name:`summary`,type:`text`,required:false,id:`param-summary`,children:_jsxs(_components.p,{children:[`A summary of the document that can be fed directly into LLMs. When retrieved from the `,_jsx(_components.code,{children:`/query`}),` endpoint, this may summarize only the sections of the document returned as highlights (i.e., parts relevant to your query). Otherwise, it will summarize the entire document.`]})}),_jsx(ResponseField,{name:`data`,type:`elements[]`,required:false,id:`param-data`,children:_jsxs(_components.p,{children:[`A structured representation of the website’s content. This field is only returned if the resource is returned from the `,_jsx(_components.code,{children:`/documents/get`}),` endpoint. If the website is returned from the `,_jsx(_components.code,{children:`/query`}),` endpoint, this field will be empty and the `,_jsx(_components.code,{children:`highlights`}),` field will contain the relevant sections of the website.`]})}),_jsx(ResponseField,{name:`highlights`,type:`highlight[]`,required:false,id:`param-highlights`,children:_jsxs(_components.p,{children:[`A list of highlights from the website relevant to the query. This field is only returned if the resource is returned from the `,_jsx(_components.code,{children:`/query`}),` endpoint.`]})})]}),`\n`,_jsx(Heading,{level:`2`,id:`additional-endpoints`,children:`Additional Endpoints`}),`\n`,_jsx(Heading,{level:`3`,id:`index-a-website-before-querying-it`,children:`Index a website before querying it`}),`\n`,_jsx(_components.p,{children:_jsx(_components.code,{children:`GET /integrations/web_crawler/index`})}),`\n`,_jsx(_components.p,{children:`Call this endpoint to index a website for indexed search. The website will be crawled recursively and added to the search index.`}),`\n`,_jsxs(Expandable,{title:`parameters`,children:[_jsx(ParamField,{query:`url`,type:`string`,required:true,id:`param-url-2`}),_jsx(ParamField,{query:`max_depth`,type:`number`,required:false,default:2,id:`param-max-depth-1`,children:_jsx(_components.p,{children:`The maximum depth of the website to crawl. 0 means only the root page will be queried, 1 means the root page and all pages linked from it will be queried, and so on.`})}),_jsx(ParamField,{query:`max_pages`,type:`number`,required:false,default:50,id:`param-max-pages`,children:_jsx(_components.p,{children:`The maximum number of pages to crawl.`})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{IndexedSearch,LiveSearch,default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Web Crawler","description":"Get answers from websites","href":"/integrations/all/web_crawler"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"Overview","slug":"overview","depth":1,"children":[]},{"title":"Authentication","slug":"authentication","depth":2,"children":[]},{"title":"Search Options","slug":"search-options","depth":2,"children":[]},{"title":"Resources","slug":"resources","depth":2,"children":[]},{"title":"Additional Endpoints","slug":"additional-endpoints","depth":2,"children":[{"title":"Index a website before querying it","slug":"index-a-website-before-querying-it","depth":3,"children":[]}]}],"codeExamples":{}},"jsLinks":[],"dynamicCss":".mint-text-emerald-600 {\n --tw-text-opacity: 1;\n color: rgb(5 150 105 / var(--tw-text-opacity, 1))\n}\n.mint-text-sky-600 {\n --tw-text-opacity: 1;\n color: rgb(2 132 199 / var(--tw-text-opacity, 1))\n}","subdomain":"hyperspell","siteName":"Hyperspell","title":"Web Crawler - Hyperspell","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-09-02T01:20:51.991Z","originalFileLocation":"integrations/all/web_crawler.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"light\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Get answers from websites\">\u003cmeta data-mint-head name=\"application-name\" content=\"Hyperspell\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Web Crawler - Hyperspell\">\u003cmeta data-mint-head property=\"og:description\" content=\"Get answers from websites\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Hyperspell\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.hyperspell.com/integrations/all/web_crawler\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://hyperspell.mintlify.site/_mintlify/api/og/v2?division=Integrations\u0026amp;title=Web+Crawler\u0026amp;description=Get+answers+from+websites\u0026amp;theme=d1877f2b57f72ecfb447ed52\u0026amp;signature=55cd73df91be8646e1352dc053a10351f947d520b5968224dd324930748dd6d0\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Web Crawler - Hyperspell\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Get answers from websites\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://hyperspell.mintlify.site/_mintlify/api/og/v2?division=Integrations\u0026amp;title=Web+Crawler\u0026amp;description=Get+answers+from+websites\u0026amp;theme=d1877f2b57f72ecfb447ed52\u0026amp;signature=55cd73df91be8646e1352dc053a10351f947d520b5968224dd324930748dd6d0\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/hyperspell/_L3SAE3y_mnPfj1R/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Hyperspell\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#eF7E75\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.hyperspell.com/integrations/all/web_crawler\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/integrations/all/web_crawler.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.hyperspell.com/#organization\",\"name\":\"Hyperspell\",\"url\":\"https://docs.hyperspell.com\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/hyperspell/v3b7lBADAA2jgsr8/logo/light.svg?fit=max\u0026auto=format\u0026n=v3b7lBADAA2jgsr8\u0026q=85\u0026s=daa1192bf1d8015701d50d2c12b5e843\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.hyperspell.com#website\",\"name\":\"Hyperspell\",\"url\":\"https://docs.hyperspell.com\",\"publisher\":{\"@id\":\"https://docs.hyperspell.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.hyperspell.com/integrations/all/web_crawler#webpage\",\"url\":\"https://docs.hyperspell.com/integrations/all/web_crawler\",\"name\":\"Web Crawler\",\"description\":\"Get answers from websites\",\"dateModified\":\"2026-09-02T01:20:51.991Z\",\"isPartOf\":{\"@id\":\"https://docs.hyperspell.com#website\"},\"breadcrumb\":{\"@id\":\"https://docs.hyperspell.com/integrations/all/web_crawler#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.hyperspell.com/integrations/all/web_crawler#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Integrations\",\"item\":\"https://docs.hyperspell.com/integrations/overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Web Crawler\",\"item\":\"https://docs.hyperspell.com/integrations/all/web_crawler\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.hyperspell.com/integrations/all/web_crawler#article\",\"headline\":\"Web Crawler\",\"name\":\"Web Crawler\",\"description\":\"Get answers from websites\",\"url\":\"https://docs.hyperspell.com/integrations/all/web_crawler\",\"mainEntityOfPage\":{\"@id\":\"https://docs.hyperspell.com/integrations/all/web_crawler#webpage\"},\"dateModified\":\"2026-09-02T01:20:51.991Z\",\"publisher\":{\"@id\":\"https://docs.hyperspell.com/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.hyperspell.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Hyperspell\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Hyperspell\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/integrations/all/web_crawler.md","siteName":"Hyperspell","nav":{"global":null,"anchors":[{"anchor":"Documentation","icon":"book","pages":[{"title":"Documentation","href":"/core/introduction"}]},{"anchor":"Integrations","icon":"circle-nodes","groups":[{"group":"Integrations","pages":[{"title":"Available Integrations","href":"/integrations/overview"},{"title":"ClickUp","href":"/integrations/all/clickup"},{"title":"Superhuman Docs","href":"/integrations/all/coda"},{"title":"Confluence","href":"/integrations/all/confluence"},{"title":"Dropbox","href":"/integrations/all/dropbox"},{"title":"Fathom","href":"/integrations/all/fathom"},{"title":"Fellow","href":"/integrations/all/fellow"},{"title":"Fireflies","href":"/integrations/all/fireflies"},{"title":"GitHub","href":"/integrations/all/github"},{"title":"GitLab","href":"/integrations/all/gitlab"},{"title":"Gong","href":"/integrations/all/gong"},{"title":"Google Calendar","href":"/integrations/all/google_calendar"},{"title":"Google Drive","href":"/integrations/all/google_drive"},{"title":"Gmail","href":"/integrations/all/google_mail"},{"title":"Granola","href":"/integrations/all/granola"},{"title":"HubSpot","href":"/integrations/all/hubspot"},{"title":"Jira","href":"/integrations/all/jira"},{"title":"Lightfield","href":"/integrations/all/lightfield"},{"title":"Linear","href":"/integrations/all/linear"},{"title":"Notion","href":"/integrations/all/notion"},{"title":"Microsoft Outlook","href":"/integrations/all/outlook"},{"title":"Pylon","href":"/integrations/all/pylon"},{"title":"Salesforce","href":"/integrations/all/salesforce"},{"title":"Slack","href":"/integrations/all/slack"},{"title":"Vault","description":"Store and search uploaded documents, saved notes, and agent work summaries","href":"/integrations/all/vault"},{"title":"Web Crawler","href":"/integrations/all/web_crawler"}]}]},{"anchor":"API Reference","icon":"terminal","pages":[{"title":"API Reference","href":"/api-reference/introduction"}]}]},"basePath":"","title":"Web Crawler - Hyperspell","buildId":"dpl_Fyv1NbzwP68g2bEH2Roy98uY1zmE","deploymentId":"dpl_Fyv1NbzwP68g2bEH2Roy98uY1zmE","dynamics":[],"chunks":[]}