Crawl a website

Crawl a website

Follows links from a starting URL and returns the content of every page it reaches, within the limits you set.

The API starts a job and returns its id; read the pages with GET /crawl/{id}. The MCP tool polls the job and returns once the crawl has finished. Set a limit and include or exclude paths to keep a crawl bounded.

Set up

Use the first option your agent supports.

MCP (official, remote)

Add this server to your agent's MCP settings, then sign in when asked.

{ "mcpServers": { "firecrawl": { "url": "https://mcp.firecrawl.dev/v2/mcp-oauth" } } }

Call the MCP tool firecrawl_crawl.

Server URL: https://mcp.firecrawl.dev/v2/mcp-oauth

CLI (official)

Install the command, then confirm it runs.

npm install -g firecrawl-cli
firecrawl --version

Run firecrawl crawl.

Set $FIRECRAWL_API_KEY in your environment first (get a key: https://www.firecrawl.dev/app/api-keys).

API (official)

• Base URL: https://api.firecrawl.dev/v2
• Endpoint: POST /crawl
• Auth: send the header Authorization: Bearer $FIRECRAWL_API_KEY
• Get a key: https://www.firecrawl.dev/app/api-keys
• Docs: https://docs.firecrawl.dev/api-reference/introduction

Before doing anything else, make one read-only call to confirm access.

Rules

• Ask the user before anything that sends messages, costs money, or changes data.
• Never print API keys.