Skip to main content

​
What it does

The Scrape URL tool extracts text content from web pages and PDF documents. Perfect for gathering information from websites, analyzing web content, or processing documents that your agents need to work with.

​
Key features

  • Extract content from any web page or PDF URL
  • Return markdown, HTML, or just the page’s links
  • Strip navigation, headers, and footers so only the article body comes back
  • Renders JavaScript-heavy pages before extracting
  • Oversized pages are saved as an attachment instead of flooding the conversation

​
Parameters

​
Common use cases

​
Extract article content

Defaults already return clean markdown of the main content — no other parameters needed.

​
Get the full page, chrome included

Use when the content you want lives in a sidebar, footer, or nav that the main-content filter would strip.

​
Process PDF documents

Extract text content from PDF files for document analysis.

​
Get raw HTML for parsing

Useful when you need the full HTML structure for custom processing. Pass ["markdown", "html"] to get both.
Returns the page’s outbound links so the agent can pick which ones to scrape next.

​
Limitations

  • Results over 100 KB are saved as a file attachment; the agent gets a ~10,000-character preview inline and reads the rest with Read File
  • PDF extraction doesn’t handle images or complex formatting
  • Some dynamic content requiring user interaction may not be captured

​
Troubleshooting

“Failed to load page”
  • Check that the URL is accessible and valid
  • Verify the website doesn’t block automated access
  • Try the URL in a browser to confirm it works
“Content truncated”
  • The page exceeded 100 KB, so the full text was attached as a file
  • Read it with Read File using the returned filename
  • Or narrow the scrape with onlyMainContent: true (the default)
“PDF extraction failed”
  • Ensure the URL points to a valid PDF file
  • Some password-protected PDFs cannot be processed
  • Try downloading and hosting the PDF elsewhere
  • Ask Web - Ask questions about web content using an LLM
  • Call API - Make API calls to web services