> ## Documentation Index
> Fetch the complete documentation index at: https://docs.asteragents.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape URL Tool

> Extract content from web pages and PDFs for analysis and processing

## What it does

The Scrape URL tool extracts text content from web pages and PDF documents. Perfect for gathering information from websites, analyzing web content, or processing documents that your agents need to work with.

## Key features

* Extract content from any web page or PDF URL
* Return markdown, HTML, or just the page's links
* Strip navigation, headers, and footers so only the article body comes back
* Renders JavaScript-heavy pages before extracting
* Oversized pages are saved as an attachment instead of flooding the conversation

## Parameters

| Parameter            | Type    | Required | Description                                                                             |
| -------------------- | ------- | -------- | --------------------------------------------------------------------------------------- |
| `url`                | string  | Yes      | The URL of the webpage or PDF to scrape                                                 |
| `formats`            | array   | No       | Which outputs to return: any of `markdown`, `html`, `links` (default: `["markdown"]`)   |
| `onlyMainContent`    | boolean | No       | Drop headers, navs, and footers, keeping the main content (default: true)               |
| `removeBase64Images` | boolean | No       | Replace inline base64 images with a placeholder, keeping their alt text (default: true) |

## Common use cases

### Extract article content

```
url: "https://example.com/article"
```

Defaults already return clean markdown of the main content — no other parameters needed.

### Get the full page, chrome included

```
url: "https://example.com/data-page"
onlyMainContent: false
```

Use when the content you want lives in a sidebar, footer, or nav that the main-content
filter would strip.

### Process PDF documents

```
url: "https://example.com/document.pdf"
```

Extract text content from PDF files for document analysis.

### Get raw HTML for parsing

```
url: "https://example.com/page"
formats: ["html"]
```

Useful when you need the full HTML structure for custom processing. Pass
`["markdown", "html"]` to get both.

### Harvest links for a crawl

```
url: "https://example.com/index"
formats: ["links"]
```

Returns the page's outbound links so the agent can pick which ones to scrape next.

## Limitations

* Results over 100 KB are saved as a file attachment; the agent gets a \~10,000-character
  preview inline and reads the rest with [Read File](/tools/read_file)
* PDF extraction doesn't handle images or complex formatting
* Some dynamic content requiring user interaction may not be captured

## Troubleshooting

**"Failed to load page"**

* Check that the URL is accessible and valid
* Verify the website doesn't block automated access
* Try the URL in a browser to confirm it works

**"Content truncated"**

* The page exceeded 100 KB, so the full text was attached as a file
* Read it with [Read File](/tools/read_file) using the returned filename
* Or narrow the scrape with `onlyMainContent: true` (the default)

**"PDF extraction failed"**

* Ensure the URL points to a valid PDF file
* Some password-protected PDFs cannot be processed
* Try downloading and hosting the PDF elsewhere

## Related tools

* [Ask Web](/tools/ask_web) - Ask questions about web content using an LLM
* [Call API](/tools/call_api) - Make API calls to web services