> ## Documentation Index
> Fetch the complete documentation index at: https://docs.asteragents.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Extraction Schemas
> Define JSON schemas to automatically extract structured data from your documents
## Overview
Your AI agents can already search and read the full contents of every file in a knowledge base. Extraction schemas add a **structured index** on top of that—a consistent set of fields extracted from each document that enables quick filtering and cross-document analysis.
Think of it this way: without an extraction schema, answering "which contracts are worth over \$1M?" requires the AI to open and read every contract. With an extraction schema that includes `total_value` as a number field, that question becomes a simple filter.
**Extraction schemas are most valuable for:**
* **Numeric fields** you want to compare, aggregate, or filter (amounts, counts, percentages, dates)
* **Categorical fields** you want to group or filter by (document type, status, region, sector)
* **Key identifiers** that help locate the right documents quickly (deal names, parties, project codes)
String fields like summaries can be useful too, but the real power is in structured data that saves you from scanning every file when asking questions across your entire knowledge base.
## Choosing What to Extract
Before writing your schema, ask yourself: **"What questions do I want to answer across all my documents?"**
| Question Type | Extract These Fields |
| ---------------------------------------------- | ------------------------------------------ |
| "Which deals are over \$10M?" | `deal_value` (number) |
| "Show me all contracts expiring this quarter" | `expiration_date` (string in YYYY-MM-DD) |
| "Break down documents by type" | `document_type` (enum) |
| "Find all projects in the healthcare sector" | `sector` (enum) |
| "What's our average contract value by region?" | `contract_value` (number), `region` (enum) |
| "List all documents involving Acme Corp" | `parties` (array of strings) |
* Frequently used for filtering or sorting
* Needed for aggregations (sums, averages, counts)
* Used to categorize or group documents
* Key identifiers for finding specific documents
* Only useful when reading the full document (the AI can already do that)
* Highly variable or unstructured across documents
* Rarely needed for cross-document questions
## Schema Basics
Extraction schemas use [JSON Schema](https://json-schema.org/) format. At minimum, your schema needs three things:
A `type` field (usually `"object"`).
A `properties` field defining what to extract.
A `required` array listing mandatory fields.
```json theme={null}
{
"type": "object",
"required": ["title", "date", "summary"],
"properties": {
"title": { "type": "string", "description": "Document title as shown on the first page" },
"date": { "type": "string", "description": "Publication or effective date in YYYY-MM-DD format" },
"summary": { "type": "string", "description": "One paragraph summary of the document's main purpose" }
}
}
```
## Field Types & Structures
The reference below covers every supported field type and the structures you can compose from them. Expand each section as you need it.
### Supported Types
Use these types for your schema fields:
| Type | Description | Example Value |
| --------- | ----------------- | -------------------- |
| `string` | Text values | `"Quarterly Report"` |
| `number` | Decimal numbers | `123.45` |
| `integer` | Whole numbers | `42` |
| `boolean` | True/false values | `true` |
| `array` | Lists of items | `["item1", "item2"]` |
| `object` | Nested structures | `{"name": "John"}` |
Use a type array with `null` to make fields optional:
```json theme={null}
{
"type": "object",
"required": ["company_name"],
"properties": {
"company_name": { "type": "string", "description": "Legal company name" },
"website": { "type": ["string", "null"], "description": "Company website URL, if mentioned" },
"employee_count": { "type": ["integer", "null"], "description": "Number of employees, if stated" }
}
}
```
When a field uses `["type", "null"]`, the AI will return `null` if the information isn't found in the document.
Enums constrain a field to specific allowed values. **Use enums whenever you have a known set of categories**:
```json theme={null}
{
"type": "object",
"required": ["document_type", "status"],
"properties": {
"document_type": {
"type": "string",
"enum": ["CONTRACT", "INVOICE", "PROPOSAL", "REPORT"],
"description": "Classification of the document"
},
"status": {
"type": "string",
"enum": ["DRAFT", "PENDING", "APPROVED", "REJECTED"],
"description": "Current approval status"
},
"priority": {
"type": ["string", "null"],
"enum": ["LOW", "MEDIUM", "HIGH", null],
"description": "Priority level, if indicated"
}
}
}
```
Enums work with strings, numbers, booleans, or null values. Include `null` in the enum array if the field should be optional.
Use arrays to extract lists of items:
```json theme={null}
{
"type": "object",
"required": ["authors", "keywords"],
"properties": {
"authors": {
"type": "array",
"items": { "type": "string" },
"description": "Full names of all authors"
},
"keywords": {
"type": "array",
"items": { "type": "string" },
"description": "Key topics or tags relevant to this document"
}
}
}
```
**Arrays of objects** — extract structured lists with typed objects:
```json theme={null}
{
"type": "object",
"required": ["line_items"],
"properties": {
"line_items": {
"type": "array",
"description": "Individual items or charges listed in the document",
"items": {
"type": "object",
"required": ["description", "amount"],
"properties": {
"description": { "type": "string", "description": "Item or service description" },
"quantity": { "type": ["integer", "null"], "description": "Number of units, if specified" },
"amount": { "type": "number", "description": "Line item total in USD" }
}
}
}
}
}
```
Group related fields using nested objects:
```json theme={null}
{
"type": "object",
"required": ["vendor", "total_amount"],
"properties": {
"vendor": {
"type": "object",
"description": "Information about the vendor or supplier",
"required": ["name"],
"properties": {
"name": { "type": "string", "description": "Vendor's legal business name" },
"address": { "type": ["string", "null"], "description": "Full mailing address" },
"contact_email": { "type": ["string", "null"], "description": "Primary contact email" }
}
},
"total_amount": { "type": "number", "description": "Total amount due in USD" }
}
}
```
## Best Practices
Mark fields as required unless they genuinely may not exist in documents. This helps the AI understand which information is essential:
```json theme={null}
{
"required": ["title", "date", "amount"],
"properties": { ... }
}
```
Whenever you have a known set of possible values, use enums instead of free-form strings:
```json theme={null}
// Good - constrained values
"status": { "type": "string", "enum": ["ACTIVE", "INACTIVE", "PENDING"] }
// Avoid - unconstrained
"status": { "type": "string" }
```
Prefer one or two levels of nesting to group related fields logically:
```json theme={null}
{
"deal_name": { "type": "string" },
"deal_type": { "type": "string" },
"buyer": {
"type": "object",
"properties": {
"name": { "type": "string" },
"contact_email": { "type": "string" }
}
}
}
```
The `description` field is your primary way to guide the extraction AI. Think of descriptions as instructions telling the AI exactly how to interpret each field when analyzing the document:
```json theme={null}
{
"type": "object",
"required": ["effective_date", "total_value", "parties"],
"properties": {
"effective_date": {
"type": "string",
"description": "Contract start date in YYYY-MM-DD format"
},
"total_value": {
"type": "number",
"description": "Total contract value in USD, excluding taxes and fees"
},
"renewal_terms": {
"type": ["string", "null"],
"description": "Summary of auto-renewal clauses. Return null if no auto-renewal provisions exist."
},
"parties": {
"type": "array",
"items": { "type": "string" },
"description": "Legal names of all signing parties, extracted exactly as written in the signature block"
}
}
}
```
**Use descriptions to specify:**
* **Date and number formats**: `"YYYY-MM-DD"`, `"in USD"`, `"as a decimal percentage (e.g., 0.15 for 15%)"`
* **What to include or exclude**: `"excluding taxes"`, `"only from the executive summary section"`
* **How to handle missing data**: `"Return null if not explicitly stated"`
* **Where to look**: `"from the signature block"`, `"as stated in the header"`
* **Level of detail**: `"one-sentence summary"`, `"verbatim quote"`, `"3-5 bullet points"`
Well-written descriptions dramatically improve extraction accuracy.
Before finalizing your schema, test it against several representative documents—not just one. A schema that works perfectly for one document may fail on others.
Testing across examples helps you identify:
* Fields that don't exist in all documents (should be optional with `null`)
* Categories that vary more than expected (broaden your enums)
* Format inconsistencies (clarify your descriptions)
* Edge cases you didn't anticipate
The goal is a schema that works generically across all documents you'll upload to the knowledge base, not one optimized for a single example.
## Example Schemas
These are complete, copy-ready schemas for common document types. Switch tabs to see each one.
```json Invoice theme={null}
{
"type": "object",
"required": ["invoice_number", "vendor_name", "total_amount", "line_items"],
"properties": {
"invoice_number": { "type": "string" },
"invoice_date": { "type": ["string", "null"] },
"due_date": { "type": ["string", "null"] },
"vendor_name": { "type": "string" },
"vendor_address": { "type": ["string", "null"] },
"total_amount": { "type": "number" },
"currency": {
"type": "string",
"enum": ["USD", "EUR", "GBP", "CAD"]
},
"line_items": {
"type": "array",
"items": {
"type": "object",
"required": ["description", "amount"],
"properties": {
"description": { "type": "string" },
"quantity": { "type": ["number", "null"] },
"unit_price": { "type": ["number", "null"] },
"amount": { "type": "number" }
}
}
}
}
}
```
```json Research Paper theme={null}
{
"type": "object",
"required": ["title", "authors", "abstract", "methodology"],
"properties": {
"title": { "type": "string" },
"authors": {
"type": "array",
"items": { "type": "string" }
},
"publication_date": { "type": ["string", "null"] },
"journal": { "type": ["string", "null"] },
"abstract": { "type": "string" },
"methodology": { "type": "string" },
"key_findings": {
"type": "array",
"items": { "type": "string" }
},
"limitations": { "type": ["string", "null"] },
"sample_size": { "type": ["integer", "null"] }
}
}
```
```json Contract theme={null}
{
"type": "object",
"required": ["contract_type", "parties", "effective_date", "key_terms"],
"properties": {
"contract_type": {
"type": "string",
"enum": ["NDA", "MSA", "SOW", "EMPLOYMENT", "LEASE", "OTHER"]
},
"parties": {
"type": "array",
"items": {
"type": "object",
"required": ["name", "role"],
"properties": {
"name": { "type": "string" },
"role": {
"type": "string",
"enum": ["PARTY_A", "PARTY_B", "GUARANTOR", "WITNESS"]
}
}
}
},
"effective_date": { "type": "string" },
"expiration_date": { "type": ["string", "null"] },
"auto_renewal": { "type": ["boolean", "null"] },
"total_value": { "type": ["number", "null"] },
"key_terms": {
"type": "array",
"items": {
"type": "object",
"required": ["term", "description"],
"properties": {
"term": { "type": "string" },
"description": { "type": "string" }
}
}
}
}
}
```
## Schema Depth: Index vs. Full Extraction
When designing your schema, consider how your agent will actually use the data.
Best when agents need to **find and filter documents**, then read the full content for detail:
```json theme={null}
{
"type": "object",
"required": ["vendor_name", "total_amount", "invoice_date"],
"properties": {
"vendor_name": { "type": "string" },
"total_amount": { "type": "number" },
"invoice_date": { "type": "string" }
}
}
```
The agent loads `extracted_data.json` to find the right files, then reads their `.md` content for line items, notes, and other detail. This is simpler to maintain and works well when documents vary in structure.
Best when agents need to **process many documents without reading raw content** — e.g., spreading financials, building comparison tables, or aggregating metrics across hundreds of files:
```json theme={null}
{
"type": "object",
"required": ["vendor_name", "total_amount", "line_items"],
"properties": {
"vendor_name": { "type": "string" },
"total_amount": { "type": "number" },
"line_items": {
"type": "array",
"items": {
"type": "object",
"properties": {
"description": { "type": "string" },
"quantity": { "type": ["number", "null"] },
"unit_price": { "type": ["number", "null"] },
"amount": { "type": "number" }
}
}
}
}
}
```
With a deep schema, the agent can build its entire output from `extracted_data.json` alone — no document reading needed. This is faster and more reliable for bulk operations, but requires more upfront schema design and re-extraction when the schema changes.
### How to decide
| Factor | Summary schema | Deep schema |
| ---------------------------------- | ----------------------------------- | ------------------------- |
| Agent reads 1-5 files per task | Good fit | Overkill |
| Agent processes 10+ files per task | Bottleneck — agent reads every file | Good fit |
| Document structure varies a lot | Easier to maintain | Harder to design |
| Speed/timeout matters | Agent still needs file reads | Single JSON load |
| Schema changes frequently | Low re-extraction cost | Must re-extract all files |
**Start summary, go deep where it hurts.** Begin with a summary-level schema for filtering. If you notice your agent spending most of its time reading files to extract the same fields repeatedly, promote those fields into the extraction schema.
## Setting Up Extraction
Go to **Control Hub** → **Knowledge Bases**.
Create a new knowledge base or edit an existing one.
Under **Structured Data Extraction**:
* Select an **Extraction Model** (e.g., GPT-4o)
* Paste your JSON schema into the **Extraction Schema** field
Save and upload your documents.
The schema only applies to **newly uploaded files**. To re-extract data from existing files, use the retry extraction option in the file details.
## Viewing Extracted Data
After processing completes:
Click on a file in your knowledge base.
View the **Extracted Data** section. The structured JSON output matches your schema definition.
You can also access extracted data programmatically through agents using the `search_knowledge_base` tool.
## Troubleshooting
Your schema isn't valid JSON. Check for:
* Missing commas between properties
* Missing closing braces `}`
* Trailing commas (not allowed in JSON)
Every schema needs a root `type`. Usually this should be `"object"`:
```json theme={null}
{
"type": "object",
...
}
```
* Ensure the field is marked as `required` if it should always be extracted
* Check that your field names clearly describe what to extract
* Verify the document actually contains the information
Large schemas with many fields take longer to process. Consider:
* Breaking into multiple smaller knowledge bases
* Removing non-essential fields
* Using a faster extraction model