> ## Documentation Index
> Fetch the complete documentation index at: https://docs.graphorlm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Ingest Sources

> Upload sources (files, web pages, GitHub, YouTube) to your project via the Graphor REST API

This page documents Graphor's **source ingestion** endpoints. All ingestion is **asynchronous**: you send a request, receive a **build ID** immediately, and then poll the **build status** endpoint until processing completes. Use these endpoints to add content to your project — whether that content is a **local file**, a **public web page URL**, a **public GitHub repository**, or a **public YouTube video**.

## Endpoints

<CardGroup cols={2}>
  <Card title="Get build status" icon="circle-info" href="/api-reference/sources/upload#get-build-status">
    **GET** `https://sources.graphorlm.com/builds/{build_id}`

    Poll status and optional parsed elements for an async ingestion
  </Card>

  <Card title="Ingest file" icon="arrow-up-from-bracket" href="/api-reference/sources/upload#ingest-file">
    **POST** `https://sources.graphorlm.com/ingest-file`

    Upload a local file; processing runs in the background
  </Card>

  <Card title="Ingest URL" icon="link" href="/api-reference/sources/upload#ingest-url">
    **POST** `https://sources.graphorlm.com/ingest-url`

    Ingest a public web page by URL (async)
  </Card>

  <Card title="Ingest GitHub" icon="code-branch" href="/api-reference/sources/upload#ingest-github">
    **POST** `https://sources.graphorlm.com/ingest-github`

    Ingest a public GitHub repository (async)
  </Card>

  <Card title="Ingest YouTube" icon="video" href="/api-reference/sources/upload#ingest-youtube">
    **POST** `https://sources.graphorlm.com/ingest-youtube`

    Ingest a public YouTube video (async)
  </Card>
</CardGroup>

## Authentication

All endpoints on this page require authentication using an API token. Include your API token as a Bearer token in the `Authorization` header.

<Note>
  Learn how to create and manage API tokens in the [API Tokens guide](/guides/api-tokens).
</Note>

## Async ingestion flow

1. **Call one of the ingest endpoints** (file, URL, GitHub, or YouTube). The request is validated and the job is scheduled; the response returns immediately with a **`build_id`**.
2. **Poll GET `/builds/{build_id}`** to check status. When `status` is `Completed`, the source is ready; when `status` indicates failure, check the `error` field.
3. **Use the returned `file_id`** (once the build has completed) for subsequent API calls (ask, extract, retrieve, delete, etc.).

<Note>
  The **Get build status** endpoint can also return paginated **parsed elements** (chunks) for a completed build when you do not set `suppress_elements=true`.
</Note>

## Get build status

Use this endpoint to poll the result of an async ingestion (or re-process). The **build\_id** is returned by:

* **POST** `/ingest-file`
* **POST** `/ingest-url`
* **POST** `/ingest-github`
* **POST** `/ingest-youtube`
* **POST** `/reprocess` (re-process)

### Endpoint overview

<CardGroup cols={2}>
  <Card title="HTTP Method" icon="arrow-up">
    **GET**
  </Card>

  <Card title="Endpoint URL" icon="link">
    **[https://sources.graphorlm.com/builds/\{build\_id}](https://sources.graphorlm.com/builds/\{build_id})**
  </Card>
</CardGroup>

### Path parameter

| Parameter  | Type   | Description                                              |
| ---------- | ------ | -------------------------------------------------------- |
| `build_id` | string | The build identifier returned when the job was scheduled |

### Query parameters

| Parameter             | Type    | Default | Description                                               |
| --------------------- | ------- | ------- | --------------------------------------------------------- |
| `suppress_elements`   | boolean | `false` | When `true`, elements are omitted from the response       |
| `suppress_img_base64` | boolean | `false` | When `true`, `img_base64` is omitted from each element    |
| `page`                | integer | —       | 1-based page number (use with `page_size` for pagination) |
| `page_size`           | integer | —       | Number of elements per page (max 100)                     |

### Success response (200 OK)

When the build has been persisted (history exists), the response includes status and optional metadata:

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "Completed",
  "success": true,
  "file_id": "f47ac10b-58cc-4372-a567-0e02b2c3d479",
  "file_name": "report.pdf",
  "error": null,
  "method": "balanced",
  "total_partitions": 42,
  "total_pages": 10,
  "created_at": "2025-03-07T12:00:00Z",
  "updated_at": "2025-03-07T12:01:30Z",
  "message": null,
  "elements": null,
  "total_elements": null,
  "page": null,
  "page_size": null,
  "total_pages_elements": null
}
```

When the build is **pending** (request received but build has not started yet):

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "Pending",
  "success": false,
  "file_id": null,
  "file_name": null,
  "error": null,
  "message": "Build is pending; processing has not started yet"
}
```

When the build is still in progress (running):

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "status": "Processing",
  "success": false,
  "file_id": null,
  "file_name": null,
  "error": null,
  "message": "Build not found or not yet persisted"
}
```

### Response fields

| Field                  | Type            | Description                                                                                      |
| ---------------------- | --------------- | ------------------------------------------------------------------------------------------------ |
| `build_id`             | string          | The requested build identifier                                                                   |
| `status`               | string          | `Pending`, `Processing`, `Completed`, `Processing failed`, or `not_found` when no history exists |
| `success`              | boolean         | `true` only when `status` is `Completed`                                                         |
| `file_id`              | string \| null  | Source file ID; present when the build has been persisted                                        |
| `file_name`            | string \| null  | Display name of the source; present when persisted                                               |
| `error`                | string \| null  | Error message from the pipeline when the build failed                                            |
| `method`               | string \| null  | Strategy used (e.g. `auto`, `fast`, `balanced`, `accurate`, `agentic`)                           |
| `total_partitions`     | integer \| null | Number of partitions; present when history exists                                                |
| `total_pages`          | integer \| null | Total pages in the source; present when history exists                                           |
| `created_at`           | string \| null  | ISO8601 timestamp when the build was created                                                     |
| `updated_at`           | string \| null  | ISO8601 timestamp when the build was last updated                                                |
| `message`              | string \| null  | Human-readable message (e.g. when status is `not_found`)                                         |
| `elements`             | array \| null   | Parsed elements (chunks) when `suppress_elements=false` and build completed                      |
| `total_elements`       | integer \| null | Total number of elements (when elements are returned)                                            |
| `page`                 | integer \| null | Current page of elements (1-based) when pagination is used                                       |
| `page_size`            | integer \| null | Elements per page when pagination is used                                                        |
| `total_pages_elements` | integer \| null | Total pages of elements when pagination is used                                                  |

### Code example: poll until complete

Poll until `success` is `true`. While `status` is `Pending` (request received, build not started) or `Processing`, keep polling. Only treat `Processing failed` or a non-null `error` (when status is not `not_found`) as failure.

```javascript theme={null}
const pollBuildStatus = async (apiToken, buildId, options = {}) => {
  const { intervalMs = 2000, maxAttempts = 120 } = options;
  const url = `https://sources.graphorlm.com/builds/${buildId}`;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const response = await fetch(url, {
      headers: { Authorization: `Bearer ${apiToken}` },
    });
    const data = await response.json();

    if (data.success) return data;
    if (data.status === "Processing failed" || (data.error && data.status !== "not_found"))
      throw new Error(data.error || data.message);

    await new Promise((r) => setTimeout(r, intervalMs));
  }

  throw new Error("Polling timed out");
};
```

```python theme={null}
import time
import requests

def poll_build_status(api_token, build_id, interval_seconds=2, max_attempts=120):
    url = f"https://sources.graphorlm.com/builds/{build_id}"
    headers = {"Authorization": f"Bearer {api_token}"}

    for _ in range(max_attempts):
        response = requests.get(url, headers=headers)
        data = response.json()

        if data.get("success"):
            return data
        if data.get("status") == "Processing failed" or (data.get("error") and data.get("status") != "not_found"):
            raise RuntimeError(data.get("error") or data.get("message", "Build failed"))

        time.sleep(interval_seconds)

    raise TimeoutError("Polling timed out")
```

***

## Ingest file

Upload a local file and schedule ingestion in the background. The API validates size (max 100 MB) and extension, stores the file, then runs the full pipeline (partitioning, chunking, embedding) asynchronously.

### Endpoint overview

<CardGroup cols={2}>
  <Card title="HTTP Method" icon="arrow-up">
    **POST**
  </Card>

  <Card title="Endpoint URL" icon="link">
    **[https://sources.graphorlm.com/ingest-file](https://sources.graphorlm.com/ingest-file)**
  </Card>
</CardGroup>

### Request format

#### Headers

| Header          | Value                   | Required |
| --------------- | ----------------------- | -------- |
| `Authorization` | `Bearer YOUR_API_TOKEN` | Yes      |
| `Content-Type`  | `multipart/form-data`   | Yes      |

#### Request body (multipart/form-data)

| Field    | Type   | Description                                                                                                           | Required |
| -------- | ------ | --------------------------------------------------------------------------------------------------------------------- | -------- |
| `file`   | File   | The document file to upload                                                                                           | Yes      |
| `method` | string | Processing method: `auto`, `fast`, `balanced`, `accurate`, or `agentic` (see [Partition methods](#partition-methods)) | No       |

### Partition methods

When provided, `partition_method` controls how the document is parsed. If omitted, the system default is used.

| Value      | Name     | Description                                                                                                                                               |
| ---------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auto`     | Auto     | Per-page routing: classifies each page and runs the cheapest parser that fits. Best for mixed PDFs (body + tables + scans in one document). **PDF-only.** |
| `fast`     | Fast     | Fast processing with heuristic classification. No OCR.                                                                                                    |
| `balanced` | Balanced | OCR-based extraction with structure classification.                                                                                                       |
| `accurate` | Accurate | Fine-tuned model for highest accuracy (Premium).                                                                                                          |
| `agentic`  | Agentic  | Highest accuracy for complex layouts, tables, and diagrams.                                                                                               |

<Note>
  For more details, see the [Process Source](/api-reference/sources/process) documentation.
</Note>

### File requirements

<AccordionGroup>
  <Accordion icon="file-text" title="Supported file types">
    **Documents**: PDF, DOC, DOCX, ODT, TXT, TEXT, MD, HTML, HTM\
    **Presentations**: PPT, PPTX\
    **Spreadsheets**: CSV, TSV, XLS, XLSX\
    **Images**: PNG, JPG, JPEG, TIFF, BMP, HEIC\
    **Audio**: MP3, WAV, M4A, OGG, FLAC\
    **Video**: MP4, MOV, AVI, MKV, WEBM
  </Accordion>

  <Accordion icon="weight-scale" title="File size limits">
    **Maximum file size**: 100 MB per file.\
    The request must include a `Content-Length` header so the server can enforce the limit.
  </Accordion>

  <Accordion icon="file-signature" title="File name requirements">
    The file must have a valid filename with extension; the extension determines allowed processing.
  </Accordion>
</AccordionGroup>

### Success response (200 OK)

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "success": true,
  "error": null
}
```

### Response fields

| Field      | Type           | Description                                                 |
| ---------- | -------------- | ----------------------------------------------------------- |
| `build_id` | string         | Use this ID to poll [Get build status](#get-build-status)   |
| `success`  | boolean        | Whether the request was successfully scheduled              |
| `error`    | string \| null | Error message if the request was not scheduled successfully |

### Code examples

#### JavaScript/Node.js

```javascript theme={null}
const ingestFile = async (apiToken, filePath) => {
  const formData = new FormData();
  const fileStream = fs.createReadStream(filePath);
  formData.append("file", fileStream);

  const response = await fetch("https://sources.graphorlm.com/ingest-file", {
    method: "POST",
    headers: { Authorization: `Bearer ${apiToken}` },
    body: formData,
  });

  if (!response.ok) throw new Error(`Ingest failed: ${response.status} ${response.statusText}`);
  const { build_id } = await response.json();
  return build_id;
};

// Usage: get build_id, then poll get_build_status until success
const buildId = await ingestFile("grlm_your_api_token_here", "./document.pdf");
console.log("Build ID:", buildId);
```

#### Python

```python theme={null}
import requests

def ingest_file(api_token, file_path, partition_method=None):
    url = "https://sources.graphorlm.com/ingest-file"
    headers = {"Authorization": f"Bearer {api_token}"}
    with open(file_path, "rb") as f:
        files = {"file": (file_path, f)}
        data = {}
        if partition_method:
            data["method"] = partition_method
        response = requests.post(url, headers=headers, files=files, data=data or None, timeout=300)
    response.raise_for_status()
    return response.json()["build_id"]

# Usage
build_id = ingest_file("grlm_your_api_token_here", "document.pdf")
print("Build ID:", build_id)
```

#### cURL

```bash theme={null}
curl -X POST https://sources.graphorlm.com/ingest-file \
  -H "Authorization: Bearer grlm_your_api_token_here" \
  -F "file=@document.pdf"
```

#### cURL with partition method

```bash theme={null}
curl -X POST https://sources.graphorlm.com/ingest-file \
  -H "Authorization: Bearer grlm_your_api_token_here" \
  -F "file=@document.pdf" \
  -F "method=balanced"
```

### Error responses

| Status Code | Description                                |
| ----------- | ------------------------------------------ |
| `400`       | Unsupported file type or missing file name |
| `411`       | Missing Content-Length header              |
| `413`       | File exceeds 100 MB limit                  |
| `500`       | Internal server error                      |

Example error body:

```json theme={null}
{
  "detail": "File type 'exe' is not supported. Allowed types: csv, doc, docx, pdf, txt, ..."
}
```

```json theme={null}
{
  "detail": "File size exceeds the maximum allowed limit of 100MB"
}
```

***

## Ingest URL

Ingest a web page (or multiple pages via crawling) as a source. The job runs in the background; use the returned **build\_id** to poll [Get build status](#get-build-status). If the URL points to a downloadable file (by extension or Content-Type), the file is downloaded and then processed in the background.

### Endpoint overview

<CardGroup cols={2}>
  <Card title="HTTP Method" icon="arrow-up">
    **POST**
  </Card>

  <Card title="Endpoint URL" icon="link">
    **[https://sources.graphorlm.com/ingest-url](https://sources.graphorlm.com/ingest-url)**
  </Card>
</CardGroup>

### Request format

#### Headers

| Header          | Value                   | Required |
| --------------- | ----------------------- | -------- |
| `Authorization` | `Bearer YOUR_API_TOKEN` | Yes      |
| `Content-Type`  | `application/json`      | Yes      |

#### Request body (JSON)

| Field       | Type    | Description                                                                                                                                                        | Required              |
| ----------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------- |
| `url`       | string  | The web page URL to ingest                                                                                                                                         | Yes                   |
| `crawlUrls` | boolean | When `true`, follow and ingest links found on the page (ignored when URL resolves to a file)                                                                       | No (default: `false`) |
| `method`    | string  | One of: `auto`, `fast`, `balanced`, `accurate`, `agentic`. `auto` is PDF-only and only effective when the URL resolves to a PDF; non-PDF URLs fall back to `fast`. | No                    |

### Success response (200 OK)

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "success": true,
  "error": null
}
```

### Code examples

#### JavaScript/Node.js

```javascript theme={null}
const ingestUrl = async (apiToken, url, crawlUrls = false) => {
  const response = await fetch("https://sources.graphorlm.com/ingest-url", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url, crawlUrls }),
  });
  if (!response.ok) throw new Error(`Ingest URL failed: ${response.status}`);
  const { build_id } = await response.json();
  return build_id;
};
```

#### Python

```python theme={null}
import requests

def ingest_url(api_token, url, crawl_urls=False, partition_method=None):
    payload = {"url": url, "crawlUrls": crawl_urls}
    if partition_method:
        payload["method"] = partition_method
    response = requests.post(
        "https://sources.graphorlm.com/ingest-url",
        headers={"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"},
        json=payload,
        timeout=300,
    )
    response.raise_for_status()
    return response.json()["build_id"]
```

#### cURL

```bash theme={null}
curl -X POST https://sources.graphorlm.com/ingest-url \
  -H "Authorization: Bearer grlm_your_api_token_here" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/","crawlUrls":false}'
```

### Error responses

| Status Code | Description                                    |
| ----------- | ---------------------------------------------- |
| `400`       | Unsupported file type detected from a file URL |
| `500`       | Internal server error during URL processing    |

<Note>
  To ingest local files (PDF, DOCX, etc.), use [Ingest file](#ingest-file).
</Note>

***

## Ingest GitHub

Ingest a public GitHub repository as a source. Processing runs in the background; use the returned **build\_id** with [Get build status](#get-build-status).

### Endpoint overview

<CardGroup cols={2}>
  <Card title="HTTP Method" icon="arrow-up">
    **POST**
  </Card>

  <Card title="Endpoint URL" icon="link">
    **[https://sources.graphorlm.com/ingest-github](https://sources.graphorlm.com/ingest-github)**
  </Card>
</CardGroup>

### Request format

#### Headers

| Header          | Value                   | Required |
| --------------- | ----------------------- | -------- |
| `Authorization` | `Bearer YOUR_API_TOKEN` | Yes      |
| `Content-Type`  | `application/json`      | Yes      |

#### Request body (JSON)

| Field | Type   | Description                                                  | Required |
| ----- | ------ | ------------------------------------------------------------ | -------- |
| `url` | string | GitHub repository URL (e.g. `https://github.com/owner/repo`) | Yes      |

### Success response (200 OK)

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "success": true,
  "error": null
}
```

### Code examples

#### JavaScript/Node.js

```javascript theme={null}
const ingestGithub = async (apiToken, repoUrl) => {
  const response = await fetch("https://sources.graphorlm.com/ingest-github", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url: repoUrl }),
  });
  if (!response.ok) throw new Error(`GitHub ingest failed: ${response.status}`);
  return (await response.json()).build_id;
};
```

#### Python

```python theme={null}
import requests

def ingest_github(api_token, repo_url):
    response = requests.post(
        "https://sources.graphorlm.com/ingest-github",
        headers={"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"},
        json={"url": repo_url},
        timeout=300,
    )
    response.raise_for_status()
    return response.json()["build_id"]
```

#### cURL

```bash theme={null}
curl -X POST https://sources.graphorlm.com/ingest-github \
  -H "Authorization: Bearer grlm_your_api_token_here" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://github.com/owner/repo"}'
```

### Error responses

| Status Code | Description                                    |
| ----------- | ---------------------------------------------- |
| `500`       | Internal server error during GitHub processing |

<Note>
  Only **public** repositories are supported.
</Note>

***

## Ingest YouTube

Ingest a public YouTube video (transcript/captions) as a source. Processing runs in the background; use the returned **build\_id** with [Get build status](#get-build-status).

### Endpoint overview

<CardGroup cols={2}>
  <Card title="HTTP Method" icon="arrow-up">
    **POST**
  </Card>

  <Card title="Endpoint URL" icon="link">
    **[https://sources.graphorlm.com/ingest-youtube](https://sources.graphorlm.com/ingest-youtube)**
  </Card>
</CardGroup>

### Request format

#### Headers

| Header          | Value                   | Required |
| --------------- | ----------------------- | -------- |
| `Authorization` | `Bearer YOUR_API_TOKEN` | Yes      |
| `Content-Type`  | `application/json`      | Yes      |

#### Request body (JSON)

| Field | Type   | Description                                                    | Required |
| ----- | ------ | -------------------------------------------------------------- | -------- |
| `url` | string | YouTube video URL (e.g. `https://www.youtube.com/watch?v=...`) | Yes      |

### Success response (200 OK)

```json theme={null}
{
  "build_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "success": true,
  "error": null
}
```

### Code examples

#### JavaScript/Node.js

```javascript theme={null}
const ingestYoutube = async (apiToken, videoUrl) => {
  const response = await fetch("https://sources.graphorlm.com/ingest-youtube", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${apiToken}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url: videoUrl }),
  });
  if (!response.ok) throw new Error(`YouTube ingest failed: ${response.status}`);
  return (await response.json()).build_id;
};
```

#### Python

```python theme={null}
import requests

def ingest_youtube(api_token, video_url):
    response = requests.post(
        "https://sources.graphorlm.com/ingest-youtube",
        headers={"Authorization": f"Bearer {api_token}", "Content-Type": "application/json"},
        json={"url": video_url},
        timeout=300,
    )
    response.raise_for_status()
    return response.json()["build_id"]
```

#### cURL

```bash theme={null}
curl -X POST https://sources.graphorlm.com/ingest-youtube \
  -H "Authorization: Bearer grlm_your_api_token_here" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.youtube.com/watch?v=VIDEO_ID"}'
```

### Error responses

| Status Code | Description                                     |
| ----------- | ----------------------------------------------- |
| `500`       | Internal server error during YouTube processing |

<Note>
  The video must be **public**; transcripts/captions are downloaded and processed in the background.
</Note>

***

## Best practices

* **Poll with backoff**: When polling [Get build status](#get-build-status), use a reasonable interval (e.g. 2–5 seconds) and a timeout to avoid tight loops.
* **Store `file_id`**: Once the build completes (`success: true`), store `file_id` for use with ask, extract, retrieve, delete, and list elements.
* **Validate before upload**: Check file type and size client-side before calling [Ingest file](#ingest-file).
* **Protect API tokens**: Never expose tokens in client-side code or public repositories; use HTTPS only.

***

## Next steps

After ingestion completes (build status `Completed`):

<CardGroup cols={2}>
  <Card title="Parse source" icon="gears" href="/api-reference/sources/process">
    Re-process a source with a different partition method (async; returns a new build\_id)
  </Card>

  <Card title="List sources" icon="list" href="/api-reference/sources/list">
    List all sources in your project
  </Card>

  <Card title="Get elements" icon="file-dashed-line" href="/api-reference/sources/list-elements">
    Retrieve parsed elements (chunks) for a source
  </Card>

  <Card title="Delete source" icon="trash" href="/api-reference/sources/delete">
    Remove a source from your project
  </Card>
</CardGroup>