Ingestion and reconciliation

Ingestion converts uploaded files into derived representations (search index, knowledge-graph nodes, organized-file outputs). This is the only call that makes files searchable. Uploading a file registers it in the ledger; ingesting it builds the representations that queries and tools read.

Ingest

POST /v1/workspaces/{workspace_id}/ingestions

Queue an ingestion run for one file, an explicit batch, or an entire folder. The ingestion lane (PDF, spreadsheet, audio, …) is chosen per file from its type — callers do not name a lane. Query parameters:

wait_seconds (optional, default 0, max 300)

Headers:

Idempotency-Key (optional)

Request body:

{
  "selection": {
    "type": "path_prefix",
    "path_prefix": "reports/",
    "stale_only": false
  },
  "reingest_unchanged": false
}

Body fields:

selection (required) — What to process. Discriminated on type:
- {"type": "file_ids", "file_ids": ["<uuid>", ...]} — Explicit list of up to 1000 files.
- {"type": "path_prefix", "path_prefix": "reports/", "stale_only": false} — All files under a folder prefix. "" or "/" means the whole workspace. Set stale_only: true to re-ingest only files whose observed hash diverged from the ingested hash.
reingest_unchanged (optional, default false) — Force re-ingestion even for files already ingested at the same content hash (normally a no-cost skip).

Response (202 Accepted): A Job with kind: "ingestion". Result (on success):

{
  "result_type": "ingestion",
  "outcomes": [
    {
      "file_id": "550e8400-e29b-41d4-a716-446655440004",
      "path": "reports/annual-report.pdf",
      "status": "ingested",
      "lane": "document",
      "units": 96,
      "error": null
    },
    {
      "file_id": "550e8400-e29b-41d4-a716-446655440006",
      "path": "reports/unchanged-report.pdf",
      "status": "skipped_unchanged",
      "lane": "document",
      "units": 0,
      "error": null
    },
    {
      "file_id": "550e8400-e29b-41d4-a716-446655440005",
      "path": "reports/draft.pdf",
      "status": "failed",
      "lane": "document",
      "units": 0,
      "error": {
        "code": "corrupt_file",
        "message": "...",
        "detail": null,
        "retryable": false,
        "request_id": "req-01j9..."
      }
    }
  ],
  "files_ingested": 1,
  "files_skipped": 1,
  "files_failed": 1,
  "units_total": 96,
  "kg_build_job_id": "550e8400-e29b-41d4-a716-446655440008"
}

Per-file outcomes are independent: one corrupt file in a folder of 400 does not fail the run — it appears as a failed entry. The job status is succeeded even when some files failed. Per-file status values: "ingested", "skipped_unchanged" (same content hash — no-cost skip), "failed". kg_build_job_id is set only when the workspace has opted into knowledge_graph.auto_build (off by default); poll it to know when the graph covers the newly ingested files. Otherwise it is null and the graph is built when you ask. Example (ingest two explicit files):

curl -X POST "$NDI_BASE_URL/v1/workspaces/$WS/ingestions" \
  -H "X-API-Key: $NDI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "selection": {
      "type": "file_ids",
      "file_ids": [
        "550e8400-e29b-41d4-a716-446655440004",
        "550e8400-e29b-41d4-a716-446655440005"
      ]
    }
  }'

Reconcile

POST /v1/workspaces/{workspace_id}/reconciliations

Detect divergence between storage and the file ledger. Reconciliation only detects — it never modifies files or representations. Query parameters:

wait_seconds (optional, default 0, max 300)

Headers:

Idempotency-Key (optional)

Request body:

{
  "path_prefix": "reports/",
  "auto_ingest": false
}

Body fields:

path_prefix (optional) — Scope the scan to a path prefix. Omit or pass null for the whole workspace.
auto_ingest (optional, default false) — Chain an ingestion for everything the scan found.

Response (202 Accepted): A Job with kind: "reconciliation". Result (on success):

{
  "result_type": "reconciliation",
  "files_scanned": 50,
  "discovered": ["550e8400-e29b-41d4-a716-446655440006"],
  "marked_stale": ["550e8400-e29b-41d4-a716-446655440004"],
  "orphans_removed": [],
  "ingestion_job_id": null
}

discovered — Objects in storage with no ledger row yet.
marked_stale — Rows whose observed_hash no longer matches the stored object.
orphans_removed — Rows whose object is gone from storage.
ingestion_job_id — Set when auto_ingest was requested.

Ingestion workflow

# 1. Upload files
FILE=$(curl -s -X POST "$NDI_BASE_URL/v1/workspaces/$WS/files" \
  -H "X-API-Key: $NDI_API_KEY" \
  -F "file=@document.pdf" \
  -F 'metadata={"path":"document.pdf"}' | jq -r '.result.file.file_id')

# 2. Ingest
JOB=$(curl -s -X POST "$NDI_BASE_URL/v1/workspaces/$WS/ingestions" \
  -H "X-API-Key: $NDI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"selection\":{\"type\":\"file_ids\",\"file_ids\":[\"$FILE\"]}}" | jq -r '.job_id')

# 3. Poll until done
until [[ $(curl -s "$NDI_BASE_URL/v1/jobs/$JOB" -H "X-API-Key: $NDI_API_KEY" | jq -r '.status') =~ succeeded|failed|cancelled ]]; do
  sleep 3
done

# 4. Query
curl -X POST "$NDI_BASE_URL/v1/workspaces/$WS/tools/hybrid-search" \
  -H "X-API-Key: $NDI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "revenue recognition"}'

Granular operations

Platform resources

Knowledge graph

Tools

Operations

Ingest

Reconcile

Ingestion workflow

​Ingest

​Reconcile

​Ingestion workflow

Ingest

Reconcile

Ingestion workflow