Ingest and reconcile

Ingestion is the only call that makes uploaded files searchable. Outcomes are per file: one corrupt document in a folder does not fail the whole run. A valid raw Parquet file with zero rows is reported as skipped_empty, counted in files_empty, and left without a searchable table representation.

Python
TypeScript
curl

from ndi_sdk import NdiClient

with NdiClient() as client:
    workspace = client.workspaces.create(name="ingest-demo")
    client.jobs.wait(
        client.files.upload(
            workspace.workspace_id,
            "report.pdf",
            path="reports/report.pdf",
        ).job_id
    )
    job = client.jobs.wait(
        client.ingestion.ingest(
            workspace.workspace_id,
            path_prefix="reports/",
        ).job_id,
        timeout=600,
    )
    print(job.result)

import { NdiClient } from "ndi-sdk";

const client = new NdiClient();
const workspace = await client.workspaces.create({ name: "ingest-demo" });
await client.jobs.wait(
  (
    await client.files.upload(workspace.workspace_id, "report.pdf", {
      path: "reports/report.pdf",
    })
  ).job_id,
);
const job = await client.jobs.wait(
  (await client.ingestion.ingest(workspace.workspace_id, { path_prefix: "reports/" })).job_id,
  { timeout: 600 },
);
console.log(job.result);

curl -s -X POST "$NDI_BASE_URL/v1/workspaces/$WS/ingestions" \
  -H "X-API-Key: $NDI_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: ingest-reports-v1" \
  -d '{"selection":{"type":"path_prefix","path_prefix":"reports/","stale_only":false}}'

Selection

Pass one of these selectors, or omit both to ingest the whole workspace:

Option	Meaning
`file_ids`	Named files to ingest. Cannot be combined with `path_prefix`
`path_prefix`	Folder prefix. An empty prefix is the whole workspace
`stale_only`	When true, a folder run only includes files whose bytes changed since the last ingest
`reingest_unchanged`	When true, rebuild derivatives even when the content hash matches

The result separates files that were imported (files_ingested), already current (files_skipped), valid but contained no rows (files_empty), and invalid (files_failed). Empty Parquet is informational; corrupt Parquet still appears as a failed per-file outcome. stale_only=True and reingest_unchanged=True can be combined with path_prefix. Combining file_ids with path_prefix is invalid_request.

Reconcile

Reconcile compares the storage tree to the file ledger. Detection only: discovered, stale, and orphan rows. auto_ingest=True chains an ingestion and returns result.ingestion_job_id only when reconcile found files to discover or mark stale. Otherwise no ingestion starts and result.ingestion_job_id is None.

from ndi_sdk import NdiClient

with NdiClient() as client:
    workspace_id = "550e8400-e29b-41d4-a716-446655440001"
    job = client.jobs.wait(
        client.ingestion.reconcile(workspace_id, auto_ingest=True).job_id
    )
    print(job.result)

When the workspace policy is ledger_understanding.trigger=auto and the ingest landed a journal-line table, the run chains a ledger-understanding job. IngestionResult.ledger_understanding_job_id names it; poll with jobs.wait. See Ledger understanding.

Errors

file_not_ingested — a tool ran before ingest finished
invalid_request — both file_ids and path_prefix on one call

See File lifecycle.

Getting started

Guides

Agents

Concepts

Troubleshooting

Selection

Reconcile

Errors

​Selection

​Reconcile

​Errors

Selection

Reconcile

Errors