Data retention

This page states what NDI stores about your documents and results, where it sits, and when it is deleted. Retention depends on how you call NDI: workspaces keep files until a lifetime you choose runs out, granular operations use temporary copies, and zero retention, available on request, keeps no document content once a document operation’s short delivery window has passed.

​

At a glance

​

Bring-your-own buckets

For a workspace backed by your own bucket, NDI does not apply source_ttl_days or job-artifact expiry: those bytes are yours to manage. NDI does delete what it wrote or staged there when its lifecycle requires it: a file’s derived folder (on derived expiry or file deletion), a deleted file’s source bytes, staged uploads, and the whole workspace prefix when you delete the workspace.

​

Workspace retention

A workspace’s retention_policy has three lifetimes:

• derived_ttl_days: counted from ingested_at. When it elapses, the file’s derived material is removed and its ingestion_status becomes expired. The source bytes are not touched. If a knowledge graph was built over the file, the file’s content is removed from it and the workspace’s graph goes offline until it is rebuilt.
• source_ttl_days: counted from created_at. When it elapses, the file is deleted from platform-managed storage: the source bytes of every version and its derived material.
• job_ttl_days: how long job input, result and processing files are kept, at most the account’s job retention (48 hours on a paid account, 7 days on a free account). It can only shorten that period; a longer value, or null, does not extend it. Deleting a job removes its payload sooner; see Jobs and artifacts.

An unset (null) derived or source TTL means no expiry. Expiry is applied by a scheduled sweep, so data is removed after its lifetime ends rather than at that exact moment. Deletion of expired derived material and source files is switched on per deployment; ask your NDI contact whether it is enforced for yours. The policy is editable with PATCH /v1/workspaces/{id} (workspaces.update). A changed derived_ttl_days is applied to files already ingested. A changed source_ttl_days takes effect at the next sweep for every file. A changed job_ttl_days applies to jobs created after the change; existing jobs keep the expiry they were given. Deleting a file soft-deletes its record: the record is kept with its metadata de-identified (see What we keep after deletion), and it no longer lists or resolves. Its source bytes (every version) and all content derived from it are removed. The same happens when the source_ttl_days lifetime ends. The PDF an Office file was converted to is deleted with the file too, unless a retained Extract result still points at it. Deleting a workspace removes all of its files.

​

Jobs and artifacts

A job’s inputs, outputs and processing files are removed 48 hours after the job completes on a paid account and 7 days after on a free account. For a failed or cancelled job the period counts from the failure or cancel. A workspace job_ttl_days can only shorten it. When a job’s payload expires, GET /v1/jobs/{id} still returns the job with result_state=expired and no result. The input and result are not returned again; NDI keeps only a de-identified record of them for audit (see De-identified payloads). In platform-managed storage, the job’s artifact files (markdown output, figure crops, converted PDFs, source snapshots) are deleted with the payload. If another job that is still retained reused this job’s Parse result, the artifact files that job uses stay until its own payload expires. Job metadata stays. DELETE /v1/jobs/{id} hides the job, de-identifies its input and result and deletes its artifact files within minutes, under the same reuse rule. The cached copies of its results that NDI keeps to answer identical requests faster are deleted with them. A running job is cancelled first. Job metadata, usage and credits stay. Deleting a file also deletes the cached results that your jobs computed from it, as long as those jobs’ payloads have not expired yet. Older cached results expire no later than 48 hours after they were computed, so they are gone well within the 72 hours a deletion request takes. A cached result that one of your retained jobs still uses is no longer reused and is deleted once that job’s payload expires.

​

De-identified payloads

When a job’s input and result are removed (their lifetime ends, you delete the job, zero retention clears the input, or the organization is purged), NDI keeps a de-identified record of their structure for audit and billing. The same applies to a legacy /api/v1 request’s input and response 48 hours after the request finishes, or sooner when its organization is purged. The record is never returned by the API. It keeps:

• the names of fields NDI defines, such as result_type, page_count or source, and the nesting and list lengths of the payload (a long list keeps its first items and a count of the rest);
• counts and measures: pages, units, sizes, token counts and durations;
• true, false and null in fields NDI defines;
• fixed values of fields NDI defines, such as the operation, mode, status, type and error code, and identifiers in UUID form.

Everything else is replaced by [redacted]: document text and markdown, file names, paths, URLs, prompts and questions, names you chose, every other number, and everything inside data shaped by you, such as extraction results and the JSON schema you sent, including its field names.

​

Staged uploads

A POST /v1/uploads handle is staging for a job. The staged bytes are deleted at the next cleanup sweep after the job that consumes the upload ends; the upload’s record stays, anonymized, and the handle stops resolving. An upload that no job is using is removed after its expires_at. See Sources and uploads.

​

Saved schemas and domains

The extraction schemas you create with POST /v1/extraction-schemas (every version of each family) and the domains you create with POST /v1/domains (drafts and published versions) are saved definitions. Workspace lifetimes and zero retention do not apply to them. A schema is kept until you delete it or your organization is closed. DELETE /v1/extraction-schemas/{schema_id} deletes every version of the schema: its name, description and JSON Schema are removed at once, and only its id, version numbers and timestamps stay, so jobs that used it still name the schema_id and schema_version they ran with. See Extraction schemas. A domain is kept until your organization is deleted. There is no endpoint to delete a domain: workspaces are created against a published domain version and keep pointing at it. To have one removed sooner, ask your NDI contact. When the organization is deleted, its schemas and domains are emptied the same way: a schema loses its name, description and JSON Schema and keeps only its id, version numbers and timestamps, and a domain loses its taxonomy and ontology and is renamed to deleted-domain- followed by its id. The rows stay as empty records marked deleted and no endpoint returns them; see Deleting an organization or tenant. The schemas and domains NDI provides to every account are not affected.

​

Zero retention

Zero retention is an account-level data-handling setting for document operations (parse, split, classify, extract, ground). It is available on request: it is off by default, and NDI enables it per deployment and per account. It is not a request field, so no individual call can opt out of it by accident. Ask your NDI contact whether your account has it. Partners can set it per tenant with data_retention once it is enabled; see Tenants. For an account with zero retention:

• Call with wait_seconds. A document operation without wait_seconds is rejected with HTTP 422 wait_required_for_zero_retention, because the result can only be delivered to the waiting request.
• The result is returned once, inline. It is handed to the waiting request through a short-lived in-memory cache and is never written to the database. A later GET /v1/jobs/{id} returns the job with result_state=not_retained and no result. Job metadata, usage and credits remain, so billing works.
• Artifact files have a delivery window. Files the result links to (images, converted PDFs, crops, content files) stay downloadable through those links for ZERO_RETENTION_DELIVERY_WINDOW_SECONDS after the job finishes, then are deleted. The window is the longest inline wait (MAX_INLINE_WAIT_SECONDS) plus the lifetime of a download link (DOWNLOAD_URL_TTL_SECONDS) plus 60 seconds of grace, so any link you received inline still works for its full lifetime. Download what you need during that window.
• The request is never stored in the database. The job record holds only the request’s de-identified record, which has no document content and no text you sent. The full request is held for the worker in the same short-lived in-memory cache, and is deleted when the delivery window below ends.
• Name and error message are never stored. The job record keeps no name; GET /v1/jobs/{id} shows a generated one such as Extract 1a2b3c4d. A failed job keeps only its error code: the full error message is held in the same short-lived in-memory cache, readable on GET /v1/jobs/{id} until the delivery window below ends, and a cancel reason is not stored.
• Upload file names are never stored. A staged upload is recorded as upload-<id> plus its extension, and its storage key carries the same name. The name you sent is held in the in-memory cache for the upload’s lifetime plus the delivery window, for the worker and your results.
• No legacy API. The legacy /api/v1 API stores every request and result, so an account with zero retention gets HTTP 403 access_denied from its processing and upload routes. Use /v1 with wait_seconds.
• No reuse caches. Results are not cached or published to shared reuse caches. Working copies a job keeps while it runs, such as per-page OCR checkpoints, expire within the delivery window below.
• Leftovers are deleted. Staged uploads, source snapshots and job artifacts are deleted at the first cleanup sweep after the delivery window ends. Where that storage keeps earlier versions of a file, every earlier version is deleted with it, so none is kept as a previous file version. For each deleted staged upload NDI keeps only the anonymized file record described in What we keep after deletion.

Everything above is gone within one hour of the result being delivered: the delivery window is ZERO_RETENTION_DELIVERY_WINDOW_SECONDS after the job finishes, and the cleanup sweep runs every five minutes. NDI does not write zero-retention content to logs or traces; they hold only usage metadata. Zero retention covers document operations. It does not change workspaces, which keep files by design under the lifetimes above.

​

What we keep after deletion

Deleting content removes the content. NDI does not delete the records that describe it: they are kept for internal audit and billing, with the parts that could identify a person or a document removed. A record you deleted is marked deleted, so it no longer lists, resolves or grants access.

• Jobs. The job record stays after the job, its workspace or its organization is deleted: id, kind, status, timestamps, pages, units, bytes, credits, token counts per model, the error code, and the ids of the user, partner user and partner account that ran it. When the job’s payload expires or is deleted, or its organization is purged, its name is cleared, its error is reduced to the error code, and its input and result are reduced to a de-identified record. 13 months after the job was created, the user and partner user ids, the name and the error message are cleared too; the partner account id stays because usage is billed to it, and the idempotency key stays so a retry still replays (see Idempotency keys below). A zero-retention job’s name and error message are cleared the same way, once its delivery window has passed (see Zero retention).
• Deleted files. When a workspace file is deleted or its lifetime ends, when its workspace is deleted or its organization purged, and when a staged upload is removed, NDI writes a record with an anonymized name of the form file-<id>.<ext>, a keyed one-way fingerprint of the original name (when the deployment’s fingerprint key is configured), the extension, the size and page count when known, and the reason the file was deleted. A deleted file’s entry in the workspace, and a removed staged upload’s record, stay under the same anonymized name. These records are kept, and the fingerprint with them for at least 7 years after the file was deleted.
• People in a purged organization. When an organization is purged, a person whose only organization it was is kept as an anonymized record, shown as deleted-user-<id>, so jobs keep pointing at one anonymous id. It has no email and no login identifiers, and cannot sign in. People who also belong to another organization keep their account. 24 months after the account was closed, the person’s console keys are revoked and renamed deleted-key-<prefix>, the invitation they accepted shows deleted-user-<id> instead of their email, and a CLI login they approved no longer holds a key. The records themselves are kept. A purged person’s console sessions, password record and reset links stay as revoked or used rows whose secrets are replaced, so none of them can sign anyone in.
• Workspaces and API keys of a purged organization. They stay as rows renamed deleted-workspace-<id> and deleted-key-<prefix>, with their settings and descriptions cleared. Each key is revoked and its hash replaced, so it can never authenticate. 24 months after the purge, they also lose the person or partner account that created or owned them; the rows stay because jobs point at them. A workspace you delete yourself stays as a deleted record under its id and name; its key grants and member shares stay as deleted records that grant nothing.
• Other records of a purged organization. Memberships, invitations, chunked upload sessions, CLI logins, webhook endpoints, webhook events and their delivery records, limit settings and access labels stay as rows marked deleted, revoked, expired or disabled. Email addresses, paths, URLs, secrets, key material, descriptions and payloads are removed from them.
• Idempotency keys. The Idempotency-Key of a job, a legacy request or a chunked upload session stays on its record for as long as the record is kept, also after the payload expires, usage is de-identified or the organization is purged, so a retry with the same key still returns the original record instead of starting new work. NDI never clears it. Do not put personal data, file names or other confidential text in an idempotency key; use an opaque value such as a UUID.
• Webhook events. After the webhook retention period an event keeps its id, type, job and time, and the record of each delivery; its payload is removed and it can no longer be re-sent.
• Usage after 13 months. Usage records are never deleted: they are de-identified. After 13 months a job keeps its kind, status, timestamps, pages, units, bytes, credits and token counts, and loses who ran it and any text you chose for it other than its idempotency key. Requests on the legacy API lose their output location and error message the same way. Usage and billing totals do not change.
• Chunked-upload sessions. Once a session is completed or aborted and past its expiry, it loses its file path, labels and partner user. Its status, sizes, timestamps, idempotency key and the job that completed it stay.

NDI does not keep file content, the content of job inputs or results, or the original names of deleted files. The deleted-file record never stores the original name. Content NDI derived from your files (search indexes, the metadata catalog, extracted records, caches, job secrets) is emptied: the stored bytes and index documents are deleted, and the database row that held it stays as an empty record marked deleted, with its text, values and secrets removed. No database row is ever deleted. The fingerprint of a file name cannot be reversed to the name. It lets NDI answer one question: when you give us a file name, was a file with that name processed? NDI computes the fingerprint of the name you give and compares it. The fingerprint is kept for at least 7 years after the file was deleted. A record written without a fingerprint cannot answer that question; the file is deleted all the same.

​

Processing by other systems

Content is processed transiently by systems NDI depends on while a job runs:

• Model providers. Parsing, classification, extraction and grounding send document content to the model providers that serve the selected lane, which they receive in order to produce the response. Provider-side retention is governed by each provider’s own terms.
• Workflow engine. NDI runs jobs on a workflow engine. Payloads it passes between steps are held in the engine’s history and, for large payloads, in an offload bucket, each under its own retention period. Where payload encryption is configured for a deployment, those payloads are encrypted with keys NDI manages before they leave NDI’s workers.

​

Deleting an organization or tenant

A partner can permanently delete a tenant’s data with DELETE /v1/tenants/{external_ref}?purge=true; see Purging a tenant. An inactive self-serve organization on the free credit allowance may be deleted after a period without activity; see Inactive organizations.

​

Related

• File lifecycle
• Workspaces
• Sources and uploads