> ## Documentation Index > Fetch the complete documentation index at: https://docs.ndi.nace.ai/llms.txt > Use this file to discover all available pages before exploring further. # Connect customer storage > Prepare an S3 bucket or Azure Blob container for a customer-owned workspace. Customer storage is not generally available. This guide applies to deployments where the feature has been enabled after qualification. Disabled deployments refuse new connections and external workspaces with `501 not_implemented`; incomplete enabled deployments return `503 storage_unavailable`. A storage connection authorizes NDI to read and write an existing destination. Select a validated connection when creating a workspace. Omitting it keeps the deployment's default storage, and existing workspaces do not move. The initial providers are AWS S3 with an IAM role and Azure Blob Storage with a Microsoft Entra application secret. Custom endpoints, S3-compatible services, account keys and SAS credentials are not accepted by this connection API. Only the AWS commercial partition (`arn:aws:` role ARNs) and Azure public cloud are supported. ## Choose the destination Create the bucket or container in your cloud account first. NDI creates neither resource. Use a dedicated destination or base prefix that permits read, write, list and delete operations. The prefix is relative, such as `customer-ndi`, with no leading or trailing slash. NDI allocates a fresh namespace for each workspace; it does not adopt or index objects already in the bucket. For base prefix `customer-ndi`, the reserved layout is: ```text theme={"dark"} customer-ndi/ndi/probes/// customer-ndi/ndi/workspaces// .ndi-owner.json workspace/ .workspace/documents/.attempts// scratch/ documents// ``` Use NDI's upload/import APIs to register files. After writing sources directly inside the allocated `workspace/` namespace, run workspace reconciliation to register their current bytes; enable `auto_ingest` to process changes. Keep other applications and lifecycle rules from changing NDI's ownership marker, `.workspace/` internals or active scratch objects. Workspace contents include derived and internal files, not just the uploads visible in the console. ## AWS S3 Create the customer IAM role in the **customer AWS account that owns the bucket**. The role is an IAM resource, not a bucket setting. NDI returns its own service role ARNs as the principals to trust; those are different from the customer role ARN you will submit. 1. Create a connection with a `name` and a `destination` of `type: "s3"` with `bucket`, `region` and optional `prefix`. The draft response supplies `aws_trusted_principal_arns` and a connection-specific `external_id`. Include every returned ARN in the trust policy below; the example shows separate API and worker roles. Do not validate until the role exists and these policies are attached. 2. Create an IAM role in your account. Set its trust policy using those exact response values: ```json theme={"dark"} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": ["", ""] }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } ``` The external ID identifies this connection; use the value issued by NDI rather than an account name or a reused value. Both API and worker validation reject a role that can be assumed without that ID or with near misses of it (one character changed or appended). Use the exact `StringEquals` condition shown above; these probes do not prove every possible trust policy safe. Expect denied `AssumeRole` attempts from NDI in CloudTrail during validation. The role must belong to your AWS account, outside NDI's account; NDI-owned buckets cannot be connected as customer storage. AWS recommends an external-ID condition for third-party access. See [AWS's cross-account role guidance](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html). 3. Attach a permissions policy to that role. This example grants access to NDI's allocations beneath `customer-ndi` in `YOUR_BUCKET`; replace both values to match the connection: ```json theme={"dark"} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:ListBucketMultipartUploads"], "Resource": "arn:aws:s3:::YOUR_BUCKET" }, { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload", "s3:ListMultipartUploadParts" ], "Resource": "arn:aws:s3:::YOUR_BUCKET/customer-ndi/ndi/*" } ] } ``` For an empty connection prefix, use `arn:aws:s3:::YOUR_BUCKET/ndi/*` for objects. Grant `s3:ListBucket` without an `s3:prefix` condition: NDI treats a missing object as normal, and S3 reports a missing key as `404` rather than `403` only when the role can list the bucket. Validation checks a deleted probe object's metadata and refuses a grant that reports that missing object as an authorization failure. Bucket-wide listing reveals object names outside NDI's prefix, so use a dedicated bucket when that matters. Bucket policies and organization policies must also permit the role. Encryption with a customer-managed KMS key may require additional key permissions. See [S3's operation-to-permission mapping](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-with-s3-policy-actions.html) and [HeadObject's 404/403 rule](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html). Automatic cleanup lists incomplete multipart uploads under the job’s scratch and abandoned, unselected publication attempts, aborts them, and checks their remaining parts after writers settle. Selected publication attempts are preserved. Workspace deletion covers the complete NDI-owned allocation. These operations require `s3:ListBucketMultipartUploads`, `s3:AbortMultipartUpload`, and `s3:ListMultipartUploadParts`; see [AWS multipart cleanup](https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html). Validation checks unfinished-upload listing; it does not prove abort or part-listing permission. Missing permissions can prevent cleanup. If you configured an AWS role from an earlier policy, add the multipart permissions and revalidate the connection before creating customer workspaces. Historical versions follow your storage policy. 4. Set the connection credential to `type: "aws_role"` with your new role's `role_arn`. NDI assumes it using temporary credentials; do not supply IAM access keys. 5. Validate the connection and wait for its validation job to succeed. Confirm that the connection reports `ready` before creating a workspace. The trust policy and permissions policy serve different purposes: the first allows NDI to assume the role for this connection; the second limits what that role can do in your bucket. ## Azure Blob Storage 1. Create a Blob container and register an application in your Microsoft Entra tenant. Record its tenant ID, application client ID and client secret value. 2. Assign the application's service principal **Storage Blob Data Contributor**, scoped to the chosen container. This grants blob read, write and delete access. Azure's management-plane Contributor role alone does not grant blob data access. See [Azure blob authorization](https://learn.microsoft.com/en-us/azure/storage/blobs/authorize-access-azure-active-directory). 3. Create an NDI connection with a `name` and a `destination` of `type: "azure_blob"` with `account` (the storage account name), `container` and optional relative `prefix`. Stage `type: "azure_service_principal"` credentials containing `tenant_id`, `client_id` and `client_secret`. 4. Validate and wait for the validation job to succeed, then confirm the connection is `ready`. Container-scoped RBAC covers every blob in that container. The connection prefix limits NDI's destination; it does not narrow the application's Azure role assignment. Use a dedicated container when that is the required cloud access boundary. Role assignment changes can take time to propagate; see [Azure's container-scope instructions](https://learn.microsoft.com/en-us/azure/storage/blobs/assign-azure-role-data-access). Cleanup includes [uncommitted blobs](https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs) and uses the normal blob delete permission, included in Storage Blob Data Contributor. Historical versions and snapshots follow your storage policy. Supply the secret through the authenticated console credential form or an SDK using your secret manager or a hidden prompt. It is a write-only input. Do not include it in command-line arguments, chat prompts or logs. ## Validate and create the workspace Validation writes and removes temporary probe objects and checks that both the API and background worker can access the destination. The API probes before it starts the validation job, so a misconfigured destination can return `storage_auth_required` or `storage_unavailable` from the validate request itself, with no job. A failed validation may leave probe objects under `ndi/probes/`. A successful credential upload alone does not make a connection ready. The destination must be reachable from both service components. If the bucket or storage account restricts network access, get the addresses to allow from the deployment operator before validating. In the console, use **Connect storage** in the workspace creation dialog, or manage connections in **Settings**. Select the ready connection under **Workspace storage**, then create the workspace. Connections that are still draft, degraded or revoked cannot be selected. Storage selection is fixed at creation. Connections can be attached only within their creator's scope. A connection created by a console member belongs to that member's console session; one created for an external account belongs to callers for that account. Connections created by console owners, admins or tenant API keys are shared by those callers, but not by members' console sessions. Administrators can inspect and revoke connections they cannot attach. Members can mint tenant API keys, so use a separate tenant if its destinations must be inaccessible to them. Create the connection with the identity that will create the workspaces. For programmatic setup, follow the [Python storage connection example](https://docs.ndi.nace.ai/sdks/python/storage-connections.md) or [TypeScript example](https://docs.ndi.nace.ai/sdks/typescript/storage-connections.md). Create the workspace with `storage: { type: "external", connection_id: "" }`. Keep one idempotency key per creation request when recovering a lost response across process restarts. ## Reuse Parse results A job that reuses a Parse result copies its artifacts into the selected workspace's scratch before processing. Its published copies remain available after the producing job expires. Reuse requires an available result; a customer-storage producer without a publication receipt returns `storage_conflict`. Reuse does not reopen an original HTTP URL. Split, Extract and Ground need retained document Markdown or a readable original file. A URL-backed spreadsheet without that input returns `result_expired`, even while its Parse result remains available. Another Parse can still copy the available result. After a direct customer-storage edit and reconciliation, another Parse can copy the available result into a managed or customer workspace. Extract and other consumers can reuse retained Markdown when it supplies their input. Operations needing the changed original return `storage_conflict`. Source deletion, loss of source-label access and revoked connections still prevent reuse; a missing mutable source returns `file_not_found`. ## Rotate, revoke and delete Stage replacement credentials and validate them before retiring the old cloud identity. A failed rotation leaves the previous active identity selected. For a revoked connection, validation must set `reactivate: true`. After administrative revocation or member offboarding, the connection reports `reactivation_requires_approval` and a console owner or admin must call allow-reactivation first; approval itself does not restore access. Revoking a connection stops subsequent workspace access; it does not delete customer objects or cancel cloud requests already in flight. Published customer objects remain after job payload expiry, whether the connection is ready or revoked. NDI needs restored access to finish pending scratch or abandoned-attempt cleanup. Reconnecting restores access to retained content but does not renew expired job payloads or artifact links. Check the job's `result_error` when operational job status remains available but content is withheld. Signed downloads for customer-workspace files and document artifacts recheck storage access when opened and stop serving after revocation or deletion. Document artifact links also stop serving after job cancellation or expiry, even if their URL has not expired. A transfer already in progress cannot be recalled. Document cleanup runs after the job's payload expires, or its zero-retention delivery window ends, and after dependent jobs release their holds. It removes temporary processing scratch once in-flight writes settle. A separate cleanup pass removes abandoned, unselected publication attempts; published customer objects and their integrity receipts survive job payload expiry. Deleting objects directly in your cloud account does not close NDI's pending cleanup record: restore access so NDI can verify the deletion, or complete verified workspace deletion. Deleting a workspace removes its NDI-owned workspace and scratch namespaces. Cleanup uses the connection, so delete workspaces before revoking it, and keep valid cloud authorization available until cleanup completes. If the connection is revoked, the `workspace_delete` job fails with `storage_auth_required` and the workspace stays `deleting`; reactivate the connection and send DELETE again. Cleanup is complete only when the `workspace_delete` job succeeds with `result.verified: true`. A failed cleanup needs revalidation and another DELETE; changing credentials does not migrate the workspace to a new destination. Deleting a connection is refused while any workspace binding or pending creation attempt remains. Verified workspace cleanup releases its binding, so delete the connection after all its workspace deletions complete. Cleanup waits for in-flight writes to settle. A failed or interrupted write whose outcome is uncertain can delay cleanup until its upload deadline and provider retry window have passed. There is no endpoint to release a pending attempt. Complete it by retrying the creation with its original idempotency key; otherwise keep the connection `ready` and wait for automatic cleanup of attempts abandoned for more than a day. A failed creation sent without an idempotency key can only wait for that cleanup. NDI keeps the staged credential while the connection exists, and a revoked connection can be reactivated from NDI. To end NDI's access permanently, delete its workspaces and wait for verified cleanup, revoke the connection, then remove access in your cloud account: delete the IAM role or its trust statement, or delete the Entra client secret or role assignment. Removing the trust statement or the client secret stops new credentials but does not invalidate sessions or tokens already issued, which stay valid until they expire. For a faster cutoff, remove the role's permissions policy or revoke its active sessions in AWS, or remove the Azure role assignment, allowing for propagation. | Error | Next step | | - | - | | `storage_auth_required` (424) | Check the role trust, permissions, application secret and network policy; reconnect and validate. | | `storage_unavailable` (503) | Retry a transient failure. If validation keeps failing, check the connection region and ask the operator which network addresses to allow. For an unconfigured deployment, contact its operator. | | `invalid_request` (422) | Use a bucket and role in your own AWS account; NDI refuses its own buckets when you create the connection and roles in its account when you set credentials. | | `storage_conflict` (409) | Inspect the connection or workspace state. Validating a revoked connection needs `reactivate: true`; wait for a running validation to finish. A changed destination or ownership conflict may require a new connection/request. A download from a non-retained (zero-retention) document job needs retained output. | | `file_not_found` (404 or failed job) | A file record, path or mutable source object is missing. Restore or upload the source, reconcile the workspace and submit a new document job. | | `storage_connection_in_use` (409) | Retry pending creations with their idempotency keys or wait for abandoned-attempt cleanup; wait for a running validation to finish. Wait for every bound workspace to finish verified cleanup before deleting its connection. | | `storage_connection_not_found` (404) | Check the connection ID and the caller scope. Use the identity that created the connection, or create one under the right identity. | | `access_denied` (403) | Use an authorized owner or administrator for the requested operation, or get reactivation approved first. Cloud credentials do not grant NDI workspace permissions. | ## Retention scope Customer storage selects where durable workspace bytes live. Temporary processing, database metadata, search indexes, job history and service logs retain their existing placement. This is not a service-wide zero-retention guarantee. Private agent activity traces are not recorded for customer-storage workspaces; the activity routes return `result_expired`. External workspaces reject both source and derived TTLs. NDI automatic retention preserves durable source files, derivatives and published results in customer storage; you control their cloud lifecycle. Workspace deletion removes current NDI-owned objects, but cloud versioning, soft deletion, backups and retention policies can keep historical bytes in your account. Configure those policies according to your retention requirements; NDI workspace deletion does not override them. Deployment operators configure `NDI_STORAGE_AWS_PRINCIPAL_ARNS` as a JSON array of every actual runtime role that accesses customer storage, for example `["arn:aws:iam::111111111111:role/api", "arn:aws:iam::111111111111:role/worker"]`. JSON preserves commas within role names. It supersedes `NDI_STORAGE_AWS_PRINCIPAL_ARN`; an unset plural setting keeps the legacy single-role configuration. The API returns `aws_trusted_principal_arn` only when exactly one role is required. Updating an ARN setting does not change the runtime AWS identity. ### Temporary document outputs Downloads of customer-storage document artifacts require retained output with a host publication receipt. Jobs configured for zero-retention output can return their inline answer during the delivery window, but file, image, Markdown and row downloads return `409 storage_conflict` because temporary scratch objects have no immutable receipt. Use retained output in the customer workspace when those downloads are needed. Workspace content still remains exempt from NDI's automatic content retention. Customer-storage Parse requests with zero-retention output must disable `return_ocr_data`. NDI rejects that combination with `422 invalid_request` before scheduling work; retain the output to request inline OCR data. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.