Blogs · Scale

File sizes have no limit

Supporting multi-terabyte access without breaking identity, policy, or the agents that depend on both.

Enterprise file infrastructure fails in two opposite directions: it either caps object size so teams fragment work across zip bombs of shards, or it accepts huge blobs and loses everything that makes a file intelligent — versioning, scans, citations, dual representation. Deliniext refuses that trade. File sizes have no practical ceiling for committed use; what changes is how bytes move, not what the ledger promises.

The demand is already here. A single uncompressed 8K production master can exceed hundreds of gigabytes. Particle-physics and genomics datasets routinely reach terabytes per instrument run. Automotive and robotics programs store multi-hour sensor logs that must be replayable for safety review. Industry surveys of media and research IT consistently show that “large file” pain — failed uploads, timeout-prone downloads, re-transfer after network blips — ranks among the top three operational complaints, often above raw storage cost. The problem is not that enterprises lack bytes on disk. It is that their file layer treats a 12 TiB object like a 12 KiB attachment with a longer timeout.

50 GiB+ per object Enterprise current supports resumable ingest up to 50 GiB per object on SaaS (higher on sovereign fleets). Identity, Harbor Scan, and versioning attach at commit — not after a fragile monolithic upload succeeds.

Deliniext separates transfer from commit. Ingest begins with a ledger reservation: collection, path, audience, policy version. For payloads above 32 MiB the API returns transfer: "resumable" and an upload_url with chunk semantics your client controls. Parts can retry independently. The file does not exist for agents until POST /v1/files/{id}/commit succeeds — at which point virus and PII scans enqueue on that version the same way they do for a spreadsheet. A videography team uploading a 4 TiB archive to a sovereign fleet uses the same contract; only the ceiling and region are configured locally.

Multi-terabyte access is not only about upload. Operators and agents need range-aware reads without pulling the entire object into a context window or a laptop disk. GET /v1/files/{id}/content supports HTTP range requests so players, transcoders, and training pipelines stream the segments they need. The differential still describes the whole: codec, duration, checksum, proxy handles, transcript rule outputs. An agent cites a timecode span; a human sees the dossier frame. Inner Keel parsing does not mean “convert everything to JSON.” It means the ledger holds truthful structure — including for binaries — so nobody re-scrapes the blob because the platform forgot large files exist.

Videography assets are an instructive example. Production libraries store camera originals, color grades, and delivery masters under one collection policy. Deliniext stores bytes durably, runs virus scan at commit, and generates proxies and waveform derivatives as processing rules — not as a separate DAM that drifts from the record. Clear Channel search can find “Scene 4 exterior, take 3” across metadata and approved transcript text without exposing raw PII from talent releases. When legal asks what left the building, audit export includes citations and scan events, not a shrug about a file too big to log.

Large ML datasets stress another dimension. Training corpora are not “a pile of files.” They are pinned versions, schema on extracted fields, and governance that blocks open-ended model writes to cleared collections. A 9 TiB image corpus might pin version 7 for a quarter while version 8 ingests new labels. Reprocess jobs (POST /v1/jobs/reprocess) batch across the corpus with webhooks for progress — throughput as a fundamental, not a demo-day trick. Teams that once rsync’d shards between buckets now hold one collection identity with differential history an agent can reason over: which snapshot trained which model, which scan cleared which partition.

Self-hosted sovereign fleets push limits further. When Restora is not operating the plane, you set cluster-level maximum object size, egress policy, and HA topology — but the API surface does not change. Platform SREs get the same idempotency keys, the same Harbor Scan events, the same MCP tools. That parity matters for regulated industries: the audit story for a 800 GiB clinical imaging study must be identical whether the bytes sit in Restora SaaS or in your VPC. File size is an operational parameter. Trust is not.

If your evaluation checklist still says “max upload 5 GB,” update it. Ask instead: Can I resume after a dropped link? Can an agent read metadata and a byte range without owning the whole object? Does scan run on the version that actually committed? Can I pin a terabyte-scale snapshot for ML without forking identity? Deliniext is built so the answer is yes — because file intelligence that stops at gigabytes is not enterprise grade. It is a consumer drive with an API key.

File sizes have no limit that matters to your architecture. The ledger does not shrink when the payload grows. It grows more necessary — one identity, two truths, lifecycle compliance, and tooling that still works when the file is larger than the team that uploaded it.