These docs track the development branch (main). Latest release: v0.23.0.
CargoShip

Appearance

CargoShipHigh-performance S3 data archiving

Stream large datasets straight to S3 — sharded, compressed, verifiable, and cost-aware. Your data lands in an open, portable archive format you can read with or without CargoShip.

Quick Start
What is CargoShip?
View on GitHub
CargoShip — a container ship sailing on a cloud
🚢

Zero-disk streaming pipeline

Scanner → Chunker → Archiver → Uploader, connected over io.Pipe. Data streams directly to S3 without staging to local disk, so memory stays bounded regardless of dataset size.

How it works

⚡

Multi-prefix parallel sharding

CargoHold spreads chunks across multiple S3 prefixes with an adaptive shard count (4–32), parallelizing request throughput instead of funnelling through one key space.

Learn more

🗜️

Content-aware compression

Streaming tar + Zstandard with per-content-type levels, plus optional Magika AI file-type detection to compress smarter.

Learn more

💰

Cost & budget controls

Estimate before you upload, track spend by project, set dual cost + volume quotas, and forecast burn-down with multiple models.

Learn more

🔒

Encryption & integrity

SSE and KMS-envelope-encrypted manifests, optional GPG, and manifest-checksum verification that proves a round trip on restore.

Learn more

📖

Open, portable format

Plain tar.zst objects plus a documented JSON manifest (v2.0). Extract your data with standard tools even without CargoShip installed.

Read the spec

Get where you're going ​

CargoShip ships large research and backup datasets to Amazon S3 quickly, cheaply, and reversibly. Pick the path that fits you:

Install ​

bash
brew install scttfrdmn/tap/cargoship
bash
go install github.com/scttfrdmn/cargoship/cmd/cargoship@latest
bash
# Download the latest release for your platform from:
# https://github.com/scttfrdmn/cargoship/releases/latest

Then send your first dataset to S3:

bash
cargoship upload ./my-data s3://my-bucket/archives/

See Install for all methods and Your first upload for a line-by-line walkthrough.