Uploading & sync commands
Commands that write data to S3. For guidance on which to use, see upload vs. create upload; for a walkthrough, Uploading data.
Generated reference
Flag tables below are generated from the CLI and kept in sync by a drift check.
cargoship upload
Upload directory to S3 with CargoHold sharding
Synopsis
Upload a directory to S3 using CargoHold's intelligent sharding system.
CargoHold divides large datasets into multiple shards for parallel uploads, providing:
- Intelligent shard distribution (round-robin, hash, size, type, or directory)
- Content-aware zstd compression, with an optional fixed-level override
- Parallel uploads for maximum throughput
- Automatic manifest generation for easy restore
- Progress tracking
Shard Count: CargoShip automatically selects between 4 and 32 shards when --shard-count is 0 (the default). If automatic selection fails, it falls back to 8.
Shard Strategies: round-robin - Distribute by chunk order (even shards, cheapest; default) hash - Hash of chunk contents (stable across runs) size - Least-loaded shard by bytes (evens out uneven chunk sizes) type - Group chunks by predominant content type directory - Group chunks by common directory prefix
Compression: By default CargoShip picks a zstd level per chunk from the content it contains — level 1 for already-compressed data, up to level 9 for source code. Passing --compression-level overrides that and pins every chunk to one level. Note that zstd has only four internal levels, so values map in bands (1-2, 3-5, 6-9, 10+); the effective setting is reported at upload start.
Examples:
Upload with defaults (auto shard count, round-robin, content-aware compression)
cargoship upload /data s3://my-bucket/dataset
Upload with custom shard count and strategy
cargoship upload /data s3://my-bucket/dataset --shard-count 20 --shard-strategy size
Pin every chunk to maximum compression (disables content-aware selection)
cargoship upload /data s3://my-bucket/dataset --compression-level 19
Quiet mode (no progress display)
cargoship upload /data s3://my-bucket/dataset --quiet
cargoship upload SOURCE_DIR DESTINATION [flags]Options
--auto-tier Enable automatic storage tier selection based on file access time
-b, --bucket string S3 bucket name (or use s3:// URL in DESTINATION)
--compression-level int Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
--congestion-control string Congestion control algorithm: bbr, cubic, auto (default "auto")
--direct-upload Enable direct upload mode (bypasses archiving/compression for small files)
--direct-upload-threshold-mb int Max total size in MB for auto direct upload (default: 500) (default 500)
--direct-upload-workers int Worker count for direct upload (default: 256) (default 256)
--disable-staging Disable adaptive staging (reduces memory usage)
--dvc-auto Auto-discover DVC stages from dvc.yaml and annotate each file entry with its stage name
--dvc-cache-dir string Local DVC cache directory (recorded in manifest; default: .dvc/cache) (default ".dvc/cache")
--dvc-output-dir string Directory to write .dvc files (default: source directory)
--dvc-stage string DVC pipeline stage name to extract provenance from (reads dvc.yaml + dvc.lock)
--enable-dedup Enable cross-shard file deduplication (10-30% space savings for redundant datasets)
--encrypt-manifest Encrypt manifest with KMS envelope encryption (requires --kms-key-id)
--force-direct-upload Force direct upload regardless of thresholds (for benchmarking)
--force-restart Ignore saved state and start fresh upload (bypasses resume detection)
--generate-dvc-files Generate DVC sidecar .dvc files after upload
--git-metadata Embed Git repository metadata (commit, branch, tag, remote) in the manifest
-h, --help help for upload
--incremental Enable incremental sync: only upload new or changed files
--kms-key-id string AWS KMS key ID or ARN for encryption (data chunks encrypted with SSE-KMS)
--no-file-checksums Disable per-file content checksums (faster uploads, but 'verify --deep' can't confirm per-file integrity)
--optimization Enable optimization features (BBR/CUBIC, adaptive staging, BDP) (default true)
--prev-manifest string Path to previous manifest JSON (or .json.gz) for incremental sync
--project string Project ID for cost tracking (e.g. 'dvc_cache' for DVC remotes)
--prometheus-addr string Prometheus metrics HTTP address (e.g., :9090)
--quiet Disable progress display
-r, --region string AWS region (default "us-west-2")
--shard-count int Shards for parallel uploads: 0 auto-selects 4-32 (falls back to 8), or set 4-32 manually
--shard-strategy string Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
--tag stringArray Custom tag in key=value format, repeatable (e.g. --tag dvc_cache=true --tag env=prod)
--tier-archive-days int Days since access to consider 'archive' (DEEP_ARCHIVE) (default 180)
--tier-cold-days int Days since access to consider 'cold' (GLACIER) (default 90)
--tier-hot-days int Days since access to consider 'hot' (STANDARD) (default 30)
--tier-max string Maximum storage tier (STANDARD, STANDARD_IA, GLACIER, DEEP_ARCHIVE) - prevents automatic selection of more restrictive tiers
--tier-strategy string Tier chunking strategy (requires --auto-tier):
youngest-file: Conservative strategy - assigns tier based on youngest file per chunk (default)
tier-aware: Optimal cost - groups files by tier before chunking (30-60% savings)
⚠️ WARNING: tier-aware uses GLACIER/DEEP_ARCHIVE with cost implications:
• GLACIER: 90-day minimum storage ($0.004/GB-month, $0.01/GB retrieval, 3-5hr access)
• DEEP_ARCHIVE: 180-day minimum ($0.00099/GB-month, $0.02/GB retrieval, 12hr access)
• Early deletion penalties apply if removed before minimum duration
• Best for long-term archives accessed <1x per year
See: https://github.com/scttfrdmn/cargoship/issues/168 (default "youngest-file")
--tracing Enable distributed tracing
--tracing-endpoint string Tracing endpoint URL (required for jaeger/otlp exporters)
--tracing-exporter string Tracing exporter: stdout, jaeger, otlp, none (default "stdout")
--tracing-sample-rate float Trace sampling rate (0.0-1.0, default: 1.0 = 100%) (default 1)
--transporter string S3 transporter type: basic, staging, adaptive, optimized, none (default "staging")
-y, --yes Skip confirmation prompts (auto-accept warnings)Options inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship create
Create something!
cargoship create [flags]Options
-d, --destination string Directory to write files in to. Defaults to the current directory
-h, --help help for createOptions inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship create upload
Upload directories to S3 using streaming pipeline
Synopsis
Upload directories to S3 using the high-performance streaming pipeline.
This command replaces the legacy suitcase/rclone system with a modern streaming architecture that provides:
- Real-time progress tracking with beautiful TUI
- Multi-prefix S3 parallel uploads (8x throughput improvement)
- Zero local disk usage (streaming directly to S3)
- Automatic compression (zstd)
- Intelligent chunking and sharding
cargoship create upload SOURCE_DIR... [flags]Examples
# Upload a directory with progress tracking
cargoship create upload /path/to/data --bucket my-bucket
# Upload with custom prefix
cargoship create upload /path/to/data --bucket my-bucket --prefix backups/2025-12-05
# Quiet mode (no progress display)
cargoship create upload /path/to/data --bucket my-bucket --quiet
# JSON progress output (for scripts)
cargoship create upload /path/to/data --bucket my-bucket --progress-format jsonOptions
--bucket string S3 bucket name (required)
--chunk-size-mb int Target chunk size in MB (0 = adaptive) (default 200)
--cleanup-on-failure Automatically delete partial uploads on error (default true)
-h, --help help for upload
--http2 Enable HTTP/2 (default true)
--http2-max-streams int Max concurrent HTTP/2 streams per connection (default 250)
--idle-conn-timeout duration Idle connection timeout (default 5m0s)
--max-idle-conns int Max idle connections per host (default 100)
--network-profile string Network tuning profile: default, aggressive, conservative (default "default")
--no-cleanup Disable automatic cleanup on failure (for debugging)
--prefix string S3 key prefix (optional)
--progress-format string Progress output format: tui, json, text (default "tui")
--quiet Disable progress display
--region string AWS region (default "us-west-2")
--resume Resume a previous incomplete upload
--shards int Number of S3 prefix shards for parallel uploads (default 8)
--skip-existing Skip chunks that already exist in S3 (HeadObject check)
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
--upload-id string Upload ID to resume (auto-detect if not specified)
--workers int Workers per stage (scanner, archiver, uploader) (default 4)Options inherited from parent commands
--context string Override execution context (local, agent, repl)
-d, --destination string Directory to write files in to. Defaults to the current directory
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship sync
Incrementally sync directory to S3 (only upload new/changed files)
Synopsis
Incrementally sync a local directory to S3 by uploading only new or modified files.
The sync command provides efficient incremental backups by:
- Downloading the latest manifest for the source path (if exists)
- Comparing local filesystem state against the manifest
- Uploading only files that are new or have changed
- Creating a new manifest that references the previous one
First sync uploads everything (like 'upload' command). Subsequent syncs only upload changed files, saving time and bandwidth.
Change detection (default: fast mode):
- Size change: File size differs from manifest
- Time change: Modification time is newer than manifest
Use --checksum for guaranteed accuracy (slower, computes SHA256).
Examples:
First sync: uploads all files
cargoship sync /home/photos s3://my-bucket/backups
Second sync: only uploads new/changed photos
cargoship sync /home/photos s3://my-bucket/backups
Dry run to see what would be synced
cargoship sync /home/photos s3://my-bucket/backups --dry-run
Use checksum comparison (slower but accurate)
cargoship sync /data s3://my-bucket/backups --checksum
Force full sync (ignore previous manifest)
cargoship sync /data s3://my-bucket/backups --force
cargoship sync SOURCE_DIR S3_URL [flags]Options
--checksum Use SHA256 checksum comparison (slower but accurate)
--compression-level int Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
--dry-run Show what would be synced without uploading
--force Force full sync (ignore previous manifest)
-h, --help help for sync
-q, --quiet Quiet mode (minimal output)
-r, --region string AWS region (default "us-west-2")
--shard-count int Number of shards for parallel uploads (1-100) (default 10)
--shard-strategy string Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
--storage-class string S3 storage class (STANDARD, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
--track-deletes Track deleted files in manifestOptions inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship migrate
Convert traditional archives to CargoHold sharded format
Synopsis
Download a traditional tar.zst archive from S3 and re-upload using CargoHold's intelligent sharding system.
The migrate command:
- Downloads the traditional archive from S3
- Extracts files to a temporary location
- Re-uploads using CargoHold sharding with compression
- Generates a manifest for selective extraction
- Optionally deletes the original archive
Examples:
Migrate archive with default settings
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded
Migrate and delete original
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --delete-original
Dry run to estimate migration
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --dry-run
Custom temp directory and shard count
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded
--temp-dir /mnt/fast-ssd --shard-count 16
Keep temp files for debugging
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --keep-temp
cargoship migrate SOURCE_ARCHIVE DESTINATION [flags]Options
--compression-level int Zstd compression level (1-22) (default 3)
--delete-original Delete original archive after successful migration
--dry-run Estimate migration without performing it
-h, --help help for migrate
--keep-temp Keep temporary files after migration
--quiet Disable progress display
-r, --region string AWS region (default "us-west-2")
--shard-count int Number of shards for CargoHold (1-100) (default 8)
--skip-validation Skip pre-flight validation checks
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
--temp-dir string Temporary directory for extraction (default: OS temp)Options inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume
Manage resumable uploads
Synopsis
Manage resumable uploads including listing, resuming, and cleaning up old states.
The resume command allows you to:
- List all resumable uploads with their progress
- Resume a specific interrupted upload by ID
- Clean up old state files to free disk space
Examples:
List all resumable uploads
cargoship resume list
Resume a specific upload
cargoship resume 20250115-143052-a3b4c5d6
Clean up state files older than 24 hours
cargoship resume clean --older-than 24h
Clean up all completed uploads
cargoship resume clean --completed
State files are stored in ~/.cargoship/state/ Each state file contains upload progress, configuration, and file hashes.
cargoship resume [upload-id] [flags]Options
-h, --help help for resumeOptions inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume list
List all resumable uploads
Synopsis
List all resumable uploads with their current progress.
Displays:
- Upload ID and status
- Source directory and S3 destination
- Progress (files and bytes completed)
- Time since upload started
- Estimated completion (if applicable)
State files are read from ~/.cargoship/state/
cargoship resume list [flags]Options
-h, --help help for listOptions inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume clean
Clean up old or completed upload states
Synopsis
Remove old state files to free up disk space.
By default, cleans up state files older than 24 hours. Use --completed to remove only fully completed uploads. Use --older-than to specify a custom age threshold.
Examples:
Clean states older than 24 hours (default)
cargoship resume clean
Clean states older than 1 week
cargoship resume clean --older-than 168h
Clean only completed uploads
cargoship resume clean --completed
State files are stored in ~/.cargoship/state/ Each file is typically 10-50 KB.
cargoship resume clean [flags]Options
--completed Clean only completed uploads
-h, --help help for clean
--older-than string Clean states older than duration (e.g., 24h, 7d, 168h)Options inherited from parent commands
--context string Override execution context (local, agent, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose output