CargoShip

Appearance

Inspection & retrieval commands ​

Read what's in an upload, verify it, and get data back out. Guides: Inspecting, Verifying, Downloading, Restoring, Browsing.

Generated reference

Flag tables below are generated from the CLI and kept in sync by a drift check.

cargoship list ​

List files from a CargoShip upload using the manifest

Synopsis ​

Query and display files from a CargoShip upload without downloading archives.

The list command downloads the lightweight manifest.json.gz file (~30KB) from S3 and displays all uploaded files with their locations and metadata.

Examples:

List all files from an upload ​

cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234

List files matching a pattern ​

cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234 --pattern "*.log"

Verbose output with chunk and shard information ​

cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234 --verbose

cargoship list [flags]

Options ​

  -b, --bucket string      S3 bucket name (required)
  -h, --help               help for list
      --pattern string     Filter files by glob pattern (e.g., '*.log')
  -p, --prefix string      S3 prefix for upload (default: empty)
  -r, --region string      AWS region (default "us-west-2")
  -u, --upload-id string   Upload ID to query (required)
      --verbose            Show verbose output with full file details

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output

cargoship info ​

Display metadata and statistics for a CargoShip upload

Synopsis ​

Display comprehensive metadata for a CargoShip upload without downloading data.

The info command downloads only the lightweight manifest (~30KB) and displays:

  • Upload identification (ID, timestamp, source)
  • File statistics (count, size, compression ratio)
  • Shard distribution (per-shard statistics)
  • Compression settings (algorithm, level, ratio achieved)
  • Storage location (bucket, prefix, region)

This is useful for:

  • Inspecting upload metadata before downloading
  • Verifying upload completed successfully
  • Planning selective extractions
  • Auditing archived datasets

Examples:

Using S3 URL (Issue #98) ​

cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234

Using flags (backward compatibility) ​

cargoship info --bucket my-bucket --prefix cargoship --upload-id 20231208-123456-abcd1234

Show detailed per-shard statistics ​

cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --verbose

Output as JSON for scripting ​

cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --json

cargoship info [s3://bucket/prefix/uploads/upload-id] [flags]

Options ​

  -b, --bucket string      S3 bucket name (or provide S3 URL as argument)
  -h, --help               help for info
      --json               Output manifest as JSON for scripting
  -p, --prefix string      S3 prefix for upload (default: empty)
  -r, --region string      AWS region (default "us-west-2")
  -u, --upload-id string   Upload ID to inspect (or provide S3 URL as argument)
      --verbose            Show detailed per-shard statistics

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output

cargoship verify ​

Verify dataset integrity using manifest checksums

Synopsis ​

Verify the integrity of a CargoShip upload by validating the manifest and checksums.

The verify command performs comprehensive integrity checks:

  • Downloads and validates manifest structure
  • Verifies manifest consistency (shard counts, file counts, size totals)
  • Checks for missing or corrupted metadata
  • Validates checksum coverage (if present)

This is useful for:

  • Ensuring upload completed successfully
  • Detecting corrupted or incomplete uploads
  • Validating data integrity before restore
  • Compliance and audit requirements

Examples:

Verify using S3 URL ​

cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234

Verify using flags ​

cargoship verify --bucket my-bucket --prefix cargoship --upload-id 20231208-123456-abcd1234

Quick validation (metadata only, fast) ​

cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --quick

Verbose output with detailed errors ​

cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --verbose

Exit Codes: 0 - All checks passed 1 - Verification failed (errors found)

cargoship verify [s3://bucket/prefix/uploads/upload-id] [flags]

Options ​

  -b, --bucket string      S3 bucket name (or provide S3 URL as argument)
      --deep               Deep verification: re-download stored objects and recompute checksums against the manifest (data-level integrity)
  -h, --help               help for verify
  -p, --prefix string      S3 prefix for upload (default: empty)
      --quick              Quick validation (metadata only, fast)
  -r, --region string      AWS region (default "us-west-2")
  -u, --upload-id string   Upload ID to verify (or provide S3 URL as argument)
      --verbose            Show detailed error and warning information

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output

cargoship balance ​

Analyze shard balance for an uploaded dataset

Synopsis ​

Analyze and rebalance shard distribution for a CargoShip upload.

This command downloads the manifest for an upload and analyzes the distribution of files across shards. It identifies imbalanced shards and can automatically rebalance them by redistributing files.

An upload is considered imbalanced if the largest shard is more than 2x the average shard size (configurable with --threshold).

Examples:

Analyze shard balance only (read-only) ​

cargoship balance s3://my-bucket/data/uploads/20240101-abc123

Show what would be rebalanced without executing ​

cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --dry-run

Execute rebalancing (redistributes files across shards) ​

cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --execute

Check with custom threshold (3x instead of 2x) ​

cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --threshold 3.0

Output as JSON ​

cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --format json

cargoship balance <s3://bucket/prefix/uploads/upload-id> [flags]

Options ​

      --dry-run           Show rebalancing plan without executing
      --execute           Execute rebalancing (modifies data)
  -f, --format string     Output format (table, json) (default "table")
  -h, --help              help for balance
      --profile string    AWS profile to use
      --region string     AWS region
      --threshold float   Imbalance threshold (max/avg ratio) (default 2)

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship download ​

Download and extract files from a CargoShip upload

Synopsis ​

Download and selectively extract files from a CargoShip upload using the manifest.

The download command provides efficient selective extraction by:

  1. Downloading the lightweight manifest first (~30KB)
  2. Identifying which chunks contain the requested files
  3. Only downloading and extracting necessary chunks (10x faster than full download)

Selective extraction options: --pattern : Glob pattern matching (e.g., ".log", "data/.csv") --files : Comma-separated list of exact file paths --shard-ids : Only download specific shard IDs (0-7 by default)

Examples:

Download all files from an upload ​

cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored

Download files matching a pattern ​

cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./logs
--pattern "*.log"

Download specific files ​

cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./reports
--files "data/report.csv,data/summary.csv"

Download specific shards only ​

cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored
--shard-ids 0,2,4

Dry run to see what would be downloaded ​

cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored
--pattern "*.csv" --dry-run

cargoship download S3_URL OUTPUT_DIR [flags]

Options ​

      --dry-run          Show what would be downloaded without actually downloading
      --files strings    Comma-separated list of exact file paths to download
      --flatten          Write downloaded files by basename into the output dir instead of recreating their directory structure
  -h, --help             help for download
      --no-verify        Skip restore-time checksum verification (faster, but won't detect corrupted stored data)
      --pattern string   Filter files by glob pattern (e.g., '*.log')
  -r, --region string    AWS region (default "us-west-2")
      --shard-ids ints   Comma-separated list of shard IDs to download (0-7)
      --verbose          Show verbose output (list each file as extracted)
      --workers int      Number of parallel download workers (future use) (default 4)

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output

cargoship restore ​

Restore specific files from a CargoShip archive using hash, path, commit, or DVC stage

Synopsis ​

Restore targeted files from a CargoShip archive without downloading the whole dataset.

Restoration modes (pick one or combine --file with others): --hash : Restore a single file by its MD5 content hash --file : Restore one or more exact file paths --git-commit : Restore all files from a specific git commit --dvc-stage : Restore all files produced by a DVC pipeline stage

Glacier/Deep Archive support: --tier : Retrieval tier: expedited (1-5 min), standard (3-5 h), bulk (5-12 h) --wait : Block until Glacier restoration completes before downloading --dry-run : Show what would be restored (size, cost) without downloading

Budget controls: --max-restore-cost : Abort if estimated retrieval cost exceeds this USD limit

Examples:

Restore a file by its MD5 hash ​

cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--hash d8e8fca2dc0f896fd7cb4cb0031ba249

Restore specific files by path ​

cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--file data/train.csv --file models/model.pkl

Restore all files from a DVC pipeline stage ​

cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage preprocess

Restore from Glacier with standard retrieval tier, wait for completion ​

cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage train --tier standard --wait

Dry-run: show estimated cost without restoring ​

cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage train --dry-run

cargoship restore S3_URL OUTPUT_DIR [flags]

Options ​

      --cache-gb int             LRU chunk cache size in GB (0 = default 10 GB) (default 10)
      --dry-run                  Show what would be restored without downloading
      --dvc-stage string         Restore all files produced by this DVC pipeline stage
      --file stringArray         Exact file path(s) to restore (repeatable)
      --flatten                  Write restored files by basename into the output dir instead of recreating their directory structure
      --git-commit string        Restore all files from this git commit SHA
      --hash string              MD5 content hash of the file to restore
  -h, --help                     help for restore
      --json                     Output restore statistics as JSON
      --max-restore-cost float   Abort if estimated retrieval cost exceeds this USD amount
      --no-verify                Skip restore-time checksum verification (faster, but won't detect corrupted stored data)
  -r, --region string            AWS region (default "us-east-1")
      --restore-days int32       Days to keep Glacier restored copy available (default 7)
      --tier string              Glacier retrieval tier: expedited, standard (default), bulk
      --wait                     Block until Glacier restoration completes before downloading

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship restore jobs ​

Manage queued Glacier restore jobs

Synopsis ​

List, check, download, and clean restore jobs created when Glacier/Deep Archive objects need time to be retrieved before they can be downloaded.

When 'cargoship restore' requests a Glacier restore without --wait, it saves a job to ~/.cargoship/restore-jobs/ and prints a job ID. Use these subcommands to track the job and trigger the download once the objects are ready.

Options ​

  -h, --help   help for jobs

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship restore jobs list ​

List all restore jobs

cargoship restore jobs list [flags]

Options ​

  -h, --help   help for list

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship restore jobs check ​

Check Glacier restore status for pending jobs

Synopsis ​

Poll S3 for each pending job and mark jobs as 'ready' when all their chunks are accessible. If a job ID is given, only that job is checked.

cargoship restore jobs check [job-id] [flags]

Options ​

  -h, --help            help for check
      --job-id string   Check only this specific job ID

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship restore jobs download ​

Download files from a ready restore job

Synopsis ​

Download the files for a restore job whose Glacier restore has completed. The job must be in 'ready' status (run 'restore jobs check' first if unsure).

cargoship restore jobs download <job-id> [flags]

Options ​

      --cache-gb int   LRU chunk cache size in GB (default 10)
  -h, --help           help for download

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship restore jobs clean ​

Remove completed and failed restore jobs

Synopsis ​

Delete completed and failed restore jobs older than the given duration (default: 24h).

cargoship restore jobs clean [flags]

Options ​

  -h, --help                help for clean
      --older-than string   Remove jobs older than this duration (e.g. 72h, 7d) (default "24h")

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship browse ​

Interactively browse and restore files from a CargoShip archive

Synopsis ​

Open an interactive terminal UI to browse manifest contents and select files for restore.

Navigation: ↑/↓ Navigate file list space Toggle selection on highlighted file enter Confirm restore of selected files / Enter incremental search mode d Cycle DVC stage filter g Cycle git commit filter a Select all visible files c Clear selection q / ctrl+c Quit without restoring

Glacier/Deep Archive: --tier Retrieval tier if files are archived: expedited, standard (default), bulk --wait Block until Glacier restoration completes before downloading --max-restore-cost Abort restore if estimated retrieval cost exceeds this USD limit

Examples:

Open the interactive browser ​

cargoship browse s3://my-bucket/uploads/20240101-abc123 ./restored

Use a larger cache for big datasets ​

cargoship browse s3://my-bucket/uploads/20240101-abc123 ./restored --cache-gb 20

Restore from Glacier with standard tier, wait for completion ​

cargoship browse s3://my-bucket/uploads/20240101-abc123 --tier standard --wait

cargoship browse S3_URL [OUTPUT_DIR] [flags]

Options ​

      --cache-gb int             LRU chunk cache size in GB (default 10)
  -h, --help                     help for browse
      --max-restore-cost float   Abort if estimated retrieval cost exceeds this USD amount
  -r, --region string            AWS region (default "us-east-1")
      --restore-days int32       Days to keep Glacier restored copy available (default 7)
      --tier string              Glacier retrieval tier: expedited, standard (default), bulk
      --wait                     Block until Glacier restoration completes before downloading

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship shell ​

Navigate a CargoShip archive or start an interactive shell

Synopsis ​

When called with an S3 URL, opens an interactive filesystem shell for browsing and inspecting a CargoShip archive without downloading files.

cargoship shell s3://my-bucket/uploads/20240101-abc123

When called without arguments, starts the generic CargoShip REPL.

Archive shell commands: ls [path] List files and directories cd <dir> Change current directory pwd Print current directory cat <file> Stream file content to stdout head <file> [n] Print first n lines (default 10) stat <file> Show file metadata (size, hash, chunk, DVC stage, git commit) find <pattern> Find files by glob pattern (e.g. .csv, data/.parquet) stage list List all DVC pipeline stages and their file counts stage <name> List files belonging to a DVC stage get <file> [dst] Extract file to a local path (default: current directory) help Show this help exit / quit Exit the shell

cargoship shell [S3_URL] [flags]

Options ​

      --cache-gb int    LRU chunk cache size in GB (default 10)
  -h, --help            help for shell
  -r, --region string   AWS region (default "us-east-1")

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output