Inspection & retrieval commands
Read what's in an upload, verify it, and get data back out. Guides: Inspecting, Verifying, Downloading, Restoring, Browsing.
Generated reference
Flag tables below are generated from the CLI and kept in sync by a drift check.
cargoship list
List files from a CargoShip upload using the manifest
Synopsis
Query and display files from a CargoShip upload without downloading archives.
The list command downloads the lightweight manifest.json.gz file (~30KB) from S3 and displays all uploaded files with their locations and metadata.
Examples:
List all files from an upload
cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234
List files matching a pattern
cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234 --pattern "*.log"
Verbose output with chunk and shard information
cargoship list --bucket my-bucket --upload-id 20231208-123456-abcd1234 --verbose
cargoship list [flags]Options
-b, --bucket string S3 bucket name (required)
-h, --help help for list
--pattern string Filter files by glob pattern (e.g., '*.log')
-p, --prefix string S3 prefix for upload (default: empty)
-r, --region string AWS region (default "us-west-2")
-u, --upload-id string Upload ID to query (required)
--verbose Show verbose output with full file detailsOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in outputcargoship info
Display metadata and statistics for a CargoShip upload
Synopsis
Display comprehensive metadata for a CargoShip upload without downloading data.
The info command downloads only the lightweight manifest (~30KB) and displays:
- Upload identification (ID, timestamp, source)
- File statistics (count, size, compression ratio)
- Shard distribution (per-shard statistics)
- Compression settings (algorithm, level, ratio achieved)
- Storage location (bucket, prefix, region)
This is useful for:
- Inspecting upload metadata before downloading
- Verifying upload completed successfully
- Planning selective extractions
- Auditing archived datasets
Examples:
Using S3 URL (Issue #98)
cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234
Using flags (backward compatibility)
cargoship info --bucket my-bucket --prefix cargoship --upload-id 20231208-123456-abcd1234
Show detailed per-shard statistics
cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --verbose
Output as JSON for scripting
cargoship info s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --json
cargoship info [s3://bucket/prefix/uploads/upload-id] [flags]Options
-b, --bucket string S3 bucket name (or provide S3 URL as argument)
-h, --help help for info
--json Output manifest as JSON for scripting
-p, --prefix string S3 prefix for upload (default: empty)
-r, --region string AWS region (default "us-west-2")
-u, --upload-id string Upload ID to inspect (or provide S3 URL as argument)
--verbose Show detailed per-shard statisticsOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in outputcargoship verify
Verify dataset integrity using manifest checksums
Synopsis
Verify the integrity of a CargoShip upload by validating the manifest and checksums.
The verify command performs comprehensive integrity checks:
- Downloads and validates manifest structure
- Verifies manifest consistency (shard counts, file counts, size totals)
- Checks for missing or corrupted metadata
- Validates checksum coverage (if present)
This is useful for:
- Ensuring upload completed successfully
- Detecting corrupted or incomplete uploads
- Validating data integrity before restore
- Compliance and audit requirements
Examples:
Verify using S3 URL
cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234
Verify using flags
cargoship verify --bucket my-bucket --prefix cargoship --upload-id 20231208-123456-abcd1234
Quick validation (metadata only, fast)
cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --quick
Verbose output with detailed errors
cargoship verify s3://my-bucket/cargoship/uploads/20231208-123456-abcd1234 --verbose
Exit Codes: 0 - All checks passed 1 - Verification failed (errors found)
cargoship verify [s3://bucket/prefix/uploads/upload-id] [flags]Options
-b, --bucket string S3 bucket name (or provide S3 URL as argument)
--deep Deep verification: re-download stored objects and recompute checksums against the manifest (data-level integrity)
-h, --help help for verify
-p, --prefix string S3 prefix for upload (default: empty)
--quick Quick validation (metadata only, fast)
-r, --region string AWS region (default "us-west-2")
-u, --upload-id string Upload ID to verify (or provide S3 URL as argument)
--verbose Show detailed error and warning informationOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in outputcargoship access-check
Report the access-control posture of a target bucket
Synopsis
Report whether the bucket CargoShip writes to (or reads from) is exposed: public access, wildcard-principal grants in the bucket policy or ACL, object ownership, and the scoping of the default SSE-KMS key policy.
This is a read-only posture report — it makes no changes and always exits 0 when it can reach the bucket (findings are informational). It is deliberately scoped to user/group access controls, not a general security audit.
Each check degrades gracefully: if the calling principal lacks permission for a check, that check reports "unknown" with the IAM action it needs, and the rest of the report still runs. Checks are bucket-level (a prefix is recorded for context but S3 policy/ACL/BPA/ownership apply to the whole bucket).
Examples:
Report on a bucket/prefix
cargoship access-check s3://my-bucket/archives
Using flags, with a specific profile and region
cargoship access-check --bucket my-bucket --region us-west-2 --profile prod
Machine-readable output
cargoship access-check s3://my-bucket/archives --format json
Inspect a specific KMS key's policy instead of the bucket default
cargoship access-check s3://my-bucket/archives --kms-key-id alias/cargoship
Exit Codes: 0 - Report produced (regardless of findings) 1 - Could not reach the bucket (missing/invalid credentials, no such bucket) 2 - Usage error
cargoship access-check [s3://bucket/prefix] [flags]Options
-b, --bucket string S3 bucket name
--format string output format: table or json (default "table")
-h, --help help for access-check
--kms-key-id string KMS key ID/ARN/alias to inspect (default: the bucket's SSE-KMS key)
-p, --prefix string S3 prefix (for context; checks are bucket-level)
--profile string AWS profile
-r, --region string AWS region (auto-detected if omitted)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship balance
Analyze shard balance for an uploaded dataset
Synopsis
Analyze and rebalance shard distribution for a CargoShip upload.
This command downloads the manifest for an upload and analyzes the distribution of files across shards. It identifies imbalanced shards and can automatically rebalance them by redistributing files.
An upload is considered imbalanced if the largest shard is more than 2x the average shard size (configurable with --threshold).
Examples:
Analyze shard balance only (read-only)
cargoship balance s3://my-bucket/data/uploads/20240101-abc123
Show what would be rebalanced without executing
cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --dry-run
Execute rebalancing (redistributes files across shards)
cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --execute
Check with custom threshold (3x instead of 2x)
cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --threshold 3.0
Output as JSON
cargoship balance s3://my-bucket/data/uploads/20240101-abc123 --format json
cargoship balance <s3://bucket/prefix/uploads/upload-id> [flags]Options
--dry-run Show rebalancing plan without executing
--execute Execute rebalancing (modifies data)
-f, --format string Output format (table, json) (default "table")
-h, --help help for balance
--profile string AWS profile to use
--region string AWS region
--threshold float Imbalance threshold (max/avg ratio) (default 2)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship download
Download and extract files from a CargoShip upload
Synopsis
Download and selectively extract files from a CargoShip upload using the manifest.
The download command provides efficient selective extraction by:
- Downloading the lightweight manifest first (~30KB)
- Identifying which chunks contain the requested files
- Only downloading and extracting necessary chunks (10x faster than full download)
Selective extraction options: --pattern : Glob pattern matching (e.g., ".log", "data/.csv") --files : Comma-separated list of exact file paths --shard-ids : Only download specific shard IDs (0-7 by default)
Examples:
Download all files from an upload
cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored
Download files matching a pattern
cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./logs
--pattern "*.log"
Download specific files
cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./reports
--files "data/report.csv,data/summary.csv"
Download specific shards only
cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored
--shard-ids 0,2,4
Dry run to see what would be downloaded
cargoship download s3://my-bucket/uploads/20231208-123456-abcd1234 ./restored
--pattern "*.csv" --dry-run
cargoship download S3_URL OUTPUT_DIR [flags]Options
--dry-run Show what would be downloaded without actually downloading
--files strings Comma-separated list of exact file paths to download
--flatten Write downloaded files by basename into the output dir instead of recreating their directory structure
-h, --help help for download
--no-verify Skip restore-time checksum verification (faster, but won't detect corrupted stored data)
--pattern string Filter files by glob pattern (e.g., '*.log')
-r, --region string AWS region (default "us-west-2")
--shard-ids ints Comma-separated list of shard IDs to download (0-7)
--verbose Show verbose output (list each file as extracted)
--workers int Number of parallel download workers (future use) (default 4)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in outputcargoship restore
Restore specific files from a CargoShip archive using hash, path, commit, or DVC stage
Synopsis
Restore targeted files from a CargoShip archive without downloading the whole dataset.
Restoration modes (pick one or combine --file with others): --hash : Restore a single file by its MD5 content hash --file : Restore one or more exact file paths --git-commit : Restore all files from a specific git commit --dvc-stage : Restore all files produced by a DVC pipeline stage --all : Restore every file in the upload (disaster recovery)
Glacier/Deep Archive support: --tier : Retrieval tier: expedited (1-5 min), standard (3-5 h), bulk (5-12 h) --wait : Block until Glacier restoration completes before downloading --dry-run : Show what would be restored (size, cost) without downloading
Budget controls: --max-restore-cost : Abort if estimated retrieval cost exceeds this USD limit
Examples:
Restore a file by its MD5 hash
cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--hash d8e8fca2dc0f896fd7cb4cb0031ba249
Restore specific files by path
cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--file data/train.csv --file models/model.pkl
Restore all files from a DVC pipeline stage
cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage preprocess
Restore an ENTIRE upload — disaster recovery. Needs only read access + the
decryption key, not the box (or writer id) that produced it.
cargoship restore s3://my-bucket/prefix/writers/lab-nas-1/uploads/20240101-abc123 ./out --all
Restore from Glacier with standard retrieval tier, wait for completion
cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage train --tier standard --wait
Dry-run: show estimated cost without restoring
cargoship restore s3://my-bucket/uploads/20240101-abc123 ./out
--dvc-stage train --dry-run
cargoship restore S3_URL OUTPUT_DIR [flags]Options
--all Restore every file recorded in the upload's manifest (disaster recovery); cannot be combined with --hash/--file/--git-commit/--dvc-stage
--as-of string With --dataset-id: restore the newest version at or before this date (YYYY-MM-DD)
--cache-gb int LRU chunk cache size in GB (0 = default 10 GB) (default 10)
--dataset-id string Restore a version of this dataset (S3_URL is then the bucket/prefix); see 'cargoship dataset list'
--dry-run Show what would be restored without downloading
--dvc-stage string Restore all files produced by this DVC pipeline stage
--file stringArray Exact file path(s) to restore (repeatable)
--flatten Write restored files by basename into the output dir instead of recreating their directory structure
--git-commit string Restore all files from this git commit SHA
--hash string MD5 content hash of the file to restore
-h, --help help for restore
--json Output restore statistics as JSON
--max-restore-cost float Abort if estimated retrieval cost exceeds this USD amount
--no-verify Skip restore-time checksum verification (faster, but won't detect corrupted stored data)
-r, --region string AWS region (default "us-east-1")
--restore-days int32 Days to keep Glacier restored copy available (default 7)
--tier string Glacier retrieval tier: expedited, standard (default), bulk
--version int With --dataset-id: the version ordinal to restore (default: latest)
--wait Block until Glacier restoration completes before downloadingOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship restore jobs
Manage queued Glacier restore jobs
Synopsis
List, check, download, and clean restore jobs created when Glacier/Deep Archive objects need time to be retrieved before they can be downloaded.
When 'cargoship restore' requests a Glacier restore without --wait, it saves a job to ~/.cargoship/restore-jobs/ and prints a job ID. Use these subcommands to track the job and trigger the download once the objects are ready.
Options
-h, --help help for jobsOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship restore jobs list
List all restore jobs
cargoship restore jobs list [flags]Options
-h, --help help for listOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship restore jobs check
Check Glacier restore status for pending jobs
Synopsis
Poll S3 for each pending job and mark jobs as 'ready' when all their chunks are accessible. If a job ID is given, only that job is checked.
cargoship restore jobs check [job-id] [flags]Options
-h, --help help for check
--job-id string Check only this specific job IDOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship restore jobs download
Download files from a ready restore job
Synopsis
Download the files for a restore job whose Glacier restore has completed. The job must be in 'ready' status (run 'restore jobs check' first if unsure).
cargoship restore jobs download <job-id> [flags]Options
--cache-gb int LRU chunk cache size in GB (default 10)
-h, --help help for downloadOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship restore jobs clean
Remove completed and failed restore jobs
Synopsis
Delete completed and failed restore jobs older than the given duration (default: 24h).
cargoship restore jobs clean [flags]Options
-h, --help help for clean
--older-than string Remove jobs older than this duration (e.g. 72h, 7d) (default "24h")Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship browse
Interactively browse and restore files from a CargoShip archive
Synopsis
Open an interactive terminal UI to browse manifest contents and select files for restore.
Navigation: ↑/↓ Navigate file list space Toggle selection on highlighted file enter Confirm restore of selected files / Enter incremental search mode d Cycle DVC stage filter g Cycle git commit filter a Select all visible files c Clear selection q / ctrl+c Quit without restoring
Glacier/Deep Archive: --tier Retrieval tier if files are archived: expedited, standard (default), bulk --wait Block until Glacier restoration completes before downloading --max-restore-cost Abort restore if estimated retrieval cost exceeds this USD limit
Examples:
Open the interactive browser
cargoship browse s3://my-bucket/uploads/20240101-abc123 ./restored
Use a larger cache for big datasets
cargoship browse s3://my-bucket/uploads/20240101-abc123 ./restored --cache-gb 20
Restore from Glacier with standard tier, wait for completion
cargoship browse s3://my-bucket/uploads/20240101-abc123 --tier standard --wait
cargoship browse S3_URL [OUTPUT_DIR] [flags]Options
--as-of string With --dataset-id: browse the newest version at or before this date (YYYY-MM-DD)
--cache-gb int LRU chunk cache size in GB (default 10)
--dataset-id string Browse a version of this dataset (S3_URL is then the bucket/prefix); see 'cargoship dataset list'
-h, --help help for browse
--max-restore-cost float Abort if estimated retrieval cost exceeds this USD amount
-r, --region string AWS region (default "us-east-1")
--restore-days int32 Days to keep Glacier restored copy available (default 7)
--tier string Glacier retrieval tier: expedited, standard (default), bulk
--version int With --dataset-id: the version ordinal to browse (default: latest)
--wait Block until Glacier restoration completes before downloadingOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship shell
Navigate a CargoShip archive or start an interactive shell
Synopsis
When called with an S3 URL, opens an interactive filesystem shell for browsing and inspecting a CargoShip archive without downloading files.
cargoship shell s3://my-bucket/uploads/20240101-abc123
When called without arguments, starts the generic CargoShip REPL.
Archive shell commands: ls [path] List files and directories cd <dir> Change current directory pwd Print current directory cat <file> Stream file content to stdout head <file> [n] Print first n lines (default 10) stat <file> Show file metadata (size, hash, chunk, DVC stage, git commit) find <pattern> Find files by glob pattern (e.g. .csv, data/.parquet) stage list List all DVC pipeline stages and their file counts stage <name> List files belonging to a DVC stage get <file> [dst] Extract file to a local path (default: current directory) help Show this help exit / quit Exit the shell
cargoship shell [S3_URL] [flags]Options
--cache-gb int LRU chunk cache size in GB (default 10)
-h, --help help for shell
-r, --region string AWS region (default "us-east-1")Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship dataset
Inspect dataset versions (incremental sync history)
Synopsis
Inspect and manage the version history that incremental sync builds under an S3 prefix.
A "dataset" is a chain of manifests linked by their previous-manifest reference; each 'cargoship sync' adds a version. 'list', 'versions', and 'diff' are read-only; 'prune' performs destructive, irreversible garbage collection.
Options
-h, --help help for datasetOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship dataset list
List datasets under an S3 prefix
cargoship dataset list S3_URL [flags]Options
-h, --help help for list
--json Output as JSON
-r, --region string AWS region (default "us-west-2")Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship dataset versions
List the versions of one dataset, newest first
cargoship dataset versions S3_URL --dataset-id ID [flags]Options
--dataset-id string Dataset ID to list versions of (from 'cargoship dataset list')
-h, --help help for versions
--json Output as JSON
-r, --region string AWS region (default "us-west-2")Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship dataset diff
Compare two versions of a dataset (added/removed/modified files)
cargoship dataset diff S3_URL --dataset-id ID --from V --to V [flags]Options
--dataset-id string Dataset ID (from 'cargoship dataset list')
--from string Baseline version: a number (v2 → 2) or a date (YYYY-MM-DD)
-h, --help help for diff
--json Output as JSON
-r, --region string AWS region (default "us-west-2")
--to string Target version: a number or date; default HEAD (latest)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship dataset prune
Delete old versions of a dataset, keeping the newest N (garbage collection)
Synopsis
Reclaim storage by deleting old versions of a dataset, keeping the newest N.
Only objects no kept version still references are removed. The oldest kept version is first rewritten self-contained (compaction) so the pruned versions' manifests can be removed without breaking it; the original is backed up to a ".pre-compact.bak" object first, and the rewrite is verified before anything is deleted.
WARNING: deletion is IRREVERSIBLE. Use --dry-run to preview. Encrypted-manifest datasets and mixed direct/chunked chains are not yet supported.
cargoship dataset prune S3_URL --dataset-id ID --keep-last N [flags]Options
--dataset-id string Dataset ID to prune (from 'cargoship dataset list')
--dry-run Preview what would be pruned without deleting
--force Skip the confirmation prompt
-h, --help help for prune
--keep-last int Keep the newest N versions; delete older ones (required, >= 1)
-r, --region string AWS region (default "us-west-2")Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose output