CargoShip

Appearance

Uploading & sync commands ​

Commands that write data to S3. For guidance on which to use, see upload vs. create upload; for a walkthrough, Uploading data.

Generated reference

Flag tables below are generated from the CLI and kept in sync by a drift check.

cargoship upload ​

Upload directory to S3 with CargoHold sharding

Synopsis ​

Upload a directory to S3 using CargoHold's intelligent sharding system.

CargoHold divides large datasets into multiple shards for parallel uploads, providing:

  • Intelligent shard distribution (round-robin, hash, size, type, or directory)
  • Content-aware zstd compression, with an optional fixed-level override
  • Parallel uploads for maximum throughput
  • Automatic manifest generation for easy restore
  • Progress tracking

Shard Count: CargoShip automatically selects between 4 and 32 shards when --shard-count is 0 (the default). If automatic selection fails, it falls back to 8.

Shard Strategies: round-robin - Distribute by chunk order (even shards, cheapest; default) hash - Hash of chunk contents (stable across runs) size - Least-loaded shard by bytes (evens out uneven chunk sizes) type - Group chunks by predominant content type directory - Group chunks by common directory prefix

Compression: By default CargoShip picks a zstd level per chunk from the content it contains — level 1 for already-compressed data, up to level 9 for source code. Passing --compression-level overrides that and pins every chunk to one level. Note that zstd has only four internal levels, so values map in bands (1-2, 3-5, 6-9, 10+); the effective setting is reported at upload start.

Examples:

Upload with defaults (auto shard count, round-robin, content-aware compression) ​

cargoship upload /data s3://my-bucket/dataset

Upload with custom shard count and strategy ​

cargoship upload /data s3://my-bucket/dataset --shard-count 20 --shard-strategy size

Pin every chunk to maximum compression (disables content-aware selection) ​

cargoship upload /data s3://my-bucket/dataset --compression-level 19

Quiet mode (no progress display) ​

cargoship upload /data s3://my-bucket/dataset --quiet

cargoship upload SOURCE_DIR DESTINATION [flags]

Options ​

      --auto-tier                        Enable automatic storage tier selection based on file access time
  -b, --bucket string                    S3 bucket name (or use s3:// URL in DESTINATION)
      --check-access                     Report the target bucket's access-control posture before uploading (see the 'access-check' command)
      --compression-level int            Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
      --congestion-control string        Congestion control algorithm: bbr, cubic, auto (default "auto")
      --direct-upload                    Enable direct upload mode (bypasses archiving/compression for small files)
      --direct-upload-threshold-mb int   Max total size in MB for auto direct upload (default: 500) (default 500)
      --direct-upload-workers int        Worker count for direct upload (default: 256) (default 256)
      --disable-staging                  Disable adaptive staging (reduces memory usage)
      --dvc-auto                         Auto-discover DVC stages from dvc.yaml and annotate each file entry with its stage name
      --dvc-cache-dir string             Local DVC cache directory (recorded in manifest; default: .dvc/cache) (default ".dvc/cache")
      --dvc-output-dir string            Directory to write .dvc files (default: source directory)
      --dvc-stage string                 DVC pipeline stage name to extract provenance from (reads dvc.yaml + dvc.lock)
      --enable-dedup                     Enable cross-shard file deduplication (10-30% space savings for redundant datasets)
      --encrypt-manifest                 Encrypt manifest with KMS envelope encryption (requires --kms-key-id)
      --fail-on string                   With --check-access, abort if the worst finding is at least this severe: none, unknown, warn, critical (warn/critical do NOT trip on checks that couldn't be evaluated — use 'unknown' to also fail when a check was denied) (default "none")
      --force-direct-upload              Force direct upload regardless of thresholds (for benchmarking)
      --force-restart                    Ignore saved state and start fresh upload (bypasses resume detection)
      --frame-size string                Cut compressed chunks into random-access zstd frames every N bytes (e.g. 16MiB, 64MB); 0 disables framing (default "16MiB")
      --generate-dvc-files               Generate DVC sidecar .dvc files after upload
      --git-metadata                     Embed Git repository metadata (commit, branch, tag, remote) in the manifest
  -h, --help                             help for upload
      --ignore-budget                    Skip the pre-upload budget/volume cap check (#629)
      --incremental                      Enable incremental sync: only upload new or changed files
      --kms-key-id string                AWS KMS key ID or ARN for encryption (data chunks encrypted with SSE-KMS)
      --no-file-checksums                Disable per-file content checksums (faster uploads, but 'verify --deep' can't confirm per-file integrity)
      --optimization                     Enable optimization features (BBR/CUBIC, adaptive staging, BDP) (default true)
      --prev-manifest string             Path to previous manifest JSON (or .json.gz) for incremental sync
      --project string                   Project ID for cost tracking (e.g. 'dvc_cache' for DVC remotes)
      --prometheus-addr string           Prometheus metrics HTTP address (e.g., :9090)
      --quiet                            Disable progress display
  -r, --region string                    AWS region (default "us-west-2")
      --resume                           Resume an interrupted upload of this source→destination without prompting (skips already-uploaded chunks)
      --shard-count int                  Shards for parallel uploads: 0 auto-selects 4-32 (falls back to 8), or set 4-32 manually
      --shard-strategy string            Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
      --storage-class string             S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
      --tag stringArray                  Custom tag in key=value format, repeatable (e.g. --tag dvc_cache=true --tag env=prod)
      --tier-archive-days int            Days since access to consider 'archive' (DEEP_ARCHIVE) (default 180)
      --tier-cold-days int               Days since access to consider 'cold' (GLACIER) (default 90)
      --tier-hot-days int                Days since access to consider 'hot' (STANDARD) (default 30)
      --tier-max string                  Maximum storage tier (STANDARD, STANDARD_IA, GLACIER, DEEP_ARCHIVE) - prevents automatic selection of more restrictive tiers
      --tier-strategy string             Tier chunking strategy (requires --auto-tier):
                                           youngest-file: Conservative strategy - assigns tier based on youngest file per chunk (default)
                                           tier-aware:    Optimal cost - groups files by tier before chunking (30-60% savings)

                                         ⚠️  WARNING: tier-aware uses GLACIER/DEEP_ARCHIVE with cost implications:
                                           • GLACIER: 90-day minimum storage ($0.004/GB-month, $0.01/GB retrieval, 3-5hr access)
                                           • DEEP_ARCHIVE: 180-day minimum ($0.00099/GB-month, $0.02/GB retrieval, 12hr access)
                                           • Early deletion penalties apply if removed before minimum duration
                                           • Best for long-term archives accessed <1x per year

                                           See: https://github.com/scttfrdmn/cargoship/issues/168 (default "youngest-file")
      --tracing                          Enable distributed tracing
      --tracing-endpoint string          Tracing endpoint URL (required for jaeger/otlp exporters)
      --tracing-exporter string          Tracing exporter: stdout, jaeger, otlp, none (default "stdout")
      --tracing-sample-rate float        Trace sampling rate (0.0-1.0, default: 1.0 = 100%) (default 1)
      --transporter string               S3 transporter type: basic, staging, adaptive, optimized, none (default "staging")
      --writer-id string                 Writer identity for fleet isolation: objects go under writers/<id>/. 'auto' derives a stable per-host id; empty = single-writer layout (default)
  -y, --yes                              Skip confirmation prompts (auto-accept warnings)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship create ​

Create something!

cargoship create [flags]

Options ​

  -d, --destination string   Directory to write files in to. Defaults to the current directory
  -h, --help                 help for create

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship create upload ​

Upload directories to S3 using streaming pipeline

Synopsis ​

Upload directories to S3 using the high-performance streaming pipeline.

This command replaces the legacy suitcase/rclone system with a modern streaming architecture that provides:

  • Real-time progress tracking with beautiful TUI
  • Multi-prefix S3 parallel uploads (8x throughput improvement)
  • Zero local disk usage (streaming directly to S3)
  • Automatic compression (zstd)
  • Intelligent chunking and sharding
cargoship create upload SOURCE_DIR... [flags]

Examples ​

  # Upload a directory with progress tracking
  cargoship create upload /path/to/data --bucket my-bucket

  # Upload with custom prefix
  cargoship create upload /path/to/data --bucket my-bucket --prefix backups/2025-12-05

  # Quiet mode (no progress display)
  cargoship create upload /path/to/data --bucket my-bucket --quiet

  # JSON progress output (for scripts)
  cargoship create upload /path/to/data --bucket my-bucket --progress-format json

Options ​

      --bucket string                S3 bucket name (required)
      --chunk-size-mb int            Target chunk size in MB (0 = adaptive) (default 200)
      --cleanup-on-failure           Automatically delete partial uploads on error (default true)
  -h, --help                         help for upload
      --http2                        Enable HTTP/2 (default true)
      --http2-max-streams int        Max concurrent HTTP/2 streams per connection (default 250)
      --idle-conn-timeout duration   Idle connection timeout (default 5m0s)
      --max-idle-conns int           Max idle connections per host (default 100)
      --network-profile string       Network tuning profile: default, aggressive, conservative (default "default")
      --no-cleanup                   Disable automatic cleanup on failure (for debugging)
      --prefix string                S3 key prefix (optional)
      --progress-format string       Progress output format: tui, json, text (default "tui")
      --quiet                        Disable progress display
      --region string                AWS region (default "us-west-2")
      --resume                       Resume a previous incomplete upload
      --shards int                   Number of S3 prefix shards for parallel uploads (default 8)
      --skip-existing                Skip chunks that already exist in S3 (HeadObject check)
      --storage-class string         S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
      --upload-id string             Upload ID to resume (auto-detect if not specified)
      --workers int                  Workers per stage (scanner, archiver, uploader) (default 4)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
  -d, --destination string           Directory to write files in to. Defaults to the current directory
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship sync ​

Incrementally sync directory to S3 (only upload new/changed files)

Synopsis ​

Incrementally sync a local directory to S3 by uploading only new or modified files.

The sync command provides efficient incremental backups by:

  1. Downloading the latest manifest for the source path (if exists)
  2. Comparing local filesystem state against the manifest
  3. Uploading only files that are new or have changed
  4. Creating a new manifest that references the previous one

First sync uploads everything (like 'upload' command). Subsequent syncs only upload changed files, saving time and bandwidth.

Change detection compares SIZE and MODIFICATION TIME against the previous manifest. Content is not hashed, so an edit that preserves both size and mtime is not detected. Use --force for a full re-upload when you need certainty.

Examples:

First sync: uploads all files ​

cargoship sync /home/photos s3://my-bucket/backups

Second sync: only uploads new/changed photos ​

cargoship sync /home/photos s3://my-bucket/backups

Dry run to see what would be synced ​

cargoship sync /home/photos s3://my-bucket/backups --dry-run

Force full sync (ignore previous manifest) ​

cargoship sync /data s3://my-bucket/backups --force

cargoship sync SOURCE_DIR S3_URL [flags]

Options ​

      --checksum                NOT IMPLEMENTED (#678): rejected if set. Change detection is size+mtime; use --force for a full re-upload
      --compression-level int   Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
      --dry-run                 Show what would be synced without uploading
      --force                   Force full sync (ignore previous manifest)
  -h, --help                    help for sync
      --no-file-checksums       Disable per-file content checksums (faster, but 'verify --deep' can't confirm per-file integrity)
  -q, --quiet                   Quiet mode (minimal output)
  -r, --region string           AWS region (default "us-west-2")
      --shard-count int         Number of shards for parallel uploads (1-100) (default 10)
      --shard-strategy string   Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
      --storage-class string    S3 storage class (STANDARD, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
      --track-deletes           Track deleted files in manifest
      --writer-id string        Writer identity for fleet isolation: objects go under writers/<id>/. 'auto' derives a stable per-host id; empty = single-writer layout (default)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship migrate ​

Convert traditional archives to CargoHold sharded format

Synopsis ​

Download a traditional tar.zst archive from S3 and re-upload using CargoHold's intelligent sharding system.

The migrate command:

  1. Downloads the traditional archive from S3
  2. Extracts files to a temporary location
  3. Re-uploads using CargoHold sharding with compression
  4. Generates a manifest for selective extraction
  5. Optionally deletes the original archive

Examples:

Migrate archive with default settings ​

cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded

Migrate and delete original ​

cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --delete-original

Dry run to estimate migration ​

cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --dry-run

Custom temp directory and shard count ​

cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded
--temp-dir /mnt/fast-ssd --shard-count 16

Keep temp files for debugging ​

cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --keep-temp

cargoship migrate SOURCE_ARCHIVE DESTINATION [flags]

Options ​

      --compression-level int   Zstd compression level (1-22) (default 3)
      --delete-original         Delete original archive after successful migration
      --dry-run                 Estimate migration without performing it
  -h, --help                    help for migrate
      --keep-temp               Keep temporary files after migration
      --quiet                   Disable progress display
  -r, --region string           AWS region (default "us-west-2")
      --shard-count int         Number of shards for CargoHold (1-100) (default 8)
      --skip-validation         Skip pre-flight validation checks
      --storage-class string    S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
      --temp-dir string         Temporary directory for extraction (default: OS temp)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship resume ​

Manage resumable uploads

Synopsis ​

Manage resumable uploads including listing, resuming, and cleaning up old states.

The resume command allows you to:

  • List all resumable uploads with their progress
  • Resume a specific interrupted upload by ID
  • Clean up old state files to free disk space

Examples:

List all resumable uploads ​

cargoship resume list

Resume a specific upload ​

cargoship resume 20250115-143052-a3b4c5d6

Clean up state files older than 24 hours ​

cargoship resume clean --older-than 24h

Clean up all completed uploads ​

cargoship resume clean --completed

State files are stored in ~/.cargoship/state/ Each state file contains upload progress, configuration, and file hashes.

cargoship resume [upload-id] [flags]

Options ​

  -h, --help   help for resume

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship resume list ​

List all resumable uploads

Synopsis ​

List all resumable uploads with their current progress.

Displays:

  • Upload ID and status
  • Source directory and S3 destination
  • Progress (files and bytes completed)
  • Time since upload started
  • Estimated completion (if applicable)

State files are read from ~/.cargoship/state/

cargoship resume list [flags]

Options ​

  -h, --help   help for list

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship resume clean ​

Clean up old or completed upload states

Synopsis ​

Remove old state files to free up disk space.

By default, cleans up state files older than 24 hours. Use --completed to remove only fully completed uploads. Use --older-than to specify a custom age threshold.

Examples:

Clean states older than 24 hours (default) ​

cargoship resume clean

Clean states older than 1 week ​

cargoship resume clean --older-than 168h

Clean only completed uploads ​

cargoship resume clean --completed

State files are stored in ~/.cargoship/state/ Each file is typically 10-50 KB.

cargoship resume clean [flags]

Options ​

      --completed           Clean only completed uploads
  -h, --help                help for clean
      --older-than string   Clean states older than duration (e.g., 24h, 7d, 168h)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

Fleet (ghostship) ​

cargoship ghostship ​

Fleet mode: run CargoShip as unattended, writer-isolated backup agents

Synopsis ​

Ghostship groups the commands for running CargoShip as a fleet of unattended backup agents that share one S3 bucket without colliding (writer isolation, #520).

Subcommands: init Scaffold a deployable, write-only writer bundle (--mint provisions IAM) scuttle Revoke a minted writer's IAM identity (optionally purge its data) run Run an unattended, writer-isolated incremental backup on a schedule iam-policy Emit the least-privilege IAM policy for one writer validate-config Check a ghostship config (and optionally its scope) before deploy config-keygen Generate an ed25519 keypair for signing fleet configs sign-config Sign a config so agents can verify it

Options ​

  -h, --help   help for ghostship

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship init ​

Scaffold a deployable, write-only fleet-writer bundle (#613)

Synopsis ​

Generate everything needed to deploy one ghostship writer against s3://BUCKET/BASE: a strict write-only IAM policy, a config skeleton, a compose file that wires credentials via a mounted file (not env), and the operator's config-signing public key baked in.

This emits artifacts only — it does not call AWS. Attach the emitted policy to a new IAM identity yourself (or use your own tooling), then fill in and sign the config. The writer runs in signed config-over-S3 (pull) mode.

The emitted IAM policy is write-only and delete-free: PutObject on the writer's own prefix, ListBucket scoped to it, read-only on its config, and kms:GenerateDataKey (no Decrypt). A compromised writer can neither read back, decrypt, nor delete any data.

Example: cargoship ghostship config-keygen --out ./keys cargoship ghostship init s3://backups/nas --writer-id lab-nas-1
--kms-key-arn arn:aws:kms:us-west-2:123456789012:key/abcd
--public-key ./keys/config-signing-public.pem --out ./lab-nas-1

cargoship ghostship init S3_URL [flags]

Options ​

  -h, --help                 help for init
      --image string         Container image to run in the emitted compose file (default "ghcr.io/scttfrdmn/cargoship:<this binary's version>")
      --kms-key-arn string   Fleet KMS key ARN; adds kms:GenerateDataKey (write-only, no Decrypt) scoped to that key
      --mint                 Provision the IAM identity live (create user + write-only policy + access key) and write credentials into the bundle; requires iam:Create* on the caller
      --out string           Directory to write the bundle into (default ./<writer-id>)
      --profile string       AWS profile to use for --mint
      --public-key string    Path to the config-signing public key (from 'ghostship config-keygen') to bake into the bundle (required)
  -r, --region string        AWS region for the emitted compose file (and --mint calls) (default "us-west-2")
      --writer-id string     Writer identity for this bundle (writers/<id>/). 'auto' derives a stable per-host id

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship scuttle ​

Revoke a minted fleet-writer IAM identity (#613)

Synopsis ​

Decommission one fleet writer by deleting the IAM identity that 'ghostship init --mint' created for it: its access keys, the attached write-only policy, and the user. Idempotent — a writer that is already gone is a no-op, and only the cargoship-managed policy is deleted (any other attached policy is detached but left intact).

With --purge-data it ALSO deletes that writer's S3 data (writers/<id>/) and config (<base>/fleet/<id>/) under the given bucket — irreversible, so it requires --yes.

This is the fleet-writer counterpart to 'cargoship scuttle' (which wipes bucket data).

Examples: cargoship ghostship scuttle lab-nas-1 cargoship ghostship scuttle lab-nas-1 --purge-data s3://backups/nas --yes

cargoship ghostship scuttle WRITER_ID [flags]

Options ​

  -h, --help                help for scuttle
      --profile string      AWS profile to use
      --purge-data string   Also delete this writer's S3 data + config under s3://BUCKET/BASE (irreversible; needs --yes)
  -r, --region string       AWS region (default "us-west-2")
      --yes                 Confirm the irreversible --purge-data deletion

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship run ​

Run an unattended, writer-isolated incremental backup on a schedule

Synopsis ​

Continuously back up on an interval using CargoShip's real incremental sync engine (chunked archives, manifests, dataset versioning) — the same engine as 'cargoship sync', just scheduled and headless. Each cycle uploads only what changed since the previous manifest.

Three forms: Single source (flags): cargoship ghostship run SOURCE_DIR s3://BUCKET/PREFIX --writer-id ID Config file (fleet): cargoship ghostship run --config box.yaml Signed config (pull): cargoship ghostship run --config-url s3://BUCKET/BASE
--public-key pub.pem --writer-id ID

With a config file, each entry in 'watch_paths' is backed up as its own source under one writer prefix (writers/<id>/). 'archival_rules' are ignored in sync mode (they belong to the legacy per-file model).

With --config-url, the config is PULLED from BASE/fleet/<id>/config.yaml and must verify against the ed25519 --public-key (the agent bakes only the public key). An unsigned, wrong-key, or tampered config is refused. Outbound-only: no inbound port. Stops cleanly on SIGINT/SIGTERM; --once runs a single cycle (for cron).

Examples: cargoship ghostship run /volume1/Documents s3://backups/nas --writer-id lab-nas-1 cargoship ghostship run --config /etc/cargoship/box.yaml --interval 30m cargoship ghostship run --config-url s3://backups/nas --public-key /etc/cargoship/fleet.pub --writer-id lab-nas-1

cargoship ghostship run [SOURCE_DIR S3_URL] [flags]

Options ​

      --compression-level int       Fixed zstd level (1-22); 0 = content-aware per-chunk selection
      --config string               Ghostship config file (fleet mode); backs up each watch_paths entry
      --config-url string           Pull a signed config from S3 (s3://BUCKET/BASE); reads BASE/fleet/<id>/config.yaml, verified against --public-key
  -h, --help                        help for run
      --ignore-budget               Skip the per-writer budget/volume cap check (#629)
      --interval duration           How often to run a backup cycle (overrides config scan_interval) (default 1h0m0s)
      --manifest-cache-dir string   Directory for the local manifest cache (default ~/.cargoship/manifest-cache)
      --no-manifest-cache           Disable the local manifest cache. A write-only writer then has no previous manifest to diff against and re-uploads every file each cycle
      --once                        Run a single cycle and exit (for cron / testing)
      --public-key string           PEM ed25519 public key the pulled config must verify against (required with --config-url)
  -r, --region string               AWS region (default "us-west-2")
      --shard-count int             Number of shards for parallel uploads (1-100) (default 10)
      --shard-strategy string       Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
      --storage-class string        Default S3 storage class (per-source storage_class in config overrides) (default "STANDARD")
      --track-deletes               Record files deleted since the last backup in the manifest
      --writer-id string            Writer identity for fleet isolation (writers/<id>/). 'auto' derives a stable per-host id; overrides the config

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship iam-policy ​

Emit a least-privilege, delete-free IAM policy for a fleet writer

Synopsis ​

Emit the least-privilege AWS IAM policy for one ghostship writer backing up to s3://BUCKET/PREFIX. Attach the emitted policy to the IAM user or role that your writer runs as — a ghostship agent, or a cron'd 'cargoship sync --writer-id'.

The policy grants only what a writer needs: PutObject/GetObject/AbortMultipartUpload scoped to writers/<id>/, plus ListBucket gated to that prefix. It is deliberately delete-free and cannot reach another writer's data — a compromised writer can only append to its own subtree, never delete or overwrite backups. With --kms-key-arn it also grants kms:GenerateDataKey and kms:Decrypt on that one key (AWS requires both to multipart-upload SSE-KMS objects).

Examples: cargoship ghostship iam-policy s3://my-bucket/backups --writer-id lab-nas-1 cargoship ghostship iam-policy s3://my-bucket/backups --writer-id auto
--kms-key-arn arn:aws:kms:us-west-2:123456789012:key/abcd-1234 -o policy.json

cargoship ghostship iam-policy S3_URL [flags]

Options ​

  -h, --help                 help for iam-policy
      --kms-key-arn string   KMS key ARN for encrypted uploads; adds kms:GenerateDataKey (+kms:Decrypt unless --write-only) scoped to that key
  -o, --output string        Write the policy JSON to this file instead of stdout
      --write-only           Emit the strict write-only policy: PutObject-only on data (no GetObject/Decrypt); incremental sync degrades to full re-scan each cycle
      --writer-id string     Writer identity to scope the policy to (writers/<id>/). 'auto' derives a stable per-host id; empty scopes to the base prefix

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship validate-config ​

Validate a ghostship config, and optionally check it doesn't widen scope

Synopsis ​

Validate a ghostship config file before deploying it. Reports errors (an invalid config) and warnings (dangerous-but-permitted combinations, e.g. delete_after_archive with a broad matcher).

With --baseline, also reports how CONFIG_FILE widens what a writer reads or deletes relative to the currently-deployed config (new/broadened watch paths, recursive flips, added includes, removed excludes, newly-enabled source deletion) — the check the fleet's config-over-S3 pull will enforce.

Read-only: no AWS calls, no network.

Examples: cargoship ghostship validate-config ghost_ship.yaml cargoship ghostship validate-config new.yaml --baseline deployed.yaml --strict

cargoship ghostship validate-config CONFIG_FILE [flags]

Options ​

      --baseline string     Path to the currently-deployed config; report how CONFIG_FILE widens watch scope vs it
  -h, --help                help for validate-config
      --public-key string   Verify the config's detached signature against this PEM ed25519 public key (fails if invalid)
      --signature string    Path to the signature sidecar (default CONFIG_FILE.sig)
      --strict              Treat warnings and scope-widenings as failures (non-zero exit)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship config-keygen ​

Generate an ed25519 keypair for signing fleet configs (#614)

Synopsis ​

Generate an ed25519 signing keypair for the config-over-S3 trust path. The operator keeps the private key and signs configs with 'ghostship sign-config'; only the public key is baked into an agent at deploy, and the agent refuses any config that doesn't verify against it.

Writes config-signing-private.pem (0600) and config-signing-public.pem; neither is overwritten if it already exists. This keypair signs configs — it is separate from the GPG keys 'cargoship create keys' makes for file encryption.

Example: cargoship ghostship config-keygen --out ./fleet-keys

cargoship ghostship config-keygen [flags]

Options ​

  -h, --help         help for config-keygen
      --out string   Directory to write the keypair into (default current directory)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship ghostship sign-config ​

Sign a ghostship config with an ed25519 private key (#614)

Synopsis ​

Produce a detached signature over a ghostship config so agents can verify it came from the operator. The config is validated first — a config with validation errors is refused, so you never sign a broken config.

Writes a signature sidecar (default CONFIG_FILE.sig) that travels with the config. Bump the config's 'version' field before signing so each rollout is identifiable in writer heartbeats.

Example: cargoship ghostship sign-config ghost_ship.yaml --key config-signing-private.pem

cargoship ghostship sign-config CONFIG_FILE [flags]

Options ​

  -h, --help            help for sign-config
      --key string      Path to the ed25519 private key PEM (required)
  -o, --output string   Write the signature to this path (default CONFIG_FILE.sig)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship fleet ​

Observe a ghostship fleet from the control side (read-only)

Synopsis ​

Inspect a fleet of unattended ghostship backup agents by reading the heartbeats they write under writers/<id>/status.json.

These commands run on your control machine and only read the bucket — they do not need any agent's write identity. See 'cargoship ghostship' for the agent side.

Options ​

  -h, --help   help for fleet

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship fleet status ​

Show the last-known status of every writer in a fleet

Synopsis ​

List every writer that has reported under the given fleet prefix, with how long ago it last checked in, whether its most recent cycle was healthy, and its per-source counts.

S3_URL is the fleet's base prefix (the same destination the agents back up to, not a writers/<id>/ sub-prefix). Writers with no readable heartbeat are omitted.

Examples: cargoship fleet status s3://backups/nas cargoship fleet status s3://backups/nas --json

cargoship fleet status S3_URL [flags]

Options ​

  -h, --help             help for status
      --json             Emit the fleet status as JSON
      --profile string   AWS profile for the S3 target
      --region string    AWS region for the S3 target (auto-detected if empty)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship fleet monitor ​

Watch a fleet and alert when a writer goes stale (#630)

Synopsis ​

Periodically read every writer's heartbeat under a fleet prefix and fire a stale-writer alert for any writer that has not checked in within the freshness threshold — a writer that is down, stuck, or offline can't report on itself, so this runs on the control side. Each pass also evaluates budgets, so both fleet-freshness and budget alerts go out through the same configured channels (see 'cargoship alerts').

Alerts are de-duplicated per writer by the alert cooldown; a writer alerts at most once per cooldown window. With no alert channel configured, this still logs the fleet state on every pass. Runs on an interval; --once does a single pass (for cron).

Examples: cargoship fleet monitor s3://backups/nas --threshold 2h --interval 15m cargoship fleet monitor s3://backups/nas --once

cargoship fleet monitor S3_URL [flags]

Options ​

  -h, --help                 help for monitor
      --interval duration    How often to check the fleet (default 15m0s)
      --once                 Run a single check and exit (for cron / testing)
      --profile string       AWS profile for the S3 target
      --region string        AWS region for the S3 target (auto-detected if empty)
      --threshold duration   Age since last check-in after which a writer is considered stale (default 2h0m0s)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship fleet lock-status ​

Audit a fleet bucket's immutability posture (versioning, Object Lock, lifecycle hygiene)

Synopsis ​

Report whether a fleet bucket can resist a stolen delete-capable credential and whether failed-upload hygiene is in place — read-only, it never changes the bucket.

Because the ghostship agent is delete-free by design, the ransomware backstop lives in the bucket itself: S3 Versioning + Object Lock protect history even if a delete-capable credential is compromised, and a lifecycle AbortIncompleteMultipartUpload rule cleans up failed uploads the agent won't. This command checks all three and suggests remediation.

Object Lock, versioning, and lifecycle are bucket-level, so any prefix in the S3 URL is ignored — the posture is reported for the whole bucket.

Examples: cargoship fleet lock-status s3://backups/nas cargoship fleet lock-status s3://backups/nas --json

cargoship fleet lock-status S3_URL [flags]

Options ​

  -h, --help             help for lock-status
      --json             Emit the posture report as JSON
      --profile string   AWS profile for the S3 target
      --region string    AWS region for the S3 target (auto-detected if empty)

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output