Uploading & sync commands
Commands that write data to S3. For guidance on which to use, see upload vs. create upload; for a walkthrough, Uploading data.
Generated reference
Flag tables below are generated from the CLI and kept in sync by a drift check.
cargoship upload
Upload directory to S3 with CargoHold sharding
Synopsis
Upload a directory to S3 using CargoHold's intelligent sharding system.
CargoHold divides large datasets into multiple shards for parallel uploads, providing:
- Intelligent shard distribution (round-robin, hash, size, type, or directory)
- Content-aware zstd compression, with an optional fixed-level override
- Parallel uploads for maximum throughput
- Automatic manifest generation for easy restore
- Progress tracking
Shard Count: CargoShip automatically selects between 4 and 32 shards when --shard-count is 0 (the default). If automatic selection fails, it falls back to 8.
Shard Strategies: round-robin - Distribute by chunk order (even shards, cheapest; default) hash - Hash of chunk contents (stable across runs) size - Least-loaded shard by bytes (evens out uneven chunk sizes) type - Group chunks by predominant content type directory - Group chunks by common directory prefix
Compression: By default CargoShip picks a zstd level per chunk from the content it contains — level 1 for already-compressed data, up to level 9 for source code. Passing --compression-level overrides that and pins every chunk to one level. Note that zstd has only four internal levels, so values map in bands (1-2, 3-5, 6-9, 10+); the effective setting is reported at upload start.
Examples:
Upload with defaults (auto shard count, round-robin, content-aware compression)
cargoship upload /data s3://my-bucket/dataset
Upload with custom shard count and strategy
cargoship upload /data s3://my-bucket/dataset --shard-count 20 --shard-strategy size
Pin every chunk to maximum compression (disables content-aware selection)
cargoship upload /data s3://my-bucket/dataset --compression-level 19
Quiet mode (no progress display)
cargoship upload /data s3://my-bucket/dataset --quiet
cargoship upload SOURCE_DIR DESTINATION [flags]Options
--auto-tier Enable automatic storage tier selection based on file access time
-b, --bucket string S3 bucket name (or use s3:// URL in DESTINATION)
--check-access Report the target bucket's access-control posture before uploading (see the 'access-check' command)
--compression-level int Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
--congestion-control string Congestion control algorithm: bbr, cubic, auto (default "auto")
--direct-upload Enable direct upload mode (bypasses archiving/compression for small files)
--direct-upload-threshold-mb int Max total size in MB for auto direct upload (default: 500) (default 500)
--direct-upload-workers int Worker count for direct upload (default: 256) (default 256)
--disable-staging Disable adaptive staging (reduces memory usage)
--dvc-auto Auto-discover DVC stages from dvc.yaml and annotate each file entry with its stage name
--dvc-cache-dir string Local DVC cache directory (recorded in manifest; default: .dvc/cache) (default ".dvc/cache")
--dvc-output-dir string Directory to write .dvc files (default: source directory)
--dvc-stage string DVC pipeline stage name to extract provenance from (reads dvc.yaml + dvc.lock)
--enable-dedup Enable cross-shard file deduplication (10-30% space savings for redundant datasets)
--encrypt-manifest Encrypt manifest with KMS envelope encryption (requires --kms-key-id)
--fail-on string With --check-access, abort if the worst finding is at least this severe: none, unknown, warn, critical (warn/critical do NOT trip on checks that couldn't be evaluated — use 'unknown' to also fail when a check was denied) (default "none")
--force-direct-upload Force direct upload regardless of thresholds (for benchmarking)
--force-restart Ignore saved state and start fresh upload (bypasses resume detection)
--frame-size string Cut compressed chunks into random-access zstd frames every N bytes (e.g. 16MiB, 64MB); 0 disables framing (default "16MiB")
--generate-dvc-files Generate DVC sidecar .dvc files after upload
--git-metadata Embed Git repository metadata (commit, branch, tag, remote) in the manifest
-h, --help help for upload
--ignore-budget Skip the pre-upload budget/volume cap check (#629)
--incremental Enable incremental sync: only upload new or changed files
--kms-key-id string AWS KMS key ID or ARN for encryption (data chunks encrypted with SSE-KMS)
--no-file-checksums Disable per-file content checksums (faster uploads, but 'verify --deep' can't confirm per-file integrity)
--optimization Enable optimization features (BBR/CUBIC, adaptive staging, BDP) (default true)
--prev-manifest string Path to previous manifest JSON (or .json.gz) for incremental sync
--project string Project ID for cost tracking (e.g. 'dvc_cache' for DVC remotes)
--prometheus-addr string Prometheus metrics HTTP address (e.g., :9090)
--quiet Disable progress display
-r, --region string AWS region (default "us-west-2")
--resume Resume an interrupted upload of this source→destination without prompting (skips already-uploaded chunks)
--shard-count int Shards for parallel uploads: 0 auto-selects 4-32 (falls back to 8), or set 4-32 manually
--shard-strategy string Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
--tag stringArray Custom tag in key=value format, repeatable (e.g. --tag dvc_cache=true --tag env=prod)
--tier-archive-days int Days since access to consider 'archive' (DEEP_ARCHIVE) (default 180)
--tier-cold-days int Days since access to consider 'cold' (GLACIER) (default 90)
--tier-hot-days int Days since access to consider 'hot' (STANDARD) (default 30)
--tier-max string Maximum storage tier (STANDARD, STANDARD_IA, GLACIER, DEEP_ARCHIVE) - prevents automatic selection of more restrictive tiers
--tier-strategy string Tier chunking strategy (requires --auto-tier):
youngest-file: Conservative strategy - assigns tier based on youngest file per chunk (default)
tier-aware: Optimal cost - groups files by tier before chunking (30-60% savings)
⚠️ WARNING: tier-aware uses GLACIER/DEEP_ARCHIVE with cost implications:
• GLACIER: 90-day minimum storage ($0.004/GB-month, $0.01/GB retrieval, 3-5hr access)
• DEEP_ARCHIVE: 180-day minimum ($0.00099/GB-month, $0.02/GB retrieval, 12hr access)
• Early deletion penalties apply if removed before minimum duration
• Best for long-term archives accessed <1x per year
See: https://github.com/scttfrdmn/cargoship/issues/168 (default "youngest-file")
--tracing Enable distributed tracing
--tracing-endpoint string Tracing endpoint URL (required for jaeger/otlp exporters)
--tracing-exporter string Tracing exporter: stdout, jaeger, otlp, none (default "stdout")
--tracing-sample-rate float Trace sampling rate (0.0-1.0, default: 1.0 = 100%) (default 1)
--transporter string S3 transporter type: basic, staging, adaptive, optimized, none (default "staging")
--writer-id string Writer identity for fleet isolation: objects go under writers/<id>/. 'auto' derives a stable per-host id; empty = single-writer layout (default)
-y, --yes Skip confirmation prompts (auto-accept warnings)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship create
Create something!
cargoship create [flags]Options
-d, --destination string Directory to write files in to. Defaults to the current directory
-h, --help help for createOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship create upload
Upload directories to S3 using streaming pipeline
Synopsis
Upload directories to S3 using the high-performance streaming pipeline.
This command replaces the legacy suitcase/rclone system with a modern streaming architecture that provides:
- Real-time progress tracking with beautiful TUI
- Multi-prefix S3 parallel uploads (8x throughput improvement)
- Zero local disk usage (streaming directly to S3)
- Automatic compression (zstd)
- Intelligent chunking and sharding
cargoship create upload SOURCE_DIR... [flags]Examples
# Upload a directory with progress tracking
cargoship create upload /path/to/data --bucket my-bucket
# Upload with custom prefix
cargoship create upload /path/to/data --bucket my-bucket --prefix backups/2025-12-05
# Quiet mode (no progress display)
cargoship create upload /path/to/data --bucket my-bucket --quiet
# JSON progress output (for scripts)
cargoship create upload /path/to/data --bucket my-bucket --progress-format jsonOptions
--bucket string S3 bucket name (required)
--chunk-size-mb int Target chunk size in MB (0 = adaptive) (default 200)
--cleanup-on-failure Automatically delete partial uploads on error (default true)
-h, --help help for upload
--http2 Enable HTTP/2 (default true)
--http2-max-streams int Max concurrent HTTP/2 streams per connection (default 250)
--idle-conn-timeout duration Idle connection timeout (default 5m0s)
--max-idle-conns int Max idle connections per host (default 100)
--network-profile string Network tuning profile: default, aggressive, conservative (default "default")
--no-cleanup Disable automatic cleanup on failure (for debugging)
--prefix string S3 key prefix (optional)
--progress-format string Progress output format: tui, json, text (default "tui")
--quiet Disable progress display
--region string AWS region (default "us-west-2")
--resume Resume a previous incomplete upload
--shards int Number of S3 prefix shards for parallel uploads (default 8)
--skip-existing Skip chunks that already exist in S3 (HeadObject check)
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER, etc.) (default "STANDARD")
--upload-id string Upload ID to resume (auto-detect if not specified)
--workers int Workers per stage (scanner, archiver, uploader) (default 4)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
-d, --destination string Directory to write files in to. Defaults to the current directory
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship sync
Incrementally sync directory to S3 (only upload new/changed files)
Synopsis
Incrementally sync a local directory to S3 by uploading only new or modified files.
The sync command provides efficient incremental backups by:
- Downloading the latest manifest for the source path (if exists)
- Comparing local filesystem state against the manifest
- Uploading only files that are new or have changed
- Creating a new manifest that references the previous one
First sync uploads everything (like 'upload' command). Subsequent syncs only upload changed files, saving time and bandwidth.
Change detection compares SIZE and MODIFICATION TIME against the previous manifest. Content is not hashed, so an edit that preserves both size and mtime is not detected. Use --force for a full re-upload when you need certainty.
Examples:
First sync: uploads all files
cargoship sync /home/photos s3://my-bucket/backups
Second sync: only uploads new/changed photos
cargoship sync /home/photos s3://my-bucket/backups
Dry run to see what would be synced
cargoship sync /home/photos s3://my-bucket/backups --dry-run
Force full sync (ignore previous manifest)
cargoship sync /data s3://my-bucket/backups --force
cargoship sync SOURCE_DIR S3_URL [flags]Options
--checksum NOT IMPLEMENTED (#678): rejected if set. Change detection is size+mtime; use --force for a full re-upload
--compression-level int Fixed zstd compression level (1-22), overriding per-chunk content-aware selection. Unset = content-aware (default 3)
--dry-run Show what would be synced without uploading
--force Force full sync (ignore previous manifest)
-h, --help help for sync
--no-file-checksums Disable per-file content checksums (faster, but 'verify --deep' can't confirm per-file integrity)
-q, --quiet Quiet mode (minimal output)
-r, --region string AWS region (default "us-west-2")
--shard-count int Number of shards for parallel uploads (1-100) (default 10)
--shard-strategy string Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
--storage-class string S3 storage class (STANDARD, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
--track-deletes Track deleted files in manifest
--writer-id string Writer identity for fleet isolation: objects go under writers/<id>/. 'auto' derives a stable per-host id; empty = single-writer layout (default)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship migrate
Convert traditional archives to CargoHold sharded format
Synopsis
Download a traditional tar.zst archive from S3 and re-upload using CargoHold's intelligent sharding system.
The migrate command:
- Downloads the traditional archive from S3
- Extracts files to a temporary location
- Re-uploads using CargoHold sharding with compression
- Generates a manifest for selective extraction
- Optionally deletes the original archive
Examples:
Migrate archive with default settings
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded
Migrate and delete original
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --delete-original
Dry run to estimate migration
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --dry-run
Custom temp directory and shard count
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded
--temp-dir /mnt/fast-ssd --shard-count 16
Keep temp files for debugging
cargoship migrate s3://bucket/archive.tar.zst s3://bucket/dataset-sharded --keep-temp
cargoship migrate SOURCE_ARCHIVE DESTINATION [flags]Options
--compression-level int Zstd compression level (1-22) (default 3)
--delete-original Delete original archive after successful migration
--dry-run Estimate migration without performing it
-h, --help help for migrate
--keep-temp Keep temporary files after migration
--quiet Disable progress display
-r, --region string AWS region (default "us-west-2")
--shard-count int Number of shards for CargoHold (1-100) (default 8)
--skip-validation Skip pre-flight validation checks
--storage-class string S3 storage class (STANDARD, INTELLIGENT_TIERING, GLACIER_IR, DEEP_ARCHIVE) (default "STANDARD")
--temp-dir string Temporary directory for extraction (default: OS temp)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume
Manage resumable uploads
Synopsis
Manage resumable uploads including listing, resuming, and cleaning up old states.
The resume command allows you to:
- List all resumable uploads with their progress
- Resume a specific interrupted upload by ID
- Clean up old state files to free disk space
Examples:
List all resumable uploads
cargoship resume list
Resume a specific upload
cargoship resume 20250115-143052-a3b4c5d6
Clean up state files older than 24 hours
cargoship resume clean --older-than 24h
Clean up all completed uploads
cargoship resume clean --completed
State files are stored in ~/.cargoship/state/ Each state file contains upload progress, configuration, and file hashes.
cargoship resume [upload-id] [flags]Options
-h, --help help for resumeOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume list
List all resumable uploads
Synopsis
List all resumable uploads with their current progress.
Displays:
- Upload ID and status
- Source directory and S3 destination
- Progress (files and bytes completed)
- Time since upload started
- Estimated completion (if applicable)
State files are read from ~/.cargoship/state/
cargoship resume list [flags]Options
-h, --help help for listOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship resume clean
Clean up old or completed upload states
Synopsis
Remove old state files to free up disk space.
By default, cleans up state files older than 24 hours. Use --completed to remove only fully completed uploads. Use --older-than to specify a custom age threshold.
Examples:
Clean states older than 24 hours (default)
cargoship resume clean
Clean states older than 1 week
cargoship resume clean --older-than 168h
Clean only completed uploads
cargoship resume clean --completed
State files are stored in ~/.cargoship/state/ Each file is typically 10-50 KB.
cargoship resume clean [flags]Options
--completed Clean only completed uploads
-h, --help help for clean
--older-than string Clean states older than duration (e.g., 24h, 7d, 168h)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputFleet (ghostship)
cargoship ghostship
Fleet mode: run CargoShip as unattended, writer-isolated backup agents
Synopsis
Ghostship groups the commands for running CargoShip as a fleet of unattended backup agents that share one S3 bucket without colliding (writer isolation, #520).
Subcommands: init Scaffold a deployable, write-only writer bundle (--mint provisions IAM) scuttle Revoke a minted writer's IAM identity (optionally purge its data) run Run an unattended, writer-isolated incremental backup on a schedule iam-policy Emit the least-privilege IAM policy for one writer validate-config Check a ghostship config (and optionally its scope) before deploy config-keygen Generate an ed25519 keypair for signing fleet configs sign-config Sign a config so agents can verify it
Options
-h, --help help for ghostshipOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship init
Scaffold a deployable, write-only fleet-writer bundle (#613)
Synopsis
Generate everything needed to deploy one ghostship writer against s3://BUCKET/BASE: a strict write-only IAM policy, a config skeleton, a compose file that wires credentials via a mounted file (not env), and the operator's config-signing public key baked in.
This emits artifacts only — it does not call AWS. Attach the emitted policy to a new IAM identity yourself (or use your own tooling), then fill in and sign the config. The writer runs in signed config-over-S3 (pull) mode.
The emitted IAM policy is write-only and delete-free: PutObject on the writer's own prefix, ListBucket scoped to it, read-only on its config, and kms:GenerateDataKey (no Decrypt). A compromised writer can neither read back, decrypt, nor delete any data.
Example: cargoship ghostship config-keygen --out ./keys cargoship ghostship init s3://backups/nas --writer-id lab-nas-1
--kms-key-arn arn:aws:kms:us-west-2:123456789012:key/abcd
--public-key ./keys/config-signing-public.pem --out ./lab-nas-1
cargoship ghostship init S3_URL [flags]Options
-h, --help help for init
--image string Container image to run in the emitted compose file (default "ghcr.io/scttfrdmn/cargoship:<this binary's version>")
--kms-key-arn string Fleet KMS key ARN; adds kms:GenerateDataKey (write-only, no Decrypt) scoped to that key
--mint Provision the IAM identity live (create user + write-only policy + access key) and write credentials into the bundle; requires iam:Create* on the caller
--out string Directory to write the bundle into (default ./<writer-id>)
--profile string AWS profile to use for --mint
--public-key string Path to the config-signing public key (from 'ghostship config-keygen') to bake into the bundle (required)
-r, --region string AWS region for the emitted compose file (and --mint calls) (default "us-west-2")
--writer-id string Writer identity for this bundle (writers/<id>/). 'auto' derives a stable per-host idOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship scuttle
Revoke a minted fleet-writer IAM identity (#613)
Synopsis
Decommission one fleet writer by deleting the IAM identity that 'ghostship init --mint' created for it: its access keys, the attached write-only policy, and the user. Idempotent — a writer that is already gone is a no-op, and only the cargoship-managed policy is deleted (any other attached policy is detached but left intact).
With --purge-data it ALSO deletes that writer's S3 data (writers/<id>/) and config (<base>/fleet/<id>/) under the given bucket — irreversible, so it requires --yes.
This is the fleet-writer counterpart to 'cargoship scuttle' (which wipes bucket data).
Examples: cargoship ghostship scuttle lab-nas-1 cargoship ghostship scuttle lab-nas-1 --purge-data s3://backups/nas --yes
cargoship ghostship scuttle WRITER_ID [flags]Options
-h, --help help for scuttle
--profile string AWS profile to use
--purge-data string Also delete this writer's S3 data + config under s3://BUCKET/BASE (irreversible; needs --yes)
-r, --region string AWS region (default "us-west-2")
--yes Confirm the irreversible --purge-data deletionOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship run
Run an unattended, writer-isolated incremental backup on a schedule
Synopsis
Continuously back up on an interval using CargoShip's real incremental sync engine (chunked archives, manifests, dataset versioning) — the same engine as 'cargoship sync', just scheduled and headless. Each cycle uploads only what changed since the previous manifest.
Three forms: Single source (flags): cargoship ghostship run SOURCE_DIR s3://BUCKET/PREFIX --writer-id ID Config file (fleet): cargoship ghostship run --config box.yaml Signed config (pull): cargoship ghostship run --config-url s3://BUCKET/BASE
--public-key pub.pem --writer-id ID
With a config file, each entry in 'watch_paths' is backed up as its own source under one writer prefix (writers/<id>/). 'archival_rules' are ignored in sync mode (they belong to the legacy per-file model).
With --config-url, the config is PULLED from BASE/fleet/<id>/config.yaml and must verify against the ed25519 --public-key (the agent bakes only the public key). An unsigned, wrong-key, or tampered config is refused. Outbound-only: no inbound port. Stops cleanly on SIGINT/SIGTERM; --once runs a single cycle (for cron).
Examples: cargoship ghostship run /volume1/Documents s3://backups/nas --writer-id lab-nas-1 cargoship ghostship run --config /etc/cargoship/box.yaml --interval 30m cargoship ghostship run --config-url s3://backups/nas --public-key /etc/cargoship/fleet.pub --writer-id lab-nas-1
cargoship ghostship run [SOURCE_DIR S3_URL] [flags]Options
--compression-level int Fixed zstd level (1-22); 0 = content-aware per-chunk selection
--config string Ghostship config file (fleet mode); backs up each watch_paths entry
--config-url string Pull a signed config from S3 (s3://BUCKET/BASE); reads BASE/fleet/<id>/config.yaml, verified against --public-key
-h, --help help for run
--ignore-budget Skip the per-writer budget/volume cap check (#629)
--interval duration How often to run a backup cycle (overrides config scan_interval) (default 1h0m0s)
--manifest-cache-dir string Directory for the local manifest cache (default ~/.cargoship/manifest-cache)
--no-manifest-cache Disable the local manifest cache. A write-only writer then has no previous manifest to diff against and re-uploads every file each cycle
--once Run a single cycle and exit (for cron / testing)
--public-key string PEM ed25519 public key the pulled config must verify against (required with --config-url)
-r, --region string AWS region (default "us-west-2")
--shard-count int Number of shards for parallel uploads (1-100) (default 10)
--shard-strategy string Shard distribution strategy (round-robin, hash, size, type, directory) (default "round-robin")
--storage-class string Default S3 storage class (per-source storage_class in config overrides) (default "STANDARD")
--track-deletes Record files deleted since the last backup in the manifest
--writer-id string Writer identity for fleet isolation (writers/<id>/). 'auto' derives a stable per-host id; overrides the configOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship iam-policy
Emit a least-privilege, delete-free IAM policy for a fleet writer
Synopsis
Emit the least-privilege AWS IAM policy for one ghostship writer backing up to s3://BUCKET/PREFIX. Attach the emitted policy to the IAM user or role that your writer runs as — a ghostship agent, or a cron'd 'cargoship sync --writer-id'.
The policy grants only what a writer needs: PutObject/GetObject/AbortMultipartUpload scoped to writers/<id>/, plus ListBucket gated to that prefix. It is deliberately delete-free and cannot reach another writer's data — a compromised writer can only append to its own subtree, never delete or overwrite backups. With --kms-key-arn it also grants kms:GenerateDataKey and kms:Decrypt on that one key (AWS requires both to multipart-upload SSE-KMS objects).
Examples: cargoship ghostship iam-policy s3://my-bucket/backups --writer-id lab-nas-1 cargoship ghostship iam-policy s3://my-bucket/backups --writer-id auto
--kms-key-arn arn:aws:kms:us-west-2:123456789012:key/abcd-1234 -o policy.json
cargoship ghostship iam-policy S3_URL [flags]Options
-h, --help help for iam-policy
--kms-key-arn string KMS key ARN for encrypted uploads; adds kms:GenerateDataKey (+kms:Decrypt unless --write-only) scoped to that key
-o, --output string Write the policy JSON to this file instead of stdout
--write-only Emit the strict write-only policy: PutObject-only on data (no GetObject/Decrypt); incremental sync degrades to full re-scan each cycle
--writer-id string Writer identity to scope the policy to (writers/<id>/). 'auto' derives a stable per-host id; empty scopes to the base prefixOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship validate-config
Validate a ghostship config, and optionally check it doesn't widen scope
Synopsis
Validate a ghostship config file before deploying it. Reports errors (an invalid config) and warnings (dangerous-but-permitted combinations, e.g. delete_after_archive with a broad matcher).
With --baseline, also reports how CONFIG_FILE widens what a writer reads or deletes relative to the currently-deployed config (new/broadened watch paths, recursive flips, added includes, removed excludes, newly-enabled source deletion) — the check the fleet's config-over-S3 pull will enforce.
Read-only: no AWS calls, no network.
Examples: cargoship ghostship validate-config ghost_ship.yaml cargoship ghostship validate-config new.yaml --baseline deployed.yaml --strict
cargoship ghostship validate-config CONFIG_FILE [flags]Options
--baseline string Path to the currently-deployed config; report how CONFIG_FILE widens watch scope vs it
-h, --help help for validate-config
--public-key string Verify the config's detached signature against this PEM ed25519 public key (fails if invalid)
--signature string Path to the signature sidecar (default CONFIG_FILE.sig)
--strict Treat warnings and scope-widenings as failures (non-zero exit)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship config-keygen
Generate an ed25519 keypair for signing fleet configs (#614)
Synopsis
Generate an ed25519 signing keypair for the config-over-S3 trust path. The operator keeps the private key and signs configs with 'ghostship sign-config'; only the public key is baked into an agent at deploy, and the agent refuses any config that doesn't verify against it.
Writes config-signing-private.pem (0600) and config-signing-public.pem; neither is overwritten if it already exists. This keypair signs configs — it is separate from the GPG keys 'cargoship create keys' makes for file encryption.
Example: cargoship ghostship config-keygen --out ./fleet-keys
cargoship ghostship config-keygen [flags]Options
-h, --help help for config-keygen
--out string Directory to write the keypair into (default current directory)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship ghostship sign-config
Sign a ghostship config with an ed25519 private key (#614)
Synopsis
Produce a detached signature over a ghostship config so agents can verify it came from the operator. The config is validated first — a config with validation errors is refused, so you never sign a broken config.
Writes a signature sidecar (default CONFIG_FILE.sig) that travels with the config. Bump the config's 'version' field before signing so each rollout is identifiable in writer heartbeats.
Example: cargoship ghostship sign-config ghost_ship.yaml --key config-signing-private.pem
cargoship ghostship sign-config CONFIG_FILE [flags]Options
-h, --help help for sign-config
--key string Path to the ed25519 private key PEM (required)
-o, --output string Write the signature to this path (default CONFIG_FILE.sig)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship fleet
Observe a ghostship fleet from the control side (read-only)
Synopsis
Inspect a fleet of unattended ghostship backup agents by reading the heartbeats they write under writers/<id>/status.json.
These commands run on your control machine and only read the bucket — they do not need any agent's write identity. See 'cargoship ghostship' for the agent side.
Options
-h, --help help for fleetOptions inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
--profile Enable performance profiling. This will generate profile files in a temp directory
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship fleet status
Show the last-known status of every writer in a fleet
Synopsis
List every writer that has reported under the given fleet prefix, with how long ago it last checked in, whether its most recent cycle was healthy, and its per-source counts.
S3_URL is the fleet's base prefix (the same destination the agents back up to, not a writers/<id>/ sub-prefix). Writers with no readable heartbeat are omitted.
Examples: cargoship fleet status s3://backups/nas cargoship fleet status s3://backups/nas --json
cargoship fleet status S3_URL [flags]Options
-h, --help help for status
--json Emit the fleet status as JSON
--profile string AWS profile for the S3 target
--region string AWS region for the S3 target (auto-detected if empty)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship fleet monitor
Watch a fleet and alert when a writer goes stale (#630)
Synopsis
Periodically read every writer's heartbeat under a fleet prefix and fire a stale-writer alert for any writer that has not checked in within the freshness threshold — a writer that is down, stuck, or offline can't report on itself, so this runs on the control side. Each pass also evaluates budgets, so both fleet-freshness and budget alerts go out through the same configured channels (see 'cargoship alerts').
Alerts are de-duplicated per writer by the alert cooldown; a writer alerts at most once per cooldown window. With no alert channel configured, this still logs the fleet state on every pass. Runs on an interval; --once does a single pass (for cron).
Examples: cargoship fleet monitor s3://backups/nas --threshold 2h --interval 15m cargoship fleet monitor s3://backups/nas --once
cargoship fleet monitor S3_URL [flags]Options
-h, --help help for monitor
--interval duration How often to check the fleet (default 15m0s)
--once Run a single check and exit (for cron / testing)
--profile string AWS profile for the S3 target
--region string AWS region for the S3 target (auto-detected if empty)
--threshold duration Age since last check-in after which a writer is considered stale (default 2h0m0s)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose outputcargoship fleet lock-status
Audit a fleet bucket's immutability posture (versioning, Object Lock, lifecycle hygiene)
Synopsis
Report whether a fleet bucket can resist a stolen delete-capable credential and whether failed-upload hygiene is in place — read-only, it never changes the bucket.
Because the ghostship agent is delete-free by design, the ransomware backstop lives in the bucket itself: S3 Versioning + Object Lock protect history even if a delete-capable credential is compromised, and a lifecycle AbortIncompleteMultipartUpload rule cleans up failed uploads the agent won't. This command checks all three and suggests remediation.
Object Lock, versioning, and lifecycle are bucket-level, so any prefix in the S3 URL is ignored — the posture is reported for the whole bucket.
Examples: cargoship fleet lock-status s3://backups/nas cargoship fleet lock-status s3://backups/nas --json
cargoship fleet lock-status S3_URL [flags]Options
-h, --help help for lock-status
--json Emit the posture report as JSON
--profile string AWS profile for the S3 target
--region string AWS region for the S3 target (auto-detected if empty)Options inherited from parent commands
--allow-public-observability Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
--context string Override execution context (local, repl)
--memory-limit string Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
--pprof Enable runtime profiling HTTP endpoint at localhost:6060
--pprof-addr string Address for runtime profiling HTTP endpoint (default "localhost:6060")
-t, --trace Enable trace messages in output
-v, --verbose Enable verbose output