CargoShip

Appearance

Incremental sync ​

cargoship sync keeps an S3 archive in step with a local directory by uploading only the files that are new or have changed since the last run. The first sync uploads everything (like cargoship upload); every sync after that reads the previous manifest, compares it against the current filesystem, and transfers just the differences — then writes a new manifest that references the prior one. The exhaustive flag list lives in the command reference.

bash
cargoship sync SOURCE_DIR s3://BUCKET/PREFIX/

How change detection works ​

A file is re-uploaded when either

  • its size differs from the manifest, or
  • its modification time is newer than the manifest.

This is quick because it never re-reads file contents.

bash
cargoship sync ./my-data s3://my-bucket/backups/

Content is not hashed, so one case is not detected

An edit that preserves both size and modification time is not detected. That happens in practice with files restored from backup with their mtime preserved, some rsync modes, and in-place edits that happen to keep the same length.

There is no content-comparison mode today. --checksum was supposed to be it, but was never implemented and now fails with an error rather than silently doing nothing (#678).

When you need certainty for a tree whose timestamps you do not trust, use --force to re-upload everything and let the manifest chain record a fresh baseline.

Previewing a sync ​

Run with --dry-run to see exactly what would transfer without uploading anything. This is the safe way to sanity-check a large sync before committing to the bandwidth and cost:

bash
cargoship sync ./my-data s3://my-bucket/backups/ --dry-run
Comparing ./my-data against previous manifest...
  new:      42 files   (1.2 GB)
  changed:   7 files   (318 MB)
  unchanged: 9,914 files (skipped)
Would upload 49 files (1.5 GB) across 10 shards. No changes made (--dry-run).

Tracking deletions ​

By default, sync is additive: files removed locally stay in the archive so old manifests remain fully restorable. Add --track-deletes to record local deletions in the new manifest, keeping it a faithful mirror of the source tree:

bash
cargoship sync ./my-data s3://my-bucket/backups/ --track-deletes

Deletions are recorded, not purged

--track-deletes marks files as deleted in the new manifest so restores from it won't include them. It does not delete chunk data from S3 — earlier versions in the chain still reference those chunks.

Reclaim storage with dataset prune, not age-based expiration

To free space, use reference-aware garbage collection:

bash
cargoship dataset prune s3://my-bucket/backups/ --dataset-id <id> --keep-last 5

dataset prune deletes only objects that no retained version still references (see dataset versioning). Do not apply an age-based S3 lifecycle expiration rule to an active dataset's uploads/ prefix: because an incremental version reuses unchanged objects from older versions, expiring objects by age can make a currently-retained version unrestorable. S3 lifecycle is still appropriate for storage-class transitions (e.g. Standard → Glacier) of still-live data — just not for deletion.

Forcing a full re-sync ​

If a manifest is missing or you want a clean baseline, --force ignores the previous manifest and uploads everything, just like a first run:

bash
cargoship sync ./my-data s3://my-bucket/backups/ --force

Sharding, compression & storage class ​

Sync accepts the same tuning knobs as upload:

bash
cargoship sync ./my-data s3://my-bucket/backups/ \
  --shard-count 16 --shard-strategy size \
  --compression-level 9 \
  --storage-class GLACIER_IR
  • --shard-count — number of parallel shards, 1–100 (default 10). Unlike upload, sync does not auto-select the count.
  • --shard-strategy — round-robin (default), hash, size, type, directory. See Sharding.
  • --compression-level — Zstandard 1–22. Passing it overrides content-aware per-chunk selection and pins every chunk to that level; omit it to keep automatic selection. See Compression.
  • --storage-class — STANDARD (default), GLACIER_IR, DEEP_ARCHIVE, etc. See Lifecycle & storage classes.
  • -q/--quiet — minimal output for cron and scripts.
  • -r/--region — AWS region (default us-west-2); match your bucket.

sync vs. upload --incremental ​

Both do incremental transfers. Reach for sync when you want a repeatable directory-mirroring workflow that manages the manifest chain for you. Reach for upload --incremental when you already hold a specific previous manifest and want to layer one incremental pass on top of it.

Best practices ​

TIP

  • Always --dry-run first on a new sync target — confirm the new/changed counts look sane before spending bandwidth.
  • Use --force when mtimes can't be trusted (restored trees, CI, rsync): there is no content-comparison mode, so a full re-upload is the only way to re-baseline.
  • Add --track-deletes only when you want the archive to mirror the source; leave it off when you want an append-only history.
  • Pin --region (or set AWS_REGION) so scheduled syncs never guess wrong.
  • Run quiet in cron: add -q and check the exit code.

See also ​