Encryption

CargoShip supports two independent layers of encryption, both keyed by AWS KMS:

1. Data encryption — S3 server-side encryption (SSE-KMS) applied to the chunk objects. The chunk bytes themselves are standard .tar.zst; S3 handles the at-rest encryption transparently.
2. Manifest encryption — KMS envelope encryption applied to the manifest JSON, producing a manifest.encrypted.json[.gz] wrapper.

Both are optional and configured independently. The manifest records which are active in its encryption block.

EncryptionMetadata

The encryption field of the manifest, when present:

type EncryptionMetadata struct {
	// Enabled indicates if encryption is active
	Enabled bool `json:"enabled"`

	// Data encryption (S3 server-side encryption)
	DataKMSKeyID string `json:"data_kms_key_id,omitempty"` // KMS key ID/ARN for data chunks

	// Manifest encryption (envelope encryption)
	ManifestEncrypted bool   `json:"manifest_encrypted"`            // Whether manifest itself is encrypted
	ManifestKMSKeyID  string `json:"manifest_kms_key_id,omitempty"` // KMS key ID/ARN for manifest
	Algorithm         string `json:"algorithm,omitempty"`           // e.g. "AES-256-GCM"
	EncryptedDEK      string `json:"encrypted_dek,omitempty"`       // Base64-encoded encrypted data encryption key
}

Field	Meaning
enabled	True when data (chunk) encryption is active — set when a data KMS key is configured.
data_kms_key_id	KMS key ID/ARN used for S3 SSE-KMS on the chunk objects.
manifest_encrypted	True when the manifest is stored as an envelope-encrypted wrapper.
manifest_kms_key_id	KMS key ID/ARN used to wrap the manifest's data key.
algorithm	Symmetric algorithm for the manifest payload — AES-256-GCM.
encrypted_dek	Base64 of the KMS-encrypted data encryption key (mirrors the wrapper's encrypted_dek).

The two layers are orthogonal

You can encrypt data without encrypting the manifest, or vice versa. enabled governs data/chunk encryption; manifest_encrypted governs the manifest wrapper.

Data encryption (chunks)

When data_kms_key_id is set, chunk objects are written with S3 server-side encryption using that KMS key. The object layout is unchanged — a chunk is still a .tar.zst (or .tar) object — and decryption is handled by S3 on GetObject for any caller with kms:Decrypt on the key. No CargoShip-specific unwrapping is needed to read the bytes.

Manifest encryption (envelope)

When manifest_encrypted is true, the manifest is not stored as plaintext JSON. Instead CargoShip writes an EncryptedManifest wrapper to manifest.encrypted.json (or .gz):

type EncryptedManifest struct {
	// Algorithm used for encryption
	Algorithm string `json:"algorithm"`

	// KMS key ID used to encrypt the data key
	KMSKeyID string `json:"kms_key_id"`

	// Base64-encoded encrypted data encryption key (DEK)
	EncryptedDEK string `json:"encrypted_dek"`

	// Base64-encoded initialization vector (IV/nonce) for AES-GCM
	IV string `json:"iv"`

	// Base64-encoded encrypted manifest data
	EncryptedData string `json:"encrypted_data"`

	// Nonzero when the manifest is cryptographically bound to its upload
	// identity (KMS EncryptionContext + GCM AAD). Absent/0 = legacy unbound.
	BindingVersion int `json:"binding_version,omitempty"`
}

Identity binding (binding_version)

Since binding scheme v1, a newly-written encrypted manifest is bound to the upload it belongs to, so an attacker with S3 write cannot substitute another upload's (legitimately encrypted) manifest — the substituted object fails to decrypt (CSH-SEC-004). The binding value is the key/value set {application: "cargoship", purpose: "manifest", v: "1", upload_id: <id>}, used both as the KMS EncryptionContext (on GenerateDataKey/Decrypt) and as the AES-GCM AAD (canonical sorted k=v\n). Only upload_id varies per upload; bucket/prefix are deliberately not bound, so an archive stays readable after a legitimate copy/rename to another location.

A reader derives the expected identity from the location it is fetching (the upload id in the object key), not from the ciphertext, and passes it to Decrypt; a binding_version ≥ 1 manifest with no expected upload id is rejected (fail closed). A binding_version of 0 or absent is decrypted with no context and nil AAD, so manifests written before the scheme remain readable.

Forward compatibility

A bound manifest cannot be decrypted by a cargoship build older than the binding scheme — the older reader passes no EncryptionContext/AAD, and KMS + GCM both refuse. This is inherent to binding and affects only the --encrypt-manifest path; unencrypted manifests and data chunks are unaffected.

Envelope encryption flow (write)

1. Call KMS GenerateDataKey with KeySpec: AES_256 against manifest_kms_key_id, passing the upload-identity EncryptionContext. This returns both a plaintext 32-byte DEK and a KMS-encrypted copy of it (the ciphertext blob) that KMS will only unwrap under the same context.
2. Encrypt the (optionally gzip-compressed) manifest JSON with AES-256-GCM using the plaintext DEK, a fresh random IV (nonce), and the identity AAD. GCM provides authenticated encryption — confidentiality and integrity — and the AAD binds the ciphertext to the upload. Set binding_version = 1.
3. Discard the plaintext DEK. Store the KMS-encrypted DEK, the IV, and the ciphertext (all base64-encoded) in the EncryptedManifest wrapper.
4. Serialize the wrapper to JSON and upload it as manifest.encrypted.json[.gz].

The wrapper's Algorithm and EncryptedDEK are also copied back into the manifest's EncryptionMetadata for reference.

Decryption flow (read)

To recover the manifest JSON:

1. Fetch manifest.encrypted.json.gz (fall back to manifest.encrypted.json), and gunzip if compressed.
2. Parse the EncryptedManifest JSON and verify Algorithm == "AES-256-GCM".
3. Base64-decode encrypted_dek, iv, and encrypted_data.
4. Call KMS Decrypt on the encrypted DEK to recover the plaintext 32-byte key. For a binding_version ≥ 1 manifest, pass the EncryptionContext derived from the upload you are reading; KMS refuses the unwrap if it doesn't match.
5. AES-256-GCM Open the ciphertext with the DEK, IV, and — for a bound manifest — the identity AAD. A GCM authentication failure means the data was tampered with, substituted from another upload, or the wrong key was used — treat it as a hard error, not a soft fallback.
6. The result is the plaintext manifest JSON (still gzip-compressed if it was compressed before encryption — gunzip again if so).

Ordering: compress-then-encrypt

When both apply, the manifest is gzip-compressed before encryption, so the GCM ciphertext wraps compressed bytes. On read, decrypt first, then gunzip.

Discovery order for a reader

A reader that supports encryption should probe keys in this order, falling back on miss:

1. manifest.encrypted.json.gz — encrypted + gzip
2. manifest.encrypted.json — encrypted, plain
3. manifest.json.gz — plaintext + gzip
4. manifest.json — plaintext, plain

This mirrors CargoShip's own DownloadFromS3WithDecryption, which tries the encrypted variants first and falls back to the plaintext download.

Wrapper example

{
  "algorithm": "AES-256-GCM",
  "kms_key_id": "arn:aws:kms:us-west-2:123456789012:key/abcd-1234",
  "encrypted_dek": "AQIDAHh...base64...",
  "iv": "n0Nc3Byt3s...base64...",
  "encrypted_data": "…base64 ciphertext of the (gzipped) manifest JSON…",
  "binding_version": 1
}

Reader requirements

• You need kms:Decrypt on the manifest key to read an encrypted manifest, and on the data key to read SSE-KMS chunks.
• Enforce Algorithm == "AES-256-GCM"; reject unknown algorithms.
• Use the authenticated-decryption result — GCM verification is your integrity check for the manifest payload.
• For a binding_version ≥ 1 manifest, pass the upload-identity EncryptionContext to KMS Decrypt and the identity AAD to GCM Open, both derived from the upload you are reading (not from the wrapper). Reject a bound manifest you cannot supply an upload id for.
• The Go implementation lives in pkg/encryption (EncryptManifest / DecryptManifest, EncryptedManifest); see Reading archives for using it.