# CargoShip documentation

> High-performance S3 data archiving. Stream large datasets straight to S3 — sharded, compressed, verifiable, cost-aware — in an open, portable archive format (tar.zst objects + a documented JSON manifest).

## Introduction

- [What is CargoShip?](https://cargoship.app/)
- [How it works (the pipeline)](https://cargoship.app/intro/how-it-works)
- [Concepts & terminology](https://cargoship.app/intro/concepts)
- [Costs & safety guarantees](https://cargoship.app/intro/costs-and-safety)

## Get Started

- [Quick Start](https://cargoship.app/start/quickstart)
- [Install](https://cargoship.app/start/install)
- [AWS setup & credentials](https://cargoship.app/start/aws-setup)
- [Your first upload](https://cargoship.app/start/first-upload)
- [Verify & restore it](https://cargoship.app/start/verify-and-restore)
- [Clean up](https://cargoship.app/start/cleanup)

## Tutorials (by use-case)

- [Overview](https://cargoship.app/tutorials/)
- [Millions of small files](https://cargoship.app/tutorials/small-files)
- [Genomics / sequencing data](https://cargoship.app/tutorials/genomics)
- [Imaging / microscopy data](https://cargoship.app/tutorials/imaging)
- [ML datasets with DVC](https://cargoship.app/tutorials/ml-dvc)
- [Lab data manager](https://cargoship.app/tutorials/lab-manager)
- [Principal investigator](https://cargoship.app/tutorials/principal-investigator)
- [Migrating from rclone / aws cli](https://cargoship.app/tutorials/migrating)

## Core Workflows

- [Uploading data](https://cargoship.app/guides/uploading)
- [upload vs. create upload](https://cargoship.app/guides/upload-vs-create-upload)
- [Incremental sync](https://cargoship.app/guides/sync)
- [Listing & inspecting uploads](https://cargoship.app/guides/inspecting)
- [Downloading & extracting](https://cargoship.app/guides/downloading)
- [Verifying integrity](https://cargoship.app/guides/verifying)
- [Restoring files (incl. Glacier)](https://cargoship.app/guides/restoring)
- [Browsing archives (TUI/shell)](https://cargoship.app/guides/browsing)
- [Resuming interrupted uploads](https://cargoship.app/guides/resuming)

## Cost & Budget

- [Estimating costs](https://cargoship.app/guides/cost/estimate)
- [Analyzing existing S3 spend](https://cargoship.app/guides/cost/analyze)
- [Cost management & reporting](https://cargoship.app/guides/cost/management)
- [Budgets & volume quotas](https://cargoship.app/guides/cost/budgets)
- [Alerts & notifications](https://cargoship.app/guides/cost/alerts)
- [Lifecycle & storage classes](https://cargoship.app/guides/cost/lifecycle)

## Features & Optimization

- [Compression & content-aware](https://cargoship.app/guides/features/compression)
- [Magika AI file detection](https://cargoship.app/guides/features/magika)
- [Multi-prefix sharding](https://cargoship.app/guides/features/sharding)
- [Multi-region (LB & failover)](https://cargoship.app/guides/features/multi-region)
- [Encryption (KMS & GPG)](https://cargoship.app/guides/features/encryption)
- [Tier-aware storage](https://cargoship.app/guides/features/tiering)
- [Performance tuning](https://cargoship.app/guides/features/optimization)
- [Observability & tracing](https://cargoship.app/guides/features/observability)
- [Benchmarking](https://cargoship.app/guides/features/benchmarking)

## DVC Integration

- [Overview](https://cargoship.app/guides/dvc/)
- [Go `dvc` command](https://cargoship.app/guides/dvc/command)
- [Python dvc-cargoship plugin](https://cargoship.app/guides/dvc/plugin)

## Configuration

- [Config files & precedence](https://cargoship.app/guides/config/files)
- [Interactive setup wizard](https://cargoship.app/guides/config/setup)
- [Execution contexts](https://cargoship.app/guides/config/contexts)
- [Shell completion](https://cargoship.app/guides/config/completion)

## Reference

- [Command reference overview](https://cargoship.app/reference/)
- [Uploading & sync commands](https://cargoship.app/reference/commands/upload)
- [Inspection & retrieval](https://cargoship.app/reference/commands/inspect)
- [Cost, budget & alerts](https://cargoship.app/reference/commands/cost)
- [DVC commands](https://cargoship.app/reference/commands/dvc)
- [Configuration & context](https://cargoship.app/reference/commands/config)
- [Destructive operations](https://cargoship.app/reference/commands/destructive)
- [Diagnostics & utilities](https://cargoship.app/reference/commands/diagnostics)
- [Global flags](https://cargoship.app/reference/commands/global-flags)
- [Exit codes](https://cargoship.app/reference/exit-codes)
- [Environment variables](https://cargoship.app/reference/environment-variables)
- [Configuration schema](https://cargoship.app/reference/configuration)
- [CargoShip vs. other tools](https://cargoship.app/reference/comparison)
- [Benchmarks & methodology](https://cargoship.app/reference/benchmarks)
- [Recovery & operations runbook](https://cargoship.app/reference/recovery)
- [Troubleshooting](https://cargoship.app/reference/troubleshooting)
- [FAQ](https://cargoship.app/reference/faq)
- [Glossary](https://cargoship.app/reference/glossary)
- [Cheat sheet](https://cargoship.app/reference/cheatsheet)

## Archive & Manifest Format Spec

- [Format spec overview](https://cargoship.app/reference/format/)
- [Archive layout & shard keys](https://cargoship.app/reference/format/archive-layout)
- [Compression levels](https://cargoship.app/reference/format/compression)
- [Manifest schema (v2.0)](https://cargoship.app/reference/format/manifest)
- [Encryption metadata](https://cargoship.app/reference/format/encryption)
- [Split-file PAX records](https://cargoship.app/reference/format/split-files)
- [Reading archives (Go library)](https://cargoship.app/reference/format/library-api)

## Distributed / Enterprise

- [Overview](https://cargoship.app/enterprise/)
- [ghost-ship](https://cargoship.app/enterprise/ghost-ship)
- [Deployment guide](https://cargoship.app/enterprise/deployment)
- [QNAP / NAS deployment](https://cargoship.app/enterprise/qnap)

## Project

- [Architecture](https://cargoship.app/project/architecture)
- [Project maturity & compatibility](https://cargoship.app/project/maturity)
- [Security model](https://cargoship.app/project/security)
- [Integrity model](https://cargoship.app/project/integrity)
- [Verification reports](https://cargoship.app/project/verification-reports)
- [API stability & versioning](https://cargoship.app/project/versioning)
- [Attribution & license](https://cargoship.app/project/attribution)