These docs track the development branch (main). Latest release: v0.38.2.
CargoShip

Appearance

DVC commands ​

Inspect and export DVC pipeline metadata from a CargoShip archive. See the DVC integration guide for the full workflow, and ML datasets with DVC for a tutorial.

Generated reference

Flag tables below are generated from the CLI and kept in sync by a drift check.

cargoship dvc ​

Inspect and export DVC stage metadata from a CargoShip archive

Synopsis ​

Commands for working with DVC (Data Version Control) stage metadata embedded in CargoShip manifests.

Use 'cargoship upload --dvc-auto' to annotate uploads with per-file stage information from dvc.yaml before using these commands.

Options ​

  -h, --help   help for dvc

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship dvc stages ​

List DVC pipeline stages and their file counts from a manifest

cargoship dvc stages S3_URL [flags]

Options ​

  -h, --help            help for stages
      --json            Output as JSON
      --region string   AWS region (default "us-east-1")

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship dvc status ​

Compare local files against a manifest (unchanged / modified / missing)

cargoship dvc status LOCAL_PATH S3_URL [flags]

Options ​

  -h, --help            help for status
      --json            Output as JSON
      --region string   AWS region (default "us-east-1")
      --stage string    Filter to files from a specific DVC stage

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output

cargoship dvc export ​

Download a manifest and generate .dvc sidecar files

cargoship dvc export S3_URL [OUTPUT_DIR] [flags]

Options ​

      --cache-dir string    Local DVC cache directory (recorded in manifest) (default ".dvc/cache")
  -h, --help                help for export
      --output-dir string   Directory to write .dvc files (default: dvc-files)
      --region string       AWS region (default "us-east-1")

Options inherited from parent commands ​

      --allow-public-observability   Permit the pprof/Prometheus endpoints to bind a non-loopback (public) interface; they are unauthenticated, so this exposes them to the network
      --context string               Override execution context (local, repl)
      --memory-limit string          Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                        Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string            Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile                      Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                        Enable trace messages in output
  -v, --verbose                      Enable verbose output