CargoShip

Appearance

DVC commands ​

Inspect and export DVC pipeline metadata from a CargoShip archive. See the DVC integration guide for the full workflow, and ML datasets with DVC for a tutorial.

Generated reference

Flag tables below are generated from the CLI and kept in sync by a drift check.

cargoship dvc ​

Inspect and export DVC stage metadata from a CargoShip archive

Synopsis ​

Commands for working with DVC (Data Version Control) stage metadata embedded in CargoShip manifests.

Use 'cargoship upload --dvc-auto' to annotate uploads with per-file stage information from dvc.yaml before using these commands.

Options ​

  -h, --help   help for dvc

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship dvc stages ​

List DVC pipeline stages and their file counts from a manifest

cargoship dvc stages S3_URL [flags]

Options ​

  -h, --help            help for stages
      --json            Output as JSON
      --region string   AWS region (default "us-east-1")

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship dvc status ​

Compare local files against a manifest (unchanged / modified / missing)

cargoship dvc status LOCAL_PATH S3_URL [flags]

Options ​

  -h, --help            help for status
      --json            Output as JSON
      --region string   AWS region (default "us-east-1")
      --stage string    Filter to files from a specific DVC stage

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output

cargoship dvc export ​

Download a manifest and generate .dvc sidecar files

cargoship dvc export S3_URL [OUTPUT_DIR] [flags]

Options ​

      --cache-dir string    Local DVC cache directory (recorded in manifest) (default ".dvc/cache")
  -h, --help                help for export
      --output-dir string   Directory to write .dvc files (default: dvc-files)
      --region string       AWS region (default "us-east-1")

Options inherited from parent commands ​

      --context string        Override execution context (local, agent, repl)
      --memory-limit string   Set a memory limit for the run. This will slow things down, but will less likely to OOM in certain situations. Avoid this unless you are having memory issues.
      --pprof                 Enable runtime profiling HTTP endpoint at localhost:6060
      --pprof-addr string     Address for runtime profiling HTTP endpoint (default "localhost:6060")
      --profile               Enable performance profiling. This will generate profile files in a temp directory
  -t, --trace                 Enable trace messages in output
  -v, --verbose               Enable verbose output