Download files from an HTC submit node — htc

htc_download() copies one or more files from a directory on an HTC submit node to a local directory via scp. It is the final step in the job submission workflow – called after htc_status() confirms all jobs have completed.

Usage

htc_download(
  files = NULL,
  cluster_id = NULL,
  remote_path = NULL,
  local_path = ".",
  config = NULL,
  dry_run = FALSE,
  verbose = FALSE,
  path = "."
)

Arguments

files: A character vector or NULL. One or more filenames or glob patterns to download from remote_path on the submit node. Examples: "results.tar.gz", c("job.log", "job.err"), "*.tar.gz". When NULL, the function uses cluster_id and the submission state to determine which files to download. Defaults to NULL.
cluster_id: A character string or NULL. The cluster ID returned by htc_submit(). When supplied without files, the function constructs the file list from the submission state. When NULL, falls back to the most recently submitted cluster ID stored in the submission state. Defaults to NULL.
remote_path: A character string or NULL. The directory on the submit node where the files are located. When NULL (the default), resolves to the remote_path recorded in the submission state by the preceding call to htc_submit(), falling back to "~/" if the submission state holds no value. Should match the remote_path used in htc_upload() and htc_submit().
local_path: A character string. The local directory where downloaded files will be saved. Defaults to "." (current working directory).
config: A named list as returned by htc_config(). Must contain username and server. If NULL (the default), uses the session config set by htc_start(). If no session config is set, the function errors with instructions.
dry_run: Logical. If TRUE, prints the scp command that would be executed without running it. Defaults to FALSE.
verbose: Logical. If TRUE, prints progress messages. Defaults to FALSE.
path: A character string. Directory holding the submission state (htc-manifest.yml). This is where the function looks for job metadata; it is not where downloaded files are written, which is local_path. Defaults to ".". If you passed a non-default output or path to htc_gen_submit() and htc_gen_executable(), pass that same directory here.

Value

Called for its side effects. Returns invisible(NULL).

Details

When cluster_id is supplied without files, the function uses the submission state built up by htc_gen_submit(), htc_gen_executable(), and htc_submit() to determine which files to download. For single-mode jobs, this includes the results tarball and the log, error, and output files. For multiple-mode jobs, the function reads the subset names from the submission state and constructs per-job tarball names and per-process log file patterns.

Glob patterns such as "*.tar.gz" are supported when using the files argument and are evaluated on the remote server, not locally.

Automatic file resolution

When files is NULL, the function resolves the file list from the submission state. The submission state is built automatically as you call htc_gen_submit(), htc_gen_executable(), and htc_submit() during the normal workflow. No extra steps are needed.

For a single-mode job:

The results tarball (e.g. "analysis-results.tar.gz")
Log files: "{cluster_id}-0-job.log", ".err", ".out"

For a multiple-mode job:

Per-subset tarballs (e.g. "analysis-adelie-results.tar.gz")
Log files for each process: "{cluster_id}-{0,1,...}-job.log", etc.

Workflow

htc_download() is the final system-facing step in the submitr workflow. Call it after htc_status() confirms all jobs have completed.

# Automatic: uses the submission state to determine what to download
htc_start()
htc_gen_submit(...)
htc_gen_executable(...)
htc_upload(...)
job <- htc_submit("analysis.sub")
htc_status(cluster_id = job, watch = TRUE)
htc_download()

Glob patterns

When using files directly, glob patterns are passed to the remote shell for evaluation so they match files on the submit node, not on your local machine. The pattern is single-quoted in the scp command to prevent local shell expansion.

SSH connection reuse

Each call to htc_download() opens a new SSH connection. If you have not configured ControlMaster in your ~/.ssh/config, this will trigger a Duo MFA prompt. Run htc_config() for setup guidance.

Examples

# \donttest{
# Preview the scp command without connecting to CHTC
cfg <- list(username = "netid", server = "ap2002.chtc.wisc.edu")
htc_download(files = "*.tar.gz", config = cfg, dry_run = TRUE)
#> ✔ Dry run -- command that would be executed:
#>   `scp 'netid@ap2002.chtc.wisc.edu:~/*.tar.gz' .`
# }

if (FALSE) { # \dontrun{
# All remaining examples require a live CHTC connection

# Automatic download after a workflow
htc_start()
htc_gen_submit(...)
htc_gen_executable(...)
htc_upload(...)
job <- htc_submit("job.sub")
htc_status(cluster_id = job, watch = TRUE)
htc_download()

# Download by cluster ID
htc_download(cluster_id = "6590895")

# Download specific files using globs
htc_download(files = "*.tar.gz", local_path = "downloads/")

# Download logs only
htc_download(files = c("*.log", "*.err", "*.out"), local_path = "logs/")
} # }