Skip to contents

Assigns Jobs to Workers (R6 Class)

Source: R/jobqueue_class.r
jobqueue_class.Rd

Jobs go in. Results come out.

Active bindings

hooks

A named list of currently registered callback hooks.

jobs

Get or set - List of jobs currently managed by this jobqueue.

state

The jobqueue's state: 'starting', 'idle', 'busy', 'stopped', or 'error.'

uid

A short string, e.g. 'Q1', that uniquely identifies this jobqueue.

tmp

The jobqueue's temporary directory.

workers

Get or set - List of workers used for processing jobs.

cnd

The error that caused the jobqueue to stop.

Methods

Public methods

Method new()

Creates a pool of background processes for handling $run() and $submit() calls. These workers are initialized according to the globals, packages, and init arguments.

Usage

jobqueue_class$new(
  globals = NULL,
  packages = NULL,
  namespace = NULL,
  init = NULL,
  max_cpus = availableCores(),
  workers = ceiling(max_cpus * 1.2),
  timeout = NULL,
  hooks = NULL,
  reformat = NULL,
  signal = FALSE,
  cpus = 1L,
  stop_id = NULL,
  copy_id = NULL
)

Arguments

globals

A named list of variables that all <job>$exprs will have access to. Alternatively, an object that can be coerced to a named list with as.list(), e.g. named vector, data.frame, or environment.

packages

Character vector of package names to load on workers.

namespace

The name of a package to attach to the worker's environment.

init

A call or R expression wrapped in curly braces to evaluate on each worker just once, immediately after start-up. Will have access to variables defined by globals and assets from packages and namespace. Returned value is ignored.

max_cpus

Total number of CPU cores that can be reserved by all running jobs (sum(<job>$cpus)). Does not enforce limits on actual CPU utilization.

workers

How many background worker processes to start. Set to more than max_cpus to enable standby workers to quickly swap out with workers that need to restart.

timeout, hooks, reformat, signal, cpus, stop_id, copy_id

Defaults for this jobqueue's $run() method. Here only, stop_id and copy_id must be either a function (job) or NULL. hooks can set jobqueue, worker, and/or job hooks - see the "Attaching" section in vignette('hooks').

Returns

A jobqueue object.

Method print()

Print method for a jobqueue.

Usage

jobqueue_class$print(...)

Arguments

...

Arguments are not used currently.

Method run()

Creates a job object and submits it to the jobqueue for running. Any NA arguments will be replaced with their value from jobqueue_class$new().

Usage

jobqueue_class$run(
  expr,
  vars = list(),
  timeout = NA,
  hooks = NA,
  reformat = NA,
  signal = NA,
  cpus = NA,
  stop_id = NA,
  copy_id = NA,
  ...
)

Arguments

expr

A call or R expression wrapped in curly braces to evaluate on a worker. Will have access to any variables defined by vars, as well as the jobqueue's globals, packages, and init configuration. See vignette('eval').

vars

A named list of variables to make available to expr during evaluation. Alternatively, an object that can be coerced to a named list with as.list(), e.g. named vector, data.frame, or environment. Or a function (job) that returns such an object.

timeout

A named numeric vector indicating the maximum number of seconds allowed for each state the job passes through, or 'total' to apply a single timeout from 'submitted' to 'done'. Can also limit the 'starting' state for workers. A function (job) can be used in place of a number. Example: timeout = c(total = 2.5, running = 1). See vignette('stops').

hooks

A named list of functions to run when the job state changes, of the form hooks = list(created = function (worker) {...}). Or a function (job) that returns the same. Names of worker hooks are typically 'created', 'submitted', 'queued', 'dispatched', 'starting', 'running', 'done', or '*' (duplicates okay). See vignette('hooks').

reformat

Set reformat = function (job) to define what <job>$result should return. The default, reformat = NULL passes <job>$output to <job>$result unchanged. See vignette('results').

signal

Should calling <job>$result signal on condition objects? When FALSE, <job>$result will return the object without taking additional action. Setting to TRUE or a character vector of condition classes, e.g. c('interrupt', 'error', 'warning'), will cause the equivalent of stop(<condition>) to be called when those conditions are produced. Alternatively, a function (job) that returns TRUE or FALSE. See vignette('results').

cpus

How many CPU cores to reserve for this job. Or a function (job) that returns the same. Used to limit the number of jobs running simultaneously to respect <jobqueue>$max_cpus. Does not prevent a job from using more CPUs than reserved.

stop_id

If an existing job in the jobqueue has the same stop_id, that job will be stopped and return an 'interrupt' condition object as its result. stop_id can also be a function (job) that returns the stop_id to assign to a given job. A stop_id of NULL disables this feature. See vignette('stops').

copy_id

If an existing job in the jobqueue has the same copy_id, the newly submitted job will become a "proxy" for that earlier job, returning whatever result the earlier job returns. copy_id can also be a function (job) that returns the copy_id to assign to a given job. A copy_id of NULL disables this feature. See vignette('stops').

...

Arbitrary named values to add to the returned job object.

Returns

The new job object.

Method submit()

Adds a job to the jobqueue for running on a background process.

Usage

jobqueue_class$submit(job)

Arguments

job

A job object, as created by job_class$new().

Returns

This jobqueue, invisibly.

Method wait()

Blocks until the jobqueue enters the given state.

Usage

jobqueue_class$wait(state = "idle", timeout = NULL, signal = TRUE)

Arguments

state

The name of a jobqueue state. Typically one of:

  • '*' - Every time the state changes.

  • '.next' - Only one time, the next time the state changes.

  • 'starting' - workers are starting.

  • 'idle' - All workers are ready/idle.

  • 'busy' - At least one worker is busy.

  • 'stopped' - Shutdown is complete.

timeout

Stop the jobqueue if it takes longer than this number of seconds, or NULL.

signal

Raise an error if encountered (will also be recorded in <jobqueue>$cnd).

Returns

This jobqueue, invisibly.

Method on()

Attach a callback function to execute when the jobqueue enters state.

Usage

jobqueue_class$on(state, func)

Arguments

state

The name of a jobqueue state. Typically one of:

  • '*' - Every time the state changes.

  • '.next' - Only one time, the next time the state changes.

  • 'starting' - workers are starting.

  • 'idle' - All workers are ready/idle.

  • 'busy' - At least one worker is busy.

  • 'stopped' - Shutdown is complete.

func

A function that accepts a jobqueue object as input. Return value is ignored.

Returns

A function that when called removes this callback from the jobqueue.

Method stop()

Stop all jobs and workers.

Usage

jobqueue_class$stop(reason = "jobqueue shut down by user", cls = NULL)

Arguments

reason

Passed to <job>$stop() for any jobs currently managed by this jobqueue.

cls

Passed to <job>$stop() for any jobs currently managed by this jobqueue.

Returns

This jobqueue, invisibly.