Jobs go in. Results come out.
Usage
jobqueue(
globals = NULL,
packages = NULL,
namespace = NULL,
init = NULL,
max_cpus = availableCores(),
workers = ceiling(max_cpus * 1.2),
timeout = NULL,
hooks = NULL,
reformat = NULL,
signal = FALSE,
cpus = 1L,
stop_id = NULL,
copy_id = NULL
)
Arguments
- globals
A named list of variables that all
<job>$expr
s will have access to. Alternatively, an object that can be coerced to a named list withas.list()
, e.g. named vector, data.frame, or environment.- packages
Character vector of package names to load on
workers
.- namespace
The name of a package to attach to the
worker's
environment.- init
A call or R expression wrapped in curly braces to evaluate on each
worker
just once, immediately after start-up. Will have access to variables defined byglobals
and assets frompackages
andnamespace
. Returned value is ignored.- max_cpus
Total number of CPU cores that can be reserved by all running
jobs
(sum(<job>$cpus)
). Does not enforce limits on actual CPU utilization.- workers
How many background
worker
processes to start. Set to more thanmax_cpus
to enable standbyworkers
to quickly swap out withworkers
that need to restart.- timeout
A named numeric vector indicating the maximum number of seconds allowed for each state the
job
passes through, or 'total' to apply a single timeout from 'submitted' to 'done'. Can also limit the 'starting' state forworkers
. Afunction (job)
can be used in place of a number. Example:timeout = c(total = 2.5, running = 1)
. Seevignette('stops')
.- hooks
A named list of functions to run when the
job
state changes, of the formhooks = list(created = function (worker) {...})
. Or afunction (job)
that returns the same. Names ofworker
hooks are typically'created'
,'submitted'
,'queued'
,'dispatched'
,'starting'
,'running'
,'done'
, or'*'
(duplicates okay). Seevignette('hooks')
.- reformat
Set
reformat = function (job)
to define what<job>$result
should return. The default,reformat = NULL
passes<job>$output
to<job>$result
unchanged. Seevignette('results')
.- signal
Should calling
<job>$result
signal on condition objects? WhenFALSE
,<job>$result
will return the object without taking additional action. Setting toTRUE
or a character vector of condition classes, e.g.c('interrupt', 'error', 'warning')
, will cause the equivalent ofstop(<condition>)
to be called when those conditions are produced. Alternatively, afunction (job)
that returnsTRUE
orFALSE
. Seevignette('results')
.- cpus
The default number of CPU cores per
job
. Or afunction (job)
that returns the number of CPU cores to reserve for a givenjob
. Used to limit the number ofjobs
running simultaneously to respect<jobqueue>$max_cpus
. Does not prevent ajob
from using more CPUs than reserved.- stop_id
If an existing
job
in thejobqueue
has the samestop_id
, thatjob
will be stopped and return an 'interrupt' condition object as its result.stop_id
can also be afunction (job)
that returns thestop_id
to assign to a givenjob
. Astop_id
ofNULL
disables this feature. Seevignette('stops')
.- copy_id
If an existing
job
in thejobqueue
has the samecopy_id
, the newly submittedjob
will become a "proxy" for that earlierjob
, returning whatever result the earlierjob
returns.copy_id
can also be afunction (job)
that returns thecopy_id
to assign to a givenjob
. Acopy_id
ofNULL
disables this feature. Seevignette('stops')
.
Value
A jobqueue
object.
Examples
jq <- jobqueue(globals = list(N = 42), workers = 2)
print(jq)
#> ── Q1 <jobqueue/R6> ────────────────────── idle ──
#>
#> • 0 jobs - 0 are running
#> • 2 workers - 0 are busy
#> • 0 of 4 CPUs are currently in use
#>
#> ────────────── 0 jobs run in 1 secs ──────────────
job <- jq$run({ paste("N is", N) })
job$result
#> [1] "N is 42"
jq$stop()