Jobs go in. Results come out.
Active bindings
hooks
A named list of currently registered callback hooks.
jobs
Get or set - List of jobs currently managed by this
jobqueue
.state
The
jobqueue's
state:'starting'
,'idle'
,'busy'
,'stopped'
, or'error.'
uid
A short string, e.g.
'Q1'
, that uniquely identifies thisjobqueue
.tmp
The
jobqueue
's temporary directory.workers
cnd
The error that caused the
jobqueue
to stop.
Methods
Method new()
Creates a pool of background processes for handling $run()
and
$submit()
calls. These workers
are initialized
according to the globals
, packages
, and init
arguments.
Usage
jobqueue_class$new(
globals = NULL,
packages = NULL,
namespace = NULL,
init = NULL,
max_cpus = availableCores(),
workers = ceiling(max_cpus * 1.2),
timeout = NULL,
hooks = NULL,
reformat = NULL,
signal = FALSE,
cpus = 1L,
stop_id = NULL,
copy_id = NULL
)
Arguments
globals
A named list of variables that all
<job>$expr
s will have access to. Alternatively, an object that can be coerced to a named list withas.list()
, e.g. named vector, data.frame, or environment.packages
Character vector of package names to load on
workers
.namespace
The name of a package to attach to the
worker's
environment.init
A call or R expression wrapped in curly braces to evaluate on each
worker
just once, immediately after start-up. Will have access to variables defined byglobals
and assets frompackages
andnamespace
. Returned value is ignored.max_cpus
Total number of CPU cores that can be reserved by all running
jobs
(sum(<job>$cpus)
). Does not enforce limits on actual CPU utilization.workers
How many background
worker
processes to start. Set to more thanmax_cpus
to enable standbyworkers
to quickly swap out withworkers
that need to restart.timeout, hooks, reformat, signal, cpus, stop_id, copy_id
Defaults for this
jobqueue's
$run()
method. Here only,stop_id
andcopy_id
must be either afunction (job)
orNULL
.hooks
can setjobqueue
,worker
, and/orjob
hooks - see the "Attaching" section invignette('hooks')
.
Method print()
Print method for a jobqueue
.
Method run()
Creates a job
object and submits it to the
jobqueue
for running. Any NA
arguments will be
replaced with their value from jobqueue_class$new()
.
Usage
jobqueue_class$run(
expr,
vars = list(),
timeout = NA,
hooks = NA,
reformat = NA,
signal = NA,
cpus = NA,
stop_id = NA,
copy_id = NA,
...
)
Arguments
expr
A call or R expression wrapped in curly braces to evaluate on a
worker
. Will have access to any variables defined byvars
, as well as thejobqueue's
globals
,packages
, andinit
configuration. Seevignette('eval')
.vars
A named list of variables to make available to
expr
during evaluation. Alternatively, an object that can be coerced to a named list withas.list()
, e.g. named vector, data.frame, or environment. Or afunction (job)
that returns such an object.timeout
A named numeric vector indicating the maximum number of seconds allowed for each state the
job
passes through, or 'total' to apply a single timeout from 'submitted' to 'done'. Can also limit the 'starting' state forworkers
. Afunction (job)
can be used in place of a number. Example:timeout = c(total = 2.5, running = 1)
. Seevignette('stops')
.hooks
A named list of functions to run when the
job
state changes, of the formhooks = list(created = function (worker) {...})
. Or afunction (job)
that returns the same. Names ofworker
hooks are typically'created'
,'submitted'
,'queued'
,'dispatched'
,'starting'
,'running'
,'done'
, or'*'
(duplicates okay). Seevignette('hooks')
.reformat
Set
reformat = function (job)
to define what<job>$result
should return. The default,reformat = NULL
passes<job>$output
to<job>$result
unchanged. Seevignette('results')
.signal
Should calling
<job>$result
signal on condition objects? WhenFALSE
,<job>$result
will return the object without taking additional action. Setting toTRUE
or a character vector of condition classes, e.g.c('interrupt', 'error', 'warning')
, will cause the equivalent ofstop(<condition>)
to be called when those conditions are produced. Alternatively, afunction (job)
that returnsTRUE
orFALSE
. Seevignette('results')
.cpus
How many CPU cores to reserve for this
job
. Or afunction (job)
that returns the same. Used to limit the number ofjobs
running simultaneously to respect<jobqueue>$max_cpus
. Does not prevent ajob
from using more CPUs than reserved.stop_id
If an existing
job
in thejobqueue
has the samestop_id
, thatjob
will be stopped and return an 'interrupt' condition object as its result.stop_id
can also be afunction (job)
that returns thestop_id
to assign to a givenjob
. Astop_id
ofNULL
disables this feature. Seevignette('stops')
.copy_id
If an existing
job
in thejobqueue
has the samecopy_id
, the newly submittedjob
will become a "proxy" for that earlierjob
, returning whatever result the earlierjob
returns.copy_id
can also be afunction (job)
that returns thecopy_id
to assign to a givenjob
. Acopy_id
ofNULL
disables this feature. Seevignette('stops')
....
Arbitrary named values to add to the returned
job
object.
Returns
The new job
object.
Method submit()
Adds a job
to the jobqueue
for
running on a background process.
Arguments
job
A
job
object, as created byjob_class$new()
.
Method wait()
Blocks until the jobqueue
enters the given state.
Method on()
Attach a callback function to execute when the
jobqueue
enters state
.