( 未来 )
Minimalist Async Evaluation Framework for R
Lightweight parallel code execution and distributed
computing.
mirai()
returns a ‘mirai’ object immediately.
Designed for simplicity, a ‘mirai’ evaluates an R expression
asynchronously, on local or network resources, resolving automatically
upon completion.
State of the art networking and
concurrency via nanonext
offers reliable and efficient scheduling over fast inter-process
communications or TCP/IP secured by TLS.
mirai パッケージを試してみたところ、かなり速くて驚きました
Install the latest release from CRAN:
install.packages("mirai")
Or the development version from R-universe:
install.packages("mirai", repos = "https://shikokuchuo.r-universe.dev")
Use mirai()
to evaluate an expression asynchronously in a separate,
clean R process.
A ‘mirai’ object is returned immediately.
library(mirai)
m <- mirai(
{
res <- rnorm(x) + y ^ 2
res / rev(res)
},
x = 10,
y = runif(1)
)
m
#> < mirai | $data >
Above, all specified name = value
pairs are passed through to the
‘mirai’.
The ‘mirai’ yields an ‘unresolved’ logical NA whilst the async operation is ongoing.
m$data
#> 'unresolved' logi NA
To check whether a mirai has resolved:
unresolved(m)
#> [1] FALSE
Upon completion, the ‘mirai’ resolves automatically to the evaluated result.
m$data
#> [1] -1.0230903 -6.8652249 -0.1568005 0.7824384 0.7390472 1.3530936
#> [7] 1.2780560 -6.3775314 -0.1456617 -0.9774309
Alternatively, explicitly call and wait for the result using
call_mirai()
.
call_mirai(m)$data
#> [1] -1.0230903 -6.8652249 -0.1568005 0.7824384 0.7390472 1.3530936
#> [7] 1.2780560 -6.3775314 -0.1456617 -0.9774309
Daemons are persistent background processes created to receive ‘mirai’ requests.
They may be deployed for:
Local parallel processing; or
Remote network distributed computing.
Launchers allow daemons to be started both on the local machine and across the network via SSH etc.
Secure TLS connections can be automatically-configured on-the-fly for remote daemon connections.
Refer to the {mirai} vignette for full package functionality. This may be accessed within R by:
vignette("mirai", package = "mirai")
The following core integrations are documented, with usage examples in the linked vignettes:
arrow
- the
custom serialization interface allows queries using the Apache Arrow
format to be handled seamlessly over database connections hosted in
daemon processes.
parallel
-
provides an alternative communications backend for R, implementing a
low-level feature request by R-Core at R Project Sprint 2023. Also
includes examples of using foreach
, which is supported via
doParallel
.
promises
-
‘mirai’ may be used interchangeably with ‘promises’ by using the promise
pipe %...>%
or the as.promise()
method.
plumber
-
serves as an asynchronous / distributed backend, scaling applications
via the use of promises.
shiny
- serves as
an asynchronous / distributed backend, plugging directly into the
reactive framework or accepted anywhere that takes a ‘promise’.
torch
- the
custom serialization interface allows tensors and complex objects such
as models and optimizers to be used seamlessly across parallel
processes.
targets
, a Make-like pipeline
tool for statistics and data science, has integrated and adopted crew
as its default high-performance computing backend.
crew
is a distributed
worker-launcher extending mirai
to different distributed computing
platforms, from traditional clusters to cloud services.
crew.cluster
enables
mirai-based workflows on traditional high-performance computing clusters
using LFS, PBS/TORQUE, SGE and SLURM.
crew.aws.batch
extends
mirai
to cloud computing using AWS Batch.
We would like to thank in particular:
Will Landau, for being instrumental in
shaping development of the package, from initiating the original request
for persistent daemons, through to orchestrating robustness testing for
the high performance computing requirements of crew
and targets
.
Joe Cheng, for optimising the promises
method to make mirai
work seamlessly within Shiny, and prototyping
non-polling promises, which is implemented across nanonext
and
mirai
.
Luke Tierney, R Core, for discussion on
L’Ecuyer-CMRG streams to ensure statistical independence in parallel
processing, and making it possible for mirai
to be the first
‘alternative communications backend for R’.
Henrik Bengtsson, for valuable insights leading to the interface accepting broader usage patterns.
Daniel Falbel, for discussion around an
efficient solution to serialization and transmission of torch
tensors.
Kirill Müller, for discussion on using ‘daemons’ to host Arrow database connections.
◈ mirai R package: https://shikokuchuo.net/mirai/
mirai is listed in CRAN Task View:
- High Performance Computing:
https://cran.r-project.org/view=HighPerformanceComputing
◈ nanonext R package: https://shikokuchuo.net/nanonext/
NNG website: https://nng.nanomsg.org/
–
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.