index.Rmd

---
title: "The drake R Package User Manual"
author:
- "Will Landau, Kirill Müller, Alex Axthelm, Jasper Clarkberg, Lorenz Walthert, Ellis Hughes, Matthew Mark Strasiotto"
- "Copyright Eli Lilly and Company"
site: bookdown::bookdown_site
documentclass: book
output:
  bookdown::gitbook:
    includes:
      in_header: google_analytics.html
      after_body: footer.html
url: "https://books.ropensci.org/drake/"
description: "In-depth walkthroughs and examples of drake, an R package for reproducible computation at scale."
github-repo: "ropensci-books/drake"
cover-image: /images/logo.png
apple-touch-icon: "images/apple-touch-icon.png"
apple-touch-icon-size: 120
favicon: "images/favicon.ico"
always_allow_html: yes
---

```{r, message = FALSE, warning = FALSE, echo = FALSE}
knitr::opts_knit$set(root.dir = fs::dir_create(tempfile()))
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = TRUE)
options(
  drake_make_menu = FALSE,
  drake_clean_menu = FALSE,
  warnPartialMatchArgs = FALSE,
  crayon.enabled = FALSE,
  readr.show_progress = FALSE,
  tidyverse.quiet = TRUE
)
```

# Introduction {#intro}

## Consider targets

<a href="https://www.tidyverse.org/lifecycle/#superseded"><img src="https://img.shields.io/badge/lifecycle-superseded-blue.svg" alt='superseded lifecycle'></a>

`drake` is [superseded](https://www.tidyverse.org/lifecycle/#superseded). The [`targets`](https://docs.ropensci.org/targets/) R package is the long-term successor of `drake`, and it is more robust and easier to use. Please visit <https://books.ropensci.org/targets/drake.html> for full context and advice on transitioning.

## Video

### That Feeling of Workflowing

<iframe width="560" height="315" src="https://www.youtube.com/embed/jU1Zv21GvT4" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

(By [Miles McBain](https://github.com/MilesMcBain); [venue](https://nyhackr.org/index.html), [resources](https://github.com/MilesMcBain/nycr_meetup_talk))

### rOpenSci Community Call

<iframe title="vimeo-player" src="https://player.vimeo.com/video/362383350" width="640" height="360" frameborder="0" allowfullscreen></iframe>

([resources](https://ropensci.org/commcalls/2019-09-24/))

## The drake R package 

Data analysis can be slow. A round of scientific computation can take several minutes, hours, or even days to complete. After it finishes, if you update your code or data, your hard-earned results may no longer be valid. How much of that valuable output can you keep, and how much do you need to update? How much runtime must you endure all over again?

For projects in R, the `drake` package can help. It [analyzes your workflow](https://books.ropensci.org/drake/plans.html), skips steps with up-to-date results, and orchestrates the rest with [optional distributed computing](https://books.ropensci.org/drake/hpc.html). At the end, `drake` provides evidence that your results match the underlying code and data, which increases your ability to trust your research.

## Installation

You can choose among different versions of `drake`. The latest CRAN release may be more convenient to install, but this manual is kept up to date with the GitHub version, so some features described here may not yet be available on CRAN.

```{r, eval = FALSE}
# Install the latest stable release from CRAN.
install.packages("drake")

# Alternatively, install the development version from GitHub.
install.packages("devtools")
library(devtools)
install_github("ropensci/drake")
```

## Why drake?

### What gets done stays done.

Too many data science projects follow a [Sisyphean loop](https://en.wikipedia.org/wiki/Sisyphus):

1. Launch the code.
2. Wait while it runs.
3. Discover an issue.
4. Restart from scratch.

For projects with long runtimes, people tend to get stuck. But with `drake`, you can automatically

1. Launch the parts that changed since last time.
2. Skip the rest.

### Reproducibility with confidence

The R community emphasizes reproducibility. Traditional themes include [scientific replicability](https://en.wikipedia.org/wiki/Replication_crisis), literate programming with [knitr](https://yihui.name/knitr/), and version control with [git](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control). But internal consistency is important too. Reproducibility carries the promise that your output matches the code and data you say you used. With the exception of any [triggers](#triggers) suppressed by the user, `drake` strives to keep this promise.

#### Evidence

Suppose you are reviewing someone else's data analysis project for reproducibility. You scrutinize it carefully, checking that the datasets are available and the documentation is thorough. But could you re-create the results without the help of the original author? With `drake`, it is quick and easy to find out.

```{r, eval = FALSE}
make(plan) # See also r_make().

outdated(plan) # See also r_outdated().
```

With everything already up to date, you have **tangible evidence** of reproducibility. Even though you did not re-create the results, you know the results are re-creatable. They **faithfully show** what the code is producing. Given the right [package environment](https://rstudio.github.io/packrat/) and [system configuration](https://stat.ethz.ch/R-manual/R-devel/library/utils/html/sessionInfo.html), you have everything you need to reproduce all the output by yourself.

#### Ease

When it comes time to actually rerun the entire project, you have much more confidence. Starting over from scratch is trivially easy.

```{r, eval = FALSE}
clean()    # Remove the original author's results.
make(plan) # Independently re-create the results from the code and input data.
```

#### Independent replication

With even more evidence and confidence, you can invest the time to independently replicate the original code base if necessary. Up until this point, you relied on basic `drake` functions such as `make()`, so you may not have needed to peek at any substantive author-defined code in advance. In that case, you can stay usefully ignorant as you reimplement the original author's methodology. In other words, `drake` could potentially improve the integrity of independent replication.

#### Big data efficiency

Select a specialized data format to increase speed and reduce memory consumption. In version 7.5.2.9000 and above, the available formats are ["fst"](https://github.com/fstpackage/fst) for data frames (example below) and "keras" for [Keras](https://keras.rstudio.com/) models ([example here](https://books.ropensci.org/drake/churn.html#plan)).

```{r, eval = FALSE}
library(drake)
n <- 1e8 # Each target is 1.6 GB in memory.
plan <- drake_plan(
  data_fst = target(
    data.frame(x = runif(n), y = runif(n)),
    format = "fst"
  ),
  data_old = data.frame(x = runif(n), y = runif(n))
)
make(plan)
#> target data_fst
#> target data_old
build_times(type = "build")
#> # A tibble: 2 x 4
#>   target   elapsed              user                 system    
#>   <chr>    <Duration>           <Duration>           <Duration>
#> 1 data_fst 13.93s               37.562s              7.954s    
#> 2 data_old 184s (~3.07 minutes) 177s (~2.95 minutes) 4.157s
```

#### History

As of version 7.5.0, `drake` tracks the history of your analysis: what you built, when you built it, how you built it, the arguments you used in your function calls, and how to get the data back. (Disable with `make(history = FALSE)`)

```{r, eval = FALSE}
drake_history(analyze = TRUE)
#> # A tibble: 7 x 8
#>   target  time        hash   exists command            runtime latest quiet
#>   <chr>   <chr>       <chr>  <lgl>  <chr>                <dbl> <lgl>  <lgl>
#> 1 data    2019-06-23… e580e… TRUE   raw_data %>% muta… 0.001   TRUE   NA   
#> 2 fit     2019-06-23… 62a16… TRUE   lm(Ozone ~ Temp +… 0.00300 TRUE   NA   
#> 3 hist    2019-06-23… 10bcd… TRUE   create_plot(data)  0.00500 FALSE  NA   
#> 4 hist    2019-06-23… 00fad… TRUE   create_plot(data)  0.00300 TRUE   NA   
#> 5 raw_da… 2019-06-23… 63172… TRUE   "readxl::read_exc… 0.00900 TRUE   NA   
#> 6 report  2019-06-23… dd965… TRUE   "rmarkdown::rende… 0.476   FALSE  TRUE 
#> 7 report  2019-06-23… dd965… TRUE   "rmarkdown::rende… 0.369   TRUE   TRUE
```

The history has arguments like `quiet` (because of the call to `knit(quiet = TRUE)`) and hashes to help you recover old data. To learn more, see the end of the [walkthrough chapter](#walkthrough) and the [`drake_history()`](https://docs.ropensci.org/drake/reference/drake_history.html) help file.

#### Reproducible recovery

`drake`'s data recovery feature is another way to avoid rerunning commands. It is useful if:

- You want to revert to your old code, maybe with `git reset`.
- You accidentally `clean()`ed a target and to get it back.
- You want to rename an expensive target.

See the [walkthrough chapter](#walkthrough) for details.

#### Readability and transparency

Ideally, independent observers should be able to read your code and understand it. `drake` helps in several ways.

- The [`drake` plan](https://docs.ropensci.org/drake/reference/drake_plan.html) explicitly outlines the steps of the analysis, and [`vis_drake_graph()`](https://docs.ropensci.org/drake/reference/vis_drake_graph.html) visualizes how those steps depend on each other.
- `drake` takes care of the parallel scheduling and high-performance computing (HPC) for you. That means the HPC code is no longer tangled up with the code that actually expresses your ideas.
- You can [generate large collections of targets](https://books.ropensci.org/drake/plans#large-plans) without necessarily changing your code base of imported functions, another nice separation between the concepts and the execution of your workflow

### Scale up and out.

Not every project can complete in a single R session on your laptop. Some projects need more speed or computing power. Some require a few local processor cores, and some need large high-performance computing systems. But parallel computing is hard. Your tables and figures depend on your analysis results, and your analyses depend on your datasets, so some tasks must finish before others even begin. `drake` knows what to do. Parallelism is implicit and automatic. See the [high-performance computing guide](https://books.ropensci.org/drake/hpc.html) for all the details.

```{r, eval = FALSE}
# Use the spare cores on your local machine.
options(clustermq.scheduler = "multicore")
make(plan, parallelism = "clustermq", jobs = 4)

# Or scale up to a supercomputer.
drake_hpc_tmpl_file("slurm_clustermq.tmpl") # https://slurm.schedmd.com/
options(
  clustermq.scheduler = "slurm",
  clustermq.template = "slurm_clustermq.tmpl"
)
make(plan, parallelism = "clustermq", jobs = 100)
```

## With Docker

`drake` and Docker are compatible and complementary. Here are some examples that run `drake` inside a Docker image.

- [`drake-gitlab-docker-example`](https://gitlab.com/ecohealthalliance/drake-gitlab-docker-example): A small pedagogical example workflow that leverages `drake`, Docker, GitLab, and continuous integration in a reproducible analysis pipeline. Created by [Noam Ross](https://www.noamross.net/).
- [`pleurosoriopsis`](https://github.com/joelnitta/pleurosoriopsis): The workflow that supports [Ebihara *et al.* 2019. "Growth Dynamics of the Independent Gametophytes of *Pleurorosiopsis makinoi* (Polypodiaceae)" *Bulletin of the National Science Museum Series B (Botany)* 45:77-86.](https://www.kahaku.go.jp/research/publication/botany.html). Created by [Joel Nitta](https://github.com/joelnitta).

Alternatively, it is possible to run `drake` outside Docker and use the [`future`](https://github.com/HenrikBengtsson/future) package to send targets to a Docker image. `drake`'s [`Docker-psock`](https://github.com/wlandau/drake-examples/tree/main/Docker-psock) example demonstrates how. Download the code with `drake_example("Docker-psock")`.

## Documentation

### Core concepts

The following resources explain what `drake` can do and how it works. The [`learndrake`](https://github.com/wlandau/learndrake) workshop devotes particular attention to `drake`'s mental model.

- The [user manual](https://books.ropensci.org/drake/).
- [`drakeplanner`](https://github.com/wlandau/drakeplanner), an R/Shiny app to help learn `drake` and create new projects. Run locally with `drakeplanner::drakeplanner()` or access it at <https://wlandau.shinyapps.io/drakeplanner>.
- [`learndrake`](https://github.com/wlandau/learndrake), an R package for teaching an extended `drake` workshop. It contains notebooks, slides, Shiny apps, the latter two of which are publicly deployed. See the [README](https://github.com/wlandau/learndrake/blob/main/README.md) for instructions and links.

### In practice

- [Miles McBain](https://github.com/MilesMcBain)'s [excellent blog post](https://milesmcbain.xyz/the-drake-post/) explains the motivating factors and practical issues {drake} addresses for most projects, how to set up a project as quickly and painlessly as possible, and how to  overcome common obstacles.
- Miles' [`dflow`](https://github.com/MilesMcBain/dflow) package generates the file structure for a boilerplate `drake` project. It is a more thorough alternative to `drake::use_drake()`.
- `drake` is heavily function-oriented by design, and Miles' [`fnmate`](https://github.com/MilesMcBain/fnmate) package automatically generates boilerplate code and docstrings for functions you mention in `drake` plans.

### Use cases

The official [rOpenSci use cases](https://discuss.ropensci.org/c/usecases) and [associated discussion threads](https://discuss.ropensci.org/c/usecases) describe applications of `drake` in the real world. Many of these use cases are linked from the [`drake` tag on the rOpenSci discussion forum](https://discuss.ropensci.org/tag/drake).

Here are some additional applications of `drake` in real-world projects.

- [efcaguab/demografia-del-voto](https://github.com/efcaguab/demografia-del-voto)
- [efcaguab/great-white-shark-nsw](https://github.com/efcaguab/great-white-shark-nsw)
- [IndianaCHE/Detailed-SSP-Reports](https://github.com/IndianaCHE/Detailed-SSP-Reports)
- [joelnitta/pleurosoriopsis](https://github.com/joelnitta/pleurosoriopsis)
- [pat-s/pathogen-modeling](https://github.com/pat-s/pathogen-modeling)
- [sol-eng/tensorflow-w-r](https://github.com/sol-eng/tensorflow-w-r)
- [tiernanmartin/home-and-hope](https://github.com/tiernanmartin/home-and-hope)

### `drake` projects as R packages

Some folks like to structure their `drake` workflows as R packages. Examples are below. In your own analysis packages, be sure to supply the namespace of your package to the `envir` argument of `make()` and friends (e.g. `make(envir = getNamespace("yourPackage")` so `drake` can watch you package's functions for changes and rebuild downstream targets accordingly.

- [b-rodrigues/coolmlproject](https://github.com/b-rodrigues/coolmlproject)
- [tiernanmartin/drakepkg](https://github.com/tiernanmartin/drakepkg)

### Frequently asked questions

The [FAQ page](https://books.ropensci.org/drake/faq.html) is an index of links to [appropriately-labeled issues on GitHub](https://github.com/ropensci/drake/issues?q=is%3Aissue+is%3Aopen+label%3A%22frequently+asked+question%22). To contribute, please [submit a new issue](https://github.com/ropensci/drake/issues/new) and ask that it be labeled as a frequently asked question.

### Reference

- The [reference website](https://docs.ropensci.org/drake/).
- The [official repository of example code](https://github.com/wlandau/drake-examples). Download an example workflow from here with `drake_example()`.
- Presentations and workshops by [Will Landau](https://github.com/wlandau), [Kirill Müller](https://github.com/krlmlr), [Amanda Dobbyn](https://github.com/aedobbyn), [Karthik Ram](http://github.com/karthik), [Sina Rüeger](https://github.com/sinarueeger), [Christine Stawitz](https://github.com/cstawitz), and others. See specific links at <https://books.ropensci.org/drake/index.html#presentations>
- The [FAQ page](https://books.ropensci.org/drake/faq.html), which links to [appropriately-labeled issues on GitHub](https://github.com/ropensci/drake/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%22frequently+asked+question%22+).

### Function reference

The [reference section](https://docs.ropensci.org/drake/reference/index.html) lists all the available functions. Here are the most important ones.

- `drake_plan()`: create a workflow data frame (like `my_plan`).
- `make()`: build your project.
- `drake_history()`: show what you built, when you built it, and the function arguments you used.
- `loadd()`: load one or more built targets into your R session.
- `readd()`: read and return a built target.
- `vis_drake_graph()`: show an interactive visual network representation of your workflow.
- `outdated()`: see which targets will be built in the next `make()`.
- `deps_code()`: check the dependencies of a command or function.
- `drake_failed()`: list the targets that failed to build in the last `make()`.
- `diagnose()`: return the full context of a build, including errors, warnings, and messages.

### Tutorials

Thanks to [Kirill](https://github.com/krlmlr) for constructing two interactive [`learnr`](https://rstudio.github.io/learnr/) tutorials: [one supporting `drake` itself](https://krlmlr.shinyapps.io/cooking-drake-tutorial/), and a [prerequisite walkthrough](https://krlmlr.shinyapps.io/cooking-tutorial/) of the [`cooking` package](https://github.com/krlmlr/cooking).

### Examples

The official [rOpenSci use cases](https://ropensci.org/usecases/) and [associated discussion threads](https://discuss.ropensci.org/c/usecases) describe applications of `drake` in action. Here are some more real-world sightings of `drake` in the wild.

- [ecohealthalliance/drake-gitlab-docker-example](https://gitlab.com/ecohealthalliance/drake-gitlab-docker-example)
- [efcaguab/demografia-del-voto](https://github.com/efcaguab/demografia-del-voto)
- [efcaguab/great-white-shark-nsw](https://github.com/efcaguab/great-white-shark-nsw)
- [IndianaCHE/Detailed-SSP-Reports](https://github.com/IndianaCHE/Detailed-SSP-Reports)
- [joelnitta/pleurosoriopsis](https://github.com/joelnitta/pleurosoriopsis)
- [pat-s/pathogen-modeling](https://github.com/pat-s/pathogen-modeling)
- [sol-eng/tensorflow-w-r](https://github.com/sol-eng/tensorflow-w-r)
- [tiernanmartin/home-and-hope](https://github.com/tiernanmartin/home-and-hope)

There are also multiple `drake`-powered example projects [available here](https://github.com/wlandau/drake-examples), ranging from beginner-friendly stubs to demonstrations of high-performance computing. You can generate the files for a project with `drake_example()` (e.g. `drake_example("gsp")`), and you can list the available projects with `drake_examples()`. You can contribute your own example project with a [fork and pull request](https://github.com/wlandau/drake-examples/pulls).

### Presentations

| Author | Venue  | Date   | Materials |
|--------|--------|--------|-----------|
| [Will Landau](https://github.com/wlandau) | [R/Medicine 2020 Conference](https://user2020.r-project.org/) | 2020-08-28 | [video](https://youtu.be/swkyS--Iho4), [source](https://github.com/wlandau/rmedicine2020), [slides](https://wlandau.github.io/rmedicine2020) |
| [Miles McBain](https://github.com/MilesMcBain) | [New York Open Statistical Programming Meetup](https://nyhackr.org/) | 2020-08-10 | [Meetup](https://www.meetup.com/nyhackr/events/272228751/), [source](https://github.com/MilesMcBain/nycr_meetup_talk), [slides](https://docs.google.com/presentation/d/18tEfBymtD50g3gTM2s2hfdRaoJTXRSohRSJxHb-FgU4/edit?usp=sharing), [video](https://www.youtube.com/watch?v=jU1Zv21GvT4) |
| [Will Landau](https://github.com/wlandau) | [useR! 2020 Conference](https://user2020.r-project.org/) | 2020-08-04 | [workspace](https://rstudio.cloud/project/627076), [source](https://github.com/wlandau/learndrake), [slides](https://wlandau.github.io/learndrake) |
| [Bruno Rodrigues](http://github.com/b-rodrigues) | YouTube | 2020-05-11 | [video](https://youtu.be/yNHwM3N8bAQ), [source](https://github.com/b-rodrigues/coolmlproject) |
| [Matt Dray](https://github.com/matt-dray) | [Bioinformatics London Meetup](https://www.meetup.com/Bioinformatics-London/) | 2020-01-30 | [slides](https://github.com/matt-dray/drake-bioinformatics), [source](https://matt-dray.github.io/drake-bioinformatics/) |
| [Matt Dray](https://github.com/matt-dray) | Coffee & Coding, [UK Dept for Transport](https://www.gov.uk/government/organisations/department-for-transport) | 2019-10-02 | [slides](https://github.com/matt-dray/drake-egg-rap/blob/master/docs/drake-presentation.pdf) |
[Patrick Schratz](https://github.com/pat-s) | [whyR Conference](http://whyr.pl/2019/) | 2019-09-27 | [workshop](https://github.com/mlr-org/mlr3-learndrake), [slides](https://rawcdn.githack.com/mlr-org/mlr3-learndrake/c46dda78d2e4177a1e458c218056c3d00ec55407/slides/drake/index.html), [source](https://github.com/mlr-org/mlr3-learndrake) |
| [Will Landau](https://github.com/wlandau) | [rOpenSci Community Calls](https://ropensci.org/commcalls) | 2019-09-24 | [Video recording and resource links](https://ropensci.org/commcalls/2019-09-24/)
| [Will Landau](https://github.com/wlandau) | [R/Pharma 2019](http://rinpharma.com/) | 2019-08-21 | [slides](https://wlandau.github.io/learndrake/#1), [workspace](https://mybinder.org/v2/gh/wlandau/learndrake/binder?urlpath=rstudio), [source](https://github.com/wlandau/learndrake) |
| [Garrick Aden-Buie](https://github.com/gadenbuie) | [Bio-Data Club at Moffitt Cancer Center](https://www.biodataclub.org/) | 2019-07-19 | [slides](https://pkg.garrickadenbuie.com/drake-intro/), [workspace](https://rstudio.cloud/project/405721), [source](https://github.com/gadenbuie/drake-intro) |
| [Tiernan Martin](https://github.com/tiernanmartin) | [Cascadia R Conference](https://cascadiarconf.org) | 2019-06-08 | [slides](https://github.com/tiernanmartin/drakepkg/raw/master/drakepkg-slides-cascadiarconf2019.pdf) |
| [Dominik Rafacz](https://github.com/DominikRafacz) | [satRday Gdansk](https://gdansk2019.satrdays.org) | 2019-05-18 | [slides](https://github.com/DominikRafacz/drake-presentation/blob/master/presentation.pdf), [source](https://github.com/DominikRafacz/drake-presentation) | 
| [Amanda Dobbyn](https://github.com/aedobbyn) | [R-Ladies NYC](http://www.rladiesnyc.org/) | 2019-02-12 | [slides](https://aedobbyn.github.io/nyc-fires/index.html#1), [source](https://github.com/aedobbyn/nyc-fires) |
| [Will Landau](https://github.com/wlandau) | [Harvard DataFest](https://projects.iq.harvard.edu/datafest2019/home) | 2019-01-22 | [slides](https://wlandau.github.io/drake-datafest-2019), [source](https://github.com/wlandau/drake-datafest-2019) |
| [Karthik Ram](http://github.com/karthik) | [RStudio Conference](https://www.rstudio.com/conference) | 2019-01-18  | [video](https://resources.rstudio.com/rstudio-conf-2019/a-guide-to-modern-reproducible-data-science-with-r), [slides](http://inundata.org/talks/rstd19/#/),  [resources](https://github.com/karthik/rstudio2019) |
| [Sina Rüeger](https://github.com/sinarueeger) | [Geneva R User Group](https://www.meetup.com/Geneve-R-User-Group) | 2018-10-04 | [slides](https://sinarueeger.github.io/20181004-geneve-rug), [example code](https://github.com/sinarueeger/workflow-example) |
| [Will Landau](https://github.com/wlandau) | [R in Pharma](http://rinpharma.com/) | 2018-08-16 | [video](https://books.ropensci.org/drake/), [slides](https://wlandau.github.io/drake-talk), [source](https://github.com/wlandau/drake-talk) |
| [Christine Stawitz](https://github.com/cstawitz) | [R-Ladies Seattle](https://www.meetup.com/rladies-seattle) | 2018-06-25 | [materials](https://github.com/cstawitz/RLadies_Sea_drake) |
| [Kirill Müller](https://github.com/krlmlr) | [Swiss Institute of Bioinformatics](https://www.sib.swiss) | 2018-03-05 | [workshop](https://www.sib.swiss/training/course/2018-03-remake), [slides](https://krlmlr.github.io/slides/drake-sib-zurich), [source](https://github.com/krlmlr/drake-sib-zurich),  [exercises](https://krlmlr.github.io/slides/drake-sib-zurich/cooking.html) |

### Context and history

For context and history, check out [this post on the rOpenSci blog](https://ropensci.org/blog/2018/02/06/drake/) and [episode 22 of the R Podcast](https://www.r-podcast.org/episode/022-diving-in-to-drake-with-will-landau/).

## Help and troubleshooting

The [GitHub issue tracker](https://github.com/ropensci/drake/issues) is the best place to request help with your use case. Please search both open and closed ones before posting a new issue. Don't be afraid to open a new issue, just please take 30 seconds to search for existing threads that could solve your problem.

# Similar work

`drake` enhances reproducibility and high-performance computing, but not in all respects. [Literate programming](https://rmarkdown.rstudio.com/), [local library managers](https://rstudio.github.io/packrat), [containerization](https://www.docker.com/), and [strict session managers](https://github.com/tidyverse/reprex) offer more robust solutions in their respective domains. And for the problems `drake` *does* solve, it stands on the shoulders of the giants that came before.

## Pipeline tools

### GNU Make

The original idea of a time-saving reproducible build system extends back at least as far as [GNU Make](https://www.gnu.org/software/make/), which still aids the work of [data scientists](http://blog.kaggle.com/2012/10/15/make-for-data-scientists/) as well as the original user base of compiled language programmers. In fact, the name "drake" stands for "Data Frames in R for Make". [Make](https://kbroman.org/minimal_make/) is used widely in reproducible research. Below are some examples from [Karl Broman's website](https://kbroman.org/minimal_make/).

- Bostock, Mike (2013). "A map of flowlines from NHDPlus." https://github.com/mbostock/us-rivers. Powered by the Makefile at https://github.com/mbostock/us-rivers/blob/master/Makefile.
- Broman, Karl W (2012). "Halotype Probabilities in Advanced Intercross Populations." *G3* 2(2), 199-202.Powered by the `Makefile` at https://github.com/kbroman/ailProbPaper/blob/master/Makefile.
- Broman, Karl W (2012). "Genotype Probabilities at Intermediate Generations in the Construction of Recombinant Inbred Lines." *Genetics 190(2), 403-412. Powered by the Makefile at https://github.com/kbroman/preCCProbPaper/blob/master/Makefile.
- Broman, Karl W and Kim, Sungjin and Sen, Saunak and Ane, Cecile and Payseur, Bret A (2012). "Mapping Quantitative Trait Loci onto a Phylogenetic Tree." *Genetics* 192(2), 267-279. Powered by the `Makefile` at https://github.com/kbroman/phyloQTLpaper/blob/master/Makefile.

Whereas [GNU Make](https://www.gnu.org/software/make/) is language-agnostic, `drake` is fundamentally designed for R.

- Instead of a [Makefile](https://github.com/kbroman/preCCProbPaper/blob/master/Makefile), `drake` supports an R-friendly [domain-specific language](https://books.ropensci.org/drake/plans.html#large-plans) for declaring targets.
- Targets in [GNU Make](https://www.gnu.org/software/make/) are files, whereas targets in `drake` are arbitrary variables in memory. (`drake` does have opt-in support for files via `file_out()`, `file_in()`, and `knitr_in()`.) `drake` caches these objects in its own [storage system](https://github.com/richfitz/storr) so R users rarely have to think about output files.

### Remake

[remake](https://github.com/richfitz/remake) itself is no longer maintained, but its founding design goals and principles live on through [drake](https://github.com/ropensci/drake). In fact, [drake](https://github.com/ropensci/drake) is a direct reimagining of [remake](https://github.com/richfitz/remake) with enhanced scalability, reproducibility, high-performance computing, visualization, and documentation.

### Factual's Drake

[Factual's Drake](https://github.com/Factual/drake) is similar in concept, but the development effort is completely unrelated to the [drake R package](https://github.com/ropensci/drake).

### Other pipeline tools

There are [countless other successful pipeline toolkits](https://github.com/pditommaso/awesome-pipeline). The `drake` package distinguishes itself with its R-focused approach, Tidyverse-friendly interface, and a [thorough selection of parallel computing technologies and scheduling algorithms](https://books.ropensci.org/drake/hpc.html).

## Memoization

Memoization is the strategic caching of the return values of functions. It is a lightweight approach to the core problem that `drake` and other pipeline tools are trying to solve. Every time a memoized function is called with a new set of arguments, the return value is saved for future use. Later, whenever the same function is called with the same arguments, the previous return value is salvaged, and the function call is skipped to save time. The [`memoise`](https://github.com/r-lib/memoise) package is the primary implementation of memoization in R.

Memoization saves time for small projects, but it arguably does not go far enough for large reproducible pipelines. In reality, the return value of a function depends not only on the function body and the arguments, but also on any nested functions and global variables, the dependencies of those dependencies, and so on upstream. `drake` tracks this deeper context, while [memoise](https://github.com/r-lib/memoise) does not.

## Literate programming

[Literate programming](https://rmarkdown.rstudio.com/) is the practice of narrating code in plain vernacular. The goal is to communicate the research process clearly, transparently, and reproducibly. Whereas commented code is still mostly code, literate [knitr](https://yihui.name/knitr/) / [R Markdown](https://rmarkdown.rstudio.com/) reports can become websites, presentation slides, lecture notes, serious scientific manuscripts, and even books.

### knitr and R Markdown

`drake` and [knitr](https://yihui.name/knitr/) are symbiotic. `drake`'s job is to manage large computation and orchestrate the demanding tasks of a complex data analysis pipeline. [knitr](https://yihui.name/knitr/)'s job is to communicate those expensive results after `drake` computes them. [knitr](https://yihui.name/knitr/) / [R Markdown](https://rmarkdown.rstudio.com/) reports are small pieces of an overarching `drake` pipeline. They should focus on communication, and they should do as little computation as possible. 

To insert a [knitr](https://yihui.name/knitr/) report in a `drake` pipeline, use the `knitr_in()` function inside your [`drake` plan](https://books.ropensci.org/drake/plans.html), and use `loadd()` and `readd()` to refer to targets in the report itself. See an [example here](https://github.com/wlandau/drake-examples/tree/main/main).

### Version control

`drake` is not a version control tool. However, it is fully compatible with [`git`](https://git-scm.com/), [`svn`](https://en.wikipedia.org/wiki/Apache_Subversion), and similar software. In fact, it is good practice to use [`git`](https://git-scm.com/) alongside `drake` for reproducible workflows.

However, data poses a challenge. The datasets created by `make()` can get large and numerous, and it is not recommended to put the `.drake/` cache or the `.drake_history/` logs under version control. Instead, it is recommended to use a data storage solution such as [DropBox](https://www.dropbox.com/) or [OSF](https://osf.io/ka7jv/wiki/home/).

### Containerization and R package environments

`drake` does not track R packages or system dependencies for changes. Instead, it defers to tools like [Docker](https://www.docker.com), [Singularity](https://sylabs.io/singularity/), [`renv`](https://github.com/rstudio/renv), and [`packrat`](https://github.com/rstudio/packrat), which create self-contained portable environments to reproducibly isolate and ship data analysis projects. `drake` is fully compatible with these tools.

### workflowr

The [`workflowr`](https://github.com/jdblischak/workflowr) package is a project manager that focuses on literate programming, sharing over the web, file organization, and version control. Its brand of reproducibility is all about transparency, communication, and discoverability. For an example of [`workflowr`](https://github.com/jdblischak/workflowr) and `drake` working together, see [this machine learning project](https://2019-feature-selection.pjs-web.de/report-defoliation.html) by [Patrick Schratz](https://github.com/pat-s) ([source](https://github.com/pat-s/2019-feature-selection)).

## Acknowledgements

Special thanks to [Jarad Niemi](http://www.jarad.me/), my advisor from [graduate school](http://stat.iastate.edu/), for first introducing me to the idea of [Makefiles](https://www.gnu.org/software/make/) for research. He originally set me down the path that led to `drake`.

Many thanks to [Julia Lowndes](https://github.com/jules32), [Ben Marwick](https://github.com/benmarwick), and [Peter Slaughter](https://github.com/gothub) for [reviewing drake for rOpenSci](https://github.com/ropensci/onboarding/issues/156), and to [Maëlle Salmon](https://github.com/maelle) for such active involvement as the editor. Thanks also to the following people for contributing early in development.

- [Alex Axthelm](https://github.com/AlexAxthelm)
- [Chan-Yub Park](https://github.com/mrchypark)
- [Daniel Falster](https://github.com/dfalster)
- [Eric Nantz](https://github.com/thercast)
- [Henrik Bengtsson](https://github.com/HenrikBengtsson)
- [Ian Watson](https://github.com/IanAWatson)
- [Jasper Clarkberg](https://github.com/dapperjapper)
- [Kendon Bell](https://github.com/kendonB)
- [Kirill M&uuml;ller](https://github.com/krlmlr)

Credit for images is [attributed here](docs.ropensci.org/drake/reference/figures/image-credit.md).

[![ropensci_footer](http://ropensci.org/public_images/github_footer.png)](https://ropensci.org)