Function always outdated

[jennysjaarda]

Sorry in advance for a question without a reproducible example. I thought I would check if you have a quick suggestion as to my mistake. I have one target that is always outdated everytime I start a fresh R session. The target looks like this: ukbb_drug_counts = count_ukbb_drug_users(ukbb_munge_drug_users, drug_codes_interest). The two arguments use upstream targets as inputs: ukbb_munge_drug_users and drug_codes_interest.

The function looks as follows:

count_ukbb_drug_users <- function(ukbb_munge_drug_users, drug_codes_interest){
  data <- ukbb_munge_drug_users %>% dplyr::select(paste0(drug_codes_interest$molecule, "_users")) %>% map_df(~length(which(.==1))) %>% t()
  return(data)
}

Its purpose is to simply count the number of "users" of each drug from a dataset of drug users (0/1 = no/yes). I've included a snapshot of the data below. Is there something obviously wrong with the function such that it would always be outdated? I've tried using deps_code on it, but I don't notice anything. It only takes <1 second to run and it has no downstream targets so for now it's not a huge deal, but I would like to know what I'm doing wrong. Thanks in advance!

loadd(ukbb_munge_drug_users)
loadd(drug_codes_interest)

head(ukbb_munge_drug_users)
      eid amisulpride_users aripiprazole_users clozapine_users
1 4845585                 0                  0               0
2 2326469                 0                  0               0
3 3665794                 0                  0               0
4 3647543                 0                  0               0
5 1303873                 0                  0               0
6 4818681                 0                  0               0
  mirtazapine_users olanzapine_users quetiapine_users risperidone_users
1                 0                0                0                 0
2                 0                0                0                 0
3                 0                0                0                 0
4                 0                0                0                 0
5                 0                0                0                 0
6                 0                0                0                 0
  valproate_users
1               0
2               0
3               0
4               0
5               0
6               0

drug_codes_interest$molecule

[1] "amisulpride"  "aripiprazole" "clozapine"    "mirtazapine"  "olanzapine"  
[6] "quetiapine"   "risperidone"  "valproate"  

count_ukbb_drug_users(ukbb_munge_drug_users, drug_codes_interest)
                   [,1]
amisulpride_users   181
aripiprazole_users   78
clozapine_users      57
mirtazapine_users  1432
olanzapine_users    615
quetiapine_users    359
risperidone_users   310
valproate_users    1495

[wlandau]

I am not sure what the issue could be. I assume the error goes away when you try to create a reprex? Could it be that you have an outdated copy of count_ukbb_drug_users() somewhere else in your code which overwrites the count_ukbb_drug_users() you expect to have?

[jennysjaarda]

I think I may have found the problem, I assume the r_make may help solve the issue, and ultimately moving to targets would be the ideal long term solution. Here I have a reproducible example:

library(drake)
library(dplyr)
  
eths <- c("CEU", "EA", "MIXED", "NA", "YRI")

test_plan <- drake_plan(
  eth_test =  tibble( eths = !!eths),
  eth_sub = eth_test %>% filter(eth!="META")
)

make(test_plan)
outdated(test_plan)
## none 

eth="X"

outdated(test_plan)
## now eth_sub is outdated

I believe the problem was that I was developing and troubleshooting code so had defined some variables within my R session (such as eth) and the filter function combined with defining this variable invalidated this target.

[jennysjaarda]

So I am trying to set up my _drake.R file to use r_make() instead. Just a question:

I have more than one plan in my project (perhaps this is bad practice?), do I need to include the drake_config(plan, verbose = 2) line for each plan? i.e. the file would look like:

setwd(here::here())

source("R/packages.R")
source("R/functions.R")
source("R/settings.R") # I assume I can also have a file with some input parameters? 
source("R/plan.R")

drake_config(plan1, verbose = 2)
drake_config(plan2, verbose = 2) ## etc 

In R, I could then interactively run:

r_make()

Is this the proper use? It's not clear to me how I would only run a single plan. It seems like if I only need to run one plan, it's time consuming to run drake_config for each plan in the project. Perhaps the intended use is to bind all plans together and then select with the targets argument?

Thanks in advance.

[wlandau]

In _drake.R, only the last call to drake_config() will take effect. So I recommend one drake_config() with plan = bind_plans(plan1, plan2). As you say, the targets argument can help select parts of the plan to run.

[jennysjaarda]

Ok thanks for the advice. And I would use the targets argument within the rmake() (not drake_config)? Meanwhile I am going to to start moving to your targets package! Looking forward to implementing the new and improved version. I assume when doing so, everything will initially be outdated (i.e. targets needs to start fresh, it can't read from an old drake project and know not to run certain targets?).

[wlandau]

And I would use the targets argument within the r_make() (not drake_config)?

drake_config() has a targets argument, but r_make() does not. So either way you would toggle the drake_config() call inside _drake.R. Alternatively, r_make() has a source argument to allow you to specify a script path other than _drake.R, but I would not generally recommend this.

I assume when doing so, everything will initially be outdated (i.e. targets needs to start fresh, it can't read from an old drake project and know not to run certain targets?).

Yes, targets has a different storage system, and so converting from drake will unfortunately require you to run all targets from scratch.

[jennysjaarda]

Sorry just one another question, I'm just trying to understand the best practices for the workflow.

I see that drake_config accepts the same arguments as make.

Previously I had a make.R file in my home directory that had a bunch of make calls to my various plans, for example. The different plans/calls are adapted based on the computing requirements:

make(init_analysis, log_make = "init_analysis.log", cache_log_file = "cache_log.csv",
  parallelism = "clustermq", jobs = 20, template = list(cpus = 8, partition = "sgg",
  log_file = "/data/sgg2/jenny/projects/PSYMETAB/init_analysis_%a_clustermq.out"))

make(process_init, log_make = "process_init.log", cache_log_file = "cache_log.csv",
  parallelism = "clustermq", jobs = 120, template = list(cpus = 1, partition = "sgg",
  log_file = "/data/sgg2/jenny/projects/PSYMETAB/process_init_%a_clustermq.out"))

make(ukbb, log_make = "ukbb.log", cache_log_file = "cache_log.csv",
  parallelism = "clustermq", jobs = 8, template = list(cpus = 12, partition = "cluster",
  log_file = "/data/sgg2/jenny/projects/PSYMETAB/ukbb_%a_clustermq.out"),
  targets = c("psy_UKBB_merge"))

make(ukbb, log_make = "ukbb.log", cache_log_file = "cache_log.csv",
  parallelism = "clustermq", jobs = 24, template = list(cpus = 1, partition = "cluster",
  log_file = "/data/sgg2/jenny/projects/PSYMETAB/ukbb_%a_clustermq.out"),
  targets = ukbb %>% filter(target!="psy_UKBB_merge") %>% pull(target)) #all targets except psy_UKBB_merge because of different computing requirments

Using your suggestion, I have bound plans together and then just selected targets with the targets argument that have specific computing requirements. But I am not sure how to adapt this into my _drake.R file since you say only the last call to drake_config will take effect.

Should the last 2 lines of the _drake.R file be:

plan <- bind_plans(plan1, plans2,...)
drake_config(plan, verbose = 2)

And then in the interactive session (or a make.R script) simply run:
r_make(plan, ... ) #with all the usual parameters from above, i.e.

r_make(plan, log_make = "plan.log", cache_log_file = "cache_log.csv", parallelism = "clustermq", jobs = 24, template = list(cpus = 1, partition = "cluster", log_file = "/data/sgg2/jenny/projects/PSYMETAB/ukbb_%a_clustermq.out"), targets = ukbb %>% filter(target!="psy_UKBB_merge") %>% pull(target))

Plus any other r_make calls with different computing requirements...

[wlandau]

I recommend that your _drake.R file look something like this:

plan <- bind_plans(plan1, plans2,...)
drake_config(
  plan, 
  verbose = 2,
  log_make = "plan.log",
  cache_log_file = "cache_log.csv",
  parallelism = "clustermq",
  jobs = 24,
  template = list(cpus = 1, partition = "cluster", log_file = "/data/sgg2/jenny/projects/PSYMETAB/ukbb_%a_clustermq.out"),
  targets = ukbb %>% filter(target!="psy_UKBB_merge") %>% pull(target)
)

Then, interactively or through make.R, you can call r_make() with no arguments (unless you want to supply an alternative callr function to r_fn).

[jennysjaarda]

Thank-you! I'm still a bit confused. I'm a bit unclear the best way to select different targets unless I am constantly changing the _drake.R file?

I tried using the r_args argument but that didn't work and I am not sure how the r_fn argument helps me? Sorry if I'm missing something obvious. Is there any way to do something like below? This way I can have a make.R script that keeps track of my computing requirements without constantly changing the targets and template arguments in the drake_config function of the _drake.R file.

r_make(..., r_args = list(log_make = "plan.log", cache_log_file = "cache_log.csv",
  parallelism = "clustermq", jobs = 120, template = list(cpus = 1, partition = "cluster2",
  log_file = "/data/sgg2/jenny/projects/PSYMETAB/plan_%a_clustermq.out"))) 

[wlandau]

The idea is to go through _drake.R to make changes to most arguments. make.R/r_make() will stay the same, which avoids duplication when you update the configuration.

r_fn and r_args control how the external callr process is deployed, and I agree they are probably not relevant here. By default, r_make() launches an external callr::r() process that sources _drake.R and runs the pipeline. r_fn allows you to supply something other than callr::r, such as callr::r_bg, and r_args is a list of named arguments to the callr function.

[jennysjaarda]

Ok got it. Thanks for the explanation. So there isn't really a way around updating the _drake.R file for the different targets/computing requirement each time then? Because previously with my make.R script, I would simply run the make functions with different arguments one after another. Hopefully what I'm trying to do makes sense.

…

On Apr 26, 2021, at 18:13, Will Landau ***@***.***> wrote:  The idea is to go through _drake.R to make changes to most arguments. make.R/r_make() will stay the same, which avoids duplication when you update the configuration. r_fn and r_args control how the external callr process is deployed, and I agree they are probably not relevant here. By default, r_make() launches an external callr::r() process that sources _drake.R and runs the pipeline. r_fn allows you to supply something other than callr::r, such as callr::r_bg, and r_args is a list of named arguments to the callr function. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[wlandau]

Going through _drake.R is part of the design, and targets is similar. In most cases, this helps avoid having to remember and retype all the arguments in an interactive call, and it helps manage those arguments reproducibility. But it's an adjustment for some, I know. targets has a convenience function tar_edit() that opens _targets.R if you are using the RStudio IDE.

[jennysjaarda]

Got it. It all makes sense. The only reason I have these different make calls in a separate script is because some of my targets require more cores than others (we've discussed this previously), and this was the best workaround I (we) could come up with. But now using the _drake.R script I'm not sure how best to adapt. The future backend (very, very slow) with transient workers wasn't a good option and clustermq is much better.

…

On Mon, Apr 26, 2021 at 8:02 PM Will Landau ***@***.***> wrote: Going through _drake.R is part of the design, and targets is similar. In most cases, this helps avoid having to remember and retype all the arguments in an interactive call, and it helps manage those arguments reproducibility. But it's an adjustment for some, I know. targets has a convenience function tar_edit() that opens _targets.R if you are using the RStudio IDE. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1366 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AITJSVUVNI24J5J4FTSBWQDTKWTCBANCNFSM43SOW4GA> .

[wlandau]

Ah, I see where you are coming from now. targets should reenable that particular workaround again because parallelism happens through tar_make_clustermq(), which has a workers argument. But in any case, I recommend keeping an eye on HenrikBengtsson/future.batchtools#65.