[help] tar_group_by works only with callr_function = NULL

[stemangiola]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

The target below works only with callr_function = NULL.

the deployment is always = "main".

 # Group samples
      tarchetypes::tar_group_by(
        metadata_grouped_pseudobulk,
        metadata_split_pseudobulk_SMALL |> bind_rows(metadata_split_pseudobulk_BIG),
        file_id,
        deployment = "main"
      ),

I am not sure what the issue might be. For deployment = "main", using callr_function = NULL should not make any different because the environment should be the same (?).

I could spend some time producing a reproducible example if the issue is not obvious.

R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

Matrix products: default
BLAS:   /stornext/System/data/apps/R/R-4.3.0/lib64/R/lib/libRblas.so 
LAPACK: /stornext/System/data/apps/R/R-4.3.0/lib64/R/lib/libRlapack.so;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: UTC
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] glmmSeq_0.5.5                   digest_0.6.35                  
 [3] magrittr_2.0.3                  edgeR_4.0.16                   
 [5] limma_3.58.1                    tidySummarizedExperiment_1.11.4
 [7] tidybulk_1.15.4                 Seurat_5.0.1                   
 [9] SeuratObject_5.0.1              sp_2.1-3                       
[11] Matrix_1.6-5                    ttservice_0.4.0                
[13] tidySingleCellExperiment_1.11.8 SingleCellExperiment_1.24.0    
[15] SummarizedExperiment_1.32.0     Biobase_2.62.0                 
[17] GenomicRanges_1.54.1            GenomeInfoDb_1.38.8            
[19] IRanges_2.36.0                  S4Vectors_0.40.2               
[21] BiocGenerics_0.48.1             MatrixGenerics_1.14.0          
[23] matrixStats_1.2.0               CuratedAtlasQueryR_1.1.1       
[25] crew.cluster_0.3.0              crew_0.9.1                     
[27] qs_0.26.1                       tarchetypes_0.7.9              
[29] glue_1.7.0                      targets_1.4.1.9002             
[31] lubridate_1.9.3                 forcats_1.0.0                  
[33] stringr_1.5.1                   dplyr_1.1.4                    
[35] purrr_1.0.2                     readr_2.1.5                    
[37] tidyr_1.3.1                     tibble_3.2.1                   
[39] ggplot2_3.5.0                   tidyverse_2.0.0                

loaded via a namespace (and not attached):
  [1] fs_1.6.3                spatstat.sparse_3.0-3   bitops_1.0-7           
  [4] httr_1.4.7              RColorBrewer_1.1-3      numDeriv_2016.8-1.1    
  [7] tools_4.3.0             sctransform_0.4.1       backports_1.4.1        
 [10] utf8_1.2.4              R6_2.5.1                HDF5Array_1.30.1       
 [13] mgcv_1.9-1              lazyeval_0.2.2          uwot_0.1.16            
 [16] rhdf5filters_1.14.1     withr_3.0.0             gridExtra_2.3          
 [19] preprocessCore_1.64.0   progressr_0.14.0        cli_3.6.2              
 [22] spatstat.explore_3.2-5  fastDummies_1.7.3       sandwich_3.1-0         
 [25] mvtnorm_1.2-4           spatstat.data_3.0-4     ggridges_0.5.6         
 [28] pbapply_1.7-2           QuickJSR_1.1.3          StanHeaders_2.32.6     
 [31] parallelly_1.37.1       rstudioapi_0.16.0       generics_0.1.3         
 [34] RApiSerialize_0.1.2     ica_1.0-3               spatstat.random_3.2-2  
 [37] car_3.1-2               inline_0.3.19           loo_2.7.0              
 [40] fansi_1.0.6             abind_1.4-5             lifecycle_1.0.4        
 [43] multcomp_1.4-25         yaml_2.3.8              carData_3.0-5          
 [46] qvalue_2.34.0           rhdf5_2.46.1            SparseArray_1.2.4      
 [49] Rtsne_0.17              grid_4.3.0              promises_1.2.1         
 [52] crayon_1.5.2            miniUI_0.1.1.1          lattice_0.22-6         
 [55] cowplot_1.1.3           pillar_1.9.0            knitr_1.45             
 [58] boot_1.3-30             estimability_1.5        future.apply_1.11.2    
 [61] codetools_0.2-19        leiden_0.4.3.1          V8_4.4.2               
 [64] data.table_1.15.2       vctrs_0.6.5             png_0.1-8              
 [67] spam_2.10-0             gtable_0.3.4            assertthat_0.2.1       
 [70] xfun_0.43               S4Arrays_1.2.1          mime_0.12              
 [73] coda_0.19-4.1           survival_3.5-8          pbmcapply_1.5.1        
 [76] statmod_1.5.0           TH.data_1.1-2           ellipsis_0.3.2         
 [79] fitdistrplus_1.1-11     ROCR_1.0-11             nlme_3.1-164           
 [82] bit64_4.0.5             RcppAnnoy_0.0.22        rstan_2.32.6           
 [85] TMB_1.9.10              irlba_2.3.5.1           KernSmooth_2.23-22     
 [88] colorspace_2.1-0        DBI_1.2.2               duckdb_0.10.0          
 [91] tidyselect_1.2.1        processx_3.8.4          emmeans_1.10.0         
 [94] bit_4.0.5               compiler_4.3.0          curl_5.2.1             
 [97] xml2_1.3.3              DelayedArray_0.28.0     plotly_4.10.3          
[100] stringfish_0.16.0       scales_1.3.0            lmtest_0.9-40          
[103] callr_3.7.6             getip_0.1-4             goftest_1.2-3          
[106] minqa_1.2.6             mirai_0.12.1            spatstat.utils_3.0-4   
[109] XVector_0.42.0          htmltools_0.5.8         pkgconfig_2.0.3        
[112] lme4_1.1-35.2           dbplyr_2.5.0            fastmap_1.1.1          
[115] rlang_1.1.3             htmlwidgets_1.6.3       shiny_1.8.0            
[118] zoo_1.8-12              jsonlite_1.8.8          RCurl_1.98-1.12        
[121] GenomeInfoDbData_1.2.11 dotCall64_1.1-1         patchwork_1.2.0        
[124] Rhdf5lib_1.24.2         munsell_0.5.0           Rcpp_1.0.12            
[127] reticulate_1.35.0       stringi_1.8.3           zlibbioc_1.48.2        
[130] MASS_7.3-60.0.1         plyr_1.8.9              pkgbuild_1.4.4         
[133] parallel_4.3.0          listenv_0.9.1           ggrepel_0.9.5          
[136] deldir_2.0-4            splines_4.3.0           tensor_1.5             
[139] hms_1.1.3               locfit_1.5-9.9          ps_1.7.6               
[142] igraph_1.5.0.1          ggpubr_0.6.0            spatstat.geom_3.2-9    
[145] ggsignif_0.6.4          base64url_1.4           RcppHNSW_0.6.0         
[148] reshape2_1.4.4          RcppParallel_5.1.7      nloptr_2.0.3           
[151] tzdb_0.4.0              nanonext_0.13.2         httpuv_1.6.15          
[154] RANN_2.6.1              polyclip_1.10-6         future_1.33.2          
[157] scattermore_1.2         broom_1.0.5             xtable_1.8-4           
[160] RSpectra_0.16-1         rstatix_0.7.2           later_1.3.2            
[163] viridisLite_0.4.2       lmerTest_3.1-3          glmmTMB_1.1.9          
[166] cluster_2.1.6           timechange_0.3.0        globals_0.16.3   ```

[wlandau]

Hard to say, except your version of targets is 1.4.1.9002. Might have been fixed as of the latest release.

[wlandau]

Same goes with tarchetypes. The current release is 0.9.0 and your session info lists 0.7.9.

[stemangiola]

updating and retesting

[stemangiola]

Updated but exactly the same behaviour. With

• 50Gb callr_function = NULL Works
• 50Gb local Fails

I think must be a bug, it cannot be resource-related