Harmonizing and modeling BrAPI TERRA REF data

This repository contains code, data, and intermediate files for harmonizing data from the TERRA-REF BrAPI endpoint into the ARDN format and running it through the DSSAT model. The data transformation pipeline is outlined below.

Learn more about this project at the ARDN website here: https://agmip.github.io/ARDN/

Pipeline

Step	Input data	Output data	Output data location	Processing tool
1. Reformatting raw data	TERRA REF JSONs from BrAPI	TERRA REF BrAPI csvs	csv_data folder	R script
2. Mapping ICASA variables	TERRA REF BrAPI csvs	SC2	dssat_files folder	VMapper
3. Getting ICASA-compliant data	TERRA REF BrAPI csvs + SC2	AgMIP zip	dssat_files folder	VMapper (AgMIP input package button)
4. Get model-specific files		DOME + linkage files	dssat_files/DOME folder	Hand-written based on DOME template & linkage info
5. Generating model input files	AgMIP zip + DOME + linkage	DSSAT files + ACEB .aceb	dssat_files/dssat_input and dssat_files	Data Factory
6. Running model	DSSAT input files + cultivar, weather, soil files	DSSAT output files	dssat_files/dssat_input folder	Compiled DSSAT

Diagram mapping pipeline steps

Repository file structure

• csv_data/
  • scripts/download_and_convert.R: script downloads raw BrAPI TERRA REF JSON data from 4 endpoints (observationunits, studies, germplasm, and events) and converts to required CSV format
  • data/: contains converted csv data files
    • studies_table.csv
    • obs_table.csv
    • germplasms_table.csv
    • events_table.csv.gz
  • mvp/: (obsolete) contains older versions of converted BrAPI TERRA REF JSON files
• dssat_files/
  • sc2.json: sidecar file that maps csv files with AgMIP ICASA variables
  • AgMIP_Input.zip: resulting csv file from apply sidecar to data csv files
  • DOME/
    • Field_Overlay_s4.csv: DSSAT-specific DOME file that contributes additional required inputs to run model
    • LINKAGE_s4_v2.csv: DDSAT-specific linkage file that enables individual DOME files to be used for multiple experiments or seasons
  • AgMIP_Input.aceb: ACEB-format file
  • dssat_input.zip
  • dssat_input/: generated from above zip file; contains DSSAT-specific input files and model run output files; many files, only including ones that are used
    • DSSBatch.v48
    • TERRAR01.SGX
    • Summary.OUT
• azmet/: contains daily weather data (not used in DSSAT run yet)
• irrigation/: contains irrigation rates for Season 4 during the drought experiment
  • season_4_irrigation.csv
    • day: date in YYYY-MM-DD format
    • early_drought: mm of water applied to plots in early drought treatment
    • late_drought: mm of water applied to plots in late drought treatment
    • note that these irrigation rates only differ for the Aug 1-14 (early drought) and Aug 15-31 (late drought) treatments
  • season_4_irrigation.csv: raw data including gallons water per day and total area irrigated; to show calculations. Date and last two columns are identical to the three columns in the season_4_irrigation_raw.csv file
• raw_brapi_json/: (obsolete) contains R script for converting raw TERRA REF JSON files into format needed by collaborator, and resulting .json files
• json_data/scripts: (obsolete) contains R scripts for converting raw TERRA REF JSON files into format needed by collaborator

Overview of relationships among tables provided as CSV files

Four tables were generated from the download_and_convert.R script that turned BrAPI JSON into tables provided as CSV. These tables and their relational keys are shown in the image below.

Generating DSSAT input files

The files in the previous section are used with the sidecar 2 file to generate input files for crop models with the AgMIP 'Data Factory' web interface, found at this website. This program is used to get inputs to multiple different models, including DSSAT (used here), APSIM, and STICS.

DSSAT input and output files

All DSSAT input files, which are generated by the Data Factory, are available in dssat_input.zip in dssat_files folder. These files are also in dssat_input subfolder, along with DSSAT output files. Lists of these files are below.

DSSAT input files

• ACMO_meta.dat
• DSSBatch.v47
• DSSBatch.v48
• Run47.bat
• Run48.bat
• TERRAR01.SGA
• TERRAR01.SGT
• TERRAR01.SGX

DSSAT output files

• DSSAT48.INH
• DSSAT48.INP
• Evaluate.OUT
• OVERVIEW.OUT
• PlantGro.OUT
• SoilCBal.OUT
• SoilCBalSum.OUT
• SoilOrg.OUT
• Summary.OUT
• WARNING.OUT
• Weather.OUT

Some of the DSSAT output files in this repo are specific to one author's model run and are provided as an example. If you want to run DSSAT with TERRA REF data, you should regenerate all DSSAT input and output files yourself, starting with deleting dssat_input.zip and dssat_input from your local copy of this repo.

How to run DSSAT

• Compile model locally (optional for Windows, required for Linux systems)
  • These instructions assume model version is 4.8, so file names contain 48
  • Clone source code from dssat-csm-os repo
  • Run make commands
  • dscsm048 is executable model file
• Move additional data files into correct location
  • Need DSSAT-specific files for cultivar (Sorghum), Arizona weather, and Arizona soils
  • Copy Data/ subfolder from dssat-csm-os repo, which has Genotype/, Soil/ and Weather/ subfolders
  • Paste into path specified in DSSATPRO.L48; alternatively move relevant data files into main compiled model folder or change file paths in DSSATPRO.L48
  • Data file suffix number should match model batch file suffix number, e.g., 48
• Run model
  • From DSSAT input files, generated in step 5 in pipeline, run /path/to/dscsm048 b DSSBatch.v48
  • If successful, there will be no errors and a summary of all simulations can be viewed in Summary.OUT

References

LeBauer, David et al. (2020), Data From: TERRA-REF, An open reference data set from high resolution genomics, phenomics, and imaging sensors, Dryad, Dataset, https://doi.org/10.5061/dryad.4b8gtht99

Hoogenboom, G., C.H. Porter, K.J. Boote, V. Shelia, P.W. Wilkens, U. Singh, J.W. White, S. Asseng, J.I. Lizaso, L.P. Moreno, W. Pavan, R. Ogoshi, L.A. Hunt, G.Y. Tsuji, and J.W. Jones. 2019. The DSSAT crop modeling ecosystem. In: p.173-216 [K.J. Boote, editor] Advances in Crop Modeling for a Sustainable Agriculture. Burleigh Dodds Science Publishing, Cambridge, United Kingdom, http://dx.doi.org/10.19103/AS.2019.0061.10