Idea for a flexible component to generate parameters dynamically from shell code

[samuell]

A current shady spot in SciPipe is when needing to generate many sets of parameter values or file names to feed a downstream pipeline. This can be done to some extent using e.g. a globber component, but that is very limited to a very specific use case.

Below is a sketch on an idea on how to implement a type of component that can generate this based on shell scripts.

The idea is that you are supposed to write a shell script that produces a set of JSON-objects, one per line, with the parameter and output filename fields populated.

The API could look like this:

looper := wf.NewLooper("looper", "for f in data/*.csv; do echo \"{ 'outfile': '{o:outfile:$f}' }\"; done;")

otherProc := wf.NewProc("other-proc", "some-command -in {i:infile} ... Etc etc")

outerProc.In("infile").From(looper.Out("outfile"))

// ... etc etc ...

The example above is basically just a globber, but the same method could be used for populating parameters as well. I will update the example shortly to illustrate the combined generation of filenames and parameters.