[WIP] Windows Compatibility Fixes

[dwmunster]

To Do:

• Replace manual testing paths, path concatenation, and absolute tests with ioutil.TempDir, filepath.Join, and filepath.IsAbs, respectively.
• Find replacements for/configure Appveyor to handle GNU utilities like sed.
• Handle absolute Windows paths for output files vis a vis temp exec dirs.

Fixes #86

[fire]

Can you give an update to this?

[samuell]

Find replacements for/configure Appveyor to handle GNU utilities like sed.

Hmm, I wonder, is the idea to be able to run SciPipe in a POSIX-environment like Git-bash/MSYS2 or even WSL, or is it to be able to run in a plain windows (MS-DOS) environment?

[dwmunster]

My intention had been to run in a plain windows environment. Sadly, I've been a bit busy and this fell by the wayside.

[samuell]

My intention had been to run in a plain windows environment. Sadly, I've been a bit busy and this fell by the wayside.

No worries at all!

Mostly I started wondering/worrying there might be some blockers to getting that to work, especially since executeCommand() is using bash to execute commands.

[dwmunster]

Easily remedied via build tags if need be.

Is there a particular reason to shell out instead of just exec'ing directly?

[samuell]

@dwmunster wrote:

Is there a particular reason to shell out instead of just exec'ing directly?

Sorry for the delay. Good question! The idea has generally been that the bash shell is such a powerful way to solve a lot of the various task that often are needed to be done when building pipelines.

Actually I think for some commands it might even be hard to do without it, such as those that only take input via standard in and write to standard out, so that you need to interact with the along the lines of:

cat {i:infile} | somecommand > {o:outfile}

... and then there is all the things one can do with piping together long commands, like:

cat somefile.csv | awk -F, '{ print $2 }' | grep someword > {o:wordsfrom2ndcolumn}

etc.

Or, for tasks that don't even write any output, creating a "done-flag" to signal finish status:

some-command && echo done > {o:done}

As far as I can see, a lot of these use cases would not really be possible if just exec'ing directly?

[dwmunster]

Without much thought put into the consequences or work required, some options presented in no particular order:

• We could add an executor interface where the default is a shell executor that calls the appropriate shell for a given OS (via build tags). This also has the benefit of being able to build (batteries included or user-written) executors that handle slurm, etc.
• The executeCommand function could be specialized via build tags to invoke the appropriate shell for the OS.
• Make executeCommand directly exec and leave it to users to invoke the shell.

In any case, I personally think it's fine if some workflows are OS-specific. The ability to compile a particular workflow for multiple OS is not the same as the ability to use the exact same workflow for all OS. Using OS-specific features will limit you just like using programs only available in a particular environment.

[samuell]

Without much thought put into the consequences or work required, some options presented in no particular order:

• We could add an executor interface where the default is a shell executor that calls the appropriate shell for a given OS (via build tags). This also has the benefit of being able to build (batteries included or user-written) executors that handle slurm, etc.
• The executeCommand function could be specialized via build tags to invoke the appropriate shell for the OS.
• Make executeCommand directly exec and leave it to users to invoke the shell.

In any case, I personally think it's fine if some workflows are OS-specific. The ability to compile a particular workflow for multiple OS is not the same as the ability to use the exact same workflow for all OS. Using OS-specific features will limit you just like using programs only available in a particular environment.

Thanks @dwmunster !

I like the idea of having a more pluggable mechanism for executors. Some of the pluggability I have wished to put in seem like they will be much easier once they get generics into Go, although perhaps executors would be possible to solve with just interfaces... will have to look closer into this.