Singularity updates

[bgruening]

This PR will:

• remove the hack way we created Singularity containers by estimating the initial container size, recent versions of Singularity do have smarter ways to do this
• but this also means we do need to push to Quay.io before we can create Singularity containers (is that ok @jmchilton?)
• use latest Singularity version
• remove the --singularity in favor of the singularity or build-singularity subtask
• the Quay.io oauth-token can be stored in an OAUTH_TOKEN env var

[jmchilton]

Amazing, thanks a bunch.

but this also means we do need to push to Quay.io before we can create Singularity containers

Right now - I think Galaxy builds Singularity containers on-demand from tool requirements if they aren't published - https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/deps/container_resolvers/mulled.py#L384. That may be useful for local development and for combinations of tool parameters that somehow or another don't currently have published Biocontainers - do you agree? It looks like this approach wouldn't account for that - things would need to be published to registry in order to build the singularity container - at least this is how I'm reading things on https://singularity.lbl.gov/docs-docker? Can we fallback on the old method for now when coming from the build-on-demand singularity path - with like a TODO to replace the invfile.lua approach when one is just building for Singularity with a Singularity native approach so that Docker doesn't need to be set up on the job runner in order to use Singularity on the workers?

Do you have any other ideas for how to approach this - I don't suppose invfile.lua will ever support Singularity directly (without relying on Docker first)? If we agree on an approach I can try to find some time to patch up the mulled container resolve to implement it.

[bgruening]

Right now - I think Galaxy builds Singularity containers on-demand from tool requirements if they aren't published - https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/deps/container_resolvers/mulled.py#L384. That may be useful for local development and for combinations of tool parameters that somehow or another don't currently have published Biocontainers - do you agree?

Yes. However, this works like --> create Docker image -> Convert Docker to Singularity the old way

It looks like this approach wouldn't account for that - things would need to be published to registry in order to build the singularity container - at least this is how I'm reading things on https://singularity.lbl.gov/docs-docker?

Yes. Let me figure out an other way for this. There should be a way to retrieve this from localhost.

Can we fallback on the old method for now when coming from the build-on-demand singularity path - with like a TODO to replace the invfile.lua approach when one is just building for Singularity with a Singularity native approach so that Docker doesn't need to be set up on the job runner in order to use Singularity on the workers?

Not sure I understand this correctly. The old approach needed Docker locally, because we have build a container before - build-and-test was building a Docker container via involucro and then converting this into Singularity in a very hacky way.
With this PR we don't need Docker anymore with the build-singularity task, iff the container is already on quay.io. I will find a way hopefully to have a local build as well, but the local build then needs to have Docker installed.

Do you have any other ideas for how to approach this - I don't suppose invfile.lua will ever support Singularity directly?

We could do this in theory, but I thought this does not scale in the long run imho, as we would need to add native builds for all containers. Currently we have the philosophy to build Docker and auto-convert everything.

Let me see if I can find a solution that restores the current behavior and keeps the one here. Any idea how to name the targets? build-singularity-online and build-singularity-offline? Or build-singularity-from-docker?

[jmchilton]

We could do this in theory, but I thought this does not scale in the long run imho, as we would need to add native builds for all containers. Currently we have the philosophy to build Docker and auto-convert everything.

I don't know how many high profile container targets will pop up - it may be a lot and I'm not suggesting we target all of them. I think the counter point would be something like Singularity has emerged as the premier container technology for HPC (an important target for Galaxy) and there isn't likely going to be a rival with the same amount of resources behind it for sometime - so we should provide the best Singularity experience we can for now which probably means building native Singularity containers when Docker is unavailable. Does that sound right, I would expect most places insisting on using Singularity aren't going to have Docker available unfortunately.

For this PR though it doesn't matter what our Singularity roadmap looks like though, if you are willing to restore the old behavior for this particular use case I think that is more than enough. build-singularity-online and build-singularity-offline sound great to me! Thanks a bunch.