The Future of DRAM

[rmFlynn]

Hi to all our DRAM users! This summer, we are working on a bunch of massive improvements to DRAM. We have our own ideas about what that will include, but we want to hear more from people in the community. Things like better logging, new databases and faster performance are in the works, and I will be back later to add info on what we are up to, but I wanted to open this up to discussion. What are your must-haves for DRAM2?

[SilasK]

Thank you @rmFlynn, I try to keep each idea in a separate comment.

Integration with Snakmake

I already use Dram with snakemake in atlas. The advantage of snakmake is clearly the scalability. One can easily annotate genomes separately and submit each to a cluster. Snakmeke takes care of all the scheduling.
For this to work, I need to do run dram on each genome separately. concatenate the output manually and then run the distil.

I see different ways to integrate snakmake and DRAM.

1. The easy, provide commands to annotate a single genome or a bunch of genomes. a merge command and then distil.
2. Use snakemake within DRAM, to make the parallelization a child's play. I'm happy to help with this.

[rmFlynn]

I am interested in both options.

1. The tools to do this are actually already in place, but are not sufficiently documented and not robust enough. These tools should probably be made into Snakemake Modules, I think that would be the most flexible and useable way to do this, but you may know better than I on this.
2. This is the complicated part. I think that some parts of DRAM should remain atomic, I am thinking the annotation process specifically, Snakemake has some good features for this, but I have yet to see a wrapper to Snakemake that, for experienced users, is better than using the snakemake command and tools in the first place, and running with all the transparency that provides.
   The Atlas approach is definitely the way to do this because it is transparent and minimalist, but Virsorter2 takes a different approach and I find the result to be less elegant in that case. Most people are still more comfortable using command line arguments, so wrapping workflows should be more elegant, and some day I think it will have purpose built support in snakemake.
   I have been looking at using Dask for some things. I think Dask will be more explicit and so less prone to breaking on edge cases, and then giving errors from a program that the user may not know they were using. I also want to parallelize things that don't write to disk, or write to database files, if Snakemake supports this I missed how, and I want to know, I also want snake make to see features in SQLight and HDF5, but that is a different point.
   IO is a big problem when most people use network attached storage with slower write speeds than they realize, so I don’t want to make more temporary files if I can help it.
   I like Snakemake a lot. Because of the above, I am hesitant to use it in DRAM its self until I am enlightened, but I may come around.

[rmFlynn]

Thank you @rmFlynn, I try to keep each idea in a separate comment.
Thanks! These are good ideas, I will respond to all of them in time.

[SilasK]

Comand line interface

I looked at the CLI of DRAM, and it was quite long. I wonder if using click could allow simplifying the use of different subcommands. I used click for atlas.

[rmFlynn]

This is easy, I like Click and agree it would help. So no notes on this, just know I expect to do this.

[SilasK]

Genecatalog for large scale annotation

It seems to me that dram is increasingly used also for large sets of genome annotation. (>1000). In such cases, it could be useful to predict all the genes, create a non-redundant gene catalog (90 -100 identity) and then annotate these genes only once.

I have some code for this.

[SilasK]

Transcriptome annotation

I recently started using meta-transcriptome data. It would be cool to somehow allow the pathway prediction to incorporate the gene expression. But this is only a vague idea.

[rmFlynn]

@SilasK are you attending ISME?

[SilasK]

Unfortunately not. But I live in the same area. If you want we could try to meet outsidenof the conference.

[rmFlynn]

Things are packed, if you want to meet for lunch tomorrow, I will be available close to the convention center on the Swiss Institute of Technology I can meet you there. Email me at rory.flynn@colostate.edu. If you can't make that admittedly specific time, I apologize but would not worry, we can colab on GitHub no problem.

[SilasK]

Hey @rmFlynn

I now added Dram 1.4 in atlas. very cool indeed.

Are you using the new dbcan-sub for CAZy subfamilies annotatioin? https://bcb.unl.edu/dbCAN_sub/
Could they be used to add infer the substrate isn't it?

I have another question you never used the diamond annotation from dbcan, is there a reason?

[rmFlynn]

That is awesome!

[rmFlynn]

We do use the subfamilies, but we actually extract the parents when it comes time to make the distant, so we're certainly not using them to their fullest extent. As for the more detailed portion of your question I'll have to pass you to one of our metabolism people @jmikayla1991 or @bmcgivern13 do you guys have any opinions?

[SilasK]

In the relese notes you state known issues with performance and merging of tables.
In atlas I run dram on each genome separately, merge the tables and then do the distilling. You can use the code if you want. You said you want to adopt snakemake eventually.

[SilasK]

Where is the developpment of dram2 going on? Would it be possible to follow /contribute to it?