Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Blog post: Moving workflows from single files to collections - a case study #2050

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Blog post: Moving workflows from single files to collections - a case study #2050

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bd6792f
init
paulzierep Jun 20, 2023
57bd1a2
3 cases
paulzierep Jun 21, 2023
c76e9ff
changed todo
paulzierep Jun 21, 2023
991c526
adding examples to show how we converted from single files to collect…
EngyNasr Jun 21, 2023
c9cf070
Merge branch 'collection-news' into addingtocollectionnews
paulzierep Jun 27, 2023
12aeba0
Merge pull request #1 from EngyNasr/addingtocollectionnews
paulzierep Jun 27, 2023
a42f9b9
* added todo
paulzierep Jun 27, 2023
d17a2f4
added WF links, minor text changes
paulzierep Jun 28, 2023
79e8e03
added cached option
paulzierep Jun 30, 2023
f1836a3
removed last part
paulzierep Jun 30, 2023
daa2a81
failed collection element only problematic downstream for Collapse Co…
paulzierep Jun 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 26 additions & 0 deletions content/news/2023-06-from-single-files-to-collections/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
title: "Moving workflows from single files to collections - a case study"
date: "2023-06-20"
tease: "Allowing a complex workflow to be used on multiple datasets using collections."
hide_tease: false
authors: 'Paul Zierep, Engy Nasr'
authors_structured:
- github: paulzierep
- github: EngyNasr
tags: [EU]
subsites: [all-eu]
main_subsite: eu
---

Collections are a great way to bundle multiple dataset into single entities (as shown in the histroy) that can be
processed collectively. In fact, when the amount of datasets rises up to 1000+ it becomes essential to use collections.
Galaxy can also use collections in tools that are not specifically designed to process
collections using the mapping-over strategy (run the tool for each of the elements in a collection).
Therefore, it should be a peace of cake to port complete workflows that
were based on processing single files to use collections as well.
However, when applying this idea on our latest metagenomics workflow [Foodborn Pathogen detection](https://training.galaxyproject.org/training-material/topics/metagenomics/tutorials/pathogen-detection-from-nanopore-foodborne-data/tutorial.html) we encountered some problems
that arise when switching from single files to collection.
In the following we would like to present some of those issues and how we solved them, in the hopes that these strategies can help
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you also want to write them as FAQs in the GTN it could be useful, just sort of "how to do X" type FAQs, then we can easily link users to them when they encounter those issues later

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the post is finalized, we will condense into a FAQ, thanks for the Idea !

others to port their workflows to collections.