Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vazovn new leads #2493

Open
wants to merge 47 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
c5b46d1
First commit
vazovn Jan 26, 2024
ba8387f
Update index.md
vazovn Feb 1, 2024
9c2a9d1
Update index.md
vazovn Feb 1, 2024
786f593
Update index.md
vazovn Feb 2, 2024
0cbc598
Update index.md
vazovn Feb 2, 2024
3815bc8
Update index.md
vazovn Feb 2, 2024
6636b95
Update index.md
vazovn Feb 5, 2024
bde692f
Update index.md
vazovn Feb 5, 2024
cdf6936
Update index.md
vazovn Feb 5, 2024
63076bf
Update index.md
vazovn Feb 5, 2024
d6f9828
Update index.md
vazovn Feb 5, 2024
65fa1ae
Update index.md
vazovn Feb 5, 2024
9487922
Update index.md
vazovn Feb 5, 2024
57d974c
Update index.md
vazovn Feb 6, 2024
442b10b
Create materials_science
vazovn Feb 13, 2024
0f900ae
Create climate
vazovn Feb 13, 2024
a73796f
Create astrophysics
vazovn Feb 13, 2024
1af9e73
Create biodiversity
vazovn Feb 13, 2024
54e4bdb
Delete content/get-started/new-leads/successful_stories directory
vazovn Feb 13, 2024
57ada99
Create astrophysics
vazovn Feb 13, 2024
d6f1f1b
Create materials-science
vazovn Feb 13, 2024
39e86cd
Create climate
vazovn Feb 13, 2024
a12782c
Create biodiversity
vazovn Feb 13, 2024
610329c
Update index.md
vazovn Feb 13, 2024
04397b6
Update astrophysics
vazovn Feb 13, 2024
cbb437e
Create astrophysics.md
vazovn Feb 14, 2024
a0f044a
Rename biodiversity to biodiversity.md
vazovn Feb 14, 2024
785309d
Delete content/get-started/new-leads/successful-stories/astrophysics.md
vazovn Feb 14, 2024
aeeadc4
Rename astrophysics to astrophysics.md
vazovn Feb 14, 2024
275c2e0
Rename climate to climate.md
vazovn Feb 14, 2024
bf80ee0
Rename materials-science to materials-science.md
vazovn Feb 14, 2024
0426380
Update index.md
vazovn Feb 14, 2024
f963e0e
Update index.md
vazovn Feb 14, 2024
e95b7c8
Update astrophysics.md
vazovn Feb 14, 2024
38d342d
Update astrophysics.md
vazovn Feb 14, 2024
8669909
Update materials-science.md
vazovn Feb 14, 2024
a2c5fc5
Create success-stories-images
vazovn Feb 14, 2024
578d110
Delete content/images/success-stories-images
vazovn Feb 14, 2024
9737ab3
Create img1.jpg
vazovn Feb 14, 2024
53f9956
Add files via upload
vazovn Feb 14, 2024
16324c4
Delete content/images/successful-stories/img1.jpg
vazovn Feb 14, 2024
c7f8841
Update materials-science.md
vazovn Feb 14, 2024
1faa3b2
Update materials-science.md
vazovn Feb 17, 2024
589f13e
Update climate.md
vazovn Feb 17, 2024
d601580
Add files via upload
vazovn Feb 17, 2024
e927ac4
Update biodiversity.md
vazovn Feb 17, 2024
3a0d88d
Optimised images with calibre/image-actions
github-actions[bot] Apr 2, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
74 changes: 59 additions & 15 deletions content/get-started/new-leads/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,40 +4,84 @@ title: Set up your own community

<div class='right'><img src="/images/undraw-illustrations/mind_map.svg" alt="" width="240" /></div>

There are many communities established within Galaxy. We call them [Special Interest Groups](/community/sig/). We have SIGs based for example on
[**region**](/community/sig/#regional-communities) or [**domain of science**](/community/sig/#communities-of-practice) but
if you feel like you just don't see your own or don't feel identified with them, you can create your own community!
There exist many scientific groups within the Galaxy project. The members of these groups are organized into _communities of practice_ - usually referred to as [Special Interest Groups](/community/sig/) (SIG).

Below you'll find a suggested path to follow before you can start your own community, a list of Do's and Don'ts and also useful links you should probably check out.
We encourage you to look through the existing [Special Interest Groups](/community/sig/) and find the one which is appropriate for your needs. You can always join an existing SIG and contribute to that community. However, if you don't find a SIG that fits your work, domain or region, or you don't feel identified with any of them, you can create your own community of practice SIG!

### Path
You are still wondering if and why you should start a new SIG? Find some arguments [here](/community/governance/gcb/#why-make-a-sig)!

### What is _good to know_ before you start building a SIG

The goal of building a new community is to establish a working environment related to your domain of expertise. In the process of creating and maintaining a SIG, regardless of your research domain, you will need to address the following issues:

1. Identify the technical challenges you want to handle. They will be the main focus for your SIG and will determine the major part of its activities. Here is a very _inexhaustive_ list of potential requirements you may want to meet:

* create relevant tools and software
* define and manage special data formats
* manage big data (specific to your domain?)
* share and reproduce the processes and the results of your work
* obtain the optimal computational resources for achieving your goals

2. Administer your group

* Set up means of communication (mailing lists, chat channels)
* Set up a group organization and nominate members responsible for SIG's routines (contact person, tool support person, etc.)

3. Publish your results and promote your experience in order to:

* extend and share the acquired expertise
* attract new people into your SIG by identifying other groups who might benefit from your experience

4. Organise a training process for all interested in your SIG activites

* set up a [training network](https://training.galaxyproject.org/) for your domain
* maintain a training documentation (slides, hands-on, automation routines)

### What is _good to do_ before you start building a SIG

#### Learn the basics about Galaxy

First of all, you should start with our dedicated [**Get Started**](/get-started/) page to get a head-start of what Galaxy is about.
First of all, you should start with our dedicated [**Get Started**](https://galaxyproject.org/get-started/) page to get a head-start of what Galaxy is about.
You'll find all the necessary resources and you could even participate in the [**Galaxy Mentorship Network**](https://galaxy-mentor-network.netlify.app/)
to get started with the help of a Galaxy Mentor.

#### Get acquainted with the Galaxy Training Network

The Galaxy Training Network ([GTN](https://training.galaxyproject.org)) is collection of tutorials developed and maintained by the worldwide Galaxy community.
It will show you how various SIGs, e.g. **Climate**, **Metabolomics**, carry out their work, prepare training materials and organize their events

#### Learn about Tool Development
Galaxy has many tools already installed and ready to be used but when creating your community, you'll probably need specific tools for your Galaxy analysis. Don't worry, see our dedicated page for [**Tool Authors**](/tools/) where you'll find everything you'll need and also reach out to the [**Tools Working Group**](/community/wg/) via Gitter.
Galaxy has many tools already installed and ready to be used but when creating your community, you'll probably need specific tools for your Galaxy analysis. Don't worry, see our dedicated page for [**Tool Authors**](/tools/) where you'll find information about tool development.

Don't forget to follow the [**Best Practices**](https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices.html) for writing [**Galaxy Tools**](https://toolshed.g2.bx.psu.edu/).

#### Get involved
- Introduce yourself and dive in our [Working Groups](/community/wg/)
- Introduce yourself and dive in our [Special Interest Groups](/community/sig/)
- Participate in our [worldwide Galaxy Events](/events/)
- If you want to start your own [Special Interest Group](/community/sig/) we encourage you to read the [Galaxy Community Board](/community/governance/gcb/#creating-a-new-sig) page.

### Do's and Don'ts

<div class='right'><img src="/images/undraw-illustrations/questions.svg" alt="" width="180" /></div>
### Please read the success stories below! They are full of inspiring examples and solutions!

| __Special Interest Group__ | __Here is how they did it__ |
|---|---|
| Biodiversity | [the story](https://galaxyproject.org/get-started/new-leads/succesful-stories/biodiversity.md) |
| Climate | [the story](https://galaxyproject.org/get-started/new-leads/succesful-stories/climate.md) |
| Material Science | [the story](https://galaxyproject.org/get-started/new-leads//succesful-stories/materials-science.md)|
| Astrophysics | [the story](https://galaxyproject.org/get-started/new-leads/succesful-stories/astrophysics.md) |



This curated table was made with the help of Community member's experience in hope to avoid obstacles in your path to create your Community.
### Some useful recomendations

| __DO__ | __DON'T__ |
<div class='right'><img src="/images/undraw-illustrations/questions.svg" alt="" width="180" /></div>

| __DO__ | __DON'T__ |
|---|---|
| Be patient! Take the time to learn the basics about Galaxy. | Hurry and dive in straight to create without knowing a lot about Galaxy and its workarounds. |
| Ask if a tool for what you need already exists before starting to develop one. | Get hazy with all your tool's possible needs as there are many installed already that can be close to what you would like.|
| Join the Galaxy [Matrix channel](https://matrix.org/docs/chat_basics/matrix-for-im/) and consult the [Galaxy help pages](https://help.galaxyproject.org/) | Spend hours or days to read and debug the galaxy code |
| Ask if a tool you need already exists before starting to develop one. | Get hazy with all your tool's possible needs as there are many installed already that can be close to what you would like.|
| Request your tool to be reviewed, avoid managing your repo alone! | Be a lone wolf. The main servers may not want to install your tool as they want to make sure it follows the [**standards**](https://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices.html) they have defined for developing tools. |
| When you happen to need a tool: Make the tool -> write the training -> set up an event. Repeat events and take advantage of Galaxy’s infrastructure (Smorgasbord, CoFests, GCC, etc.) to improve your tool ,maximize its use and receive feedback from the community. | Set to create very specific tools and avoid feedback. |
| When you happen to need a tool: Make the tool -> write the training -> set up an event. Repeat events and take advantage of Galaxy’s infrastructure [Smorgasbord](https://gallantries.github.io/video-library/events/smorgasbord3/) , [CoFests](https://galaxyproject.org/events/cofests/), [GCC](https://galaxyproject.org/gcc/), etc. to improve your tool, maximize its use and receive feedback from the community. | Set to create very specific tools and avoid feedback. |
| Use already established servers. Asking for a subdomain within the [**usegalaxy.eu**](https://usegalaxy.eu/) server is a great option! |Set up a server all by yourself (consider resources, complexity, security issues and more).|


70 changes: 70 additions & 0 deletions content/get-started/new-leads/successful-stories/astrophysics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@

## Our community onboarding

### How did we get to know about Galaxy project and framework and its potential

Diverse web-based data analysis platforms are reasonably well advanced and accepted in astrophysics. As we continued to develop our own research and infrastructure projects in this area, we got advice from some of our colleagues involved in EOSC about the new EuroScienceGateway project, which in particular advances the Galaxy platform in broad range communities. We joined EuroScienceGateway to learn more about Galaxy and make it useful in the astrophysical community.

Prior to EuroScienceGateway we did not know about the Galaxy, and never heard about it from anyone in the broad astrophysical community. Later we learned that Galaxy was considered as a possible “science platform” for SKA, but was not so far selected (but the potential for adoption remains).


### What were our needs / challenges :


Each telescope/infrastructure relies on a data reduction pipeline, typically developed and maintained by telescope collaboration and/or a telescope’s Science Data Center. **The telescope tools typically share few reusable components**, except for some common libraries for manipulating common data formats (such as [astropy](https://github.com/astropy/astropy)). A notable exception is the [HEASOFT](https://heasarc.gsfc.nasa.gov/lheasoft/) package, which includes software for data reduction of most NASA space telescopes. The situation became more complex as **telescopes became more diverse** in the last decades, with proliferation of Gravitational Wave and Neutrino observatories, with very different data reduction techniques and practices. Telescope data reduction is often a **resource-consuming process** and requires dedicated infrastructures.

Sky objects are more or less the same for all observers, and **telescopes often combine their observations**. Astronomers do not control their subject of study, and observations of transient phenomena are often **opportunistic**, and even **small telescopes can make big discoveries**, meaning that **interoperability is relevant** for big and small infrastructures. The practice of combining observations from very different telescopes to derive a complete view of astrophysical sources is encompassed within “**multi-wavelength**” and “**multi-messenger**” astrophysics discipline. Inter-telescope interoperability is primarily concerned with applying joint analysis techniques (cross-correlation, broad-band modelling, etc) to high-level scientific products: images, emission spectra, light curves.

The understanding that astrophysical questions can be best answered by combining diverse data led to adoption of FAIR practices in astrophysics. To enable them, platforms and portals were developed by big ([ESA](https://datalabs.esa.int/), [NASA](https://heasarc.gsfc.nasa.gov/)) and smaller (just in Switzerland [MMODA](https://www.astro.unige.ch/mmoda/), [DACE](https://dace.unige.ch/), [Renku](https://renkulab.io/)) actors. Several EOSC projects made important strides in this direction: ASTERICS-OBELICS, ESCAPE.

High level data are usually (although not always) smaller and can be more easily shared. On the other hand, new astrophysical questions occasionally require re-analysis of lower-level data, meaning that low-level analysis also needs to fit in the FAIR paradigm.

Since astrophysical data are inherently non-repeatable and unique, both low-level and high-level data need to be preserved in a reusable long-term way. Astrophysical Archives are taking care of this preservation in accordance with suitable standards, in particular those developed by International Virtual Observatory Alliance ([IVOA](https://www.ivoa.net/)).

### What were the steps we made:

Since we were dealing with this specialised data format, it was suggested to us to try making Galaxy recognize this format, and visualise it, and [we did this](https://github.com/usegalaxy-eu/galaxy/issues/194), making Galaxy much more familiar to astronomers.

As an initial tool development attempt, we [took](https://github.com/esg-epfl-apc/tools-astro/tree/main/tools/astropytools) simple transformations of astrophysical data, part of any real analysis workflow, as provided by [astropy](https://github.com/astropy/astropy) package. These tools also allow converting astrophysical data into formats consumable by many of the Galaxy tools.


While considering the next tools to add, we realised that many of our [typical workflows](https://doi.org/10.48550/arXiv.2002.12895) contain **hundreds of different tools, repeated hundreds of thousands of times**. Furthermore, many of these tools are unique and not especially useful outside a single workflow. Instead, **we decided to focus on tools and workflows producing and consuming standard reusable data types** (images, spectra, light curves). This way, the analysis complexity is hidden inside some of the tools.

Converting tools to Galaxy format, we noticed that much of the work consists in mechanical mapping of astronomical tool annotation into galaxy tools. To simplify this process, we decided to make a converter from existing collections of tools into Galaxy. Examples of tool collections are [EOSSR](https://pypi.org/project/eossr/), [HEASOFT](https://heasarc.gsfc.nasa.gov/lheasoft/), [Renku Projects](https://gitlab.renkulab.io/astronomy/mmoda). First case we took creates [PRs for tools](https://github.com/esg-epfl-apc/tools-astro/pulls) for workflows in Renkulab. It quickly became apparent that some of our collaborators benefit from a public instance for previewing our tools before delivering them, and we implemented a [small galaxy instance](https://galaxy.odahub.fr/).

As of now, we are continuing implementation of astronomical galaxy tools, especially those useful in multi-messenger analysis, when interoperability and reusability of workflows is especially crucial.

While reaching-out to various astronomical RIs (e.g. CTA, SKA, ESA) we realized that one of the key concerns of the astro community is making use of large data volumes: getting them into the galaxy, running massively data-parallel workflows. For discovering and selecting data we adopted **IVOA TAP protocol**, and implemented a **galaxy tool** making use of it.

We are also exploring the possibility of using Galaxy as a GUI for particular telescope workflows, in cases when such a GUI is needed and when it is not yet available.

### What have we achieved given our level of maturity:

With the features and tools we added to Galaxy, it now provides a much more familiar environment for an astronomer.

With the prototype versions of several astronomical telescope data reduction tools, we are demonstrating how Galaxy can produce publication-ready astronomical results fitting real-life needs of astronomers.

### Our setup (technical file)

We are relying on usegalaxy.eu for making our tools broadly available.
To facilitate galaxy live tool review in galaxy tool flow, we setup a small [preview instance](https://galaxy.odahub.fr/) .

On preview instance - local users only, **authentication**, **basic job configuration** and
**storage**


### Problems to solve

We are still trying to understand how to best use Galaxy with “**Big Data**”: data which is costly to relocate and hence should be treated by reference (following “**deferred data**” Galaxy concept). We are reaching out to other communities with similar needs.

We want to make use of our dedicated compute resources, much of them within the “Grid” computing paradigm (relying on technologies like DIRAC, ARC, Rucio), fitting well within plans of WP4 of ESG.

There is a generic difficulty in choosing a suitable degree of detail for the workflow producing scientific outcome. Since telescope data reduction pipelines are quite unique and have few reusable pieces, we focused on the workflows **combining high-level products**, while **specific telescope workflows are contained within individual tools**. It is to be determined if a high degree of detail is needed for each individual telescope workflow.

### Our community outreach

- Personal contacts
- We are participating in CTAO and SKAO research infrastructures.
- We have close contacts with ESA and preparing tools for data reduction of several space telescopes