Skip to content

A curated list of datasets, tools and papers for machine learning in heliophysics.

Notifications You must be signed in to change notification settings

i4Ds/awesome-helio

Repository files navigation

Awesome Helio

A curated list of datasets, tools and papers for machine learning in heliophysics.

About

The primary goal of awesome-helio is to provide a compiled list for all the great resources contributing to the state of the art in heliophysics that are out there but are often hard to find. awesome-helio is mainly focussed on SDO and IRIS data and targets machine learning groups but can also provide important insights for other users.

Contributing

Please take a quick look at the contribution guidelines first.

If you find any datasets, tools and papers that is missing or is not a good fit, please submit a pull request to improve this file. Thank you!

Contents

Missions

SDO and IRIS are part of a larger fleet of spacecraft strategically placed throughout our heliosphere:

Heliophysics Mission Fleet Heliophysics Mission Fleet, Source NASA

A comprehensive list of different missions can be found here.

Books

Datasets

For datasets at FHNW refer to this link.

Data formats

JPEG vs. Fits (Flexible Image Transport System - commonly used digital file format in astronomy)

e-Callisto

  • e-Callisto Data access - The CALLISTO spectrometer is a programmable heterodyne receiver used for observation of solar radio bursts for astronomical science, education, outreach and citizen science as well as rfi-monitoring.

IRIS

  • LMSAL data search - Useful search interface by Lockheed Martin with links to various resources for each observation. Use our internal mirror for data download or work directly on one of the dedicated machines.
  • IRISdataset (Currently only for internal access) - Dataset with selected representative observations for different observation classes.

RHESSI

TODO

SDO

Before working with SDO data make sure to checkout the Guide to SDO Data Analysis, which contains a set of instructions on how to browse, find, download, and analyze SDO data.

When working with AIA data and analyzing short-lived phenomena, make sure to consider respiking the data as the AIA despiking algorithm might remove some of the features (Young, Peter R., et al. "An Analysis of Spikes in Atmospheric Imaging Assembly (AIA) Data." Solar Physics 296.12 (2021): 1-21.).

SOHO

  • Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021).

Solar Orbiter

  • Solar Orbiter Notebooks - Repository holding all the tutorial notebooks for the data analysis day of the Solar Orbiter 8 Workshop.
  • STIX Data Center - Spectrometer Telescope for Imaging X-rays (STIX) on Solar Orbiter is a hard X-ray imaging spectrometer covering the energy range from 4 to 150 keV

Links

Papers

Datasets

  • Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "A large-scale solar dynamics observatory image dataset for computer vision applications." Scientific data 4.1 (2017): 1-9. link
  • McGranaghan, R., et al. "Machine learning databases used for Journal of Geophysical Research: Space Physics manuscript: New capabilities for prediction of high-latitude ionospheric scintillation: A novel approach with machine learning.". (2018). figshare. Dataset. link
  • Ahmadzadeh, Azim, Dustin J. Kempton, and Rafal A. Angryk. "A Curated Image Parameter Data Set from the Solar Dynamics Observatory Mission." The Astrophysical Journal Supplement Series 243.1 (2019): 18. link
  • Galvez, Richard, et al. "A machine-learning data set prepared from the NASA solar dynamics observatory mission." The Astrophysical Journal Supplement Series 242.1 (2019): 7. link
  • Baek, Ji-Hye, et al. "Solar Event Detection Using Deep-Learning-Based Object Detection Methods." Solar Physics 296.11 (2021): 1-15. link
  • Angryk, Rafal A., et al. "Multivariate time series dataset for space weather data analytics." Scientific data 7.1 (2020): 1-13. link code
  • Mahajan, Sushant S., et al. "Improved Measurements of the Sun's Meridional Flow and Torsional Oscillation from Correlation Tracking on MDI and HMI Magnetograms." The Astrophysical Journal 917.2 (2021): 100. link data
  • Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021). link
  • Bobra, Monica G., et al. "SMARPs and SHARPs: Two Solar Cycles of Active Region Data." The Astrophysical Journal Supplement Series 256.2 (2021): 26. link code

Event detection, classification and tracking

  • Martens, P. C. H., et al. "Computer vision for the solar dynamics observatory (SDO)." Solar Physics 275.1 (2012): 79-113. link
  • Banda, Juan M., and Rafal A. Angryk. "Unsupervised learning techniques for detection of regions of interest in Solar Images." 2015 IEEE International Conference on Data Mining Workshop (ICDMW). IEEE, 2015. link
  • Schuh, Michael A., Dustin Kempton, and Rafal A. Angryk. "A Region-Based Retrieval System for Heliophysics Imagery." FLAIRS Conference. 2017. link
  • Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "Solar event classification using deep convolutional neural networks." International Conference on Artificial Intelligence and Soft Computing. Springer, Cham, 2017. link
  • Illarionov, Egor A., and Andrey G. Tlatov. "Segmentation of coronal holes in solar disc images with a convolutional neural network." Monthly Notices of the Royal Astronomical Society 481.4 (2018): 5014-5021. link
  • Kempton, Dustin J., Michael A. Schuh, and Rafal A. Angryk. "Tracking solar phenomena from the sdo." The Astrophysical Journal 869.1 (2018): 54. link
  • Armstrong, John A., and Lyndsay Fletcher. "Fast solar image classification using deep learning and its importance for automation in solar physics." Solar Physics 294.6 (2019): 1-23. link
  • Gitiaux, Xavier, et al. "Probabilistic Super-Resolution of Solar Magnetograms: Generating Many Explanations and Measuring Uncertainties." arXiv preprint arXiv:1911.01486 (2019). link
  • Jungbluth, Anna, et al. "Single-frame super-resolution of solar magnetograms: Investigating physics-based metrics& losses." arXiv preprint arXiv:1911.01490 (2019). link
  • Love, Teri, Thomas Neukirch, and Clare E. Parnell. "Analyzing AIA Flare Observations Using Convolutional Neural Networks." Frontiers in Astronomy and Space Sciences 7 (2020): 34. link
  • Mackovjak, Šimon, et al. "SCSS-Net: solar corona structures segmentation by deep learning." Monthly Notices of the Royal Astronomical Society 508.3 (2021): 3111-3124. code link
  • Broock, Elena García, Tobías Felipe, and A. Asensio Ramos. "Performance of solar far-side active region neural detection." Astronomy & Astrophysics 652 (2021): A132. link
  • Innocenti, Maria Elena, et al. "Unsupervised classification of simulated magnetospheric regions." Annales Geophysicae. Vol. 39. No. 5. Copernicus GmbH, 2021. link

Missions

  • Pesnell, W. Dean, B. J. Thompson, and P. C. Chamberlin. "The solar dynamics observatory (SDO)." The Solar Dynamics Observatory. Springer, New York, NY, 2011. 3-15. link
  • Lemen, James R., et al. "The atmospheric imaging assembly (AIA) on the solar dynamics observatory (SDO)." The solar dynamics observatory. Springer, New York, NY, 2011. 17-40. link

Space weather prediction

  • Bobra, Monica G., and Sebastien Couvidat. "Solar flare prediction using SDO/HMI vector magnetic field data with a machine-learning algorithm." The Astrophysical Journal 798.2 (2015): 135. link
  • McGregor, Sean, et al. "Flarenet: A deep learning framework for solar phenomena prediction." Neural Information Processing Systems (NIPS) 2017 workshop on Deep Learning for Physical Sciences (DLPS), Long Beach, CA, US. 2017. link
  • Nagem, Tarek AM, et al. "Deep learning technology for predicting solar flares from (Geostationary Operational Environmental Satellite) data." (2018) link
  • Panos, Brandon, et al. "Identifying typical Mg II flare spectra using machine learning." The Astrophysical Journal 861.1 (2018): 62. link
  • Jonas, Eric, et al. "Flare prediction using photospheric and coronal image data." Solar Physics 293.3 (2018): 1-22. link
  • McGranaghan, Ryan M., et al. "New capabilities for prediction of high‐latitude ionospheric scintillation: A novel approach with machine learning." Space Weather 16.11 (2018): 1817-1846. link
  • Camporeale, Enrico. "The challenge of machine learning in space weather: Nowcasting and forecasting." Space Weather 17.8 (2019): 1166-1207. link
  • Chen, Yang, et al. "Identifying solar flare precursors using time series of SDO/HMI Images and SHARP Parameters." Space Weather 17.10 (2019): 1404-1426. link
  • Panos, Brandon, and Lucia Kleint. "Real-time flare prediction based on distinctions between flaring and non-flaring active region spectra." The Astrophysical Journal 891.1 (2020): 17. link
  • Ivanov, Sergey, et al. "Solar activity classification based on Mg II spectra: towards classification on compressed data." arXiv preprint arXiv:2009.07156 (2020). link
  • Wang, Jingjing, et al. "Solar Flare Predictive Features Derived from Polarity Inversion Line Masks in Active Regions Using an Unsupervised Machine Learning Algorithm." The Astrophysical Journal 892.2 (2020): 140. link
  • Ahmadzadeh, Azim, et al. "How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events." The Astrophysical Journal Supplement Series 254.2 (2021): 23. link
  • Li, Xuebao, et al. "Predicting solar flares using a novel deep convolutional neural network." The Astrophysical Journal 891.1 (2020): 10. link
  • Wang, Xiantong, et al. "Predicting solar flares with machine learning: Investigating solar cycle dependence." The Astrophysical Journal 895.1 (2020): 3. link
  • McGranaghan, Ryan M., et al. "Toward a next generation particle precipitation model: Mesoscale prediction through machine learning (a case study and framework for progress)." Space Weather 19.6 (2021): e2020SW002684. link
  • Krista, Larisza D., and Matthew Chih. "A DEFT Way to Forecast Solar Flares." The Astrophysical Journal 922.2 (2021): 218. link
  • Abduallah, Yasser, et al. "DeepSun: machine-learning-as-a-service for solar flare prediction." Research in Astronomy and Astrophysics 21.7 (2021): 160. link
  • Pandey, Chetraj, Rafal A. Angryk, and Berkay Aydin. "Solar Flare Forecasting with Deep Neural Networks using Compressed Full-disk HMI Magnetograms." 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021. link
  • Abed, Ali K., Rami Qahwaji, and Ahmed Abed. "The automated prediction of solar flares from SDO images using deep learning." Advances in Space Research 67.8 (2021): 2544-2557. link
  • Nishizuka, Naoto, et al. "Operational solar flare prediction model using Deep Flare Net." Earth, Planets and Space 73.1 (2021): 1-12. link
  • Brown, Edward JE, et al. "Attention‐Based Machine Vision Models and Techniques for Solar Wind Speed Forecasting Using Solar EUV Images." Space Weather 20.3 (2022): e2021SW002976. link
  • Hu, Andong, et al. "Probabilistic prediction of Dst storms one-day-ahead using Full-Disk SoHO Images" code link
  • Huwyler, Cédric, and Martin Melchior. "Using Multiple Instance Learning for Explainable Solar Flare Prediction." arXiv preprint arXiv:2203.13896 (2022). link
  • Sun, Zeyu, et al. "Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data." The Astrophysical Journal 931.2 (2022): 163. link
  • Deshmukh, Varad, et al. "Decreasing False-alarm Rates in CNN-based Solar Flare Prediction Using SDO/HMI Data." The Astrophysical Journal Supplement Series 260.1 (2022): 9. link
  • Guastavino, Sabrina, et al. "Implementation paradigm for supervised flare forecasting studies: A deep learning application with video data." Astronomy & Astrophysics 662 (2022): A105. link

Other Applications

  • Wright, Paul J., et al. "DeepEM: Demonstrating a Deep Learning Approach to DEM Inversion." Zenodo. link
  • Guedes dos Santos, L. F., et al. "Multi-Channel Auto-Calibration for the Atmospheric Imaging Assembly instrument with Deep Learning." AGU Fall Meeting Abstracts. Vol. 2020. 2020. link
  • Brown, Edward JE, et al. "Learning the solar latent space: sigma-variational autoencoders for multiple channel solar imaging." link
  • Jarolim, Robert, et al. "Instrument-To-Instrument translation: Instrumental advances drive restoration of solar observation series via deep learning." (2022). link code

Tools

  • Aiapy - Package for analyzing calibrated (level 1) EUV imaging data from AIA.
  • IRISreader - Python library that allows for efficient browsing through IRIS satellite data in order to simplify machine learning applications.
  • Global High-Resolution Hα network
  • Helioviewer - Solar and heliospheric image visualization tool. code
  • Integrated Solar Database - Solar and heliospheric image visualization tool including image parameters with Extended Spatiotemporal Querying Capabilities.
  • integrated Space Weather Analysis (iSWA) system
  • SolarMonitor - Provides realtime information about the solar activity. The most recent images of the sun are shown, together with the description of the different NOAA AR of the day and which flares has been associated to them.
  • SpaceML - A Machine Learning toolbox and developer community building the next generation AI applications for space science and exploration containing a set of code examples.
  • Sunpy - Set of tools for solar data analysis with Python.
  • SWPC CME Analysis Tool (SWPC_CAT) - Primary tool being used by NOAA SWPC in measuring key parameters of a Coronal Mass Ejection (CME) code
  • The Heliophysics KNOWledge Network - Collection of software and systems for improved information representation in Heliophysics.

Videos

About

A curated list of datasets, tools and papers for machine learning in heliophysics.

Topics

Resources

Code of conduct

Stars

Watchers

Forks

Contributors 4

  •  
  •  
  •  
  •  

Languages