Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a new Fido client that downloads ADAPT magnetograms. #7152

Open
GillySpace27 opened this issue Aug 17, 2023 · 18 comments
Open

Create a new Fido client that downloads ADAPT magnetograms. #7152

GillySpace27 opened this issue Aug 17, 2023 · 18 comments
Labels
Effort Medium Requires a moderate time investment Feature Request New feature wanted! net Affects the net submodule Package Intermediate Requires some knowledge of the internal structure of SunPy Priority Low Slow action required

Comments

@GillySpace27
Copy link

Describe the feature

I want to use FIDO to download ADAPT magnetograms.

Proposed solution

There are many clients. This one would work as a subclass of Generic Client!
@GillySpace27
Copy link
Author

I have written this code and I need to integrate it into the main sunpy code.

@nabobalis nabobalis added Feature Request New feature wanted! net Affects the net submodule labels Aug 17, 2023
@nabobalis
Copy link
Contributor

We would need to discuss with @sunpy/lead-developers to see if we want to accept more clients into sunpy directly but if you would like, sending in a pull request would be great.

If you need help with that, please let me know and I can help get the client added to sunpy!

@GillySpace27
Copy link
Author

GillySpace27 commented Aug 17, 2023 via email

@nabobalis
Copy link
Contributor

Hi Nabil, I was talking to Stuart and Will Barnes at the SPD meeting on Monday and they thought it would be really useful since the ADAPT maps are being used more and more over time.

That is good to know!

I’m not really sure how to make real tests, but I have some code that runs and makes sure everything is working.

I would suggest just copying one of the other test files and then adjusting it for the output from your client. Then we can go from there.

Am I supposed to make a branch, commit to that, then eventually make the pull request? Or is it ok to just work on the main branch since I’m only touching the one thing? Any other advice would be appreciated! Thanks, Gilly

Ideally it is better if you use a new branch, this allows you to keep your fork's main up to date.
However, in the grand scheme of things, it doesn't matter if you create a branch on your fork or just use the main branch. So do whatever is easier for you.

@Cadair
Copy link
Member

Cadair commented Aug 17, 2023

I had a chat to some VSO people, and they are planning on having these data in the VSO, but they are not sure when. I am happy to have a client added until it becomes redundant.

@ryuusama09
Copy link
Contributor

Hey @nabobalis , I dont see the pr here . Is this feature request still relevant ?

@nabobalis
Copy link
Contributor

Hey @nabobalis , I dont see the pr here . Is this feature request still relevant ?

Yes

@ryuusama09
Copy link
Contributor

Great , as far as I understand we need to make a subclass of the generic scraper client .. I suppose There wont be much difference and the skeletal code could be the same as other sub clients ? @nabobalis .

@nabobalis
Copy link
Contributor

Hopefully

@ryuusama09
Copy link
Contributor

ryuusama09 commented Feb 29, 2024 •
edited

@nabobalis , I don't have much experience with clients , can you suggest a client whose behaviour should be the same as this one . I mean I would like to go through the structure of the directory once , what all attributes will be required etc ...

@nabobalis
Copy link
Contributor

@nabobalis , I don't have much experience with clients , can you suggest a client whose behaviour should be the same as this one . I mean I would like to go through the structure of the directory once , what all attributes will be required etc ...

Err, I'm not familiar with the ADATA data server, but I would suggest any client under the datasources folder. All of them implement a scraper client that is useful to look at.

Do say if that's not helpful.

@GillySpace27
Copy link
Author

GillySpace27 commented Feb 29, 2024 •
edited

Hi Folks,

I actually already implemented the ADAPTClient, which I'll share here with you as a first step. I'll try to figure out the "correct" way to share it tomorrow.

from sunpy.net import attr, attrs, Fido
from sunpy.net.dataretriever import GenericClient

class ADAPTFileType(attr.SimpleAttr):
    """
    ADAPT file type: Public.
    """
    pass

class ADAPTLngType(attr.SimpleAttr):
    """
    ADAPT longitude type: Carrington Fixed, Central Meridian, East Limb.
    """
    pass

class ADAPTInputSource(attr.SimpleAttr):
    """
    ADAPT input source: All, KPVT, VSM, GONG, HMI, MDI, MWO.
    """
    pass

class ADAPTDataAssimilation(attr.SimpleAttr):
    """
    ADAPT data assimilation: WH, enLS, enkf, enLAKF.
    """
    pass

class ADAPTResolution(attr.SimpleAttr):
    """
    ADAPT model spatial resolution: 1.0 deg, 0.2 deg.
    """
    pass

class ADAPTVersionYear(attr.SimpleAttr):
    """
    ADAPT code version year.
    """
    pass

class ADAPTVersionMonth(attr.SimpleAttr):
    """
    ADAPT code version month.
    """
    pass

class ADAPTEvolutionMode(attr.SimpleAttr):
    """
    ADAPT evolution mode: Data assimilation step, Intermediate step, Forecast step.
    """
    pass

class ADAPTHelioData(attr.SimpleAttr):
    """
    ADAPT helioseismic data: Not added or no data, Far-side, Emergence, Both emergence & far-side.
    """
    pass

class ADAPTRealizations(attr.SimpleAttr):
    """
    ADAPT realizations: number of model/file realizations, e.g., 16 -> "016"
    """
    pass


class ADAPTMagData(attr.SimpleAttr):
    """
    ADAPT magnetic data: Not added or no data, Mag-los, Mag-vector, Mag- both los & vector, Mag- polar avg obs, Mag- los & polar, Mag- vector & polar, Mag- both los and vector & polar.
    """
    pass


class ADAPTClient(GenericClient):

    baseurl = r'https://gong.nso.edu/adapt/maps/gong/%Y/adapt(\d){5}_(\d){2}(\w){1}(\d){3}_(\d){12}_(\w){1}(\d){8}(\w){1}(\d){1}\.fts\.gz'
    pattern = '{}adapt{ADAPTFileType:1d}{ADAPTLngType:1d}{ADAPTInputSource:1d}{ADAPTDataAssimilation:1d}{ADAPTResolution:1d}_{ADAPTVersionYear:2d}{ADAPTVersionMonth:1l}{ADAPTRealizations:3d}_{year:4d}{month:2d}{day:2d}{hour:2d}{minute:2d}_{ADAPTEvolutionMode:1l}{days_since_last_obs:2d}{hours_since_last_obs:2d}{minutes_since_last_obs:2d}{seconds_since_last_obs:2d}{ADAPTHelioData:1l}{ADAPTMagData:1d}.fts.gz'
    


    @classmethod
    def register_values(cls):
        from sunpy.net import attrs as a

        adict ={attrs.Instrument: [('ADAPT', 'ADvanced Adaptive Prediction Technique.')],
                attrs.Source: [('NSO', 'National Solar Observatory.')],
                attrs.Provider: [('GONG', 'Global Oscillation Network Group.')],
                ADAPTFileType: [('4', 'Public')],
                ADAPTLngType: [('0', 'Carrington Fixed'), ('1', 'Central Meridian'), ('2', 'East Limb')],
                ADAPTInputSource: [('0', 'All'), ('1', 'KPVT'), ('2', 'VSM'), ('3', 'GONG'), ('4', 'HMI'), ('5', 'MDI'), ('6', 'MWO')],
                ADAPTDataAssimilation: [('0', 'WH'), ('1', 'enLS'), ('2', 'enkf'), ('3', 'enLAKF')],
                ADAPTResolution: [('1', '1.0 deg'), ('2', '0.2 deg')],
                ADAPTVersionYear: [(str(i), f"Code version year -> {2000 + i}") for i in range(1, 20)],
                ADAPTRealizations: [(str(i), f"Number of Realizations -> {i}") for i in range(1, 20)],
                ADAPTVersionMonth: [(chr(i+96), f"Code version month -> {i}") for i in range(1, 13)],
                ADAPTEvolutionMode: [('a', 'Data assimilation step'), ('i', 'Intermediate step'), ('f', 'Forecast step')],
                ADAPTHelioData: [('n', 'Not added or no data'), ('f', 'Far-side'), ('e', 'Emergence'), ('b', 'Both emergence & far-side')],
                ADAPTMagData: [('0', 'Not added or no data'), ('1', 'Mag-los'), ('2', 'Mag-vector'), ('3', 'Mag- both los & vector'), ('4', 'Mag- polar avg obs'), ('5', 'Mag- los & polar'), ('6', 'Mag- vector & polar'), ('7', 'Mag- both los and vector & polar')]
                }
        return adict
    
    @classmethod
    def _can_handle_query(cls, *query):
        required = {attrs.Instrument, attrs.Time}

        optional = {ADAPTFileType, ADAPTLngType, ADAPTInputSource, ADAPTDataAssimilation, 
                    ADAPTResolution, ADAPTVersionYear, ADAPTVersionMonth, ADAPTEvolutionMode, 
                    ADAPTHelioData, ADAPTMagData}

        all_attrs = {type(x) for x in query}
        # print(all_attrs)
        return required.issubset(all_attrs) and all_attrs.issubset(required.union(optional))
    
def carrington_time(CR=2193, frames=1):
    """Get the start date for the start of a carrington rotation, 
    and a duration that will retrieve a given number of frames

    Returns:
        _type_: _description_
    """
    import astropy.units as u
    import sunpy.coordinates
    date = sunpy.coordinates.sun.carrington_rotation_time(CR)
    date_end = date + frames*(3*1.9999999 * u.hour)

    # Format the Search Dates
    tstring = r"%Y-%m-%dT%H:%M:%S"
    get_date    =date.strftime(tstring)
    get_date_end=date_end.strftime(tstring)
    return get_date, get_date_end

def test_client(CR=2193, frames=1, ask=False):
    res = test_search(CR, frames)
    print("Found {} files".format(len(res)))
    print(res)
    # ask the user to continue or not
    if len(res) > 0:
        if ask:
            print("Do you want to continue?")
            inp = input("y/n: ")
            if inp == 'y':
                out = test_fetch(res)
            else:
                print("Aborting")
                return None
        else:
            return test_fetch(res)
    

def test_search(CR=2193, frames=1):
    # Get the date range
    get_date, get_date_end = carrington_time(CR, frames)
    LngType = '0' # 0 is carrington, 1 is central meridian
    res = Fido.search(attrs.Instrument('adapt'), attrs.Time(get_date, get_date_end), ADAPTLngType(LngType))
    print(res)
    return res

def test_fetch(res, path="../flux-extra/downloads"):
    import os
    directory = os.path.join(os.getcwd(), path)
    if not os.path.exists(directory):
        os.makedirs(directory)
    ret =  Fido.fetch(res, path=directory)
    test_download(ret)
    return ret


def test_download(out):
    print(out)
    pass

if __name__ == "__main__":
    test_client()

@ryuusama09
Copy link
Contributor

woah , great @GillySpace27 . Since I have little to no idea of adapt maps I am sorry that I am asking such question !. what queryresponsetype are we planning to use ? . Like is there a special way to handle adapt map objects or will it be handled by the generic-client ? .

@GillySpace27 GillySpace27 mentioned this issue Feb 29, 2024
Open
@GillySpace27
Copy link
Author

GillySpace27 commented Mar 1, 2024 •
edited

I've just made a draft pull request here: #7463
I'm not exactly sure what to do next.
Ryuusama09, I think that the response type is handled by the GenericClient, but feel free to take a look around the code.
Most of the code I touched is in these three green files:
image

@ryuusama09
Copy link
Contributor

ryuusama09 commented Mar 1, 2024 •
edited

I've just made a draft pull request here: #7463 I'm not exactly sure what to do next. Ryuusama09, I think that the response type is handled by the GenericClient, but feel free to take a look around the code. Most of the code I touched is in these three green files: image

oh , great . I was planning to raise the pr but nevermind 😅. Moreover I don't see the search and the fetch methods implemented (I hope i am not missing out something ). @nabobalis can you suggest some issue for me ? . It seems i am finding a tough time to find one .

@nabobalis
Copy link
Contributor

I've just made a draft pull request here: #7463 I'm not exactly sure what to do next. Ryuusama09, I think that the response type is handled by the GenericClient, but feel free to take a look around the code. Most of the code I touched is in these three green files: ![image](https://private-user-images.githubusercontent.
oh , great . I was planning to raise the pr but nevermind 😅. Moreover I don't see the search and the fetch methods implemented (I hope i am not missing out something ). @nabobalis can you suggest some issue for me ? . It seems i am finding a tough time to find one .

I would suggest looking at good first issues or package novice issues for something that catches your eye.

@ryuusama09

This comment was marked as outdated.

Sign in to view
@nabobalis

This comment was marked as outdated.

Sign in to view
@nabobalis nabobalis added Priority Low Slow action required Effort Medium Requires a moderate time investment Package Intermediate Requires some knowledge of the internal structure of SunPy labels Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Effort Medium Requires a moderate time investment Feature Request New feature wanted! net Affects the net submodule Package Intermediate Requires some knowledge of the internal structure of SunPy Priority Low Slow action required
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Cadair @nabobalis @GillySpace27 @ryuusama09