add all filter transmission functions from SVO

[dannygoldstein]

@kbarbary I propose that we make all filters from

http://svo2.cab.inta-csic.es/svo/theory/fps/index.php?mode=browse

available through sncosmo. This would result in about 6000 total filters exposed by the program, so wanted to discuss it as an issue first.

thoughts?

[kbarbary]

Whoa, awesome. I didn't know about the filter profile service. Looks like what I was imagining the stuff in sncosmo could eventually become, but they've already done it, which is great. Some questions:

• Can all filters be accessed programatically in the same way (enabling us to write minimal code on the sncosmo side to download directly from the service)?
• Are you thinking they would be accessible through sncosmo.get_bandpass(), like existing filters? That is, would the difference between filters from the service and those from other sources be transparent to users? (I'm guessing yes.)
• What are you thinking of for a naming scheme and how would it mesh with the existing filter naming scheme? What do we do for filters that already exist in sncosmo?
• For filters that already exist in sncosmo, how do their definitions compare to the corresponding filters from the service?
• What happens if a filter disappears from the service or if the service disappears? With other filters, we had the option to put them on sncosmo.github.io, but with 6000+ filters this becomes infeasible. (I admit it seems unlikely that this would happen, but it would be good to know what we would do.)

I'd like to hear others' opinions as well.

[dannygoldstein]

cc @mansikasliwal @profjsb

[dannygoldstein]

Whoa, awesome. I didn't know about the filter profile service. Looks like what I was imagining the stuff in sncosmo could eventually become, but they've already done it, which is great. Some questions:

• Can all filters be accessed programatically in the same way (enabling us to write minimal code on the sncosmo side to download directly from the service)?

Yes, API docs here: http://svo2.cab.inta-csic.es/svo/theory/fps/index.php?mode=voservice

• Are you thinking they would be accessible through sncosmo.get_bandpass(), like existing filters? That is, would the difference between filters from the service and those from other sources be transparent to users? (I'm guessing yes.)

Definitely. No API changes proposed here.

• What are you thinking of for a naming scheme and how would it mesh with the existing filter naming scheme? What do we do for filters that already exist in sncosmo?

for a naming scheme, I propose two approaches:

1. for new filters that do not already exist in sncosmo:

facility::instrument-filtername

e.g.,

lasilla::combo17-b

2. for filters that already exist in sncosmo, i propose making the versions available from SVO as a new version of the already existing filter in the registry, with the same name, e.g.,

bessellb, version='svo-1'

• For filters that already exist in sncosmo, how do their definitions compare to the corresponding filters from the service?

See above for a proposed handling of this.

• What happens if a filter disappears from the service or if the service disappears? With other filters, we had the option to put them on sncosmo.github.io, but with 6000+ filters this becomes infeasible. (I admit it seems unlikely that this would happen, but it would be good to know what we would do.)

I propose we download all the filters and host them at NERSC on a project directory that will never go away, and can handle large amounts of data.

I'd like to hear others' opinions as well.

[profjsb]

Posting a static version of all the bandpasses at Zenodo would be my default for longevity—becomes versionable too. Could also host/version using git LFS.

User could pip install bandpasses or sncosmo.download_allbandpasses() as a preamble (like what is done for some large scikit-learn datasets or large nltk models). For lighter weight usage, using a named bandpass in sncosmo would then amount to checking if that bandpass exists locally and if not downloading the bandpass from the site mentioned above.

[dannygoldstein]

More useful information here

http://ivoa.net/documents/Notes/SVOFPS/NOTE-SVOFPS-1.0.20121015.pdf
http://ivoa.net/documents/Notes/SVOFPSDAL/20130510/NOTE-SVOFPSDAL-v2-1.0-20130510.pdf

[rbiswas4]

Having access to more bandpasses from different observatories is great!

I like @dannygoldstein 's proposals on naming.

But, beyond naming, this makes me think of a couple of issues that are more general:

• Since this is a large repository, it would be good to know that bands came from this repository even when the bandpass was the first instance in sncosmo
• Since, we are well-connected with certain surveys, and might prefer to use bandpasses from direct sources, can users express preferences somewhere?

This is almost like a conda channel, and maybe this could be addressed by having a source attribute in Bandpass, and users setting up a source preference somewhere ?

Returning to @profjsb 's suggestion sncosmo releases are already on zenodo, so having the bandpasses that would be obtained by a simple get_bandpass call on a release should also be similarly maintained. git_lfs could be a solution for intermediate times.

[kbarbary]

So I think this is a great idea and the details can be ironed out. However, there are enough tricky details about backwards compatibility, documentation, etc that it might be worthwhile to write up as an enhancement proposal before implementing. For example:

• What will the local storage directory structure (in $SNCOSMO_DATA_DIR/bandpasses) look like for these filters? (Currently that directory is organized by instrument, not facility)
• get_bandpass doesn't currently have versioning. Are we adding one and what will that look like in detail? What will the default version be?
• Currently, when get_bandpass() receives an unknown filter, it throws an exception that lists all the known filters. This will become infeasible with 6000+ filters. What will the new behavior be?
• Documentation: currently all the bandpasses are documented at https://sncosmo.readthedocs.io/en/latest/bandpass-list.html along with their transmission plots. Presumably we'll no longer do this for all 6000+ filters, since it is already done on the SVO site. How will the bandpass page change? I'm guessing we'll want to still document the non-SVO filters... but how will it be clear to users which filters are available?
• Internals: how will this be implemented? This is a somewhat major architectural change because we won't be explicitly registering every single existing filter as we do now in https://github.com/sncosmo/sncosmo/blob/master/sncosmo/builtins.py
• Fallback mechanism: currently if a file disappears from an external host, the fallback mechanism is clear: we remove it from redirects.json and add it to sncosmo.github.io/data. This is nice because it fixes the problem for all existing versions of sncosmo and doesn't require releasing a new version of sncosmo. Will we have something similar? Will we use the same mechanism?

There are probably several other things to consider that I haven't thought of here. Would you @dannygoldstein (or anyone else interested it implementing this) be willing to write up an enhancement proposal?

[dannygoldstein]

@kbarbary I will write up a proposal

[dannygoldstein]

Update: the only filters that are present in sncosmo that are not present in the SVO are:

• radially dependent megacampsf (radially averaged are present)
• 4shooter2
• jwst miri tophat
• snls3-landolt

[kboone]

I'm going to reopen this. The bandpasses in SNCosmo are kind of a mess right now with different naming conventions and formats, and it would be good to think about how to improve this. This Issue has a lot of insightful discussion about this.