Changes to Eumetsat L2 BUFR reader

[samain-eum]

This PL carries the following feature changes:

• Adapt the EUM BUFR reader to make it generic and no longer SEVIRI speficic (hence the name change)
• Add functionality and configuration yaml file for the FCI BUFR products
• Include a fix / workaround for the case a variable within a BUFR message is all set to fill values, which results in an unexpected array size as returned by eccodes

[strandgren]

Good job, see comments in-line.

[strandgren]

see some more comments in-line.

[strandgren]

Add resolution of AMV product

[strandgren]

Add resolution of AMV product

[strandgren]

Thanks @samain-eum, we're getting close now :) Some more comments in-line, There are also a few minor open comments from the last review, see here: #2603 (review) - only the ones related to renaming SEVIRI --> EUMETSAT, the rest should be fixed, or tracked otherwise.

I also noticed that the units of the SEVIRI GII datasets are not set correctly, could you maybe align this with the latest in the fci_l2_nc reader? i.e.

ki: deg Celcius
ko: deg Celcius
li: deg Celcius
mb: deg Celcius
lpw1: kg/m2 (mm is also correct, but kg/m2 is in line with fci_l2_nc and also seems more inline with CF)
lpw2: kg/m2 
lpw3: kg/m2
tpw1: kg/m2

[strandgren]

A few more comments after testing the code locally.

[strandgren]

Set to simply Alpha since AMV products are supported

[strandgren]

none is not allowed here, breaks for example the Scene.available_dataset_names() helper function. I think it should be enough to simply remove it and only keep [32000]. Confirm with Youva that the AMV reading still works.

[strandgren]

none is not allowed here, breaks for example the Scene.available_dataset_names() helper function. I think it should be enough to simply remove it and only keep [32000]. Confirm with Youva that the AMV reading still works.

[samain-eum]

@mraspaud : We have a question on what do for the resolution of wind products
For the various dedicated variables, we chose to set it to "none", as a spatial resolution doesn't make sense for a wind vector. It shows up as such in the variable attributes imported into satpy.
However we have also "common" variables, i.e. latitude and longitude, where the resolution key is defined as an array, with one element per product in the file_type key. It doesn't seem possible to mix different types for the resolution (it's an array, not a list).
The solutions I have found are either:

• Put a placeholder value for the AMV resolution (something that won't break anything, i.e. 0 or 1 perhaps)
• Create an other kind of latitude / longitude bloc entry for resolution-less data. Same name, just different file types allocated, and resolution: ["none"].

Do you have any preference of advice?

[strandgren]

@mraspaud : We have a question on what do for the resolution of wind products For the various dedicated variables, we chose to set it to "none", as a spatial resolution doesn't make sense for a wind vector. It shows up as such in the variable attributes imported into satpy. However we have also "common" variables, i.e. latitude and longitude, where the resolution key is defined as an array, with one element per product in the file_type key. It doesn't seem possible to mix different types for the resolution (it's an array, not a list). The solutions I have found are either:

• Put a placeholder value for the AMV resolution (something that won't break anything, i.e. 0 or 1 perhaps)
• Create an other kind of latitude / longitude bloc entry for resolution-less data. Same name, just different file types allocated, and resolution: ["none"].

Do you have any preference of advice?

@mraspaud : We have a question on what do for the resolution of wind products For the various dedicated variables, we chose to set it to "none", as a spatial resolution doesn't make sense for a wind vector. It shows up as such in the variable attributes imported into satpy. However we have also "common" variables, i.e. latitude and longitude, where the resolution key is defined as an array, with one element per product in the file_type key. It doesn't seem possible to mix different types for the resolution (it's an array, not a list). The solutions I have found are either:

• Put a placeholder value for the AMV resolution (something that won't break anything, i.e. 0 or 1 perhaps)
• Create an other kind of latitude / longitude bloc entry for resolution-less data. Same name, just different file types allocated, and resolution: ["none"].

Do you have any preference of advice?

To add to this: what is actually the idea of having a list of resolutions for a datasets resolution variable for a dataset in the reader yaml-file? How would satpy know what resolution is the correct one when loading e.g. latitude if there is a list of three values? I first thought there was a relationship between the file_type list and the resolution list, but since equal lengths of these lists/arrays is not needed, that cannot be the case.

If you know the logic behind this @mraspaud or could point to the relevant code that would help us understand better what the best approach is for our use case.

[samain-eum]

@strandgren Actually my idea of separate different lat/lon blocs for resolution-less products doesn't work. It seems satpy overloads the dataset info with the last bloc found in the yaml file and the same name. So it breaks when trying to load lat/lon for ASR for example. So I think I will instead use a dummy value of 0 in the yaml file, and replace it by 'none' when creating the attributes.

As for your question above, the resolution is currently linked to the file pattern itself, so we can't manage datasets with different resolutions within one file.

[mraspaud]

That's a tough question!
The resolution is indeed quite central in satpy, and I don't believe None is an option.

But just for curiosity, why doesn't resolution make sense for wind data? Is it like a continuous field? or is it motion vector derived from pixels at a certain resolution? (in which case maybe the original pixel resolution makes a little sense?)

Otherwise, you can always define custom data identification keys for your reader, like in here: https://github.com/pytroll/satpy/blob/main/satpy/etc/readers/sgli_l1b.yaml#L13-L31
And from there, maybe you can have a resolution-less one? I'm quite sure it will break somewhere, but maybe worth a try?

[samain-eum]

Hi @mraspaud
Wind vectors are produced by finding cloud features that are moving during successive images and a product is a "list" of winds with their lat/lon information, not a gridded product. So yes, resolution don't really make sense.
I'm aware that a None (i.e. the python reserved type) may cause issues, that's why I opted for a "none" string instead. In the scope of our unit tests, it seems to work fine now. It just need dedicated lat/lon coordinates variables with the same value for the resolution.
Maybe it's worth noting that for those product, the 'with_area_definition' flag when creating a scene instance is ignored, so we make no attempt at building an area definition (which requires a resolution). We are not aware of anything else that could break satpy.

This is how the definitions look like now for AMV:

  latitude_amv:
    name: latitude_amv
    key: '#1#latitude'
    resolution: none
    file_type: fci_l2_bufr_amv
    standard_name: latitude
    units: degree_north
    fill_value: -1.e+100
 
  longitude_amv:
    name: longitude_amv
    key: '#1#longitude'
    resolution: none
    file_type: fci_l2_bufr_amv
    standard_name: longitude
    units: degree_east
    fill_value: -1.e+100
 
  pressure:
    name: pressure
    long_name: Pressure of AMV feature
    standard_name: air_pressure_at_wind_level
    resolution: none
    file_type: fci_l2_bufr_amv
    key: '#1#pressure'
    units: Pa
    fill_value: -1.0e+100
    coordinates:
      - longitude_amv
      - latitude_amv