Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

dnaviap · 2024-01-11T12:55:06Z

Add combined GRIB reader that can be used for both SEVIRI and FCI L2 products.

Tests added
Fully documented
Add @dnaviap to AUTHORS.md

Feature/eum l2 grib reader

strandgren · 2024-01-11T16:20:27Z

Thanks for working on this @dnaviap :) Before reviewing in more detail, I think we need some refactoring of the files:

Instead of having a eum_l2_grib.yaml files I suggest to keep the seviri_l2_grib.yaml file (which is still there) and add the FCI cloud mask dataset in a new fci_l2_grib.yaml file. That way we don't change the user interface to the reader. That's also how it's being done for the BUFR reader (Changes to Eumetsat L2 BUFR reader #2603)
eum_l2_grib.py should be compatible with both FCI and SEVIRI data, meaning that seviri_l2_grib.py should become obsolete and should therefore be removed.
Similarly, the new test file test_eum_l2_grib.py should make test_seviri_l2_grib.py obsolete, meaning it can be removed as well.

…ing the user interface to the reader

…and SEVIRI data

Feature/eum l2 grib refactor

dnaviap · 2024-01-12T15:51:20Z

The refactoring is complete and tested.

strandgren

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:

    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """

That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

strandgren · 2024-01-15T12:29:06Z

satpy/readers/eum_l2_grib.py

@@ -62,14 +75,23 @@ def start_time(self):
    @property
    def end_time(self):
        """Return the sensing end time."""
-        return self.start_time + timedelta(minutes=REPEAT_CYCLE_DURATION)
+        if self.sensor == "seviri":
+            return self.start_time + timedelta(minutes=REPEAT_CYCLE_DURATION)


This is not correct for RSS data where the temporal resolution is 5min. Could you add REPEAT_CYCLE_DURATION_RSS = 5 in seviri_base.py and then use the sub-satellite longitude to determine the end_time, ie.e. something like this?

delta = REPEAT_CYCLE_DURATION_RSS if self._ssp_lon == 9.5 else REPEAT_CYCLE_DURATION return self.start_time + delta

strandgren · 2024-01-15T12:41:02Z

satpy/readers/fci_base.py

+def calculate_area_extent(area_dict):
+        """Calculate the area extent seen by MTG FCI instrument.
+
+        Since the center of the FCI L2 grid is located at the interface between the pixels, there are equally many


FCI L2 grid --> FCI grids

strandgren · 2024-01-15T14:04:06Z

satpy/tests/reader_tests/test_eum_l2_grib.py

+FAKE_GID = [0, 1, 2, 3, None]
+
+
+class Test_EUML2GribFileHandler(unittest.TestCase):


It's quite a lot of duplicate code for the SEVIRI and FCI tests, can you please have a look and see if this can be refactored and optimized a bit?

I moved the duplicated code to a common method, please check if it is necessary to do something else

dnaviap · 2024-05-13T08:52:46Z

Thanks for the refactoring! :) See a couple of comments inline.

Can you please also modify the doc-string of the _scale_earth_axis method to read:
    def _scale_earth_axis(data):
        """Scale Earth axis data to make sure the value matched the expected unit [m].
        
        The earthMinorAxis value stored in the MPEF aerosol over sea product prior to December 12, 2022 has the wrong unit and this method provides a flexible work-around by making sure that all earth axis values are scaled such
        that they are on the order of millions of meters as expected by the reader. 
        
        """
That was a fix implemented by me a while ago when we noticed some issues in one of the products, and although it has been fixed in the product, the archived files will still have the issue meaning that the fix is still useful to have in the reader.

strandgren

Thanks for the fixes and the refactoring @dnaviap!

I left a couple of minor comments in-line. Could you please also merge the main branch into this branch and resolve the merge conflict such that all unit tests etc. run?

strandgren · 2024-05-27T08:22:09Z

satpy/readers/eum_l2_grib.py

-        if self.sensor == "seviri":
-            return self.start_time + timedelta(minutes=REPEAT_CYCLE_DURATION)
-        elif self.sensor == "fci":
-            return self.filename_info["end_time"]
+        delta = REPEAT_CYCLE_DURATION_RSS if self._ssp_lon == 9.5 else REPEAT_CYCLE_DURATION
+        return self.start_time + delta


My comment only considered SEVIRI, since for SEVIRI we have a nominal repeat cycle duration of 15 minutes for FES and 5 minutes for RSS. For FCI, however, it's 10minutes. I also think we need to convert the imported integers to datetime.timedela objects? So we should have something like this instead:

if self.sensor == "seviri": delta = REPEAT_CYCLE_DURATION_RSS if self._ssp_lon == 9.5 else REPEAT_CYCLE_DURATION return self.start_time + timedelta(minutes=delta) elif self.sensor == "fci": return self.filename_info["end_time"]

strandgren · 2024-05-27T08:24:30Z

satpy/readers/eum_l2_grib.py

@@ -33,7 +32,7 @@
 from satpy.readers.eum_base import get_service_mode
 from satpy.readers.fci_base import calculate_area_extent as fci_calculate_area_extent
 from satpy.readers.file_handlers import BaseFileHandler
-from satpy.readers.seviri_base import PLATFORM_DICT, REPEAT_CYCLE_DURATION
+from satpy.readers.seviri_base import PLATFORM_DICT, REPEAT_CYCLE_DURATION, REPEAT_CYCLE_DURATION_RSS


To avoid confusion (see e.g. comment below), perhaps rename the variables to make clear that these are SEVIRI variables:

from satpy.readers.seviri_base import ( PLATFORM_DICT as SEVIRI_PLATFORM_DICT, REPEAT_CYCLE_DURATION as SEVIRI_REPEAT_CYCLE_DURATION, REPEAT_CYCLE_DURATION_RSS as SEVIRI_REPEAT_CYCLE_DURATION_RSS )

strandgren · 2024-05-27T08:32:16Z

satpy/tests/reader_tests/test_eum_l2_grib.py

-                # Checks that codes_release has been called after each codes_grib_new_from_file call
-                # (except after the last one which has returned a None)
-                assert self.ec_.codes_grib_new_from_file.call_count == self.ec_.codes_release.call_count + 1
+                self.common_checks(mock_file, dataset_id)

                # Checks the basic data reading
                assert REPEAT_CYCLE_DURATION == 15


Not sure we really need this assert? It's just a constant imported from eum_base.py, and now we also have REPEAT_CYCLE_DURATION_RSS. Perhaps rather verify the start_time and end_time, which would be more useful.

In this case we have ssp_lon=9.5, so start_time should be 2019-10-20 17:45 and end_time 2019-10-20 17:50` as far as I can tell from the fake message.

David Navia and others added 6 commits January 11, 2024 09:57

Add common functionality for FCI data readers

28f44c7

Add reader for both SEVIRI and FCI L2 products in GRIB2 format

d3ccedc

Add EUM L2 GRIB-reader test package

de65b63

Add my name to AUTHORS.md

0474fd0

Merge pull request #1 from dnaviap/feature/eum_l2_grib_reader

1150b44

Feature/eum l2 grib reader

Merge branch 'pytroll:main' into main

d8043db

dnaviap requested review from djhoese and mraspaud as code owners January 11, 2024 12:55

David Navia and others added 5 commits January 12, 2024 16:33

Delete eum_l2_grib.yaml and update seviri_l2_grib.yaml to avoid chang…

767aeab

…ing the user interface to the reader

Add fci_l2_grib.yaml reader

a53ddea

Delete seviri_l2_grib.py since eum_l2_grib.py is compatible with FCI …

f94c4f7

…and SEVIRI data

Delete obsolete test_seviri_l2_grib.py

66946ad

Merge pull request #2 from dnaviap/feature/eum_l2_grib_refactor

e7009de

Feature/eum l2 grib refactor

strandgren reviewed Jan 15, 2024

View reviewed changes

David Navia added 3 commits May 13, 2024 08:47

Refactor duplicate code in tests

5bbb421

Correct for RSS data

486b3a6

Modify fci_base doc-string

c6322fa

dnaviap requested review from sfinkens and adybbroe as code owners May 13, 2024 08:47

dnaviap closed this May 13, 2024

dnaviap reopened this May 13, 2024

strandgren reviewed May 27, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

dnaviap commented Jan 11, 2024

strandgren commented Jan 11, 2024

dnaviap commented Jan 12, 2024

strandgren left a comment

strandgren Jan 15, 2024

dnaviap May 13, 2024

strandgren Jan 15, 2024

dnaviap May 13, 2024

strandgren Jan 15, 2024

dnaviap May 13, 2024

dnaviap commented May 13, 2024

strandgren left a comment

strandgren May 27, 2024

strandgren May 27, 2024

strandgren May 27, 2024

		FAKE_GID = [0, 1, 2, 3, None]


		class Test_EUML2GribFileHandler(unittest.TestCase):

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Are you sure you want to change the base?

Add combined GRIB reader for both SEVIRI and FCI L2 products #2717

Conversation

dnaviap commented Jan 11, 2024

strandgren commented Jan 11, 2024

dnaviap commented Jan 12, 2024

strandgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnaviap commented May 13, 2024

strandgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment