Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add provider eumetsat_ds #1060

Merged
merged 20 commits into from
May 16, 2024
Merged

feat: add provider eumetsat_ds #1060

merged 20 commits into from
May 16, 2024

Conversation

anesson-cs
Copy link
Collaborator

@anesson-cs anesson-cs commented Mar 14, 2024

Fixes #989

Add EUMETSAT data store as a new provider

Note: ssl_verify parameter must be added to search and download plugins of the provider

Copy link
Contributor

github-actions bot commented Mar 14, 2024

Test Results

    4 files  ± 0      4 suites  ±0   4m 57s ⏱️ +9s
  529 tests + 4    526 ✅ + 4   3 💤 ±0  0 ❌ ±0 
2 116 runs  +16  2 022 ✅ +16  94 💤 ±0  0 ❌ ±0 

Results for commit 94cf0ad. ± Comparison against base commit 0440f20.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Mar 14, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 646-692, 794-845, 849
config.py                                      342      27  92.11%   81-83, 92, 100, 104-106, 179, 191, 445-447, 511-514, 561-562, 571-572, 651, 720-725, 727
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    739      76  89.72%   88-97, 370, 597, 641-644, 682, 787, 791-796, 822, 892, 969, 1082, 1172-1184, 1224, 1226, 1230, 1253, 1257-1268, 1281-1287, 1377-1380, 1409-1429, 1486, 1503-1506, 1518-1521, 1889, 1922-1928, 2193, 2197-2200, 2214-2216, 2251
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      84  87.59%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             132      14  89.39%   49-51, 105-110, 156, 195, 215, 241, 280-281
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      57  78.08%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py          179      24  86.59%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 527, 529
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     551      58  89.47%   96, 368-374, 382-383, 490-496, 551-554, 627-628, 669, 688, 703, 756, 777, 780-781, 790, 801, 810, 833, 895-900, 904-905, 933, 1004, 1051, 1127-1131, 1197-1198, 1219, 1244-1256, 1263, 1294-1295, 1306-1312, 1355, 1370, 1390, 1468
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   206      16  92.23%   66-72, 261, 345, 605, 607, 610-612, 684, 691-695
rest/server.py                                 299      54  81.94%   80-81, 107, 130-131, 244-246, 262, 302-303, 315-331, 423-428, 460, 627-634, 667, 711-712, 803-805, 822-827, 858, 860, 864-865, 869-870
rest/stac.py                                   436      86  80.28%   60-62, 272, 294, 366, 403-405, 428, 463-464, 550-595, 640, 660-661, 841, 906-908, 1127, 1137-1149, 1162-1184, 1198-1243, 1402-1403
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/stac_queryables.py                   32       2  93.75%   28, 116
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                          96      13  86.46%   51, 108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                              111      12  89.19%   53, 70, 129-132, 199, 214-218, 245, 250, 258
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      42  91.63%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1186, 1204, 1380
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8941    1540  82.78%

Diff against develop

Filename                           Stmts    Miss  Cover
-------------------------------  -------  ------  -------
config.py                             +1       0  +0.03%
api/product/_product.py              -12      -2  +0.05%
api/product/metadata_mapping.py      +24       0  +0.45%
plugins/authentication/token.py       +4       0  +0.88%
plugins/search/qssearch.py            -2       0  -0.04%
TOTAL                                +15      -2  +0.06%

Results for commit: 94cf0ad

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Mar 14, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 646-692, 794-845, 849
config.py                                      342      28  91.81%   81-83, 92, 100, 104-106, 179, 191, 445-447, 511-514, 561-562, 571-572, 651, 685, 720-725, 727
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    739      76  89.72%   88-97, 370, 597, 641-644, 682, 787, 791-796, 822, 892, 969, 1082, 1172-1184, 1224, 1226, 1230, 1253, 1257-1268, 1281-1287, 1377-1380, 1409-1429, 1486, 1503-1506, 1518-1521, 1889, 1922-1928, 2193, 2197-2200, 2214-2216, 2251
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      85  87.44%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1486, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             132      14  89.39%   49-51, 105-110, 156, 195, 215, 241, 280-281
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      59  77.31%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py          179      31  82.68%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 519-534
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     551      82  85.12%   96, 368-374, 382-383, 490-496, 551-554, 627-628, 669, 688, 703, 756, 777, 780-781, 790, 801, 810, 833, 895-900, 904-905, 933, 1004, 1051, 1127-1131, 1197-1198, 1219, 1244-1256, 1263, 1294-1295, 1306-1312, 1355, 1370, 1390, 1428-1497
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   206      95  53.88%   66-72, 153, 155, 157, 163-164, 185-193, 199-205, 252-306, 319-355, 505-536, 554, 604-643, 684, 691-695
rest/server.py                                 299     299  0.00%    18-883
rest/stac.py                                   436     145  66.74%   60-62, 214, 272, 294, 366, 403-405, 428, 463-464, 550-595, 640, 648-649, 653-661, 783, 841, 906-908, 925-927, 935-937, 950-952, 966-983, 993-1014, 1024-1046, 1054-1071, 1094-1117, 1127, 1137-1149, 1162-1184, 1198-1243, 1396-1422
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/stac_queryables.py                   32       6  81.25%   28, 55-60, 116
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                          96      31  67.71%   51, 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                              111      36  67.57%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-221, 226, 245, 250, 258, 268
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      43  91.43%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1185-1186, 1204, 1380
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8941    2044  77.14%

Diff against develop

Filename                           Stmts    Miss  Cover
-------------------------------  -------  ------  -------
config.py                             +1       0  +0.02%
api/product/_product.py              -12      -2  +0.05%
api/product/metadata_mapping.py      +24       0  +0.46%
plugins/authentication/token.py       +4       0  +0.88%
plugins/search/qssearch.py            -2       0  -0.05%
TOTAL                                +15      -2  +0.06%

Results for commit: 94cf0ad

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@anesson-cs anesson-cs force-pushed the add-eumetsat-provider branch 2 times, most recently from 7883c7e to f0cd767 Compare March 15, 2024 09:46
@anesson-cs anesson-cs changed the title Add eumetsat provider as a new provider feat: add provider eumetsat Mar 15, 2024
@sbrunato sbrunato marked this pull request as draft March 27, 2024 13:57
@anesson-cs anesson-cs force-pushed the add-eumetsat-provider branch 2 times, most recently from 49e9607 to d9c3a23 Compare March 28, 2024 09:18
@anesson-cs anesson-cs marked this pull request as ready for review March 28, 2024 10:04
docs/getting_started_guide/providers.rst Outdated Show resolved Hide resolved
eodag/resources/user_conf_template.yml Outdated Show resolved Hide resolved
docs/getting_started_guide/register.rst Outdated Show resolved Hide resolved
eodag/api/product/_product.py Outdated Show resolved Hide resolved
@sbrunato sbrunato changed the title feat: add provider eumetsat feat: add provider eumetsat_ds May 16, 2024
@sbrunato sbrunato merged commit 05697ac into develop May 16, 2024
10 checks passed
@sbrunato sbrunato deleted the add-eumetsat-provider branch May 16, 2024 08:40
bjgiraudon pushed a commit that referenced this pull request May 16, 2024
Co-authored-by: Sylvain Brunato <sylvain.brunato@c-s.fr>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add provider EUMETSAT
2 participants