feat: get queryables for wekeo with a constraints file #1104

anesson-cs · 2024-04-18T18:06:16Z

It is possible to get queryables of wekeo using a constraints file.

discover_queryables() is now also available for QueryStringSearch search plugin.

Now, we are using product type specific metadata mapping instead of general provider one to add queryables after getting the ones given by the provider file if exists.

Note:

common queryables (productType, id, start, end, geom) are overwritten by eodag CommonQueryables model. We may prevent these querybales from being overwritten if they are given by queryables or constraints file.
for now, when there is a intersection between queryables from several providers, the ones of the provider with the highest priority are taken. In that case, we may lose some information (possible values or default for instance) so we may study a more optimal way to deal with queryables intersections.

github-actions · 2024-04-18T18:10:06Z

Test Results

4 files ± 0 4 suites ±0 5m 12s ⏱️ ±0s
536 tests + 3 533 ✅ +3 3 💤 ±0 0 ❌ ±0
2 144 runs +12 2 046 ✅ +8 98 💤 +4 0 ❌ ±0

Results for commit 8988a3c. ± Comparison against base commit be8f361.

♻️ This comment has been updated with latest results.

github-actions · 2024-04-18T18:10:16Z

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 646-692, 794-845, 849
config.py                                      345      27  92.17%   81-83, 92, 100, 104-106, 179, 191, 448-450, 514-517, 564-565, 574-575, 654, 723-728, 730
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    739      76  89.72%   88-97, 371, 598, 642-645, 683, 788, 792-797, 823, 893, 970, 1083, 1173-1185, 1225, 1227, 1231, 1254, 1258-1269, 1282-1288, 1378-1381, 1410-1430, 1487, 1504-1507, 1519-1522, 1890, 1923-1929, 2194, 2198-2201, 2215-2217, 2252
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      84  87.59%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             132      14  89.39%   49-51, 105-110, 156, 195, 215, 241, 280-281
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      57  78.08%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         132      13  90.15%   49-54, 108, 112, 125, 275, 295, 354-355, 375, 384
plugins/search/build_search_result.py          179      24  86.59%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 527, 529
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 278-281, 289-300, 317, 319, 326-327, 329-330, 348-352, 385, 392, 403, 416, 422-434, 439
plugins/search/qssearch.py                     606      67  88.94%   101, 373-379, 387-388, 495-501, 549, 565, 575, 602, 604, 654-657, 730-731, 772, 791, 806, 859, 880, 883-884, 893, 904, 913, 936, 998-1003, 1007-1008, 1036, 1107, 1154, 1230-1234, 1300-1301, 1322, 1347-1359, 1366, 1397-1398, 1409-1415, 1458, 1473, 1493, 1594-1604
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   207      17  91.79%   66-72, 261, 345, 605, 607, 610-612, 684, 691-696
rest/server.py                                 299      53  82.27%   80-81, 107, 130-131, 244-246, 302-303, 315-331, 423-428, 460, 627-634, 667, 711-712, 803-805, 822-827, 858, 860, 864-865, 869-870
rest/stac.py                                   464      90  80.60%   64-66, 276, 298, 370, 407-409, 432, 467-468, 554-599, 636-641, 672, 692-693, 886, 951-953, 1172, 1182-1194, 1207-1229, 1243-1288, 1447-1448
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/stac_queryables.py                   32       2  93.75%   28, 116
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                          96      13  86.46%   51, 108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                              111      11  90.09%   53, 70, 129-132, 199, 214-218, 245, 258
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      42  91.63%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1186, 1204, 1380
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         9084    1562  82.80%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  -------
config.py                        +2       0  +0.04%
plugins/search/base.py           -1      -1  +0.68%
plugins/search/qssearch.py      +47      +9  -0.68%
rest/server.py                    0      -1  +0.33%
utils/constraints.py             +1      -1  +1.11%
TOTAL                           +49      +6  +0.02%

Results for commit: 8988a3c

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

github-actions · 2024-04-18T18:10:26Z

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         303      51  83.17%   60, 646-692, 794-845, 849
config.py                                      345      28  91.88%   81-83, 92, 100, 104-106, 179, 191, 448-450, 514-517, 564-565, 574-575, 654, 688, 723-728, 730
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    739      76  89.72%   88-97, 371, 598, 642-645, 683, 788, 792-797, 823, 893, 970, 1083, 1173-1185, 1225, 1227, 1231, 1254, 1258-1269, 1282-1288, 1378-1381, 1410-1430, 1487, 1504-1507, 1519-1522, 1890, 1923-1929, 2194, 2198-2201, 2215-2217, 2252
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        203      32  84.24%   52-59, 63-65, 169-176, 260-261, 343, 372, 433, 447-450, 463, 487-490, 533-539
api/product/metadata_mapping.py                677      84  87.59%   67-69, 130-132, 233, 265-266, 312-313, 323-335, 337, 348, 354-366, 407-410, 447, 468-471, 494, 502-503, 579-580, 604-605, 611-614, 629-630, 779, 825, 999-1004, 1135, 1149-1169, 1189, 1194, 1304, 1326, 1340, 1353-1372, 1411, 1463, 1501-1505, 1524
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             132      14  89.39%   49-51, 105-110, 156, 195, 215, 241, 280-281
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           98      15  84.69%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       117      11  90.60%   40-42, 153, 159, 172-173, 184-189
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 90      17  81.11%   35-37, 79, 107, 109, 131-143, 198-201
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        492     165  66.46%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 459-460, 466-470, 503, 538, 542, 549, 579-587, 591, 629-637, 644-646, 687-761, 779-840, 851-856, 872-885, 914, 929-931, 934, 944-952, 960-973, 983-1002, 1009-1021, 1062, 1088, 1133-1135, 1355
plugins/download/base.py                       260      59  77.31%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 688, 710, 718
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       529     132  75.05%   85-91, 124, 203-213, 215-216, 251-254, 318-321, 323-324, 331-336, 354-369, 386-388, 400, 448, 455-461, 479, 493, 507, 515-517, 533-538, 549, 567, 609-613, 635, 675, 720, 734-740, 776-837, 855, 888-897, 923-924, 951-956, 962, 978, 995-996, 1023-1024, 1031, 1093-1099, 1154-1155, 1161, 1171, 1237, 1241, 1255-1271
plugins/download/s3rest.py                     118      27  77.12%   55-58, 121, 162, 197, 227-234, 237-239, 243, 256-262, 270-271, 274-278, 301, 322-325
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         132      18  86.36%   49-54, 108, 112, 125, 275, 295, 354-355, 375, 378-386, 388
plugins/search/build_search_result.py          179      31  82.68%   62, 97, 142-143, 149, 160, 296-299, 333, 390-407, 469, 472, 482, 499, 519-534
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          197      65  67.01%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 278-281, 289-300, 317, 319, 326-327, 329-330, 348-352, 385, 392, 403, 416, 422-434, 439
plugins/search/qssearch.py                     606      95  84.32%   101, 373-379, 387-388, 495-501, 549, 552, 565, 575, 594-609, 654-657, 730-731, 772, 791, 806, 859, 880, 883-884, 893, 904, 913, 936, 998-1003, 1007-1008, 1036, 1107, 1154, 1230-1234, 1300-1301, 1322, 1347-1359, 1366, 1397-1398, 1409-1415, 1458, 1473, 1493, 1555-1624
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 5       2  60.00%   21-22
rest/core.py                                   207      96  53.62%   66-72, 153, 155, 157, 163-164, 185-193, 199-205, 252-306, 319-355, 505-536, 554, 604-643, 684, 691-696
rest/server.py                                 299     299  0.00%    18-883
rest/stac.py                                   464     149  67.89%   64-66, 218, 276, 298, 370, 407-409, 432, 467-468, 554-599, 636-641, 672, 680-681, 685-693, 828, 886, 951-953, 970-972, 980-982, 995-997, 1011-1028, 1038-1059, 1069-1091, 1099-1116, 1139-1162, 1172, 1182-1194, 1207-1229, 1243-1288, 1441-1467
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/stac_queryables.py                   32       6  81.25%   28, 55-60, 116
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                          96      31  67.71%   51, 79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                              111      36  67.57%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-221, 226, 245, 250, 258, 268
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              502      43  91.43%   84, 89, 110-112, 193-194, 203-230, 233, 247, 329-333, 409-413, 434-436, 518, 533, 571-572, 968-971, 979-980, 1021-1022, 1102, 1185-1186, 1204, 1380
utils/constraints.py                           119      37  68.91%   94-103, 144, 149, 153, 164, 190-192, 202, 216-232, 241-252
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/requests.py                               55      11  80.00%   69, 96, 98, 100, 102, 104, 123, 128-130, 138
utils/rest.py                                   28       1  96.43%   49
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         9084    2076  77.15%

Diff against develop

Filename                           Stmts    Miss  Cover
-------------------------------  -------  ------  -------
config.py                             +2       0  +0.04%
api/product/metadata_mapping.py        0      -1  +0.15%
plugins/search/base.py                -1      +3  -2.36%
plugins/search/qssearch.py           +47     +13  -1.01%
utils/constraints.py                  +1      -1  +1.11%
TOTAL                                +49     +14  -0.03%

Results for commit: 8988a3c

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

…e configuration with wekeo to make its queryables discovery works

…e of the provider

this check was unjustified

…ryable value if no kwarg was found

sbrunato marked this pull request as draft April 22, 2024 08:25

sbrunato assigned anesson-cs Apr 22, 2024

anesson-cs force-pushed the wekeo2-constraints-file branch from bbe3e30 to d677422 Compare May 4, 2024 13:21

anesson-cs added 3 commits May 24, 2024 11:56

feat: get queryables for wekeo with a constraints file

e4383a4

fix: change a default parameter of 'SEASONAL_ORIGINAL_SL' product typ…

ca11a4f

…e configuration with wekeo to make its queryables discovery works

test: fix and add tests on queryables

f6c181c

anesson-cs force-pushed the wekeo2-constraints-file branch from d677422 to 7b158d2 Compare May 24, 2024 10:19

anesson-cs added 8 commits May 24, 2024 14:43

fix: use metadata mapping of the product type instead of the whole on…

b4cc84a

…e of the provider

fix: remove authentication verification before queryables discovery

69e865b

this check was unjustified

fix: try to use values from the provider configuration as default que…

7283307

…ryable value if no kwarg was found

test: fix queryables search plugin tests

649f073

refactor: use get_metadata_mapping() method from search plugin

01a4351

test: make tests on constraints discovery works again after rebase

08e47ae

fix: remove libraries not used after the rebase anymore

28ce437

test: fix a mock issue due to the rebase

8988a3c

anesson-cs force-pushed the wekeo2-constraints-file branch from c7e0a05 to 8988a3c Compare May 24, 2024 12:46

anesson-cs marked this pull request as ready for review May 24, 2024 12:51

sbrunato approved these changes May 24, 2024

View reviewed changes

sbrunato merged commit 3bcdd79 into develop May 24, 2024
10 checks passed

sbrunato deleted the wekeo2-constraints-file branch May 24, 2024 17:18

sbrunato mentioned this pull request May 24, 2024

queryables refactor #1161

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: get queryables for wekeo with a constraints file #1104

feat: get queryables for wekeo with a constraints file #1104

anesson-cs commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

feat: get queryables for wekeo with a constraints file #1104

feat: get queryables for wekeo with a constraints file #1104

Conversation

anesson-cs commented Apr 18, 2024 • edited

github-actions bot commented Apr 18, 2024 • edited

Test Results

github-actions bot commented Apr 18, 2024 • edited

Code Coverage (Ubuntu)

Diff against develop

github-actions bot commented Apr 18, 2024 • edited

Code Coverage (Windows)

Diff against develop

anesson-cs commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited

github-actions bot commented Apr 18, 2024 •

edited