Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat[server]: caching LRU + redis #1073

Open
wants to merge 17 commits into
base: develop
Choose a base branch
from

Conversation

alambare-csgroup
Copy link
Collaborator

  • feat: config management for server mode
  • feat: caching of /collections, /catalogs, /conformance and /queryables with a unique local LRU cache
  • feat: caching of search_stac_items in redis
  • feat: use of cached products while downloading
  • fix: 404 error when product not found in collection or catalog
  • refactor: init EOProduct with get_geometry_from_various

Copy link
Contributor

Test Results

    4 files  ±0      4 suites  ±0   5m 57s ⏱️ -6s
  506 tests  - 1    503 ✅  - 1   3 💤 ±0  0 ❌ ±0 
2 024 runs   - 4  1 938 ✅  - 4  86 💤 ±0  0 ❌ ±0 

Results for commit 070126d. ± Comparison against base commit 2ef1ad1.

This pull request removes 1 test.
tests.units.test_stac_core.TestStacCore ‑ test_get_product_types

Copy link
Contributor

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      315      27  91.43%   80-82, 91, 99, 103-105, 178, 190, 388-390, 454-457, 504-505, 514-515, 594, 663-668, 670
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    731      76  89.60%   86-95, 368, 581, 625-628, 666, 771, 775-780, 806, 876, 946, 1063-1068, 1154-1166, 1206, 1208, 1212, 1233-1235, 1239-1250, 1263-1269, 1359-1362, 1391-1411, 1459, 1465-1468, 1477, 1843, 1876-1882, 2147, 2151-2154, 2168-2170, 2205
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       6  86.36%   27-29, 79, 119, 129
api/product/_product.py                        193      27  86.01%   48-55, 59-61, 231-232, 322, 358, 419, 433-436, 449, 473-476, 519-525
api/product/metadata_mapping.py                649      84  87.06%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 348-360, 401-404, 441, 462-465, 488, 496-497, 570-571, 595-596, 602-605, 620-621, 770, 816, 889-890, 933-938, 1069, 1083-1103, 1123, 1128, 1238, 1260, 1274, 1287-1306, 1345, 1397, 1435-1439, 1458
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             127      14  88.98%   49-51, 95-100, 146, 185, 205, 231, 270-271
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/cds.py                            206      49  76.21%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 487, 515, 517
plugins/apis/ecmwf.py                           99      15  84.85%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 81      13  83.95%   35-37, 86, 99, 101, 114-117, 165-168
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        491     165  66.40%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 458-459, 465-469, 502, 537, 541, 548, 578-586, 590, 628-636, 643-645, 686-760, 778-839, 850-855, 871-884, 913, 928-930, 933, 943-951, 959-972, 982-1001, 1008-1020, 1061, 1087, 1132-1134, 1354
plugins/download/base.py                       261      57  78.16%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       435     102  76.55%   81-88, 122, 171, 192-199, 209-222, 262-267, 291, 293, 365, 392-394, 404, 410, 412, 434, 473-477, 534, 607-663, 677, 702-711, 737-738, 746, 749, 765, 788, 796-801, 819-820, 827, 888-894, 949-950, 956, 966, 1032, 1050-1066
plugins/download/s3rest.py                     117      27  76.92%   55-58, 124, 165, 199, 229-236, 239-241, 245, 258-264, 272-273, 276-280, 303, 324-327
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          198      65  67.17%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     560      65  88.39%   89, 361-367, 375-376, 483-489, 544-547, 620-621, 662, 680, 695, 748, 769, 772-773, 782, 793, 802, 825, 885-890, 894-895, 923, 994, 1013-1030, 1069, 1145-1146, 1210-1211, 1232, 1257, 1263, 1294-1295, 1305-1311, 1354, 1368, 1388, 1398, 1478
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 1       0  100.00%
rest/cache.py                                   97      38  60.82%   37-49, 61-63, 67, 76, 98-109, 113, 128-144, 148, 164-167, 171
rest/config.py                                  21       0  100.00%
rest/constants.py                                7       0  100.00%
rest/core.py                                   208      13  93.75%   267, 274, 295-297, 307-309, 436, 663, 670-674
rest/server.py                                 281      54  80.78%   81-82, 106, 129-131, 242-244, 260, 300-301, 313-329, 408-410, 437, 590-597, 627, 670-671, 693, 764-766, 783-788, 816, 818, 822-823, 827-828
rest/stac.py                                   412     111  73.06%   59-61, 219-221, 263, 276-285, 307-313, 332-335, 354-376, 381-387, 410-412, 435, 470-471, 499, 512, 557-603, 777, 845-847, 1066, 1076-1088, 1101-1123, 1137-1182, 1336-1337
rest/types/__init__.py                           1       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/queryables.py                        50       2  96.00%   37, 151
rest/types/stac_search.py                      128      10  92.19%   50-53, 178, 193-195, 203, 207
rest/utils/__init__.py                         116      14  87.93%   53, 108-109, 128-130, 180, 190-204, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              495      41  91.72%   83, 88, 109-111, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1182, 1200
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8854    1635  81.53%

Diff against develop

Filename                            Stmts    Miss  Cover
--------------------------------  -------  ------  --------
api/product/_assets.py                  0      +1  -2.28%
api/product/_product.py               -15      -5  +1.39%
plugins/base.py                         0      +1  -4.35%
plugins/authentication/base.py          0      +1  -5.26%
rest/cache.py                         +97     +38  +60.82%
rest/config.py                        +21       0  +100.00%
rest/constants.py                      +7       0  +100.00%
rest/core.py                          +39      +3  -0.33%
rest/server.py                        -17       0  -1.10%
rest/stac.py                          -24     +16  -5.15%
rest/types/collections_search.py      +13     +13  +100.00%
rest/types/queryables.py              +50      +2  +96.00%
rest/types/stac_search.py              +6       0  +0.39%
utils/__init__.py                       0      -2  +0.41%
TOTAL                                +177     +68  -0.45%

Results for commit: 070126d

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      315      28  91.11%   80-82, 91, 99, 103-105, 178, 190, 388-390, 454-457, 504-505, 514-515, 594, 628, 663-668, 670
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    731      76  89.60%   86-95, 368, 581, 625-628, 666, 771, 775-780, 806, 876, 946, 1063-1068, 1154-1166, 1206, 1208, 1212, 1233-1235, 1239-1250, 1263-1269, 1359-1362, 1391-1411, 1459, 1465-1468, 1477, 1843, 1876-1882, 2147, 2151-2154, 2168-2170, 2205
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          5       0  100.00%
api/product/_assets.py                          44       6  86.36%   27-29, 79, 119, 129
api/product/_product.py                        193      27  86.01%   48-55, 59-61, 231-232, 322, 358, 419, 433-436, 449, 473-476, 519-525
api/product/metadata_mapping.py                649      85  86.90%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 348-360, 401-404, 441, 462-465, 488, 496-497, 570-571, 595-596, 602-605, 620-621, 770, 816, 889-890, 933-938, 1069, 1083-1103, 1123, 1128, 1238, 1260, 1274, 1287-1306, 1345, 1397, 1420, 1435-1439, 1458
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             127      14  88.98%   49-51, 95-100, 146, 185, 205, 231, 270-271
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/cds.py                            206      56  72.82%   72-80, 126-129, 176-188, 244, 309, 317-322, 335-342, 360-362, 383-385, 395-397, 411-412, 457, 460, 470, 487, 507-522
plugins/apis/ecmwf.py                           99      15  84.85%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 81      13  83.95%   35-37, 86, 99, 101, 114-117, 165-168
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        491     165  66.40%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 458-459, 465-469, 502, 537, 541, 548, 578-586, 590, 628-636, 643-645, 686-760, 778-839, 850-855, 871-884, 913, 928-930, 933, 943-951, 959-972, 982-1001, 1008-1020, 1061, 1087, 1132-1134, 1354
plugins/download/base.py                       261      59  77.39%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       435     102  76.55%   81-88, 122, 171, 192-199, 209-222, 262-267, 291, 293, 365, 392-394, 404, 410, 412, 434, 473-477, 534, 607-663, 677, 702-711, 737-738, 746, 749, 765, 788, 796-801, 819-820, 827, 888-894, 949-950, 956, 966, 1032, 1050-1066
plugins/download/s3rest.py                     117      27  76.92%   55-58, 124, 165, 199, 229-236, 239-241, 245, 258-264, 272-273, 276-280, 303, 324-327
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py           56       6  89.29%   67, 112-113, 121, 132, 176
plugins/search/creodias_s3.py                   55       3  94.55%   56, 74, 108
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          198      65  67.17%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     560      89  84.11%   89, 361-367, 375-376, 483-489, 544-547, 620-621, 662, 680, 695, 748, 769, 772-773, 782, 793, 802, 825, 885-890, 894-895, 923, 994, 1013-1030, 1069, 1145-1146, 1210-1211, 1232, 1257, 1263, 1294-1295, 1305-1311, 1354, 1368, 1388, 1398, 1438-1507
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 1       0  100.00%
rest/cache.py                                   97      82  15.46%   37-49, 56-78, 90-150, 157-172
rest/config.py                                  21       1  95.24%   54
rest/constants.py                                7       0  100.00%
rest/core.py                                   208     126  39.42%   150-237, 261-327, 367-405, 425-441, 464-475, 484-520, 539, 585-624, 663, 670-674
rest/server.py                                 281     281  0.00%    18-839
rest/stac.py                                   412     350  15.05%   59-61, 93-99, 107-130, 183, 203-319, 329-388, 406-445, 461-514, 524-547, 557-603, 629, 652-726, 755-780, 788-804, 812-817, 827-854, 864-866, 874-876, 889-891, 905-922, 932-953, 963-985, 993-1010, 1033-1056, 1066, 1076-1088, 1101-1123, 1137-1182, 1190-1207, 1218-1367
rest/types/__init__.py                           1       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     185      19  89.73%   52-55, 126, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/queryables.py                        50      12  76.00%   37, 50-51, 57-58, 64-65, 90-95, 151
rest/types/stac_search.py                      128      22  82.81%   50-53, 130-132, 156-157, 162-163, 178, 193-195, 203, 207, 250-255
rest/utils/__init__.py                         116      31  73.28%   53, 79-85, 105, 108-109, 128-130, 147, 173-181, 188-209, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       6  80.65%   70, 73-74, 78, 90, 92
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
utils/__init__.py                              495      43  91.31%   83, 88, 109-111, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1051, 1182, 1200, 1375
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8854    2378  73.14%

Diff against develop

Filename                            Stmts    Miss  Cover
--------------------------------  -------  ------  --------
api/product/_assets.py                  0      +1  -2.28%
api/product/_product.py               -15      -5  +1.39%
rest/cache.py                         +97     +82  +15.46%
rest/config.py                        +21      +1  +95.24%
rest/constants.py                      +7       0  +100.00%
rest/core.py                          +39     +65  -24.49%
rest/server.py                        -17     -17  +100.00%
rest/stac.py                          -24    +196  -49.63%
rest/types/collections_search.py      +13     +13  +100.00%
rest/types/eodag_search.py              0      +1  -0.54%
rest/types/queryables.py              +50     +12  +76.00%
rest/types/stac_search.py              +6     +10  -7.35%
rest/utils/rfc3339.py                   0      +1  -3.22%
TOTAL                                +177    +360  -3.61%

Results for commit: 070126d

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

avoid search by id before every server mode download
1 participant