-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
853 opentelemetry metrics for server mode #944
base: develop
Are you sure you want to change the base?
Conversation
Code Coverage (Ubuntu)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 2fa09fc |
Code Coverage (Windows)
Minimum allowed coverage is Generated by 🐒 cobertura-action against 7edfabe |
79d31be
to
8cf83ef
Compare
7afb97b
to
08eee64
Compare
2fa09fc
to
a515210
Compare
a515210
to
0205ce5
Compare
abc6c8e
to
34848a5
Compare
Code Coverage (Ubuntu)
Diff against develop
Results for commit: 96e5a8a Minimum allowed coverage is ♻️ This comment has been updated with latest results |
Code Coverage (Windows)
Diff against develop
Results for commit: 96e5a8a Minimum allowed coverage is ♻️ This comment has been updated with latest results |
This decorator is used by the auto-instrumentation of EODAG
The env var OTEL_EXPORTER_OTLP_ENDPOINT must be set to init the metrics
…-search operation
This reverts commit db5a991.
The attribute can be used to have a list of the providers and product types.
TODO: the callback functions continue to be called after removing the instrumentation
0c83fc7
to
b87329b
Compare
Instructions to run the metrics. Start the OTEL collector:
Set the following environment variables:
Run EODAG in server mode. The metrics are available on |
{{- if .Values.otel.endpoint }} | ||
- name: OTEL_EXPORTER_OTLP_ENDPOINT | ||
value: {{ .Values.otel.endpoint | quote }} | ||
{{- end }} | ||
{{- if .Values.otel.timeout }} | ||
- name: OTEL_EXPORTER_OTLP_TIMEOUT | ||
value: {{ .Values.otel.timeout | quote }} | ||
{{- end }} | ||
{{- if .Values.otel.interval }} | ||
- name: OTEL_METRIC_EXPORT_INTERVAL | ||
value: {{ .Values.otel.interval | quote }} | ||
{{- end }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can simplify that and require OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TIMEOUT and OTEL_METRIC_EXPORT_INTERVAL if otel is enabled. No need to verify parameters individually.
eodag/rest/server.py
Outdated
def telemetry_init(fastapi_app: Optional[FastAPI] = None) -> None: | ||
"""Init telemetry | ||
|
||
:param fastapi_app: FastAPI to automatically instrument. | ||
:type fastapi_app: FastAPI""" | ||
|
||
if not os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT"): | ||
return None | ||
if "opentelemetry" not in sys.modules: | ||
return None | ||
|
||
# Start OTLP exporter | ||
resource = Resource(attributes={SERVICE_NAME: "eodag-serve-rest"}) | ||
# trace | ||
tracer_provider = TracerProvider(resource=resource) | ||
processor = BatchSpanProcessor(OTLPSpanExporter()) | ||
tracer_provider.add_span_processor(processor) | ||
trace.set_tracer_provider(tracer_provider) | ||
# metrics | ||
reader = PeriodicExportingMetricReader(OTLPMetricExporter()) | ||
view_histograms: View = View( | ||
instrument_type=Histogram, | ||
aggregation=ExplicitBucketHistogramAggregation( | ||
boundaries=( | ||
0.25, | ||
0.50, | ||
0.75, | ||
1.0, | ||
1.5, | ||
2.0, | ||
3.0, | ||
4.0, | ||
5.0, | ||
6.0, | ||
7.0, | ||
8.0, | ||
9.0, | ||
10.0, | ||
) | ||
), | ||
) | ||
view_overhead_histograms: View = View( | ||
instrument_type=Histogram, | ||
instrument_name="*overhead*", | ||
aggregation=ExplicitBucketHistogramAggregation( | ||
boundaries=( | ||
0.030, | ||
0.040, | ||
0.050, | ||
0.060, | ||
0.070, | ||
0.080, | ||
0.090, | ||
0.100, | ||
0.125, | ||
0.150, | ||
0.175, | ||
0.200, | ||
0.250, | ||
0.500, | ||
) | ||
), | ||
) | ||
meter_provider = MeterProvider( | ||
resource=resource, | ||
metric_readers=[reader], | ||
views=( | ||
view_histograms, | ||
view_overhead_histograms, | ||
), | ||
) | ||
metrics.set_meter_provider(meter_provider) | ||
|
||
# Auto instrumentation | ||
if fastapi_app: | ||
logger.debug("Instrument FastAPI app") | ||
FastAPIInstrumentor.instrument_app( | ||
app=fastapi_app, | ||
tracer_provider=tracer_provider, | ||
meter_provider=meter_provider, | ||
) | ||
logger.debug("Instrument EODAG app") | ||
|
||
from eodag.utils.instrumentation.eodag import EODAGInstrumentor | ||
|
||
EODAGInstrumentor(eodag_api).instrument( | ||
tracer_provider=tracer_provider, | ||
meter_provider=meter_provider, | ||
) | ||
return None | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this function to a separate file
# -*- coding: utf-8 -*- | ||
# Copyright 2023, CS GROUP - France, https://www.csgroup.eu/ | ||
# | ||
# This file is part of EODAG project | ||
# https://www.github.com/CS-SI/EODAG | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
"""OpenTelemetry auto-instrumentation packages""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be removed, we take the instrumentation from https://github.com/CS-SI/opentelemetry-instrumentation-eodag
:rtype: Iterable[Observation] | ||
""" | ||
new_available_product_types: List[str] = [ | ||
p["ID"] for p in self._eodag_api.list_product_types() | ||
] | ||
observations_dict: Dict[str, int] = { | ||
p: 0 for p in self._last_available_product_types | ||
} | ||
for p in new_available_product_types: | ||
observations_dict[p] = 1 | ||
self._last_available_product_types = new_available_product_types | ||
observations = [ | ||
Observation( | ||
v, | ||
{"product_type_id": k}, | ||
) | ||
for k, v in observations_dict.items() | ||
] | ||
return observations | ||
|
||
def _instrument(self, **kwargs) -> None: | ||
"""Instruments EODAG""" | ||
tracer_provider = kwargs.get("tracer_provider") | ||
tracer = get_tracer(__name__, tracer_provider=tracer_provider) | ||
meter_provider = kwargs.get("meter_provider") | ||
meter = get_meter(__name__, meter_provider=meter_provider) | ||
|
||
if self._eodag_api: | ||
meter.create_observable_gauge( | ||
name="eodag.core.available_providers", | ||
callbacks=[self._available_providers_callback], | ||
description="The number available providers", | ||
) | ||
meter.create_observable_gauge( | ||
name="eodag.core.available_product_types", | ||
callbacks=[self._available_product_types_callback], | ||
description="The number available product types", | ||
) | ||
|
||
request_duration_seconds = meter.create_histogram( | ||
name="eodag.server.request_duration_seconds", | ||
unit="s", | ||
description="Measures the duration of the inbound HTTP request", | ||
) | ||
outbound_request_duration_seconds = meter.create_histogram( | ||
name="eodag.core.outbound_request_duration_seconds", | ||
unit="s", | ||
description="Measure the duration of the outbound HTTP request", | ||
) | ||
request_overhead_duration_seconds = meter.create_histogram( | ||
name="eodag.server.request_overhead_duration_seconds", | ||
unit="s", | ||
description="Measure the duration of the EODAG overhead on the inbound HTTP request", | ||
) | ||
|
||
downloaded_data_counter = meter.create_counter( | ||
name="eodag.download.downloaded_data_bytes_total", | ||
description="Measure data downloaded from each provider and product type", | ||
) | ||
_instrument_download( | ||
tracer, | ||
downloaded_data_counter, | ||
request_duration_seconds, | ||
outbound_request_duration_seconds, | ||
request_overhead_duration_seconds, | ||
) | ||
|
||
searched_product_types_counter = meter.create_counter( | ||
name="eodag.core.searched_product_types_total", | ||
description="The number of searches by provider and product type", | ||
) | ||
_instrument_search( | ||
tracer, | ||
searched_product_types_counter, | ||
request_duration_seconds, | ||
outbound_request_duration_seconds, | ||
request_overhead_duration_seconds, | ||
) | ||
|
||
def _uninstrument(self, **kwargs) -> None: | ||
"""Uninstrument the library. | ||
|
||
This only works if no other module also patches eodag""" | ||
from eodag.rest import server | ||
|
||
patches = [ | ||
(server, "search_stac_items"), | ||
(server, "download_stac_item"), | ||
(QueryStringSearch, "_request"), | ||
(Download, "progress_callback_decorator"), | ||
] | ||
for p in patches: | ||
instr_func = getattr(p[0], p[1]) | ||
if not getattr( | ||
instr_func, | ||
"opentelemetry_instrumentation_eodag_applied", | ||
False, | ||
): | ||
continue | ||
setattr(p[0], p[1], instr_func.__wrapped__) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this as well
# -*- coding: utf-8 -*- | ||
# Copyright 2023, CS GROUP - France, https://www.csgroup.eu/ | ||
# | ||
# This file is part of EODAG project | ||
# https://www.github.com/CS-SI/EODAG | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
_instruments = ("eodag ~= 2.0",) | ||
|
||
_supports_metrics = True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
remove this as well
ae8bba6
to
96e5a8a
Compare
Implemented OpenTelemetry metrics with Prometheus exporter.
Metrics implemented: