Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CSW Harvester: Inconsistent and Incorrect Metrics under certain conditions #8039

Open
tobias-hotz opened this issue May 14, 2024 · 0 comments · May be fixed by #8069
Open

CSW Harvester: Inconsistent and Incorrect Metrics under certain conditions #8039

tobias-hotz opened this issue May 14, 2024 · 0 comments · May be fixed by #8069

Comments

@tobias-hotz
Copy link

Describe the bug
The CSW harvester of the geonetwork sometimes increments two metrics for a single record. This is not desired and does not happen in other harvesters AFAICT.
This can happen if a new or existing record cannot be retrived, validated or is a duplicate. In both addMetadata and updatingLocalMetadata, the function retriveMetadata is called.


This function increments one of the metrics and returns null if retrival or validation fails or if the record is a duplicate.
The error is that the calling functions also increment either unretrievable in case of new metadata or unchanged in case of updating existing metadata when null is returned, which causes the erronious behavior:
private void addMetadata(RecordInfo ri, String uuidToAssign) throws Exception {
if (cancelMonitor.get()) {
return;
}
Element md = retrieveMetadata(ri.uuid);
if (md == null) {
result.unretrievable++;
return;
}

boolean updatingLocalMetadata(RecordInfo ri, String id, Boolean force) throws Exception {
Element md = retrieveMetadata(ri.uuid);
if (md == null) {
result.unchangedMetadata++;
return false;
}

The solution seems to just not increment any counter if retriveMetadata return null

To Reproduce
Steps to reproduce the behavior:

  1. Add a harvester that harvests a CSW with known-unretrivable records or records that dont validate
  2. Run the harvester
  3. Notice that the number of total records is smaller than all added + doesNotValidate + unretrievable

Expected behavior
A single record only increments the number of total records and one of the other status metrics.

Screenshots
Screenshot Harvester Metrics Mismatch
The count of all other metrics besides total combined in this screenshot is 2969, which is 4 entries more than should be possible

Desktop (please complete the following information):

  • GeoNetwork Version 4.2.5 (but latest main branch affected as well)

Additional context
This bug makes the metrics way less useful, as you cannot say for certain what errors occured and what is just a byproduct of the current behaviour, which makes automated monitoring of these metrics impossible.

josegar74 added a commit to GeoCat/core-geonetwork that referenced this issue May 19, 2024
…rtain conditions.

Retrieve metadata method increments metrics under certain conditions and returns null. Methods calling the retrieve metadata method increments additionally the unretrievable metric if null is returned. Sonarlint fixeas are already applied. Fixes geonetwork#8039
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant