add action workflow to update citation.cff when doing a release

[vuillaut]

Description
Introduces an action to update CITATION.cff release date when doing a new release in order to solve #3953

Dear reviewer

I am not sure either this is the solution gammapy team would prefer to solve that issue, nor if it would actually work, so some feedback would be welcome.

Potential issue with that PR:

• if that action breaks for some reason (e.g. invalid CITATION.cff), it will break the tag release
• the codemeta conversion action should come after
• I am not actually sure if the action are run before Zenodo fetch the new tag and produces a new record

[codecov]

Codecov Report

Merging #3954 (61aee48) into master (109931f) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3954   +/-   ##
=======================================
  Coverage   93.98%   93.98%           
=======================================
  Files         162      162           
  Lines       21354    21354           
=======================================
  Hits        20070    20070           
  Misses       1284     1284           

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[bkhelifi]

Dear all,
The citation scheme will be discussed during the next CC meeting. Maybe one can wait for the outcome before making a merge...

[adonath]

I think some solution like this should be in place for v1.0, so I put the v1.0 milestone here...

[adonath]

Thanks @vuillaut! What is not clear to me is, whether it make sense to have that as an action that runs on pushing a tag. This basically introduces the change after the tag, however shouldn't it be included with the tag? So the citationcff.py should be run manually before tagging?

[bkhelifi]

Indeed, there are 3 cases (for the work after the V1.0):

• Just after a feature release, the CITATION.cff should be started from scratch, with global information coming from a TEMPLATE. The first merged PR could lead to the addition of the first author
• Just after a bug release, the CITATION.cff should not erased but appended.
• In the other case, an append can be done.
  Can a CI know that? A very good script should be set up for that...
  Do you share this view?

[adonath]

@bkhelifi No, I think independent of any version number the script should just always create a new CITATION.cff following the policy in PIG 24. Whether the author list is "appended" or not follows completely from the git history, there is no need to append an existing CITATION.cff ever. The info can always be created from the git history and the AUTHORS_ALL.yml file...

What is less clear to me is when the update to the CITATION.cff should happen. The options are:

• "Manually" before the tag using dev/prepapre-release.py. Then all the info is updated and included for the next version. However this requires "reserving" a DOI in Zenodo and then putting it in the CITATION.cff. In this case we do not need a CI as proposed in this PR. We just need the update_citation_cff.py script, which is run manually before defining the release tag.
• After the tag e.g. using a CI, but then the CITATION.cff is always lagging and one version behind.
• Remove all version specific info from the CITATION.cff, I guess mostly the identifiers section with the version specific DOIs.

[bkhelifi]

Agree... as long as we follow the PIG24.
And this script should build at the same time codemeta.json, built on the same list of authors as the one of citation.cff and contains extra information that are important.

[vuillaut]

Hi @adonath

I have thought about this, let me try to explain:

Here the current situation with the metadata and the release process:

Metadata are partially handled by gammapy (CITATION.cff, codemeta.json).
Other metadata directly come from GitHub (repo name, URLs, release version...).

This is more flexible, lighter and less error prone for the Zenodo publication, because the GitHub-Zenodo integration is doing a lot for you in the background, and if the CITATION.cff is not readable, Zenodo will just use whatever comes from GitHub.

If you want to do the publication to Zenodo yourself (required if you want to prereserve a DOI and update your metadata before upload), you then need to have full control over your metadata. If the metadata payload is not exactly what Zenodo expects, the publication will fail. However, this gives you more control - so I let you decide the benefit / risk balance.

The process might then look like this:

To handle the publication to Zenodo, you need:

• a Zenodo token (created by the Zenodo account owning gammapy)
• create the metadata payload for Zenodo

To handle the upload, you may use the ESCAPE eossr library and then code would look like this:

from eossr.api import zenodo
zen = zenodo.ZenodoAPI(access_token=token)
deposit = zen.create_new_entry()
doi = deposit.json()['metadata']['prereserve_doi']['doi']
record_id = deposit.json()['record_id']
gammapy_metadata['doi'] = doi
zen.upload_dir_content(path_to_gammapy_dir, record_id, gammapy_metadata)

The gammapy_metadata should be a dictionary following Zenodo expected metadata schema.
Taking advantage of the ESCAPE environment, you can read this metadata directly from your codemeta.json (given that it is correctly written):

import json
from eossr.metadata import codemeta2zenodo
gammapy_metadata = codemeta2zenodo.converter(json.load(open('codemeta.json')))

You can finish the publication in the script, or later manually in Zenodo API:

zen.publish_entry(record_id)

Note that in this workflow, CITATION.cff is not used anymore to handle the upload, the source of truth is the codemeta.json file.

[vuillaut]

Interestingly, I opened an issue several months ago already to update the codemeta.json file with the prereserved DOI: https://gitlab.in2p3.fr/escape2020/wp3/eossr/-/issues/64

That would simplify your life, simply using eossr CLI in gammay CI when doing a release:

eossr-upload-repository --token $TOKEN --input-dir . 

If this is what you would like to have as a workflow, I could work on it. However, I think that would mean dropping CITATION.cff entirely, since eossr does not handle that format.

[bkhelifi]

Does this code be able to use a template to fill other fields? like this one: https://github.com/bkhelifi/gammapy/blob/Authors/dev/FOOTER_CODEMETA.json

[vuillaut]

Does this code be able to use a template to fill other fields? like this one: https://github.com/bkhelifi/gammapy/blob/Authors/dev/FOOTER_CODEMETA.json

Yes, you would just

1. load the base metadata from what you are calling here FOOTER_CODEMETA.json
2. update that metadata with whatever information (here probably release version, DOI, publication date...)
3. write codemeta.json
4. do the rest as proposed up there

[adonath]

Thanks @vuillaut! I included the release date correction in #4137 as a step to do before a tag. For v1.0 we will not get a fully automatized creation o the meta data so I'll keep the PR open for discussion on solving for the DOI pre-registration etc.