PIG 17 - Provenance capture #2458
Conversation
There was a problem hiding this comment.
@Bultako - Thanks!
Is this something you want to implement within the next ~ month, i.e. for v1.0?
Or is the plan what to implement next year?
We're short on time for v1.0, so we need to decide whether to put time into reviewing and discussing this now, or later.
docs/development/pigs/pig-017.rst
Outdated
| Task list | ||
| ========= | ||
|
|
||
| - General cleaning of Gammapy code-base to replace ``print()`` functions with standard Python ``logging``. |
There was a problem hiding this comment.
You can remove this, we already use logging consistently instead of print. The places where we use print is for cases where we explicitly write information to the user screen, e.g. when they print out the information for a catalog source, i.e. someone explicitly calls print(source_catalogs["hgps"]["crab"]). Is it clear that those are cases where we in the library should call print instead of log, or does this need discussion?
I agree we are short on time for v1.0 Nevertheless, there are some open issues related with provenance, and I'd like those discussions to be aware of this PIG. |
|
Discussed in Gammapy CC call just now. Agreed to table this PIG and work for now to allow us to focus on other things for v1.0, and then to discuss and work on this in early 2020. |
|
I discussed this with @cboisson, it would be good to keep an eye on this implementation regularly as it is already advanced prototyping work, and it is not too intrusive or conflicting with other gammapy features. Though there is capture and inspection of provenance, I think gammapy is mainly concerned by the capture following a standard (IVOA/W3C). Having the provenance information compatible with a standard, it will be possible for it to be explored/used/disseminated with external tools, or more importantly combined with external provenance information (what happened before or after a gammapy analysis). |
Bultako
force-pushed
the
prov-pig
branch
2 times, most recently
from
2ad6877
to
79561c4
Compare
Codecov Report
@@ Coverage Diff @@
## master #2458 +/- ##
=========================================
Coverage ? 92.19%
=========================================
Files ? 142
Lines ? 16159
Branches ? 0
=========================================
Hits ? 14897
Misses ? 1262
Partials ? 0 Continue to review full report at Codecov.
|
|
@bkhelifi, @mservillat, @cboisson |
|
This is a work I prepared with Jose Enrique, this also started the logprov project (https://github.com/mservillat/logprov). |
This PIG proposes to develop a solution for automatic capturing provenance when using the high-level interface. For the moment only provenance capture and storage and it only covers the high-level interface. The prov information would be logged in a structured way following the IVOA Prov model (which is based on W3C prov model). The logging settings declared in the config file of the high-level interface would activate prov logging and where to log this info. There is no need for external dependencies.
There has been been some prototyping work already, and standard provenance files and graphs may be easily built from the log file content with what is identified as prov info.