Mep19
Table of Contents
Michael Droettboom
July 17, 2013
This MEP template is a guideline of the sections that a MEP should contain. Extra sections may be added if appropriate, and unnecessary sections may be noted as such.
Discussion
matplotlib could benefit from better and more reliable continuous integration, both for testing and building installers and documentation.
Testing
matplotlib currently uses Travis-CI for automated tests. While Travis-CI should be praised for how much it does as a free service, it has a number of shortcomings:
- It often fails due to network timeouts when installing dependencies.
- It often fails for inexplicable reasons.
- It does not save any build or test products, so it is often difficult to "post mortem" analyse what went wrong. This is particularly frustrating when the failure can not be subsequently reproduced locally.
- It is not extremely fast. matplotlib's cpu and memory requirements for testing are much higher than the average Python project.
- It only tests on Ubuntu Linux, and we have only minimal control over the specifics of the platform. It can be upgraded at any time outside of our control, causing unexpected delays at times that may not be convenient in our release schedule.
On the plus side, Travis-CI's integration with github -- automatically testing all pending pull requests -- is exceptional.
Builds
There is no centralized effort for automated binary builds for matplotlib. However, the following disparate things are being done [If the authors mentioned here could fill in detail, that would be great!]:
- @sandrotosi: builds Debian packages
- @tacaswell: Has automated Ubuntu builds on Launchpad
- @cgohlke: Makes Windows builds (don't know how automated that is)
- @r-owen: Makes OS-X builds (don't know how automated that is)
Documentation
None of the documentation on matplotlib.org is generated automatically. @mdboom regenerates it when making a release.
@NelleV, I believe, generates the docs automatically and posts them on the web to chart MEP10 progress.
matplotlib has complex requirements that make testing and building more taxing than many other Python projects.
- The CPU time to run the tests is quite high. It puts us beyond the free accounts of many CI services (e.g. ShiningPanda)
- It has a large number of dependencies, and testing the full matrix of all combinations is impractical. We need to be clever about what space we test and guarantee to support.
This section outlines the requirements that we would like to have.
# Testing all pull requests by hooking into the Github API, as Travis-CI does
# Testing on all major platforms: Linux, Mac OS-X, MS Windows (in that order of priority, based on user survey)
# Retain the last n days worth of build and test products, to aid in post-mortem debugging.
# Automated nightly binary builds, so that users can test the bleeding edge without installing a complete compilation environment.
# Automated benchmarking. It would be nice to have a standard benchmark suite (separate from the tests) whose performance could be tracked over time, in different backends and platforms. While this is separate from building and testing, ideally it would run on the same infrastructure.
# Automated nightly building and publishing of documentation (or as part of testing, to ensure PRs don't introduce documentation bugs). (This would not replace the static documentation for stable releases as a default).
# The test systems should be managable by multiple developers, so that no single person becomes a bottleneck. (Travis-CI's design does this well -- storing build configuration in the git repository, rather than elsewhere, is a very good design.)
# Make it easy to test a large but sparse matrix of different versions of matplotlib's dependencies. The matplotlib user survey provides some good data as to where to focus our efforts: https://docs.google.com/spreadsheet/ccc?key=0AjrPjlTMRTwTdHpQS25pcTZIRWdqX0pNckNSU01sMHc#gid=0
# Nice to have: A decentralized design so that those with more obscure platforms can publish build results to a central dashboard.
This part is yet-to-be-written.
However, ideally, the implementation would be a third-party service, to avoid adding system administration to our already stretched time. As we have some donated funds, this service may be a paid one if it offers significant time-saving advantages over free offerings.
Backward compatibility is not a major concern for this MEP. We will replace current tools and procedures with something better and throw out the old.