Integration Tests

We have lots of integration tests, but they need to be refactored so we can use them as CI, and also so that downstream packagers can use them when testing Cockpit as part of their OS delivery.

Test plan steps

QE owner: jscotka@redhat.com

Summary of brainstorming

Steps

• build packages like: http://openbuildservice.org/ , koji, breq
• for fedora
• for RedHat Enterprise Linux 6,7
• Trigger test automation after build (jenkins)
• Use stable trees (no updates) only cockpit to proper version (beaker, openstack instances)
• divide setup of machine part (rewrite to use autotest framework http://autotest.github.io/)
• think about which test really need more machines, or can be done locally (less resource usage -> faster)
• divide to (to be able run more test on one instance)
• setup environment
• testing part
• cleanup environment (optional)

Divide tests to various levels:

• unit tests: These run during the build phase, and are driven by 'make check' in the cockpit repo
• basic: siglehost (localhost) (could be used for build classification, could be enought)
• integration (multihost) based on VMs or native multihost testing (IPA, https://github.com/googlecloudplatform/kubernetes, multiserver tasks) (take longer time, no so important to use it for CI)
• semiautomated (per release)
• testing of browsers
• epiphany
• firefox
• chrome
• konqueror (not supported)
• IE10
• moving to automated framework http://www.seleniumhq.org/