Jenkins-CI is not trustworthy

[eugenk]

Tests pass on our Jenkins-CI while they should be failing. See http://ohj.iks.cs.ovgu.de:8080/job/ontohub-PRs/162/consoleFull and https://travis-ci.org/ontohub/ontohub/builds/151223864.
These two test the same commit. Travis fails (because there actually is something wrong in our code) while Jenkins says that everything is just fine.

[jelmd]

Well, not surprising for me because travis env is precise aka 12.04 release ( not supported by hets anymore) and thus is probably using an outdated hets version (actually it should fail, because AFAICS the hets PPA does not provide any precise packages anymore) ...

[eugenk]

The errors are not related to Hets. They are pure misused Rails 3.2 methods.

[jelmd]

jenkins is just java - has nothing to do with rails, just executes the tests. So if application tests fail, you need to find out first, why your tests do not produce the expected results. Since I don't have a clue, what you wanna test and what you expect, what side conditions need to be met, etc., I can't help solving this problem.

[eugenk]

To investigate this, we need ssh access to ohj. We need to run a rails console in the project directory (in the environment "test") and debug the test that actually should fail. Without this ssh access and the knowledge, where the project directory is, we can't find out what's wrong.

[eugenk]

We took a look at the console output of jenkins. The first few lines of ontohub-PRs/206 say:

17:13:17 
GitHub pull request #1768 of commit fc3b9c2987d971904e8cf96f8f3a2035b617689c, no merge conflicts.
17:13:17 Notifying endpoint 'HTTP:https://webhooks.gitter.im/e/79753809d9edbd6834f9'
17:13:18 Setting status of fc3b9c2987d971904e8cf96f8f3a2035b617689c to PENDING with url http://ohj.iks.cs.ovgu.de:8080/job/ontohub-PRs/206/ and message: 'Build started sha1 is merged.'
17:13:18 [EnvInject] - Loading node environment variables.
17:13:18 [EnvInject] - Preparing an environment for the build.
17:13:18 [EnvInject] - Keeping Jenkins system variables.
17:13:18 [EnvInject] - Keeping Jenkins build variables.
17:13:18 [EnvInject] - Injecting contributions.
17:13:18 Building in workspace /data/jenkins/jobs/ontohub-PRs/workspace_4
17:13:18  > git rev-parse --is-inside-work-tree # timeout=10
17:13:18 Fetching changes from the remote Git repository
17:13:18  > git config remote.origin.url https://github.com/ontohub/ontohub.git # timeout=10
17:13:18 Fetching upstream changes from https://github.com/ontohub/ontohub.git
17:13:18  > git --version # timeout=10
17:13:18  > git -c core.askpass=true fetch --tags --progress https://github.com/ontohub/ontohub.git +refs/pull/*:refs/remotes/origin/pr/*
17:13:19  > git rev-parse refs/remotes/origin/staging^{commit} # timeout=10
17:13:19  > git rev-parse refs/remotes/origin/origin/staging^{commit} # timeout=10
17:13:19 Checking out Revision 035247355c21663281553a063a130bcc8c37b43c (refs/remotes/origin/staging)
17:13:19  > git config core.sparsecheckout # timeout=10
17:13:19  > git checkout -f 035247355c21663281553a063a130bcc8c37b43c
17:13:19  > git rev-list 035247355c21663281553a063a130bcc8c37b43c # timeout=10

Here,

17:13:19  > git checkout -f 035247355c21663281553a063a130bcc8c37b43c

basically checks out staging instead of the commit that needs to be changed (which is mentioned in the first line).

In conclusion, Jenkins-CI always tests staging and never tests the pull request.

Edit: I see the checked-out commit is not HEAD of staging, but it is a commit that is merged into staging.