SQLite test failures on GitHub Actions

[mstenta]

Recently SQLite3 tests have been failing on GitHub Actions. However, they seem to be inconsistent, because sometimes they pass, although they fail more often than not.

Originally encountered in #765. See the last few comments in that thread.

[mstenta]

This demonstrates the tests failing 3 times before passing on the 4th attempt: https://github.com/farmOS/farmOS/actions/runs/7424363072

They seem to consistently fail with the following log:

Testing /opt/drupal/web/profiles/farm
.................................................F.............  63 / 111 ( 56%)
................................................                111 / 111 (100%)

Time: 09:44.933, Memory: 24.54 MB

There was 1 failure:

1) Drupal\Tests\farm_inventory\Functional\InventoryTest::testApiInventory
Failed asserting that an array is not empty.

/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121
/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/opt/drupal/web/profiles/farm/modules/core/inventory/tests/src/Functional/InventoryTest.php:111
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

FAILURES!
Tests: 111, Assertions: 3041, Failures: 1.

[mstenta]

I tested creating a branch that disables the SQLite tests entirely, and that passed (PostgreSQL and MariaDB worked).

https://github.com/mstenta/farmOS/actions/runs/7424981157

I also tested creating a branch that comments out the Drupal\Tests\farm_inventory\Functional\InventoryTest::testApiInventory, and that also passed. I ran tests 4 times with that commit, and SQLite tests passed 3 times. They didn't finish the 4th time because a PostgreSQL test failed, which I think is unrelated.

https://github.com/mstenta/farmOS/actions/runs/7426500941

This was the PostgreSQL error:

 Testing /opt/drupal/web/profiles/farm
........................................................F......  63 / 111 ( 56%)
................................................                111 / 111 (100%)

Time: 10:27.705, Memory: 24.54 MB

There was 1 failure:

1) Drupal\Tests\farm_location\Functional\LocationTest::testLocationFieldVisibility
Failed asserting that a NULL is not empty.

/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:121
/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/opt/drupal/web/profiles/farm/modules/core/location/tests/src/Functional/LocationTest.php:103
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

FAILURES!
Tests: 111, Assertions: 3037, Failures: 1.

Confusing.... 🤔

[mstenta]

One more data point for the record: I just tagged and released farmOS 3.0.0, which triggered tests to run on the 3.x branch and the 3.0.0 tag. The SQLite test failed 3 times on 3.x and 4 times on 3.0.0 before finally passing and allowing the job to finish (and the release to be packaged).

[mstenta]

I wonder if this is at all related to https://www.drupal.org/project/drupal/issues/3405976

[mstenta]

I wonder if this is at all related to https://www.drupal.org/project/drupal/issues/3405976

I don't think it is. The reason I was wondering is because that issue mentions that it caused tests to start failing on SQLite, and:

we need to verify whether the problem can happen on 10.2.x-dev as well, since there is a new transaction manager.

The "new transaction manager" bit piqued my interest, so I still wonder if that's related.

But that specific issue seems to be related to XDebug 3.3.0. I tested with XDebug enabled on SQLite tests (because I encountered the reported issue locally with it disabled, so it made me wonder), and with XDebug pinned to 3.2.2, and encountered the same errors as before, so I don't think it's related.

[symbioquine]

It's interesting that it sounds like the failures are just for that one test case. Is there anything special about what that test is doing?

In case it lends any extra datapoints, I'm running the tests for my farmOS_wfs module against SQLite too without any particular problems yet. https://github.com/symbioquine/farmOS_wfs/actions/runs/8672159801/job/23782148327

[mstenta]

Here is the specific line that is failing in the test:

farmOS/modules/core/inventory/tests/src/Functional/InventoryTest.php

Line 111 in e86545c

$this->assertNotEmpty($response['data']['attributes']['inventory']);

The test overall is checking that updates to an asset's inventory are reflected in the JSON serialization of the asset in its API endpoint.

In this case, it is first checking to be sure that the inventory array is empty (which passes), then it creates a log that resets inventory, and tests to see if inventory is no longer empty. It should pass, but this "not empty" check is sometimes failing with SQLite3 tests.

The fact that it passes/fails inconsistently is the most confusing part. And only with SQLite3... 🤔