selenium/chrome-standalone:latest causes automated tests to fail

[paul121]

Describe the bug
Automated tests have been failing the past couple days. It seems like it is just two test cases:

There were 2 errors:

1) Drupal\Tests\farm_location\Functional\LocationTest::testLocationFieldVisibility
Behat\Mink\Exception\ResponseTextException: The text "Current geometry" was not found anywhere in the text of the current page.

/opt/drupal/vendor/behat/mink/src/WebAssert.php:785
/opt/drupal/vendor/behat/mink/src/WebAssert.php:262
/opt/drupal/web/core/tests/Drupal/Tests/WebAssert.php:898
/opt/drupal/web/profiles/farm/modules/core/location/tests/src/Functional/LocationTest.php:87
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

2) Drupal\Tests\farm_settings\Functional\ModulesFormTest::testInstallFunctionality
Drupal\Core\Extension\Exception\UnknownExtensionException: The module farm_plant does not exist or is not installed.

/opt/drupal/web/core/lib/Drupal/Core/Extension/ExtensionList.php:346
/opt/drupal/web/profiles/farm/modules/core/settings/tests/src/Functional/ModulesFormTest.php:162
/opt/drupal/web/profiles/farm/modules/core/settings/tests/src/Functional/ModulesFormTest.php:73
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:726
There was 1 failure:

1) Drupal\Tests\farm_group\Functional\GroupTest::testGroupFieldVisibility
Failed asserting that a NULL is not empty.

/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:[121](https://github.com/farmOS/farmOS/runs/5623699703?check_suite_focus=true#step:12:121)
/opt/drupal/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:55
/opt/drupal/web/profiles/farm/modules/asset/group/tests/src/Functional/GroupTest.php:76
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:726

FAILURES!
Tests: 88, Assertions: 1723, Failures: 1, Errors: 2.
Error: Process completed with exit code 2.

Additional context
Add any other context about the problem here.

[mstenta]

These all seem to be tests that extend from WebDriverTestBase. I wonder if something changed upstream?

I am not set up to test these locally right now, so I'll have to look another time. I get:

The test wasn't able to connect to your webdriver instance. For more information read core/tests/README.md.

The original message while starting Mink: Could not open connection: Curl error thrown for http POST to http://chrome:4444/wd/hub/session with params: {"desiredCapabilities":{"browserName":"chrome","name":"Behat Test","chromeOptions":{"w3c":false,"args":["--disable-gpu","--headless","--no-sandbox"]}}}

Could not resolve host: chrome

[paul121]

@mstenta do you know if you had the docker container running at that time? These lines:

farmOS/docker/docker-compose.development.yml

Line 29 in 11cc36a

# Enable this service when executing javascript tests.

But looking into things a bit... it seems that our tests stopped working 13 days ago, the same day there was a new release of the docker-selenium docker image we use: https://github.com/SeleniumHQ/docker-selenium/releases/tag/4.1.2-20220317

[paul121]

And the tests pass when I pin the docker image to the previous version! So what changed? hmm. image: selenium/standalone-chrome:4.1.2-20220217

https://github.com/paul121/farmOS/actions/runs/2068962423

I'm now reading that they don't recommend using the latest tag..

[paul121]

I came across this issue which seems like it could be related: SeleniumHQ/selenium#10448

But changing our w3c option to true or even removing it entirely created different errors. So I'm not sure if that is the issue after all.

So... the only difference between the 4.1.2-20220217 and 4.1.2-20220317` releases is that Chrome and ChromeDriver went from v98 to v99. Maybe there is something there

[mstenta]

Ah good sleuthing @paul121!

Can we pin the Selenium version temporarily (referencing this issue in the commit message), but leave this issue open while we figure out the root cause? That would at least unblock the tests from running on other PRs etc.

[paul121]

Yes I suppose that is the best for now. I'll open a PR from my branch momentarily..

[mstenta]

I merged @paul121's PR that pins the selenium/chrome-standalone version for now. Leaving this open so we can get to the root cause of this issue and possibly revert back to latest.

[mstenta]

I wonder if this is still an issue?

[mstenta]

I just tested this again and it seems to still be an issue. Although the failure looks a bit different now:

There were 4 skippeds:

1) Drupal\Tests\farm_group\Functional\GroupTest::testGroupFieldVisibility

/opt/drupal/web/profiles/farm/modules/asset/group/tests/src/Functional/GroupTest.php:48

2) Drupal\Tests\farm_settings\Functional\ModulesFormTest::testInstallFunctionality

/opt/drupal/web/profiles/farm/modules/core/settings/tests/src/Functional/ModulesFormTest.php:50

3) Drupal\Tests\farm_location\Functional\LocationTest::testComputedAssetLocation

/opt/drupal/web/profiles/farm/modules/core/location/tests/src/Functional/LocationTest.php:55

4) Drupal\Tests\farm_location\Functional\LocationTest::testLocationFieldVisibility

/opt/drupal/web/profiles/farm/modules/core/location/tests/src/Functional/LocationTest.php:68

OK, but incomplete, skipped, or risky tests!
Tests: 114, Assertions: 3007, Skipped: 4.   
Error: Process completed with exit code 1.

The tests were skipped instead of failing, it seems. 🤔

My test commit: mstenta@5bda215

GitHub Actions test results: https://github.com/mstenta/farmOS/actions/runs/6566137399/job/17836347868