Post 3.0 docker and CI discussion

[ianthomas23]

Just after the 3.0 release will be a good time to revisit the docker image idea. Now that I have easy access to all 3 major OSes I would like to be able to build and test locally using docker for repeatable results regardless of OS.

There isn't much to change in the docker image creation process for this, we mostly just need to agree (or not) on creating a 3.0-dev image, uploading it somewhere so that it is easily available (docker hub?) and keep on using it until whatever the next Bokeh release will be whereupon we repeat the process.

But we would need to decide on what version of chrome to use for testing and update all of the baseline images to use that version.

We could leave GitHub Actions running on Ubuntu 20.04 or switch to using the docker image instead?

[mattpap]

But we would need to decide on what version of chrome to use for testing and update all of the baseline images to use that version.

We really need to figure out in general how to move forward regards this. We can't be stuck on Chromium 95 forever.

[bryevdv]

Which specific jobs/tests require a pinned chromium version? I would say lets use a docker image for just those jobs in CI, and let's nominally plan to publish a docker image for at/around the beginning of each branch-x.y (but of course we can publish updated post1 or -1 versions any time that's ever needed)

docker hub?

I would personally suggest looking at Amazon ECR, we already have service accounts etc set up there, it would streamline maintenance somewhat to keep as much as possible in one place under AWS

[mattpap]

I think it is just the bokehjs integration tests that generate image files (...)

Unit tests may depend on specific chromium version as well (e.g. taking measurements; sometimes they use mocks, but sometimes they depend on real measurements). Given that all bokehjs' tests run in a browser, then we can safely assume all depend on the version of Chromium. However, that's not the point. Currently, due to changes to webgl support in Chromium >= 96, we can't use any new version.

[ianthomas23]

I've just tried Chromium 103.0.5060.53 via snap on my Ubuntu 20.04 dev machine and WebGL works. If I run the bokehjs integration tests there are 115 changes with respect to the baseline images. I'd need the check the differences carefully, but at first glance they look like mostly slightly different positioning of text and slightly different antialiasing at the edges of WebGL glyphs. But it is definitely using WebGL as this can be confirmed by looking at hatch patterns which are visibly different for canvas, SVG and WebGL.

I recall Chromium 96 was a big problem for WebGL and although I haven't been following it closely it does look like we might be OK now?

[mattpap]

Indeed it's resolved with Chromium 103. I updated baselines in PR #12240. Myself I stopped testing this at 101.

[mattpap]

But it is definitely using WebGL (...)

It has no choice since PR #11823. If webgl was requested, but is not available, in tests, then you will get a hard failure and you won't be able to compare any images. But of course, we still need to investigate all changes, no matter how minor they are. Diffs look good though (normal changes from version to version).

[ianthomas23]

Thanks, this is great!

[ianthomas23]

I would personally suggest looking at Amazon ECR, we already have service accounts etc set up there, it would streamline maintenance somewhat to keep as much as possible in one place under AWS

@brydev I don't have any experience of ECR, so do you have an idea of how this might work from our end? Could we have a github secret that allows our CI to push new docker images to ECR? I suppose we should have a boolean upload flag on the image-generating github action as we may not always want to upload if just testing out some configuration changes.

[bryevdv]

We already have an AWS CI user, currently it uploads BokehJS, docs, etc. We wouldn't need any new credentials here, only to update some permissions within AWS to allow the existing CI user to push to ECR. Instructions for pushing to ECR seem fairly straightfoward:

https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html

I suppose we should have a boolean upload flag on the image-generating github action

How often are we pushing images? I thought the image generation / publish job was going to be a manually started one that we use at the beginning of new dev branches (and for any manual updates an existing branch might occasionally need). If the job is kicked off manually, do we need a flag? That said it would be simple to have the GH action kickoff dialog have a flag (similar to how the release/publish actions accept the version)

[ianthomas23]

Yes, you are right about the frequency of generating images and that they will be generated manually when required. I am just being defensive, I would like an option to either push to ECR or save the image as an artifact so that I can check it myself as happens now. This option will not be required very often, only when changing the workflow. Docker images can have multiple tags which we specify but there is always a latest tag so each new image is essentially the default image for any user who isn't specific about what they want, so when updating the workflow I'd rather avoid the possibility of pushing a default image that works but does something incorrect.

[ianthomas23]

I've made some improvements to the docker container in this branch: https://github.com/bokeh/bokeh/commits/ianthomas23/docker_updates. Now when running the container on Linux all the bokehjs tests pass including the visual integration tests. For python tests, pytest test/codebase is fine, as are unit tests if you exclude firefox i.e. pytest test/unit -k "not firefox". I assume it would be possible to get headless firefox working here but it is probably a similar level of complexity as getting headless chrome working which been most of the effort so far.

Python integration tests are OK if you exclude the "test_server_on_click_round_trip" tests, i.e. pytest test/integration -k "not on_click_round_trip". These tests fail and once they have done so they leave the headless browser in a state that causes all subsequent tests to fail. There are occasionally other flaky tests e.g. test_checkbox_button_group.

So as we stand it would be possible to create a github action that uses the docker container for all of the tests except the firefox and on_click_round_trip tests.

I've tried the docker container on my M1 mac (different OS and processor architecture) and you can run bokeh serve examples without a problem. But headless chrome isn't working so running the test suite isn't possible yet. It is also really slow to build, etc, as it is using rosetta emulation.

(Latest details on headless chrome on M1: when run node make test:spawn:headless with the extra --single-process arg added alongside the --no-sandbox arg in bokejhs/make/tasks/test.ts the error is CRITICAL ERROR: Error: connect ECONNREFUSED 127.0.0.1:9222 so it might be a network issue rather than headless chrome per se.)

Ideally I would test it now on Windows (which I can do) and Intel mac (which I can't).

[bryevdv]

Thanks for the updates @ianthomas23 do you have any ideas what might be going on with the on_click_round_trip tests specifically? I guess we could get a version of the image published somewhere soon and add a new CI job to start running tests with it soon?

[mattpap]

CRITICAL ERROR: Error: connect ECONNREFUSED 127.0.0.1:9222 so it might be a network issue rather than headless chrome per se.

@ianthomas23, can you spawn headless chrome on M1 outside doker at all?

[ianthomas23]

@mattpap Yes, if I build Bokeh natively on the M1 I can node make test:spawn:headless successfully.

@bryevdv The on_click_round_trip tests are failing as the data returned after toggling the button hasn't changed. But it is labelled as a flaky test, maybe it is just more flaky in these situations. And yes, I agree that publishing the image and getting regular CI runs with it should give us more data to play with. I will also switch to using the Docker container as my primary dev platform. Maybe, as you suggested, we can chat next week and get the push to ECR set up. Er, Tuesday perhaps? Any time.

[Karel-van-de-Plassche]

Unfortunately I don't have AWS experience either. Reminder to @bryevdv and @ianthomas23 that we still have https://hub.docker.com/repository/docker/bokeh/demo.bokeh.org and https://hub.docker.com/repository/docker/bokeh/bokeh floating around, that I just use for testing purposes. Now that 3.0 is released, I can take some time to get back into the development workflow(s) in due time.

[bryevdv]

This image is for dev/CI and will be fairly large, I don't think it will be so useful for end users. We should prioritize ease of maintenance (for us), and that means AWS since almost all the CI infra and secrets, etc are already there.

We may want to publish a very lightweight image for end-users to play around with or deploy, and maybe that gets published to dockerhub as part of releases. But that is a separate conversation.

[ianthomas23]

If we are going to go with the ECR solution then we may want to get rid of the docker hub so that we don't confuse users with the two possibilities.

[bryevdv]

We can discuss it, I see two different use case for two different audiences though:

• dev/CI image (for devs and CI) heavyweight, chrome and all testing deps, prioritize ease of maintenance and use in CI (so, AWS)
• user/release image, lightweight, for using/playing around with Bokeh, not for dev, prioritize ease of access, "standard" workflows (so, dockerhub)

[bryevdv]

But I should add: I am not invested in the decision beyond being the person who usually has to deal with infra administration, and trying to keep my personal commitment from growing too much. If others want to use docker hub for everything, and can commit to maintaining it however it needs, that's 100% fine by me.

[Karel-van-de-Plassche]

The reason I chose for Dockerhub at the time is because it's the "default". For ease of maintenance, I'd just choose one platform and stick with it; I do not mind AWS. If needed, we can set up a pipeline to push finished images to Dockerhub, but I would propose only on "tagged" releases.