Integrating with Google/OSS-fuzz #12819
Replies: 4 comments 3 replies
-
cc @bokeh/dev for discussion At least for me personally, I would want to see a concrete, targeted, and scoped proposal. Bokeh has nearly 10k unit tests, just on the Python side. Re-instrumenting all of them would would be an insurmountable task. But just randomly adding a single test or two here and there also does not seem worth the overhead and complication of dealing with yet another service integration. Before embarking on something like this I'd like to see clearly enunciated strategy that guides:
laid out and documented in the dev guide. |
Beta Was this translation helpful? Give feedback.
-
Not a problem. It sounds like you are after some specific information in this proposal, could you please send an example proposal? |
Beta Was this translation helpful? Give feedback.
-
I'd be happy for any new testing to be run on the Bokeh codebase outside of our CI, and any problems found to be reported as github issues in the usual way. But I would not want to add it to our CI as it is already complicated enough. It might be fine if an existing or new maintainer were to commit to taking full responsibility for maintaining it going forward, but that seems unlikely. Personally I am not a fan of nondeterministic tests as part of CI workflow. Bokeh is already difficult enough for newcomers to contribute to without the danger that their first PR fails CI with an unrelated failure. |
Beta Was this translation helpful? Give feedback.
-
It seems unlikely that there will be any movement here so I will close this discussion |
Beta Was this translation helpful? Give feedback.
-
Hi,
I would like to help keep bokeh running smoothly by adding fuzz testing and integrating it with google/oss-fuzz. In case you are not familiar with fuzz testing, briefly, it is an additional layer of testing (in addition to unit testing) that will help uncover any hidden or hard-to-reach bugs that might have been missed during unit testing. Wikipedia explains it nicely:
Google offers a free, continuous fuzzing service called OSS-fuzz. If bokeh is integrated into oss-fuzz, the fuzz tests under bokeh will be built and then run once a day, to search for bugs and vulnerabilities in bokeh. This service can be integrated with the CI for bokeh, so that the fuzz tests are run for 10min or so for every pull request, preventing buggy code from being merged.
I've opened up a pull request to add a basic fuzz-testing harness here #12811. If you are keen on adding bokeh to oss-fuzz I'd be happy to champion the integration :)
Beta Was this translation helpful? Give feedback.
All reactions