More top-level cleanup

[bryevdv]

I'd like to propose the following changes for 3.2

• Move bokehjs to src/bokehjs it does not make sense for one project's source code to be under "src" but not the other's
• Move release to scripts/release I could possibly also get behind renaming scripts to more general tools

cc @bokeh/core

[ianthomas23]

👍 from me on both proposals. I marginally prefer tools to scripts.

[mattpap]

Move bokehjs to src/bokehjs it does not make sense for one project's source code to be under "src" but not the other's

bokehjs/ directory is not just the source code of bokehjs, as src/bokeh/ is to bokeh, but it is an entire separate project, just a part of a monorepo. bokehjs/ already has src/ directory with various packages, so we would end up with having src/bokehjs/src/, which to me is a weird organizational structure. Such move would also further emphasize the Python-centric bias of bokeh repository.

To me a much more productive change would be to have only two (for the significant part) top-level directories: bokeh and bokehjs, and a generic README and licence file, etc. This would remove the bias and avoid the src/.../src situation. However, if we want to have bokehjs' source code under the top-level src/, then it would make sense to me, to have bokehjs' project files (e.g. the top-level package.json) and other relevant files and directories at the top-level, which would only add to the clutter.

Before we make any moves, I would like the following questions to be answered:

1. If/When we are going to have compiled extensions (C++, Rust, Cython, etc.) for bokeh, e.g. to improve speed of validation (especially with data validation on the horizon) or to improve speed and robustness of serialization, then where those would live?
2. When the extensions compiler part of bokeh is going to be split-off from main bokeh, then where that is going to live?
3. If we ever split-off bokeh "core" from the rest of bokeh, then how that would be organized?
4. If/When we will again have multiple implementations of bokeh's protocol in other programming languages and run-times (e.g. legacy bokeh-scala for JVM, rbokeh; I'm slowly moving towards bokehrs for native and WASI), then where those are going to live?

If we are going to have multiple Python projects under src/, then it doesn't make sense to mix bokehjs into that, especially given bokehjs itself is a collection of packages. If we are going to keep implementations for other languages and run-times in the monorepo, then there should be no Python related files and directories at the top-level.

I would like any moves to be postponed until we have a clearer picture of the long term organization of this project and especially until the status of bokehjs is clarified. If we want bokehjs to be a real standalone project (it's not currently), then we cannot sideline it like this. If having only two directories at the top-level, as I proposed earlier, isn't an option, then a potential alternative is move bokehjs out into its own repository again, which I think is not a real option, in particular given the limitations of GitHub.

In any case, this discussion is something that shouldn't be targeted for 3.2 release, as it's not enough time to resolve those issues.

[mattpap]

Move release to scripts/release I could possibly also get behind renaming scripts to more general tools

+1. scripts/ seems better in my opinion.

[bryevdv]

HI @mattpap you raise some good points. I am in a bit of a hurry today but I did want to get out a few initial thoughts for more context, while I have them in my head.

First, I just want to make it clear that my suggestion is not tilted in any way towards sidelining BokehJS. In fact I believe the opposite, that promoting BokehJS as its own viable standalone project, capable of attracting its own contributors independent of (Python) Bokeh, is one of the most important things we can work toward.

Next, I guess it's worth explicitly stating what I would consider the "ideal" structure something like:

dev-311 ❯ tree bokeh
bokeh
├── README.md
├── docs
│   ├── bokeh
│   └── bokehjs
├── examples
├── projects
│   ├── bokeh
│   │   ├── pyproject.toml
│   │   ├── setup.py
│   │   ├── src
│   │   └── tests
│   └── bokehjs
│       ├── package.json
│       ├── src
│       └── test
├── tests
└── tools

I'd regard the current suggestion, which "leaks" a few python project details like setup.py etc into the top level to explicitly be a compromise, and I acknowledge it as such. I am open to the idea of going whole-hog down this route, but it's not currently my favored outcome. On that:

If we are going to have multiple Python projects under src/, then it doesn't make sense to mix bokehjs into that, especially given bokehjs itself is a collection of packages. If we are going to keep implementations for other languages and run-times in the monorepo, then there should be no Python related files and directories at the top-level.

Well, the reasons in which it makes sense are "history and pragmatism". I know that is not satisfactory in some ways, but those things cannot be discounted wholly. For operational, promotional, communication, governance, and fundraising reasons I think it makes sense to keep (Python) Bokeh and BokehJS as the primary center focus of the overarching meta-project. We need a coherent locus of attention, and Python Bokeh and BokehJS are those things that do exist right now. Nothing else will be close to either of them in the foreseeable future.

More specifically e.g. we have to direct people to the main repo for grant applications and publicity, etc. There has to be something there of substance and interest in that top README.

I think other language bindings should generally go in separate repos, so that whoever ends up maintaining them can do so more independently (release schedules, CI, expertise, etc). Maybe there can be some acceptance criteria for moving things into the monorepo. But we can't get into a situation where we can't release Python Bokeh for months because we are waiting on an update to bokeh-tcl or whatever. Or maybe further in the future we can invest in tooling to coordinate more sub-projects together in a unified way. On that:

potential alternative is move bokehjs out into its own repository again, which I think is not a real option, in particular given the limitations of GitHub.

I am severely conflicted here. On the one hand, we are not where we were ten years ago when development apart was truly impossible. I can even imagine a future where separating the projects affords benefits in terms of enforcing more discipline on runtime interfaces and compatibility over time. But I do think that splitting (Python) Bokeh and BohehJS, specifically, would incur very heavy operational cost (ops, tooling, communication, etc) that was absolutely do not have the organizational capacity to absorb right now. So I have to be 👎 on this for now.

I'll have more to say soon about your other specific questions.

[bryevdv]

I also agree the larger question can wait but I will plan to move release and scripts right before the 3.2 release, so that the time when the "latest" dev docs are out of sync with the repo structure is minimized.