Example Showing Interoperability with d3js

[gmerritt123]

In response to @bryevdv 's request here, I've cooked up an example showing how you could use d3 to do JS-side contouring.

The idea was to take 100 random xy points to start, and map the equation from the bokeh contouring example to them, serving as a starting point.

Then add a PointDrawTool to move those points around, and a slider that controls the desired number of contour intervals.

The callback shows how:

• you can transform CDS data to an array of objects, which seems to be the defacto data structure for a lot of d3 (I'm no whiz at this stuff);
• you can pass that to some already developed d3 module (tricontour in this case)
• you can post process that result and transform it back to CDS ready data, which you use to update a different CDS (csrc)

Finally at the end the script shows a really basic function to build a template that allows embedding of additional CDN resources etc (Probably a better way to do this)

The output:

I dumped the code into a gist here .

I'm posting this in a discussion because I'd like to solicit feedback about it as an example (I know it's pretty rough), and get some opinions on how this example (or something along these lines) could be used to promote interoperability between d3 and bokeh (e.g. is this something to post in a section in the docs, or should I just post as a discourse showcase, etc. etc.).

G

[ianthomas23]

Thanks @gmerritt123, that is a really useful example. I've already learnt a lot from it about combining Bokeh with 3rd party libraries; something I would like to know more about but don't want to spend any time doing!

I note in passing that d3-tricontour is a port of my matplotlib tricontour code and examples without any acknowledgement. This is understandable but disappointing nevertheless.

I think there are two options here. You could just post it on discourse as it is, and the task is done. But a better approach would be for the Bokeh docs to be extended (the Advanced Usage section?) to include a new section on "BokehJS third-party integrations" or similar. This could be the first of a number of examples.

If that is the preferred option, I would want to see separate examples of quad and tri grid contouring in this way, and eventually some non-d3 examples too. I'd also want to make the words used about this example more precise, e.g. your initial sentence "use d3 to do JS-side contouring" needs to explicitly state that this is tri grid contouring. There is a much bigger demand for quad than tri grid contouring, so an unqualified "contouring" on its own usually means quad grid.

The timing of this is awkward happening just after we've added Python contouring, as some users might think that this is what we are talking about as the new functionality. When they discover that this isn't fully featured or is not geometrically robust then I'd expect we'd have to field questions that I do not wish to spend time answering as d3 is not my responsibility. But the approach of a new "BokehJS third-party integrations" section helps here as it makes the third-party aspect explicit.

[gmerritt123]

Thanks @ianthomas23 I very much agree, and that is indeed very disappointing re lack of acknowledgement.

From an optics/clarity perspective I should probably cook up something similar that avoids contours entirely, but stay focused on illustrating the key point (i.e. that bringing in a 3rd party library and using it to transform a CDS is quite powerful). I largely went to this tricontour example because a) it's a pretty simple setup, b) I already had the core JS functions on hand c) I had just used it for a day-job application (as seen in showcase), and d) JS-side contouring was recently asked about on discourse (I believe you responded).

Some other ideas:

1. Basically this , using d3.bin and d3.rollup to resample temporal data. I likely have a more involved/kickass example I can share with a bit of sanitation too using 75 years of local weather data.

2. Use the above contouring dataset as a base (i.e. the 100 random points sampling that math equation), but instead of contouring it, using https://github.com/d3/d3-delaunay to draw voronoi polygons.

3. Some sort of more involved contouring/2D interpolation example, where those 100 random points get interpolated to a grid (which could then drive a CDS that drives an Image glyph). Pick and interpolation technique, maybe IDW or kriging or radial basis functions.

Thoughts/preferences/other ideas solicited please... thanks!

[gmerritt123]

I quickly implemented 2. (voronoi) here

I like it as an example because it actually avoids writing too many JS functions. Just d3 transpose the point geometries, pass into voronoi, push the resulting polygon geoms. EZ.

[bryevdv]

This is really tremendous! It almost makes me want to reconsider including d3 (optionally) in order to have a built-in voronoi capability :D

But for now I agree with @ianthomas23 a new examples/advanced/integration/d3-vornoi.py example to kick off a corresponding section in the advanced user guide chapter would be ideal.

Edit: for 3.1 timeframe (soon) we could start by just adding the example and getting it in the gallery, and plan to author the section for the next release

[bryevdv]

It seemed to have installed bokeh 0.1.0

@gmerritt123 You are running into this:

• Add instruction to include tags in dev environment setup #12771

Evidently, some ways of forking/cloning a GH repo leave the local repo without any of the tag history, and that breaks our package build machinery.

Once we finally figured out this issue, we added new instructions a few weeks ago, but they are not published in a release yet. But if you follow the updated instruction in the PR above, then the 0.0.1 version issue should go away.

Sorry for the wasted cycles, I think this is a somewhat recent new default behavior from GH because I am nearly certain tags used to to get included by default in the past.

As for the hooks, maybe just disable/uninstall them for now. Perhaps we need to revisit the recommendation for installing them by default. Maybe it's better to just let CI catch any codebase issues for new contributors to lower the barrier to getting the PR pushed up in the first place. (cc @tcmetzger)

I'm not entirely sure about the git push issue, can you provide the output of running git remote -v ?

Edit This SO question seems to indicate that the git issue might come down to a text case problem:

https://stackoverflow.com/questions/38034866/cant-push-to-remote-branch-cannot-be-resolved-to-branch

which maybe has something to do with Windows in some way? I unfortunately do not have much windows dev experience to help here.

[gmerritt123]

Thanks for responding. I fixed the tag issue and made it to pushing my update to GH. I have not disabled/uninstalled the hooks because I don't know how.

I used the git gui to commit/push so I could "see" what i was doing. The printout is really long, mostly all "this script has carriage returns" messages. Posting the top and bottom of it below.

And as requested:

[mattpap]

This https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings?platform=windows should help with line endings issue. I suppose we should get that set up in the repository (via .gitattributes).

[gmerritt123]

OK, what configuration for line endings do I need to set up then?

Plenty of other errors (bokeh.util.strings ModuleNotFound) stopping this from "working" still...

I have not disabled/uninstalled the hooks because I don't know how.

How do I do this, will it help me bypass these push errors, and is that acceptable for me to do for now?

[bryevdv]

@gmerritt123 Yes, that's fine to do, I think it would be:

pre-commit uninstall -t pre-commit  -t pre-push

For lin endings, it should be line feeds (LF) only, without any carriage returns (i.e. not CRLF which is what windows uses by default AFAIK) From the link above, it looks like you can run this on windows:

git config --global core.autocrlf true

[mattpap]

Seeing this example, I started PR #12812, which allows to use import statements in JS callbacks, which makes creating such codes much simpler. Technically it's possible to achieve similar results with CustomJS and dynamic imports (import()), but it's a chore comparing to the new approach.

[gmerritt123]

Very cool. That would make my life a lot easier.

As an aside, having documentation on all the core/util functions would be amazing for me. I know about their existence and that its functions would shortcut tonnes of effort from me but it's too "in the dark" for me to leverage. For example I've written my own mercator/lat long projection functions despite knowing that those kinds of transforms exist already somewhere in there. My cross section tool is all in web mercator for basemaps, but distances are calculated by converting to lat long then applying haversine's formula... and I wrote all that math (poorly and inefficiently most likely) to get what I needed even when I knew that sort of utility already existed simply because a) I don't know exactly where to look and b) even if I did I wouldn't know the easiest way to use it. And, as you surely noted, in other cases I resorted to d3 for array functions etc. when I could have been using built in bokeh stuff.

[ianthomas23]

@gmerritt123 I really like the interactive voronoi example, it would be a great first example of integrating BokehJS and an external JS module. Please don't be too disheartened by the git dev environment trouble, we can provide as much help as you need. I could even dust down my Windows box tomorrow (which I don't do lightly!) and use that to try to reproduce what you are seeing if that helps.

We have been talking about the idea of fully documenting BokehJS for a while and recently I added it to one of our roadmaps: https://github.com/bokeh/bokeh/wiki/Individual-Plans--H1-2023#bokehjs-documentation. That is very nearly a commitment to do it, so hopefully that will be available soon-ish for us all to benefit from.

Thanks for looking at improving the infrastructure @mattpap. Maybe the best way forward here is to allow @gmerritt123 some time and space to get a dev environment up and running and submit a first PR. It might be possible to get this in before the upcoming 3.1 release. And then @mattpap could look at improving the infrastructure to make this easier, ready for the next release, and perhaps with a few more examples from a number of us.

[bryevdv]

re: JS api docs

One thing I have been thinking of proposing is to carve out a much smaller place for "stable" API to live. The main reason that BokehJS functions are not documented is that BokehJS is both sprawling and still a moving target under active development. Trying to document the whole thing is a sisyphean task.

So instead I'd like to propose making a new, purpose-created, initially empty place (Bokeh.api ? something like that) that can be the home for all supported, user-facing functions (that might just be trivial wrappers around internal versions). We can add new things too as they seem especially useful for users, and (critically) put everything in the new place under test with documentation at the same time is added (then even if BokehJS internals change, the user api module can remain insulated and decoupled from that churn).

I believe hat would be a tractable process that could actually be approached and successfully kept up to date incrementally.