Force Direction

[g-parki]

Edit: Cleaning up original post since discussion has moved on from a high-level "swarm plot" to providing a lower-level force direction API.

Swarm plots are one thing I wish I had when I first used Bokeh. I know now we generally stray away from providing pre-schemed plots, and, of course, new features don't happen without resources/time/etc.etc. but I'd like to run the idea past @bokeh/dev, maybe see what the HoloViz folks think... I spent some time compiling some "market research" below to see what's out there.

Plotly.express.box()

• This box plot is able to display all data points in a swarm using parameter points=all
• Swarm doesn't have overlap avoidance, but it seems to be density-normalized nonetheless (i.e. bounded by a violin plot? I haven't looked at the code myself...).

Seaborn.swarmplot()

• Very "traditional" in that it doesn't distort data to save space. This results in longer "spikes" to preserve accuracy.
• This seems to be the standard for the Python ecosystem. It's the top Google result at least...

RawGraphs

• Not Python, but open source and I found it interesting
• They use a D3 force simulation to generate the layout, which results in some data distortion from the points "settling" into their resting place. I've seen several other articles/tutorials mentioning this method, so it seems common in the D3 realm.

Bokeh models.scatter() with y=jitter()

• Jitter allows random scattering within rectangular bounds
• No overlap avoidance
• This is analogous to seaborn.stripplot()

Brainstorming...

• Instead of asking for an entire new plot type models.swarm() I wonder if most the functionality to make this happen can be added directly to jitter(). Maybe we can accomplish a faux swarm plot similar to Plotly by making the jitter radius at each point proportional to data density?
• We don't have any D3 in Bokeh, but I think the force-simulation results are very clean/aesthetic. Is it a crazy idea to integrate some D3 (or other force simulation layout) into BokehJS?

[g-parki]

This is part of a larger set of common statistical plots (e.g. pie charts, box plots, histograms, heat maps), so perhaps it's unfair for me to latch on to this one in particular. They're all possible to do in Bokeh, but we don't provide a higher level API for them. Is this something in the "we may want to do this sometime" bucket, or the "Bokeh doesn't want to be this" bucket?

[bryevdv]

Offhand, I would say the latter. In the past, this is exactly what we tried to do with the high level bokeh.charts API. But that API lacked a complete vision and resources to maintain it. It ended up a mudled mess that we had to deprecate and ultimately remove entirely.

I don't really want to have a repeat of that, so to zero-th order I think this space is better filled by higher level packages like Holoviews, pandas_bokeh, Chartify, etc. We are/have considered adding a few higher level charts on an ad-hoc basis, e.g. we would like to add a contour plot, because computing the required multi-polygons is just too much work for users unless it is wrapped up somehow. We added hexbin for a similar reason.

I am not categorically opposed to considering higher level API in a more general way "for stats", but lots of questions/conditions would need to be met. There would have to be a real commitment from someone with a detailed vision, including what testing and maintenance would look like, how it could be implemented in a decoupled way, would it be better as a separate package, woudl it be better as simply some data-centric functions to adapt data in common ways for Bokeh, rather than schematized plots, etc.

[bryevdv]

Regarding the OP, I think it would be good to support force-direction in at least a couple of places, so it's definitely worth looking into, but it would be 100% preferable if we can find a small code or library to use w/o bringing in d3 proper.

[g-parki]

Well this is fun. D3 has their force stuff in a standalone package. Using it with the BokehJS API is relatively easy. CodePen here

[bryevdv]

Ok that's pretty neat. It's nice that is is split out, and I guess it's only 15kb? I think I am still 👎 on adding a high level "swarm plot" as it's own named thing, but definitely I would not be opposed to including standalone force layout library and exposing its functionality in some way, so that users could build a swarm plot (it also seems like a graph layout provider could make use of it).

[g-parki]

I am still 👎 on adding a high level "swarm plot"

💯 with you on that. Forget I said anything 😄

including standalone force layout library

Conceptually it seems easy - there are only a handful of forces available. Plumbing will become more involved for a Bokeh server case of adding/modifying forces on the fly. Likewise with interacting with editing tools for dragging points, which can become important when a linked graph folds on itself. But yeah an initial implementation seems totally doable.

[jbednar]

Seems like something that could be added to HoloViews, with perhaps some low-level plumbing in Bokeh...

[bryevdv]

Just clarifying my own main interest: I would really like to have better layout options for graphs.

[g-parki]

Thinking about features to include in the API, but first I'll write up a high-level summary on how the simulation works. For full documentation: https://github.com/d3/d3-force

Prepare Nodes

Each node is an object with and index, bi-variate position, bi-variate velocity, and optional fixed position. If velocities aren't provided, they're initialized at 0. The nodes are then arranged in a list of nodes. Example node:

const node = {index: 0, x: 5, y: 10, vx: 0, vy: 0, fx: 5, fy: 0}

Declare simulation and apply forces

const simulation = forceSimulation(nodes)

Available forces are:

• forceX
• forceY
• forceCenter - central gravity of system
• forceCollide - triggers at radius of each node
• forceManyBody - attraction/repulsion between each node
• forceLink - for linking nodes together in a graph
  Each force can be configured with a strength (positive or negative), or a default will be accepted.

Loop through simulation

Start/Stop Conditions

The simulation runs by decaying a temperature/energy parameter alpha, and "ticking" through time steps. alpha is decayed at each tick, and each nodes' position and velocity attributes are updated at each tick. The simulation stops when alpha drops below threshold alpha_min. This threshold can be raised to automatically stop the simulation early, but you run the risk of the simulation getting "stuck" in a state that looks unfinished.

Ticking with internal timer

A typical "animated" simulation is started with simulation.on("tick", callback) where callback is whatever function we need to retrieve new node data and update the canvas coordinates. This runs on an internal timer to provide results over time. Example:

// Ticking using internal timer
const simulation = forceSimulation(nodes)
simulation.force('x', forceX().strength(0.2).x(15))
simulation.on("tick", ticker)

function ticker() {
   // code_to_update_coordinates
}

Ticking programmatically

Instead of ticking with the internal timer, the simulation can be ticked in a loop:

// Ticking in a for loop
const simulation = forceSimulation(nodes)
simulation.force('x', forceY().y(15))
simulation.stop()

for (var i = 0, n = some_alpha_decay_constraint; i < n; ++i) {
    simulation.tick()
}
// code_to_update_coordinates