[BUG] Problem copying a plot from one jupyter cell to another

[axil]

Software versions

Python version : 3.7.7 (tags/v3.7.7:d7c567b08f, Mar 10 2020, 10:41:24) [MSC v.1900 64 bit
(AMD64)]
IPython version : 7.30.1
Tornado version : 6.0.3
Bokeh version : 2.4.3
BokehJS static path : C:\Python37-64\lib\site-packages\bokeh\server\static
node.js version : (not installed)
npm version : (not installed)
Operating system : Windows-10-10.0.19041-SP0

Browser name and version

No response

Jupyter notebook / Jupyter Lab version

No response

Expected behavior

When I copy-paste a cell with a plot with Ctrl-C, Ctrl-V, the plot is copied to a new cell

Observed behavior

The plot is duplicated in the cell being copied and does not appear in the target cell

Example code

from bokeh.plotting import figure, output_file, show
p = figure()
x = [1,2,3]
y = [3,5,4]
p.scatter(x, y)
show(p)

Stack traceback or browser console output

No response

Screenshots

[bryevdv]

Bokeh plots are actually JavaScript programs. I would not expect this to work, and to be honest I am not sure it can ever be made to work.

cc @bokeh/dev

Edit: I would expect it to work if you copy the text from the original cell to a new cell, and execute the new cell. If that is not working, please clarify. But "copy cell" does some jupyter-specific thing and that is not what I would necessarily expect to work.

[axil]

Matplotlib plots are copied correctly, so it is sort of expected that bokeh also does something 'right' when copy-pasting.

I would not be surprised if the new cell would not display any output and that would be it. But duplicating the output of the old cell looks like a bug.

It is not something critical, but when I do it the millionth time, it is sort of annoying )

[bryevdv]

@axil Matplotlib plots are static images, Bokeh plots are interactive JavaScript programs. Can you clarify your exact sequence of operations? I can think of at least two different interpretations of "copying" and I want to make sure we are talking about the exact same thing.

[axil]

(I tried all kind of copy-pastes, but this one is the simplest) I press "C" keyboard button on a cell, then I press "V".

[bryevdv]

@axil Can you confirm that copying the text in the cell (not the "cell itself") and pasting that text into a new cell then executing the new cell works as expected?

[axil]

Yes. I confirm that. That's what I have to do every time I need to copy-paste a cell with a plot.

Also, I think I've found the problem. It is in the 'id' attribute of the 'bk-root' div. If I delete it, copy-pasting works as expected.

[axil]

What is the purpose of those id attributes? What will break if I delete them from the code?

[bryevdv]

Right, I expect the issue is that Jupyter copies the output cell in some way or other. But embedding Bokeh plots requires specifying a div id to embed into. So if jupyter just copies the output as-is, then it embeds into the previous div (which is in the old output cell).

The only only hope of making this work is if Jupyter provides some sort of hook that can customize the copy behaviour. I expect if that is possible at all, that it will also only be possible with the bokeh_jupyter extension installed (and possibly only in jupyterlab, not classic notebook)

[axil]

If I find this hook, what should it do? Generate a unique id? Should it register this id somewhere?

[bryevdv]

If I find this hook, what should it do? Generate a unique id?

AFAIK roughly speaking: currently Bokeh generates a div and a script with matching ids. The div is added to the output cell, and then the script is run (which then embeds the plot in the div with the specified id). So offhand speculation is that: the new "copied" div in the new output cell will need to be updated with a differnet ID, and the script will need to be updated to match.

But the way this is handled in jupyter_bokeh may also be incomplete. Other people have worked on jupyter intergration more recently than I. Regardless, this will require some investigation.

[axil]

So ok, it is used to populate the proper div. Now the next question is: is the id used after the div has been populated (eg in push_notebook or whatever) or it is ok to just delete it afterwards?

[axil]

I'm also sort of working on jupyter integration ;) Almost finished an article with the description of the new wrapper lib. Would you mind having a look before it goes online? I expect it to be ready enough in a couple of days.

[bryevdv]

So ok, it is used to populate the proper div. Now the next question is: is the id used after the div has been populated (eg in push_notebook or whatever) or it is ok to just delete it afterwards?

I am afraid I don't know the answer to that offhand. Bokeh is a complicated cross-runtime project, and Jupyter is a complicated cross-runtime project. Things will require some investigation to sort out just what is feasible.

I'm also sort of working on jupyter integration ;) Almost finished an article with the description of the new wrapper lib. Would you mind having a look before it goes online? I expect it to be ready enough in a couple of days.

Sure, I can read an article, but please know that I am far from an expert on jupyter integration. To be completely honest, I am not much of a jupyter user at all. There are probably other people who can comment much better than I.

[axil]

Point me to a person who can :) @mattpap?

[axil]

Btw, ref. your recent answer

AFAIK porting Tornado to pyodide would be huge undertaking, and I have not heard of anyone claiming to be working on it

There's a project called jupyterlite which ported the whole jupyter to webassembly.
https://blog.jupyter.org/jupyterlite-jupyter-%EF%B8%8F-webassembly-%EF%B8%8F-python-f6e2e41ab3fa

[philippjfr]

To be clear jupyterlite simply bypasses Tornado it does not attempt to actually run a tornado server. You'll find the discussion and various references/links here very relevant: holoviz/pyviz_comms#3

Copying a cell and using output_view run into exactly the same problem.

[axil]

I've tested push_notebook and jupyter_bokeh.BokehModel - they don't rely on id. @philippjfr do you think it is safe to remove the id after the div has been populated?

Do you have a community chat or meetings or something as I feel I'm somewhat offtopic here?

[bryevdv]

Some of us spend time on the Bokeh dev slack, you can request an invite here: https://slack-invite.bokeh.org/

But to reach the most people, it might make sense to convert this from and issue to a Github Discussion (or start a new one)

[axil]

I'm ok with converting this issue to a discussion, but I think I don't have enough permissions to do it (I don't have the button in the right margin)

[axil]

Is there any kind of a regression testing to check if removing the id after populating the div breaks anything?

[bryevdv]

All Jupyter testing is currently manual. There used to be some static image-diffs to make sure notebooks rendered at all, but those got broken at some point in the past. Automated notebook testing that involves interactive JS execution seems like a nightmare, unfortunately, and no one has had an opportunity to dive into it.

[jbednar]

AFAIK roughly speaking: currently Bokeh generates a div and a script with matching ids. The div is added to the output cell, and then the script is run (which then embeds the plot in the div with the specified id). So offhand speculation is that: the new "copied" div in the new output cell will need to be updated with a different ID, and the script will need to be updated to match.

OMG! This problem has bothered me for so long, and has seemed like such horrible behavior that made utterly no sense, and finally it makes sense! Thanks, @bryevdv , for that explanation. Why duplicating a cell would ever put twice the output into a single cell had always, always mystified me, but I now finally see how that can happen. And yes, if there is some way to avoid that, please, please let's do it; it's so confusing and irritating that cells don't behave properly like cells, with spooky action at a distance instead of working reasonably.

[axil]

@jbednar I've made a workaround for this issue: PR12196

As it is unlikely to be merged in soon (it is just a quick fix), and the annoyance level of this issue is quite high, I've created a patch so that anyone could test the new behavior manually (and have copy-paste working properly—at last!) until a proper solution is found.

Installation:

• Either copy the patched bokeh.min.js (unzip it from bokeh.min.js.zip) to <python_dir>/site-packages/bokeh/server/static/js

• Or copy patch.py (unzip it rom patch.py.zip) to the <python_dir>/site-packages/bokeh/server/static/js and launch it from there (it will patch bokeh.min.js and create a backup in bokeh.min.js.bak).

Then output_notebook(INLINE) — copy-pasting works properly now!

PS The attached bokeh.min.js corresponds to Bokeh v2.4.3; patch.py will hopefully work with future versions of Bokeh as well.

[jbednar]

Cool! I have no knowledge about whether that workaround is safe to merge, but I bet @philippjfr would have an opinion (as his Panel code might need to make use of the ID after the cell is rendered). In any case I primarily develop code for others to use and so I can't make use of any private or local workarounds; it always needs to be something I'm comfortable telling others to use. But the fact that this works is promising!

[philippjfr]

Honestly it might actually be pretty safe. The removal of the ID won't persist when saving the notebook so on reload everything should work just fine. I'll trial this in Panel and see how that goes.

[bryevdv]

I was thinking the same. My only question is whether the one change here is sufficient, or if a similar corresponding change will be needed in jupyter_bokeh as well.

[axil]

@bryevdv jupyter_bokeh is also affected by this bug and the patch is appliable and is working (according to my manual testing).

Updated the patch: https://github.com/axil/bokeh-copy-paste/blob/master/fix_copy_paste.py - now it works from any directory.
Uploaded some (manual) tests: https://github.com/axil/bokeh-copy-paste/blob/master/tests