Custom pre-built extension in a package

[ndmlny-qs]

This was first posted on the discourse page: https://discourse.bokeh.org/t/custom-pre-built-extension-in-a-package/9450 and bryan asked to place it here in a discussion thread. Thanks again for the help @bryevdv.

The basic need is that: I want to have a custom prebuilt Bokeh extension in an already existing python package, so when users install the Python package, they get the Bokeh extensions for use in Jupyter. I cannot have a user install the Python package, and then install the Bokeh extension in a manner similar to Panel or the JSMol extension (this is a hard requirement that I unfortunately cannot change).

first post from discourse

I have the following skeletal structure to my project, where a custom pre-built Bokeh extension exists in a sub-package of a Python package.

package
├── __init__.py
├── notebooks
│   └── nb.ipynb
└── sub_package
    ├── bokeh_extensions
    │   ├── extension.py
    │   ├── extension.ts
    │   ├── index.ts
    │   └── __init__.py
    ├── bokeh.ext.json
    ├── index.ts
    ├── __init__.py
    ├── package.json
    └── tsconfig.json

I am getting the following error when I try to use the extension in a notebook, e.g. the nb.ipynb in the notebooks folder. I should note that the package has been locally installed via pip, and bokeh build was run manually in the sub_package folder before trying to have the extension display in the Jupyter notebook.

Javascript Error: Model 'package.subpackage.bokeh_extensions.extension.Extension' does not exist.
This could be due to a widget or a custom model not being registered before first usage.

I’ve come across this error before, when experimenting with the custom Bokeh extension workflow, and the fix was to add the line;

__module__ = 'package.sub_package.bokeh_extensions.extension'

to the TypeScript class named Extension that contains the boilerplate properties, __view_type__, constructor, this.prototype.default_view property definitions. I always seem to get the same error when I update the __module__ property definition to anything, and was wondering if there is a path issue I am not properly fixing by modifying the __module__ property since this is a sub-package in a Python package.

I should also ask if creating a pre-built custom Bokeh extension in a package, and it not being its own installable package, is a viable solution? An outline of the relevant files for the workflow are below.

// package/sub_package/bokeh_extensions/extensions.ts
import {HTMLBox, HTMLBoxView} from '@bokehjs/models/layouts/html_box';
export class ExtensionView extends HTMLBoxView {
  model: Extension
  ...
};
export class Extension extends HTMLBox {
  __module__: 'package.sub_package.bokeh_extensions.extension';
  ...
};

# package/sub_package/bokeh_extensions/extension.py
class Extension:
    ...

// package/sub_package/bokeh_extensions/index.ts
export {Extension} from './extension'

// package/sub_package/index.ts
import {register_models} from '@bokehjs/base';
import * as Models from './bokeh_extensions';
register_models(Models as any);
export {Models};

# package/sub_package/__init__.py
from .extension import Extension

I know this is an odd use case, but I was not able to make Python Bokeh apps for use in a notebook as the tools being created in the custom extension had to be written in TypeScript. I’ve combed over the work done by MarcSkovMadsen for inspiration, but I cannot seem to find anyone making custom Bokeh extensions specifically for an existing Python package.

second post from discourse

Here’s a little more info on what I’ve tried, and the results of those attempts.

I tried to work around the ...does not exist. error and supplied the TypeScript directly to the Python object. I did this by refactoring the following;

# package/sub_package/bokeh_extensions/extension.py
from pathlib import Path
from bokeh.util.compiler import TypeScript
def get_ts():
    cwd = Path(__file__).parent.resolve()
    ts_path = cwd.joinpath("extension.ts")
    with open(ts_path, "r") as f:
        ts = f.read()
    return ts
class Extension:
    __implementation__ = TypeScript(get_ts())

This required an update to the extension’s TypeScript property as follows.

// package/sub_package/bokeh_extensions/extension.ts
...
export class Extension extends HTMLBox {
  ...
  static __module__ = 'extension';
}

After these updates, I no longer get the ...does not exist. error, but I get a new one having to deal with the TypeScript compiler. The new error is now;

error TS2550: Property 'flat' does not exist on type 'number[][]'.
Do you need to change your target library? Try changing the 'lib'
compiler option to 'es2019' or later.

This only sort of makes sense to me, since I’m supplying the TypeScript directly to the Python object. I’m guessing there is no build step in which the compiler would have been informed that my target in tsconfig.json is es2020. If someone knows for certain, it would be great to have a definitive correct/incorrect response to the following statement:

One should not supply any TypeScript directly to the Python object if creating a pre-compiled Bokeh extension.

If it is a definitive correct, then the statement in the documentation about not having to supply a name property as it gets “filled in” by TypeScript make more sense now.

If the above statement is not definitively correct, then I’m even more lost than before the path error in the original post, and any insights would be appreciated.

[ndmlny-qs]

I've experimented with the structure of a prebuilt custom Bokeh extension in an existing
project, and I am still not able to get the paths correct after building the extension.
Below are the two layouts I've experimented with: a flat-layout and a src-layout for the
existing package.

The read me files show how the example package is generated, but for brevity I show it
below.

mamba env create --file environment.yml
mamba activate bokeh-[flat|src]-layout
pip install --editable ".[dev]"
cd package
bokeh build
cd ..
jupyter lab

When I run the Jupyter notebook for either layout I still get the following JavaScript
console error:

VM302 bokeh-2.4.3.min.js:166 Uncaught Error: Model
'package.sub_package.bokeh_extensions.extension.Extension' does not exist. This could be
due to a widget or a custom model not being registered before first usage.
    at _.get (VM302 bokeh-2.4.3.min.js:166:1015)
    at E._instantiate_object (VM302 bokeh-2.4.3.min.js:165:4956)
    at E._instantiate_references_json (VM302 bokeh-2.4.3.min.js:165:5144)
    at E.from_json (VM302 bokeh-2.4.3.min.js:165:8794)
    at t.embed_items_notebook (VM302 bokeh-2.4.3.min.js:591:2451)
    at embed_document (<anonymous>:5:20)
    at <anonymous>:8:5
    at <anonymous>:24:3
    at a.renderModel (809.dc36b734dbd5fc902f01.js?v=dc36b734dbd5fc902f01:1:2099)

Following the instructions for setting the __module__ property, I tried to fix this
issue by changing the TypeScript file to all of the following:

1. // __module__ = 'extension';
2. __module__ = 'extension';
3. __module__ = '.bokeh_extension.extension';
4. __module__ = 'sub_package.bokeh_extension.extension';
5. __module__ = 'package.sub_package.bokeh_extension.extension';

Every time I modify the TypeScript file, I rebuild the Bokeh extension, reinstall the
package using pip, restart the Jupyter kernel, and conduct a hard refresh of the
browser. All attempts still fail with the error, given above.

Do I need to wrap the custom prebuilt Bokeh extension in a Panel app?

flat-layout

flat-layout/
├── environment.yml
├── pyproject.toml
├── README.md
├── setup.cfg
├── notebooks
│   └── flat-layout-testing.ipynb
├── package
│   ├── __init__.py
│   └── sub_package
│       ├── bokeh_extensions
│       │   ├── extension.py
│       │   ├── extension.ts
│       │   ├── index.ts
│       │   ├── __init__.py
│       ├── bokeh.ext.json
│       ├── example.py
│       ├── index.ts
│       ├── __init__.py
│       ├── package.json
│       └── tsconfig.json
└── tests

src-layout

src-layout/
├── environment.yml
├── pyproject.toml
├── README.md
├── setup.cfg
├── notebooks
│   └── src-layout-testing.ipynb
├── src
│   └── package
│       ├── __init__.py
│       └── sub_package
│           ├── bokeh_extensions
│           │   ├── extension.py
│           │   ├── extension.ts
│           │   ├── index.ts
│           │   ├── __init__.py
│           ├── bokeh.ext.json
│           ├── example.py
│           ├── index.ts
│           ├── __init__.py
│           ├── package.json
│           └── tsconfig.json
└── tests

relevant file contents

environment.yml

name: bokeh-[flat|src]-layout
channels:
  - conda-forge
dependencies:
  # dependencies
  - bokeh
  - ipywidgets
  - jupyter_bokeh
  - jupyterlab
  - pip
  - setuptools
  # dev dependencies
  - black
  - flake8
  - isort
  - nodejs
  - pip:
      - Flake8-pyproject

pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "package"
version = "0.1.0"
description = "Custom pre-built Bokeh extension using BokehJS in a [flat|src]-layout."
readme = "README.md"
requires-python = ">=3.7"
dependencies = [
    "bokeh",
    "ipywidgets",
    "jupyter_bokeh",
    "jupyterlab",
    "pip",
    "setuptools",
]

[project.optional-dependencies]
dev = [
    "black",
    "flake8",
    "Flake8-pyproject",
    "isort",
]

README.md

# Installation

Run the following on the command line.

mamba env create --file environment.yml
mamba activate bokeh-[flat|src]-layout
pip install --editable ".[dev]"
cd package
bokeh build
cd ..

# Notebooks

Run the following in the command line.

jupyter lab

Once Jupyter opens, navigate to the notebook called `testing.ipynb` in the `notebook`
folder.

setup.cfg

empty

notebook/*.ipynb

from bokeh.io import output_notebook, show
from package.sub_package.bokeh_extensions.extension import Extension
output_notebook()
extension = Extension()
output_notebook(hide_banner=True)
show(extension)

extension.ts

import { build_view } from "@bokehjs/core/build_views";
import { div } from "@bokehjs/core/dom";
import * as p from "@bokehjs/core/properties";
import { HTMLBox, HTMLBoxView } from "@bokehjs/models/layouts/html_box";
import { Plot } from "@bokehjs/models/plots/plot";
import { LinearAxis } from "@bokehjs/models/axes/linear_axis";
import { Grid } from "@bokehjs/models/grids/grid";
import { ColumnDataSource } from "@bokehjs/models/sources/column_data_source";
import { Line } from "@bokehjs/models/glyphs/line";
import { HoverTool } from "@bokehjs/models/tools/inspectors/hover_tool";
// import {Column} from '@bokehjs/models/layouts/column';

const createData = (): { x: number[]; y: number[] } => {
  const x = Array(500)
    .fill(0)
    .map((_, index) => {
      return index;
    });
  const y = x.map((value) => {
    return Math.sin(value);
  });
  return { x: x, y: y };
};

const createFigure = (): Plot => {
  const figure = new Plot({
    width: 300,
    height: 300,
    outline_line_color: "black",
  });
  const xaxis = new LinearAxis({
    axis_label: "x",
    axis_line_color: "black",
    minor_tick_line_color: "grey",
  });
  figure.add_layout(xaxis, "below");
  const yaxis = new LinearAxis({
    axis_label: "y",
    axis_line_color: "black",
    minor_tick_line_color: "grey",
  });
  figure.add_layout(yaxis, "left");
  const xgrid = new Grid({
    ticker: xaxis.ticker,
    dimension: 0,
    grid_line_alpha: 0.3,
    grid_line_color: "grey",
    grid_line_width: 0.3,
  });
  figure.add_layout(xgrid);
  const ygrid = new Grid({
    ticker: yaxis.ticker,
    dimension: 1,
    grid_line_alpha: 0.3,
    grid_line_color: "grey",
    grid_line_width: 0.3,
  });
  figure.add_layout(ygrid);
  return figure;
};

const createSource = (data: { x: number[]; y: number[] }): ColumnDataSource => {
  return new ColumnDataSource({ data: data });
};

const createGlyph = (): { glyph: Line; hoverGlyph: Line } => {
  const glyph = new Line({
    x: { field: "x" },
    y: { field: "y" },
    line_alpha: 0.7,
    line_color: "steelblue",
    line_width: 1.0,
    name: "glyph",
  });
  const hoverGlyph = new Line({
    x: { field: "x" },
    y: { field: "y" },
    line_alpha: 1.0,
    line_color: "orange",
    line_width: 1.0,
    name: "hoverGlyph",
  });
  return { glyph: glyph, hoverGlyph: hoverGlyph };
};

const addGlyph = (
  glyph: { glyph: Line; hoverGlyph: Line },
  figure: Plot,
  source: ColumnDataSource
): void => {
  figure.add_glyph(glyph.glyph, source, {
    hover_glyph: glyph.hoverGlyph,
    name: glyph.glyph.name,
  });
};

const createTooltips = (figure: Plot): HoverTool => {
  const renderers = figure.data_renderers;
  return new HoverTool({
    renderers: renderers,
    tooltips: [
      ["y", "@y"],
      ["x", "@x"],
    ],
    name: "tooltips",
  });
};

const addTooltips = (figure: Plot, tips: HoverTool): void => {
  figure.add_tools(tips);
};

export class ExtensionView extends HTMLBoxView {
  // eslint-disable-next-line no-use-before-define
  model: Extension;

  containerDiv: HTMLDivElement;

  plotDiv: HTMLDivElement;

  data: { x: number[]; y: number[] };

  figure: Plot;

  source: ColumnDataSource;

  glyph: { glyph: Line; hoverGlyph: Line };

  tips: HoverTool;

  initialize(): void {
    super.initialize();
    this.containerDiv = div({});
    this.plotDiv = div({});
    this.containerDiv = div(
      { style: "height: 500px; width: 500px" },
      this.plotDiv
    );
  }

  async render(): Promise<void> {
    super.render();
    this.el.appendChild(this.containerDiv);
    await this.createPlot();
  }

  private async createPlot() {
    this.data = createData();
    (window as any).data = this.data;
    this.figure = createFigure();
    this.source = createSource(this.data);
    this.glyph = createGlyph();
    addGlyph(this.glyph, this.figure, this.source);
    this.tips = createTooltips(this.figure);
    addTooltips(this.figure, this.tips);
    // const plotView = new Column({children: [this.figure]});

    const view = await build_view(this.figure);
    this.plotDiv.innerHTML = "";
    view.renderTo(this.plotDiv);
  }
}

export namespace Extension {
  export type Props = HTMLBox.Props;
  export type Attrs = p.AttrsOf<Props>;
}

// eslint-disable-next-line no-redeclare
export interface Extension extends Extension.Attrs {}

// eslint-disable-next-line no-redeclare
export class Extension extends HTMLBox {
  // eslint-disable-next-line no-use-before-define
  properties: Extension.Props;

  __view_type__: ExtensionView;

  static __module__ = "package.sub_package.bokeh_extensions.extension";

  // eslint-disable-next-line no-useless-constructor
  constructor(attrs?: Partial<Extension.Attrs>) {
    super(attrs);
  }

  static {
    this.prototype.default_view = ExtensionView;
    // eslint-disable-next-line arrow-body-style
    this.define<Extension.Props>(() => ({}));
  }
}