[BUG] Nanosecond-precision datetime is truncated to microsecond

[douglas-raillard-arm]

Software versions

Python version        :  3.11.7 (main, Dec  8 2023, 18:56:57) [GCC 9.4.0]
IPython version       :  8.22.2
Tornado version       :  6.4
Bokeh version         :  3.3.4
BokehJS static path   :  /work/projects/lisa/.lisa-venv-3.11/lib/python3.11/site-packages/bokeh/server/static
node.js version       :  (not installed)
npm version           :  (not installed)
jupyter_bokeh version :  (not installed)
Operating system      :  Linux-5.15.0-92-generic-x86_64-with-glibc2.31

Browser name and version

Mozilla Firefox 122.0.1

Jupyter notebook / Jupyter Lab version

4.1.3

Expected behavior

The plot should display an X=Y line.

Observed behavior

Instead of getting a line, I get a staircase output.

Example code

from bokeh.plotting import figure, show
import pandas as pd

df = pd.DataFrame(dict(
    x=list(range(10000)),
    y=list(range(10000)),
))
df['x'] = pd.to_datetime(df['x'], unit='ns')

p = figure(
    x_axis_type="datetime", 
)

p.line('x', 'y', source=df)
show(p)

Stack traceback or browser console output

No response

Screenshots

[bryevdv]

some background discussion: #10379

TLDR; Fixing this properly will be a huge lift, and there are fundamental questions without answers still. It's not clear at all when this work might happen.

There are potential workarounds that might be applicable, but it really depends on some specifics of your use case. However, that's a support question so I'd ask you to take that question to the community discourse: https://discourse.bokeh.org which is a better place for us to have that discussion.

[douglas-raillard-arm]

Thanks, I started a thread here: https://discourse.bokeh.org/t/nanoseconds-on-x-axis/11397

[bryevdv]

@bokeh/dev maybe this is a crazy idea, but rather than embarking on some hard probably-breaking overall change to datetime serialization, what if we just made the base unit configurable for time-deltas. i.e. if a user wants floating point picoseconds, they can just ask for that. The default would stay base unit could stay microseconds, preserving compatibility with existing code, but users that need finer precision have a path. This won't cover the case where a user needs span centuries down to femtosecond precision (or whatever) but I am not sure anyone actually has that case.

Maybe there is some simple way to pair this with a specified global "large scale" offset (i.e a specific date), if absolute high-precision times are necessary. But that seems secondary, or at least my impression of use-cases involving very high precision is that they are often time-deltas e.g. between measurements of some kind.

To keep things tractable for built-in codepaths we could just have a few prescribed options: seconds, microseconds, picoseconds.