Datetime Representation #12190
Replies: 4 comments
-
Further aside about webgl: the "baked offset" is only needed because we send the data space coords to be mapped by the shaders. But we currently only handle linear scales, which is also quite limiting. Options there:
|
Beta Was this translation helpful? Give feedback.
-
I see my mistake now. Introduction of |
Beta Was this translation helpful? Give feedback.
-
All other discussion aside, this is what you should do and what I've been suggesting from the beginning, with the possibility of future optimizations, but not considering them as a part of the current design. |
Beta Was this translation helpful? Give feedback.
-
I do agree that this distinction makes good sense
After a little thought this was my own conclusion as well, and how I intend to proceed. FWIW I will say I did not manage to catch any notion of this particular aspect of the webgl work from our previous discussions.
Certainly, but please don't feel like you have to do the thinking alone. I created this issue so that we can all benefit from a discussion around these issues, both to be able to provide input on the up-front design of a plan, and to be able to be informed as to the plan as any work progresses. In any case please share any thoughts about how we might improve the representation of datetimes when you have formulated some ideas. |
Beta Was this translation helpful? Give feedback.
-
cc @bokeh/dev @mattpap @philippjfr
Regarding the recent PR to change
NumberArray
toFloat32Array
, I think this will require some more consideration. Specifically, with respect to our current choice to represent datetimes as milliseconds-since-epoch. At this scale, values for current datetimes are e.g. around1596688253232
. However, this is somewhat substantially larger than the largest integer that can be faithfully represented infloat32
. Consider:As it happens, this is, purely by accident, not a problem yet. This is because there is not a
int64
typed array type, so we send these values over as plain lists, and the "materialized" coordinates get stores as such, even thought the type of e.g.this._x
is supposed to beNumberArray
:I believe if at some point we forced those values to be
Float32Array
then we would lose the ability to distinguish times smaller than (at least) a minute, which is obviously unacceptable. I'd like to start figuring out a better, more rigorous way to represent datetimes that is amenable to use of some combination of 32-bit buffers, since that is what we can send to webgl, and what can be more efficiently transported via websocket or base64 encoding.As an aside, this exact problem was handled in the current webgl code by "baking" in an arbitrary offset that moves the values into an acceptable range. It's all rather ad-hoc, and also something I want to keep in mind as we develop a plan forward.
Some quick ideas:
We could do nothing, and continue to represent datetimes as 64 bit integers in non-typed arrays. We should actively make sure this is maintained under test, and the webgl support for scaling/offsetting will probably need to be improved and made better.
We could represent datetimes with two arrays, e.g an
int32
number of seconds andfloat32
number of nanoseconds. This would unquestionably be breaking on the BokehJS side. It's possible we could make many basic use-cases on the Python side handle things transparently (but almost certainly not all possible scenarios). Exploration would be needed, though.For 2.x the only option is the first option, so this is really a question about what we can do for eventual 3.x
Beta Was this translation helpful? Give feedback.
All reactions