Scattergl points disappear when reaching a certain threshold in size difference

[luggie]

I noticed, that in Scattergl points start to disappear from the graph when resizing and reaching certain thresholds in size difference.

Minimal example:

from dash import Dash, html, dcc, callback
from dash.dependencies import Input, Output
import plotly.graph_objects as go
from plotly.graph_objs import Scattergl
import numpy as np
import pandas as pd


np.random.seed(41)
x = np.random.uniform(-10, 10, 10)
y = np.random.uniform(-10, 10, 10)
sizes = np.random.uniform(0, 1000, 10)

df = pd.DataFrame({
    'x': x,
    'y': y,
    'sizes': sizes
})

app = Dash(__name__)

app.layout = html.Div([
    dcc.Slider(min=13, max=14, value=1, id='slider-sizes', step=0.1),
    dcc.Graph(id='graph', style={"width": "100vw", "height": "80vh"}),
])


def update_marker_sizes(fig, size):
    for trace in fig.data:
        if 'marker' in trace and 'size' in trace.marker:
            trace.marker.size = [s * size for s in trace.marker.size]
    return fig


@callback(
    Output('graph', 'figure'),
    Input('slider-sizes', 'value')
)
def update_graph(size):
    figure = go.Figure(data=Scattergl(x=df["x"], y=df["y"], mode='markers',
                                      marker=dict(size=df['sizes']*size, sizemode='area')))
    print(f"{50*'-'}\nslider size:{size}")
    num_points = sum(
        len(trace['x']) for trace in figure.full_figure_for_development()['data'] if
        len(trace["x"]) > 1)
    print(f"num points in dev data (x): {num_points}")

    minsize, maxsize = float("inf"), float("-inf")
    for trace in figure.data:
        if hasattr(trace, 'marker') and hasattr(trace.marker, 'size'):
            if len(trace.marker.size) > 1:
                print(f"num points in data: {len(trace.marker.size)}")
            minsize = min(minsize, min(trace.marker.size))
            maxsize = max(maxsize, max(trace.marker.size))
    print(f"min: {minsize}, max: {maxsize}, diff: {maxsize - minsize}")

    return figure


if __name__ == '__main__':
    app.run_server(debug=True, port=8000)

The points are still present in the underlying data structure:

Output:

--------------------------------------------------
slider size:13.5
num points in dev data (x): 10
num points in data: 10
min: 1350.4235691765416, max: 10008.250444470288, diff: 8657.826875293747
--------------------------------------------------
slider size:13.6
num points in dev data (x): 10
num points in data: 10
min: 1360.4267067259975, max: 10082.385632947846, diff: 8721.958926221849
--------------------------------------------------
slider size:13.5
num points in dev data (x): 10
num points in data: 10
min: 1350.4235691765416, max: 10008.250444470288, diff: 8657.826875293747
--------------------------------------------------

Visually, this is what happens:

[Coding-with-Adam]

hi @luggie I'm getting a blank screen when I try to run your code. How did you get the graph to populate?

[luggie]

@Coding-with-Adam
the callback should fire when slider-sizes it is loaded into the DOM. I just tried the code again. For me it works with this conda env:

channels:
  - plotly
  - anaconda
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - alsa-lib=1.2.8=h166bdaf_0
  - attr=2.5.1=h166bdaf_1
  - boost=1.78.0=py39h7c9e3ff_4
  - boost-cpp=1.78.0=h75c5d50_1
  - brotli=1.0.9=h5eee18b_7
  - brotli-bin=1.0.9=h5eee18b_7
  - brotli-python=1.0.9=py39h5a03fae_8
  - bzip2=1.0.8=h7f98852_4
  - ca-certificates=2023.12.12=h06a4308_0
  - cachelib=0.9.0=py39h06a4308_0
  - cairo=1.16.0=ha61ee94_1014
  - click=8.1.3=unix_pyhd8ed1ab_2
  - cycler=0.11.0=pyhd3eb1b0_0
  - cython=0.29.28=py39h295c915_0
  - dash=2.9.3=pyhd8ed1ab_0
  - dash-bootstrap-components=1.4.1=pyhd8ed1ab_0
  - dash-daq=0.5.0=pyh9f0ad1d_1
  - dash-extensions=1.0.12=pyhd8ed1ab_0
  - dash-table=5.0.0=pyhd8ed1ab_1
  - dataclass-wizard=0.22.3=pyhd8ed1ab_0
  - dbus=1.13.18=hb2f20db_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - dill=0.3.6=pyhd8ed1ab_1
  - editorconfig=0.12.3=pyhd8ed1ab_0
  - et_xmlfile=1.1.0=py39h06a4308_0
  - expat=2.5.0=h27087fc_0
  - fftw=3.3.10=nompi_hf0379b8_106
  - flask=2.2.2=pyhd8ed1ab_0
  - flask-caching=2.0.2=pyhd8ed1ab_0
  - flask-compress=1.13=pyhd8ed1ab_0
  - font-ttf-dejavu-sans-mono=2.37=hd3eb1b0_0
  - font-ttf-inconsolata=2.001=hcb22688_0
  - font-ttf-source-code-pro=2.030=hd3eb1b0_0
  - font-ttf-ubuntu=0.83=h8b1ccd4_0
  - fontconfig=2.14.1=hc2a2eb6_0
  - fonts-anaconda=1=h8fa9717_0
  - fonts-conda-ecosystem=1=hd3eb1b0_0
  - fonttools=4.25.0=pyhd3eb1b0_0
  - freetype=2.12.1=hca18f0e_1
  - gensim=4.2.0=py39h6a678d5_0
  - gettext=0.21.1=h27087fc_0
  - giflib=5.2.1=h7b6447c_0
  - glib=2.74.1=h6239696_0
  - glib-tools=2.74.1=h6239696_0
  - greenlet=2.0.1=py39h5a03fae_0
  - gst-plugins-base=1.21.2=h3e40eee_0
  - gstreamer=1.21.2=hd4edc92_0
  - gstreamer-orc=0.4.33=h166bdaf_0
  - gunicorn=20.1.0=py39h06a4308_0
  - icu=70.1=h27087fc_0
  - importlib-metadata=5.1.0=pyha770c72_0
  - itsdangerous=2.1.2=pyhd8ed1ab_0
  - jack=1.9.21=h583fa2b_2
  - jinja2=3.1.2=pyhd8ed1ab_1
  - joblib=1.2.0=pyhd8ed1ab_0
  - jpeg=9e=h7f8727e_0
  - jsbeautifier=1.14.9=pyhd8ed1ab_0
  - keyutils=1.6.1=h166bdaf_0
  - kiwisolver=1.4.2=py39h295c915_0
  - krb5=1.19.3=h08a2579_0
  - lame=3.100=h7b6447c_0
  - lcms2=2.12=h3be6417_0
  - ld_impl_linux-64=2.39=hcc3a1bd_1
  - libblas=3.9.0=16_linux64_openblas
  - libbrotlicommon=1.0.9=h5eee18b_7
  - libbrotlidec=1.0.9=h5eee18b_7
  - libbrotlienc=1.0.9=h5eee18b_7
  - libcap=2.66=ha37c62d_0
  - libcblas=3.9.0=16_linux64_openblas
  - libclang=15.0.6=default_h2e3cab8_0
  - libclang13=15.0.6=default_h3a83d3e_0
  - libcups=2.3.3=h3e49a29_2
  - libdb=6.2.32=h6a678d5_1
  - libedit=3.1.20210910=h7f8727e_0
  - libevent=2.1.10=h28343ad_4
  - libffi=3.4.2=h7f98852_5
  - libflac=1.4.2=h27087fc_0
  - libgcc-ng=12.2.0=h65d4601_19
  - libgcrypt=1.10.1=h166bdaf_0
  - libgfortran-ng=12.2.0=h69a702a_19
  - libgfortran5=12.2.0=h337968e_19
  - libglib=2.74.1=h7a41b64_0
  - libgomp=12.2.0=h65d4601_19
  - libgpg-error=1.45=hc0c96e0_0
  - libiconv=1.17=h166bdaf_0
  - liblapack=3.9.0=16_linux64_openblas
  - libllvm15=15.0.6=h63197d8_0
  - libnsl=2.0.0=h7f98852_0
  - libogg=1.3.5=h27cfd23_1
  - libopenblas=0.3.21=pthreads_h78a6416_3
  - libopus=1.3.1=h7b6447c_0
  - libpng=1.6.39=h753d276_0
  - libpq=15.1=h67c24c5_1
  - libsndfile=1.1.0=hcb278e6_1
  - libsqlite=3.40.0=h753d276_0
  - libstdcxx-ng=12.2.0=h46fd767_19
  - libsystemd0=252=h2a991cd_0
  - libtiff=4.2.0=h2818925_1
  - libtool=2.4.6=h295c915_1008
  - libudev1=252=h166bdaf_0
  - libuuid=2.32.1=h7f98852_1000
  - libvorbis=1.3.7=h7b6447c_0
  - libwebp=1.2.2=h55f646e_0
  - libwebp-base=1.2.2=h7f8727e_0
  - libxcb=1.13=h7f98852_1004
  - libxkbcommon=1.0.3=he3ba5ed_0
  - libxml2=2.10.3=h7463322_0
  - libzlib=1.2.13=h166bdaf_4
  - lz4-c=1.9.3=h295c915_1
  - markupsafe=2.1.1=py39hb9d737c_2
  - matplotlib=3.5.1=py39h06a4308_1
  - matplotlib-base=3.5.1=py39ha18d171_1
  - more-itertools=9.1.0=pyhd8ed1ab_0
  - mpg123=1.31.1=h27087fc_0
  - munkres=1.1.4=py_0
  - mysql-common=8.0.31=h26416b9_0
  - mysql-libs=8.0.31=hbc51c84_0
  - ncurses=6.3=h27087fc_1
  - nspr=4.35=h27087fc_0
  - nss=3.82=he02c5a1_0
  - numpy=1.23.5=py39h3d75532_0
  - odfpy=1.4.1=py_0
  - openpyxl=3.0.10=py39h5eee18b_0
  - openssl=3.1.0=h0b41bf4_0
  - packaging=21.3=pyhd3eb1b0_0
  - pandarallel=1.6.4=pyhd8ed1ab_0
  - pandas=1.5.2=py39h4661b88_0
  - pcre2=10.37=he7ceb23_1
  - pillow=9.2.0=py39hace64e9_1
  - pip=22.3.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - plotly=5.14.0=py_0
  - ply=3.11=py39h06a4308_0
  - psutil=5.9.4=py39hb9d737c_0
  - pthread-stubs=0.3=h0ce48e5_1
  - pulseaudio=16.1=h126f2b6_0
  - pycairo=1.23.0=py39h23c5bb2_0
  - pyparsing=3.0.4=pyhd3eb1b0_0
  - pyqt=5.15.7=py39h18e9c17_0
  - pyqt5-sip=12.11.0=py39h5a03fae_0
  - python=3.9.15=hba424b6_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=3_cp39
  - pytz=2022.6=pyhd8ed1ab_0
  - qt-main=5.15.6=hafeba50_4
  - rdkit=2022.09.1=py39h0179058_1
  - readline=8.1.2=h0f457ee_0
  - reportlab=3.6.12=py39ha99c2b1_2
  - scikit-learn=1.2.0=py39h86b2a18_0
  - scipy=1.9.3=py39hddc5342_2
  - seaborn=0.11.2=pyhd3eb1b0_0
  - setuptools=65.5.1=pyhd8ed1ab_0
  - sip=6.6.2=py39h6a678d5_0
  - six=1.16.0=pyh6c4a22f_0
  - smart_open=5.2.1=py39h06a4308_0
  - sqlalchemy=1.4.45=py39h72bdee0_0
  - tenacity=8.1.0=pyhd8ed1ab_0
  - threadpoolctl=3.1.0=pyh8a188c0_0
  - tk=8.6.12=h27826a3_0
  - toml=0.10.2=pyhd3eb1b0_0
  - tornado=6.1=py39h27cfd23_0
  - tqdm=4.64.0=py39h06a4308_0
  - typing-extensions=4.9.0=py39h06a4308_1
  - typing_extensions=4.9.0=py39h06a4308_1
  - tzdata=2022g=h191b570_0
  - werkzeug=2.2.2=pyhd8ed1ab_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xcb-util=0.4.0=h166bdaf_0
  - xcb-util-image=0.4.0=h166bdaf_0
  - xcb-util-keysyms=0.4.0=h166bdaf_0
  - xcb-util-renderutil=0.3.9=h166bdaf_0
  - xcb-util-wm=0.4.1=h166bdaf_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h7f98852_1
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h7f98852_1002
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - zipp=3.11.0=pyhd8ed1ab_0
  - zlib=1.2.13=h166bdaf_4
  - zstd=1.5.2=ha4553b6_0
  - pip:
      - asttokens==2.4.1
      - dash-ag-grid==31.0.1
      - dash-bootstrap-templates==1.1.2
      - dash-core-components==2.0.0
      - dash-draggable==0.1.2
      - dash-html-components==2.0.0
      - dash-split==0.0.4
      - decorator==5.1.1
      - exceptiongroup==1.2.0
      - executing==2.0.1
      - flask-login==0.6.3
      - ipython==8.18.1
      - jedi==0.19.1
      - kaleido==0.2.1
      - matplotlib-inline==0.1.6
      - mol2vec==0.1
      - parso==0.8.3
      - pexpect==4.9.0
      - prompt-toolkit==3.0.43
      - ptyprocess==0.7.0
      - pure-eval==0.2.2
      - pygments==2.17.2
      - pygoslin==2.1.0
      - stack-data==0.6.3
      - traitlets==5.14.1
      - wcwidth==0.2.13

However I created an example where it gets little more clear what happens:

from dash import Dash, html, dcc, callback
from dash.dependencies import Input, Output, State
import plotly.graph_objects as go
from plotly.graph_objs import Scattergl
import numpy as np
import pandas as pd

sizes = np.linspace(start=1, stop=100, num=5)
coords = [c for c in range(5)]

df = pd.DataFrame({
    'x': coords,
    'y': coords,
    'sizes': sizes
})

app = Dash(__name__)

app.layout = html.Div([
    dcc.Slider(min=70, max=300, value=70, id='slider-sizes', step=10),
    dcc.Graph(id='graph', style={"width": "100vw", "height": "80vh"}),
    dcc.Interval(id='interval', interval=200, n_intervals=0)
])


def update_marker_sizes(fig, size):
    for trace in fig.data:
        if 'marker' in trace and 'size' in trace.marker:
            trace.marker.size = [s * size for s in trace.marker.size]
    return fig


@callback(
    Output('graph', 'figure'),
    Output('slider-sizes', 'value'),
    Input('interval', 'n_intervals'),
    State('slider-sizes', 'value'),
)
def update_graph(n_intervals, size):
    figure = go.Figure(
        data=Scattergl(x=df["x"], y=df["y"], mode='markers',
                       marker=dict(size=df['sizes']*size, sizemode='area', sizeref=None)
                       )
    )
    if size == 300:
        size = 70
    else:
        size += 10
    return figure, size


if __name__ == '__main__':
    app.run_server(debug=True, port=8000)

[Coding-with-Adam]

@luggie thank you for sharing the last code snippet. Indeed, when the slider value is equal to 150, the third marker disappears.

I wonder if this is related to the callback, the dcc.Graph, or whether this is only a Scattergl issue.

Did you face the same issue when building the Scattergl figure outside of a Dash app?

[luggie]

@Coding-with-Adam Yes, it also happens when not using a dash app around the figure like so:

import plotly.graph_objects as go
import numpy as np
import pandas as pd

sizes = np.linspace(start=1, stop=100*150, num=5)
coords = [c for c in range(5)]
df = pd.DataFrame({
    'x': coords,
    'y': coords,
    'sizes': sizes
})

scatter = go.Scattergl(
    x=df["x"],
    y=df["y"],
    mode='markers',
    marker=dict(
        size=df['sizes'],
        sizemode='area',
        sizeref=1,
        symbol=1
    )
)

fig = go.Figure(data=scatter)

print(fig)

fig.show()

print(fig):

Figure({
    'data': [{'marker': {'size': array([1.000000e+00, 3.750750e+03, 7.500500e+03, 1.125025e+04, 1.500000e+04]),
                         'sizemode': 'area',
                         'sizeref': 1,
                         'symbol': 1},
              'mode': 'markers',
              'type': 'scattergl',
              'x': array([0, 1, 2, 3, 4]),
              'y': array([0, 1, 2, 3, 4])}],
    'layout': {'template': '...'}
})

In this example, I realized that the marker never really disappear but their size is reduced to 0 or near 0 (on the plot) when reaching a certain limit or turn over point until they start growing again with rising marker size.
The same happens in my example with dash.
The underlying data structure on python side is not affected but the way that plotly.js interprets it.

[Coding-with-Adam]

hi @archmoj
Here's the codepen to replicate the issue.

[luggie]

@Coding-with-Adam any news on this or anything that I could contribute to? I really think that this is a rather critical bug isn't it?

Is there a way to retrieve the actual drawn marker size? I'd want to iteratively investigate when exactly the marker size breaks down to 0.

[luggie]

I figured out that it seems more like that there is a absolute limit on how big the markers can get before their sizes roll over which is between 10039.0 and 10039.5 with sizeref=1 and sizemode="area"

import plotly.graph_objects as go

sizes_still_there = [10039 for c in range(5)]
sizes_gone = [10039.5 for c in range(5)]
coords = [c for c in range(5)]

scatter = go.Scattergl(
    x=coords,
    y=coords,
    mode='markers',
    marker=dict(
        size=sizes_gone,
        sizeref=1,
        sizemode="area",
    )
)

fig = go.Figure(data=scatter)

fig.show()