View/filter ergonomics clunky

[bryevdv]

I was prompted by this question: https://discourse.bokeh.org/t/best-way-to-use-widgets-in-jupyter-notebooks-in-2022-2023/9828 to try and create a simple example that filtered based on a Select, and was not happy with the result:

from bokeh.plotting import figure, show
from bokeh.sampledata.penguins import data
from bokeh.layouts import column
from bokeh.models import AllIndices, CDSView, CustomJS, GroupFilter, Select

SPECIES = list(sorted(data.species.unique()))

plot = figure(background_fill_color="#fafafa")

all_filter = AllIndices()
group_filter = GroupFilter(column_name="species")
view = CDSView(filter=all_filter)

plot.scatter("flipper_length_mm", "body_mass_g", source=data,
             fill_alpha=0.4, size=12, view=view)

select = Select(value="All", options = ["All"] + SPECIES)
select.js_on_change("value", CustomJS(args=dict(view=view, af=all_filter, gf=group_filter), code="""
    if (cb_obj.value == "All") {
        view.filter = af
    }
    else {
        view.filter = af  // hack

        gf.group = cb_obj.value
        view.filter = gf
    }
"""))

show(column(plot, select))

• I could not force an update without the view.filter = af hack. I tried at least a half a dozen other ways with emit, etc. The problem is that in that branch sometimes the view is assigned a new value, and sometimes only the group is assigned a new value but the code has to happen in one order.
• Having to explicitly set AllIndicies to mean "empty filter" (and thus also be forced to manually manage the group filter too) is very clunky and tedious. This is why filters was nice in the past — an empty list meant "no filtering".

So, questions:

• Are there any ways to improve this in existing releases that I just don't know about?
• What improvements can we make to Bokeh itself to allow this to be cleaner?

E.g. group filter group could be None to indicate "no filter" and js_link could get and optional map arg, then perhaps the CustomJS could be avoided entirely:

select.js_link('value', filter, 'group', map={"All": None})

As an aside I also think that the group filter only accepting one group is also super clunky. I guess the idea is that users use UnionFilter for everything? That is really very over-verbose. But maybe even with that, a map param would go a long way:

select.js_link('value', filter, 'operands', map={
    "All": [AllIndices],
    "adielie": [adelie_filter],
    ...
})

But now users have to create N group filters up front instead of just setting one group filters group. Still seems like a lot of work for users to just be able to filter data based on a select. Anyway, ideas welcome.

cc @bokeh/dev

[mattpap]

gf.group = cb_obj.value
view.filter = gf

CDSViewView doesn't listen to changes to filter's properties, as it is very common for bokehjs not to listen to transitive properties' changes, so the first line does nothing (it's a bug, or perhaps a design limitation for the moment). Second line will do nothing when switching between groups (there is value comparison, but for models it boils down to ID comparison). The right code to resolve this is:

gf.group = cb_obj.value
view.setv({filter: gf}, {check_eq: false})

However, given that all the filters are manipulated only in JS, then perhaps it would have been easier to just construct them in JS, i.e.:

select.js_on_change("value", CustomJS(args=dict(view=view, column_name="species"), code="""
  const AllIndices = Bokeh.Models.get("AllIndices")
  const GroupFilter = Bokeh.Models.get("GroupFilter")

  const {value} = cb_obj
  if (value == "All") {
    view.filter = new AllIndices()
  } else {
    view.filter = new GroupFilter({column_name, group: value})
  }
"""))

Though currently importing bokeh's models is a bit tedious, but nothing that can't be improved.

Having to explicitly set AllIndicies to mean "empty filter" (and thus also be forced to manually manage the group filter too) is very clunky and tedious. This is why filters was nice in the past — an empty list meant "no filtering".

AllIndices doesn't mean an empty filter. It means all indices. Perhaps think about it as a list comprehension with if true clause. CDSView.filter is based on set operations, so you can take all indices and then use set operations to construct other sets of indices. In bokeh it's very convenient, because there are operators that we override. In JS, one has to explicitly construct relevant models like UnionFilter, IntersectionFilter, InversionFilter, etc., because there is no operator overloading, though there could be equivalent methods added to Filter base class.

What did previously CDSView(filter=[]) mean? For you it was a full set of indices, but for me an empty set. What did CDSView(filter=[filter0, filter1, filter2]) mean? A union or an intersection? I'm guessing the later, but it can go either way. The new CDSView's API is explicit about the meaning and how to construct different sets of indices.

As an aside I also think that the group filter only accepting one group is also super clunky. I guess the idea is that users use UnionFilter for everything? That is really very over-verbose. But maybe even with that, a map param would go a long way:

I think that UnionFilter would be a workaround, but the main problem is not having group subsets in GroupFilter. This makes the overall example skewed towards coercing the APIs. If one could say GroupFilter(group=SPECIES) and use All label in Select (to cover the list), and bokehjs would listen to transitive property changes, then this example would have become much more straightforward to write. Also note that PR #12565 prohibits partial models like group_filter = GroupFilter(column_name="species"), so this example wouldn't run at all after that PR is merged. Thus having group subset support in GroupFilter makes even more sense. With all this, the logic in JS would be reduced to:

select.js_on_change("value", CustomJS(args=dict(view=view), code="""
  view.filter.group = cb_obj.value
"""))

[bryevdv]

view.setv({filter: gf}, {check_eq: false})

I'm not sure you are suggesting as much, but FWIW just to be clear I think this level of internal detail is fairly unreasonable for users to have to know about. And I think the "import and create filter objects" is too much code to expect for what should be a very simple operation due to its common-ness.

What did previously CDSView(filter=[]) mean? For you it was a full set of indices, but for me an empty set.

Why? A filter is not a list of indices, a filter is a thing that produces a list of indices that are a subset of a previous list of indices. I agree that whether the list was a union or an intersection was up to convention, and that's not great. But an empty list has a clear meaning to me: there are no filters to apply to the original list of indices.

I understand how AllIndices is useful with the operator overloading in Python but it's very tedious in JS and I expect for actually updating filters, JS may be more common. Whatever, this ship has sailed. In this specific case I think allowing group=None (or whatever, something) to indicate "empty group subset" would go a long way.

---

re:

select.js_on_change("value", CustomJS(args=dict(view=view), code="""
  view.filter.group = cb_obj.value
"""))

I forgot that labels and values can differ for Select, in that case it seems like this below (together with "group subsets") ought to be workable:

select.js_link('value', filter, 'group')

That, I think would be ideal. As an aside I do actually still think that map would be a good addition to js_link since not all widgets have that capability to separate labels/values.

[bryevdv]

@mattpap BTW what is your proposal for "group subsets"? Would you change group to be Either(String, Sequence(String)) ? Or would you allow the current string to be empty or delimited? Or something else?

[DSgUY]

Maybe add as Plotly a 'x' in the Select widget to reset the filter?

[bryevdv]

@DSgUY this question is really about the programmatic API for filters and views, not any particular UI widget (filters could be used in conjunction with many different widgets, buttons, sliders, (multi-)select, choice, toggles...)

[gmerritt123]

I have a few thoughts. First, a question: if I were to say put two GroupFilters on a view:

src = ColumnDataSource(data={'A':['banana','orange','apple']})

f1 = GroupFilter(column_name='A', group='banana')
f2 = GroupFilter(column_name='A',group='orange')

v = CDSView(filters=[f1,f2])

Internally, is src being iterated through twice to apply the filters?

Thought 1: to throw another suggestion/idea out there, maybe modifying GroupFilter to accomodate a list of groups or a singular group would greatly improve its usability. @bryevdv 's example would become a bit less janky, more flexible, and more straightforward (to me at least):

from bokeh.plotting import figure, show
from bokeh.sampledata.penguins import data
from bokeh.layouts import column
from bokeh.models import CDSView, CustomJS, GroupFilter, Select

SPECIES = list(sorted(data.species.unique()))

plot = figure(background_fill_color="#fafafa")


group_filter = GroupFilter(column_name="species",group=SPECIES)
view = CDSView(filter=GroupFilter)

plot.scatter("flipper_length_mm", "body_mass_g", source=data,
             fill_alpha=0.4, size=12, view=view)

select = Select(value="All", options = ["All"] + SPECIES)
select.js_on_change("value", CustomJS(args=dict(gf=group_filter,species=SPECIES), code="""
    if (cb_obj.value == "All") {
        gf.group = species
    }
    else {
        gf.group = cb_obj.value
    }
    //would still need to trigger a rerender with src.change.emit() etc here though I think 
"""))

show(column(plot, select))

It would also become intuitively obvious that GroupFilter is the best filter for multiselect widgets (e.g. CheckboxGroups etc,) callbacks would just have retrieve the selected groups and update the filter's group property to that.

Thought number 2: In general my user experience with filters has been to avoid putting more than one filter on a view as I inevitably end up being slayed by unpredictable rerender behaviour (unpredictable to me the humble user at least). I have been confounded by this many times, see #7273 , https://discourse.bokeh.org/t/filtering-cdsviewing-streaming/9359 , and this https://discourse.bokeh.org/t/understanding-change-emit-with-filters-sources-glyphs/7070/2 . So in general my approach has been when in doubt, make an IndexFilter, make it the only filter on the view, and update its indices property in CustomJS according to the state of all widgets etc. This is not what you're trying to bring to the table (I have to write a lot of CustomJS etc), but I inevitably find it's the only thing that works consistently, and if the answer to my first question about iterating through the dataset twice is "yes", then it's more computationally efficient.

[mattpap]

Internally, is src being iterated through twice to apply the filters?

Yes. Every filter works independently.