Proposal: BEP 10: Policy for Experimental Features

[bryevdv]

I believe it is important that we adopt defined polices regarding experimental features, in order to effectively communicate the experimental nature to users in a consistent manner. I have drafted BEP 10 for this purpose:

https://github.com/bokeh/bokeh/wiki/BEP-10:-Policy-for-Experimental-Features

cc @bokeh/core please provide feedback here regarding the wording, contents, etc.

[bryevdv]

I'll note that it might be nice to define things like an Experimental property wrapper that automatically adds the admonition helpstrings and also issues the warning on explicit usage, or experimental decorators for functions or classes that do the same. However, I have purposefully stuck only to outcomes in the BEP, and avoided specifying or suggesting specific implementations, in order to afford the most flexibility.

[ianthomas23]

I would be inclined to add some examples of the sorts of thing that could change, to make it clear that it could be anything and everything. So changes to behaviour and API, the latter could be changes to function and class names, argument names and order, existence of arguments, import location. And that changes can occur from one release to another without any formal deprecation process.

[bryevdv]

I will try to add History / Examples sections tonight. I am trying to straddle practical vs purity with this. We all aspire to maintain compatibility, and we do our best with available time and resources, but sometimes there are unforeseen changes that are required. This might be e.g. because GMaps changed to disallow anonymous usage, or because Jupyter made some change we have to contend with, or maybe we just unwittingly backed ourselves into a corner with some previous decision, that we now have to get out of to move forward.

So I do want to make it clear this policy is about foreseen changes. I.e. when we know right now that something is only partially fleshed out, and we want to add it anyway, with the exepectation that it will change. My informal litmus test here is more or less "if you think it needs a warning in a docstring" (then it needs all the rest as well). I can address that in a "Scope" section as well. Does that make sense and seem reasonable?

[ianthomas23]

Certainly.

[bryevdv]

@ianthomas23 I have finally made the promised edits

[ianthomas23]

Thanks, it looks good to me.

[bryevdv]

cc @bokeh/core if there is no more discussion I intend to put up a recorded vote to accept or reject this proposal on Loomio in the next few days.

[mattpap]

This warning shall be issued whenever an experimental feature is explicitly invoked by a user (e.g. by setting a particular property, or by calling a specific API function): (...)

This is excessive. The point of experimental status of features is to let users know about them and encourage them to try them out. This will likely discourage such behavior, annoy people and/or make them disable this warning (and possibly all bokeh's warnings altogether for a good measure while they are at it).

Experimental features are not deprecations. We should reasonably expect that a user should consult documentation before using a new feature. This is different from deprecations, where users are already using existing features and are not expected to consult the documentation with every new release, thus we have to let them know more vocally. What I would do instead of a runtime warning, is having experimental features stand out in the documentation. This will possibly require a sphinx extension, as after a cursory search I don't see sphinx supporting anything like this.

[bryevdv]

This is excessive.

In fact, projects often go further, disabling experimental features entirely unless they are explicitly opted into. I'm not opposed to that, however my preference is for default enabled, with a warning.

This will likely discourage such behavior, annoy people and/or make them disable this warning (and possibly all bokeh's warnings altogether for a good measure while they are at it).

The good news is this is easily solved by making the runtime warning explicitly actionable:

BokehExperimentalWarning(
    f"Feature {feature_name} is experimental and may change or be removed in the future. Users are 
    encouraged to try out experimental features and report their experiences. To disable warnings
    for experimental features, set the environment variable BOKEH_EXPERIMENTAL_WARNING=no, or 
    bokeh.settings.experimental_warning = False"
)

I will update the BEP proposal with this for now. We could discuss making warning silencing more granular, but that would be more work to actively "tag" experimental features with a unique identifier.

We should reasonably expect that a user should consult documentation before using a new feature.

Not even remotely a reasonable expectation. Mere notes in documentation are not sufficient, because users often do not see them. Users learn about Bokeh on discussion forums, StackOverflow and other coding sites, colleagues, and tutorials not put out by us. Most of the time users have no way to even know whether a feature is "new", it is just the solution to their problem they found online (until is isn't, then the support and complaint burden lands back us).

[mattpap]

In fact, projects often go further, disabling experimental features entirely unless they are explicitly opted into. I'm not opposed to that, however my preference is for default enabled, with a warning.

There are different concerns for different kinds of projects. If we were developing security related software, then sure I would do that or probably go even further. But then I would also be much stricter on other fronts, e.g. requiring a CLA or even not accepting any contributions from untrusted third-parties, etc.

[bryevdv]

There are different concerns for different kinds of projects. If we were developing security related software, then sure I would do that or probably go even further.

Feature flags are common in all kinds of software, for instance this very browser I am typing this on.

requiring a CLA

CLAs were not common when the project was first started, and the ship has sailed on that in any case. There are also reasonable debates around the necessity or utility of CLAs outside the context of "corporate" OSS.

not accepting any contributions from untrusted third-parties, etc.

FWIW I would never agree to this. For long-term viability, we must cultivate more new involved contributors, not put up barriers to entry.

[mattpap]

We should allow entire modules to be experimental as well.

[bryevdv]

I have opened a Loomio poll to reach a decision on this proposal.

[bryevdv]

This proposal passed 3-1 with two abstentions: https://www.loomio.com/p/sIJa1NSH/bep-10-policy-for-experimental-features

I will update the BEP this week and I also plan to take on implementing the features described in the BEP, unless anyone else expresses interest in taking it on.