[WIP] Require using the Jupyter Enterprise Org

[jasongrout]

Work in progress

Questions to answer ❓

Background or context to help others understand the change.

Currently Project Jupyter development on GitHub is spread across a number of GitHub orgs. This provides autonomy for subprojects to maintain their own contributor lists, roles, and processes. However, as the number of GitHub orgs in Jupyter increases, it also causes confusion about the scope of official Jupyter activities and increases friction for Jupyter-wide policies.

Recently, GitHub offered Jupyter (and other open-source projects) a free upgrade to their GitHub Enterprise Cloud plan. The basic idea is that Project Jupyter would have an Enterprise org, and inside that enterprise org we have our existing GitHub orgs. The enterprise org provides a place to centrally manage policies and roles across all of the Jupyter GitHub orgs, while still enabling individual subprojects to maintain their individual GitHub orgs.

This upgrade helps solve pain points we've had with multiple GitHub orgs in Jupyter. For example:

• Recently the Jupyter Security subproject enabled 2FA across all of the Jupyter GitHub orgs, and this required a nontrivial amount of work to follow through on each individual GitHub org. With an enterprise org, enabling 2FA across all of the Jupyter GitHub orgs would have been single checkbox at the enterprise org level.
• We've had a tension between creating more orgs to better reflect the autonomy of subprojects and the desire to keep the orgs to a manageable number (see discussion here, here, and here, for example). A centralized place to manage policies and roles would hopefully resolve that tension, and we can give all subprojects the autonomy of their own GitHub org if desired.
• Sometimes this tension for creating new GitHub orgs for subprojects has led to having multiple subprojects share a single GitHub org. When this happens, org-based roles for one subproject often grant permissions for another subproject in the same GitHub org that were not intended, infringing on the autonomy of subprojects.
• With the number of GitHub orgs in Jupyter, there is confusion about the scope and spaces of official Jupyter activities (see comments here, for example). We maintain manually a list of some of the GitHub orgs in the list of subprojects, but there are additional orgs under the Jupyter umbrella, such as the incubator org and the EC org. If all Jupyter orgs belonged to a single umbrella Enterprise org, it would clarify where official Jupyter development spaces are.

Furthermore, using the enterprise org gives additional benifits, such as:

• higher limits for CI and other resources
• the ability to create custom permission roles

Based on this information, the Executive Council recently made the decision to create a Project Jupyter enterprise org, and we reached out to the SSC about having various Jupyter GitHub orgs join the enterprise org. At this point, many have joined.

There is further discussion of this at #219.

A brief summary of the change.

In this change, we require official Jupyter GitHub orgs to belong to the new Jupyter enterprise GitHub org. We also simplify other text in the subproject governance.

What is the reason for this change?

To make it easier to have consistency across Jupyter GitHub orgs, while still enabling autonomy for subprojects.

Alternatives to making this change and other considerations.

• We could continue with our current practice of creating new GitHub orgs as we adopt new subprojects. However, it seems that every time a new subproject joins Jupyter, the discussion of the proliferation of GitHub orgs comes up. For example, here, and here.
• We could consolidate all development under a single or very few GitHub orgs, as suggested here. However, as mentioned above, this makes it harder for Subprojects to have autonomy in their permissions and operations.

The process ❗

The process for changing the governance pages is as follows:

• Open a pull request in draft state. This triggers a discussion and iteration phase
  for your proposed changes.
• When you believe enough discussion has happened,
  move the pull request to an active state. This triggers a vote.
• During the voting phase, no substantive changes may be made to the pull request.
• The Executive Council and Software Steering Council will vote, and at the end of voting the pull request is merged or closed.

The discussion phase is meant to gather input and multiple perspectives from the community.
Make sure that the community has had an opportunity to weigh in on
the change before calling a vote. A good rule of thumb is to ask several Council
members if they believe that it is time for a vote, and to let at least one person review
the pull request for structural quality and typos.

[fperez]

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

[Carreau]

I can't see https://github.com/enterprises/jupyter/ it insist on redirecting me to accept the invite to the Jupyter Enterprise for JupyterHub.

So some question I can't answer as I can't visit the jupyter entreprise page.

• Is there a way/how do you know all the official orgs that are part of Jupyter Enterprise ?
• What does the global management interface looks like ? In particular can you manage things in one place and have all the org "inherit" the enterprise setting, or is it just a way to "impersonate" for every org and thus still have us check setting 13 times ?

This looks like a positive to me

[fcollonval]

Coming here from a discussion started on JupyterLab gitter, looking for advice on the best path for a specific setting now managed at the entreprise level:

@fcollonval (Frédéric Collonval)

I'm not able to publish a new version of the language packs due to the above actions setting being disabled (and not changeable in JupyterLab GitHub org). It may be a consequence of the move under the Jupyter entreprise umbrella - that setting is, according to stackoverflow, handled there now.

Not sure what is the best solution.
Xref: https://stackoverflow.com/questions/72376229/github-actions-is-not-permitted-to-create-or-approve-pull-requests-createpullre

I don't know if it impacts the releaser

@jtpio (Jeremy Tuloup)
this release of notebook went through yesterday, using the releaser: https://github.com/jupyter/notebook/releases/tag/v7.2.0rc1

[jasongrout]

Is there a way/how do you know all the official orgs that are part of Jupyter Enterprise ?

I've got a question into GitHub asking about this.

What does the global management interface looks like ? In particular can you manage things in one place and have all the org "inherit" the enterprise setting, or is it just a way to "impersonate" for every org and thus still have us check setting 13 times ?

It looks like there are typically three choices for settings: "No policy" -> deferred to org owners, like today; "Enabled"/"Disabled" -> overrides org setting. For example:

[jasongrout]

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

[rpwagner]

Tiny edit in review. I also would like to repeat my previous suggestion:

My personal opinion is that now we have this option [the Enterprise feature], we should use MUST in that language. That would make it unambiguously clear whether something is or isn't "jupyter official".

I agree with the MUST language regarding GitHub as the service used to host Project Jupyter's source code and related work. There will be other services (e.g., Read the Docs, PyPI, conda-forge, etc.) used for different purposes, while GitHub is the service for the source code. As someone else commented, this helps to define the scope of which repositories may be in the scope of requirements including accessibility and security policies.

I don't have a suggested edit yet, I think it would go in the final section of "responsibilities", indicating specifically how we ask subprojects to tag their org(s) as "Jupyter official" (I'm not even sure we know yet how that will be done, I know @jasongrout was asking some questions about the new enterprise setup).

Once we have the policies and technical implications nailed down for adding orgs to Jupyter Enterprise org, the SSC may be the body to work with subprojects on being added.

[jasongrout]

I put the requirement to maintain source code on GitHub in the Project Jupyter enterprise org down in the list of subproject responsibilities. Is that better?

[rpwagner]

Is it worth being explicit here about the availability of the repositories? (Without the emphasis, of course.)

Maintain Subproject source code in publicly readable GitHub repositories in the Project Jupyter GitHub enterprise organization.

[fperez]

Change looks good to me, thanks @jasongrout! As for the explicit availability requirement, I guess a few of the security-related repos are private, so I don't know how many caveats are necessary? I don't have a strong opinion either way, I'll leave that point for you (@rpwagner) & @jasongrout to discuss.

Otherwise all good with me, thx!!

[rpwagner]

I think it's unlikely a subproject would want to be part of Jupyter and to keep their code private. However, I also like making things clear up front to avoid edge cases. And some repositories should be private, this would just be for the primary code base. I'm fine either way as well, because this looks great overall.

[jasongrout]

I think we can trust subprojects on this one, and we do have general oversight from the SSC and EC as well.

[jasongrout]

I think having publicly accessible code also falls generally under "Conduct its activities in a manner that is open, transparent, and inclusive"

[rpwagner]

Good point. Yes, that covers it.

[fperez]

With latest changes, 👍 from me, many thanks!!

[jasongrout]

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

[rpwagner]

As a policy matter, who should have owner privileges on the enterprise org? @rpwagner, from the security standpoint, what do you think?

I think it makes sense for at least some members of the EC/SSC to be enterprise org owners, but I don't think it is necessary for all of them to have administrative permissions, and I think it makes sense that you could be a github admin without being a member of the EC/SSC. I'd rather we have a small group of active administrators particularly picked for this purpose, plus a few people for oversight/backup from EC/SSC.

My suggestion is that the EC and SSC define a list of Jupyter communities members that are trusted to be enterprise owners, with at least one EC representative and at least one SSC representative. Equally as important, the list of enterprise owners should be reviewed by the EC and SSC on some regular interval, say quarterly. That review is as important as defining the initial list.

[rpwagner]

Thanks, @jasongrout!

I'm really glad to see this happening.

[jasongrout]

I think we should wait a couple more weeks to call a vote on this requirement to join the enterprise org to give us more time to surface and work out any other problems, and give subprojects that haven't joined (like JupyterHub) a chance to meet and discuss. In the meantime, I think we can proceed with figuring out the administration policy with the SSC.

[Carreau]

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

[jasongrout]

Can @jupyterhub owners refuse or accept the Jupyter enterprise invite, it's completely preventing me to access some of the links @jasongrout put in the description.

That's weird. I'll cancel the invite to JupyterHub for now. But JupyterHub folks, feel free to request an invite at any time! I think it's great to take your time here and discuss and see how it is working out.

CC @minrk as the SSC JupyterHub representative.

[fcollonval]

@fcollonval - do you want to hop on a quick video conference to figure out the issue with actions you raised?

Edit: I just switched this setting on. Do things work now?

Sorry for the intertwine discussion, we are now able to set that option at the JupyterLab org level. Thanks for the quick change.

[minrk]

We've no objections in jupyterhub/team-compass#719, so I'll accept the invitation if you re-send it. I'm also okay waiting for this PR to land before joining.

[Carreau]

That's weird.

Yeah, my guess it's a GitHub bug/issue, not the first one I encountered recently.

Here is a few redacted screenshot of what it looks like. "Getting started" is 404 for me:

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

[jasongrout]

I don't see a way to 'enforce' some setting on all orgs/repositories (like public security issue reporting), but it does give some overview to all the orgs/repos.

You don't have the organization owner permission currently, so you won't see the settings menu option. We're figuring out who should have admin access to the enterprise org (see conversation above). Here's the main page for me, for example:

Let's file a few bugs with github. Can you post a screenshot of what the Getting Started page looks like for you, Matthias? Also, after that, I'll resend the invite to JupyterHub (fyi, @minrk), and it would be great if you could screenshot that "approve/deny" page that prevented you from seeing the org.

[Carreau]

For getting to https://github.com/enterprises/jupyter , It would redirect me to https://github.com/enterprises/jupyter/invited which would show the (i guess), normal GitHub invitation accept page with a link to go to JupyterHub billing and plans

For getting started I get the usual 404 falling octocat, my guess is that's because i'm not an owner.

(for sec reason github often return a 404 instead of 403 so I'm guessing this is actually a 403 permission denied)

[jasongrout]

Thanks. @minrk, I went ahead and invited JupyterHub to join the enterprise org again.

[jasongrout]

For the record, the Getting Started page seems to mostly be focused to org owners in various setup tasks.

The links are:

GitHub Actions for enterprises
Manage code security
Monitor activity logs
Enterprise policies
Migrate your data to GitHub

I clicked the "Dismiss Onboarding" button at the bottom, and now it seems that page has disappeared.

[Carreau]

Here is where I'm redirected to

"billing and plan" links to https://github.com/organizations/jupyterhub/settings/billing/summary and there no other actionable links in hamburger menu...

[manics]

@minrk has accepted the invite for JupyterHub, but I'm seeing the same as @Carreau in #221 (comment)

[minrk]

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

[rpwagner]

I accepted the invite, but I think the accepted invitation has to be confirmed by someone on the Enterprise, too, to finish it off.

You are probably correct. I just accepted the transfer.

[manics]

https://github.com/enterprises/jupyter looks better now

[jasongrout]

At this point, the following orgs have joined the enterprise org:
IPython
Project Jupyter
Jupyter Governance
jupyter-incubator
Jupyter Server
Jupyter Widgets
Jupyter Xeus
JupyterHub
JupyterLab
Voilà Dashboards
Jupyter Attic

I think that is all of the official orgs of Jupyter. Are there any other orgs currently under Jupyter's purview?

[manics]

We've got a couple more Binder related orgs:

• https://github.com/binderhub-ci-repos (there's a reason we made a separate org but I can't remember why)
• https://github.com/binder-examples

[jasongrout]

We've got a couple more Binder related orgs:

@manics - I invited both of those orgs

[jtpio]

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

[jasongrout]

For the Voila Dashboards sub-project, there is also the voila-gallery organization: https://github.com/voila-gallery

I invited it. It looks like @rpwagner also invited a few other orgs like JupyterLite, jupyter-lsp, and Jupyter Standards.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

[jtpio]

I invited it

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

@jtpio, is JupyterLite part of Jupyter at this point? I know we were all discussing it at some point, but I forget where that discussion ended up.

Good question. I think this is still not clear yet, and iirc it is not listed as an official sub-project (or part of a sub-project) at the moment. Which is why I haven't accepted the invitation yet. Maybe it's currently at the same level as jupyter-incubator, as some parts of JupyterLite might end up being upstreamed in other existing Jupyter sub-projects in the long run.

But I'm happy to accept the invitation to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

[jasongrout]

Thanks! I got the email, but the invitation does not show up in the voila-gallery organization settings to be able to accept it. Not sure why.

Looks like you sorted it out, and I approved, so we're good on voila-gallery now.

But I'm happy to accept the invitation [for JupyterLite] to join the Jupyter Enterprise Org, if it can help simplify processes for Jupyter related orgs.

I suggest that orgs in the enterprise org be limited to those that are currently officially part of Jupyter, i.e., they consider themselves under Project Jupyter governance (including code of conduct, licenses, SSC and EC oversight, etc.), and they are accepted and acknowledged as under stewardship of a specific council (a subproject council, SSC, or EC). In other words, I see the enterprise org as being the place where we can draw a line around github activities and say "this is the official Project Jupyter".

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

[jasongrout]

@krassowski noted here that he accepted the invitation for the jupyter-lsp org to join the jupyter enterprise org. In that same thread, there has been recent discussion of how jupyter-lsp could officially be adopted as under the Project Jupyter umbrella.

Like JupyterLite, can we clarify the status of jupyter-lsp first before approving its transfer to the jupyter enterprise org? For jupyter-lsp, it seemed in that thread that the current option under discussion is having a vote in the Jupyter Frontend subproject about adopting the jupyter-lsp org under the frontend subproject.

CC @rpwagner

[rpwagner]

If we're not sure (like with JupyterLite, it seems), I suggest we first clarify if JupyterLite is under the Project Jupyter jurisdiction. We have a process for creating a new subproject that I think several projects are currently pursuing, for example, or existing subproject councils (like the frontends subproject) can vote to adopt JupyterLite under its umbrella.

@jasongrout I think JupyterLite has made itself a de facto part of Project Jupyter, especially because JupyterLite is provided on the https://jupyter.org/try as the entry point to learning about Jupyter. If JupyterLite is not officially part of Project Jupyter, we should consider removing that. From a security or risk perspective, having some oversight of the JupyterLite development seems necessary if we're promoting its use this directly.

That was logic we went through on Tuesday in the Security Subproject on Tuesday as we reviewed the GitHub Enterprise features. Of course, it would be even better if JupyterLite were brought in to Project Jupyter in a more official capacity.

[jtpio]

Do we need to decide on the state of jupyter-lsp and jupyterlite now? Or could these orgs be left out of the Jupyter Enterprise org for now, and added later if they are accepted as Jupyter sub-project (or join an existing sub-project)?

To not block this PR, since it might take some time and discussions.

[fperez]

👍 to @jtpio's suggestion above. I think this is a good sign that we have a vibrant ecosystem of "adjacent" projects that have been incubated semi-informally, and we should work on processing that as cleanly as possible, but that shouldn't block this PR.

For example, @choldgraf just posted a a full JEP for JupyterBook, after a good amount of preliminary discussion and even meetings with the SSC. The jupyter-ai team similarly posted a vote for inclusion within the "umbrella" of jupyterlab, back last year. Those are examples of good practices in terms of allowing for discussion, consensus seeking and process we should keep.

I'm always open to figuring out where our processes become unnecessarily slow/cumbersome and to remove pointless red tape, but that's different from rushing things. Let's decouple the two things and get the (I think entirely non-controversial) Enterprise part done as a rubber band enclosing the things that are clearly, unambiguously, today official Jupyter projects. And we leave any new inclusions to proceed at their pace.

[jasongrout]

+1 on both Jeremy's and Fernando's comments. I'll cancel the invites to the enterprise org for jupyter-lsp and jupyterlite until we have more clarity on their status. CC @krassowski as well.