Jupyter(Hub) conceptual intro #2726

rkdarst · 2019-09-09T12:04:33Z

At the JupyterHub/BinderHub workshop, one of our ideas was to make a conceptual intro to JupyterHub so that people could know what it does, and in particular what it doesn't do (what is handled by other components). We get many issues that end up misdirected or that have a root cause of not understanding what the components are.

This PR is my initial draft - comments welcome. I've written it informally and with a certain opinion - it's supposed to be like a teaching giving a first lesson, and not technical reference. It goes far beyond just JupyterHub, but the reason for having it here is that you really need to start knowing this stuff once you start administrating a JupyterHub, before that you can sort of get by without a perfect mental model.

Issues I know of:

What is a proper title?
Where should it go under the table of contents?
I used the very verbose "single-user notebook server", I'd like a shorter term that still avoids ambiguity.
All parts need fact-checking, because I haven't done that yet
After the content is somewhat stable, we need to add links and references.
Intro needs improvement and we should make sure it explains the purpose well
Conclusion needs improvement and guides to what is next

consideRatio · 2019-09-09T15:35:52Z

@rkdarst thank you for putting in this excellent work to give an cohesive overview how things relate! I appreciate how it is written in a readable non-reference like manner, I'm thinking about the clarifications in the early part of the text for example where you clarify the purpose of the text is to give overview etc.

rkdarst · 2020-04-13T23:32:39Z

I did a big pass on this, and I think we should working on polishing. It's late and I'm sure everyone will be able to find little improvements - instead of making things perfect, I'll just let everyone read and make their suggestions.

The CI failures seem unrelated, but at least the docs one should be fixed.

betatim · 2020-04-15T22:09:37Z

Thanks for coming back to this!

The docs build error is:

Warning, treated as error:
/home/circleci/project/jupyterhub/auth.py:docstring of jupyterhub.auth.Authenticator.admin_users:1:duplicate object description of jupyterhub.auth.Authenticator.admin_users, other instance in api/auth, use :noindex: for one of them
Makefile:64: recipe for target 'html' failed
make: *** [html] Error 2

which I don't fully understand. There is one definition https://github.com/jupyterhub/jupyterhub/blob/master/docs/source/api/auth.rst (the "other instance"). But where is the first definition?

betatim · 2020-04-15T22:14:05Z

The weird thing is that on master the build seems to work. Maybe rebase your branch on the latest master ?

rkdarst · 2020-04-15T22:40:30Z

On Wed, Apr 15, 2020 at 03:09:51PM -0700, Tim Head wrote: There is one definition https://github.com/jupyterhub/jupyterhub/blob/master/docs/source/api/auth.rst (the "other instance"). But where is the first definition?

My current hypothesis is that it is about autodoc somehow: it's parsing the docstring to turn it into docs, but then one of the `.. autoconfigurable:: Authenticator` includes the same thing. And actually. scratch that. Look at the current docs: https://jupyterhub.readthedocs.io/en/stable/api/auth.html jupyterhub.auth.Authenticator has *every* attribute, including admin_users, duplicated! So, there *is* some actual, existing duplication. I guess something is stricter on warnings?

rkdarst · 2020-04-15T22:55:07Z

https://github.com/jupyterhub/autodoc-traits/blob/master/autodoc_traits/autodoc_traits.py#L28

I added some "print" statements right before this line, and sure enough, admin_users is in both trait_members and members. Same for Authenticator and LocalAuthenticator, but not for PAMAuthenticator or DummyAuthenticator. The first two have :members: in the rst file. Anyone know details of which way is correct?

@willingc ?

- When run with the `:members:`, then traitlets traits are duplicated, because they are added to the autodoc list from both from `:members:` and this autodetection. (I think) - Discussed in at least jupyterhub/jupyterhub#2726. Currently causing JupyterHub docs to fail, because sphinx gives an error if there are duplicate autodoced traits and it is run with `-W`. - This seems to be started in a new version of sphinx, but we aren't completly sure. JH has been using the `-W` option since 2017. - I'm unsure if this is the right solution, but it works and gets me past these errors.

rkdarst · 2020-04-15T23:11:47Z

jupyterhub/autodoc-traits#1

After this, I am able to get the build to run further - but then there are other new errors. I guess sphinx really has gotten a lot stricter in more ways than one. Easy fix is removing -W from docs/Makefile...

betatim · 2020-04-16T05:56:46Z

Thanks for diving into autodocs! We introduced the -W flag a few years back to help keep the docs in "reasonable shape". At the time there the number of warnings was so big it was hard to tell if a change added more warnings or not. So the weary maintainer in me wants to keep -W to help us keep improving the quality of the docs or at least not adding new debt to them.

However we don't want to block progress with this. Maybe as a compromise we can limit ourselves to sphinx<3 for now? A next step would be to start a (hopefully) small coordinated effort to get the docs building with v3.

Curiously the last build on master still uses Sphinx-2.4.4 and it is green.

rkdarst · 2020-04-16T09:45:55Z

Discussion of docs failure moved to #3021.

betatim · 2020-04-18T16:21:51Z

Hurrah! All CI robots are happy again. Great work digging into sphinx and autotraits!

rkdarst · 2020-06-01T11:38:27Z

Now that CI works, would someone like to take a look at this? It is ready now, I've done multiple passes.

Some hints on what to check is at the top, but roughly a) is the placement within the docs good? Should it entirely be somewhere else?

... and b) fact-checking. I've done lots with Jupyter so I think there's not much wrong, but of course there's just so much that any other eyes will of course help.

betatim · 2020-06-02T06:09:14Z

docs/source/getting-started/what-is-jupyterhub.md

+## JupyterHub
+
+**JupyterHub** is the central piece that provides multi-user
+login. Despite this, the end user only briefly interacts with


"... provides multi-user login $thing" The sentence somehow ends abruptly. Could it be "...multi-user login capabilities." or "...functionality."?

docs/source/getting-started/what-is-jupyterhub.md

betatim · 2020-06-02T06:14:14Z

docs/source/getting-started/what-is-jupyterhub.md

+[reference](../reference/authenticators)) if the
+username/password is valid(&).  The authenticator can also return user
+groups and admin status of users, so that JupyterHub can do some
+higher-level management.  The authenticator returns a username(&),


Can we move the "The authenticator can also return..." sentence to the end of the paragraph? Lets tell people about the main thing the authenticator does "return a username" and then afterwards tell them about "but wait there is more".

docs/source/getting-started/what-is-jupyterhub.md

betatim · 2020-06-02T06:34:23Z

docs/source/getting-started/what-is-jupyterhub.md

+the user's notebook servers.  It actually isn't directly between,
+because the JupyterHub **proxy** relays connections between the users
+and their single-user notebook servers.  What this basically means is
+that the hub itself can shut down, and if the proxy can continue to


Missing words in the second half of the sentence?

I think the problem was an extra "if". Removed.

docs/source/getting-started/what-is-jupyterhub.md

rcthomas

@rkdarst as promised I took a read through of the docs here. I think this puts a lot of useful information in one place to try to dispel some folks' confusion about what's what. I made a few comments where some things may need to be clarified.

rcthomas · 2021-04-09T19:56:45Z

docs/source/getting-started/what-is-jupyterhub.md

@@ -0,0 +1,465 @@
+# What is Jupyter and JupyterHub?
+
+JupyterHub is not what you think it is.  Most things you think are


"is not" -> "may not be" ?

rcthomas · 2021-04-09T19:57:24Z

docs/source/getting-started/what-is-jupyterhub.md

+part of JupyterHub are actually handled by some other component, for
+example the spawner or notebook server itself, and it's not always
+obvious how the parts relate.  The knowledge contained here hasn't
+been assembled in one place before, and is essential to understand


rcthomas · 2021-04-09T19:58:59Z

docs/source/getting-started/what-is-jupyterhub.md

+In this document, we occasionally leave things out or bend the truth
+where it helps in explanation, and give our explanations in terms of
+Python even though Jupyter itself is language-neutral.  The "(&)"
+symbol highlights important points where there is more.


"where there is more."

Not sure what there is more of? Did the ending get chopped off or should it be "there is more to it" or something?

rcthomas · 2021-04-09T19:59:53Z

docs/source/getting-started/what-is-jupyterhub.md

+Before we get too far, let's remember what our end goal is.  A
+**Jupyter Notebook** is really nothing more than a Python(&) process
+which is getting commands from a web browser and displaying the output
+via that browser.  What the process actually sees can roughly like


"can roughly like" -> "can roughly be thought of like" ?

docs/source/getting-started/what-is-jupyterhub.md

rcthomas · 2021-04-09T20:05:57Z

docs/source/getting-started/what-is-jupyterhub.md

+JupyterHub: when someone wants a notebook server, the spawner allocates
+resources and starts the server.  The notebook server could run on the
+same machine as JupyterHub, on another machine, on some cloud service,
+or even more.  They can limit resources (CPU, memory) or isolate users


Not sure who "They" refers to, is this the administrator who configured the hub?

docs/source/getting-started/what-is-jupyterhub.md

rcthomas · 2021-04-09T20:14:15Z

docs/source/getting-started/what-is-jupyterhub.md

+opens in a separate tab.  It is traditionally started by `jupyter
+notebook`.
+
+Does anything need to be said here?


I think you don't have enough for 2 ###-level sections, maybe just smush them into the single user notebook server.

agree - I don't think we need much depth on the interfaces, maybe beyond mentioning that they'll live at different URL prefixes.

I removed the sections but not the text, someone else can do that later. Anyway, there is jupyter_server now, which will some sort of updates here, right?

rcthomas · 2021-04-09T20:16:22Z

docs/source/getting-started/what-is-jupyterhub.md

+
+## I want to...
+
+TODO: answers to common cross-layer questions.


I think the foregoing text actually does a good job of laying things out. You might want to omit this section for now to move forward with getting these docs integrated and then add this section later if things come up that aren't handled better any other way.

rcthomas · 2021-04-09T20:16:35Z

docs/source/getting-started/what-is-jupyterhub.md

+there are still plenty of details, implementations, and exceptions.
+When setting up JupyterHub, the first step is to consider the above
+layers, decide the right option for each of them, then begin putting
+everything together.


Maybe cite the JupyterCon talk?

Does that mean this one? https://www.youtube.com/watch?v=JxyKBNJnfVM
Since it's mine and perhaps old, I'll let someone else decide to add it.

willingc · 2021-04-10T00:52:12Z

Great to see this PR moving forward again. I will take a more detailed look. One thing that I'm thinking is that we come up with an improved title. "JupyterHub Concepts for New Users", "JupyterHub: A Conceptual Look"

@choldgraf added you to the reviewers as well.

choldgraf

Thanks @willingc for pinging me. I took a pass through the document and made several suggestions and comments.

I think that this content will be really helpful for people trying to wrap their heads around JupyterHub (and the broader Jupyter ecosystem). In my opinion we should do a round or two to make sure the content is of "MVP quality" and then get it in the docs, and iterate on it over time. The PR is big enough that I worry it'll get bogged down for a long time if we try to make it perfect. Does that make sense to others?

docs/source/index.rst

choldgraf · 2021-04-20T21:22:01Z

docs/source/getting-started/what-is-jupyterhub.md

@@ -0,0 +1,465 @@
+# What is Jupyter and JupyterHub?
+
+JupyterHub is not what you think it is.  Most things you think are


I think that we should avoid narrative flourishes like this. I think it's good writing and I enjoy it, but I don't think it's the most helpful for newcomers who are learning technical concepts. It may also make things harder to understand for non-native english speakers

e.g., rather than saying "Most things you think are part of JupyterHub are actually handled by some other component", we could say "JupyterHub is designed in a modular fashion, and much of its functionality is handled by pluggable components."

choldgraf · 2021-04-20T21:22:31Z

docs/source/getting-started/what-is-jupyterhub.md

+
+JupyterHub is not what you think it is.  Most things you think are
+part of JupyterHub are actually handled by some other component, for
+example the spawner or notebook server itself, and it's not always


can we link to sections where we discuss these in the docs?

choldgraf · 2021-04-20T21:22:55Z

docs/source/getting-started/what-is-jupyterhub.md

+part of JupyterHub are actually handled by some other component, for
+example the spawner or notebook server itself, and it's not always
+obvious how the parts relate.  The knowledge contained here hasn't
+been assembled in one place before, and is essential to understand


I think we can remove the "story of this document" stuff like "hasn't been assembled in one place before"

choldgraf · 2021-04-20T21:24:08Z

docs/source/getting-started/what-is-jupyterhub.md

+
+In this document, we occasionally leave things out or bend the truth
+where it helps in explanation, and give our explanations in terms of
+Python even though Jupyter itself is language-neutral.  The "(&)"


Does the & symbol have a functional use on the page? (e.g., does it create a hyperlink or something like that?) if not, I think we should either:

Turn these into footnotes or in-line links to other sections

Remove the & symbol because I think many readers won't really understand what to do with it

I got this idea form regex(7), which uses (!) to indicate a certain thing that is commonly said but shouldn't be spelled out each time (possibly non-portable decisions). I thought I had two choices: a) say thing that are not entirely true or the whole story or b) very often say "but there is more/this is a simplification". Perhaps the definition of (&) could be c) emphasized in an admonition, or d) it could be changed to "everything on this page may be simplified and expect some inaccuracies here". But I'd expect (d) could lead to lots of updates adding more details, making the page too long for it's main purpose...

docs/source/getting-started/what-is-jupyterhub.md

docs/source/index.rst

docs/source/installation-guide-hard.md

docs/source/getting-started/what-is-jupyterhub.md

choldgraf · 2021-04-20T21:55:22Z

also - somebody (@willingc ? @minrk ?) should enable ReadTheDocs builds for PRs so we can preview these changes! https://readthedocs.org/projects/jupyterhub/ (I don't have permissions)

manics · 2022-05-21T14:50:45Z

RTD build is active, this needs a rebase since docs/source/installation-guide-hard.md was moved into a separate repo, and some of the RTD deps may have changed

rkdarst · 2023-01-08T23:12:16Z

All of a sudden I'm reminded that this exists. I'll try to make more improvements to it based on the suggestions, but I'm not very good at following up with things these days, so if anyone wants to push things forward, by all means go ahead!

Does anyone know if rebasing (but not renaming the file) to resolve the conflicts above will mess up the per-line issues? My thought is "probably not" but just want to make sure...

choldgraf · 2023-01-09T13:45:57Z

No ideas re: rebasing but I usually find it to work sensibly for this kind of thing.

I am a big fan of merging this one quickly. It has a lot of useful information and I'd prefer merging something imperfect and then iterating, rather than having all of this knowledge locked up in a PR draft.

- Single-user servers are same you get with `jupyter notebook`. - Kernels by default in single-user server environment but don't have to be.

- Apparently recommonmark does intelligently uses links like sphinx+rst, and you shouldn't use `.html` on the links.

@betatim

Thanks to @betatim Co-authored-by: Tim Head <betatim@gmail.com>

@betatim

- Thanks to @betatim for the suggestions.

Co-authored-by: Chris Holdgraf <choldgraf@gmail.com>

docs/source/index.rst

rkdarst · 2023-01-09T20:48:11Z

docs/source/getting-started/what-is-jupyterhub.md

+obvious how the parts relate.  The knowledge contained here hasn't
+been assembled in one place before, and is essential to understand
+when setting up a sufficiently complex Jupyter(Hub) setup.
+
+This document was originally written to assist in debugging: very
+often, the actual problem is not where one thinks it is and thus
+people can't easily debug.  In order to tell this story, we start at
+JupyterHub and go all the way down to the fundamental components of
+Jupyter.


Suggested change

obvious how the parts relate. The knowledge contained here hasn't

been assembled in one place before, and is essential to understand

when setting up a sufficiently complex Jupyter(Hub) setup.

This document was originally written to assist in debugging: very

often, the actual problem is not where one thinks it is and thus

people can't easily debug. In order to tell this story, we start at

JupyterHub and go all the way down to the fundamental components of

Jupyter.

obvious how the parts relate.

Removing as per suggestion

rkdarst · 2023-01-09T20:51:22Z

docs/source/getting-started/what-is-jupyterhub.md

+company), or whitelist only the allowed users (e.g. your group's
+Github usernames).  Some other popular authenticators include:
+
+- **OAuthenticator** uses the standard OAuth protocol to verify users.


yes, we should (not just here but many places scattered around here), but to get it out I'll save that for later...

rkdarst · 2023-01-09T20:54:21Z

docs/source/getting-started/what-is-jupyterhub.md

+  what it does out of the box) and makes the hub not too dissimilar to
+  an advanced ssh server.
+
+There are many more advanced spawners:


I added a link to /reference/spawners, but didn't remove any from here since they somehow were chosen to represent the diversity of what's available... I'll let someone else pick what to remove.

rkdarst · 2023-01-09T20:58:28Z

docs/source/getting-started/what-is-jupyterhub.md

+
+The proxy always runs as a separate process to JupyterHub (even though
+JupyterHub can start it for you).  JupyterHub has one set of
+configuration options for the proxy addresses (`bind_url`) and one for


I think I got this right now (in the push that is upcoming), might be worth someone checking that it matches modern standards...

docs/source/getting-started/what-is-jupyterhub.md

rkdarst · 2023-01-09T21:01:49Z

docs/source/getting-started/what-is-jupyterhub.md

+opens in a separate tab.  It is traditionally started by `jupyter
+notebook`.
+
+Does anything need to be said here?


I removed the sections but not the text, someone else can do that later. Anyway, there is jupyter_server now, which will some sort of updates here, right?

rkdarst · 2023-01-09T21:02:58Z

docs/source/getting-started/what-is-jupyterhub.md

+there are still plenty of details, implementations, and exceptions.
+When setting up JupyterHub, the first step is to consider the above
+layers, decide the right option for each of them, then begin putting
+everything together.


Does that mean this one? https://www.youtube.com/watch?v=JxyKBNJnfVM
Since it's mine and perhaps old, I'll let someone else decide to add it.

… review

for more information, see https://pre-commit.ci

rkdarst · 2023-01-09T21:04:44Z

In the sprit of "getting it out and update", I made rebased to current upstream and did the quick revisions from the reviews. There are still more extensive things to do, but I'll leave that for someone else for later.

rkdarst · 2023-01-09T21:08:06Z

linkcheck failures seem unrelated

rkdarst force-pushed the conceptual-intro branch from e86f1a8 to c129841 Compare September 9, 2019 14:50

This was referenced Sep 24, 2019

Need of a more detailed user guide jupyterhub/batchspawner#162

Open

Proxy startup and configuration is required for init_spawners, right? #2749

Closed

rkdarst force-pushed the conceptual-intro branch from cece31d to 78f424f Compare April 13, 2020 22:53

rkdarst mentioned this pull request Apr 15, 2020

Remove duplicates in documenting trait members jupyterhub/autodoc-traits#1

Merged

betatim mentioned this pull request Apr 16, 2020

Support kubespawner running on a IPv6 only cluster #3020

Merged

This was referenced Apr 16, 2020

Fix user_row endblock in admin template #3015

Merged

Fix docs CI test failure: duplicate object description #3021

Merged

rkdarst force-pushed the conceptual-intro branch from 78f424f to 87704e3 Compare April 17, 2020 21:20

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

betatim reviewed Jun 2, 2020

View reviewed changes

docs/source/getting-started/what-is-jupyterhub.md Outdated Show resolved Hide resolved

rcthomas suggested changes Apr 9, 2021

View reviewed changes

willingc requested review from willingc and choldgraf April 10, 2021 00:44

rcthomas mentioned this pull request Apr 16, 2021

Add April 2021 notes jupyterhub/team-compass#401

Merged

choldgraf requested changes Apr 20, 2021

View reviewed changes

manics marked this pull request as draft May 21, 2022 14:37

manics closed this May 21, 2022

manics reopened this May 21, 2022

GeorgianaElena mentioned this pull request Oct 12, 2022

Sketching ideas for the content of a Community Handbook jupyterhub/team-compass#572

Closed

minrk mentioned this pull request Nov 24, 2022

migrate remaining rst docs to myst #4228

Closed

10 tasks

sgibson91 mentioned this pull request Nov 24, 2022

Create a new JupyterHub "landing page" at hub.jupyter.org jupyterhub/team-compass#444

Open

2 tasks

minrk mentioned this pull request Dec 7, 2022

convert remaining rst files to myst #4249

Merged

rkdarst and others added 7 commits January 9, 2023 21:52

what-is-jupyterhub initial draft

0774f1a

what-is-jupyterhub: clarifications (single-user and kernels)

7f980f6

- Single-user servers are same you get with `jupyter notebook`. - Kernels by default in single-user server environment but don't have to be.

what-is-jupyterhub: Full revision

2dccd07

what-is-jupyterhub: Fix links

4eb9a3a

- Apparently recommonmark does intelligently uses links like sphinx+rst, and you shouldn't use `.html` on the links.

Apply suggestions from code review

ffb4f51

Thanks to @betatim Co-authored-by: Tim Head <betatim@gmail.com>

docs/.../what-is-jupyterhub: Updates based on code review

92c8e08

- Thanks to @betatim for the suggestions.

Apply suggestions from code review

ff4b9c6

Co-authored-by: Chris Holdgraf <choldgraf@gmail.com>

rkdarst commented Jan 9, 2023

View reviewed changes

docs/source/getting-started/what-is-jupyterhub: Suggestions from code…

e3d645d

… review

rkdarst force-pushed the conceptual-intro branch from 51f8129 to e3d645d Compare January 9, 2023 21:03

[pre-commit.ci] auto fixes from pre-commit.com hooks

474df33

for more information, see https://pre-commit.ci

		@@ -0,0 +1,465 @@
		# What is Jupyter and JupyterHub?

		JupyterHub is not what you think it is. Most things you think are


		## I want to...

		TODO: answers to common cross-layer questions.

Jupyter(Hub) conceptual intro #2726

Are you sure you want to change the base?

Jupyter(Hub) conceptual intro #2726

Conversation

rkdarst commented Sep 9, 2019 • edited

consideRatio commented Sep 9, 2019

rkdarst commented Apr 13, 2020

betatim commented Apr 15, 2020

betatim commented Apr 15, 2020

rkdarst commented Apr 15, 2020 via email

rkdarst commented Apr 15, 2020

rkdarst commented Apr 15, 2020

betatim commented Apr 16, 2020 • edited

rkdarst commented Apr 16, 2020

betatim commented Apr 18, 2020

rkdarst commented Jun 1, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcthomas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

willingc commented Apr 10, 2021

choldgraf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

choldgraf commented Apr 20, 2021 • edited

manics commented May 21, 2022

rkdarst commented Jan 8, 2023

choldgraf commented Jan 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rkdarst commented Jan 9, 2023

rkdarst commented Jan 9, 2023

rkdarst commented Sep 9, 2019 •

edited

betatim commented Apr 16, 2020 •

edited

choldgraf commented Apr 20, 2021 •

edited