Tutorials infrastructure: switch to text-based notebook workflow #59

alexdesiqueira · 2021-06-23T21:48:42Z

Changes the workflow to use a text-based format for Jupyter notebooks and sphinx+myst for running and testing them, continuing changes added by @jni.
Kindly stolen from @rossbar's numpy-tutorials.

pep8speaks · 2021-06-23T22:38:27Z

Hello @alexdesiqueira! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-12-10 07:09:55 UTC

alexdesiqueira · 2021-06-23T22:54:56Z

Website looks like this so far:

alexdesiqueira · 2021-06-23T23:53:58Z

How it looks so far:

alexdesiqueira · 2021-06-23T23:55:25Z

Alright, I think that's it as a starting point.
We need to discuss how the tutorials will be structured, but the main machinery is here already.

jni

@alexdesiqueira in addition to the minor comments below, why are the existing tutorials not migrated here? I would prefer that ongoing PRs only add content rather than removing it! 😂

jni · 2021-06-29T00:54:23Z

.github/workflows/deploy.yml

      - name: GitHub Pages action
        if: github.repository == 'scikit-image/skimage-tutorials' && github.ref == 'refs/heads/main' && github.event_name == 'push'
        uses: peaceiris/actions-gh-pages@v3.6.1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
-          publish_dir: ./_build/html
+          publish_dir: ./website/_build/html


Why this move? I don't love it...

I felt it'd make things more organized. I don't mind, though. Would you like me to leave it as it was?

_toc.yml

content/00_images_are_arrays.md

alexdesiqueira · 2021-09-13T18:49:38Z

why are the existing tutorials not migrated here? I would prefer that ongoing PRs only add content rather than removing it!

@jni* We're not removing anything yet! 😂 If I convert everything now, this PR will be huge; the idea here was only to add the infrastructure — the tutorial(s) presented here are examples 🙂

alexdesiqueira · 2021-12-02T22:49:12Z

Thank you @stefanv!

stefanv · 2021-12-03T00:49:39Z

Something isn't right with the caching: it rebuilds all notebooks on each make.

stefanv · 2021-12-03T00:50:47Z

The Makefile is definitely not right... refers to a lessons dir that does not exist, directories are off, etc.

alexdesiqueira · 2021-12-03T00:56:42Z

The Makefile is definitely not right... refers to a lessons dir that does not exist, directories are off, etc.

@stefanv thanks; gonna check that.

stefanv · 2021-12-03T01:07:35Z

I will push up a patch, then you can work on top of that.

stefanv · 2021-12-03T01:25:32Z

Actually, the whole thing is completely broken :)

alexdesiqueira · 2021-12-03T03:57:00Z

😂 Sorry about that.

stefanv · 2021-12-03T06:43:26Z

OK, I pushed up a new Makefile and switched to using jupytext to render directly. Jupytext correctly passes through failing cells with the raises-exception tag. Other tags (remove-input, remove-output) are handled by myst_nb.

Process now is:

Files are stored as myst notebooks (.md)
Jupytext converts to ipynb and executes inline
myst_nb renders those notebooks inside Sphinx

There's no caching, but make will not rebuild a notebook unless it has changed. I.e., if a .md source file is modified, the notebook is re-rendered and re-executed.

There will be failures on CI. These are real failures that need to be fixed.

rossbar · 2021-12-14T19:54:42Z

Adding my two cents from afar...

Though the build system is a departure from the jupyter-book or pure-sphinx workflows, it LGTM and shouldn't make a difference to tutorial authors one way or the other once everything is working. From trying it out, there are two main features that you get from the sphinx/jb workflows that are missing here:

For whatever reason, the execution caching doesn't seem to be working. It seems to me based on the Makefile that the notebooks should only be executed on a re-build if their content has changed, but in practice all the notebooks are re-executed on each build. Maybe the myst-nb sphinx extension is touch-ing or otherwise modifying the timestamps of the generated .ipynb files during the sphinx build? It's not immediately obvious to me why this isn't working...
Another feature for the build system that would be nice would be the ability to continue building even after execution of a notebook fails. Currently if the jupytext --execute fails, then the full make command exits and the outputs aren't generated. With the sphinx workflow, the default behavior is that execution failures are treated as warnings and the full execute-and-convert-to-html procedure continues to completion. This is a nice feature because authors/reviewers can see the execution failures in the output html, which IME has helped a lot with reviewing etc. in numpy-tutorials and nx-guides.

stefanv · 2021-12-14T21:04:31Z

@rossbar do you want to update the build system to use the Sphinx workflow? Fine with me, as long as we have the ability to error on failed build for CI purposes.

stefanv · 2021-12-14T21:05:05Z

I am a bit surprised at the rebuilds. That's the whole point of make, but I observe the same thing.

rossbar · 2021-12-14T21:19:55Z

do you want to update the build system to use the Sphinx workflow?

Sure, I'm happy to do so, if for no other reason than to provide a point of comparison.

Fine with me, as long as we have the ability to error on failed build for CI purposes.

Yeah this is a bit of a sore point: I do this with the SPHINXOPTS=-W --keep-going flags: -W to elevate warnings to errors, and --keep-going to ensure that everything runs to completion. The downside is that it elevates all sphinx warnings to errors, not just execution failures. FWIW this is a requested enhancement in myst-nb, see e.g. executablebooks/MyST-NB#248

alexdesiqueira · 2022-02-01T02:05:14Z

Superseded by #66; closing this one in favor of #66 then. Thank you @rossbar!

Adding jupytext options to Makefile

7ee54d5

alexdesiqueira added the wip Work in progress label Jun 23, 2021

alexdesiqueira added 6 commits June 23, 2021 14:57

Migrating from the "old" format: toc is not a mapping

0803983

Moving builds to their folder

55ff765

Building site instead of book

f3bf746

Adding Makefile on website

de2d146

Removing website Makefile refs

e345a61

Adding website/conf.py

1d4e64c

alexdesiqueira added 3 commits June 23, 2021 15:42

Starting index.md

37c53d5

Adding Indices and tables to index.md

3086296

Ignoring website local builds

0c22b39

alexdesiqueira linked an issue Jun 23, 2021 that may be closed by this pull request

Migrating tutorials to website todo #57

Open

4 tasks

alexdesiqueira added 5 commits June 23, 2021 15:57

An example — testing jupytext conversion

568fe59

Adding example to toctree on index.md

2437a6a

Creating a link from website/images to ../images

831741f

Testing toc structure

e818bc4

Organizing folders

747d842

alexdesiqueira removed the wip Work in progress label Jun 23, 2021

alexdesiqueira requested review from jni and stefanv June 23, 2021 23:55

Changing intro.md

f217e16

jni reviewed Jun 29, 2021

View reviewed changes

hmaarrfk mentioned this pull request Sep 13, 2021

Pip dependency resolution conflict with requirements.txt #63

Open

rossbar mentioned this pull request Sep 13, 2021

Update toc using jb toc migrate. #65

Merged

Testing python instead of ipython3 on {code-cell}s

9244886

alexdesiqueira requested a review from jni December 3, 2021 00:36

stefanv force-pushed the notes_to_text branch from bb2a1e8 to 560d599 Compare December 3, 2021 06:36

Refactor the build system

d4325e7

stefanv force-pushed the notes_to_text branch from 560d599 to d4325e7 Compare December 3, 2021 06:39

stefanv added 2 commits December 2, 2021 22:58

Fix skdemo

9edcefd

Fix up color_and_exposure module

ef683cc

stefanv force-pushed the notes_to_text branch from 3ca72c8 to ef683cc Compare December 3, 2021 06:59

stefanv added 8 commits December 3, 2021 00:51

Fix image links

f95be21

Fix notebooks; fix skdemo; fix notebook build path

28d0e4c

Require imagecodecs for reading tiffs

4103ab0

Fix filters imports

db180a5

threshold_adaptive is not threshold_local

64cea04

Final fixes to adv0_chromosomes

bfcc8c5

Fix adv1_lesion_quantification

ad248c4

Fix adv2_microarray

6edeb56

rossbar mentioned this pull request Dec 15, 2021

Alternative build system to #59 #66

Open

alexdesiqueira closed this Feb 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tutorials infrastructure: switch to text-based notebook workflow #59

Tutorials infrastructure: switch to text-based notebook workflow #59

alexdesiqueira commented Jun 23, 2021

pep8speaks commented Jun 23, 2021 •

edited

alexdesiqueira commented Jun 23, 2021

alexdesiqueira commented Jun 23, 2021

alexdesiqueira commented Jun 23, 2021

jni left a comment

jni Jun 29, 2021

alexdesiqueira Sep 13, 2021

alexdesiqueira commented Sep 13, 2021 •

edited

alexdesiqueira commented Dec 2, 2021

stefanv commented Dec 3, 2021

stefanv commented Dec 3, 2021

alexdesiqueira commented Dec 3, 2021

stefanv commented Dec 3, 2021

stefanv commented Dec 3, 2021

alexdesiqueira commented Dec 3, 2021

stefanv commented Dec 3, 2021

rossbar commented Dec 14, 2021

stefanv commented Dec 14, 2021

stefanv commented Dec 14, 2021

rossbar commented Dec 14, 2021

alexdesiqueira commented Feb 1, 2022

Tutorials infrastructure: switch to text-based notebook workflow #59

Tutorials infrastructure: switch to text-based notebook workflow #59

Conversation

alexdesiqueira commented Jun 23, 2021

pep8speaks commented Jun 23, 2021 • edited

Comment last updated at 2021-12-10 07:09:55 UTC

alexdesiqueira commented Jun 23, 2021

alexdesiqueira commented Jun 23, 2021

alexdesiqueira commented Jun 23, 2021

jni left a comment

Choose a reason for hiding this comment

jni Jun 29, 2021

Choose a reason for hiding this comment

alexdesiqueira Sep 13, 2021

Choose a reason for hiding this comment

alexdesiqueira commented Sep 13, 2021 • edited

alexdesiqueira commented Dec 2, 2021

stefanv commented Dec 3, 2021

stefanv commented Dec 3, 2021

alexdesiqueira commented Dec 3, 2021

stefanv commented Dec 3, 2021

stefanv commented Dec 3, 2021

alexdesiqueira commented Dec 3, 2021

stefanv commented Dec 3, 2021

rossbar commented Dec 14, 2021

stefanv commented Dec 14, 2021

stefanv commented Dec 14, 2021

rossbar commented Dec 14, 2021

alexdesiqueira commented Feb 1, 2022

pep8speaks commented Jun 23, 2021 •

edited

alexdesiqueira commented Sep 13, 2021 •

edited