indented magic commands block all magic commands

[carandraug]

Magic commands can be indented as described in #694. According to #717 this should be supported. However, I've found that is not the case. I'm using Jupytext 1.16.1 (currently that is the latest version) so maybe there's a regression?

In addition, I've found magic commands are converted fine as long as there are no indented magic commands. If there is any magic command in the python script, then all magic commands are not converted to the notebook.

Minimal example:

1. Showing input files:

$ cat test1.py   # has unindented magic command
# %%

# !ls
$ cat test2.py  # has magic indented command
# %%

if 1:
    # !ls
$ cat test3.py  # has indented and unindented magic command in two cells
# %%

# !ls

# %%

if 1:
    # !ls

2. Converting to python notebooks:

$ jupytext --version
1.16.1
$ jupytext --to ipynb test1.py 
[jupytext] Reading test1.py in format py
[jupytext] Writing test1.ipynb
($ jupytext --to ipynb test2.py 
[jupytext] Reading test2.py in format py
[jupytext] Writing test2.ipynb
$ jupytext --to ipynb test3.py 
[jupytext] Reading test3.py in format py
[jupytext] Writing test3.ipynb

3. Inspecting notebooks:

$ jq '.cells[].source' test1.ipynb  # unindented magic command was uncommented (correct)
[
  "\n",
  "!ls"
]
$ jq '.cells[].source' test2.ipynb  # indented magic command not uncommented (incorrect)
[
  "\n",
  "if 1:\n",
  "    # !ls"
]
$ jq '.cells[].source' test3.ipynb  # indented and unindented magic commands remain commented (incorrect and surprising)
[
  "\n",
  "# !ls"
]
[
  "\n",
  "if 1:\n",
  "    # !ls"
]

[carandraug]

As a workaround, I can pipe the output to

sed -E  's,^(\s*"\s*)# !,\1!,'

This is fine if the ! commands are a single line but it doesn't work for cases like:

# !some-command \
#   arg1 \
#   arg2