Fix unexpected whitespace removal outside line range #1102

limwz01 · 2023-06-06T02:26:47Z

save the whitespace prefix in the pytree node even for those occurring with comments
disable the comment splicer's removal of trailing whitespace
add back the whitespace before emitting unformatted line
remove trailing whitespace in comments for enabled lines later when printing

TODO:

extensive testing is necessary as this method is like a hack
this still will remove trailing whitespace in line just before line range and will ignore trailing whitespace in last line of line range

* save the whitespace prefix in the pytree node even for those occurring with comments * disable the comment splicer's removal of trailing whitespace * add back the whitespace before emitting unformatted line * remove trailing whitespace in comments for enabled lines later when printing TODO: this still will remove trailing whitespace in line just before line range and will ignore trailing whitespace in last line of line range

bwendling · 2023-10-25T17:21:38Z

Could you give an example of what this is fixing?

limwz01 · 2023-10-25T17:28:26Z

The example is in the linked issue.

Spitfire1900 · 2024-01-20T20:16:51Z

Resolve conflicts

limwz01 · 2024-01-22T07:10:39Z

After a long while of using this with only the problems mentioned in my first post, I've found a further problem that occurs with the following:

def main():
    if 1:
        if 1:
            pass

        # extra
        elif 1:
            pass
    pass

Applying the modified yapf on anywhere except including the extra comment line, we will get extra trailing space on the line before the extra comment line. I've not figured out the bug yet.

bwendling

This needs testcases.

bwendling · 2024-01-22T20:55:45Z

yapf/yapflib/reformatter.py

@@ -19,18 +19,19 @@
  Reformat(): the main function exported by this module.
 """

+import json


Please remove.

bwendling · 2024-01-22T20:57:21Z

yapf/yapflib/reformatter.py

-
-from yapf_third_party._ylib2to3 import pytree
-from yapf_third_party._ylib2to3.pgen2 import token
+import sys


Please remove.

bwendling · 2024-01-22T20:57:49Z

yapf/yapflib/reformatter.py


 from yapf.pytree import pytree_utils
 from yapf.yapflib import format_decision_state
 from yapf.yapflib import format_token
 from yapf.yapflib import line_joiner
 from yapf.yapflib import style
+from yapf_third_party._ylib2to3 import pytree
+from yapf_third_party._ylib2to3.pgen2 import token


Please remove.

bwendling · 2024-01-22T20:59:17Z

yapf/yapflib/reformatter.py

@@ -72,6 +73,21 @@ def Reformat(llines, lines=None):
    if lline.disable or _LineHasContinuationMarkers(lline):
      _RetainHorizontalSpacing(lline)
      _RetainRequiredVerticalSpacing(lline, prev_line, lines)
+      if not _LineHasContinuationMarkers(lline):
+        for token in lline.tokens:
+          # print(


Please remove the debugging comments.

bwendling · 2024-01-22T20:59:59Z

yapf/pytree/comment_splicer.py

+                comment_prefix,
+                comment_lineno,
+                comment_column,
+                "",


Use single-quotes for consistency.

bwendling · 2024-01-22T21:02:01Z

yapf/pytree/pytree_unwrapper.py

@@ -85,6 +85,7 @@ class PyTreeUnwrapper(pytree_visitor.PyTreeVisitor):
  def __init__(self):
    # A list of all logical lines finished visiting so far.
    self._logical_lines = []
+    self.prefix = ""


Single quotes.

bwendling · 2024-01-22T21:02:14Z

yapf/pytree/pytree_unwrapper.py

    if leaf.type in _WHITESPACE_TOKENS:
      self._StartNewLine()
+      if leaf.type == grammar_token.NEWLINE:
+        self.prefix += "\n"


Single quotes.

bwendling · 2024-01-22T21:02:21Z

yapf/pytree/pytree_unwrapper.py

-          format_token.FormatToken(leaf, pytree_utils.NodeName(leaf)))
+          format_token.FormatToken(leaf, pytree_utils.NodeName(leaf),
+                                   self.prefix))
+      self.prefix = ""


bwendling · 2024-01-22T21:03:59Z

yapf/pytree/comment_splicer.py

@@ -206,6 +223,7 @@ def _VisitNodeRec(node):
 def _CreateCommentsFromPrefix(comment_prefix,
                              comment_lineno,
                              comment_column,
+                              prefix,


Add a default prefix='' so you only have to pass it in when it's not empty.

bwendling · 2024-01-22T21:04:14Z

yapf/pytree/comment_splicer.py

+        if not (index < len(lines) and lines[index].lstrip().startswith("#")):
+          break
+
+      # print(repr(comment_block), repr(prefix), file=sys.stderr)


limwz01 mentioned this pull request Jun 6, 2023

Trailing whitespace changes are made outside of --lines #613

Open

This was referenced Dec 15, 2023

Format Selection is formatting the entire file EeyoreLee/vscode-extension-yapf#101

Closed

[Bug] Some space out of range strip when using -l argument. #1190

Open

Merge branch 'main' into whitespace_line_range_fix

d3ea45b

bwendling reviewed Jan 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix unexpected whitespace removal outside line range #1102

Fix unexpected whitespace removal outside line range #1102

limwz01 commented Jun 6, 2023

bwendling commented Oct 25, 2023

limwz01 commented Oct 25, 2023

Spitfire1900 commented Jan 20, 2024

limwz01 commented Jan 22, 2024 •

edited

bwendling left a comment

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

bwendling Jan 22, 2024

Fix unexpected whitespace removal outside line range #1102

Are you sure you want to change the base?

Fix unexpected whitespace removal outside line range #1102

Conversation

limwz01 commented Jun 6, 2023

bwendling commented Oct 25, 2023

limwz01 commented Oct 25, 2023

Spitfire1900 commented Jan 20, 2024

limwz01 commented Jan 22, 2024 • edited

bwendling left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

limwz01 commented Jan 22, 2024 •

edited