TokenizeContext.sliceSerialize for Token.type of setextHeading includes non-heading content from outside the range of [startLine, endLine]

[DavidAnson]

Initial checklist

• I read the support docs
• I read the contributing guide
• I agree to follow the code of conduct
• I searched issues and couldn’t find anything (or linked relevant results below)

Affected packages and versions

4.0.0

Link to runnable example

No response

Steps to reproduce

user@HOST micromark-setext % npm ls micromark
micromark-setext@ /Users/user/Documents/micromark-setext
└── micromark@4.0.0

user@HOST micromark-setext % cat issue.mjs 
import { parse } from "micromark";
import { postprocess } from "micromark";
import { preprocess } from "micromark";

const markdown = `
Text

Setext
======

Text
`;

const encoding = undefined;
const end = true;
const options = undefined;
const chunks = preprocess()(markdown, encoding, end);
const parseContext = parse(options).document().write(chunks);
const events = postprocess(parseContext);
for (const event of events) {
  const [ kind, token, context ] = event;
  if (kind === "enter") {
    const { type, start, end } = token;
    const { "line": startLine } = start;
    const { "line": endLine } = end;
    console.dir(`${type} (${startLine}-${endLine}): ${context.sliceSerialize(token)}`);
  }
}
user@HOST micromark-setext % node issue.mjs  
'lineEndingBlank (1-2): \n'
'content (2-2): Text'
'paragraph (2-2): Text'
'data (2-2): Text'
'lineEnding (2-3): \n'
'lineEndingBlank (3-4): \n'
'setextHeading (4-5): Text\n\nSetext\n======'
'setextHeadingText (4-4): Setext'
'data (4-4): Setext'
'lineEnding (4-5): \n'
'setextHeadingLine (5-5): ======'
'setextHeadingLineSequence (5-5): ======'
'lineEnding (5-6): \n'
'lineEndingBlank (6-7): \n'
'content (7-7): Text'
'paragraph (7-7): Text'
'data (7-7): Text'
'lineEnding (7-8): \n'
user@HOST micromark-setext %

Expected behavior

Note specifically this part of the output: 'setextHeading (4-5): Text\n\nSetext\n======'

While the start and end lines are correct, the output of sliceSerialize includes "Text\n\n" from lines 2 and 3 which is not part of the heading (confirmed by the associated setextHeadingText token which contains only "Setext").

Actual behavior

See above.

Runtime

Node v16

Package manager

npm v7

OS

macOS

Build and bundle tools

No response

[wooorm]

Here’s the code https://github.com/micromark/micromark/blob/4346e010d537b05f622e636d001bd25c1140f6a1/packages/micromark-core-commonmark/dev/lib/setext-underline.js#L62C1-L62C1.

Might be that markdown-rs does have this better. I ported most of its changes over to micromark, but not some of the more complex stuff handling events, which is exactly what has to happen for setext headings