Skip to content

Mistune 3 parser refactor + CI tooling#11

Open
philippb wants to merge 19 commits into
masterfrom
refactor/mistune3-api
Open

Mistune 3 parser refactor + CI tooling#11
philippb wants to merge 19 commits into
masterfrom
refactor/mistune3-api

Conversation

@philippb

@philippb philippb commented Jan 19, 2026

Copy link
Copy Markdown
Contributor

Goal

Upgrade to the Mistune 3 parser stack while preserving master behavior for structure diff output.

What Changed

  • Refactored parser internals for Mistune 3 token APIs while keeping public sdiff behavior stable.
  • Fixed inline token buffering regression (links/images/markers/linebreaks) that could duplicate rendered text.
  • Restored legacy-compatible parsing semantics for key markdown edge cases.
  • Added/expanded parser and compatibility-focused tests.
  • Aligned lint configuration with branch CI expectations.
  • Added coverage-focused tests to satisfy repository coverage gate.

Validation

  • make test passes on refactor/mistune3-api.
  • python -m flake8 --config .flake8 sdiff tests passes.
  • python -m coverage report -m passes with total coverage above fail_under=96.

Compatibility Notes

  • Objective is behavioral compatibility with master for structure diff, not markdown auto-formatting.
  • Any behavior differences intended as improvements should be called out explicitly before expectation changes.

@philippb philippb changed the title refactor/mistune3 api Mistune 3 parser refactor + CI tooling Jan 19, 2026
@philippb

Copy link
Copy Markdown
Contributor Author

Compatibility audit result: do not merge this revision.

I compared master at 12e7782208e4b458c8c4242882fda2377d9cba6b with this PR at ada6821b16f56c20490e8d2f44879c75d902cad3 in isolated Python 3.11.13 environments.

  • Baseline suite: 53/53 passed.
  • PR suite: 150/150 passed.
  • Direct black-box comparator: 23,185 identical public calls per environment across 1,149 texts and 21 unique pairs (33 recorded pair provenances).
  • Corpus: existing fixtures and test literals, README examples, curated Markdown/Zendesk/error boundaries, and deterministic block combinations.
  • Result: 11,462 exact signature mismatches. One is nondeterministic inspect.signature output; 11,461 are behavioral mismatches in parse trees, rendered bytes, diffs, extension behavior, or errors.

Representative differences:

input:    hello\nworld
baseline: hello\nworld
PR:       hello world

fenced-code rendering:
baseline: ```\ncode\n```
PR:       ``` code ```

MdParser().parse(None):
baseline: AttributeError("'NoneType' object has no attribute 'rstrip'")
PR:       []

Other reproduced changes include thematic breaks disappearing, autolink escaping changing, link-title quote normalization, legacy fixture byte changes, and reference definitions inside lists moving into the list item.

Compatibility confidence is 2/10, below the required 9/10 merge bar. The smallest next step is to define which legacy outputs are contractual versus explicitly approved improvements, retain a reduced dual-environment comparator as a regression gate, fix or adapt each unapproved mismatch class, and rerun the same exact comparison on the current base. I am leaving this PR open and unmerged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant