executablebooks/markdown-it-py v4.3.0

on GitHub

This release fixes two quadratic-time parsing paths that let crafted input stall rendering for minutes (security),
aligns several parsing rules with CommonMark 0.31.2 and markdown-it 15.0.x,
turns parser rules that never advance into a RuntimeError instead of an infinite loop,
makes rendering linear on PyPy and free-threaded CPython, and adds support for Python 3.14.
Python 3.10 is still supported.
If you write plugins or render rules, or edit SyntaxTreeNode trees, please read the upgrade notes at the end.

🔒 Security

• 👌 Fix quadratic-time inline parsing of long runs of characters that start an inline rule without completing one (such as &, <, ~ or incomplete entities), which let a few hundred kilobytes of input stall rendering for seconds to minutes, in #423, thanks to @hdimer (#411)
• 👌 Reject unterminated inline HTML (<!--, <?, <![CDATA[, <!X) in constant time when html is enabled (as in the default commonmark preset), instead of rescanning to the end of the input for every opener, in #428

✨ New

• ✨ Add --enable-tables to the CLI for file, standard input and interactive parsing in #425, thanks to @user01010111 (#422); requested by @hartwork in #352
• ⬆️ Add support for Python 3.14 in #462

👌 Improved

• 👌 Raise RuntimeError instead of looping forever when a block or inline rule reports a match without advancing (or moves backwards), or when no block rule matches a line, as markdown-it does, in #444, thanks to @rupayon123, @hukkin (#336) and @inchang-ing (#443); requested by @acaprino in #441
• 👌 Fix quadratic-time HTML rendering on PyPy, GraalPy and free-threaded CPython by joining the rendered fragments once, in #458, thanks to @DicksonWu654
• 👌 Make StateBase.src a plain attribute, for about 4–11% faster parsing on CPython, in #450, thanks to @deadstrobe5
• 👌 Build block line markers with one scan per line instead of one step per character in #449, thanks to @John-Cusack
• 👌 Name the Windows end-of-input keys (Ctrl-Z then Enter) in the CLI's interactive prompt and -h text in #463

🐛 Fixed

• 🐛 Fix HTML blocks of types 1–5 (comments, processing instructions, declarations, CDATA, script/pre/style/textarea) ending at a blank line inside a list item or blockquote, porting markdown-it 14.2.0, in #406, thanks to @binggao1230; reported by @Jayman2000 in #377
• 🐛 Fix HTML block starts per CommonMark 0.31.2: lowercase declarations such as <!doctype html> start a block and interrupt a paragraph, and tag names match case-insensitively in ASCII only, in #440, thanks to @rupayon123; reported by @p5ych0 in #437
• 🐛 Fix code spans rendered as literal backticks after an unclosed link or image label, and count only U+0020 spaces (not tabs or no-break spaces) when deciding whether to strip a code span's padding, porting markdown-it 15.0.1, in #439, thanks to @SparkUiX; reported by @felzab in #438
• 🐛 Trim and split with the CommonMark whitespace set instead of Python's, so U+001C–U+001F and U+0085 are no longer dropped from paragraphs, headings, table cells and fence info strings, and distinct reference labels no longer match each other, in #433, thanks to @Nexory (#418)
• 🐛 Preserve IPv6 host brackets (e.g. http://[::1]:8000/) in normalized link and image URLs, porting markdown-it 15.0.1, in #459, thanks to @Jinyong-Li
• 🐛 Fix task-list detection consuming a literal [ ]/[x] at the start of an indented code block in a list item in #451, thanks to @AryanRG
• 🐛 Fix an IndexError in the html_block and heading rules at the end of the input (a table in a blockquote that ends on >) in #416, thanks to @saket3395; reported by @WilcoPancras in #415
• 🐛 Keep backslash escapes and entities in image alt text (![a \* b](/u) gives alt="a * b"), porting markdown-it 15.0.0, in #446, thanks to @gyanu2507; reported by @kdeldycke in #445
• 🐛 Keep code span text in image alt text, porting markdown-it 15.0.0, in #460, thanks to @jayhemnani9910
• 🐛 Keep hard breaks and inline HTML in image alt text in #409, thanks to @chuenchen309
• 🐛 Keep edits to the children of inline and image nodes in SyntaxTreeNode.to_tokens() in #453, thanks to @KennyMcSimpson; edited tokens are copied, so the token stream the tree was built from is left unchanged (#464)
• 🐛 Fix rule chains staying stale after Ruler.enable, disable or enableOnly raised on an unknown rule name in #457, thanks to @IlyaKhalafi; reported in #456
• 🐛 Fix MarkdownIt.enable() and disable() applying a generator of rule names only to the core chain in #447, thanks to @muneikh
• 🐛 Fix the CLI's interactive mode joining input lines with an extra newline, which split every line into its own paragraph and broke hard line breaks, in #425; reported by @jamesquilty in #172
• 🐛 CLI helpers create a default parser only when none is given in #455, thanks to @Quinnod345

📚 Documentation

• 📚 Add a "Third-party renderers" section (mdformat, mdit_plain, MyST-Parser) and document RendererProtocol for writing your own renderer, in #463
• 📚 Document that MarkdownIt passes itself to renderer_cls, unlike markdown-it's renderer constructor, in #421, thanks to @user01010111
• 📚 Clarify that the name given to add_render_rule must match a token type in #432, thanks to @hikmetba-bit
• 📚 Fix typos and use HTTPS for the CommonMark spec links in #402, #407 and #408, thanks to @nikolas, @maxtaran2010 and @deeferentleeg

🔧 Maintenance

• 🔧 Publish to PyPI with trusted publishing and attestations, uploading distributions built with python -m build and checked on every pull request, in #397 and #461
• 🔧 Include tests/ in the sdist (next to tox.ini and CHANGELOG.md), so packagers can run the test suite from it, in #426, thanks to @BetterAndBetterII (#420), and #461; requested by @mtelka in #261
• 🔧 Move coverage and pytest-cov from the testing extra to a new coverage extra, and skip the linkify tests when linkify-it-py is not installed, in #463; requested by @befeleme in #195
• 🔧 Read the CommonMark spec fixtures as UTF-8, so the tests pass on Windows with a non-UTF-8 locale, in #454, thanks to @Aschenbath
• 🔧 Drop the optional dependency on the unmaintained commonmark package from the compare extra in #427, thanks to @nikolas (#401)
• 🔧 Build the documentation on Python 3.13 and Sphinx 9 in #462

Upgrade notes

• Rules that do not advance now raise.
  A block or inline rule that returns True without moving state.line or state.pos forward now raises RuntimeError (block rule didn't increment state.line, inline rule didn't increment state.pos), where it used to hang.
  So does a line that no block rule accepts, which is only possible with the paragraph rule disabled (none of the block rules matched).
• Image children hold joined text tokens.
  As in markdown-it 15, text_join now also joins the text_special tokens among an image token's children, so code that reads alt text from them gets escaped characters and entities as plain text tokens (a nested image's own children are still not joined, as upstream).
• Render rules must return str.
  The rendered fragments are now joined with str.join, so a rule that returns a str subclass (such as markupsafe.Markup) gives a plain str from render(), and any non-str result (even one that defines __radd__, which 4.2.0 let through) raises TypeError when the fragments are joined.
• SyntaxTreeNode.to_tokens() follows the tree.
  The tree's children are the source of truth: a change made directly to a token's children list no longer appears in the output, and to_tokens() does not modify the token stream the tree was built from.
• Rule names given as a generator to MarkdownIt.enable() or disable() are validated like a list, so an unknown name now raises ValueError.
• Internals plugins may touch.
  StateInline.pending is now a property backed by a buffer: state.pending += text still works, and the new state.append_pending(text) appends in constant time.
  HTML_TAG_RE (markdown_it.common.html_re) and DIGITAL_RE and NAMED_RE (markdown_it.rules_inline.entity) no longer start with ^, because they are applied with .match(src, pos); code that calls .search() on them now matches anywhere in the string.
  The private StateBase._src and _srcCharCode attributes are gone (srcCharCode itself is still there, and still deprecated).
• Packaging.
  The testing extra no longer installs coverage and pytest-cov: install .[testing,coverage] to run the tests with --cov.
  The sdist now contains tests/, and no longer contains repository files such as .github/, .pre-commit-config.yaml and scripts/.

Full Changelog: v4.2.0...v4.3.0

Full changelog: v4.2.0...v4.3.0