fix(jit): conditionally apply match/fullmatch anchoring workaround#97
Conversation
Some PCRE2 builds (e.g. libpcre2-8 10.46, and also 10.39 here) ignore PCRE2_ANCHORED/PCRE2_ENDANCHORED when passed as match-time options to pcre2_jit_match(). Rather than always paying for the post-JIT ovector check and possible pcre2_match() re-run, probe the linked library once at module load and only enable the workaround when the probe detects a non-compliant JIT. - Add jit_anchor_fixup_needed() which compiles two tiny probe patterns, enables JIT, and calls pcre2_jit_match() with ANCHORED/ENDANCHORED to see whether the returned ovector respects the flags. - Cache the result in an atomic int set during PyInit_pcre_ext_c(). - Apply the ovector verification + interpreter re-run in Pattern_execute() only when the probe returns non-compliant. - Expose _jit_anchor_fixup_needed() for tests/debugging. - Add tests/test_jit_anchoring.py from the upstream fork to ensure match()/fullmatch() remain correct with and without the fix active.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| } else if (mode == EXEC_MODE_FULLMATCH && jit_ovector[1] != offset_limit) { | ||
| jit_endanchor_uncertain = 1; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (!pattern_jit_get(self)) { | ||
| if (!pattern_jit_get(self) || jit_endanchor_uncertain) { | ||
| PCRE2_CALL_MAYBE_RELEASE_GIL(pcre2_match(self->code, | ||
| (PCRE2_SPTR)buffer, | ||
| exec_length, |
There was a problem hiding this comment.
🔍 Fullmatch end-anchor check relies on offset_limit equaling the subject end used by pcre2_match
The new FULLMATCH workaround compares jit_ovector[1] != offset_limit and, when unequal, re-runs pcre2_match() with PCRE2_ANCHORED | PCRE2_ENDANCHORED (pcre_ext/pcre2.c:1789-1803). This is correct in the truncation path (offset-limit-option disabled) because exec_length is set to offset_limit (pcre_ext/pcre2.c:1725-1730), so PCRE2_ENDANCHORED anchors exactly at offset_limit. However, when offset_limit_option_enabled() is true (PCRE2_USE_OFFSET_LIMIT path, pcre_ext/pcre2.c:1706-1723), exec_length remains the full subject length while offset_limit is byte_end (endpos). PCRE2's offset limit constrains the match start, and PCRE2_ENDANCHORED anchors to the end of the whole subject (exec_length), not to offset_limit. In that path the endpos-fullmatch semantics can be subtly wrong (both for the compliant-JIT ovector[1] comparison and for the fallback pcre2_match re-run). This is a pre-existing concern shared with the non-JIT path (!pattern_jit_get) rather than something introduced by this PR — behavior stays consistent with the interpreter path — but it is worth confirming that endpos-fullmatch with the offset-limit option active is actually exercised by the tests (they appear to use the truncation path).
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Thanks for the review. I checked this and it appears to work as intended because the match context has PCRE2_USE_OFFSET_LIMIT set with offset_limit whenever use_offset_limit_option is active. On the 10.39 system lib I verified:
p.fullmatch(b"abX", endpos=2) # -> Match span=(0, 2)
p.fullmatch(b"2025-10-08X", endpos=10) # -> Match span=(0, 10)
p.fullmatch(b"2025-10-08XX", endpos=11) # -> NoneI also added regression tests in tests/test_jit_anchoring.py covering fullmatch with endpos and trailing content, with and without JIT, and they pass locally and in CI. If there is a specific PCRE2 build where PCRE2_USE_OFFSET_LIMIT does not make PCRE2_ENDANCHORED respect offset_limit, the new tests will catch it.
There was a problem hiding this comment.
The concern about the PCRE2_USE_OFFSET_LIMIT re-run path is addressed in commit 21f563f (pushed after the review snapshot af76399):
- The
pcre2_match()re-run now usesinterpreter_length = offset_limitwheneverjit_endanchor_uncertainis set, regardless of whetherPCRE2_USE_OFFSET_LIMITis active. This makesPCRE2_ENDANCHOREDanchor at the requestedendposboundary even on builds wherePCRE2_USE_OFFSET_LIMITis interpreted as only a start constraint. - The new
TestEndposWithTrailingContenttests exercisefullmatch(..., endpos=N)with trailing content under both JIT and no-JIT, and CI is green. On the tested 10.39 runtimeoffset_limit_option_enabled()is true, so these tests do cover thePCRE2_USE_OFFSET_LIMITpath as well as the truncation path.
If there is a specific PCRE2 build where the PCRE2_USE_OFFSET_LIMIT re-run still misbehaves, the added tests will catch it.
…ndpos fallback When the JIT anchoring workaround needs to re-run fullmatch through pcre2_match(), pass only the requested endpos prefix as the subject length. This makes PCRE2_ENDANCHORED anchor at the intended boundary regardless of whether the active PCRE2 build treats PCRE2_USE_OFFSET_LIMIT as influencing end-anchored matches. Adds endpos regression tests for match/fullmatch/search/finditer with trailing content, pos+endpos, and multibyte text.
|
Pushed a fix and expanded coverage for this:
|
1 participant
Summary
Some PCRE2 builds (e.g. the system
libpcre2-8on this Ubuntu host, 10.39, and the upstream-reported 10.46) ignorePCRE2_ANCHORED/PCRE2_ENDANCHOREDwhen passed as match-time options topcre2_jit_match(). That madePattern.match()andPattern.fullmatch()report wrong-position matches or matches that did not reach the required end.Instead of always paying for the post-JIT ovector check and possible
pcre2_match()re-run, we now probe the linked library once duringPyInit_pcre_ext_c()and only enable the workaround when the probe detects a non-compliant JIT:jit_anchor_fixup_needed()compiles two tiny probe patterns, enables JIT, and callspcre2_jit_match()withPCRE2_ANCHORED/PCRE2_ENDANCHOREDto verify the returned ovector respects the flags.jit_anchor_fixup_needed_state).Pattern_execute()applies the ovector verification + interpreter re-run only whenjit_anchor_fixup_needed()returns 1.pcre_ext_c._jit_anchor_fixup_needed()is exposed so tests/debugging can see whether the workaround is active.The regression tests from the fork (
tests/test_jit_anchoring.py) are included; they pass with and without the workaround active.Verification
Local build against
libpcre2-810.39:python setup.py build_ext --inplacesucceeds.pcre_ext_c._jit_anchor_fixup_needed()returns1for this library, confirming the probe flags the non-compliant JIT.python -m pytest tests/test_jit_anchoring.py -vpasses (16 tests).test_basic,test_bytes,test_core,test_jit,test_utf8,test_pattern,test_module,test_re_compat,test_errors,test_flags,test_api_parity,test_version,test_memory(199 passed, 10 skipped).Link to Devin session: https://app.devin.ai/sessions/3cf75f5f1108416eb288c8dcc282c0e4
Requested by: @Qubitium