Skip to content

Fix crash rewriting relative document URI on ..// in querystring#314

Open
goosfrabba wants to merge 1 commit into
openzim:mainfrom
goosfrabba:fix-querystring-dotdot-relative-uri
Open

Fix crash rewriting relative document URI on ..// in querystring#314
goosfrabba wants to merge 1 commit into
openzim:mainfrom
goosfrabba:fix-querystring-dotdot-relative-uri

Conversation

@goosfrabba

Copy link
Copy Markdown

What

ArticleUrlRewriter.get_document_uri crashes when the document being rewritten (or a linked item) has a querystring that contains a .. segment, e.g. xtree.html?css=../../prg:

ValueError: '..' segment in 'host/common/nessComponents/xtree.html?css=../..' cannot be walked

This is the bug reported (and confirmed as legitimate by @benoit74) in openzim/warc2zim#380 — the rewriting code has since moved into this library, so the fix belongs here.

Why it happened

get_document_uri built the string path + "?" + querystring and handed it to PurePosixPath. PurePosixPath splits on / and treats .. as directory navigation, so any / or .. inside a querystring was interpreted as path structure. PurePosixPath.relative_to(..., walk_up=True) then raises ValueError as soon as it has to walk up through a .. segment coming from the querystring.

Fix

A querystring is part of the ZIM entry name (a leaf), not a navigable directory, so its content must not take part in the relative-path walk. The relative path is now computed from the path components only, and the querystring is re-appended (url-encoded together with the path) once the relative path is known.

Output is unchanged for querystrings that don't contain / or .. segments (verified by the existing test suite).

Tests

Added test_get_document_uri_querystring_segments covering .. in the document querystring (the crash case), .. in a linked item's querystring, and / in a querystring.

Before the fix (source reverted, test kept):

FAILED tests/rewriting/test_url_rewriting.py::...::test_get_document_uri_querystring_segments[dotdot_in_article_querystring]
E   ValueError: '..' segment in 'kiwix.org/common/sub/page.html?css=../..' cannot be walked

After the fix:

tests/rewriting/ .......................................... 477 passed

Full tests/rewriting/ suite passes; ruff==0.15.14 check + format clean.


Disclosure: implemented with the help of an AI coding assistant (Claude); verified locally with the included test.

Fixes openzim/warc2zim#380

`ArticleUrlRewriter.get_document_uri` computed the relative path between two
ZIM entries by feeding `path + "?" + querystring` to `PurePosixPath`, which
splits on `/` and interprets `..` segments as directory navigation. When the
document being rewritten (or a linked item) had a querystring containing `..`
(e.g. `xtree.html?css=../../prg`), `PurePosixPath.relative_to(..., walk_up=True)`
raised `ValueError: '..' segment ... cannot be walked`, aborting the scrape.

A querystring is part of the ZIM entry name (a leaf), not a navigable
directory, so its content must not take part in the relative-path walk. Compute
the relative path from the path components only, then re-append the querystring
(url-encoded together with the path) once the relative path is known. Existing
outputs are unchanged for querystrings without `/` or `..` segments.

Fixes openzim/warc2zim#380

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

@benoit74 benoit74 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the valuable PR.

Two things to fix to move PR forward:

  • you've fixed URL rewriting in Python which is good ; we need to have the same logic and fixes applied to URL rewriting in Javascript (see
    export function urlRewriteFunction(
    )
  • your comments currently explain why you've done these changes ; they should be meant to help future maintainer which barely know about this PR ; for instance "it used to raide ValueError ..." is quite verbose and I don't feel like it help future maintainer, which probably does not care what the code used to misbehave at some point in the past.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rewriter choke on a relative URL inside a querystring

2 participants