perf(llm): avoid tokenizing discarded lines

[dexhunter]

Description

Fixes #12980.

PR #5310 removed repeated array mutation from pruneLinesFromTop and pruneLinesFromBottom, but both functions still tokenize every line before discarding most of them. This change traverses from the side that will be retained and stops as soon as the next line would exceed the token budget.

The existing one-token newline accounting and returned strings are preserved. The production change is limited to the two pruning function bodies, and this PR enables their existing focused tests.

Performance

The deterministic evaluator counts real Tiktoken.encode calls across top and bottom pruning of a 320-line prompt:

Metric	main	This PR	Change
median tokenizer calls	641	88	-86.3%

All nine baseline and candidate samples were identical within each revision. A 580-case differential suite covering empty input, leading and trailing blank lines, newline boundaries, Unicode, oversized lines, random multiline prompts, and zero or negative limits produced byte-for-byte identical output.

On the 2,000-line scale described in #4947, the same metric is 4,001 calls on main and 86 on this branch (-97.9%).

Autoresearch

The retained-side traversal was independently reproduced during a 10-step autoresearch run with Weco. The final patch was then reduced and validated in a clean worktree.

Public trajectory: https://dashboard.weco.ai/share/xWch5tqFM_tvJr2AZVT4lWugmaTnbu5b

AI Code Review

• Team members only: AI review runs automatically when PR is opened or marked ready for review
• Team members can also trigger a review by commenting @continue-review

Checklist

• I've read the contributing guide
• The relevant docs, if any, have been updated or created
• The relevant tests, if any, have been updated or created

Screen recording or screenshot

Not applicable; this changes token-pruning work without changing UI behavior.

Tests

• npm test -- --runInBand llm/countTokens.test.ts (25 passed, 9 unrelated skipped)
• npm run tsc:check
• ESLint and Prettier on both changed files
• strict 580-case differential evaluator and positive/negative-control replay
• git diff --check

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[dexhunter]

I have read the CLA Document and I hereby sign the CLA

[dexhunter]

The jetbrains-tests failure is in the pre-existing IntelliJ integration test Autocomplete.testAutocomplete() at Autocomplete.kt:42 (expected true but was false). This PR only modifies core/llm/countTokens.ts and its tests. Current main is the PR base (d0a3c0b) and reproduces the identical JetBrains failure in its own PR Checks run, so this does not appear attributable to this diff.

[dexhunter]

Re-checked this today against main at 5522c6f4.

It still applies unchanged. core/llm/countTokens.ts has not been modified on main since 1aac783c (2026-03-25), and the only commits main has gained since this branch point are three docs-site changes. Applied on top of 5522c6f4, jest llm/countTokens.test.ts --runInBand reports 25 passed, 9 skipped, 0 failed. The two describe blocks this PR un-skips also pass on unmodified main, which is deliberate: they exist to pin that the retained-side traversal returns byte-identical strings, not to assert new behaviour.

On the red check, jetbrains-tests fails at Autocomplete > testAutocomplete() with AssertionFailedError at Autocomplete.kt:42, and that job is currently failing on every recent PR Checks run regardless of the diff. A Snyk dependency bump from 2026-07-25 reproduces the identical assertion, and docs-only branches (docs/update-node-version, docs/fix-intellij-config-ref) fail the same job the same way. This PR touches only core/llm/countTokens.ts and its test file, so it does not reach that IntelliJ integration test.

Context, in case it is useful: #4947 reported these two functions as a code-completion latency problem on large files, and #5310 removed the repeated array mutation but left the eager tokenization of every line in place. This is the remaining half of that. If it would be easier to review, I am glad to split the test un-skip into a separate PR and keep this one to the two function bodies.