idna 2.10.post1: backport CVE-2024-3651

[icanhasmath]

ActiveState security release of idna 2.10 — the last upstream release supporting Python 2.7 (3.0 requires 3.5+).

CVE	Level	Fixed upstream in	Status
CVE-2024-3651 (GHSA-jjg7-2v4v-x38h)	Moderate	3.7	Patched

The vulnerability

A crafted argument to idna.encode() costs O(n²). Measured on the pristine 2.10 base with a label of n joiners:

n	time
250	0.043 s
500	0.159 s
1000	0.565 s
2000	2.085 s
4000	8.431 s

4× per doubling — a few kilobytes of input stalls the process. After the fix: 0.0001 s, flat.

RFC 5892 Appendix A.1 requires a ZWNJ to match

(Joining_Type:{L,D})(Joining_Type:T)*ZWNJ(Joining_Type:T)*(Joining_Type:{R,D})

so the scan on each side of the joiner must stop at the first codepoint that is neither Transparent nor a valid terminator. valid_contextj() instead fell through on anything that was not Transparent and not L/D (resp. R/D) and kept scanning to the end of the label. That was both the performance bug and a correctness bug: labels with a Non_Joining codepoint between the joiner and its context were wrongly accepted.

Two changes are required together

1. idna/core.py — add the else: break arm to both scan loops, as upstream did.

2. idna/idnadata.py — regenerate joining_types. This is not optional. The table was built from ArabicShaping.txt, which lists joining types only for the Arabic script, so it held just 4 Transparent entries out of 775. With only change 1 applied, every combining mark would terminate the scan and valid IDNs would be rejected. The table now derives from extracted/DerivedJoiningType.txt: 2722 entries (T=1997, D=586, R=130, L=5, C=4).

Regenerated at Unicode 13.0.0 — the version 2.10's other tables are built on — rather than lifting upstream 3.7's Unicode 15.1.0 table, so the release stays internally consistent. Verified to reproduce every one of 2.10's existing D/R/L/C entries exactly (586/130/5/4, zero diffs). The 46 explicit Non_Joining (U) entries are dropped, matching upstream: with the else: break arm, an absent entry and an explicit U now behave identically.

3. tools/idna-data — point _load_arabicshaping() at extracted/DerivedJoiningType.txt and handle its range syntax, so the committed table is reproducible from the generator.

Testing (Python 2.7.18)

The full IdnaTest.txt conformance corpus runs 7787 tests, 1227 pre-existing skips — identical to the pristine 2.10 base, so no behavioural drift. Note this corpus is driven by load_tests, which pytest ignores; run it with python -m unittest discover -s tests -t ..

Two regression tests added (7789 total). Both fail on the pristine base and pass with the fix. Test literals use \uXXXX escapes so tests/test_idna.py stays pure ASCII and needs no PEP 263 declaration. Both idna/ and tests/ compile under Python 2 and Python 3.

Upstream fix: kjd/idna commit 5beb28b9dd77 (PR kjd#172).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Backports the upstream fix for CVE-2024-3651 to the Python-2.7-compatible idna 2.10 line by correcting valid_contextj()’s joining-type scan behavior and regenerating the joining-types data to preserve correctness while eliminating the O(n²) worst-case.

Changes:

• Fix valid_contextj() to stop scanning at the first non-Transparent, non-terminating joining type (eliminating quadratic behavior and correcting acceptance rules).
• Regenerate joining_types from extracted/DerivedJoiningType.txt and update the generator to load that source (including range syntax handling).
• Add regression tests for correctness and non-quadratic behavior; bump version to 2.10.post1 and document the security release in HISTORY.rst.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tools/idna-data	Switch joining-type source to DerivedJoiningType.txt and support range parsing so regenerated data is reproducible.
tests/test_idna.py	Add regression coverage for joining-type scan termination and guard against quadratic regressions.
idna/package_data.py	Bump library version to 2.10.post1.
idna/idnadata.py	Regenerate joining_types table to include correct Transparent entries needed for ContextJ scanning.
idna/core.py	Implement early-termination (else: break) in ContextJ scan loops to match RFC 5892 expectations and avoid O(n²).
HISTORY.rst	Record the security fix and release notes for 2.10.post1.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.