Add Schedule 14D-9 data object for tender offer recommendations

[SurafelAnshebo]

Closes #922

Implements Schedule14D9 per the boundary agreed in #922 SC 14D-9 only, not the tender-offer side (SC TO-T/SC TO-I).

Approach

• Parser: lxml.html on the filing's primary document (SC 14D-9 has no structured XML cover page like Schedule 13D/G, so this parses rendered text directly).
• recommendation is a nullable classification ("accept" / "reject" / "neutral" / None) alongside the raw Item 4 text returns None on ambiguous/hedged board language rather than guessing, per the design note in Add data objects for tender offer filings (SC TO-T, SC TO-I, SC 14D9) #922.
• Item 4 section extraction filters out quoted cross-references before picking the real section (same failure mode as the fabricated item-anchor bug in 10-Q Part II Item 1 replaced by an MD&A cross-reference; numbered-index rows fabricate item anchors (ICE/CSCO/CTSH 10-Qs, FMCC 10-Ks) #918 a naive "first match" approach picks up a table-of-contents/cross-reference occurrence instead of the real section).
• Registered SC 14D9 / SC 14D9/A in form_map and obj() dispatch in edgar/__init__.py, plus the skill YAML (edgar/ai/skills/forms.yaml).

Ground truth

Lisata Therapeutics, Inc. (CIK 320017), SC 14D-9 filed 2026-06-10, accession 0001140361-26-024737. Hand-verified against the primary document and the board unanimously recommended shareholders accept the offer, and .recommendation == "accept" matches.

Testing

• New tests: tests/test_tender_offers.py, 11/11 passing
• Full fast suite: 2733 passed, 0 failed, 25 skipped (pre-existing) — no regressions
• ruff check / ruff format clean on new files

Marking as draft per @dgunning's note in #922 and happy to adjust the object shape/API before it grows further.

Thanks!

[dgunning]

Sorry this sat — thanks for your patience, and for marking it draft and asking about the shape before growing it. Reviewed properly now.

The structure is right. lxml.html on the primary document matches where this codebase is heading for new parsers, edgar/tender_offers/ parallels edgar/beneficial_ownership/ sensibly, and the Optional[str] recommendation with an explicit None for hedged language is the correct call — that instinct is the best thing in the PR. Two things I checked before raising them and then didn't:

• SC 14D9/A dispatch: fine. matches_form appends /A itself, so the single matches_form(sec_filing, "SC 14D9") covers both.
• assert filing.form in SC_14D9_FORMS: consistent with Schedule13D, Schedule13G, Form144 and MunicipalAdvisorForm today. It'll get swept when the 6.0 error-policy work lands, along with all of those — not something for you to fix here.

Your 11 tests pass, and the Lisata ground truth is genuine.

One real defect, in classify_recommendation

It searches the entire Item 4 — 126,597 characters in your own fixture, which includes Background of the Offer, prior board positions and the fairness opinion summary — in the fixed order reject → neutral → accept. So a hedging phrase anywhere in those 126k chars outranks the board's actual recommendation:

'neutral'  <- "The Board unanimously recommends that stockholders accept the Offer and
                tender their Shares. Background of the Offer. At the January meeting,
                the Board determined to express no opinion pending further review."

'neutral'  <- "The Board recommends that you accept the offer. Previously the company
                had made no recommendation regarding the earlier proposal."

Both are confident and wrong, which is the outcome your own design note argues against: "A confident wrong classification is worse than an honest unknown." The narrative history of a deal routinely contains exactly this language, so it isn't a contrived case.

Worth knowing: the fixture passes by luck. I checked all three neutral patterns against those 126k chars and they get 0 hits each. So the green test isn't evidence the approach works — it's evidence this particular filing happens not to contain the trigger words.

Suggested fix

Classify from a bounded window at the top of Item 4 rather than the whole section. The recommendation statement is essentially always in the opening paragraphs; the background that follows is what poisons the match. On your fixture, classify_recommendation(item4_text[:3000]) still returns 'accept'.

Pick the bound by measuring rather than by feel — where the recommendation sentence actually sits across a handful of filings — and say in a comment what the number is calibrated against, so the next person can requalify it when it drifts.

While you're there, the reject → neutral → accept precedence deserves a comment explaining why that order. Once the window is bounded it matters much less, but a reader will wonder.

Two smaller shape notes, since you asked

recommendation_text truncates at 2000 chars with no signal. A caller who gets exactly 2000 characters can't tell a complete short recommendation from a clipped long one. Either return the full opening statement by finding its actual end, or make the truncation visible (an ellipsis, or a truncated flag). Silent truncation of something named _text is the kind of thing that gets built on.

No rich rendering. Schedule13D has six __rich__/repr_rich touchpoints; this has a plain __repr__. Every other data object renders in the console, and users will notice this one doesn't. Worth adding before it grows.

Suggested next step

Bound the classification window, add a second and third ground-truth filing — ideally one genuine reject and one genuine neutral, since right now the accept path is the only one verified against a real document — and I think this comes out of draft in good shape. The parsing side I have no concerns about.

[dgunning]

Apologies for the long silence on this — you asked for a read on the object shape before growing it further and then waited five weeks. That's on us, not you.

Short version: the design is right, the code is better commented than most of what's already in the tree, and I'd take it. There are three CI gates it trips and one property whose behaviour doesn't match its docstring. None is structural.

What I checked first: does it still apply?

It's branched from 6ac6f4e5 (5.44.0) and main is now past 5.58, so I merged main into it before judging anything. It merges cleanly — one auto-merge in edgar/__init__.py, no conflicts — and your 20 tests pass unchanged against current main. Nothing has rotted.

The design

recommendation as a nullable classification alongside the raw text is the right call, and returning None on hedged language rather than guessing is the part I'd have pushed for if you hadn't done it. The comments carry their evidence — accessions for Woodbridge and Genco, the measured 722–1067 char statement range, the explicit note that reject/neutral are checked before accept because accept phrasing is the common case. That's the house style and it's rarer than it should be.

Registration is in the right place: both the form_map entry and the obj() if/elif chain in edgar/__init__.py. That's the dispatch people reliably get wrong (it is not the table in _filings.py). matches_form handles the amendment correctly — I checked SC 14D9/A matches and SC 14D1 / SC 14D9-X don't. Updating forms.yaml in the same PR covers the solvability half of the Definition of Done, which contributors almost never do unprompted.

One thing I want to be explicit about because it looks like a defect and isn't: assert filing.form in SC_14D9_FORMS is the established pattern here — ten_k.py, ten_q.py, current_report.py, schedule13.py, sixk.py, twenty_f.py, forty_f.py, form144.py and proxy/core.py all do exactly this. Don't change it; you'd be the odd one out.

Blocking: three CI gates

These all fail on the merged branch and all three are caused by the PR.

1. test_obj_still_returns_none_for_a_form_we_do_not_model uses form = "SC 14D9" as its example of a form we don't model. You now model it, so obj() routes into Schedule14D9.from_filing and dies on the stub:

AttributeError: '_Unmodelled' object has no attribute 'html'
edgar/tender_offers/schedule14d9.py:237

That test needs a different unmodelled form. It's a required companion change, not a sign anything is wrong with your code.

2. The raw-ValueError ratchet (tests/issues/regression/test_raw_valueerror_ratchet.py) goes 133 → 135, one for each raise ValueError in schedule14d9.py.

The ratchet's message suggests ValidationError, but I don't think that's the right one here — the caller's input wasn't invalid, the document failed to parse. DataObjectError in edgar/exceptions.py:497 is purpose-built for this: "filing.obj() could not build a data object for a form it supports", and it takes form= and accession_no=, which is exactly what both of your raises carry.

Worth knowing before you switch: DataObjectError is a ParsingError, not a ValueError, so your two pytest.raises(ValueError, match=...) assertions will need updating too. ValidationError would keep them working since it IS-A ValueError — but it would be the wrong name for the condition, and I'd rather have the right exception and an updated test.

3. No changelog fragment. Add changelog.d/922.added.md — a bold headline plus a sentence or two with one measured value, 500 chars max. Never touch CHANGELOG.md directly; concurrent PRs conflict on it.

One real finding: recommendation_text_truncated is inverted

The docstring promises "the recommendation statement itself may run longer than the calibrated window. Check this before treating recommendation_text as a complete statement." The implementation answers a different question — whether Item 4 as a whole exceeds the cap — and Item 4 is essentially always huge, so the flag fires when it shouldn't.

Measured on your own fixtures:

fixture	Item 4	window	cut by	reports truncated
lisata_therapeutics	125,815	1,066	marker	True
moody_national_reit_ii_neutral	1,741	845	marker	False
moody_national_reit_ii_reject	1,588	692	marker	False

Lisata's window was cut at the Background of the heading, so the statement is complete — and the flag tells the caller to distrust it. Since your own docstring says Item 4 routinely runs past 100,000 characters, this will read True for nearly every real filing while being False only for unusually short ones, which is backwards from useful.

The question the docstring asks is "did the cap make the cut, rather than a heading?":

normalized = re.sub(r"\s+", " ", self.item4_text).strip()
lowered = normalized.lower()
hits = [m.start() for m in (re.search(p, lowered) for p in _SECTION_BOUNDARY_MARKERS) if m]
boundary = min(hits) if hits else len(normalized)
return boundary > _RECOMMENDATION_WINDOW_CHARS

I tested it: all three fixtures report False (all cut cleanly at a heading), a synthetic 4,852-char Item 4 with no boundary heading reports True, and a short one with no heading reports False. test_recommendation_text_truncated_flag asserts is True for Lisata, so it locks the current behaviour in and would need to flip with the fix.

Not your problem

tests/xbrl/test_xbrl_unit_pointintime.py::TestPerformanceOverhead::test_reasonable_overhead_when_enabled also failed in my full -m fast run (7553 passed, 3 failed). It passes in isolation on both your branch and main, and your PR touches no XBRL files — it's load-sensitive. Ignore it.

Summary

Fix the two gates, add the changelog fragment, and decide on the truncation flag, and I'll take this. If you'd rather ship the object without recommendation_text_truncated and add it later once the semantics are settled, that's fine too — the rest stands on its own.

Thanks for the care you put into this, and sorry again for the wait.

[dgunning]

Follow-up: I ran the parser against real filings rather than only the committed fixtures, and found something worth having before you touch anything else. It's a one-flag change and it moves the numbers a lot.

_find_real_item_starts is case-sensitive

pattern = re.compile(rf"Item\s*{item_number}\s*[.\-—]")

A large share of filers set the heading in caps, so that pattern never sees it. The only Item 4 it matches in those documents is a quoted cross-reference, which your quote filter then correctly discards — leaving zero candidates and the ValueError:

Tourmaline Bio    case-sensitive=1  ignorecase=3   'ITEM 4.\n THE SOLICITATION OR RECOMMENDATION'
IGM Biosciences   case-sensitive=0  ignorecase=2   'ITEM 4. THE SOLICITATION OR RECOMMENDATION'
Allakos Inc.      case-sensitive=0  ignorecase=2   'ITEM 4.THE SOLICITATION OR RECOMMENDATION'

Adding re.IGNORECASE to that one compile, measured over 25 SC 14D9 originals filed in 2025:

	before	after
classified (accept/neutral/reject)	13 (52%)	20 (80%)
parsed, recommendation is None	3	4
raised ValueError	9 (36%)	1 (4%)

Your 20 tests still pass unchanged. The table-of-contents entries do start matching too, but extract_item_section's longest-span rule discards them exactly as designed — nice that it already handled the case.

Worth adding a fixture with an uppercase heading so this can't regress; none of the three committed ones use that spelling, which is why the suite was green while a third of real filings failed.

The design question you asked about, with numbers

You offered to adjust the object shape, and here is the place it matters. Of the 229 SC 14D9 filings in 2025, 174 are SC 14D9/A. Sampling 20 amendments with the flag applied, 15 still raise: an amendment usually restates only the items it changes, so Item 4 genuinely isn't there.

Since both forms are registered, filing.obj() raises for the dominant filing type of the form. That's worse than it sounds, because obj() is the generic entry point people reach for without knowing the form. Two ways out:

1. Make Item 4 optional — construct the object with item4_text=None and recommendation=None, and let is_amendment explain why. Keeps obj() total, at the cost of an object that sometimes carries nothing.
2. Register only SC 14D9 and leave amendments unmodelled.

I lean towards (1): an amendment still carries the subject company, date and amendment flag, and raising for a structurally normal document reads as a library bug to the caller. But it's your design and I'd rather you picked.

Where that leaves the PR

With the case fix, this is four small items from mergeable — the two CI gates and the changelog fragment from my earlier review, plus the truncation-flag semantics. Happy for you to take them; they're yours and you clearly know this form better than I do now.

If you'd rather not, say so or just go quiet for a week and we'll finish it on your branch with your authorship intact. No obligation either way after the wait we put you through.

[dgunning]

@SurafelAnshebo I've pushed the fixes to this branch rather than leaving you another list — CI had gone red and all of it was mechanical. Your parser, the classification design and the ground truth are untouched and remain yours. Revert any of it if you disagree; none of it is precious.

What CI was failing on

Both gates live only on main, and this branch was cut at 5.44.0 — CI tests the pull-request merge commit, which is why they fired here and not in your local run. I merged main in so the branch and CI now see the same tree.

• test_obj_still_returns_none_for_a_form_we_do_not_model used "SC 14D9" as its example of an unmodelled form. You model it now, so obj() routed into Schedule14D9.from_filing and died on the stub. Changed to "SC 14D1", with a comment saying why the choice matters.
• The raw-ValueError ratchet went 133 → 135. I used DataObjectError rather than the ValidationError the ratchet suggests: the caller's input wasn't invalid, the document failed to parse, and DataObjectError is the canonical "filing.obj() could not build a data object for a form it supports" — it takes form= and accession_no=, exactly what both of your raises already carried. It's a ParsingError, not a ValueError, so your two pytest.raises assertions moved with it.

The one that actually mattered

_find_real_item_starts compiled Item\s*N\s*[.\-—] case-sensitively, and a large share of filers set the heading in caps. In those documents the only surviving Item 4 was a quoted cross-reference, which your quote filter then correctly discarded — so the whole filing raised.

Over 25 SC 14D9 originals filed in 2025:

	before	after
classified	13 (52%)	20 (80%)
raised	9 (36%)	1 (4%)

Your extract_item_section longest-span rule handles the table-of-contents entries that now also match, exactly as designed — nothing else needed changing.

A latent one your tests couldn't reach

Writing a case-insensitivity test surfaced this:

prev_char = text[start - 1] if start > 0 else ""
if prev_char in _QUOTE_CHARS:   # "" is a substring of every string → True

A heading at offset 0 was discarded as quoted. Not reachable in a whole filed document, where Item 4 never starts at byte 0, but reachable from any already-sliced text — and it's why my synthetic fixture failed on the title-case spelling too, which is what put me onto it. Guarded with prev_char and ..., plus a test that a genuine quoted cross-reference is still skipped.

The truncation flag

recommendation_text_truncated promised "the statement may run longer than the window" but measured whether Item 4 exceeded the cap. Item 4 carries the whole background narrative and routinely passes 100,000 characters, so it read True for nearly every real filing — including Lisata, whose window ends cleanly at a heading after 1,066 characters. It now reports whether the cap, rather than a heading, made the cut. Your old test asserted is True for Lisata, so it flips, and there's a new one for the case the flag is actually for.

Plus the changelog fragment. Full fast suite: 7566 passed.

Still yours to decide

The design question from your original post, with numbers behind it now: of the 229 SC 14D9 filings in 2025, 174 are SC 14D9/A, and 15 of 20 sampled amendments still raise, because an amendment restates only the items it changes. filing.obj() therefore raises on the dominant filing type for this form. Either Item 4 becomes optional on the object, or SC 14D9/A comes off the registration. I've left it alone and only documented it in the Raises: block — it's your call, and I'd rather you made it than have me pick while you were away.

Thanks again for the work, and for your patience with how long we took to look at it.