Intercept out= ufunc writes: close the silent callback bypass (#379 item 4a)

[lmoresi]

Last of the #379 series, stacked on #383 (base is bugfix/synchronised-update-collective-flush; retarget as the stack merges). Deliberately a single, independently revertable commit.

What this does

np.add(x, 1, out=x) — and every in-place operator, which numpy routes through the same ufunc machinery — wrote straight into the buffer with no __setitem__ and no callback. Values landed, because the canonical array wraps the PETSc vec memory, but the ghost synchronisation and the state increment silently did not happen: stale ghosts and stale caches rather than hangs.

NDArray_With_Callback now implements __array_ufunc__ using the standard numpy override recipe:

• operands are unwrapped to base-class views for the computation (numpy's default implementation refuses to operate when another overriding operand is present);
• each requested out= is returned as the original object, so x += 1 keeps its subclass and callbacks — this is the recipe's load-bearing detail;
• each out= target of ours is notified after the write. The notification composes with the Callback-array parallel safety: central view/copy guard, honest exceptions, no dead snapshots (#379 items 3+4b) #381 canonical guard (an out= view resolves to the canonical array; an out= detached copy is skipped) and with the synchronised_array_update: collective dirty-variable flush replaces rank-local event replay (#379 item 2) #383 flush (inside uw.synchronised_array_update it dirty-marks for the single deferred flush).

The twelve in-place operators drop their explicit triggers — with the hook in place they would notify twice per operation.

Remaining documented bypasses: np.copyto and ufunc.at do not pass out= and are not intercepted; they are recorded in the class docstring and the test module.

Verification

• New tests/test_0061_ufunc_out_callback.py (level_1 / tier_a, written first and shown to fail): out= on canonical / view / detached copy, plain arithmetic stays silent and returns plain ndarrays, += fires exactly once, masked += still nets one notification via the parent write-back, deferral inside synchronised_array_update, and an end-to-end mesh-variable out= write reaching the PETSc vec.
• Units suite (0700/0710/0730/0800/0814/0850): 240 passed, pre-existing skips/xfails only.
• Quick gate level_1 and tier_a: 419 passed, 1 pre-existing xfail.
• Matplotlib quiver + contourf + masked-array smoke: clean.
• Ufunc dispatch overhead: a fixed ~2-3 microseconds per call (vs ~1.5 for plain ndarray) — noise against any PETSc pack. If real-model profiling ever disagrees, revert this one commit and the bypass reverts to a documented limitation.

Underworld development team with AI support from Claude Code

[lmoresi]

Adversarial review — round 1

A 36-operation differential battery (pre/post hook, including UnitAwareArray arithmetic, reductions, divmod, accumulate, outer, matmul, where=, out= tuples, pickle/deepcopy, sympy operands) came back 33/36 identical, and notification counts are exactly-once in every idiom checked. Three items:

1. Should-fix — the hook does not honour disable_inplace_operators. np.add(pts, delta, out=pts) bypasses the guard that pts += delta enforces — and on this branch the only user of that flag is the legacy swarm.points wrapper whose callback runs collective migrate(), so the bypass re-arms the exact per-write-collective hazard the flag exists to prevent (verified: the out= spelling moved PETSc coordinates). Retire the swarm.points writable stack: read-only snapshot, coords is the write path (#379 item 1) #382 deletes the only current flag user, but the flag's contract should be enforceable regardless. Fix: raise the same RuntimeError when any out= target carries the flag.

2. Note — reductions now return np.float64 where they previously returned Python float (the old __array_wrap__ scalar conversion is bypassed for ufunc reductions). Same values; only exact type(x) is float checks or direct JSON serialisation would notice. No in-repo consumer does.

3. Note — arr += pint.Quantity previously rebound the name to a pint Quantity (losing the subclass and callbacks); it now stays an NDArray_With_Callback and notifies. An improvement, recorded as a semantic change. ufunc.at remains a documented no-notify bypass (it now flows through the hook, so closing it later is trivial).

Fix for item 1 follows on this branch.

Underworld development team with AI support from Claude Code

[lmoresi]

Round 1 — response (fixed; branch rebased onto the updated #383)

Item 1 fixed: any out= target carrying disable_inplace_operators now raises the same RuntimeError as the in-place operators, before any computation — pinned by test_out_honours_disable_inplace_operators.

Items 2 and 3 stand as recorded notes: the reduction scalar type change (np.float64 vs Python float, no in-repo consumer affected) and the += pint.Quantity improvement (subclass and callbacks survive where they used to be silently dropped). ufunc.at remains a documented no-notify bypass.

Units suite 37 passed; quick gate 424 passed (the growing count reflects the accumulated stack tests).

Underworld development team with AI support from Claude Code

[lmoresi]

Round 2 — verified closed

The verifier confirmed the disable_inplace_operators fix: the raise happens before any computation, numpy normalises out= to a tuple (single and multi-output forms covered, None entries skipped), flagged VIEWS inherit the flag via __array_finalize__, and the explicit in-place operator guards raise a compatible message first — no double notification. No new findings.

The stack is review-clean end to end: round 1 (4 reviewers, 15 findings, all fixed) and round 2 (all fixes verified, 2 new defects found and fixed, 3 notes recorded). Ready to merge in order #381 → #382 → #383 → #385.

Underworld development team with AI support from Claude Code