fix: Boolean difference silently does nothing on near-coplanar/step cuts (#11410)

[BenJule]

What

Fixes #11410. The interactive Boolean Difference tool silently does nothing for certain geometries (deep step or near-coplanar cuts). The model stays unchanged while the UI reports success. Setting the cutter as a negative part works because that path is unrelated (CSG at slice time), which is why it looked inconsistent.

Root cause

MeshBoolean::mcut::do_boolean splits multi-component meshes and subtracts each cutter via do_boolean_single (mcut, A_NOT_B with MC_DISPATCH_FILTER_FRAGMENT_LOCATION_ABOVE). For near-coplanar or step cuts mcut returns zero connected components, so do_boolean_single returns false and leaves src_part untouched.

The bug: for A_NOT_B and UNION the return value was ignored and the unchanged src_part was merged back, so the subtraction was silently dropped. The INTERSECTION branch already checks success.

Fix

Capture the return value; when mcut yields nothing for a difference, fall back to the CGAL corefinement path that already exists in this file (cgal::minus on an indexed_triangle_set accumulator) instead of dropping the op. The fallback only runs when mcut already failed, so it cannot regress anything that works today. Union and intersection paths are unchanged.

Validation

I could not run a full GUI build here, so I validated the approach offline against the reporter's exact model. The object contains the body (26,346 faces, 3 solids) and the cutter cylinder (1,440 faces) as two separate, still overlapping solids, i.e. the difference genuinely never got baked in. Running the subtraction with an independent robust engine (Manifold) succeeds cleanly:

Body and Cylinder overlap volume = 2025.1  (they really overlap)
Body minus Cylinder: volume 121900.2 to 119875.1  (removed 2025.1), result watertight

So the geometry is booleanable by a robust backend, mcut is the weak link. CGAL corefinement (used by this fix) is that kind of robust backend and is already linked and used in the codebase.

Note

Not yet exercised in a full BambuStudio GUI build on my side, CI will cover compilation. The change is deliberately low-risk (fallback only triggers on an existing mcut failure). Happy to adjust if you would prefer surfacing an error instead of the fallback, or gating it behind a flag.

[ArthurBambulab]

Please address the cancel-vs-failure ambiguity before merge, and ideally verify the CGAL path on the reporter's model (Manifold offline success ≠ CGAL in-tree).

Must fix

Cancel is conflated with mcut failure (MeshBoolean.cpp, do_boolean A_NOT_B branch)
do_boolean_single() also returns false when cancel_cb() fires mid-loop (around the connected-component traversal). This PR treats every !ok as a geometric mcut failure and runs CGAL fallback, so a user cancel can:

1. still run an expensive cgal::minus, and/or
2. complete successfully via CGAL, contradicting cancel semantics.
   Suggested minimal guard:

bool ok = do_boolean_single(*src_part, *cut_part, boolean_opts, cancel_cb, temp_progress_cb);
if (cancel_cb && cancel_cb())
    return false;
if (boolean_opts == "A_NOT_B") {
    // existing fallback...
}

[BenJule]

Thanks for catching this.

I added an explicit cancel check immediately after do_boolean_single() and before interpreting false as an mcut failure. A cancellation now returns from do_boolean() without entering the CGAL fallback.

I also built the updated commit successfully with the Linux PR pipeline and tested the resulting Bambu Studio artifact against the reporter's original 3MF from #11410. The subtraction now completes successfully and produces the expected deeper cut, confirming the in-tree CGAL fallback on the actual model.

I attempted to exercise cancellation manually as well, but this particular operation completes too quickly to reliably hit the Cancel button. I therefore cannot claim a manual cancellation test, but the cancel/failure ambiguity identified in the review is now explicitly guarded in the code.