Fix Polygon::offsetInward for non-convex polygons and collinear vertices

[andreolf]

Fixes #514

Implements the approach @Arjan-TNO analyzed and prototyped in #514, as requested by @Ryanf55 ("Can you submit a pull request with perhaps a unit test that verifies the fix?").

Problem

offsetInward offset each vertex along v1 + v2 (unit vectors towards its neighbours) scaled by margin / sin(angle). That is only correct for convex vertices of CCW-ordered polygons:

• Reflex vertices were offset outward (the red dashed result in the issue's picture) — acos(v1·v2) cannot distinguish a reflex corner from its convex mirror.
• Collinear vertices did not move at all: v1 + v2 degenerates to the zero vector, and sin(angle) = 0 additionally produces NaNs.

Fix

Each vertex is offset along the bisector of the inward normals of its two adjacent edges at the miter distance margin / cos(α/2), where α is the angle between the normals (|n1 + n2| = 2·cos(α/2), so no trigonometric calls are needed). This is correct for convex, reflex, and collinear vertices alike, because by construction the offset vertex keeps the distance margin to the lines through both adjacent edges, on the interior side.

Additionally:

• Winding order is detected with the shoelace formula, so clockwise-ordered polygons are now also offset towards the interior (previously they were silently offset outward).
• Degenerate inputs — fewer than 3 vertices, zero area, repeated vertices, or spike vertices whose edges fold back — now return false and leave the polygon unchanged, matching the documented @return contract (the old code always returned true).
• Negative margin still offsets outward, as documented.

Tests

The pre-existing offsetInward.triangle expectations pass unchanged (convex CCW behavior is identical). New tests:

• nonConvexPolygon — the exact 10-vertex polygon from unexpected/wrong output of Polygon::offsetInward function #514; asserts every offset vertex keeps signed distance margin to the lines through both adjacent original edges (signed = interior side, so a wrong-direction offset cannot pass), plus exact positions for the two collinear vertices.
• collinearVertices — square with a collinear mid-edge vertex (NaN in the old code).
• clockwiseOrder — mirrored triangle test with CW ordering.
• outwardWithNegativeMargin — documented negative-margin behavior.
• degenerateInputs — false + polygon unchanged for <3 vertices, repeated vertices, zero area.

All 17 tests in PolygonTest.cpp pass locally (compiled standalone against Eigen 3.4 / googletest with clang on macOS; full colcon CI deferred to this repo's pipeline).

No other code in the repository calls offsetInward, so the behavior change for previously-broken inputs (CW polygons, reflex/collinear vertices) has no in-tree callers to migrate.

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request