Share vertices for flat shaded materials

[Taldariner]

Closes the vertex efficiency half of #696. As @Lilaa3 put it there:

Kaze's issue is not the lack of implementation of disabling G_SHADE. That's already implemented. The issue is the efficiency of vertice usage.

What it does

With G_SHADING_SMOOTH cleared, the microcode takes the shade of a whole triangle from the vertex the flag argument of the triangle commands selects. The shade of the other two corners is never read. The exporter did not use that: it wrote one vertex per normal and always passed flag 0, so a faceted model paid for vertices it did not need.

This adds fast64_internal/f3d/flat_shading.py, which picks the provoking corner of each triangle and tracks which buffer vertices may be shared. Two steps:

1. Shades are matched to keys they can own. A per-face choice is not enough: on a cube every corner is a candidate for three faces, so any fixed order wastes one and ends at 9 vertices instead of 8.
2. Faces are then processed in export order, reusing a vertex whenever its shade is already right, or has not been read by anything yet.

getInfoDict compares edges without shade for flat materials, so saveTriangleStrip keeps faces that can share vertices next to each other.

Behind scene.optimizeFlatShading, off by default.

Results

C export, F3DEX2 (F3DEX and F3DEX3 give the same counts), with and without material bleeding:

mesh	vertices before	after
hexagonal prism, 16 tris	30	12
Bob-omb style top, 20 tris	42	16
cube	24	8
tetrahedron	12	4
one mesh across two bones (skinned)	54	36
large texture mode, 128 tris	280	152 (24 → 13 vertex loads)
12800 triangle mesh	26074	13647 (848 → 428 vertex loads)

Planar meshes are untouched, since one normal means nothing to share. The 12800 triangle export also got faster, by a fifth to a third depending on the run: half as many vertices means half as many buffer scans.

Scope

Enabled for every F3DEX generation, which is exactly where the triangle commands encode flag as a cyclic rotation of the indices. Original F3D writes it as a separate byte and is left out. T3D was not looked at.

Skinned meshes are included. A parent limb's vertices arrive as existingVertData already written out, so they enter the pool claimed: reusable as a corner, allowed to provoke only when their shade already matches, never rewritten. OOTTriangleConverter opts out, because OoT skinning computes skin vertex normals separately and stores them in SkinLimbModif structures this does not model.

Why the behaviour is what I say it is

F3DEX3's microcode is public, and it spells the rule out:

tri_flat_shading:
    vlt     $v29, $v31, $v31[3]       // Set vcc to 11100000
    vmrg    tHAtI, $v25, tHAtI        // RGB from original vtx 1, alpha from $1
    vmrg    tMAtI, $v25, tMAtI        // RGB from original vtx 1, alpha from $2
    vmrg    tLAtI, $v25, tLAtI        // RGB from original vtx 1, alpha from $3

with $v25 loaded from origV1Addr + flatV1Offset. So RGB comes from vertex 1 and alpha stays per vertex, which is why alpha is part of the merge key here and vertices are never merged across it. flatV1Offset is what keeps this correct through clipping. GLideN64 reads the flag the same way for the older microcodes, including the separate flag byte of original F3D.

How it was checked

A generated corpus of 22 meshes (flat and smooth, vertex colors, fog, UV seams, large texture mode, two materials on one object, F3DEX3 packed normals and cel shading) plus a skinned geolayout export, for F3D, F3DEX, F3DEX2 and F3DEX3, with and without bleeding, exported through the real operators in Blender 4.2.16:

• with the switch off, every exported file is byte identical to before this change;
• with it on, smooth materials are still byte identical, and every model draws the same: both exports are expanded into triangle soups by emulating gSPVertex and the flag rule, then compared as multisets up to a cyclic rotation of the corners;
• indices stay inside the vertex buffer and flags stay in 0..2 everywhere;
• insertable binary export was checked the same way for F3DEX and F3DEX2: 948 → 660 bytes, every index inside the buffer.

There is also a debug switch, scene.poisonFlatShading, which writes a deliberately wrong shade into vertices that provide shade to no triangle. On the corpus it poisons 160 of 416 vertices and nothing about the drawn result changes, which is the property the optimization rests on.

What I could not do: run this on hardware or an accurate RDP. I have no ROM to build a decomp project from. Everything above checks my model of the microcode, and for F3DEX3 that model comes from the microcode source, but F3DEX2 is still only corroborated by GLideN64. If someone with a decomp setup can run it, I have a ready .blend with the test shapes and a fixed camera, plus a script that compares two screenshots pixel by pixel; the interesting run is with poisoning on, where a wrong provoking vertex shows up as a magenta or unlit face instead of a subtle shading difference. Happy to attach both.

Tests

tests/test_flat_shading.py covers the assignment: the known optima above, agreement with a brute force optimum on random small meshes, the invariants (the provoking vertex carries its face's shade, corners keep their vertex), alpha staying in the key, the claimed and inherited rules, and that poisoning leaves provoking and inherited vertices and alpha alone.

They are plain scripts run by tests/run_tests.py, not pytest: the repository root is itself a Python package whose __init__.py imports bpy, so pytest resolves test modules as fast64.tests.… and fails to collect. This keeps the dependency count at zero, matching the rest of the repo. .github/workflows/tests.yml runs them.

The Blender level harness that produced the corpus numbers lives outside this PR, since it is larger than the change itself and overlaps with #317. I can contribute it separately in whatever shape suits.

Not included

OoT skinned limbs, original F3D, T3D. Vertices are also still duplicated across vertex buffer boundaries, and the triangulation is whatever Blender produced, so a model may need its diagonals flipped by hand to reach the optimum.

[Dragorn421]

Sorry for the review delay.

Overall I'm not convinced by adding tests at this point in fast64's development, the addon is pretty mature. Also not convinced by making a module bpy-less just for the sake of testing it