Repository navigation
[Lane 2 · kotlin-core] Precision/Recall/F1 — Averaging enum + internal ConfusionMatrixAccumulator #1226
Copy link
Copy link
Closed
Labels
codingImplementation task (DARC: C)Implementation task (DARC: C)size:sA few hoursA few hoursskill:kotlin-coreKotlin implementation in commonMainKotlin implementation in commonMainsub-issueSub-issue of a tracking issueSub-issue of a tracking issuetrainingTraining pipeline, optimizers, lossTraining pipeline, optimizers, loss
Description
Sub-issue of #1222 (Precision, Recall and F1Score metrics).
Lane: 2 · Kotlin core
Skill needed: Kotlin; read Accuracy.kt and the helpers from #1225 first.
Size: s (~3–4 h)
Blocked by: #1225 (helpers) and the contract decisions from #1224 (zero-division, macro class set, binary mode).
What to do
- Add
public enum class Averaging { BINARY, MACRO, MICRO }in
sk.ainet.lang.nn.metrics(own file, KDoc on each value — this is public API).
Semantics per [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224; the working definitions are:BINARY— report class1(the positive class) only; requires ≤ 2 classes;MACRO— unweighted mean of the per-class value over the class set decided in [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224;MICRO— pool TP / FP / FN over all classes, then apply the formula once.
- Add
internal class ConfusionMatrixAccumulator(dim: Int = -1, threshold: Float? = null)
withfun <T : DType, V> update(predictions: Tensor<T, V>, targets: Tensor<out DType, *>)
— walks every sample via the [Lane 2 · kotlin-core] Precision/Recall/F1 — extract Accuracy's class-matching helpers into a shared internal file #1225 helper and increments per-classtp/fp/fn
(Longcounters keyed by class index; a wrong prediction is one FP for the predicted
class and one FN for the target class);fun counts(): Map<Int, ClassCounts>(a smallinternal data class ClassCounts(tp, fp, fn));fun reset().
- Add
internal fun reduce(counts, averaging, perClass: (tp: Long, fp: Long, fn: Long) -> Double): Double
implementing the three averaging modes and the zero-division convention from [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224
in one place, soPrecision,RecallandF1Scoreeach contribute only a formula. - Tests: the accumulator is
internaltoskainet-lang-core, and the metric tests live
inskainet-backend-cpu(they needDirectCpuExecutionContextto build tensors). Two
options — pick one and say which in the PR:- test it through
Precision/Recallin [Lane 2 · kotlin-core] Precision metric #1227 / [Lane 2 · kotlin-core] Recall metric #1228 (then this PR ships
with areduce()unit test only, which needs no tensors), or - put an accumulator test in
skainet-lang-core'scommonTest, building tensors with
DenseTensorDataFactory().fromFloatArray(...)+SimpleTensor(...)the way
OperationExecutor.createTensordoes.
- test it through
./gradlew :skainet-lang:skainet-lang-core:apiDump— the dump gainsAveragingand
nothing else.
Acceptance
-
Averagingpublic, documented, in the API dump -
ConfusionMatrixAccumulator+reduce()internal, no public surface -
reduce()tested for macro / micro / binary and the zero-division case - Reference fixture from [Lane 1 · numerics] Precision/Recall/F1 — averaging-mode and zero-division conventions #1224 (
pred [2,0,1,2],target [2,1,1,0]) yields counts
{0: (0,1,1), 1: (1,0,1), 2: (1,1,0)}as (tp, fp, fn)
Notes
Don't build Precision / Recall / F1Score here even though it's tempting — they are
separate good first issues (#1227, #1228, #1229) precisely so three people can each
ship one small PR against this accumulator.
Metadata
Metadata
Assignees
Labels
codingImplementation task (DARC: C)Implementation task (DARC: C)size:sA few hoursA few hoursskill:kotlin-coreKotlin implementation in commonMainKotlin implementation in commonMainsub-issueSub-issue of a tracking issueSub-issue of a tracking issuetrainingTraining pipeline, optimizers, lossTraining pipeline, optimizers, loss
Activity
Merge pull request #1458 from EdinEd/feature/1226-confusion-matrix-ac…