diff --git a/.github/workflows/flow.yml b/.github/workflows/flow.yml index 0b27b5f..c87d677 100644 --- a/.github/workflows/flow.yml +++ b/.github/workflows/flow.yml @@ -157,6 +157,69 @@ jobs: benchmarks/parity_diagnostics.json benchmarks/headline_environment.json + estimator-matrix: + name: Wide estimator matrix + runs-on: ubuntu-latest + # Both sides call into OpenBLAS, so the thread count is pinned here for the + # reason the canonical job pins it. + env: + OPENBLAS_NUM_THREADS: "4" + OMP_NUM_THREADS: "4" + MKL_NUM_THREADS: "4" + steps: + - uses: actions/checkout@v4 + + - uses: actions/checkout@v4 + with: + repository: flooooooooooow/flow + ref: 88aac5095488309813c7173d268c1f8260421c6e + path: .flow-toolchain + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install benchmark dependencies + run: | + pip install -r .flow-toolchain/requirements.txt + pip install numpy scikit-learn + sudo apt-get update && sudo apt-get install -y libopenblas-dev + + - name: Rebuild the registry and the generated benchmarks + run: | + python benchmarks/estimator_coverage.py + python benchmarks/generate_estimator_bench.py + git diff --exit-code benchmarks/estimator_coverage.json benchmarks/generated + + - name: Time every estimator on the Flow side + env: + FLOW_BIN: ${{ github.workspace }}/.flow-toolchain/flow + FLOW_HOST: python + FLOW_OPT_LEVEL: "3" + FLOW_LDFLAGS: "-lm -lopenblas lib/scikit/flow_time.c lib/scikit/flow_parallel.c" + run: python benchmarks/run_estimator_bench.py --rounds 3 --out benchmarks/estimator_flow_raw.txt + + - name: Time every estimator on the scikit-learn side + run: python benchmarks/bench_estimators_sklearn.py --repeats 5 + + - name: Join the two sides + run: | + python benchmarks/compare_estimators.py benchmarks/estimator_flow_raw.txt \ + --note "GitHub Actions ubuntu-latest, Flow at -O3 with adaptive repeats, fastest of 3 rounds; scikit-learn wheel, fastest of 5; OpenBLAS pinned to 4 threads" + + - name: Gate every ranked row against scikit-learn + run: python benchmarks/check_estimator_matrix.py benchmarks/estimator_comparison.json + + - uses: actions/upload-artifact@v4 + if: always() + with: + name: estimator-matrix-${{ github.run_id }} + path: | + benchmarks/estimator_flow_raw.txt + benchmarks/estimators_sklearn.json + benchmarks/estimator_comparison.json + benchmarks/estimator_coverage.json + scaled-report: name: Scaled benchmark report runs-on: ubuntu-latest diff --git a/README.md b/README.md index d2eaa7e..dc5c66d 100644 --- a/README.md +++ b/README.md @@ -73,6 +73,27 @@ The ten rows at 10000 samples, the only sizes in the matrix large enough to meas `benchmarks/scaled_flow_baseline.json` is a separate Flow-against-itself gate that does fail the build. Refresh it from a CI artifact rather than from a developer machine, or the gate will read a fast laptop as the standard and fail every CI run. +## The whole library + +The 19 canonical rows race twelve estimators. `lib/scikit` exports 203, and a +claim about Flow against scikit-learn covers six percent of the library while +the rest go unmeasured. A registry maps every exported `fit` to its +scikit-learn counterpart, times both sides on the same data, and a CI job +fails the build when a ranked row is slower. + +**Flow is faster on 188 of the 188 ranked rows.** The rows that carry no ratio +carry a reason instead: four implementations say in their own comments that +they are simplified, so they are timed and shown without being ranked; five +Flow functions do part of what their scikit-learn namesake does, such as a +voting estimator that takes models already fitted; six have no scikit-learn +counterpart at all. + +These rows carry no parity contract and no declared tolerances. Each library +runs its own defaults over the same data, which answers whether an +implementation is in the same performance league and says nothing about +whether it computes the same thing. The canonical rows above are where +numerical equivalence is established. + Detailed artifacts: - [`benchmarks/SKLEARN_EXECUTION_INVENTORY.md`](benchmarks/SKLEARN_EXECUTION_INVENTORY.md) diff --git a/benchmarks/README.md b/benchmarks/README.md index 094b1d8..20fd16f 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -241,11 +241,19 @@ reason. | bucket | meaning | | --- | --- | | `runnable` | arguments resolved and a scikit-learn counterpart exists | -| `different_shape` | takes a pipeline, a vectorizer input or a list of fitted models first, so it is not an estimator over a feature matrix | +| `shaped` | its fit does not begin with a feature matrix, so the registry carries the call written out, and the row is raced and ranked like any other | +| `different_shape` | the Flow function does part of what the scikit-learn class does, such as a voting estimator that takes models already fitted | | `flow_only` | Flow implements it and scikit-learn has no equivalent | | `simplified` | the implementation's own comments call it a simplified stand-in | | `blocked` | the signature is not resolved yet, with the missing parameter named | +A `shaped` entry gives the Flow call in terms of the variables the generated +harness declares, what the scikit-learn side fits and transforms, and where +needed a preamble that builds the input: a pipeline of a scaler and a logistic +regression, a corpus for the two text vectorizers, a list of dicts for the dict +vectorizer. The corpus lives in the registry so the Flow file and the +scikit-learn harness read one copy of it. + The `simplified` bucket is detected from the source rather than listed, so it stays true as the implementations are filled in. It matters: `spectral_biclustering` thresholds row and column means where scikit-learn does an SVD and k-means, and @@ -266,11 +274,23 @@ scikit-learn side of the same registry, and ``` python benchmarks/estimator_coverage.py python benchmarks/generate_estimator_bench.py -for f in benchmarks/generated/bench_estimators_*.flow; do flow run "$f"; done | tee /tmp/flow.txt +python benchmarks/run_estimator_bench.py --rounds 3 python benchmarks/bench_estimators_sklearn.py -python benchmarks/compare_estimators.py /tmp/flow.txt +python benchmarks/compare_estimators.py benchmarks/estimator_flow_raw.txt +python benchmarks/check_estimator_matrix.py ``` +[`run_estimator_bench.py`](run_estimator_bench.py) checks every chunk's exit +status, so a chunk that dies takes its own row count down with it in the +summary rather than disappearing, and reuses the binary the first round leaves +behind so later rounds pay for timing instead of for compiling the library +again. [`check_estimator_matrix.py`](check_estimator_matrix.py) fails on a +ranked row slower than scikit-learn, on a row that reported no timing, and on a +ranked count that has quietly shrunk. The `Wide estimator matrix` job in +`.github/workflows/flow.yml` runs all of it on a runner that is not competing +with anything, which is where a published number belongs. A developer machine +under load recorded the same row at 1.72x and at 0.96x in consecutive runs. + Read the result for what it is. These rows have no parity contract, no declared tolerances and no disparity report: each library runs its own defaults over the same data. That answers whether a Flow implementation is in the same diff --git a/benchmarks/bench_estimators_sklearn.py b/benchmarks/bench_estimators_sklearn.py index a99c9cc..a516f87 100644 --- a/benchmarks/bench_estimators_sklearn.py +++ b/benchmarks/bench_estimators_sklearn.py @@ -33,7 +33,8 @@ # this data. A shallow tree keeps the wrapper's own overhead visible rather than # burying it under the base estimator's work. def _constructors() -> dict: - from sklearn.linear_model import Ridge + from sklearn.linear_model import LogisticRegression, Ridge + from sklearn.preprocessing import FunctionTransformer, StandardScaler from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor import numpy as np @@ -57,6 +58,16 @@ def _constructors() -> dict: "SparseCoder": lambda c: c(dictionary=np.eye(4)), # nu=0.5 is infeasible for this class balance. "NuSVC": lambda c: c(nu=0.1), + # A selector needs something to read importances from. + "SelectFromModel": lambda c: c(tree_c(), threshold=-np.inf, max_features=2), + # The composed rows, built to match what the Flow harness composes: + # a scaler and a logistic regression, a scaler over the columns, and a + # scaler beside a passthrough. + "Pipeline": lambda c: c([("scaler", StandardScaler()), + ("classifier", LogisticRegression(max_iter=50))]), + "ColumnTransformer": lambda c: c([("scaler", StandardScaler(), [0, 1, 2, 3])]), + "FeatureUnion": lambda c: c([("scaler", StandardScaler()), + ("passthrough", FunctionTransformer())]), } @@ -91,7 +102,10 @@ def main() -> int: args = ap.parse_args() registry = json.loads(REGISTRY.read_text()) - runnable = [e for e in registry["entries"] if e["bucket"] == "runnable"] + # Matches generate_estimator_bench.py: shaped rows are raced and ranked, + # simplified rows are timed on both sides and shown without a ratio. + runnable = [e for e in registry["entries"] + if e["bucket"] in ("runnable", "shaped", "simplified")] classes = dict(all_estimators()) constructors = _constructors() @@ -115,19 +129,53 @@ def main() -> int: rows.append({"flow_estimator": entry["flow_estimator"], "sklearn_estimator": name, "status": "unavailable", "reason": "not in sklearn.utils.all_estimators()"}) continue - kind = dataset_kind(entry) + shape = entry.get("shape") + kind = shape["dataset"] if shape else dataset_kind(entry) X, y = data[kind] + ctor_kwargs = {} + if shape and shape.get("sklearn_ctor"): + # The string is the constructor's own keyword list, kept next to the + # Flow call it matches so the two cannot drift apart. + ctor_kwargs = eval(f"dict({shape['sklearn_ctor']})") # noqa: S307 + # A row whose scikit-learn side fits a target vector or a single ordered + # variable rather than a design. The Flow side of these is written out + # in the registry for the same reason. + fit_input = (shape or {}).get("sklearn_input", "X") + if fit_input == "y": + first, second = y, None + elif fit_input == "x1d": + first, second = X[:, 0], y + elif fit_input == "docs": + # The corpus travels in the registry, so the Flow file and this one + # read one copy of it. + first, second = list(shape["corpus"]), None + elif fit_input == "dicts": + first = [{f"f{j}": float(v) for j, v in enumerate(row)} for row in X] + second = None + elif fit_input == "labelsets": + first = [tuple(int(v) for v in row) for row in y] + second = None + else: + first, second = X, y try: with warnings.catch_warnings(): warnings.simplefilter("ignore") build = constructors.get(name) - model = build(cls) if build else cls() - fit_ms = timed((lambda: model.fit(X)) if y is None else (lambda: model.fit(X, y)), args.repeats) + model = build(cls) if build else cls(**ctor_kwargs) + fit_ms = timed( + (lambda: model.fit(first)) if second is None + else (lambda: model.fit(first, second)), + args.repeats, + ) pred_ms = 0.0 - for method in ("predict", "transform"): + # A recipe can name the method to time, for a class whose work + # is called something other than predict or transform. + methods = [shape["sklearn_work"]] if shape and shape.get("sklearn_work") \ + else ["predict", "transform"] + for method in methods: if hasattr(model, method): try: - pred_ms = timed(lambda m=method: getattr(model, m)(X), args.repeats) + pred_ms = timed(lambda m=method: getattr(model, m)(first), args.repeats) except Exception: pred_ms = 0.0 break diff --git a/benchmarks/check_estimator_matrix.py b/benchmarks/check_estimator_matrix.py new file mode 100755 index 0000000..6cd1a31 --- /dev/null +++ b/benchmarks/check_estimator_matrix.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python3 +"""Gate the wide estimator matrix: every ranked row has to favour Flow. + +The matrix was measured by hand for a long time, which is why a published +number sat at 145 of 166 while the working tree was at 155. A gate in CI is +what keeps the claim and the code in step: a row that turns slower than +scikit-learn fails the run, and a row that stops reporting fails it too, +because a chunk that dies mid-run is indistinguishable from a chunk with +nothing to say. + +Rows the registry marks simplified carry no ratio and are not ranked. Rows +under the clock's floor carry no ratio either. Both are reported and neither +fails the run. +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("comparison", nargs="?", type=Path, + default=ROOT / "benchmarks" / "estimator_comparison.json") + ap.add_argument("--min-compared", type=int, default=160, + help="fail if fewer rows than this produced a ratio") + ap.add_argument("--tolerance", type=float, default=1.0, + help="the ratio a row has to reach, sklearn_ms / flow_ms") + args = ap.parse_args() + + payload = json.loads(args.comparison.read_text()) + rows = payload["rows"] + ranked = [r for r in rows if r.get("status") == "ok" and r.get("speedup") is not None] + losers = sorted((r for r in ranked if r["speedup"] < args.tolerance), + key=lambda r: r["speedup"]) + missing = [r for r in rows if r.get("status") in ("flow_missing", "sklearn_missing")] + + print(f"ranked {len(ranked)} rows, {len(ranked) - len(losers)} at or above " + f"{args.tolerance:.2f}x") + for r in losers: + print(f" SLOWER {r['speedup']:6.3f}x {r['flow_estimator']:32s} " + f"flow={r['flow_ms']:9.3f} sklearn={r['sklearn_ms']:9.3f}") + for r in missing: + print(f" MISSING {r['flow_estimator']:32s} {r['status']}: {r.get('reason', '')}") + + failed = False + if losers: + print(f"{len(losers)} rows are slower than scikit-learn") + failed = True + if missing: + print(f"{len(missing)} rows reported no timing") + failed = True + if len(ranked) < args.min_compared: + print(f"only {len(ranked)} rows produced a ratio, expected at least {args.min_compared}") + failed = True + if failed: + return 1 + print("every ranked row favours Flow") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/compare_estimators.py b/benchmarks/compare_estimators.py index 3973b57..2fff435 100644 --- a/benchmarks/compare_estimators.py +++ b/benchmarks/compare_estimators.py @@ -55,7 +55,7 @@ def main() -> int: rows = [] for entry in registry["entries"]: name = entry["flow_estimator"] - if entry["bucket"] != "runnable": + if entry["bucket"] not in ("runnable", "shaped"): row = {"flow_estimator": name, "status": entry["bucket"], "reason": entry.get("reason", "")} if entry.get("sklearn_estimator"): row["sklearn_estimator"] = entry["sklearn_estimator"] diff --git a/benchmarks/estimator_comparison.json b/benchmarks/estimator_comparison.json index 8b3338c..6a95f2f 100644 --- a/benchmarks/estimator_comparison.json +++ b/benchmarks/estimator_comparison.json @@ -2,13 +2,13 @@ "schema_version": 1, "contract": "breadth timing only; no parity contract, no declared tolerances, each library on its own defaults over the same data", "measurement": "fastest of several rounds per estimator on both sides, which is what survives a machine that is not idle", - "note": "Apple M4 Max; Flow at -O3 with adaptive repeats, fastest of 3 rounds; scikit-learn 1.9.0 wheel, fastest of 3; developer machine under load. CI is the authority for anything published.", + "note": "GitHub Actions ubuntu-latest, Flow at -O3 with adaptive repeats, fastest of 3 rounds; scikit-learn wheel, fastest of 5; OpenBLAS pinned to 4 threads", "counts": { "registry_estimators": 203, - "compared": 166, - "flow_wins": 145, - "sklearn_wins": 21, - "below_resolution": 2, + "compared": 188, + "flow_wins": 188, + "sklearn_wins": 0, + "below_resolution": 0, "simplified": 4 }, "rows": [ @@ -16,9 +16,9 @@ "flow_estimator": "adaboost_classifier", "sklearn_estimator": "AdaBoostClassifier", "dataset": "classification", - "flow_ms": 0.293650003, - "sklearn_ms": 23.29525000823196, - "speedup": 79.32998389321304, + "flow_ms": 0.815846328, + "sklearn_ms": 80.84233899990068, + "speedup": 99.09015488012429, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -27,27 +27,32 @@ "flow_estimator": "adaboost_regressor", "sklearn_estimator": "AdaBoostRegressor", "dataset": "regression", - "flow_ms": 8.074000001, - "sklearn_ms": 15.209208999294788, - "speedup": 1.8837266531348849, - "flow_repeats": 1, + "flow_ms": 14.230996682999999, + "sklearn_ms": 39.021172000047954, + "speedup": 2.7419844772124566, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "additive_chi2_sampler", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "AdditiveChi2Sampler" + "sklearn_estimator": "AdditiveChi2Sampler", + "dataset": "classification", + "flow_ms": 0.020321969000000002, + "sklearn_ms": 0.32264300000406365, + "speedup": 15.876561961297334, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "affinity_propagation", "sklearn_estimator": "AffinityPropagation", "dataset": "unsupervised", - "flow_ms": 139.29699707, - "sklearn_ms": 4.076251003425568, - "speedup": 0.029263021379973875, - "flow_repeats": 1, + "flow_ms": 6.470405579, + "sklearn_ms": 7.170991000066351, + "speedup": 1.1082753488189572, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -55,10 +60,10 @@ "flow_estimator": "agglomerative_clustering", "sklearn_estimator": "AgglomerativeClustering", "dataset": "unsupervised", - "flow_ms": 0.128594995, - "sklearn_ms": 0.3488329966785386, - "speedup": 2.7126483163558475, - "flow_repeats": 200, + "flow_ms": 0.220218897, + "sklearn_ms": 0.7726449999836404, + "speedup": 3.5085317859149954, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -66,10 +71,10 @@ "flow_estimator": "ard_regression", "sklearn_estimator": "ARDRegression", "dataset": "regression", - "flow_ms": 0.63499999, - "sklearn_ms": 3.3072089863708243, - "speedup": 5.208203210161979, - "flow_repeats": 1, + "flow_ms": 0.758061467, + "sklearn_ms": 10.090251000065109, + "speedup": 13.310597411047551, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -77,9 +82,9 @@ "flow_estimator": "bagging_classifier", "sklearn_estimator": "BaggingClassifier", "dataset": "classification", - "flow_ms": 0.14565, - "sklearn_ms": 6.647166999755427, - "speedup": 45.63794713186012, + "flow_ms": 0.351454919, + "sklearn_ms": 18.44309399996291, + "speedup": 52.476414478532064, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -88,10 +93,10 @@ "flow_estimator": "bagging_regressor", "sklearn_estimator": "BaggingRegressor", "dataset": "regression", - "flow_ms": 7.729000173, - "sklearn_ms": 12.465458989026956, - "speedup": 1.6128164976076727, - "flow_repeats": 1, + "flow_ms": 11.993147566000001, + "sklearn_ms": 27.779086000009556, + "speedup": 2.3162464938530345, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -99,10 +104,10 @@ "flow_estimator": "bayesian_gaussian_mixture", "sklearn_estimator": "BayesianGaussianMixture", "dataset": "unsupervised", - "flow_ms": 0.115610001, - "sklearn_ms": 1.1429580044932663, - "speedup": 9.886324665746402, - "flow_repeats": 200, + "flow_ms": 0.201650655, + "sklearn_ms": 4.248096000083024, + "speedup": 21.06661146269534, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -110,9 +115,9 @@ "flow_estimator": "bayesian_ridge", "sklearn_estimator": "BayesianRidge", "dataset": "regression", - "flow_ms": 0.096429996, - "sklearn_ms": 0.45233400305733085, - "speedup": 4.690801844037521, + "flow_ms": 0.109020204, + "sklearn_ms": 1.6822580000734888, + "speedup": 15.430699433230641, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -121,9 +126,9 @@ "flow_estimator": "bernoulli_nb", "sklearn_estimator": "BernoulliNB", "dataset": "classification", - "flow_ms": 0.002665, - "sklearn_ms": 0.5652910040225834, - "speedup": 212.1166994456223, + "flow_ms": 0.005007714999999999, + "sklearn_ms": 2.3629600000276696, + "speedup": 471.8639139862532, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -132,9 +137,9 @@ "flow_estimator": "bernoulli_rbm", "sklearn_estimator": "BernoulliRBM", "dataset": "unsupervised", - "flow_ms": 0.456850013, - "sklearn_ms": 8.775667010922916, - "speedup": 19.20907685499599, + "flow_ms": 1.2019327039999999, + "sklearn_ms": 21.10552299984647, + "speedup": 17.559654487816044, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -143,9 +148,9 @@ "flow_estimator": "birch", "sklearn_estimator": "Birch", "dataset": "unsupervised", - "flow_ms": 0.012905001, - "sklearn_ms": 1.0970829898724332, - "speedup": 85.01223594422297, + "flow_ms": 0.01437398, + "sklearn_ms": 4.229422000094019, + "speedup": 294.24153923228073, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -154,9 +159,9 @@ "flow_estimator": "bisecting_kmeans", "sklearn_estimator": "BisectingKMeans", "dataset": "unsupervised", - "flow_ms": 0.019985, - "sklearn_ms": 4.4658750121016055, - "speedup": 223.46134661504158, + "flow_ms": 0.031568501, + "sklearn_ms": 4.861612999889076, + "speedup": 154.00202245551907, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -165,10 +170,10 @@ "flow_estimator": "calibrated_classifier_cv", "sklearn_estimator": "CalibratedClassifierCV", "dataset": "classification", - "flow_ms": 0.34106499, - "sklearn_ms": 10.508707986446097, - "speedup": 30.811453226102444, - "flow_repeats": 200, + "flow_ms": 0.724029661, + "sklearn_ms": 44.45090599995183, + "speedup": 61.393763811496434, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -176,9 +181,9 @@ "flow_estimator": "categorical_nb", "sklearn_estimator": "CategoricalNB", "dataset": "classification", - "flow_ms": 0.003265, - "sklearn_ms": 0.68374999682419, - "speedup": 209.41806947142112, + "flow_ms": 0.00601415, + "sklearn_ms": 2.5394489999825964, + "speedup": 422.24570387878526, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -187,9 +192,9 @@ "flow_estimator": "cca", "sklearn_estimator": "CCA", "dataset": "multioutput", - "flow_ms": 0.122035002, - "sklearn_ms": 0.4619169922079891, - "speedup": 3.785118897347083, + "flow_ms": 0.140699784, + "sklearn_ms": 1.4727139999877181, + "speedup": 10.467066530732685, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -198,44 +203,54 @@ "flow_estimator": "classifier_chain", "sklearn_estimator": "ClassifierChain", "dataset": "multioutput_class", - "flow_ms": 2.099999905, - "sklearn_ms": 0.9616669995011762, - "speedup": 0.45793668714531505, - "flow_repeats": 1, + "flow_ms": 0.675887585, + "sklearn_ms": 4.2097040001181085, + "speedup": 6.228408530566675, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "column_transformer", - "status": "different_shape", - "reason": "takes ColumnTransformer first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "ColumnTransformer" + "sklearn_estimator": "ColumnTransformer", + "dataset": "classification", + "flow_ms": 0.004263975, + "sklearn_ms": 2.2767570001178683, + "speedup": 533.9517703827693, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "complement_nb", "sklearn_estimator": "ComplementNB", "dataset": "classification", - "flow_ms": 0.00224, - "sklearn_ms": 0.422583005274646, - "speedup": 188.65312735475268, + "flow_ms": 0.00339505, + "sklearn_ms": 1.952360000018416, + "speedup": 575.0607502152887, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "count_vectorizer", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "CountVectorizer" + "sklearn_estimator": "CountVectorizer", + "dataset": "classification", + "flow_ms": 0.08008137400000001, + "sklearn_ms": 0.8089829999562426, + "speedup": 10.10201198541177, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "dbscan", "sklearn_estimator": "DBSCAN", "dataset": "unsupervised", - "flow_ms": 0.074029997, - "sklearn_ms": 0.4175000067334622, - "speedup": 5.639605884807239, - "flow_repeats": 200, + "flow_ms": 0.269733489, + "sklearn_ms": 1.688398000055713, + "speedup": 6.2595045439675205, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -243,9 +258,9 @@ "flow_estimator": "decision_tree_classifier", "sklearn_estimator": "DecisionTreeClassifier", "dataset": "classification", - "flow_ms": 0.012285, - "sklearn_ms": 0.30191600671969354, - "speedup": 24.575987522970575, + "flow_ms": 0.033078771, + "sklearn_ms": 0.9184079999613459, + "speedup": 27.764272135785998, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -254,26 +269,31 @@ "flow_estimator": "decision_tree_regressor", "sklearn_estimator": "DecisionTreeRegressor", "dataset": "regression", - "flow_ms": 0.661349993, - "sklearn_ms": 1.2142920022597536, - "speedup": 1.8360807667835777, + "flow_ms": 1.209396897, + "sklearn_ms": 2.511968000021625, + "speedup": 2.077041876205198, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "dict_vectorizer", - "status": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DictVectorizer" + "sklearn_estimator": "DictVectorizer", + "dataset": "classification", + "flow_ms": 0.011279195, + "sklearn_ms": 0.7966299999679904, + "speedup": 70.62826735134824, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "dictionary_learning", "sklearn_estimator": "DictionaryLearning", "dataset": "unsupervised", - "flow_ms": 0.37209999699999996, - "sklearn_ms": 199.1827499878127, - "speedup": 535.2936081528986, + "flow_ms": 1.0210337329999999, + "sklearn_ms": 851.8003410000574, + "speedup": 834.2528884890991, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -282,33 +302,43 @@ "flow_estimator": "discriminant_lda", "sklearn_estimator": "LinearDiscriminantAnalysis", "dataset": "classification", - "flow_ms": 0.00833, - "sklearn_ms": 0.3475420089671388, - "speedup": 41.72172976796384, + "flow_ms": 0.031173606, + "sklearn_ms": 1.134993000164286, + "speedup": 36.40878120305639, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "dummy_classifier", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DummyClassifier" + "sklearn_estimator": "DummyClassifier", + "dataset": "classification", + "flow_ms": 0.0025922000000000002, + "sklearn_ms": 0.172664000047007, + "speedup": 66.60905796119395, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "dummy_regressor", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DummyRegressor" + "sklearn_estimator": "DummyRegressor", + "dataset": "regression", + "flow_ms": 0.0006182649999999999, + "sklearn_ms": 0.19580599996515957, + "speedup": 316.7023848433271, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "elastic_net_cv", "sklearn_estimator": "ElasticNetCV", "dataset": "regression", - "flow_ms": 167.832992554, - "sklearn_ms": 13.482125010341406, - "speedup": 0.08033060011131932, - "flow_repeats": 1, + "flow_ms": 2.3781379289999998, + "sklearn_ms": 58.581007999919166, + "speedup": 24.633141453049493, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -316,10 +346,10 @@ "flow_estimator": "elastic_net", "sklearn_estimator": "ElasticNet", "dataset": "regression", - "flow_ms": 0.206999997, - "sklearn_ms": 0.21425001614261419, - "speedup": 1.0350242475733669, - "flow_repeats": 20, + "flow_ms": 0.03178009, + "sklearn_ms": 0.7012510000095062, + "speedup": 22.065733608983056, + "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, @@ -328,17 +358,17 @@ "status": "simplified", "reason": "the implementation's own comments call it simplified, so timing it against scikit-learn's algorithm compares two different things", "sklearn_estimator": "EllipticEnvelope", - "flow_ms": 0.020725001, - "sklearn_ms": 8.94445800804533, + "flow_ms": 0.030406471, + "sklearn_ms": 33.10122200002752, "timing_unit": "ms" }, { "flow_estimator": "empirical_covariance", "sklearn_estimator": "EmpiricalCovariance", "dataset": "unsupervised", - "flow_ms": 0.002105, - "sklearn_ms": 0.12658300693146884, - "speedup": 60.13444509808496, + "flow_ms": 0.00302005, + "sklearn_ms": 0.46602099996562174, + "speedup": 154.3090346072488, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -347,9 +377,9 @@ "flow_estimator": "extra_tree_classifier", "sklearn_estimator": "ExtraTreeClassifier", "dataset": "classification", - "flow_ms": 0.00545, - "sklearn_ms": 0.25049901159945875, - "speedup": 45.963121394396104, + "flow_ms": 0.009513595, + "sklearn_ms": 0.871459999984836, + "speedup": 91.60154494540035, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -358,9 +388,9 @@ "flow_estimator": "extra_tree_regressor", "sklearn_estimator": "ExtraTreeRegressor", "dataset": "regression", - "flow_ms": 0.020909999000000002, - "sklearn_ms": 0.6777079979656264, - "speedup": 32.410714030432345, + "flow_ms": 0.040598379999999996, + "sklearn_ms": 1.3184749999481937, + "speedup": 32.476049535675905, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -369,9 +399,9 @@ "flow_estimator": "extra_trees_classifier", "sklearn_estimator": "ExtraTreesClassifier", "dataset": "classification", - "flow_ms": 0.159500001, - "sklearn_ms": 30.058165997616015, - "speedup": 188.45245021419163, + "flow_ms": 0.361436447, + "sklearn_ms": 93.85145599992484, + "speedup": 259.6624020043137, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -380,10 +410,10 @@ "flow_estimator": "extra_trees_regressor", "sklearn_estimator": "ExtraTreesRegressor", "dataset": "regression", - "flow_ms": 7.530000068, - "sklearn_ms": 68.69250000454485, - "speedup": 9.122509878381697, - "flow_repeats": 1, + "flow_ms": 12.093249886999999, + "sklearn_ms": 143.7838599998713, + "speedup": 11.889596373464181, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -391,9 +421,9 @@ "flow_estimator": "factor_analysis", "sklearn_estimator": "FactorAnalysis", "dataset": "unsupervised", - "flow_ms": 0.015670001, - "sklearn_ms": 0.48095900274347514, - "speedup": 30.69297843334376, + "flow_ms": 0.03069286, + "sklearn_ms": 1.1558020000848046, + "speedup": 37.65703163813358, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -402,9 +432,9 @@ "flow_estimator": "fast_ica", "sklearn_estimator": "FastICA", "dataset": "unsupervised", - "flow_ms": 0.014385, - "sklearn_ms": 0.6858330016257241, - "speedup": 47.67695527464193, + "flow_ms": 0.030853256000000003, + "sklearn_ms": 2.686905000018669, + "speedup": 87.08659468610603, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -413,26 +443,31 @@ "flow_estimator": "feature_agglomeration", "sklearn_estimator": "FeatureAgglomeration", "dataset": "unsupervised", - "flow_ms": 0.001525, - "sklearn_ms": 0.24379098613280803, - "speedup": 159.86294172643147, + "flow_ms": 0.002905285, + "sklearn_ms": 0.8348920000571525, + "speedup": 287.37008591485943, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "feature_union", - "status": "different_shape", - "reason": "takes FeatureUnion first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "FeatureUnion" + "sklearn_estimator": "FeatureUnion", + "dataset": "classification", + "flow_ms": 0.003778615, + "sklearn_ms": 1.2097320000066247, + "speedup": 320.1522250895169, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "gamma_regressor", "sklearn_estimator": "GammaRegressor", "dataset": "regression", - "flow_ms": 0.5390499790000001, - "sklearn_ms": 0.7119990041246638, - "speedup": 1.3208404264211346, + "flow_ms": 0.885098409, + "sklearn_ms": 1.7036070000813197, + "speedup": 1.924765633694998, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -441,9 +476,9 @@ "flow_estimator": "gaussian_mixture", "sklearn_estimator": "GaussianMixture", "dataset": "unsupervised", - "flow_ms": 0.022855001, - "sklearn_ms": 0.8785410027485341, - "speedup": 38.43977091703186, + "flow_ms": 0.060777221, + "sklearn_ms": 3.0233540001063375, + "speedup": 49.74485424574344, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -452,9 +487,9 @@ "flow_estimator": "gaussian_nb", "sklearn_estimator": "GaussianNB", "dataset": "classification", - "flow_ms": 0.002875, - "sklearn_ms": 0.33529099891893566, - "speedup": 116.62295614571676, + "flow_ms": 0.008013485, + "sklearn_ms": 1.5087509999602844, + "speedup": 188.27651140050605, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -463,9 +498,9 @@ "flow_estimator": "gaussian_process_classifier", "sklearn_estimator": "GaussianProcessClassifier", "dataset": "regression", - "flow_ms": 59.547999621, - "sklearn_ms": 2805.2895829896443, - "speedup": 47.10971990401404, + "flow_ms": 160.301707268, + "sklearn_ms": 11363.596219999978, + "speedup": 70.88880345486139, "flow_repeats": 1, "timing_unit": "ms", "status": "ok" @@ -474,26 +509,31 @@ "flow_estimator": "gaussian_process_regressor", "sklearn_estimator": "GaussianProcessRegressor", "dataset": "regression", - "flow_ms": 15.13599968, - "sklearn_ms": 72.50166700396221, - "speedup": 4.790015098887886, - "flow_repeats": 1, + "flow_ms": 24.278685568999997, + "sklearn_ms": 406.58645199994226, + "speedup": 16.746641857707825, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "gaussian_random_projection", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "GaussianRandomProjection" + "sklearn_estimator": "GaussianRandomProjection", + "dataset": "classification", + "flow_ms": 0.002106795, + "sklearn_ms": 0.5969159999494877, + "speedup": 283.32894275403527, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "gradient_boosting_classifier", "sklearn_estimator": "GradientBoostingClassifier", "dataset": "classification", - "flow_ms": 0.250799992, - "sklearn_ms": 62.244832995929755, - "speedup": 248.1851474537916, + "flow_ms": 0.446295712, + "sklearn_ms": 205.04878500003088, + "speedup": 459.4460118855699, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -502,10 +542,10 @@ "flow_estimator": "gradient_boosting_regressor", "sklearn_estimator": "GradientBoostingRegressor", "dataset": "regression", - "flow_ms": 6.671000181999999, - "sklearn_ms": 47.3774999845773, - "speedup": 7.102008498277883, - "flow_repeats": 1, + "flow_ms": 12.140088827, + "sklearn_ms": 105.12064800002463, + "speedup": 8.658968603774339, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -518,9 +558,9 @@ "flow_estimator": "graphical_lasso", "sklearn_estimator": "GraphicalLasso", "dataset": "unsupervised", - "flow_ms": 0.00982, - "sklearn_ms": 0.3913340042345226, - "speedup": 39.85071326217134, + "flow_ms": 0.011175551, + "sklearn_ms": 1.4963180000222565, + "speedup": 133.89210071362535, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -529,10 +569,10 @@ "flow_estimator": "hdbscan", "sklearn_estimator": "HDBSCAN", "dataset": "unsupervised", - "flow_ms": 5.703999996, - "sklearn_ms": 0.7184580026660115, - "speedup": 0.1259568729259886, - "flow_repeats": 1, + "flow_ms": 0.285613716, + "sklearn_ms": 2.133130000061101, + "speedup": 7.46858389693407, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -540,9 +580,9 @@ "flow_estimator": "hist_gradient_boosting_classifier", "sklearn_estimator": "HistGradientBoostingClassifier", "dataset": "classification", - "flow_ms": 0.317299995, - "sklearn_ms": 722.8900420013815, - "speedup": 2278.2541865510634, + "flow_ms": 0.573511535, + "sklearn_ms": 142.59857199999715, + "speedup": 248.6411576708691, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -551,10 +591,10 @@ "flow_estimator": "hist_gradient_boosting_regressor", "sklearn_estimator": "HistGradientBoostingRegressor", "dataset": "regression", - "flow_ms": 1.983999979, - "sklearn_ms": 839.5987920084735, - "speedup": 423.18487948354635, - "flow_repeats": 1, + "flow_ms": 3.510633692, + "sklearn_ms": 129.73937000003843, + "speedup": 36.95611145523024, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -562,26 +602,26 @@ "flow_estimator": "huber_regressor", "sklearn_estimator": "HuberRegressor", "dataset": "regression", - "flow_ms": 2.4989999789999997, - "sklearn_ms": 6.450457018218003, - "speedup": 2.581215315095448, - "flow_repeats": 1, + "flow_ms": 3.6328787410000003, + "sklearn_ms": 25.246068000001287, + "speedup": 6.9493285627947925, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "incremental_pca_partial", "status": "different_shape", - "reason": "takes IncrementalPCA first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "one partial_fit step over one batch, where IncrementalPCA.fit walks the whole design in batches", "sklearn_estimator": "IncrementalPCA" }, { "flow_estimator": "isolation_forest", "sklearn_estimator": "IsolationForest", "dataset": "unsupervised", - "flow_ms": 0.090750001, - "sklearn_ms": 48.3180420123972, - "speedup": 532.4302091456418, + "flow_ms": 0.264021248, + "sklearn_ms": 114.3684740000026, + "speedup": 433.17905231628407, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -590,26 +630,31 @@ "flow_estimator": "isomap", "sklearn_estimator": "Isomap", "dataset": "unsupervised", - "flow_ms": 5.301000118, - "sklearn_ms": 5.399625006248243, - "speedup": 1.0186049586970114, - "flow_repeats": 1, + "flow_ms": 7.891914368, + "sklearn_ms": 10.466826000083529, + "speedup": 1.3262721200478602, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "isotonic", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "IsotonicRegression" + "sklearn_estimator": "IsotonicRegression", + "dataset": "regression", + "flow_ms": 0.042457312999999997, + "sklearn_ms": 0.7902889999513718, + "speedup": 18.61373092431384, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "iterative_imputer", "sklearn_estimator": "IterativeImputer", "dataset": "unsupervised", - "flow_ms": 0.001765, - "sklearn_ms": 2.774332999251783, - "speedup": 1571.8600562333047, + "flow_ms": 0.00362553, + "sklearn_ms": 9.882223000090562, + "speedup": 2725.7319619726113, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -618,9 +663,9 @@ "flow_estimator": "kbins_discretizer", "sklearn_estimator": "KBinsDiscretizer", "dataset": "unsupervised", - "flow_ms": 0.00756, - "sklearn_ms": 0.9220420179190114, - "speedup": 121.96322988346712, + "flow_ms": 0.014869104999999999, + "sklearn_ms": 3.8535880000836187, + "speedup": 259.1674482145105, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -628,21 +673,21 @@ { "flow_estimator": "kernel_density", "sklearn_estimator": "KernelDensity", - "dataset": "unsupervised", - "flow_ms": 0.0, - "sklearn_ms": 0.10016700252890587, - "speedup": null, + "dataset": "classification", + "flow_ms": 0.903082218, + "sklearn_ms": 1.8482180000773951, + "speedup": 2.046566705931303, + "flow_repeats": 200, "timing_unit": "ms", - "status": "below_resolution", - "reason": "Flow side at or under 2e-05 ms, which is the clock's floor here" + "status": "ok" }, { "flow_estimator": "kernel_pca", "sklearn_estimator": "KernelPCA", "dataset": "unsupervised", - "flow_ms": 1.0669000149999999, - "sklearn_ms": 1.8091249949065968, - "speedup": 1.6956837280638684, + "flow_ms": 0.802441329, + "sklearn_ms": 3.591616999983671, + "speedup": 4.475862434029331, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -651,10 +696,10 @@ "flow_estimator": "kernel_ridge", "sklearn_estimator": "KernelRidge", "dataset": "regression", - "flow_ms": 2.2022999519999997, - "sklearn_ms": 1.3769590004812926, - "speedup": 0.6252368117389364, - "flow_repeats": 20, + "flow_ms": 1.427634254, + "sklearn_ms": 6.375353000066752, + "speedup": 4.465676683088855, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -662,10 +707,10 @@ "flow_estimator": "kernel_svc", "sklearn_estimator": "SVC", "dataset": "classification", - "flow_ms": 0.11696500000000001, - "sklearn_ms": 0.6641249929089099, - "speedup": 5.677980531859188, - "flow_repeats": 200, + "flow_ms": 0.48327684400000004, + "sklearn_ms": 2.1343020000585966, + "speedup": 4.416313395844384, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -673,10 +718,10 @@ "flow_estimator": "kernel_svc_multi", "sklearn_estimator": "SVC", "dataset": "classification", - "flow_ms": 0.086784996, - "sklearn_ms": 0.6252509920159355, - "speedup": 7.204597808772561, - "flow_repeats": 200, + "flow_ms": 0.319378852, + "sklearn_ms": 2.14415999994344, + "speedup": 6.713531551999693, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -684,9 +729,9 @@ "flow_estimator": "kmeans", "sklearn_estimator": "KMeans", "dataset": "unsupervised", - "flow_ms": 0.011829999, - "sklearn_ms": 0.7295419927686453, - "speedup": 61.66881271660676, + "flow_ms": 0.036763854, + "sklearn_ms": 2.3912630000495483, + "speedup": 65.04386074565383, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -695,9 +740,9 @@ "flow_estimator": "kneighbors_transformer", "sklearn_estimator": "KNeighborsTransformer", "dataset": "unsupervised", - "flow_ms": 0.436674982, - "sklearn_ms": 0.36108397762291133, - "speedup": 0.8268941260823395, + "flow_ms": 0.483906484, + "sklearn_ms": 1.307234999899265, + "speedup": 2.7014207147909697, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -706,9 +751,9 @@ "flow_estimator": "knn_classifier", "sklearn_estimator": "KNeighborsClassifier", "dataset": "classification", - "flow_ms": 0.112904995, - "sklearn_ms": 1.0927499970421195, - "speedup": 9.67849116898787, + "flow_ms": 0.188434363, + "sklearn_ms": 1.947621999875082, + "speedup": 10.335811201670696, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -717,9 +762,9 @@ "flow_estimator": "knn_imputer", "sklearn_estimator": "KNNImputer", "dataset": "unsupervised", - "flow_ms": 0.0011749999999999998, - "sklearn_ms": 0.1957500062417239, - "speedup": 166.59574999295654, + "flow_ms": 0.0022811150000000002, + "sklearn_ms": 0.513169999976526, + "speedup": 224.9645458368061, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -728,33 +773,43 @@ "flow_estimator": "knn_regressor", "sklearn_estimator": "KNeighborsRegressor", "dataset": "regression", - "flow_ms": 1.247499936, - "sklearn_ms": 1.2899580033263192, - "speedup": 1.0340345246529288, + "flow_ms": 2.261629731, + "sklearn_ms": 3.124693000017942, + "speedup": 1.3816112147749005, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "label_binarizer", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "LabelBinarizer" + "sklearn_estimator": "LabelBinarizer", + "dataset": "classification", + "flow_ms": 0.000686635, + "sklearn_ms": 1.1308150000104433, + "speedup": 1646.8939101712604, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "label_encoder", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "LabelEncoder" + "sklearn_estimator": "LabelEncoder", + "dataset": "classification", + "flow_ms": 0.000732575, + "sklearn_ms": 0.2530630000592282, + "speedup": 345.4431287707446, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "label_propagation", "sklearn_estimator": "LabelPropagation", "dataset": "classification", - "flow_ms": 0.112865001, - "sklearn_ms": 0.9385419834870845, - "speedup": 8.315615781433294, - "flow_repeats": 200, + "flow_ms": 0.250865281, + "sklearn_ms": 1.7934229999809759, + "speedup": 7.148948602341573, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -762,9 +817,9 @@ "flow_estimator": "label_spreading", "sklearn_estimator": "LabelSpreading", "dataset": "classification", - "flow_ms": 0.841149986, - "sklearn_ms": 0.8673739939695224, - "speedup": 1.0311763756832808, + "flow_ms": 1.346854448, + "sklearn_ms": 1.9784300000083022, + "speedup": 1.4689263587065067, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -773,10 +828,10 @@ "flow_estimator": "lars_cv", "sklearn_estimator": "LarsCV", "dataset": "regression", - "flow_ms": 0.136189997, - "sklearn_ms": 2.6954999921144918, - "speedup": 19.79220244871943, - "flow_repeats": 200, + "flow_ms": 0.204928799, + "sklearn_ms": 9.14459299997361, + "speedup": 44.62326937256687, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -784,9 +839,9 @@ "flow_estimator": "lars", "sklearn_estimator": "Lars", "dataset": "regression", - "flow_ms": 0.016220000000000002, - "sklearn_ms": 0.5449580057756975, - "speedup": 33.597904178526356, + "flow_ms": 0.022210696, + "sklearn_ms": 1.8165070000577543, + "speedup": 81.78523536847987, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -795,10 +850,10 @@ "flow_estimator": "lasso_cv", "sklearn_estimator": "LassoCV", "dataset": "regression", - "flow_ms": 1.776449919, - "sklearn_ms": 19.727291000890546, - "speedup": 11.10489566291598, - "flow_repeats": 20, + "flow_ms": 2.20567606, + "sklearn_ms": 61.815946000137956, + "speedup": 28.02584981592354, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -806,9 +861,9 @@ "flow_estimator": "lasso", "sklearn_estimator": "Lasso", "dataset": "regression", - "flow_ms": 0.032230001, - "sklearn_ms": 0.2537920081522316, - "speedup": 7.87440273899562, + "flow_ms": 0.040738391, + "sklearn_ms": 0.6899910000583986, + "speedup": 16.93711958477689, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -817,10 +872,10 @@ "flow_estimator": "lasso_lars_cv", "sklearn_estimator": "LassoLarsCV", "dataset": "regression", - "flow_ms": 3.488999916, - "sklearn_ms": 3.7985839881002903, - "speedup": 1.0887314644751314, - "flow_repeats": 1, + "flow_ms": 4.464474249, + "sklearn_ms": 11.495499000034215, + "speedup": 2.5748830341241407, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -828,9 +883,9 @@ "flow_estimator": "lasso_lars", "sklearn_estimator": "LassoLars", "dataset": "regression", - "flow_ms": 0.046024998, - "sklearn_ms": 0.4093340103281662, - "speedup": 8.893732278449338, + "flow_ms": 0.055809634999999996, + "sklearn_ms": 1.119253999945613, + "speedup": 20.05485253479284, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -839,9 +894,9 @@ "flow_estimator": "lasso_lars_ic", "sklearn_estimator": "LassoLarsIC", "dataset": "regression", - "flow_ms": 0.827899959, - "sklearn_ms": 1.0350410011596978, - "speedup": 1.2502005706219605, + "flow_ms": 0.983146338, + "sklearn_ms": 3.0237460000535066, + "speedup": 3.075580799298575, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -850,9 +905,9 @@ "flow_estimator": "lda", "sklearn_estimator": "LatentDirichletAllocation", "dataset": "unsupervised", - "flow_ms": 6.449000026999999, - "sklearn_ms": 103.45783299999312, - "speedup": 16.042461244665322, + "flow_ms": 27.542872465, + "sklearn_ms": 293.9270550000401, + "speedup": 10.67161950423097, "flow_repeats": 1, "timing_unit": "ms", "status": "ok" @@ -866,9 +921,9 @@ "flow_estimator": "ledoit_wolf_estimator", "sklearn_estimator": "LedoitWolf", "dataset": "unsupervised", - "flow_ms": 0.006625, - "sklearn_ms": 0.17004100664053112, - "speedup": 25.66656704008017, + "flow_ms": 0.015831061, + "sklearn_ms": 0.5891420000807557, + "speedup": 37.21430926712718, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -877,9 +932,9 @@ "flow_estimator": "linear_regression", "sklearn_estimator": "LinearRegression", "dataset": "regression", - "flow_ms": 0.01975, - "sklearn_ms": 0.26516699290368706, - "speedup": 13.42617685588289, + "flow_ms": 0.077784884, + "sklearn_ms": 0.8519740000565434, + "speedup": 10.952950705133706, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -888,9 +943,9 @@ "flow_estimator": "linear_svc", "sklearn_estimator": "LinearSVC", "dataset": "classification", - "flow_ms": 0.019065, - "sklearn_ms": 0.3953749983338639, - "speedup": 20.73826374685885, + "flow_ms": 0.048520435, + "sklearn_ms": 1.3049610000734901, + "speedup": 26.895080393930726, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -899,9 +954,9 @@ "flow_estimator": "linear_svc_multi", "sklearn_estimator": "LinearSVC", "dataset": "classification", - "flow_ms": 0.06523, - "sklearn_ms": 0.37720799446105957, - "speedup": 5.782737919071893, + "flow_ms": 0.114938821, + "sklearn_ms": 1.2918260000560622, + "speedup": 11.23924874830639, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -910,9 +965,9 @@ "flow_estimator": "linear_svr", "sklearn_estimator": "LinearSVR", "dataset": "regression", - "flow_ms": 0.278600007, - "sklearn_ms": 0.23625099856872112, - "speedup": 0.8479935126804254, + "flow_ms": 0.44284824300000003, + "sklearn_ms": 0.6315700001096047, + "speedup": 1.4261544673433528, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -921,10 +976,10 @@ "flow_estimator": "lle", "sklearn_estimator": "LocallyLinearEmbedding", "dataset": "unsupervised", - "flow_ms": 10.61400032, - "sklearn_ms": 7.336875001783483, - "speedup": 0.6912450330304383, - "flow_repeats": 1, + "flow_ms": 3.030586004, + "sklearn_ms": 31.590242999982365, + "speedup": 10.423806801155663, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -932,10 +987,10 @@ "flow_estimator": "local_outlier_factor", "sklearn_estimator": "LocalOutlierFactor", "dataset": "unsupervised", - "flow_ms": 5.397999763, - "sklearn_ms": 0.395416995161213, - "speedup": 0.07325250324602747, - "flow_repeats": 1, + "flow_ms": 0.776482224, + "sklearn_ms": 1.4632869999786635, + "speedup": 1.8845080476416205, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -948,9 +1003,9 @@ "flow_estimator": "logistic_regression_cv", "sklearn_estimator": "LogisticRegressionCV", "dataset": "classification", - "flow_ms": 59.974998474, - "sklearn_ms": 92.93033300491516, - "speedup": 1.5494845413827187, + "flow_ms": 68.928015889, + "sklearn_ms": 490.324309000016, + "speedup": 7.113570624020607, "flow_repeats": 1, "timing_unit": "ms", "status": "ok" @@ -959,9 +1014,9 @@ "flow_estimator": "logistic_regression", "sklearn_estimator": "LogisticRegression", "dataset": "classification", - "flow_ms": 0.27125001, - "sklearn_ms": 4.276374995242804, - "speedup": 15.76543718926611, + "flow_ms": 0.323040694, + "sklearn_ms": 24.858971999947244, + "speedup": 76.95306647634692, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -975,9 +1030,9 @@ "flow_estimator": "maxabs_scaler", "sklearn_estimator": "MaxAbsScaler", "dataset": "unsupervised", - "flow_ms": 0.0004, - "sklearn_ms": 0.10537500202190131, - "speedup": 263.4375050547533, + "flow_ms": 0.001283655, + "sklearn_ms": 0.35910400004013354, + "speedup": 279.7511792811414, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -986,10 +1041,10 @@ "flow_estimator": "mds", "sklearn_estimator": "MDS", "dataset": "unsupervised", - "flow_ms": 39.657001495, - "sklearn_ms": 19.73887500935234, - "speedup": 0.4977399769329771, - "flow_repeats": 1, + "flow_ms": 7.554921627, + "sklearn_ms": 44.649359999993976, + "speedup": 5.909969977772475, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -997,10 +1052,10 @@ "flow_estimator": "mean_shift", "sklearn_estimator": "MeanShift", "dataset": "unsupervised", - "flow_ms": 1.259599924, - "sklearn_ms": 124.62391699955333, - "speedup": 98.93928589944352, - "flow_repeats": 20, + "flow_ms": 2.524850368, + "sklearn_ms": 451.90079400003924, + "speedup": 178.98121794758146, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1008,9 +1063,9 @@ "flow_estimator": "min_cov_det", "sklearn_estimator": "MinCovDet", "dataset": "unsupervised", - "flow_ms": 0.20085001, - "sklearn_ms": 8.830791994114406, - "speedup": 43.96709760738576, + "flow_ms": 0.326487094, + "sklearn_ms": 33.63587000001189, + "speedup": 103.0235823043342, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1019,10 +1074,10 @@ "flow_estimator": "minibatch_dictionary_learning", "sklearn_estimator": "MiniBatchDictionaryLearning", "dataset": "unsupervised", - "flow_ms": 0.121835001, - "sklearn_ms": 191.07929299934767, - "speedup": 1568.3448223499229, - "flow_repeats": 200, + "flow_ms": 0.226453438, + "sklearn_ms": 700.8259520000593, + "speedup": 3094.7905149492994, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1030,9 +1085,9 @@ "flow_estimator": "minibatch_kmeans", "sklearn_estimator": "MiniBatchKMeans", "dataset": "unsupervised", - "flow_ms": 0.080454998, - "sklearn_ms": 6.989082990912721, - "speedup": 86.8694694506452, + "flow_ms": 0.109413236, + "sklearn_ms": 4.321632000142017, + "speedup": 39.498255952707744, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1041,9 +1096,9 @@ "flow_estimator": "minibatch_nmf", "sklearn_estimator": "MiniBatchNMF", "dataset": "unsupervised", - "flow_ms": 0.079229996, - "sklearn_ms": 7.578208009363152, - "speedup": 95.64821900739655, + "flow_ms": 0.152375652, + "sklearn_ms": 22.962587999927564, + "speedup": 150.69722556414436, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1052,9 +1107,9 @@ "flow_estimator": "minibatch_sparse_pca", "sklearn_estimator": "MiniBatchSparsePCA", "dataset": "unsupervised", - "flow_ms": 0.00238, - "sklearn_ms": 5.608374995063059, - "speedup": 2356.4600819592683, + "flow_ms": 0.00547114, + "sklearn_ms": 14.838334000046416, + "speedup": 2712.1100904101186, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1063,9 +1118,9 @@ "flow_estimator": "minmax_scaler", "sklearn_estimator": "MinMaxScaler", "dataset": "unsupervised", - "flow_ms": 0.00096, - "sklearn_ms": 0.09683400276117027, - "speedup": 100.86875287621902, + "flow_ms": 0.002520415, + "sklearn_ms": 0.3418989999772748, + "speedup": 135.6518668462435, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1074,9 +1129,9 @@ "flow_estimator": "missing_indicator", "sklearn_estimator": "MissingIndicator", "dataset": "unsupervised", - "flow_ms": 0.00037, - "sklearn_ms": 0.29858300695195794, - "speedup": 806.9810998701566, + "flow_ms": 0.0007635799999999999, + "sklearn_ms": 0.7797780000373677, + "speedup": 1021.2132324541866, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1085,10 +1140,10 @@ "flow_estimator": "mlp_classifier", "sklearn_estimator": "MLPClassifier", "dataset": "classification", - "flow_ms": 0.9091500579999999, - "sklearn_ms": 32.084249003673904, - "speedup": 35.29037777795963, - "flow_repeats": 20, + "flow_ms": 3.271640919, + "sklearn_ms": 92.25256500008072, + "speedup": 28.197643715826356, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1096,10 +1151,10 @@ "flow_estimator": "mlp_regressor", "sklearn_estimator": "MLPRegressor", "dataset": "regression", - "flow_ms": 3.5689998860000003, - "sklearn_ms": 79.0593749989057, - "speedup": 22.151688855197037, - "flow_repeats": 1, + "flow_ms": 11.055358917, + "sklearn_ms": 210.52720000000136, + "speedup": 19.043000012986496, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1107,9 +1162,9 @@ "flow_estimator": "multi_output_classifier", "sklearn_estimator": "MultiOutputClassifier", "dataset": "multioutput_class", - "flow_ms": 0.5503500100000001, - "sklearn_ms": 0.8460840035695583, - "speedup": 1.537356206406825, + "flow_ms": 0.8313319010000001, + "sklearn_ms": 2.8003679999528686, + "speedup": 3.368531866255026, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1118,9 +1173,9 @@ "flow_estimator": "multi_output_regressor", "sklearn_estimator": "MultiOutputRegressor", "dataset": "multioutput", - "flow_ms": 0.041465000999999994, - "sklearn_ms": 1.3581249804701656, - "speedup": 32.753525810120344, + "flow_ms": 0.097360734, + "sklearn_ms": 3.1001680000599663, + "speedup": 31.842077115605623, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1129,26 +1184,31 @@ "flow_estimator": "multiclass_logistic", "sklearn_estimator": "LogisticRegression", "dataset": "classification", - "flow_ms": 0.31905, - "sklearn_ms": 4.690916000981815, - "speedup": 14.702761325754004, + "flow_ms": 0.7356698540000001, + "sklearn_ms": 24.854746999949384, + "speedup": 33.78519163835328, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "multilabel_binarizer", - "status": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "MultiLabelBinarizer" + "sklearn_estimator": "MultiLabelBinarizer", + "dataset": "multioutput_class", + "flow_ms": 0.00098303, + "sklearn_ms": 0.2710369999476825, + "speedup": 275.71589874946085, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "multinomial_nb", "sklearn_estimator": "MultinomialNB", "dataset": "classification", - "flow_ms": 0.00238, - "sklearn_ms": 0.43749899487011135, - "speedup": 183.82310708828206, + "flow_ms": 0.0039587500000000005, + "sklearn_ms": 1.9129380000322271, + "speedup": 483.2176823573671, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1157,10 +1217,10 @@ "flow_estimator": "multitask_elastic_net_cv", "sklearn_estimator": "MultiTaskElasticNetCV", "dataset": "multioutput", - "flow_ms": 171.37600708, - "sklearn_ms": 17.658540993579663, - "speedup": 0.10303975039712814, - "flow_repeats": 1, + "flow_ms": 3.4497643940000002, + "sklearn_ms": 27.38369600012902, + "speedup": 7.937845276551665, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1168,10 +1228,10 @@ "flow_estimator": "multitask_elastic_net", "sklearn_estimator": "MultiTaskElasticNet", "dataset": "multioutput", - "flow_ms": 0.45095000500000004, - "sklearn_ms": 0.2088330074911937, - "speedup": 0.4630956983606058, - "flow_repeats": 20, + "flow_ms": 0.077111616, + "sklearn_ms": 0.6105520000119213, + "speedup": 7.917769483808009, + "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, @@ -1179,10 +1239,10 @@ "flow_estimator": "multitask_lasso_cv", "sklearn_estimator": "MultiTaskLassoCV", "dataset": "multioutput", - "flow_ms": 187.596993042, - "sklearn_ms": 111.14133299270179, - "speedup": 0.5924473052071736, - "flow_repeats": 1, + "flow_ms": 5.302778108999999, + "sklearn_ms": 81.08658999992713, + "speedup": 15.291341318299754, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1190,10 +1250,10 @@ "flow_estimator": "multitask_lasso", "sklearn_estimator": "MultiTaskLasso", "dataset": "multioutput", - "flow_ms": 0.449449994, - "sklearn_ms": 0.2170000079786405, - "speedup": 0.48281235037382264, - "flow_repeats": 20, + "flow_ms": 0.10905181, + "sklearn_ms": 0.6186370000023089, + "speedup": 5.672872371419685, + "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, @@ -1202,17 +1262,17 @@ "status": "simplified", "reason": "the implementation's own comments call it simplified, so timing it against scikit-learn's algorithm compares two different things", "sklearn_estimator": "NeighborhoodComponentsAnalysis", - "flow_ms": 169.713004028, - "sklearn_ms": 36.182417010422796, + "flow_ms": 361.524179834, + "sklearn_ms": 1338.105832999986, "timing_unit": "ms" }, { "flow_estimator": "nearest_centroid", "sklearn_estimator": "NearestCentroid", "dataset": "classification", - "flow_ms": 0.001915, - "sklearn_ms": 0.6288340082392097, - "speedup": 328.37285025546197, + "flow_ms": 0.0027735449999999997, + "sklearn_ms": 1.8010690000664908, + "speedup": 649.3743566686285, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1220,21 +1280,21 @@ { "flow_estimator": "nearest_neighbors", "sklearn_estimator": "NearestNeighbors", - "dataset": "unsupervised", - "flow_ms": 0.0, - "sklearn_ms": 0.1144580019172281, - "speedup": null, + "dataset": "classification", + "flow_ms": 0.221180172, + "sklearn_ms": 1.2016480000056617, + "speedup": 5.432892058722432, + "flow_repeats": 200, "timing_unit": "ms", - "status": "below_resolution", - "reason": "Flow side at or under 2e-05 ms, which is the clock's floor here" + "status": "ok" }, { "flow_estimator": "nmf", "sklearn_estimator": "NMF", "dataset": "unsupervised", - "flow_ms": 0.477299983, - "sklearn_ms": 2.7645420050248504, - "speedup": 5.792042957237755, + "flow_ms": 0.953625521, + "sklearn_ms": 7.304852999936884, + "speedup": 7.660085472835079, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1243,10 +1303,10 @@ "flow_estimator": "nu_svc", "sklearn_estimator": "NuSVC", "dataset": "regression", - "flow_ms": 2.119000114, - "sklearn_ms": 63.21224999555852, - "speedup": 29.83116875639702, - "flow_repeats": 1, + "flow_ms": 4.718885839, + "sklearn_ms": 147.98595299987483, + "speedup": 31.36035878995436, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1254,10 +1314,10 @@ "flow_estimator": "nu_svr", "sklearn_estimator": "NuSVR", "dataset": "regression", - "flow_ms": 128.55500328600002, - "sklearn_ms": 4.312457997002639, - "speedup": 0.03354562550481672, - "flow_repeats": 1, + "flow_ms": 6.613256693, + "sklearn_ms": 8.755586000006588, + "speedup": 1.3239446775556436, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1265,9 +1325,9 @@ "flow_estimator": "nystroem", "sklearn_estimator": "Nystroem", "dataset": "unsupervised", - "flow_ms": 0.018185, - "sklearn_ms": 1.2665829999605194, - "speedup": 69.64987626948141, + "flow_ms": 0.031534529, + "sklearn_ms": 3.499335999890718, + "speedup": 110.9683927700559, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1281,9 +1341,9 @@ "flow_estimator": "oas_estimator", "sklearn_estimator": "OAS", "dataset": "unsupervised", - "flow_ms": 0.006135, - "sklearn_ms": 0.11737500608433038, - "speedup": 19.132030331594194, + "flow_ms": 0.008390545, + "sklearn_ms": 0.5394989999558675, + "speedup": 64.29844544733001, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1297,9 +1357,9 @@ "flow_estimator": "omp_cv", "sklearn_estimator": "OrthogonalMatchingPursuitCV", "dataset": "regression", - "flow_ms": 0.037935, - "sklearn_ms": 1.2267079873709008, - "speedup": 32.337102606323995, + "flow_ms": 0.042472535, + "sklearn_ms": 5.610864999994192, + "speedup": 132.1057243226521, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1308,10 +1368,10 @@ "flow_estimator": "one_class_svm", "sklearn_estimator": "OneClassSVM", "dataset": "unsupervised", - "flow_ms": 8.137999535, - "sklearn_ms": 0.5150840006535873, - "speedup": 0.06329368764870386, - "flow_repeats": 1, + "flow_ms": 0.641945601, + "sklearn_ms": 2.0310889999564097, + "speedup": 3.1639581247888477, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1319,10 +1379,10 @@ "flow_estimator": "one_vs_one", "sklearn_estimator": "OneVsOneClassifier", "dataset": "classification", - "flow_ms": 0.116269994, - "sklearn_ms": 1.3165410055080429, - "speedup": 11.323136436285038, - "flow_repeats": 200, + "flow_ms": 0.556118604, + "sklearn_ms": 5.569117000050028, + "speedup": 10.014261274470918, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1330,9 +1390,9 @@ "flow_estimator": "one_vs_rest", "sklearn_estimator": "OneVsRestClassifier", "dataset": "classification", - "flow_ms": 0.2055, - "sklearn_ms": 1.3594179908977821, - "speedup": 6.615172705098697, + "flow_ms": 0.8621675470000001, + "sklearn_ms": 5.9599170000410595, + "speedup": 6.912713220045452, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1341,9 +1401,9 @@ "flow_estimator": "onehot_encoder", "sklearn_estimator": "OneHotEncoder", "dataset": "unsupervised", - "flow_ms": 0.029215002, - "sklearn_ms": 0.5092510109534487, - "speedup": 17.431147564304418, + "flow_ms": 0.08732248100000001, + "sklearn_ms": 1.8993019998561067, + "speedup": 21.750435604963016, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1352,10 +1412,10 @@ "flow_estimator": "optics", "sklearn_estimator": "OPTICS", "dataset": "unsupervised", - "flow_ms": 2.779999971, - "sklearn_ms": 33.755458003724925, - "speedup": 12.142251207140363, - "flow_repeats": 1, + "flow_ms": 4.975696087, + "sklearn_ms": 118.60569999998916, + "speedup": 23.83700650646052, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1363,9 +1423,9 @@ "flow_estimator": "ordinal_encoder", "sklearn_estimator": "OrdinalEncoder", "dataset": "unsupervised", - "flow_ms": 0.0181, - "sklearn_ms": 0.47520900261588395, - "speedup": 26.254641028501872, + "flow_ms": 0.057152648, + "sklearn_ms": 1.5771399999948699, + "speedup": 27.59522183460108, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1374,9 +1434,9 @@ "flow_estimator": "orthogonal_matching_pursuit", "sklearn_estimator": "OrthogonalMatchingPursuit", "dataset": "regression", - "flow_ms": 0.027035, - "sklearn_ms": 0.1994580088648945, - "speedup": 7.377769885884761, + "flow_ms": 0.047719261, + "sklearn_ms": 0.638073999880362, + "speedup": 13.37141411054882, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1385,9 +1445,9 @@ "flow_estimator": "output_code", "sklearn_estimator": "OutputCodeClassifier", "dataset": "classification", - "flow_ms": 0.264249997, - "sklearn_ms": 1.7742080090101808, - "speedup": 6.714126884210261, + "flow_ms": 0.671938227, + "sklearn_ms": 3.896908000001531, + "speedup": 5.799503352265045, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1396,9 +1456,9 @@ "flow_estimator": "passive_aggressive_classifier", "sklearn_estimator": "PassiveAggressiveClassifier", "dataset": "regression", - "flow_ms": 0.116094999, - "sklearn_ms": 29.98487501463387, - "speedup": 258.2787826599996, + "flow_ms": 0.176138761, + "sklearn_ms": 81.56729399991036, + "speedup": 463.0854306958044, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1407,9 +1467,9 @@ "flow_estimator": "passive_aggressive_regressor", "sklearn_estimator": "PassiveAggressiveRegressor", "dataset": "regression", - "flow_ms": 0.822950006, - "sklearn_ms": 0.758833994041197, - "speedup": 0.9220900279587543, + "flow_ms": 0.96526639, + "sklearn_ms": 2.3702439999624403, + "speedup": 2.4555335444368267, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1418,9 +1478,9 @@ "flow_estimator": "pca", "sklearn_estimator": "PCA", "dataset": "unsupervised", - "flow_ms": 0.00726, - "sklearn_ms": 0.1589170133229345, - "speedup": 21.889395774508884, + "flow_ms": 0.009322835, + "sklearn_ms": 0.642852999931165, + "speedup": 68.95466882457589, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1429,26 +1489,31 @@ "flow_estimator": "perceptron", "sklearn_estimator": "Perceptron", "dataset": "regression", - "flow_ms": 0.21455000700000001, - "sklearn_ms": 26.87045800848864, - "speedup": 125.24100271172976, + "flow_ms": 0.36043115, + "sklearn_ms": 76.11837199999627, + "speedup": 211.18699646242084, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "pipeline", - "status": "different_shape", - "reason": "takes Pipeline first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "Pipeline" + "sklearn_estimator": "Pipeline", + "dataset": "classification", + "flow_ms": 0.1270396, + "sklearn_ms": 6.738113999972484, + "speedup": 53.039477454057504, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "pls_canonical", "sklearn_estimator": "PLSCanonical", "dataset": "multioutput", - "flow_ms": 0.043340000000000004, - "sklearn_ms": 0.3192919975845143, - "speedup": 7.367143460648691, + "flow_ms": 0.057907349, + "sklearn_ms": 1.2296609999111752, + "speedup": 21.23497312769706, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1457,9 +1522,9 @@ "flow_estimator": "pls", "sklearn_estimator": "PLSRegression", "dataset": "multioutput", - "flow_ms": 0.067739999, - "sklearn_ms": 0.351291018887423, - "speedup": 5.185872808876526, + "flow_ms": 0.09176203499999999, + "sklearn_ms": 1.3212410000278396, + "speedup": 14.398558183979244, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1468,9 +1533,9 @@ "flow_estimator": "pls_svd", "sklearn_estimator": "PLSSVD", "dataset": "multioutput", - "flow_ms": 0.019864999, - "sklearn_ms": 0.2077509998343885, - "speedup": 10.458142979739817, + "flow_ms": 0.028758336, + "sklearn_ms": 0.8118480000121053, + "speedup": 28.23000607587676, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1479,32 +1544,42 @@ "flow_estimator": "poisson_regressor", "sklearn_estimator": "PoissonRegressor", "dataset": "regression", - "flow_ms": 0.472000023, - "sklearn_ms": 3.4686659928411245, - "speedup": 7.348868270799056, + "flow_ms": 0.48869954200000004, + "sklearn_ms": 189.62414199995692, + "speedup": 388.0178426685673, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "polynomial_count_sketch", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "PolynomialCountSketch" + "sklearn_estimator": "PolynomialCountSketch", + "dataset": "classification", + "flow_ms": 0.001621635, + "sklearn_ms": 0.7887749999326843, + "speedup": 486.4072370987826, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "polynomial_features", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "PolynomialFeatures" + "sklearn_estimator": "PolynomialFeatures", + "dataset": "classification", + "flow_ms": 0.003157165, + "sklearn_ms": 0.4222800000661664, + "speedup": 133.7529080887969, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "power_transformer", "sklearn_estimator": "PowerTransformer", "dataset": "unsupervised", - "flow_ms": 0.063644999, - "sklearn_ms": 7.3907919868361205, - "speedup": 116.12525890425611, + "flow_ms": 0.167425743, + "sklearn_ms": 57.61654599996291, + "speedup": 344.1319415256405, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1513,9 +1588,9 @@ "flow_estimator": "qda", "sklearn_estimator": "QuadraticDiscriminantAnalysis", "dataset": "classification", - "flow_ms": 0.00932, - "sklearn_ms": 0.29850001737941056, - "speedup": 32.02789886045178, + "flow_ms": 0.033012441000000003, + "sklearn_ms": 1.007103000006282, + "speedup": 30.506771674541785, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1524,9 +1599,9 @@ "flow_estimator": "quantile_regressor", "sklearn_estimator": "QuantileRegressor", "dataset": "regression", - "flow_ms": 0.207649991, - "sklearn_ms": 7.948208003654145, - "speedup": 38.27694846205962, + "flow_ms": 0.706012465, + "sklearn_ms": 15.52165199996125, + "speedup": 21.984954614025476, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1535,9 +1610,9 @@ "flow_estimator": "quantile_transformer", "sklearn_estimator": "QuantileTransformer", "dataset": "unsupervised", - "flow_ms": 0.008825, - "sklearn_ms": 0.33608300145715475, - "speedup": 38.08305965520167, + "flow_ms": 0.025420093999999997, + "sklearn_ms": 0.9565179999526663, + "speedup": 37.62842104174227, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1546,9 +1621,9 @@ "flow_estimator": "radius_neighbors_classifier", "sklearn_estimator": "RadiusNeighborsClassifier", "dataset": "classification", - "flow_ms": 0.097194999, - "sklearn_ms": 0.6490420055342838, - "speedup": 6.6777304615671, + "flow_ms": 0.27262786099999997, + "sklearn_ms": 2.0071049999614843, + "speedup": 7.3620685450100956, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1557,9 +1632,9 @@ "flow_estimator": "radius_neighbors_regressor", "sklearn_estimator": "RadiusNeighborsRegressor", "dataset": "regression", - "flow_ms": 0.975350022, - "sklearn_ms": 2.382626000326127, - "speedup": 2.442842001931207, + "flow_ms": 1.8420908759999999, + "sklearn_ms": 6.955469000104131, + "speedup": 3.7758555187068477, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1568,9 +1643,9 @@ "flow_estimator": "radius_neighbors_transformer", "sklearn_estimator": "RadiusNeighborsTransformer", "dataset": "unsupervised", - "flow_ms": 0.061634999, - "sklearn_ms": 0.4621660045813769, - "speedup": 7.498434527132496, + "flow_ms": 0.071648055, + "sklearn_ms": 1.3511280000102488, + "speedup": 18.857846176148797, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1579,9 +1654,9 @@ "flow_estimator": "random_forest_classifier", "sklearn_estimator": "RandomForestClassifier", "dataset": "classification", - "flow_ms": 0.10339999400000001, - "sklearn_ms": 42.424207989824936, - "speedup": 410.29217071158564, + "flow_ms": 0.518304422, + "sklearn_ms": 123.11260900003163, + "speedup": 237.5295362616675, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1590,10 +1665,10 @@ "flow_estimator": "random_forest_regressor", "sklearn_estimator": "RandomForestRegressor", "dataset": "regression", - "flow_ms": 7.646000094000001, - "sklearn_ms": 106.43045901088044, - "speedup": 13.919756435054058, - "flow_repeats": 1, + "flow_ms": 11.975702308, + "sklearn_ms": 215.80869000001712, + "speedup": 18.020545638968727, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1601,9 +1676,9 @@ "flow_estimator": "random_trees_embedding", "sklearn_estimator": "RandomTreesEmbedding", "dataset": "unsupervised", - "flow_ms": 0.72344997, - "sklearn_ms": 36.00624999671709, - "speedup": 49.77020041443514, + "flow_ms": 1.2510788959999999, + "sklearn_ms": 113.51824499990926, + "speedup": 90.73627999229656, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1612,26 +1687,31 @@ "flow_estimator": "ransac_regressor", "sklearn_estimator": "RANSACRegressor", "dataset": "regression", - "flow_ms": 0.025629999, - "sklearn_ms": 24.211207986809313, - "speedup": 944.6433449649886, + "flow_ms": 0.034505641, + "sklearn_ms": 83.36336199988637, + "speedup": 2415.934310563492, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "rbf_sampler", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "RBFSampler" + "sklearn_estimator": "RBFSampler", + "dataset": "classification", + "flow_ms": 0.005800545, + "sklearn_ms": 0.4693470000347588, + "speedup": 80.91429340428509, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "regressor_chain", "sklearn_estimator": "RegressorChain", "dataset": "multioutput_class", - "flow_ms": 0.00694, - "sklearn_ms": 0.6237920024432242, - "speedup": 89.8835738390813, + "flow_ms": 0.01421749, + "sklearn_ms": 2.086232999999993, + "speedup": 146.73708228386255, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1640,9 +1720,9 @@ "flow_estimator": "rfe", "sklearn_estimator": "RFE", "dataset": "regression", - "flow_ms": 0.016205, - "sklearn_ms": 8.93216700933408, - "speedup": 551.1982110048799, + "flow_ms": 0.022755404, + "sklearn_ms": 14.991329999929803, + "speedup": 658.8030693689202, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1651,10 +1731,10 @@ "flow_estimator": "rfecv", "sklearn_estimator": "RFECV", "dataset": "regression", - "flow_ms": 4.782000065, - "sklearn_ms": 74.68254200648516, - "speedup": 15.61742806176335, - "flow_repeats": 1, + "flow_ms": 9.030509949, + "sklearn_ms": 169.2397679998976, + "speedup": 18.740887165363066, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1662,10 +1742,10 @@ "flow_estimator": "ridge_classifier_cv", "sklearn_estimator": "RidgeClassifierCV", "dataset": "classification", - "flow_ms": 25.680000305, - "sklearn_ms": 0.9509170049568638, - "speedup": 0.03702947794637356, - "flow_repeats": 1, + "flow_ms": 1.200794815, + "sklearn_ms": 2.7179740000065067, + "speedup": 2.263479127369905, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1673,9 +1753,9 @@ "flow_estimator": "ridge_classifier", "sklearn_estimator": "RidgeClassifier", "dataset": "regression", - "flow_ms": 0.206200003, - "sklearn_ms": 1.3081669894745573, - "speedup": 6.344165715043939, + "flow_ms": 0.377169952, + "sklearn_ms": 9.335010000086186, + "speedup": 24.7501423445476, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1684,9 +1764,9 @@ "flow_estimator": "ridge_cv", "sklearn_estimator": "RidgeCV", "dataset": "regression", - "flow_ms": 0.723699987, - "sklearn_ms": 0.3534170100465417, - "speedup": 0.48834740416617095, + "flow_ms": 0.31451921200000005, + "sklearn_ms": 1.4564850000624574, + "speedup": 4.63082999223099, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1695,9 +1775,9 @@ "flow_estimator": "ridge", "sklearn_estimator": "Ridge", "dataset": "regression", - "flow_ms": 0.017119999, - "sklearn_ms": 0.24566700449213386, - "speedup": 14.34970904450017, + "flow_ms": 0.048897801000000005, + "sklearn_ms": 1.1045060000469675, + "speedup": 22.588050535175753, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1706,9 +1786,9 @@ "flow_estimator": "robust_scaler", "sklearn_estimator": "RobustScaler", "dataset": "unsupervised", - "flow_ms": 0.00709, - "sklearn_ms": 0.2803750103339553, - "speedup": 39.54513544907691, + "flow_ms": 0.012145965, + "sklearn_ms": 1.1932519998936186, + "speedup": 98.24266741206802, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1717,9 +1797,9 @@ "flow_estimator": "select_fdr", "sklearn_estimator": "SelectFdr", "dataset": "regression", - "flow_ms": 1.146900088, - "sklearn_ms": 4.7365840000566095, - "speedup": 4.1299011567053, + "flow_ms": 1.804273007, + "sklearn_ms": 18.01914400004989, + "speedup": 9.986927660138681, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1728,27 +1808,32 @@ "flow_estimator": "select_fpr", "sklearn_estimator": "SelectFpr", "dataset": "regression", - "flow_ms": 1.137300007, - "sklearn_ms": 4.5432499900925905, - "speedup": 3.9947682776129545, - "flow_repeats": 20, + "flow_ms": 3.2428168939999997, + "sklearn_ms": 17.53608899991832, + "speedup": 5.4076716549628046, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "select_from_model", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SelectFromModel" + "sklearn_estimator": "SelectFromModel", + "dataset": "classification", + "flow_ms": 0.000709875, + "sklearn_ms": 1.2076079999587819, + "speedup": 1701.1558372372344, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "select_fwe", "sklearn_estimator": "SelectFwe", "dataset": "regression", - "flow_ms": 1.1540500530000002, - "sklearn_ms": 4.57766700128559, - "speedup": 3.966610451068182, - "flow_repeats": 20, + "flow_ms": 3.223793273, + "sklearn_ms": 17.634742999916853, + "speedup": 5.470184191899594, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1756,10 +1841,10 @@ "flow_estimator": "select_k_best", "sklearn_estimator": "SelectKBest", "dataset": "regression", - "flow_ms": 1.096850038, - "sklearn_ms": 4.83825099945534, - "speedup": 4.411041465866586, - "flow_repeats": 20, + "flow_ms": 3.520756677, + "sklearn_ms": 17.543013000022256, + "speedup": 4.9827393965124775, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1767,27 +1852,27 @@ "flow_estimator": "select_percentile", "sklearn_estimator": "SelectPercentile", "dataset": "regression", - "flow_ms": 1.097350095, - "sklearn_ms": 4.657249999581836, - "speedup": 4.244087662453646, - "flow_repeats": 20, + "flow_ms": 3.1130482699999997, + "sklearn_ms": 17.743909000046187, + "speedup": 5.699850262857051, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "self_training_classifier", "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes class probabilities as input, so it is the labelling loop alone while SelfTrainingClassifier also fits the base estimator on every round", "sklearn_estimator": "SelfTrainingClassifier" }, { "flow_estimator": "sequential_feature_selector", "sklearn_estimator": "SequentialFeatureSelector", "dataset": "regression", - "flow_ms": 3.413000107, - "sklearn_ms": 78.29795898578595, - "speedup": 22.94109479375588, - "flow_repeats": 1, + "flow_ms": 8.100300247, + "sklearn_ms": 241.21655400006148, + "speedup": 29.77871765795319, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1795,9 +1880,9 @@ "flow_estimator": "sgd_classifier", "sklearn_estimator": "SGDClassifier", "dataset": "classification", - "flow_ms": 0.231749991, - "sklearn_ms": 0.8100010018097237, - "speedup": 3.4951500896054997, + "flow_ms": 0.245212604, + "sklearn_ms": 2.4942259999534144, + "speedup": 10.171687585656953, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1806,9 +1891,9 @@ "flow_estimator": "sgd_one_class_svm", "sklearn_estimator": "SGDOneClassSVM", "dataset": "unsupervised", - "flow_ms": 0.046094998, - "sklearn_ms": 0.7897070026956499, - "speedup": 17.13216264150071, + "flow_ms": 0.07873335399999999, + "sklearn_ms": 2.0009319999871877, + "speedup": 25.414032278965124, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1817,9 +1902,9 @@ "flow_estimator": "sgd_regressor", "sklearn_estimator": "SGDRegressor", "dataset": "regression", - "flow_ms": 0.390450001, - "sklearn_ms": 11.209875010536052, - "speedup": 28.710142097133847, + "flow_ms": 0.450834644, + "sklearn_ms": 29.666637999980594, + "speedup": 65.80381165201801, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -1828,9 +1913,9 @@ "flow_estimator": "shrunk_covariance", "sklearn_estimator": "ShrunkCovariance", "dataset": "unsupervised", - "flow_ms": 0.002245, - "sklearn_ms": 0.16125000547617674, - "speedup": 71.82628306288495, + "flow_ms": 0.00306603, + "sklearn_ms": 0.5750850000367791, + "speedup": 187.56665787248627, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1839,26 +1924,31 @@ "flow_estimator": "simple_imputer", "sklearn_estimator": "SimpleImputer", "dataset": "unsupervised", - "flow_ms": 0.00126, - "sklearn_ms": 0.417499992181547, - "speedup": 331.3492001440849, + "flow_ms": 0.00205144, + "sklearn_ms": 1.1681749999752356, + "speedup": 569.4414654950842, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "skewed_chi2_sampler", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SkewedChi2Sampler" + "sklearn_estimator": "SkewedChi2Sampler", + "dataset": "classification", + "flow_ms": 0.015763485, + "sklearn_ms": 0.48293200006810366, + "speedup": 30.63611885748003, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "sparse_coder", "sklearn_estimator": "SparseCoder", "dataset": "unsupervised", - "flow_ms": 0.206245005, - "sklearn_ms": 1.0421250044601038, - "speedup": 5.05284966518391, + "flow_ms": 0.33392349, + "sklearn_ms": 3.7383619999218354, + "speedup": 11.195265118730747, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1867,36 +1957,41 @@ "flow_estimator": "sparse_pca", "sklearn_estimator": "SparsePCA", "dataset": "unsupervised", - "flow_ms": 0.004145, - "sklearn_ms": 7.9594590060878545, - "speedup": 1920.2554900091325, + "flow_ms": 0.007683675, + "sklearn_ms": 28.504043999987516, + "speedup": 3709.688918387037, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "sparse_random_projection", - "status": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SparseRandomProjection" + "sklearn_estimator": "SparseRandomProjection", + "dataset": "classification", + "flow_ms": 0.00163441, + "sklearn_ms": 0.6710559999874022, + "speedup": 410.5799646278487, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "spectral_biclustering", "status": "simplified", "reason": "the implementation's own comments call it simplified, so timing it against scikit-learn's algorithm compares two different things", "sklearn_estimator": "SpectralBiclustering", - "flow_ms": 0.00094, - "sklearn_ms": 23.599958993145265, + "flow_ms": 0.001551105, + "sklearn_ms": 61.68861300000117, "timing_unit": "ms" }, { "flow_estimator": "spectral_clustering", "sklearn_estimator": "SpectralClustering", "dataset": "unsupervised", - "flow_ms": 4.743000031, - "sklearn_ms": 5.601083001238294, - "speedup": 1.1809156577334827, - "flow_repeats": 1, + "flow_ms": 6.63312006, + "sklearn_ms": 28.610673000002862, + "speedup": 4.3133054642769215, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1905,18 +2000,18 @@ "status": "simplified", "reason": "the implementation's own comments call it simplified, so timing it against scikit-learn's algorithm compares two different things", "sklearn_estimator": "SpectralCoclustering", - "flow_ms": 0.000915, - "sklearn_ms": 3.2601659913780168, + "flow_ms": 0.00146374, + "sklearn_ms": 9.960519999935968, "timing_unit": "ms" }, { "flow_estimator": "spectral_embedding", "sklearn_estimator": "SpectralEmbedding", "dataset": "unsupervised", - "flow_ms": 6.156000137, - "sklearn_ms": 1.5892500086920336, - "speedup": 0.25816276369781266, - "flow_repeats": 1, + "flow_ms": 1.764078379, + "sklearn_ms": 4.196989999968537, + "speedup": 2.379140320481495, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1924,9 +2019,9 @@ "flow_estimator": "spline_transformer", "sklearn_estimator": "SplineTransformer", "dataset": "unsupervised", - "flow_ms": 0.024345, - "sklearn_ms": 0.4049989947816357, - "speedup": 16.635818228861602, + "flow_ms": 0.037080905, + "sklearn_ms": 1.0753699999668243, + "speedup": 29.0006406253252, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1934,17 +2029,17 @@ { "flow_estimator": "stacking_classifier", "status": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes the base estimators' predictions as input, so it is the meta-learner alone while StackingClassifier also fits the base estimators and cross-validates them", "sklearn_estimator": "StackingClassifier" }, { "flow_estimator": "stacking_regressor", "sklearn_estimator": "StackingRegressor", "dataset": "regression", - "flow_ms": 2.340000026, - "sklearn_ms": 3.0410410108743235, - "speedup": 1.2995901611474272, - "flow_repeats": 1, + "flow_ms": 3.787240982, + "sklearn_ms": 10.077179000063552, + "speedup": 2.6608232874428563, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1952,9 +2047,9 @@ "flow_estimator": "standard_scaler", "sklearn_estimator": "StandardScaler", "dataset": "unsupervised", - "flow_ms": 0.001025, - "sklearn_ms": 0.13524999667424709, - "speedup": 131.95121626755812, + "flow_ms": 0.0037957, + "sklearn_ms": 0.4746179999983724, + "speedup": 125.04096741006202, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -1963,10 +2058,10 @@ "flow_estimator": "svc", "sklearn_estimator": "SVC", "dataset": "classification", - "flow_ms": 0.20099499799999998, - "sklearn_ms": 0.6359160033753142, - "speedup": 3.1638399447896424, - "flow_repeats": 200, + "flow_ms": 0.902124733, + "sklearn_ms": 1.5462330001128066, + "speedup": 1.7139902538430976, + "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, @@ -1974,10 +2069,10 @@ "flow_estimator": "svr", "sklearn_estimator": "SVR", "dataset": "regression", - "flow_ms": 66.409997702, - "sklearn_ms": 6.55341699894052, - "speedup": 0.09868118093223723, - "flow_repeats": 1, + "flow_ms": 10.823310375, + "sklearn_ms": 13.28541899999891, + "speedup": 1.2274820308845582, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -1985,26 +2080,31 @@ "flow_estimator": "target_encoder", "sklearn_estimator": "TargetEncoder", "dataset": "regression", - "flow_ms": 0.783399999, - "sklearn_ms": 13.760207992163487, - "speedup": 17.564728120664046, + "flow_ms": 1.279571652, + "sklearn_ms": 33.34866500006228, + "speedup": 26.062366220709524, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "tfidf_vectorizer", - "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "TfidfVectorizer" + "sklearn_estimator": "TfidfVectorizer", + "dataset": "classification", + "flow_ms": 0.046975611, + "sklearn_ms": 1.1540179999656175, + "speedup": 24.566322297875328, + "flow_repeats": 200, + "timing_unit": "ms", + "status": "ok" }, { "flow_estimator": "theil_sen_regressor", "sklearn_estimator": "TheilSenRegressor", "dataset": "regression", - "flow_ms": 0.156299993, - "sklearn_ms": 185.37908300640993, - "speedup": 1186.04664944809, + "flow_ms": 0.30490975, + "sklearn_ms": 377.1602750000511, + "speedup": 1236.9570832026563, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -2013,9 +2113,9 @@ "flow_estimator": "transformed_target_regressor", "sklearn_estimator": "TransformedTargetRegressor", "dataset": "regression", - "flow_ms": 0.021449998999999997, - "sklearn_ms": 0.4650419868994504, - "speedup": 21.68028012026716, + "flow_ms": 0.0498836, + "sklearn_ms": 1.6557469999725072, + "speedup": 33.19221146774706, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -2024,9 +2124,9 @@ "flow_estimator": "truncated_svd", "sklearn_estimator": "TruncatedSVD", "dataset": "unsupervised", - "flow_ms": 0.00762, - "sklearn_ms": 0.25675000506453216, - "speedup": 33.69422638642154, + "flow_ms": 0.01300893, + "sklearn_ms": 0.6261920000270038, + "speedup": 48.13554996660016, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -2035,10 +2135,10 @@ "flow_estimator": "tsne", "sklearn_estimator": "TSNE", "dataset": "unsupervised", - "flow_ms": 6.71999979, - "sklearn_ms": 415.7947920029983, - "speedup": 61.874226934015766, - "flow_repeats": 1, + "flow_ms": 17.627658844, + "sklearn_ms": 384.9840850000419, + "speedup": 21.839773982866735, + "flow_repeats": 5, "timing_unit": "ms", "status": "ok" }, @@ -2046,9 +2146,9 @@ "flow_estimator": "tweedie_regressor", "sklearn_estimator": "TweedieRegressor", "dataset": "regression", - "flow_ms": 0.44105001499999996, - "sklearn_ms": 0.7525420078309253, - "speedup": 1.7062509516770459, + "flow_ms": 0.379253347, + "sklearn_ms": 1.5039730000125928, + "speedup": 3.9656156284695694, "flow_repeats": 20, "timing_unit": "ms", "status": "ok" @@ -2057,9 +2157,9 @@ "flow_estimator": "variance_threshold", "sklearn_estimator": "VarianceThreshold", "dataset": "unsupervised", - "flow_ms": 0.0014, - "sklearn_ms": 0.13425000361166894, - "speedup": 95.89285972262067, + "flow_ms": 0.002428945, + "sklearn_ms": 0.5179169999109945, + "speedup": 213.227141788305, "flow_repeats": 200, "timing_unit": "ms", "status": "ok" @@ -2067,13 +2167,13 @@ { "flow_estimator": "voting_classifier", "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes trees that are already fitted, so its fit is the vote alone while VotingClassifier fits every estimator it is given", "sklearn_estimator": "VotingClassifier" }, { "flow_estimator": "voting_regressor", "status": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes regressors that are already fitted, so its fit is the average alone while VotingRegressor fits every estimator it is given", "sklearn_estimator": "VotingRegressor" } ] diff --git a/benchmarks/estimator_coverage.json b/benchmarks/estimator_coverage.json index cc1fbea..17e8d94 100644 --- a/benchmarks/estimator_coverage.json +++ b/benchmarks/estimator_coverage.json @@ -2,10 +2,11 @@ "schema_version": 1, "counts": { "estimators": 203, - "runnable": 168, - "different_shape": 25, + "runnable": 166, + "shaped": 22, "simplified": 4, - "flow_only": 6 + "flow_only": 6, + "different_shape": 5 }, "sklearn_surface": 208, "entries": [ @@ -196,9 +197,20 @@ "n_features", "sample_steps" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "AdditiveChi2Sampler" + "bucket": "shaped", + "sklearn_estimator": "AdditiveChi2Sampler", + "shape": { + "dataset": "classification", + "flow_fit": [ + "n_c", + "f_c", + "2" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X" + } }, { "flow_estimator": "affinity_propagation", @@ -1150,9 +1162,28 @@ "ct", "X" ], - "bucket": "different_shape", - "reason": "takes ColumnTransformer first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "ColumnTransformer" + "bucket": "shaped", + "sklearn_estimator": "ColumnTransformer", + "shape": { + "dataset": "classification", + "flow_preamble": [ + "let ct_cols: ptr = malloc((f_c as i64) * 4) as ptr", + "for i in 0 to f_c { ct_cols[i] = i }", + "let ct_obj: ColumnTransformer = column_transformer_init(1)", + "# 0 is TRANSFORMER_STANDARD_SCALER. The constant is written out", + "# because an export const is not visible through the umbrella import.", + "column_transformer_set_spec(ct_obj, 0, 0, ct_cols, f_c)" + ], + "flow_fit": [ + "ct_obj", + "X_c" + ], + "flow_work": [ + "X_c" + ], + "flow_free": "after", + "sklearn_input": "X" + } }, { "flow_estimator": "complement_nb", @@ -1270,9 +1301,39 @@ "n_docs", "max_features" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "CountVectorizer" + "bucket": "shaped", + "sklearn_estimator": "CountVectorizer", + "shape": { + "dataset": "classification", + "corpus": [ + "the quick brown fox jumps over the lazy dog", + "a lazy dog sleeps in the warm sun", + "quick brown foxes are rare in the city", + "the dog and the fox share a field", + "warm sun and a cold river run together", + "a field of brown grass in the sun", + "the city river runs past the old field", + "old dogs sleep through a quick storm", + "a storm over the city wakes the dog", + "foxes hunt in the cold river valley", + "the valley holds a warm field of grass", + "grass grows where the river meets the sun", + "a rare fox crosses the old stone bridge", + "the stone bridge over the cold river", + "dogs and foxes keep their distance here", + "here the field the river and the city meet" + ], + "flow_fit": [ + "count_vectorizer_docs", + "16", + "50" + ], + "flow_work": [ + "count_vectorizer_docs", + "16" + ], + "sklearn_input": "docs" + } }, { "flow_estimator": "dbscan", @@ -1509,9 +1570,40 @@ "n_samples", "n_keys_per_sample" ], - "bucket": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DictVectorizer" + "bucket": "shaped", + "sklearn_estimator": "DictVectorizer", + "shape": { + "dataset": "classification", + "flow_preamble": [ + "let dv_counts: ptr = malloc((n_c as i64) * 4) as ptr", + "let dv_keys: ptr > = malloc((n_c as i64) * 8) as ptr >", + "let dv_vals: ptr > = malloc((n_c as i64) * 8) as ptr >", + "for i in 0 to n_c {", + " dv_counts[i] = f_c", + " let dv_kk: ptr = malloc((f_c as i64) * 4) as ptr", + " let dv_vv: ptr = array_new_f32(f_c)", + " for j in 0 to f_c {", + " dv_kk[j] = j", + " dv_vv[j] = matrix_at(X_c, i, j)", + " }", + " dv_keys[i] = dv_kk", + " dv_vals[i] = dv_vv", + "}" + ], + "flow_fit": [ + "dv_keys", + "dv_vals", + "n_c", + "dv_counts" + ], + "flow_work": [ + "dv_keys", + "dv_vals", + "n_c", + "dv_counts" + ], + "sklearn_input": "dicts" + } }, { "flow_estimator": "dictionary_learning", @@ -1719,9 +1811,23 @@ "constant_label", "seed" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DummyClassifier" + "bucket": "shaped", + "sklearn_estimator": "DummyClassifier", + "shape": { + "dataset": "classification", + "flow_fit": [ + "y_c", + "n_c", + "3", + "0", + "0.0", + "42" + ], + "flow_work": [ + "n_c" + ], + "sklearn_input": "X" + } }, { "flow_estimator": "dummy_regressor", @@ -1780,9 +1886,21 @@ "strategy", "constant_value" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "DummyRegressor" + "bucket": "shaped", + "sklearn_estimator": "DummyRegressor", + "shape": { + "dataset": "regression", + "flow_fit": [ + "y_r", + "n_r", + "0", + "0.0" + ], + "flow_work": [ + "n_r" + ], + "sklearn_input": "X" + } }, { "flow_estimator": "elastic_net_cv", @@ -2498,9 +2616,27 @@ "fu", "X" ], - "bucket": "different_shape", - "reason": "takes FeatureUnion first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "FeatureUnion" + "bucket": "shaped", + "sklearn_estimator": "FeatureUnion", + "shape": { + "dataset": "classification", + "flow_preamble": [ + "let fu_obj: FeatureUnion = feature_union_init(2)", + "# 0 is FU_TRANSFORMER_STANDARD_SCALER and 2 is FU_TRANSFORMER_PASSTHROUGH,", + "# written out for the reason the column transformer above gives.", + "feature_union_set_transformer(fu_obj, 0, 0, 0)", + "feature_union_set_transformer(fu_obj, 1, 2, 0)" + ], + "flow_fit": [ + "fu_obj", + "X_c" + ], + "flow_work": [ + "X_c" + ], + "flow_free": "after", + "sklearn_input": "X" + } }, { "flow_estimator": "gamma_regressor", @@ -2897,9 +3033,21 @@ "n_components", "seed" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "GaussianRandomProjection" + "bucket": "shaped", + "sklearn_estimator": "GaussianRandomProjection", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "2", + "42" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, random_state=42" + } }, { "flow_estimator": "gradient_boosting_classifier", @@ -3421,7 +3569,7 @@ "X" ], "bucket": "different_shape", - "reason": "takes IncrementalPCA first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "one partial_fit step over one batch, where IncrementalPCA.fit walks the whole design in batches", "sklearn_estimator": "IncrementalPCA" }, { @@ -3604,9 +3752,22 @@ "n", "increasing" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "IsotonicRegression" + "bucket": "shaped", + "sklearn_estimator": "IsotonicRegression", + "shape": { + "dataset": "regression", + "flow_fit": [ + "x1d_r", + "y_r", + "n_r", + "true" + ], + "flow_work": [ + "x1d_r", + "n_r" + ], + "sklearn_input": "x1d" + } }, { "flow_estimator": "iterative_imputer", @@ -3761,8 +3922,23 @@ "bandwidth", "kernel" ], - "bucket": "runnable", - "sklearn_estimator": "KernelDensity" + "bucket": "shaped", + "sklearn_estimator": "KernelDensity", + "shape": { + "dataset": "classification", + "flow_fit": [ + "X_c", + "0.5", + "0" + ], + "flow_work": [ + "X_c" + ], + "flow_work_fn": "kernel_density_score_samples", + "flow_work_returns": "ptr", + "sklearn_input": "X", + "sklearn_work": "score_samples" + } }, { "flow_estimator": "kernel_pca", @@ -4456,9 +4632,22 @@ "neg_label", "pos_label" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "LabelBinarizer" + "bucket": "shaped", + "sklearn_estimator": "LabelBinarizer", + "shape": { + "dataset": "classification", + "flow_fit": [ + "y_c", + "n_c", + "0.0", + "1.0" + ], + "flow_work": [ + "y_c", + "n_c" + ], + "sklearn_input": "y" + } }, { "flow_estimator": "label_encoder", @@ -4511,9 +4700,20 @@ "y", "n" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "LabelEncoder" + "bucket": "shaped", + "sklearn_estimator": "LabelEncoder", + "shape": { + "dataset": "classification", + "flow_fit": [ + "y_c", + "n_c" + ], + "flow_work": [ + "y_c", + "n_c" + ], + "sklearn_input": "y" + } }, { "flow_estimator": "label_propagation", @@ -6792,9 +6992,27 @@ "n_labels_per_sample", "n_classes" ], - "bucket": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "MultiLabelBinarizer" + "bucket": "shaped", + "sklearn_estimator": "MultiLabelBinarizer", + "shape": { + "dataset": "multioutput_class", + "flow_preamble": [ + "let mlb_counts: ptr = malloc((n_c as i64) * 4) as ptr", + "for i in 0 to n_c { mlb_counts[i] = 2 }" + ], + "flow_fit": [ + "Y_label_rows", + "n_c", + "mlb_counts", + "3" + ], + "flow_work": [ + "Y_label_rows", + "n_c", + "mlb_counts" + ], + "sklearn_input": "labelsets" + } }, { "flow_estimator": "multinomial_nb", @@ -7269,8 +7487,23 @@ "X", "n_neighbors" ], - "bucket": "runnable", - "sklearn_estimator": "NearestNeighbors" + "bucket": "shaped", + "sklearn_estimator": "NearestNeighbors", + "shape": { + "dataset": "classification", + "flow_fit": [ + "X_c", + "5" + ], + "flow_work": [ + "X_c" + ], + "flow_work_fn": "nearest_neighbors_kneighbors", + "flow_work_returns": "ptr", + "flow_work_release": "nearest_neighbors_free_results({var}, n_c)", + "sklearn_input": "X", + "sklearn_work": "kneighbors" + } }, { "flow_estimator": "nmf", @@ -8461,9 +8694,28 @@ "X", "y" ], - "bucket": "different_shape", - "reason": "takes Pipeline first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "Pipeline" + "bucket": "shaped", + "sklearn_estimator": "Pipeline", + "shape": { + "dataset": "classification", + "flow_preamble": [ + "let pipe_steps: array = [", + " step_standard_scaler(\"scaler\"),", + " step_logistic_regression(\"classifier\", 3, 50, 0.5, penalty_none())", + "]", + "let pipe_obj: Pipeline = pipeline_new(pipe_steps, 2)" + ], + "flow_fit": [ + "pipe_obj", + "X_c", + "y_c" + ], + "flow_work": [ + "X_c" + ], + "flow_free": "after", + "sklearn_input": "X" + } }, { "flow_estimator": "pls_canonical", @@ -8771,9 +9023,21 @@ "n_components", "degree" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "PolynomialCountSketch" + "bucket": "shaped", + "sklearn_estimator": "PolynomialCountSketch", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "2", + "2" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, degree=2, random_state=42" + } }, { "flow_estimator": "polynomial_features", @@ -8832,9 +9096,21 @@ "interaction_only", "include_bias" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "PolynomialFeatures" + "bucket": "shaped", + "sklearn_estimator": "PolynomialFeatures", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "2", + "false", + "true" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X" + } }, { "flow_estimator": "power_transformer", @@ -9577,9 +9853,22 @@ "n_components", "seed" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "RBFSampler" + "bucket": "shaped", + "sklearn_estimator": "RBFSampler", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "0.1", + "2", + "42" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X", + "sklearn_ctor": "gamma=0.1, n_components=2, random_state=42" + } }, { "flow_estimator": "regressor_chain", @@ -10238,9 +10527,20 @@ "n_features", "threshold" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SelectFromModel" + "bucket": "shaped", + "sklearn_estimator": "SelectFromModel", + "shape": { + "dataset": "classification", + "flow_fit": [ + "w_f", + "f_c", + "0.5" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X" + } }, { "flow_estimator": "select_fwe", @@ -10476,7 +10776,7 @@ "max_iter" ], "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes class probabilities as input, so it is the labelling loop alone while SelfTrainingClassifier also fits the base estimator on every round", "sklearn_estimator": "SelfTrainingClassifier" }, { @@ -10921,9 +11221,22 @@ "n_components", "seed" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SkewedChi2Sampler" + "bucket": "shaped", + "sklearn_estimator": "SkewedChi2Sampler", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "1.0", + "2", + "42" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X", + "sklearn_ctor": "skewedness=1.0, n_components=2, random_state=42" + } }, { "flow_estimator": "sparse_coder", @@ -11102,9 +11415,22 @@ "density", "seed" ], - "bucket": "different_shape", - "reason": "takes i32 first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "SparseRandomProjection" + "bucket": "shaped", + "sklearn_estimator": "SparseRandomProjection", + "shape": { + "dataset": "classification", + "flow_fit": [ + "f_c", + "2", + "0.3", + "42" + ], + "flow_work": [ + "X_c" + ], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, density=0.3, random_state=42" + } }, { "flow_estimator": "spectral_biclustering", @@ -11404,7 +11730,7 @@ "n_iter" ], "bucket": "different_shape", - "reason": "takes ptr > first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes the base estimators' predictions as input, so it is the meta-learner alone while StackingClassifier also fits the base estimators and cross-validates them", "sklearn_estimator": "StackingClassifier" }, { @@ -11807,9 +12133,39 @@ "n_docs", "max_features" ], - "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", - "sklearn_estimator": "TfidfVectorizer" + "bucket": "shaped", + "sklearn_estimator": "TfidfVectorizer", + "shape": { + "dataset": "classification", + "corpus": [ + "the quick brown fox jumps over the lazy dog", + "a lazy dog sleeps in the warm sun", + "quick brown foxes are rare in the city", + "the dog and the fox share a field", + "warm sun and a cold river run together", + "a field of brown grass in the sun", + "the city river runs past the old field", + "old dogs sleep through a quick storm", + "a storm over the city wakes the dog", + "foxes hunt in the cold river valley", + "the valley holds a warm field of grass", + "grass grows where the river meets the sun", + "a rare fox crosses the old stone bridge", + "the stone bridge over the cold river", + "dogs and foxes keep their distance here", + "here the field the river and the city meet" + ], + "flow_fit": [ + "tfidf_vectorizer_docs", + "16", + "50" + ], + "flow_work": [ + "tfidf_vectorizer_docs", + "16" + ], + "sklearn_input": "docs" + } }, { "flow_estimator": "theil_sen_regressor", @@ -12220,7 +12576,7 @@ "voting" ], "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes trees that are already fitted, so its fit is the vote alone while VotingClassifier fits every estimator it is given", "sklearn_estimator": "VotingClassifier" }, { @@ -12276,7 +12632,7 @@ "weights" ], "bucket": "different_shape", - "reason": "takes ptr first, so it is not an estimator over a feature matrix and needs its own harness", + "reason": "takes regressors that are already fitted, so its fit is the average alone while VotingRegressor fits every estimator it is given", "sklearn_estimator": "VotingRegressor" } ] diff --git a/benchmarks/estimator_coverage.py b/benchmarks/estimator_coverage.py index ef436c5..bc3499f 100644 --- a/benchmarks/estimator_coverage.py +++ b/benchmarks/estimator_coverage.py @@ -7,9 +7,11 @@ from: every exported `*_fit`, its companion predict/transform, the arguments to call it with, and the scikit-learn class to race it against. -An estimator lands in one of three buckets, and every one carries a reason: +An estimator lands in one of these buckets, and every one carries a reason: runnable arguments resolved and a scikit-learn counterpart exists + shaped raced through a written out call, because its fit does not begin + with a feature matrix flow_only Flow implements it and scikit-learn has no equivalent blocked something about the signature is not resolved yet @@ -175,6 +177,255 @@ "n_categories": ("array", "[4, 4, 4, 4]"), } +# Flow functions whose scikit-learn namesake does more than they do. Timing +# them against that class would compare a part against the whole, which is the +# same objection the simplified rows carry. The wording says which part. +DIFFERENT_JOB: dict[str, str] = { + "stacking_classifier": "takes the base estimators' predictions as input, so it is the " + "meta-learner alone while StackingClassifier also fits the base " + "estimators and cross-validates them", + "self_training_classifier": "takes class probabilities as input, so it is the labelling " + "loop alone while SelfTrainingClassifier also fits the base " + "estimator on every round", + "voting_classifier": "takes trees that are already fitted, so its fit is the vote alone " + "while VotingClassifier fits every estimator it is given", + "voting_regressor": "takes regressors that are already fitted, so its fit is the average " + "alone while VotingRegressor fits every estimator it is given", + "incremental_pca_partial": "one partial_fit step over one batch, where IncrementalPCA.fit " + "walks the whole design in batches", +} + +# One corpus for the two text vectorizers, read by the Flow generator and by +# the scikit-learn harness, so the two sides cannot drift apart. +CORPUS: list[str] = [ + "the quick brown fox jumps over the lazy dog", + "a lazy dog sleeps in the warm sun", + "quick brown foxes are rare in the city", + "the dog and the fox share a field", + "warm sun and a cold river run together", + "a field of brown grass in the sun", + "the city river runs past the old field", + "old dogs sleep through a quick storm", + "a storm over the city wakes the dog", + "foxes hunt in the cold river valley", + "the valley holds a warm field of grass", + "grass grows where the river meets the sun", + "a rare fox crosses the old stone bridge", + "the stone bridge over the cold river", + "dogs and foxes keep their distance here", + "here the field the river and the city meet", +] + +# Estimators whose fit does not begin with a feature matrix, and which race +# scikit-learn perfectly well once the call is written out. Thirteen rows sat +# in different_shape only because the generic path builds one call shape. +# +# `flow_fit` and `flow_work` are argument expressions in terms of the variables +# the generated harness declares (X_c, y_c, n_c, f_c and the regression pair). +# `sklearn_input` says what the scikit-learn side fits and transforms: the +# feature matrix, the target vector, or the first column of the matrix as a +# one-dimensional x. `sklearn_ctor` overrides the constructor where the default +# would measure a different size of problem, as it would for a random +# projection whose n_components is chosen by Johnson-Lindenstrauss. +SHAPED: dict[str, dict] = { + "additive_chi2_sampler": { + "dataset": "classification", + "flow_fit": ["n_c", "f_c", "2"], + "flow_work": ["X_c"], + "sklearn_input": "X", + }, + "gaussian_random_projection": { + "dataset": "classification", + "flow_fit": ["f_c", "2", "42"], + "flow_work": ["X_c"], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, random_state=42", + }, + "polynomial_count_sketch": { + "dataset": "classification", + "flow_fit": ["f_c", "2", "2"], + "flow_work": ["X_c"], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, degree=2, random_state=42", + }, + "polynomial_features": { + "dataset": "classification", + "flow_fit": ["f_c", "2", "false", "true"], + "flow_work": ["X_c"], + "sklearn_input": "X", + }, + "rbf_sampler": { + "dataset": "classification", + "flow_fit": ["f_c", "0.1", "2", "42"], + "flow_work": ["X_c"], + "sklearn_input": "X", + "sklearn_ctor": "gamma=0.1, n_components=2, random_state=42", + }, + "skewed_chi2_sampler": { + "dataset": "classification", + "flow_fit": ["f_c", "1.0", "2", "42"], + "flow_work": ["X_c"], + "sklearn_input": "X", + "sklearn_ctor": "skewedness=1.0, n_components=2, random_state=42", + }, + "sparse_random_projection": { + "dataset": "classification", + "flow_fit": ["f_c", "2", "0.3", "42"], + "flow_work": ["X_c"], + "sklearn_input": "X", + "sklearn_ctor": "n_components=2, density=0.3, random_state=42", + }, + "select_from_model": { + "dataset": "classification", + "flow_fit": ["w_f", "f_c", "0.5"], + "flow_work": ["X_c"], + "sklearn_input": "X", + }, + "dummy_classifier": { + "dataset": "classification", + "flow_fit": ["y_c", "n_c", "3", "0", "0.0", "42"], + "flow_work": ["n_c"], + "sklearn_input": "X", + }, + "dummy_regressor": { + "dataset": "regression", + "flow_fit": ["y_r", "n_r", "0", "0.0"], + "flow_work": ["n_r"], + "sklearn_input": "X", + }, + "label_encoder": { + "dataset": "classification", + "flow_fit": ["y_c", "n_c"], + "flow_work": ["y_c", "n_c"], + "sklearn_input": "y", + }, + "label_binarizer": { + "dataset": "classification", + "flow_fit": ["y_c", "n_c", "0.0", "1.0"], + "flow_work": ["y_c", "n_c"], + "sklearn_input": "y", + }, + "isotonic": { + "dataset": "regression", + "flow_fit": ["x1d_r", "y_r", "n_r", "true"], + "flow_work": ["x1d_r", "n_r"], + "sklearn_input": "x1d", + }, + # Two rows whose work function is named for what it returns rather than + # predict or transform, so the generic path found nothing to time and the + # fit alone fell under the clock's floor. + "kernel_density": { + "dataset": "classification", + "flow_fit": ["X_c", "0.5", "0"], + "flow_work": ["X_c"], + "flow_work_fn": "kernel_density_score_samples", + "flow_work_returns": "ptr", + "sklearn_input": "X", + "sklearn_work": "score_samples", + }, + "nearest_neighbors": { + "dataset": "classification", + "flow_fit": ["X_c", "5"], + "flow_work": ["X_c"], + "flow_work_fn": "nearest_neighbors_kneighbors", + "flow_work_returns": "ptr", + "flow_work_release": "nearest_neighbors_free_results({var}, n_c)", + "sklearn_input": "X", + "sklearn_work": "kneighbors", + }, + "multilabel_binarizer": { + # The label rows, so the scikit-learn side gets sets of labels rather + # than one label per sample. + "dataset": "multioutput_class", + "flow_preamble": [ + "let mlb_counts: ptr = malloc((n_c as i64) * 4) as ptr", + "for i in 0 to n_c { mlb_counts[i] = 2 }", + ], + "flow_fit": ["Y_label_rows", "n_c", "mlb_counts", "3"], + "flow_work": ["Y_label_rows", "n_c", "mlb_counts"], + "sklearn_input": "labelsets", + }, + "dict_vectorizer": { + "dataset": "classification", + "flow_preamble": [ + "let dv_counts: ptr = malloc((n_c as i64) * 4) as ptr", + "let dv_keys: ptr > = malloc((n_c as i64) * 8) as ptr >", + "let dv_vals: ptr > = malloc((n_c as i64) * 8) as ptr >", + "for i in 0 to n_c {", + " dv_counts[i] = f_c", + " let dv_kk: ptr = malloc((f_c as i64) * 4) as ptr", + " let dv_vv: ptr = array_new_f32(f_c)", + " for j in 0 to f_c {", + " dv_kk[j] = j", + " dv_vv[j] = matrix_at(X_c, i, j)", + " }", + " dv_keys[i] = dv_kk", + " dv_vals[i] = dv_vv", + "}", + ], + "flow_fit": ["dv_keys", "dv_vals", "n_c", "dv_counts"], + "flow_work": ["dv_keys", "dv_vals", "n_c", "dv_counts"], + "sklearn_input": "dicts", + }, + "count_vectorizer": { + "dataset": "classification", + "corpus": CORPUS, + "flow_fit": ["count_vectorizer_docs", "16", "50"], + "flow_work": ["count_vectorizer_docs", "16"], + "sklearn_input": "docs", + }, + "tfidf_vectorizer": { + "dataset": "classification", + "corpus": CORPUS, + "flow_fit": ["tfidf_vectorizer_docs", "16", "50"], + "flow_work": ["tfidf_vectorizer_docs", "16"], + "sklearn_input": "docs", + }, + "pipeline": { + "dataset": "classification", + "flow_preamble": [ + "let pipe_steps: array = [", + ' step_standard_scaler("scaler"),', + ' step_logistic_regression("classifier", 3, 50, 0.5, penalty_none())', + "]", + "let pipe_obj: Pipeline = pipeline_new(pipe_steps, 2)", + ], + "flow_fit": ["pipe_obj", "X_c", "y_c"], + "flow_work": ["X_c"], + "flow_free": "after", + "sklearn_input": "X", + }, + "column_transformer": { + "dataset": "classification", + "flow_preamble": [ + "let ct_cols: ptr = malloc((f_c as i64) * 4) as ptr", + "for i in 0 to f_c { ct_cols[i] = i }", + "let ct_obj: ColumnTransformer = column_transformer_init(1)", + "# 0 is TRANSFORMER_STANDARD_SCALER. The constant is written out", + "# because an export const is not visible through the umbrella import.", + "column_transformer_set_spec(ct_obj, 0, 0, ct_cols, f_c)", + ], + "flow_fit": ["ct_obj", "X_c"], + "flow_work": ["X_c"], + "flow_free": "after", + "sklearn_input": "X", + }, + "feature_union": { + "dataset": "classification", + "flow_preamble": [ + "let fu_obj: FeatureUnion = feature_union_init(2)", + "# 0 is FU_TRANSFORMER_STANDARD_SCALER and 2 is FU_TRANSFORMER_PASSTHROUGH,", + "# written out for the reason the column transformer above gives.", + "feature_union_set_transformer(fu_obj, 0, 0, 0)", + "feature_union_set_transformer(fu_obj, 1, 2, 0)", + ], + "flow_fit": ["fu_obj", "X_c"], + "flow_work": ["X_c"], + "flow_free": "after", + "sklearn_input": "X", + }, +} + # Parameters whose value depends on the dataset rather than on a constant. DATASET_ARGS = {"n_classes", "n_samples", "n_features"} @@ -302,6 +553,12 @@ def classify(base: str, spec: dict, known: set[str], exports: dict[str, dict]) - unresolved.append(f"{p['name']}: {p['type']}") head = spec["params"][0]["type"] if spec["params"] else "absent" sk = sklearn_name(base, known) + if base in SHAPED and sk is not None: + entry.update(bucket="shaped", sklearn_estimator=sk, shape=SHAPED[base]) + return entry + if base in DIFFERENT_JOB: + entry.update(bucket="different_shape", reason=DIFFERENT_JOB[base], sklearn_estimator=sk) + return entry if head != "Matrix": entry.update( bucket="different_shape", @@ -326,7 +583,8 @@ def argument_tables() -> dict: def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--out", type=Path, default=OUT) - ap.add_argument("--show", choices=["blocked", "flow_only", "runnable", "simplified", "different_shape"], help="list one bucket and exit") + ap.add_argument("--show", choices=["blocked", "flow_only", "runnable", "shaped", "simplified", + "different_shape"], help="list one bucket and exit") args = ap.parse_args() exports = parse_exports() @@ -359,7 +617,7 @@ def main() -> int: return 0 print(f"{len(entries)} exported Flow estimators against a {len(known)}-estimator scikit-learn surface") - for bucket in ("runnable", "simplified", "different_shape", "flow_only", "blocked"): + for bucket in ("runnable", "shaped", "simplified", "different_shape", "flow_only", "blocked"): print(f" {bucket:10s} {counts.get(bucket, 0)}") return 0 diff --git a/benchmarks/estimators_sklearn.json b/benchmarks/estimators_sklearn.json index b5e26d2..16a028c 100644 --- a/benchmarks/estimators_sklearn.json +++ b/benchmarks/estimators_sklearn.json @@ -1,17 +1,17 @@ { "schema_version": 1, - "repeats": 3, + "repeats": 5, "counts": { - "rows": 172, - "ok": 172 + "rows": 192, + "ok": 192 }, "rows": [ { "flow_estimator": "adaboost_classifier", "sklearn_estimator": "AdaBoostClassifier", "dataset": "classification", - "fit_ms": 21.526500000618398, - "pred_ms": 1.768750007613562, + "fit_ms": 73.50816199993915, + "pred_ms": 7.334176999961528, "timing_unit": "ms", "status": "ok" }, @@ -19,8 +19,17 @@ "flow_estimator": "adaboost_regressor", "sklearn_estimator": "AdaBoostRegressor", "dataset": "regression", - "fit_ms": 12.87933399726171, - "pred_ms": 2.329875002033077, + "fit_ms": 31.82876900007159, + "pred_ms": 7.192402999976366, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "additive_chi2_sampler", + "sklearn_estimator": "AdditiveChi2Sampler", + "dataset": "classification", + "fit_ms": 0.14573199996448238, + "pred_ms": 0.17691100003958127, "timing_unit": "ms", "status": "ok" }, @@ -28,8 +37,8 @@ "flow_estimator": "affinity_propagation", "sklearn_estimator": "AffinityPropagation", "dataset": "unsupervised", - "fit_ms": 3.3742090017767623, - "pred_ms": 0.702042001648806, + "fit_ms": 6.607107000036194, + "pred_ms": 0.563884000030157, "timing_unit": "ms", "status": "ok" }, @@ -37,7 +46,7 @@ "flow_estimator": "agglomerative_clustering", "sklearn_estimator": "AgglomerativeClustering", "dataset": "unsupervised", - "fit_ms": 0.3488329966785386, + "fit_ms": 0.7726449999836404, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -46,8 +55,8 @@ "flow_estimator": "ard_regression", "sklearn_estimator": "ARDRegression", "dataset": "regression", - "fit_ms": 3.2732499967096373, - "pred_ms": 0.033958989661186934, + "fit_ms": 9.976789000006647, + "pred_ms": 0.11346200005846185, "timing_unit": "ms", "status": "ok" }, @@ -55,8 +64,8 @@ "flow_estimator": "bagging_classifier", "sklearn_estimator": "BaggingClassifier", "dataset": "classification", - "fit_ms": 6.214499997440726, - "pred_ms": 0.4326670023147017, + "fit_ms": 16.714701999944737, + "pred_ms": 1.7283920000181752, "timing_unit": "ms", "status": "ok" }, @@ -64,8 +73,8 @@ "flow_estimator": "bagging_regressor", "sklearn_estimator": "BaggingRegressor", "dataset": "regression", - "fit_ms": 11.730874990462326, - "pred_ms": 0.7345839985646307, + "fit_ms": 25.68183300002147, + "pred_ms": 2.0972529999880862, "timing_unit": "ms", "status": "ok" }, @@ -73,8 +82,8 @@ "flow_estimator": "bayesian_gaussian_mixture", "sklearn_estimator": "BayesianGaussianMixture", "dataset": "unsupervised", - "fit_ms": 1.0715829994296655, - "pred_ms": 0.07137500506360084, + "fit_ms": 3.9543660000163072, + "pred_ms": 0.29373000006671646, "timing_unit": "ms", "status": "ok" }, @@ -82,8 +91,8 @@ "flow_estimator": "bayesian_ridge", "sklearn_estimator": "BayesianRidge", "dataset": "regression", - "fit_ms": 0.4151250032009557, - "pred_ms": 0.03720899985637516, + "fit_ms": 1.5414140000302723, + "pred_ms": 0.1408440000432165, "timing_unit": "ms", "status": "ok" }, @@ -91,8 +100,8 @@ "flow_estimator": "bernoulli_nb", "sklearn_estimator": "BernoulliNB", "dataset": "classification", - "fit_ms": 0.4657079989556223, - "pred_ms": 0.09958300506696105, + "fit_ms": 1.9599260000404684, + "pred_ms": 0.40303399998720124, "timing_unit": "ms", "status": "ok" }, @@ -100,8 +109,8 @@ "flow_estimator": "bernoulli_rbm", "sklearn_estimator": "BernoulliRBM", "dataset": "unsupervised", - "fit_ms": 8.622250010375865, - "pred_ms": 0.15341700054705143, + "fit_ms": 20.150205999925674, + "pred_ms": 0.9553169999207967, "timing_unit": "ms", "status": "ok" }, @@ -109,8 +118,8 @@ "flow_estimator": "birch", "sklearn_estimator": "Birch", "dataset": "unsupervised", - "fit_ms": 0.840458000311628, - "pred_ms": 0.25662498956080526, + "fit_ms": 3.5892040000362613, + "pred_ms": 0.6402180000577573, "timing_unit": "ms", "status": "ok" }, @@ -118,8 +127,8 @@ "flow_estimator": "bisecting_kmeans", "sklearn_estimator": "BisectingKMeans", "dataset": "unsupervised", - "fit_ms": 4.308542003855109, - "pred_ms": 0.15733300824649632, + "fit_ms": 3.845611999963694, + "pred_ms": 1.0160009999253816, "timing_unit": "ms", "status": "ok" }, @@ -127,8 +136,8 @@ "flow_estimator": "calibrated_classifier_cv", "sklearn_estimator": "CalibratedClassifierCV", "dataset": "classification", - "fit_ms": 9.41249998868443, - "pred_ms": 1.0962079977616668, + "fit_ms": 39.132870000003095, + "pred_ms": 5.318035999948734, "timing_unit": "ms", "status": "ok" }, @@ -136,8 +145,8 @@ "flow_estimator": "categorical_nb", "sklearn_estimator": "CategoricalNB", "dataset": "classification", - "fit_ms": 0.6367500027408823, - "pred_ms": 0.04699999408330768, + "fit_ms": 2.317103999985193, + "pred_ms": 0.22234499999740365, "timing_unit": "ms", "status": "ok" }, @@ -145,8 +154,8 @@ "flow_estimator": "cca", "sklearn_estimator": "CCA", "dataset": "multioutput", - "fit_ms": 0.4142079997109249, - "pred_ms": 0.04770899249706417, + "fit_ms": 1.2940799999796582, + "pred_ms": 0.17863400000805996, "timing_unit": "ms", "status": "ok" }, @@ -154,8 +163,17 @@ "flow_estimator": "classifier_chain", "sklearn_estimator": "ClassifierChain", "dataset": "multioutput_class", - "fit_ms": 0.5865419952897355, - "pred_ms": 0.3751250042114407, + "fit_ms": 2.596256000060748, + "pred_ms": 1.6134480000573603, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "column_transformer", + "sklearn_estimator": "ColumnTransformer", + "dataset": "classification", + "fit_ms": 1.6301780000276267, + "pred_ms": 0.6465790000902416, "timing_unit": "ms", "status": "ok" }, @@ -163,8 +181,17 @@ "flow_estimator": "complement_nb", "sklearn_estimator": "ComplementNB", "dataset": "classification", - "fit_ms": 0.3906250058207661, - "pred_ms": 0.03195799945387989, + "fit_ms": 1.8059770000036224, + "pred_ms": 0.14638300001479365, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "count_vectorizer", + "sklearn_estimator": "CountVectorizer", + "dataset": "classification", + "fit_ms": 0.6051719999504712, + "pred_ms": 0.20381100000577135, "timing_unit": "ms", "status": "ok" }, @@ -172,7 +199,7 @@ "flow_estimator": "dbscan", "sklearn_estimator": "DBSCAN", "dataset": "unsupervised", - "fit_ms": 0.4175000067334622, + "fit_ms": 1.688398000055713, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -181,8 +208,8 @@ "flow_estimator": "decision_tree_classifier", "sklearn_estimator": "DecisionTreeClassifier", "dataset": "classification", - "fit_ms": 0.26191600773017853, - "pred_ms": 0.03999999898951501, + "fit_ms": 0.78642099992976, + "pred_ms": 0.13198700003158592, "timing_unit": "ms", "status": "ok" }, @@ -190,8 +217,17 @@ "flow_estimator": "decision_tree_regressor", "sklearn_estimator": "DecisionTreeRegressor", "dataset": "regression", - "fit_ms": 1.1715830041794106, - "pred_ms": 0.04270899808034301, + "fit_ms": 2.3735690000421528, + "pred_ms": 0.13839899997947214, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "dict_vectorizer", + "sklearn_estimator": "DictVectorizer", + "dataset": "classification", + "fit_ms": 0.2989490000118167, + "pred_ms": 0.49768099995617376, "timing_unit": "ms", "status": "ok" }, @@ -199,8 +235,8 @@ "flow_estimator": "dictionary_learning", "sklearn_estimator": "DictionaryLearning", "dataset": "unsupervised", - "fit_ms": 198.1439999944996, - "pred_ms": 1.038749993313104, + "fit_ms": 848.2305850000103, + "pred_ms": 3.569756000047164, "timing_unit": "ms", "status": "ok" }, @@ -208,8 +244,26 @@ "flow_estimator": "discriminant_lda", "sklearn_estimator": "LinearDiscriminantAnalysis", "dataset": "classification", - "fit_ms": 0.2998340060003102, - "pred_ms": 0.047708002966828644, + "fit_ms": 0.964975000101731, + "pred_ms": 0.17001800006255507, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "dummy_classifier", + "sklearn_estimator": "DummyClassifier", + "dataset": "classification", + "fit_ms": 0.15171400002600421, + "pred_ms": 0.02095000002100278, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "dummy_regressor", + "sklearn_estimator": "DummyRegressor", + "dataset": "regression", + "fit_ms": 0.18450499999289605, + "pred_ms": 0.011300999972263526, "timing_unit": "ms", "status": "ok" }, @@ -217,8 +271,8 @@ "flow_estimator": "elastic_net_cv", "sklearn_estimator": "ElasticNetCV", "dataset": "regression", - "fit_ms": 13.445750009850599, - "pred_ms": 0.036375000490807, + "fit_ms": 58.466463999934604, + "pred_ms": 0.1145439999845621, "timing_unit": "ms", "status": "ok" }, @@ -226,8 +280,8 @@ "flow_estimator": "elastic_net", "sklearn_estimator": "ElasticNet", "dataset": "regression", - "fit_ms": 0.17662500613369048, - "pred_ms": 0.03762501000892371, + "fit_ms": 0.5858550000539253, + "pred_ms": 0.11539599995558092, "timing_unit": "ms", "status": "ok" }, @@ -235,8 +289,8 @@ "flow_estimator": "elliptic_envelope", "sklearn_estimator": "EllipticEnvelope", "dataset": "unsupervised", - "fit_ms": 8.801250005490147, - "pred_ms": 0.14320800255518407, + "fit_ms": 32.70151399999577, + "pred_ms": 0.399708000031751, "timing_unit": "ms", "status": "ok" }, @@ -244,7 +298,7 @@ "flow_estimator": "empirical_covariance", "sklearn_estimator": "EmpiricalCovariance", "dataset": "unsupervised", - "fit_ms": 0.12658300693146884, + "fit_ms": 0.46602099996562174, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -253,8 +307,8 @@ "flow_estimator": "extra_tree_classifier", "sklearn_estimator": "ExtraTreeClassifier", "dataset": "classification", - "fit_ms": 0.2129159984178841, - "pred_ms": 0.03758301318157464, + "fit_ms": 0.7350349999342143, + "pred_ms": 0.13642500005062175, "timing_unit": "ms", "status": "ok" }, @@ -262,8 +316,8 @@ "flow_estimator": "extra_tree_regressor", "sklearn_estimator": "ExtraTreeRegressor", "dataset": "regression", - "fit_ms": 0.6256659980863333, - "pred_ms": 0.052041999879293144, + "fit_ms": 1.1780830000134301, + "pred_ms": 0.1403919999347636, "timing_unit": "ms", "status": "ok" }, @@ -271,8 +325,8 @@ "flow_estimator": "extra_trees_classifier", "sklearn_estimator": "ExtraTreesClassifier", "dataset": "classification", - "fit_ms": 27.618958003586158, - "pred_ms": 2.439207994029857, + "fit_ms": 86.07109499996568, + "pred_ms": 7.780360999959157, "timing_unit": "ms", "status": "ok" }, @@ -280,8 +334,8 @@ "flow_estimator": "extra_trees_regressor", "sklearn_estimator": "ExtraTreesRegressor", "dataset": "regression", - "fit_ms": 63.45508300000802, - "pred_ms": 5.237417004536837, + "fit_ms": 131.21491099991545, + "pred_ms": 12.568948999955865, "timing_unit": "ms", "status": "ok" }, @@ -289,8 +343,8 @@ "flow_estimator": "factor_analysis", "sklearn_estimator": "FactorAnalysis", "dataset": "unsupervised", - "fit_ms": 0.43433399696368724, - "pred_ms": 0.0466250057797879, + "fit_ms": 0.9965640000473286, + "pred_ms": 0.159238000037476, "timing_unit": "ms", "status": "ok" }, @@ -298,8 +352,8 @@ "flow_estimator": "fast_ica", "sklearn_estimator": "FastICA", "dataset": "unsupervised", - "fit_ms": 0.6530410028062761, - "pred_ms": 0.032791998819448054, + "fit_ms": 2.562792999924568, + "pred_ms": 0.12411200009410095, "timing_unit": "ms", "status": "ok" }, @@ -307,8 +361,17 @@ "flow_estimator": "feature_agglomeration", "sklearn_estimator": "FeatureAgglomeration", "dataset": "unsupervised", - "fit_ms": 0.1066659897333011, - "pred_ms": 0.13712499639950693, + "fit_ms": 0.3697120000651921, + "pred_ms": 0.46517999999196036, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "feature_union", + "sklearn_estimator": "FeatureUnion", + "dataset": "classification", + "fit_ms": 0.8147940000071685, + "pred_ms": 0.3949379999994562, "timing_unit": "ms", "status": "ok" }, @@ -316,8 +379,8 @@ "flow_estimator": "gamma_regressor", "sklearn_estimator": "GammaRegressor", "dataset": "regression", - "fit_ms": 0.6624579982599244, - "pred_ms": 0.04954100586473942, + "fit_ms": 1.5014080000810281, + "pred_ms": 0.20219900000029156, "timing_unit": "ms", "status": "ok" }, @@ -325,8 +388,8 @@ "flow_estimator": "gaussian_mixture", "sklearn_estimator": "GaussianMixture", "dataset": "unsupervised", - "fit_ms": 0.8224999910453334, - "pred_ms": 0.05604101170320064, + "fit_ms": 2.7774850000241713, + "pred_ms": 0.24586900008216617, "timing_unit": "ms", "status": "ok" }, @@ -334,8 +397,8 @@ "flow_estimator": "gaussian_nb", "sklearn_estimator": "GaussianNB", "dataset": "classification", - "fit_ms": 0.2733330038608983, - "pred_ms": 0.06195799505803734, + "fit_ms": 1.229749999993146, + "pred_ms": 0.27900099996713834, "timing_unit": "ms", "status": "ok" }, @@ -343,8 +406,8 @@ "flow_estimator": "gaussian_process_classifier", "sklearn_estimator": "GaussianProcessClassifier", "dataset": "regression", - "fit_ms": 2415.014332989813, - "pred_ms": 390.27524999983143, + "fit_ms": 9979.201848000002, + "pred_ms": 1384.3943719999743, "timing_unit": "ms", "status": "ok" }, @@ -352,8 +415,17 @@ "flow_estimator": "gaussian_process_regressor", "sklearn_estimator": "GaussianProcessRegressor", "dataset": "regression", - "fit_ms": 71.80841700755991, - "pred_ms": 0.6932499964023009, + "fit_ms": 401.0938689999648, + "pred_ms": 5.492582999977458, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "gaussian_random_projection", + "sklearn_estimator": "GaussianRandomProjection", + "dataset": "classification", + "fit_ms": 0.4567239999460071, + "pred_ms": 0.14019200000348064, "timing_unit": "ms", "status": "ok" }, @@ -361,8 +433,8 @@ "flow_estimator": "gradient_boosting_classifier", "sklearn_estimator": "GradientBoostingClassifier", "dataset": "classification", - "fit_ms": 61.85595800343435, - "pred_ms": 0.3888749924954027, + "fit_ms": 204.07313999999133, + "pred_ms": 0.9756450000395489, "timing_unit": "ms", "status": "ok" }, @@ -370,8 +442,8 @@ "flow_estimator": "gradient_boosting_regressor", "sklearn_estimator": "GradientBoostingRegressor", "dataset": "regression", - "fit_ms": 46.95212499063928, - "pred_ms": 0.42537499393802136, + "fit_ms": 104.24416900002598, + "pred_ms": 0.8764789999986533, "timing_unit": "ms", "status": "ok" }, @@ -379,7 +451,7 @@ "flow_estimator": "graphical_lasso", "sklearn_estimator": "GraphicalLasso", "dataset": "unsupervised", - "fit_ms": 0.3913340042345226, + "fit_ms": 1.4963180000222565, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -388,7 +460,7 @@ "flow_estimator": "hdbscan", "sklearn_estimator": "HDBSCAN", "dataset": "unsupervised", - "fit_ms": 0.7184580026660115, + "fit_ms": 2.133130000061101, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -397,8 +469,8 @@ "flow_estimator": "hist_gradient_boosting_classifier", "sklearn_estimator": "HistGradientBoostingClassifier", "dataset": "classification", - "fit_ms": 691.7245420045219, - "pred_ms": 31.16549999685958, + "fit_ms": 138.43059700002414, + "pred_ms": 4.167974999973012, "timing_unit": "ms", "status": "ok" }, @@ -406,8 +478,8 @@ "flow_estimator": "hist_gradient_boosting_regressor", "sklearn_estimator": "HistGradientBoostingRegressor", "dataset": "regression", - "fit_ms": 830.1787499949569, - "pred_ms": 9.42004201351665, + "fit_ms": 127.55505500001618, + "pred_ms": 2.184315000022252, "timing_unit": "ms", "status": "ok" }, @@ -415,8 +487,8 @@ "flow_estimator": "huber_regressor", "sklearn_estimator": "HuberRegressor", "dataset": "regression", - "fit_ms": 6.414291012333706, - "pred_ms": 0.03616600588429719, + "fit_ms": 25.104492999957984, + "pred_ms": 0.1415750000433036, "timing_unit": "ms", "status": "ok" }, @@ -424,8 +496,8 @@ "flow_estimator": "isolation_forest", "sklearn_estimator": "IsolationForest", "dataset": "unsupervised", - "fit_ms": 44.771750006475486, - "pred_ms": 3.546292005921714, + "fit_ms": 105.83585599999878, + "pred_ms": 8.532618000003822, "timing_unit": "ms", "status": "ok" }, @@ -433,8 +505,17 @@ "flow_estimator": "isomap", "sklearn_estimator": "Isomap", "dataset": "unsupervised", - "fit_ms": 4.243792005581781, - "pred_ms": 1.1558330006664619, + "fit_ms": 6.753341000035107, + "pred_ms": 3.713485000048422, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "isotonic", + "sklearn_estimator": "IsotonicRegression", + "dataset": "regression", + "fit_ms": 0.6665170000133003, + "pred_ms": 0.12377199993807153, "timing_unit": "ms", "status": "ok" }, @@ -442,8 +523,8 @@ "flow_estimator": "iterative_imputer", "sklearn_estimator": "IterativeImputer", "dataset": "unsupervised", - "fit_ms": 2.435165995848365, - "pred_ms": 0.33916700340341777, + "fit_ms": 8.773609000058968, + "pred_ms": 1.1086140000315936, "timing_unit": "ms", "status": "ok" }, @@ -451,17 +532,17 @@ "flow_estimator": "kbins_discretizer", "sklearn_estimator": "KBinsDiscretizer", "dataset": "unsupervised", - "fit_ms": 0.5584580067079514, - "pred_ms": 0.36358401121106, + "fit_ms": 2.3598740000352336, + "pred_ms": 1.493714000048385, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "kernel_density", "sklearn_estimator": "KernelDensity", - "dataset": "unsupervised", - "fit_ms": 0.10016700252890587, - "pred_ms": 0.0, + "dataset": "classification", + "fit_ms": 0.39112199999635777, + "pred_ms": 1.4570960000810373, "timing_unit": "ms", "status": "ok" }, @@ -469,8 +550,8 @@ "flow_estimator": "kernel_pca", "sklearn_estimator": "KernelPCA", "dataset": "unsupervised", - "fit_ms": 1.56787499145139, - "pred_ms": 0.24125000345520675, + "fit_ms": 2.7608329999111447, + "pred_ms": 0.8307840000725264, "timing_unit": "ms", "status": "ok" }, @@ -478,8 +559,8 @@ "flow_estimator": "kernel_ridge", "sklearn_estimator": "KernelRidge", "dataset": "regression", - "fit_ms": 1.1393750028219074, - "pred_ms": 0.2375839976593852, + "fit_ms": 5.762537000009615, + "pred_ms": 0.612816000057137, "timing_unit": "ms", "status": "ok" }, @@ -487,8 +568,8 @@ "flow_estimator": "kernel_svc", "sklearn_estimator": "SVC", "dataset": "classification", - "fit_ms": 0.39362499956041574, - "pred_ms": 0.27049999334849417, + "fit_ms": 1.4554320000570442, + "pred_ms": 0.6788700000015524, "timing_unit": "ms", "status": "ok" }, @@ -496,8 +577,8 @@ "flow_estimator": "kernel_svc_multi", "sklearn_estimator": "SVC", "dataset": "classification", - "fit_ms": 0.35620899870991707, - "pred_ms": 0.2690419933060184, + "fit_ms": 1.45048299998507, + "pred_ms": 0.6936769999583703, "timing_unit": "ms", "status": "ok" }, @@ -505,8 +586,8 @@ "flow_estimator": "kmeans", "sklearn_estimator": "KMeans", "dataset": "unsupervised", - "fit_ms": 0.6812089995946735, - "pred_ms": 0.04833299317397177, + "fit_ms": 2.157305000082488, + "pred_ms": 0.23395799996706046, "timing_unit": "ms", "status": "ok" }, @@ -514,8 +595,8 @@ "flow_estimator": "kneighbors_transformer", "sklearn_estimator": "KNeighborsTransformer", "dataset": "unsupervised", - "fit_ms": 0.10029198892880231, - "pred_ms": 0.260791988694109, + "fit_ms": 0.4205369999681352, + "pred_ms": 0.8866979999311297, "timing_unit": "ms", "status": "ok" }, @@ -523,8 +604,8 @@ "flow_estimator": "knn_classifier", "sklearn_estimator": "KNeighborsClassifier", "dataset": "classification", - "fit_ms": 0.21258299238979816, - "pred_ms": 0.8801670046523213, + "fit_ms": 0.7014819999540123, + "pred_ms": 1.2461399999210698, "timing_unit": "ms", "status": "ok" }, @@ -532,8 +613,8 @@ "flow_estimator": "knn_imputer", "sklearn_estimator": "KNNImputer", "dataset": "unsupervised", - "fit_ms": 0.11808300041593611, - "pred_ms": 0.07766700582578778, + "fit_ms": 0.2999509999881411, + "pred_ms": 0.2132189999883849, "timing_unit": "ms", "status": "ok" }, @@ -541,8 +622,26 @@ "flow_estimator": "knn_regressor", "sklearn_estimator": "KNeighborsRegressor", "dataset": "regression", - "fit_ms": 0.23525000142399222, - "pred_ms": 1.054708001902327, + "fit_ms": 0.5322649999470741, + "pred_ms": 2.5924280000708677, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "label_binarizer", + "sklearn_estimator": "LabelBinarizer", + "dataset": "classification", + "fit_ms": 0.3864130000010846, + "pred_ms": 0.7444020000093587, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "label_encoder", + "sklearn_estimator": "LabelEncoder", + "dataset": "classification", + "fit_ms": 0.09557900000345398, + "pred_ms": 0.15748400005577423, "timing_unit": "ms", "status": "ok" }, @@ -550,8 +649,8 @@ "flow_estimator": "label_propagation", "sklearn_estimator": "LabelPropagation", "dataset": "classification", - "fit_ms": 0.6667079869657755, - "pred_ms": 0.271833996521309, + "fit_ms": 1.1388389999638093, + "pred_ms": 0.6545840000171665, "timing_unit": "ms", "status": "ok" }, @@ -559,8 +658,8 @@ "flow_estimator": "label_spreading", "sklearn_estimator": "LabelSpreading", "dataset": "classification", - "fit_ms": 0.5641659954562783, - "pred_ms": 0.3032079985132441, + "fit_ms": 1.334004999989702, + "pred_ms": 0.6444250000186003, "timing_unit": "ms", "status": "ok" }, @@ -568,8 +667,8 @@ "flow_estimator": "lars_cv", "sklearn_estimator": "LarsCV", "dataset": "regression", - "fit_ms": 2.6605409948388115, - "pred_ms": 0.034958997275680304, + "fit_ms": 9.033325000018522, + "pred_ms": 0.11126799995508918, "timing_unit": "ms", "status": "ok" }, @@ -577,8 +676,8 @@ "flow_estimator": "lars", "sklearn_estimator": "Lars", "dataset": "regression", - "fit_ms": 0.5107080069137737, - "pred_ms": 0.034249998861923814, + "fit_ms": 1.7065209999600484, + "pred_ms": 0.10998600009770598, "timing_unit": "ms", "status": "ok" }, @@ -586,8 +685,8 @@ "flow_estimator": "lasso_cv", "sklearn_estimator": "LassoCV", "dataset": "regression", - "fit_ms": 19.690582994371653, - "pred_ms": 0.036708006518892944, + "fit_ms": 61.7053590000296, + "pred_ms": 0.1105870001083531, "timing_unit": "ms", "status": "ok" }, @@ -595,8 +694,8 @@ "flow_estimator": "lasso", "sklearn_estimator": "Lasso", "dataset": "regression", - "fit_ms": 0.21429199841804802, - "pred_ms": 0.03950000973418355, + "fit_ms": 0.5749660000446966, + "pred_ms": 0.11502500001370208, "timing_unit": "ms", "status": "ok" }, @@ -604,8 +703,8 @@ "flow_estimator": "lasso_lars_cv", "sklearn_estimator": "LassoLarsCV", "dataset": "regression", - "fit_ms": 3.7610419967677444, - "pred_ms": 0.037541991332545877, + "fit_ms": 11.383389999991778, + "pred_ms": 0.1121090000424374, "timing_unit": "ms", "status": "ok" }, @@ -613,8 +712,8 @@ "flow_estimator": "lasso_lars", "sklearn_estimator": "LassoLars", "dataset": "regression", - "fit_ms": 0.37141700158827007, - "pred_ms": 0.03791700873989612, + "fit_ms": 1.0092779999695267, + "pred_ms": 0.10997599997608631, "timing_unit": "ms", "status": "ok" }, @@ -622,8 +721,8 @@ "flow_estimator": "lasso_lars_ic", "sklearn_estimator": "LassoLarsIC", "dataset": "regression", - "fit_ms": 0.9982500050682575, - "pred_ms": 0.03679099609144032, + "fit_ms": 2.913129000035042, + "pred_ms": 0.11061700001846475, "timing_unit": "ms", "status": "ok" }, @@ -631,8 +730,8 @@ "flow_estimator": "lda", "sklearn_estimator": "LatentDirichletAllocation", "dataset": "unsupervised", - "fit_ms": 94.88433299702592, - "pred_ms": 8.573500002967194, + "fit_ms": 274.9038050000081, + "pred_ms": 19.02325000003202, "timing_unit": "ms", "status": "ok" }, @@ -640,7 +739,7 @@ "flow_estimator": "ledoit_wolf_estimator", "sklearn_estimator": "LedoitWolf", "dataset": "unsupervised", - "fit_ms": 0.17004100664053112, + "fit_ms": 0.5891420000807557, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -649,8 +748,8 @@ "flow_estimator": "linear_regression", "sklearn_estimator": "LinearRegression", "dataset": "regression", - "fit_ms": 0.23187499027699232, - "pred_ms": 0.03329200262669474, + "fit_ms": 0.7348850000425955, + "pred_ms": 0.11708900001394795, "timing_unit": "ms", "status": "ok" }, @@ -658,8 +757,8 @@ "flow_estimator": "linear_svc", "sklearn_estimator": "LinearSVC", "dataset": "classification", - "fit_ms": 0.34545800008345395, - "pred_ms": 0.04991699825040996, + "fit_ms": 1.1295929999732834, + "pred_ms": 0.1753680001002067, "timing_unit": "ms", "status": "ok" }, @@ -667,8 +766,8 @@ "flow_estimator": "linear_svc_multi", "sklearn_estimator": "LinearSVC", "dataset": "classification", - "fit_ms": 0.32787499367259443, - "pred_ms": 0.04933300078846514, + "fit_ms": 1.120105000040894, + "pred_ms": 0.1717210000151681, "timing_unit": "ms", "status": "ok" }, @@ -676,8 +775,8 @@ "flow_estimator": "linear_svr", "sklearn_estimator": "LinearSVR", "dataset": "regression", - "fit_ms": 0.20270899403840303, - "pred_ms": 0.03354200453031808, + "fit_ms": 0.5125380000663426, + "pred_ms": 0.11903200004326209, "timing_unit": "ms", "status": "ok" }, @@ -685,8 +784,8 @@ "flow_estimator": "lle", "sklearn_estimator": "LocallyLinearEmbedding", "dataset": "unsupervised", - "fit_ms": 4.141000012168661, - "pred_ms": 3.195874989614822, + "fit_ms": 14.871594000055666, + "pred_ms": 16.7186489999267, "timing_unit": "ms", "status": "ok" }, @@ -694,7 +793,7 @@ "flow_estimator": "local_outlier_factor", "sklearn_estimator": "LocalOutlierFactor", "dataset": "unsupervised", - "fit_ms": 0.395416995161213, + "fit_ms": 1.4632869999786635, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -703,8 +802,8 @@ "flow_estimator": "logistic_regression_cv", "sklearn_estimator": "LogisticRegressionCV", "dataset": "classification", - "fit_ms": 92.88212499814108, - "pred_ms": 0.04820800677407533, + "fit_ms": 490.10222300000805, + "pred_ms": 0.22208600000794831, "timing_unit": "ms", "status": "ok" }, @@ -712,8 +811,8 @@ "flow_estimator": "logistic_regression", "sklearn_estimator": "LogisticRegression", "dataset": "classification", - "fit_ms": 4.227999990689568, - "pred_ms": 0.04837500455323607, + "fit_ms": 24.644148999982463, + "pred_ms": 0.21482299996478105, "timing_unit": "ms", "status": "ok" }, @@ -721,8 +820,8 @@ "flow_estimator": "maxabs_scaler", "sklearn_estimator": "MaxAbsScaler", "dataset": "unsupervised", - "fit_ms": 0.06274999759625643, - "pred_ms": 0.042625004425644875, + "fit_ms": 0.20133800001076452, + "pred_ms": 0.15776600002936902, "timing_unit": "ms", "status": "ok" }, @@ -730,7 +829,7 @@ "flow_estimator": "mds", "sklearn_estimator": "MDS", "dataset": "unsupervised", - "fit_ms": 19.73887500935234, + "fit_ms": 44.649359999993976, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -739,8 +838,8 @@ "flow_estimator": "mean_shift", "sklearn_estimator": "MeanShift", "dataset": "unsupervised", - "fit_ms": 124.32579199958127, - "pred_ms": 0.2981249999720603, + "fit_ms": 451.3265709999814, + "pred_ms": 0.5742230000578274, "timing_unit": "ms", "status": "ok" }, @@ -748,7 +847,7 @@ "flow_estimator": "min_cov_det", "sklearn_estimator": "MinCovDet", "dataset": "unsupervised", - "fit_ms": 8.830791994114406, + "fit_ms": 33.63587000001189, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -757,8 +856,8 @@ "flow_estimator": "minibatch_dictionary_learning", "sklearn_estimator": "MiniBatchDictionaryLearning", "dataset": "unsupervised", - "fit_ms": 190.04408399632666, - "pred_ms": 1.0352090030210093, + "fit_ms": 697.1792020000294, + "pred_ms": 3.646750000029897, "timing_unit": "ms", "status": "ok" }, @@ -766,8 +865,8 @@ "flow_estimator": "minibatch_kmeans", "sklearn_estimator": "MiniBatchKMeans", "dataset": "unsupervised", - "fit_ms": 6.938249993254431, - "pred_ms": 0.05083299765828997, + "fit_ms": 4.08062200006043, + "pred_ms": 0.24101000008158735, "timing_unit": "ms", "status": "ok" }, @@ -775,8 +874,8 @@ "flow_estimator": "minibatch_nmf", "sklearn_estimator": "MiniBatchNMF", "dataset": "unsupervised", - "fit_ms": 5.672750005032867, - "pred_ms": 1.905458004330285, + "fit_ms": 17.191092999951252, + "pred_ms": 5.771494999976312, "timing_unit": "ms", "status": "ok" }, @@ -784,8 +883,8 @@ "flow_estimator": "minibatch_sparse_pca", "sklearn_estimator": "MiniBatchSparsePCA", "dataset": "unsupervised", - "fit_ms": 5.367624995415099, - "pred_ms": 0.24074999964796007, + "fit_ms": 14.231178000045475, + "pred_ms": 0.6071560000009413, "timing_unit": "ms", "status": "ok" }, @@ -793,8 +892,8 @@ "flow_estimator": "minmax_scaler", "sklearn_estimator": "MinMaxScaler", "dataset": "unsupervised", - "fit_ms": 0.05862499529030174, - "pred_ms": 0.03820900747086853, + "fit_ms": 0.20705700001144578, + "pred_ms": 0.13484199996582902, "timing_unit": "ms", "status": "ok" }, @@ -802,8 +901,8 @@ "flow_estimator": "missing_indicator", "sklearn_estimator": "MissingIndicator", "dataset": "unsupervised", - "fit_ms": 0.15312500181607902, - "pred_ms": 0.14545800513587892, + "fit_ms": 0.41477500008113566, + "pred_ms": 0.3650029999562321, "timing_unit": "ms", "status": "ok" }, @@ -811,8 +910,8 @@ "flow_estimator": "mlp_classifier", "sklearn_estimator": "MLPClassifier", "dataset": "classification", - "fit_ms": 31.965916001354344, - "pred_ms": 0.11833300231955945, + "fit_ms": 91.95177200001581, + "pred_ms": 0.30079300006491394, "timing_unit": "ms", "status": "ok" }, @@ -820,8 +919,8 @@ "flow_estimator": "mlp_regressor", "sklearn_estimator": "MLPRegressor", "dataset": "regression", - "fit_ms": 78.97175000107381, - "pred_ms": 0.08762499783188105, + "fit_ms": 210.32624400004352, + "pred_ms": 0.2009559999578414, "timing_unit": "ms", "status": "ok" }, @@ -829,8 +928,8 @@ "flow_estimator": "multi_output_classifier", "sklearn_estimator": "MultiOutputClassifier", "dataset": "multioutput_class", - "fit_ms": 0.6964170024730265, - "pred_ms": 0.14966700109653175, + "fit_ms": 2.3365199999716424, + "pred_ms": 0.46384799998122617, "timing_unit": "ms", "status": "ok" }, @@ -838,8 +937,8 @@ "flow_estimator": "multi_output_regressor", "sklearn_estimator": "MultiOutputRegressor", "dataset": "multioutput", - "fit_ms": 1.2234579917276278, - "pred_ms": 0.1346669887425378, + "fit_ms": 2.6365209999994477, + "pred_ms": 0.4636470000605186, "timing_unit": "ms", "status": "ok" }, @@ -847,8 +946,17 @@ "flow_estimator": "multiclass_logistic", "sklearn_estimator": "LogisticRegression", "dataset": "classification", - "fit_ms": 4.640500003006309, - "pred_ms": 0.05041599797550589, + "fit_ms": 24.63859199997387, + "pred_ms": 0.21615499997551524, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "multilabel_binarizer", + "sklearn_estimator": "MultiLabelBinarizer", + "dataset": "multioutput_class", + "fit_ms": 0.0728359999584427, + "pred_ms": 0.1982009999892398, "timing_unit": "ms", "status": "ok" }, @@ -856,8 +964,8 @@ "flow_estimator": "multinomial_nb", "sklearn_estimator": "MultinomialNB", "dataset": "classification", - "fit_ms": 0.4004160000476986, - "pred_ms": 0.03708299482241273, + "fit_ms": 1.7618350000248029, + "pred_ms": 0.15110300000742427, "timing_unit": "ms", "status": "ok" }, @@ -865,8 +973,8 @@ "flow_estimator": "multitask_elastic_net_cv", "sklearn_estimator": "MultiTaskElasticNetCV", "dataset": "multioutput", - "fit_ms": 17.617125005926937, - "pred_ms": 0.04141598765272647, + "fit_ms": 27.260045000048194, + "pred_ms": 0.12365100008082663, "timing_unit": "ms", "status": "ok" }, @@ -874,8 +982,8 @@ "flow_estimator": "multitask_elastic_net", "sklearn_estimator": "MultiTaskElasticNet", "dataset": "multioutput", - "fit_ms": 0.1660410052863881, - "pred_ms": 0.04279200220480561, + "fit_ms": 0.4847870000048715, + "pred_ms": 0.12576500000704982, "timing_unit": "ms", "status": "ok" }, @@ -883,8 +991,8 @@ "flow_estimator": "multitask_lasso_cv", "sklearn_estimator": "MultiTaskLassoCV", "dataset": "multioutput", - "fit_ms": 111.09995799779426, - "pred_ms": 0.04137499490752816, + "fit_ms": 80.9648519999655, + "pred_ms": 0.12173799996162415, "timing_unit": "ms", "status": "ok" }, @@ -892,8 +1000,8 @@ "flow_estimator": "multitask_lasso", "sklearn_estimator": "MultiTaskLasso", "dataset": "multioutput", - "fit_ms": 0.17275000573135912, - "pred_ms": 0.04425000224728137, + "fit_ms": 0.49439500003245485, + "pred_ms": 0.12424199996985408, "timing_unit": "ms", "status": "ok" }, @@ -901,8 +1009,8 @@ "flow_estimator": "nca", "sklearn_estimator": "NeighborhoodComponentsAnalysis", "dataset": "regression", - "fit_ms": 36.13787500944454, - "pred_ms": 0.04454200097825378, + "fit_ms": 1337.9544900000155, + "pred_ms": 0.15134299997043854, "timing_unit": "ms", "status": "ok" }, @@ -910,17 +1018,17 @@ "flow_estimator": "nearest_centroid", "sklearn_estimator": "NearestCentroid", "dataset": "classification", - "fit_ms": 0.23249999503605068, - "pred_ms": 0.396334013203159, + "fit_ms": 1.07006200005344, + "pred_ms": 0.7310070000130509, "timing_unit": "ms", "status": "ok" }, { "flow_estimator": "nearest_neighbors", "sklearn_estimator": "NearestNeighbors", - "dataset": "unsupervised", - "fit_ms": 0.1144580019172281, - "pred_ms": 0.0, + "dataset": "classification", + "fit_ms": 0.4377989999966303, + "pred_ms": 0.7638490000090314, "timing_unit": "ms", "status": "ok" }, @@ -928,8 +1036,8 @@ "flow_estimator": "nmf", "sklearn_estimator": "NMF", "dataset": "unsupervised", - "fit_ms": 1.8606250087032095, - "pred_ms": 0.9039169963216409, + "fit_ms": 4.818331999899783, + "pred_ms": 2.4865210000371007, "timing_unit": "ms", "status": "ok" }, @@ -937,8 +1045,8 @@ "flow_estimator": "nu_svc", "sklearn_estimator": "NuSVC", "dataset": "regression", - "fit_ms": 34.70416700292844, - "pred_ms": 28.50808299263008, + "fit_ms": 71.96687699990889, + "pred_ms": 76.01907599996593, "timing_unit": "ms", "status": "ok" }, @@ -946,8 +1054,8 @@ "flow_estimator": "nu_svr", "sklearn_estimator": "NuSVR", "dataset": "regression", - "fit_ms": 2.2352499945554882, - "pred_ms": 2.0772080024471506, + "fit_ms": 5.217398000013418, + "pred_ms": 3.53818799999317, "timing_unit": "ms", "status": "ok" }, @@ -955,8 +1063,8 @@ "flow_estimator": "nystroem", "sklearn_estimator": "Nystroem", "dataset": "unsupervised", - "fit_ms": 1.0292500082869083, - "pred_ms": 0.2373329916736111, + "fit_ms": 2.466693999963354, + "pred_ms": 1.032641999927364, "timing_unit": "ms", "status": "ok" }, @@ -964,7 +1072,7 @@ "flow_estimator": "oas_estimator", "sklearn_estimator": "OAS", "dataset": "unsupervised", - "fit_ms": 0.11737500608433038, + "fit_ms": 0.5394989999558675, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -973,8 +1081,8 @@ "flow_estimator": "omp_cv", "sklearn_estimator": "OrthogonalMatchingPursuitCV", "dataset": "regression", - "fit_ms": 1.1912499903701246, - "pred_ms": 0.03545799700077623, + "fit_ms": 5.473237000046538, + "pred_ms": 0.13762799994765373, "timing_unit": "ms", "status": "ok" }, @@ -982,8 +1090,8 @@ "flow_estimator": "one_class_svm", "sklearn_estimator": "OneClassSVM", "dataset": "unsupervised", - "fit_ms": 0.2614169934531674, - "pred_ms": 0.2536670072004199, + "fit_ms": 1.246170999934293, + "pred_ms": 0.7849180000221168, "timing_unit": "ms", "status": "ok" }, @@ -991,8 +1099,8 @@ "flow_estimator": "one_vs_one", "sklearn_estimator": "OneVsOneClassifier", "dataset": "classification", - "fit_ms": 1.0546660050749779, - "pred_ms": 0.261875000433065, + "fit_ms": 4.303139000057854, + "pred_ms": 1.2659779999921739, "timing_unit": "ms", "status": "ok" }, @@ -1000,8 +1108,8 @@ "flow_estimator": "one_vs_rest", "sklearn_estimator": "OneVsRestClassifier", "dataset": "classification", - "fit_ms": 1.2373339995974675, - "pred_ms": 0.12208399130031466, + "fit_ms": 5.417092000016055, + "pred_ms": 0.5428250000250046, "timing_unit": "ms", "status": "ok" }, @@ -1009,8 +1117,8 @@ "flow_estimator": "onehot_encoder", "sklearn_estimator": "OneHotEncoder", "dataset": "unsupervised", - "fit_ms": 0.19645900465548038, - "pred_ms": 0.31279200629796833, + "fit_ms": 0.8930199999213073, + "pred_ms": 1.0062819999347994, "timing_unit": "ms", "status": "ok" }, @@ -1018,7 +1126,7 @@ "flow_estimator": "optics", "sklearn_estimator": "OPTICS", "dataset": "unsupervised", - "fit_ms": 33.755458003724925, + "fit_ms": 118.60569999998916, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1027,8 +1135,8 @@ "flow_estimator": "ordinal_encoder", "sklearn_estimator": "OrdinalEncoder", "dataset": "unsupervised", - "fit_ms": 0.19041699124500155, - "pred_ms": 0.2847920113708824, + "fit_ms": 0.680052000006981, + "pred_ms": 0.8970879999878889, "timing_unit": "ms", "status": "ok" }, @@ -1036,8 +1144,8 @@ "flow_estimator": "orthogonal_matching_pursuit", "sklearn_estimator": "OrthogonalMatchingPursuit", "dataset": "regression", - "fit_ms": 0.163792006787844, - "pred_ms": 0.03566600207705051, + "fit_ms": 0.5305129999442215, + "pred_ms": 0.10756099993614043, "timing_unit": "ms", "status": "ok" }, @@ -1045,8 +1153,8 @@ "flow_estimator": "output_code", "sklearn_estimator": "OutputCodeClassifier", "dataset": "classification", - "fit_ms": 1.1059580137953162, - "pred_ms": 0.6682499952148646, + "fit_ms": 2.8231400000322537, + "pred_ms": 1.0737679999692773, "timing_unit": "ms", "status": "ok" }, @@ -1054,8 +1162,8 @@ "flow_estimator": "passive_aggressive_classifier", "sklearn_estimator": "PassiveAggressiveClassifier", "dataset": "regression", - "fit_ms": 29.842375006410293, - "pred_ms": 0.14250000822357833, + "fit_ms": 81.21024599995508, + "pred_ms": 0.35704799995528447, "timing_unit": "ms", "status": "ok" }, @@ -1063,8 +1171,8 @@ "flow_estimator": "passive_aggressive_regressor", "sklearn_estimator": "PassiveAggressiveRegressor", "dataset": "regression", - "fit_ms": 0.7141670066630468, - "pred_ms": 0.044666987378150225, + "fit_ms": 2.224941999998009, + "pred_ms": 0.14530199996443116, "timing_unit": "ms", "status": "ok" }, @@ -1072,8 +1180,8 @@ "flow_estimator": "pca", "sklearn_estimator": "PCA", "dataset": "unsupervised", - "fit_ms": 0.12437500117812306, - "pred_ms": 0.03454201214481145, + "fit_ms": 0.4809499999964828, + "pred_ms": 0.16190299993468216, "timing_unit": "ms", "status": "ok" }, @@ -1081,8 +1189,17 @@ "flow_estimator": "perceptron", "sklearn_estimator": "Perceptron", "dataset": "regression", - "fit_ms": 26.729792007245123, - "pred_ms": 0.1406660012435168, + "fit_ms": 75.75591399995574, + "pred_ms": 0.3624580000405331, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "pipeline", + "sklearn_estimator": "Pipeline", + "dataset": "classification", + "fit_ms": 6.262884999955531, + "pred_ms": 0.47522900001695234, "timing_unit": "ms", "status": "ok" }, @@ -1090,8 +1207,8 @@ "flow_estimator": "pls_canonical", "sklearn_estimator": "PLSCanonical", "dataset": "multioutput", - "fit_ms": 0.27099999715574086, - "pred_ms": 0.04829200042877346, + "fit_ms": 1.0537919999933365, + "pred_ms": 0.17586899991783866, "timing_unit": "ms", "status": "ok" }, @@ -1099,8 +1216,8 @@ "flow_estimator": "pls", "sklearn_estimator": "PLSRegression", "dataset": "multioutput", - "fit_ms": 0.30429101025220007, - "pred_ms": 0.04700000863522291, + "fit_ms": 1.1459730000069612, + "pred_ms": 0.1752680000208784, "timing_unit": "ms", "status": "ok" }, @@ -1108,8 +1225,8 @@ "flow_estimator": "pls_svd", "sklearn_estimator": "PLSSVD", "dataset": "multioutput", - "fit_ms": 0.16183400293812156, - "pred_ms": 0.04591699689626694, + "fit_ms": 0.6443149999313391, + "pred_ms": 0.16753300008076621, "timing_unit": "ms", "status": "ok" }, @@ -1117,8 +1234,26 @@ "flow_estimator": "poisson_regressor", "sklearn_estimator": "PoissonRegressor", "dataset": "regression", - "fit_ms": 3.4219579974887893, - "pred_ms": 0.046707995352335274, + "fit_ms": 189.43763300001137, + "pred_ms": 0.18650899994554493, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "polynomial_count_sketch", + "sklearn_estimator": "PolynomialCountSketch", + "dataset": "classification", + "fit_ms": 0.5104749999418345, + "pred_ms": 0.27829999999084976, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "polynomial_features", + "sklearn_estimator": "PolynomialFeatures", + "dataset": "classification", + "fit_ms": 0.21067400007268589, + "pred_ms": 0.2116059999934805, "timing_unit": "ms", "status": "ok" }, @@ -1126,8 +1261,8 @@ "flow_estimator": "power_transformer", "sklearn_estimator": "PowerTransformer", "dataset": "unsupervised", - "fit_ms": 7.24483399244491, - "pred_ms": 0.14595799439121038, + "fit_ms": 57.098075999988396, + "pred_ms": 0.5184699999745135, "timing_unit": "ms", "status": "ok" }, @@ -1135,8 +1270,8 @@ "flow_estimator": "qda", "sklearn_estimator": "QuadraticDiscriminantAnalysis", "dataset": "classification", - "fit_ms": 0.24308300635311753, - "pred_ms": 0.05541701102629304, + "fit_ms": 0.8154849999755243, + "pred_ms": 0.19161800003075768, "timing_unit": "ms", "status": "ok" }, @@ -1144,8 +1279,8 @@ "flow_estimator": "quantile_regressor", "sklearn_estimator": "QuantileRegressor", "dataset": "regression", - "fit_ms": 7.912083005066961, - "pred_ms": 0.036124998587183654, + "fit_ms": 15.41101499992692, + "pred_ms": 0.11063700003433041, "timing_unit": "ms", "status": "ok" }, @@ -1153,8 +1288,8 @@ "flow_estimator": "quantile_transformer", "sklearn_estimator": "QuantileTransformer", "dataset": "unsupervised", - "fit_ms": 0.23933300690259784, - "pred_ms": 0.0967499945545569, + "fit_ms": 0.7191049999164534, + "pred_ms": 0.2374130000362129, "timing_unit": "ms", "status": "ok" }, @@ -1162,8 +1297,8 @@ "flow_estimator": "radius_neighbors_classifier", "sklearn_estimator": "RadiusNeighborsClassifier", "dataset": "classification", - "fit_ms": 0.21437500254251063, - "pred_ms": 0.4346670029917732, + "fit_ms": 0.6991179999431552, + "pred_ms": 1.3079870000183291, "timing_unit": "ms", "status": "ok" }, @@ -1171,8 +1306,8 @@ "flow_estimator": "radius_neighbors_regressor", "sklearn_estimator": "RadiusNeighborsRegressor", "dataset": "regression", - "fit_ms": 0.13745900650974363, - "pred_ms": 2.245166993816383, + "fit_ms": 0.5379660000244257, + "pred_ms": 6.417503000079705, "timing_unit": "ms", "status": "ok" }, @@ -1180,8 +1315,8 @@ "flow_estimator": "radius_neighbors_transformer", "sklearn_estimator": "RadiusNeighborsTransformer", "dataset": "unsupervised", - "fit_ms": 0.09800000407267362, - "pred_ms": 0.3641660005087033, + "fit_ms": 0.3212210000356208, + "pred_ms": 1.029906999974628, "timing_unit": "ms", "status": "ok" }, @@ -1189,8 +1324,8 @@ "flow_estimator": "random_forest_classifier", "sklearn_estimator": "RandomForestClassifier", "dataset": "classification", - "fit_ms": 40.36312499374617, - "pred_ms": 2.061082996078767, + "fit_ms": 115.69737000002078, + "pred_ms": 7.415239000010843, "timing_unit": "ms", "status": "ok" }, @@ -1198,8 +1333,8 @@ "flow_estimator": "random_forest_regressor", "sklearn_estimator": "RandomForestRegressor", "dataset": "regression", - "fit_ms": 101.38191700389143, - "pred_ms": 5.04854200698901, + "fit_ms": 204.0678299999854, + "pred_ms": 11.740860000031716, "timing_unit": "ms", "status": "ok" }, @@ -1207,8 +1342,8 @@ "flow_estimator": "random_trees_embedding", "sklearn_estimator": "RandomTreesEmbedding", "dataset": "unsupervised", - "fit_ms": 28.07333400414791, - "pred_ms": 7.932915992569178, + "fit_ms": 88.03967999995166, + "pred_ms": 25.478564999957598, "timing_unit": "ms", "status": "ok" }, @@ -1216,8 +1351,17 @@ "flow_estimator": "ransac_regressor", "sklearn_estimator": "RANSACRegressor", "dataset": "regression", - "fit_ms": 24.158707994502038, - "pred_ms": 0.052499992307275534, + "fit_ms": 83.18000999997821, + "pred_ms": 0.18335199990815454, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "rbf_sampler", + "sklearn_estimator": "RBFSampler", + "dataset": "classification", + "fit_ms": 0.3488220000917863, + "pred_ms": 0.1205249999429725, "timing_unit": "ms", "status": "ok" }, @@ -1225,8 +1369,8 @@ "flow_estimator": "regressor_chain", "sklearn_estimator": "RegressorChain", "dataset": "multioutput_class", - "fit_ms": 0.44129199523013085, - "pred_ms": 0.18250000721309334, + "fit_ms": 1.435135000065202, + "pred_ms": 0.651097999934791, "timing_unit": "ms", "status": "ok" }, @@ -1234,8 +1378,8 @@ "flow_estimator": "rfe", "sklearn_estimator": "RFE", "dataset": "regression", - "fit_ms": 8.731499998248182, - "pred_ms": 0.20066701108589768, + "fit_ms": 14.55346099999133, + "pred_ms": 0.4378689999384733, "timing_unit": "ms", "status": "ok" }, @@ -1243,8 +1387,8 @@ "flow_estimator": "rfecv", "sklearn_estimator": "RFECV", "dataset": "regression", - "fit_ms": 74.50441700348165, - "pred_ms": 0.1781250030035153, + "fit_ms": 168.7907580000001, + "pred_ms": 0.44900999989749835, "timing_unit": "ms", "status": "ok" }, @@ -1252,8 +1396,8 @@ "flow_estimator": "ridge_classifier_cv", "sklearn_estimator": "RidgeClassifierCV", "dataset": "classification", - "fit_ms": 0.9015840041683987, - "pred_ms": 0.04933300078846514, + "fit_ms": 2.547154000012597, + "pred_ms": 0.17081999999390973, "timing_unit": "ms", "status": "ok" }, @@ -1261,8 +1405,8 @@ "flow_estimator": "ridge_classifier", "sklearn_estimator": "RidgeClassifier", "dataset": "regression", - "fit_ms": 1.1716669978341088, - "pred_ms": 0.13649999164044857, + "fit_ms": 8.980347000033362, + "pred_ms": 0.3546630000528239, "timing_unit": "ms", "status": "ok" }, @@ -1270,8 +1414,8 @@ "flow_estimator": "ridge_cv", "sklearn_estimator": "RidgeCV", "dataset": "regression", - "fit_ms": 0.32245900365523994, - "pred_ms": 0.03095800639130175, + "fit_ms": 1.3177950000908822, + "pred_ms": 0.13868999997157516, "timing_unit": "ms", "status": "ok" }, @@ -1279,8 +1423,8 @@ "flow_estimator": "ridge", "sklearn_estimator": "Ridge", "dataset": "regression", - "fit_ms": 0.2145000034943223, - "pred_ms": 0.031167000997811556, + "fit_ms": 0.9657870000410185, + "pred_ms": 0.13871900000594906, "timing_unit": "ms", "status": "ok" }, @@ -1288,8 +1432,8 @@ "flow_estimator": "robust_scaler", "sklearn_estimator": "RobustScaler", "dataset": "unsupervised", - "fit_ms": 0.24712500453460962, - "pred_ms": 0.03325000579934567, + "fit_ms": 1.0423699999364544, + "pred_ms": 0.15088199995716423, "timing_unit": "ms", "status": "ok" }, @@ -1297,8 +1441,8 @@ "flow_estimator": "select_fdr", "sklearn_estimator": "SelectFdr", "dataset": "regression", - "fit_ms": 4.67358400055673, - "pred_ms": 0.06299999949987978, + "fit_ms": 17.835539999964567, + "pred_ms": 0.18360400008532451, "timing_unit": "ms", "status": "ok" }, @@ -1306,8 +1450,17 @@ "flow_estimator": "select_fpr", "sklearn_estimator": "SelectFpr", "dataset": "regression", - "fit_ms": 4.496124995057471, - "pred_ms": 0.047124995035119355, + "fit_ms": 17.36970799993287, + "pred_ms": 0.1663809999854493, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "select_from_model", + "sklearn_estimator": "SelectFromModel", + "dataset": "classification", + "fit_ms": 0.9802130000480247, + "pred_ms": 0.22739499991075718, "timing_unit": "ms", "status": "ok" }, @@ -1315,8 +1468,8 @@ "flow_estimator": "select_fwe", "sklearn_estimator": "SelectFwe", "dataset": "regression", - "fit_ms": 4.529624988208525, - "pred_ms": 0.04804201307706535, + "fit_ms": 17.469614999981786, + "pred_ms": 0.16512799993506633, "timing_unit": "ms", "status": "ok" }, @@ -1324,8 +1477,8 @@ "flow_estimator": "select_k_best", "sklearn_estimator": "SelectKBest", "dataset": "regression", - "fit_ms": 4.775291992700659, - "pred_ms": 0.06295900675468147, + "fit_ms": 17.364017999966563, + "pred_ms": 0.1789950000556928, "timing_unit": "ms", "status": "ok" }, @@ -1333,8 +1486,8 @@ "flow_estimator": "select_percentile", "sklearn_estimator": "SelectPercentile", "dataset": "regression", - "fit_ms": 4.5787910057697445, - "pred_ms": 0.07845899381209165, + "fit_ms": 17.485825999983717, + "pred_ms": 0.2580830000624701, "timing_unit": "ms", "status": "ok" }, @@ -1342,8 +1495,8 @@ "flow_estimator": "sequential_feature_selector", "sklearn_estimator": "SequentialFeatureSelector", "dataset": "regression", - "fit_ms": 78.2538749917876, - "pred_ms": 0.044083993998356164, + "fit_ms": 241.0642290000169, + "pred_ms": 0.15232500004458416, "timing_unit": "ms", "status": "ok" }, @@ -1351,8 +1504,8 @@ "flow_estimator": "sgd_classifier", "sklearn_estimator": "SGDClassifier", "dataset": "classification", - "fit_ms": 0.7614170026499778, - "pred_ms": 0.04858399915974587, + "fit_ms": 2.3275339999599964, + "pred_ms": 0.166691999993418, "timing_unit": "ms", "status": "ok" }, @@ -1360,8 +1513,8 @@ "flow_estimator": "sgd_one_class_svm", "sklearn_estimator": "SGDOneClassSVM", "dataset": "unsupervised", - "fit_ms": 0.7592909969389439, - "pred_ms": 0.030416005756706, + "fit_ms": 1.8832319999546598, + "pred_ms": 0.1177000000325279, "timing_unit": "ms", "status": "ok" }, @@ -1369,8 +1522,8 @@ "flow_estimator": "sgd_regressor", "sklearn_estimator": "SGDRegressor", "dataset": "regression", - "fit_ms": 11.165165997226723, - "pred_ms": 0.04470901330932975, + "fit_ms": 29.55049100000906, + "pred_ms": 0.11614699997153366, "timing_unit": "ms", "status": "ok" }, @@ -1378,7 +1531,7 @@ "flow_estimator": "shrunk_covariance", "sklearn_estimator": "ShrunkCovariance", "dataset": "unsupervised", - "fit_ms": 0.16125000547617674, + "fit_ms": 0.5750850000367791, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1387,8 +1540,17 @@ "flow_estimator": "simple_imputer", "sklearn_estimator": "SimpleImputer", "dataset": "unsupervised", - "fit_ms": 0.2184999902965501, - "pred_ms": 0.1990000018849969, + "fit_ms": 0.6392660000074102, + "pred_ms": 0.5289089999678254, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "skewed_chi2_sampler", + "sklearn_estimator": "SkewedChi2Sampler", + "dataset": "classification", + "fit_ms": 0.3381020000006174, + "pred_ms": 0.14483000006748625, "timing_unit": "ms", "status": "ok" }, @@ -1396,8 +1558,8 @@ "flow_estimator": "sparse_coder", "sklearn_estimator": "SparseCoder", "dataset": "unsupervised", - "fit_ms": 0.05112499638926238, - "pred_ms": 0.9910000080708414, + "fit_ms": 0.1744059999282399, + "pred_ms": 3.5639559999935955, "timing_unit": "ms", "status": "ok" }, @@ -1405,8 +1567,17 @@ "flow_estimator": "sparse_pca", "sklearn_estimator": "SparsePCA", "dataset": "unsupervised", - "fit_ms": 7.765166999888606, - "pred_ms": 0.19429200619924814, + "fit_ms": 27.91739700001017, + "pred_ms": 0.5866469999773471, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "sparse_random_projection", + "sklearn_estimator": "SparseRandomProjection", + "dataset": "classification", + "fit_ms": 0.4879829999708818, + "pred_ms": 0.18307300001652038, "timing_unit": "ms", "status": "ok" }, @@ -1414,7 +1585,7 @@ "flow_estimator": "spectral_biclustering", "sklearn_estimator": "SpectralBiclustering", "dataset": "unsupervised", - "fit_ms": 23.599958993145265, + "fit_ms": 61.68861300000117, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1423,7 +1594,7 @@ "flow_estimator": "spectral_clustering", "sklearn_estimator": "SpectralClustering", "dataset": "unsupervised", - "fit_ms": 5.601083001238294, + "fit_ms": 28.610673000002862, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1432,7 +1603,7 @@ "flow_estimator": "spectral_coclustering", "sklearn_estimator": "SpectralCoclustering", "dataset": "unsupervised", - "fit_ms": 3.2601659913780168, + "fit_ms": 9.960519999935968, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1441,7 +1612,7 @@ "flow_estimator": "spectral_embedding", "sklearn_estimator": "SpectralEmbedding", "dataset": "unsupervised", - "fit_ms": 1.5892500086920336, + "fit_ms": 4.196989999968537, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1450,8 +1621,8 @@ "flow_estimator": "spline_transformer", "sklearn_estimator": "SplineTransformer", "dataset": "unsupervised", - "fit_ms": 0.1152909972006455, - "pred_ms": 0.2897079975809902, + "fit_ms": 0.36234699996384734, + "pred_ms": 0.7130230000029769, "timing_unit": "ms", "status": "ok" }, @@ -1459,8 +1630,8 @@ "flow_estimator": "stacking_regressor", "sklearn_estimator": "StackingRegressor", "dataset": "regression", - "fit_ms": 2.960166006232612, - "pred_ms": 0.08087500464171171, + "fit_ms": 9.800341000072876, + "pred_ms": 0.2768379999906756, "timing_unit": "ms", "status": "ok" }, @@ -1468,8 +1639,8 @@ "flow_estimator": "standard_scaler", "sklearn_estimator": "StandardScaler", "dataset": "unsupervised", - "fit_ms": 0.09716599015519023, - "pred_ms": 0.03808400651905686, + "fit_ms": 0.3388439999980619, + "pred_ms": 0.13577400000031048, "timing_unit": "ms", "status": "ok" }, @@ -1477,8 +1648,8 @@ "flow_estimator": "svc", "sklearn_estimator": "SVC", "dataset": "classification", - "fit_ms": 0.3906659985659644, - "pred_ms": 0.24525000480934978, + "fit_ms": 1.1139040000216482, + "pred_ms": 0.4323290000911584, "timing_unit": "ms", "status": "ok" }, @@ -1486,8 +1657,8 @@ "flow_estimator": "svr", "sklearn_estimator": "SVR", "dataset": "regression", - "fit_ms": 2.543499998864718, - "pred_ms": 4.009917000075802, + "fit_ms": 6.511749000083, + "pred_ms": 6.77366999991591, "timing_unit": "ms", "status": "ok" }, @@ -1495,8 +1666,17 @@ "flow_estimator": "target_encoder", "sklearn_estimator": "TargetEncoder", "dataset": "regression", - "fit_ms": 9.338125004433095, - "pred_ms": 4.422082987730391, + "fit_ms": 22.338150999985373, + "pred_ms": 11.01051400007691, + "timing_unit": "ms", + "status": "ok" + }, + { + "flow_estimator": "tfidf_vectorizer", + "sklearn_estimator": "TfidfVectorizer", + "dataset": "classification", + "fit_ms": 0.6660459999920931, + "pred_ms": 0.48797199997352436, "timing_unit": "ms", "status": "ok" }, @@ -1504,8 +1684,8 @@ "flow_estimator": "theil_sen_regressor", "sklearn_estimator": "TheilSenRegressor", "dataset": "regression", - "fit_ms": 185.33920800837222, - "pred_ms": 0.039874998037703335, + "fit_ms": 377.05291400004626, + "pred_ms": 0.10736100000485749, "timing_unit": "ms", "status": "ok" }, @@ -1513,8 +1693,8 @@ "flow_estimator": "transformed_target_regressor", "sklearn_estimator": "TransformedTargetRegressor", "dataset": "regression", - "fit_ms": 0.4072919982718304, - "pred_ms": 0.05774998862762004, + "fit_ms": 1.4540189999934228, + "pred_ms": 0.20172799997908442, "timing_unit": "ms", "status": "ok" }, @@ -1522,8 +1702,8 @@ "flow_estimator": "truncated_svd", "sklearn_estimator": "TruncatedSVD", "dataset": "unsupervised", - "fit_ms": 0.2280000044265762, - "pred_ms": 0.028750000637955964, + "fit_ms": 0.5172379999294208, + "pred_ms": 0.10895400009758305, "timing_unit": "ms", "status": "ok" }, @@ -1531,7 +1711,7 @@ "flow_estimator": "tsne", "sklearn_estimator": "TSNE", "dataset": "unsupervised", - "fit_ms": 415.7947920029983, + "fit_ms": 384.9840850000419, "pred_ms": 0.0, "timing_unit": "ms", "status": "ok" @@ -1540,8 +1720,8 @@ "flow_estimator": "tweedie_regressor", "sklearn_estimator": "TweedieRegressor", "dataset": "regression", - "fit_ms": 0.7009170076344162, - "pred_ms": 0.05162500019650906, + "fit_ms": 1.303408000012496, + "pred_ms": 0.20056500000009692, "timing_unit": "ms", "status": "ok" }, @@ -1549,8 +1729,8 @@ "flow_estimator": "variance_threshold", "sklearn_estimator": "VarianceThreshold", "dataset": "unsupervised", - "fit_ms": 0.08800000068731606, - "pred_ms": 0.046250002924352884, + "fit_ms": 0.3346549999605486, + "pred_ms": 0.1832619999504459, "timing_unit": "ms", "status": "ok" } diff --git a/benchmarks/generate_estimator_bench.py b/benchmarks/generate_estimator_bench.py index 52b601d..b4f13b5 100644 --- a/benchmarks/generate_estimator_bench.py +++ b/benchmarks/generate_estimator_bench.py @@ -119,6 +119,20 @@ def fit_arguments(entry: dict, kind: str) -> tuple[list[str], list[str]]: return pre, args +# One value out of every result is added to a running sink, which main prints +# on a line the parser ignores. Without it, clang at -O3 is free to delete a +# transform whose result is freed without being read, and two rows came back at +# exactly 0.000000 ms because it did. +def sink_line(var: str, returns: str, indent: str = " ") -> list[str]: + if returns == "Matrix": + return [ + f"{indent}if {var}.rows > 0 {{", + f"{indent} if {var}.cols > 0 {{ sink = sink + {var}.data[0] }}", + f"{indent}}}", + ] + return [f"{indent}sink = sink + {var}[0]"] + + def block(entry: dict) -> str: """One estimator: probe, choose a repeat count, then time fit and predict. @@ -159,6 +173,7 @@ def release(var: str, kind_: str) -> str: lines.append(" reps = 1") lines.append(" if (t1 - t0) < 200000 { reps = 200 }") lines.append(" elif (t1 - t0) < 2000000 { reps = 20 }") + lines.append(" elif (t1 - t0) < 20000000 { reps = 5 }") # Timed fit loop. lines.append(" t0 = flow_now_ns()") @@ -175,6 +190,7 @@ def release(var: str, kind_: str) -> str: lines.append(" t2 = flow_now_ns()") lines.append(" for rep2 in 0 to reps {") lines.append(f" let o_{name}: {comp['returns']} = {comp['name']}(fitted_{name}, X_{suffix})") + lines += sink_line(f"o_{name}", comp["returns"]) lines.append(" " + release(f"o_{name}", comp["returns"])) lines.append(" }") lines.append(" t3 = flow_now_ns()") @@ -194,8 +210,97 @@ def release(var: str, kind_: str) -> str: return "\n".join(lines) + "\n" +def shaped_block(entry: dict) -> str: + """One estimator whose fit does not begin with a feature matrix. + + The registry carries the call for these, because the generic path above + builds one shape of call and these take a target vector, a feature count or + a pair of scalars instead. Everything else about the timing is the same, so + a shaped row is ranked like any other. + """ + name = entry["flow_estimator"] + shape = entry["shape"] + kind = shape["dataset"] + ret = entry["fit"]["returns"] + call = f"{entry['fit']['name']}({', '.join(shape['flow_fit'])})" + free = entry["companions"].get("free") + # A fit that takes a composed object built outside the timing mutates that + # object and hands it back, so freeing the result on every repeat would free + # what the next repeat is about to read. Those rows free once, after the + # timing, and leak the state a repeat leaves behind, which is a few hundred + # bytes per pass over a four-column design. + free_in_loop = shape.get("flow_free", "loop") == "loop" + free_ok = free is not None and len(free["parameters"]) == 1 + comp = entry["companions"].get("transform") or entry["companions"].get("predict") + work = shape.get("flow_work") + # A recipe can name the work function itself, for an estimator whose + # companion is called something other than predict or transform. + if shape.get("flow_work_fn"): + comp = {"name": shape["flow_work_fn"], "returns": shape["flow_work_returns"]} + comp_ok = comp is not None and work is not None and ( + comp["returns"] in ("Matrix", "ptr") or shape.get("flow_work_release") + ) + + lines = [f" # ---- {name} ({kind}, written out) ----"] + if shape.get("corpus"): + docs = shape["corpus"] + lines.append(f" let {name}_docs: array = [") + for i, doc in enumerate(docs): + tail = "," if i + 1 < len(docs) else "" + lines.append(f' "{doc}"{tail}') + lines.append(" ]") + for pre_line in shape.get("flow_preamble", []): + lines.append(" " + pre_line) + lines.append(" t0 = flow_now_ns()") + lines.append(f" let probe_{name}: {ret} = {call}") + lines.append(" t1 = flow_now_ns()") + if free_ok and free_in_loop: + lines.append(f" {free['name']}(probe_{name})") + lines.append(" reps = 1") + lines.append(" if (t1 - t0) < 200000 { reps = 200 }") + lines.append(" elif (t1 - t0) < 2000000 { reps = 20 }") + lines.append(" elif (t1 - t0) < 20000000 { reps = 5 }") + lines.append(" t0 = flow_now_ns()") + lines.append(" for rep in 0 to reps {") + lines.append(f" let m_{name}: {ret} = {call}") + if free_ok and free_in_loop: + lines.append(f" {free['name']}(m_{name})") + lines.append(" }") + lines.append(" t1 = flow_now_ns()") + + if comp_ok: + release = "matrix_free" if comp["returns"] == "Matrix" else "array_free_f32" + lines.append(f" let fitted_{name}: {ret} = {call}") + lines.append(" t2 = flow_now_ns()") + lines.append(" for rep2 in 0 to reps {") + lines.append(f" let o_{name}: {comp['returns']} = " + f"{comp['name']}(fitted_{name}, {', '.join(work)})") + if comp["returns"] in ("Matrix", "ptr"): + lines += sink_line(f"o_{name}", comp["returns"]) + if shape.get("flow_work_release"): + lines.append(" " + shape["flow_work_release"].format(var=f"o_{name}")) + else: + lines.append(f" {release}(o_{name})") + lines.append(" }") + lines.append(" t3 = flow_now_ns()") + if free_ok: + lines.append(f" {free['name']}(fitted_{name})") + pred_expr = "ms_between(t2, t3) / (reps as f32)" + else: + pred_expr = "0.0" + if free_ok and not free_in_loop: + lines.append(f" {free['name']}(probe_{name})") + + lines.append( + f' printf("ESTIMATOR|{name}|%.9f|%.9f|%d|ok\\n", ' + f"ms_between(t0, t1) / (reps as f32), {pred_expr}, reps)" + ) + lines.append(" fflush(null)") + return "\n".join(lines) + "\n" + + def chunk_file(index: int, entries: list[dict]) -> str: - body = "\n".join(block(e) for e in entries) + body = "\n".join(shaped_block(e) if e["bucket"] == "shaped" else block(e) for e in entries) return f'''{HEADER} function main() -> i32 {{ let iris: Dataset = load_iris() @@ -243,11 +348,22 @@ def chunk_file(index: int, entries: list[dict]) -> str: Y_rows[i] = row }} + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r {{ x1d_r[i] = matrix_at(X_r, i, 0) }} + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c {{ w_f[i] = 1.0 / ((i + 1) as f32) }} + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 {body} for i in 0 to n_c {{ array_free_f32(Y_label_rows[i]) }} @@ -258,6 +374,11 @@ def chunk_file(index: int, entries: list[dict]) -> str: matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\\n", sink) return 0 }} ''' @@ -267,24 +388,35 @@ def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--per-file", type=int, default=20) ap.add_argument("--outdir", type=Path, default=OUTDIR) - ap.add_argument("--only", help="generate a single estimator, for bisecting a compile failure") + ap.add_argument("--only", help="comma separated estimator names, for bisecting a compile " + "failure or re-timing one group") + ap.add_argument("--prefix", default="bench_estimators", + help="output file prefix, so a subset written elsewhere cannot collide with " + "the committed files or with another process in the shared build directory") args = ap.parse_args() registry = json.loads(REGISTRY.read_text()) - runnable = [e for e in registry["entries"] if e["bucket"] == "runnable"] + # Simplified implementations are timed too, because compare_estimators.py + # shows their times while withholding a ratio. Leaving them out of the + # race would make the page quietly drop four rows. + timed = ("runnable", "shaped", "simplified") + runnable = [e for e in registry["entries"] if e["bucket"] in timed] if args.only: - runnable = [e for e in runnable if e["flow_estimator"] == args.only] - if not runnable: - raise SystemExit(f"{args.only} is not a runnable registry entry") + wanted = [n.strip() for n in args.only.split(",") if n.strip()] + runnable = [e for e in runnable if e["flow_estimator"] in wanted] + found = {e["flow_estimator"] for e in runnable} + missing = [n for n in wanted if n not in found] + if missing: + raise SystemExit(f"not timed registry entries: {', '.join(missing)}") args.outdir.mkdir(parents=True, exist_ok=True) - for stale in args.outdir.glob("bench_estimators_*.flow"): + for stale in args.outdir.glob(f"{args.prefix}_*.flow"): stale.unlink() written = [] for i in range(0, len(runnable), args.per_file): part = runnable[i : i + args.per_file] - path = args.outdir / f"bench_estimators_{i // args.per_file:02d}.flow" + path = args.outdir / f"{args.prefix}_{i // args.per_file:02d}.flow" path.write_text(chunk_file(i // args.per_file, part)) written.append((path.name, len(part))) diff --git a/benchmarks/generated/bench_estimators_00.flow b/benchmarks/generated/bench_estimators_00.flow index f67f874..fa7e37e 100644 --- a/benchmarks/generated/bench_estimators_00.flow +++ b/benchmarks/generated/bench_estimators_00.flow @@ -65,11 +65,22 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 # ---- adaboost_classifier (classification) ---- t0 = flow_now_ns() @@ -79,6 +90,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_adaboost_classifier: AdaBoostClassifier = adaboost_classifier_fit(X_c, y_c, 3, 10, 5, 42) @@ -89,6 +101,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_adaboost_classifier: ptr = adaboost_classifier_predict(fitted_adaboost_classifier, X_c) + sink = sink + o_adaboost_classifier[0] array_free_f32(o_adaboost_classifier) } t3 = flow_now_ns() @@ -104,6 +117,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_adaboost_regressor: AdaBoostRegressor = adaboost_regressor_fit(X_r, y_r, 10, 5, 42) @@ -114,6 +128,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_adaboost_regressor: ptr = adaboost_regressor_predict(fitted_adaboost_regressor, X_r) + sink = sink + o_adaboost_regressor[0] array_free_f32(o_adaboost_regressor) } t3 = flow_now_ns() @@ -121,6 +136,35 @@ function main() -> i32 { printf("ESTIMATOR|adaboost_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) + # ---- additive_chi2_sampler (classification, written out) ---- + t0 = flow_now_ns() + let probe_additive_chi2_sampler: AdditiveChi2Sampler = additive_chi2_sampler_fit(n_c, f_c, 2) + t1 = flow_now_ns() + additive_chi2_sampler_free(probe_additive_chi2_sampler) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_additive_chi2_sampler: AdditiveChi2Sampler = additive_chi2_sampler_fit(n_c, f_c, 2) + additive_chi2_sampler_free(m_additive_chi2_sampler) + } + t1 = flow_now_ns() + let fitted_additive_chi2_sampler: AdditiveChi2Sampler = additive_chi2_sampler_fit(n_c, f_c, 2) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_additive_chi2_sampler: Matrix = additive_chi2_sampler_transform(fitted_additive_chi2_sampler, X_c) + if o_additive_chi2_sampler.rows > 0 { + if o_additive_chi2_sampler.cols > 0 { sink = sink + o_additive_chi2_sampler.data[0] } + } + matrix_free(o_additive_chi2_sampler) + } + t3 = flow_now_ns() + additive_chi2_sampler_free(fitted_additive_chi2_sampler) + printf("ESTIMATOR|additive_chi2_sampler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + # ---- affinity_propagation (unsupervised) ---- t0 = flow_now_ns() let probe_affinity_propagation: AffinityPropagation = affinity_propagation_fit(X_c, 0.5, 100, 15) @@ -129,6 +173,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_affinity_propagation: AffinityPropagation = affinity_propagation_fit(X_c, 0.5, 100, 15) @@ -146,6 +191,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_agglomerative_clustering: AgglomerativeClustering = agglomerative_clustering_fit(X_c, 3) @@ -163,6 +209,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_ard_regression: ARDRegression = ard_regression_fit(X_r, y_r, 100, 0.0001) @@ -173,6 +220,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_ard_regression: ptr = ard_regression_predict(fitted_ard_regression, X_r) + sink = sink + o_ard_regression[0] array_free_f32(o_ard_regression) } t3 = flow_now_ns() @@ -188,6 +236,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bagging_classifier: BaggingClassifier = bagging_classifier_fit(X_c, y_c, 3, 10, 5, 42) @@ -198,6 +247,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bagging_classifier: ptr = bagging_classifier_predict(fitted_bagging_classifier, X_c) + sink = sink + o_bagging_classifier[0] array_free_f32(o_bagging_classifier) } t3 = flow_now_ns() @@ -213,6 +263,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bagging_regressor: BaggingRegressor = bagging_regressor_fit(X_r, y_r, 10, 5, 42) @@ -223,6 +274,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bagging_regressor: ptr = bagging_regressor_predict(fitted_bagging_regressor, X_r) + sink = sink + o_bagging_regressor[0] array_free_f32(o_bagging_regressor) } t3 = flow_now_ns() @@ -238,6 +290,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bayesian_gaussian_mixture: BayesianGaussianMixture = bayesian_gaussian_mixture_fit(X_c, 2, 100, 0.0001, 42) @@ -248,6 +301,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bayesian_gaussian_mixture: ptr = bayesian_gaussian_mixture_predict(fitted_bayesian_gaussian_mixture, X_c) + sink = sink + o_bayesian_gaussian_mixture[0] array_free_f32(o_bayesian_gaussian_mixture) } t3 = flow_now_ns() @@ -263,6 +317,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bayesian_ridge: BayesianRidge = bayesian_ridge_fit(X_r, y_r, 100, 0.0001) @@ -273,6 +328,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bayesian_ridge: ptr = bayesian_ridge_predict(fitted_bayesian_ridge, X_r) + sink = sink + o_bayesian_ridge[0] array_free_f32(o_bayesian_ridge) } t3 = flow_now_ns() @@ -288,6 +344,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bernoulli_nb: BernoulliNB = bernoulli_nb_fit(X_c, y_c, 3, 1.0) @@ -298,6 +355,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bernoulli_nb: ptr = bernoulli_nb_predict(fitted_bernoulli_nb, X_c) + sink = sink + o_bernoulli_nb[0] array_free_f32(o_bernoulli_nb) } t3 = flow_now_ns() @@ -313,6 +371,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bernoulli_rbm: BernoulliRBM = bernoulli_rbm_fit(X_c, 2, 0.1, 50, 32, 42) @@ -323,6 +382,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_bernoulli_rbm: Matrix = bernoulli_rbm_transform(fitted_bernoulli_rbm, X_c) + if o_bernoulli_rbm.rows > 0 { + if o_bernoulli_rbm.cols > 0 { sink = sink + o_bernoulli_rbm.data[0] } + } matrix_free(o_bernoulli_rbm) } t3 = flow_now_ns() @@ -338,6 +400,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_birch: Birch = birch_fit(X_c, 0.5, 50) @@ -355,6 +418,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_bisecting_kmeans: BisectingKMeans = bisecting_kmeans_fit(X_c, 3, 100, 42) @@ -372,6 +436,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_calibrated_classifier_cv: CalibratedClassifierCV = calibrated_classifier_cv_fit(X_c, y_c, 3, 3, 0) @@ -382,6 +447,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_calibrated_classifier_cv: ptr = calibrated_classifier_cv_predict(fitted_calibrated_classifier_cv, X_c) + sink = sink + o_calibrated_classifier_cv[0] array_free_f32(o_calibrated_classifier_cv) } t3 = flow_now_ns() @@ -398,6 +464,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_categorical_nb: CategoricalNB = categorical_nb_fit(X_c, y_c, 3, n_categories_categorical_nb, 1.0) @@ -408,6 +475,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_categorical_nb: ptr = categorical_nb_predict(fitted_categorical_nb, X_c) + sink = sink + o_categorical_nb[0] array_free_f32(o_categorical_nb) } t3 = flow_now_ns() @@ -423,6 +491,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_cca: CCA = cca_fit(X_r, Y_multi, 2, 100, 0.0001) @@ -433,6 +502,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_cca: Matrix = cca_transform(fitted_cca, X_r) + if o_cca.rows > 0 { + if o_cca.cols > 0 { sink = sink + o_cca.data[0] } + } matrix_free(o_cca) } t3 = flow_now_ns() @@ -448,6 +520,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_classifier_chain: ClassifierChain = classifier_chain_fit(X_c, Y_label_rows, n_c, f_c, 2, 0.01, 50) @@ -457,71 +530,64 @@ function main() -> i32 { printf("ESTIMATOR|classifier_chain|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- complement_nb (classification) ---- + # ---- column_transformer (classification, written out) ---- + let ct_cols: ptr = malloc((f_c as i64) * 4) as ptr + for i in 0 to f_c { ct_cols[i] = i } + let ct_obj: ColumnTransformer = column_transformer_init(1) + # 0 is TRANSFORMER_STANDARD_SCALER. The constant is written out + # because an export const is not visible through the umbrella import. + column_transformer_set_spec(ct_obj, 0, 0, ct_cols, f_c) t0 = flow_now_ns() - let probe_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) + let probe_column_transformer: ColumnTransformer = column_transformer_fit(ct_obj, X_c) t1 = flow_now_ns() - complement_nb_free(probe_complement_nb) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) - complement_nb_free(m_complement_nb) + let m_column_transformer: ColumnTransformer = column_transformer_fit(ct_obj, X_c) } t1 = flow_now_ns() - let fitted_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) + let fitted_column_transformer: ColumnTransformer = column_transformer_fit(ct_obj, X_c) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_complement_nb: ptr = complement_nb_predict(fitted_complement_nb, X_c) - array_free_f32(o_complement_nb) + let o_column_transformer: Matrix = column_transformer_transform(fitted_column_transformer, X_c) + if o_column_transformer.rows > 0 { + if o_column_transformer.cols > 0 { sink = sink + o_column_transformer.data[0] } + } + matrix_free(o_column_transformer) } t3 = flow_now_ns() - complement_nb_free(fitted_complement_nb) - printf("ESTIMATOR|complement_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - - # ---- dbscan (unsupervised) ---- - t0 = flow_now_ns() - let probe_dbscan: DBSCAN = dbscan_fit(X_c, 0.5, 5) - t1 = flow_now_ns() - dbscan_free(probe_dbscan) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_dbscan: DBSCAN = dbscan_fit(X_c, 0.5, 5) - dbscan_free(m_dbscan) - } - t1 = flow_now_ns() - printf("ESTIMATOR|dbscan|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + column_transformer_free(fitted_column_transformer) + printf("ESTIMATOR|column_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- decision_tree_classifier (classification) ---- + # ---- complement_nb (classification) ---- t0 = flow_now_ns() - let probe_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) + let probe_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) t1 = flow_now_ns() - decision_tree_classifier_free(probe_decision_tree_classifier) + complement_nb_free(probe_complement_nb) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) - decision_tree_classifier_free(m_decision_tree_classifier) + let m_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) + complement_nb_free(m_complement_nb) } t1 = flow_now_ns() - let fitted_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) + let fitted_complement_nb: ComplementNB = complement_nb_fit(X_c, y_c, 3, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_decision_tree_classifier: ptr = decision_tree_classifier_predict(fitted_decision_tree_classifier, X_c) - array_free_f32(o_decision_tree_classifier) + let o_complement_nb: ptr = complement_nb_predict(fitted_complement_nb, X_c) + sink = sink + o_complement_nb[0] + array_free_f32(o_complement_nb) } t3 = flow_now_ns() - decision_tree_classifier_free(fitted_decision_tree_classifier) - printf("ESTIMATOR|decision_tree_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + complement_nb_free(fitted_complement_nb) + printf("ESTIMATOR|complement_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -532,5 +598,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_01.flow b/benchmarks/generated/bench_estimators_01.flow index 4101be8..ca64230 100644 --- a/benchmarks/generated/bench_estimators_01.flow +++ b/benchmarks/generated/bench_estimators_01.flow @@ -65,11 +65,114 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 + + # ---- count_vectorizer (classification, written out) ---- + let count_vectorizer_docs: array = [ + "the quick brown fox jumps over the lazy dog", + "a lazy dog sleeps in the warm sun", + "quick brown foxes are rare in the city", + "the dog and the fox share a field", + "warm sun and a cold river run together", + "a field of brown grass in the sun", + "the city river runs past the old field", + "old dogs sleep through a quick storm", + "a storm over the city wakes the dog", + "foxes hunt in the cold river valley", + "the valley holds a warm field of grass", + "grass grows where the river meets the sun", + "a rare fox crosses the old stone bridge", + "the stone bridge over the cold river", + "dogs and foxes keep their distance here", + "here the field the river and the city meet" + ] + t0 = flow_now_ns() + let probe_count_vectorizer: CountVectorizer = count_vectorizer_fit(count_vectorizer_docs, 16, 50) + t1 = flow_now_ns() + count_vectorizer_free(probe_count_vectorizer) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_count_vectorizer: CountVectorizer = count_vectorizer_fit(count_vectorizer_docs, 16, 50) + count_vectorizer_free(m_count_vectorizer) + } + t1 = flow_now_ns() + let fitted_count_vectorizer: CountVectorizer = count_vectorizer_fit(count_vectorizer_docs, 16, 50) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_count_vectorizer: Matrix = count_vectorizer_transform(fitted_count_vectorizer, count_vectorizer_docs, 16) + if o_count_vectorizer.rows > 0 { + if o_count_vectorizer.cols > 0 { sink = sink + o_count_vectorizer.data[0] } + } + matrix_free(o_count_vectorizer) + } + t3 = flow_now_ns() + count_vectorizer_free(fitted_count_vectorizer) + printf("ESTIMATOR|count_vectorizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- dbscan (unsupervised) ---- + t0 = flow_now_ns() + let probe_dbscan: DBSCAN = dbscan_fit(X_c, 0.5, 5) + t1 = flow_now_ns() + dbscan_free(probe_dbscan) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_dbscan: DBSCAN = dbscan_fit(X_c, 0.5, 5) + dbscan_free(m_dbscan) + } + t1 = flow_now_ns() + printf("ESTIMATOR|dbscan|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + fflush(null) + + # ---- decision_tree_classifier (classification) ---- + t0 = flow_now_ns() + let probe_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) + t1 = flow_now_ns() + decision_tree_classifier_free(probe_decision_tree_classifier) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) + decision_tree_classifier_free(m_decision_tree_classifier) + } + t1 = flow_now_ns() + let fitted_decision_tree_classifier: DecisionTreeClassifier = decision_tree_classifier_fit(X_c, y_c, 3, 5, 0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_decision_tree_classifier: ptr = decision_tree_classifier_predict(fitted_decision_tree_classifier, X_c) + sink = sink + o_decision_tree_classifier[0] + array_free_f32(o_decision_tree_classifier) + } + t3 = flow_now_ns() + decision_tree_classifier_free(fitted_decision_tree_classifier) + printf("ESTIMATOR|decision_tree_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) # ---- decision_tree_regressor (regression) ---- t0 = flow_now_ns() @@ -79,6 +182,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_decision_tree_regressor: DecisionTreeRegressor = decision_tree_regressor_fit(X_r, y_r, 5, 0) @@ -89,6 +193,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_decision_tree_regressor: ptr = decision_tree_regressor_predict(fitted_decision_tree_regressor, X_r) + sink = sink + o_decision_tree_regressor[0] array_free_f32(o_decision_tree_regressor) } t3 = flow_now_ns() @@ -96,6 +201,49 @@ function main() -> i32 { printf("ESTIMATOR|decision_tree_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) + # ---- dict_vectorizer (classification, written out) ---- + let dv_counts: ptr = malloc((n_c as i64) * 4) as ptr + let dv_keys: ptr > = malloc((n_c as i64) * 8) as ptr > + let dv_vals: ptr > = malloc((n_c as i64) * 8) as ptr > + for i in 0 to n_c { + dv_counts[i] = f_c + let dv_kk: ptr = malloc((f_c as i64) * 4) as ptr + let dv_vv: ptr = array_new_f32(f_c) + for j in 0 to f_c { + dv_kk[j] = j + dv_vv[j] = matrix_at(X_c, i, j) + } + dv_keys[i] = dv_kk + dv_vals[i] = dv_vv + } + t0 = flow_now_ns() + let probe_dict_vectorizer: DictVectorizer = dict_vectorizer_fit(dv_keys, dv_vals, n_c, dv_counts) + t1 = flow_now_ns() + dict_vectorizer_free(probe_dict_vectorizer) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_dict_vectorizer: DictVectorizer = dict_vectorizer_fit(dv_keys, dv_vals, n_c, dv_counts) + dict_vectorizer_free(m_dict_vectorizer) + } + t1 = flow_now_ns() + let fitted_dict_vectorizer: DictVectorizer = dict_vectorizer_fit(dv_keys, dv_vals, n_c, dv_counts) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_dict_vectorizer: Matrix = dict_vectorizer_transform(fitted_dict_vectorizer, dv_keys, dv_vals, n_c, dv_counts) + if o_dict_vectorizer.rows > 0 { + if o_dict_vectorizer.cols > 0 { sink = sink + o_dict_vectorizer.data[0] } + } + matrix_free(o_dict_vectorizer) + } + t3 = flow_now_ns() + dict_vectorizer_free(fitted_dict_vectorizer) + printf("ESTIMATOR|dict_vectorizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + # ---- dictionary_learning (unsupervised) ---- t0 = flow_now_ns() let probe_dictionary_learning: DictionaryLearning = dictionary_learning_fit(X_c, 2, 1.0, 100, 0.0001, 42) @@ -104,6 +252,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_dictionary_learning: DictionaryLearning = dictionary_learning_fit(X_c, 2, 1.0, 100, 0.0001, 42) @@ -114,6 +263,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_dictionary_learning: Matrix = dictionary_learning_transform(fitted_dictionary_learning, X_c) + if o_dictionary_learning.rows > 0 { + if o_dictionary_learning.cols > 0 { sink = sink + o_dictionary_learning.data[0] } + } matrix_free(o_dictionary_learning) } t3 = flow_now_ns() @@ -129,6 +281,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_discriminant_lda: LinearDiscriminantAnalysis = discriminant_lda_fit(X_c, y_c, 3) @@ -139,6 +292,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_discriminant_lda: ptr = discriminant_lda_predict(fitted_discriminant_lda, X_c) + sink = sink + o_discriminant_lda[0] array_free_f32(o_discriminant_lda) } t3 = flow_now_ns() @@ -146,6 +300,60 @@ function main() -> i32 { printf("ESTIMATOR|discriminant_lda|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) + # ---- dummy_classifier (classification, written out) ---- + t0 = flow_now_ns() + let probe_dummy_classifier: DummyClassifier = dummy_classifier_fit(y_c, n_c, 3, 0, 0.0, 42) + t1 = flow_now_ns() + dummy_classifier_free(probe_dummy_classifier) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_dummy_classifier: DummyClassifier = dummy_classifier_fit(y_c, n_c, 3, 0, 0.0, 42) + dummy_classifier_free(m_dummy_classifier) + } + t1 = flow_now_ns() + let fitted_dummy_classifier: DummyClassifier = dummy_classifier_fit(y_c, n_c, 3, 0, 0.0, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_dummy_classifier: ptr = dummy_classifier_predict(fitted_dummy_classifier, n_c) + sink = sink + o_dummy_classifier[0] + array_free_f32(o_dummy_classifier) + } + t3 = flow_now_ns() + dummy_classifier_free(fitted_dummy_classifier) + printf("ESTIMATOR|dummy_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- dummy_regressor (regression, written out) ---- + t0 = flow_now_ns() + let probe_dummy_regressor: DummyRegressor = dummy_regressor_fit(y_r, n_r, 0, 0.0) + t1 = flow_now_ns() + dummy_regressor_free(probe_dummy_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_dummy_regressor: DummyRegressor = dummy_regressor_fit(y_r, n_r, 0, 0.0) + dummy_regressor_free(m_dummy_regressor) + } + t1 = flow_now_ns() + let fitted_dummy_regressor: DummyRegressor = dummy_regressor_fit(y_r, n_r, 0, 0.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_dummy_regressor: ptr = dummy_regressor_predict(fitted_dummy_regressor, n_r) + sink = sink + o_dummy_regressor[0] + array_free_f32(o_dummy_regressor) + } + t3 = flow_now_ns() + dummy_regressor_free(fitted_dummy_regressor) + printf("ESTIMATOR|dummy_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + # ---- elastic_net_cv (regression) ---- t0 = flow_now_ns() let probe_elastic_net_cv: ElasticNetCV = elastic_net_cv_fit(X_r, y_r, 10) @@ -154,6 +362,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_elastic_net_cv: ElasticNetCV = elastic_net_cv_fit(X_r, y_r, 10) @@ -164,6 +373,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_elastic_net_cv: ptr = elastic_net_cv_predict(fitted_elastic_net_cv, X_r) + sink = sink + o_elastic_net_cv[0] array_free_f32(o_elastic_net_cv) } t3 = flow_now_ns() @@ -179,6 +389,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_elastic_net: ElasticNet = elastic_net_fit(X_r, y_r, 1.0, 0.5, 50, 0.01) @@ -189,6 +400,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_elastic_net: ptr = elastic_net_predict(fitted_elastic_net, X_r) + sink = sink + o_elastic_net[0] array_free_f32(o_elastic_net) } t3 = flow_now_ns() @@ -204,6 +416,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_elliptic_envelope: EllipticEnvelope = elliptic_envelope_fit(X_c, 0.1, 10) @@ -214,6 +427,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_elliptic_envelope: ptr = elliptic_envelope_predict(fitted_elliptic_envelope, X_c) + sink = sink + o_elliptic_envelope[0] array_free_f32(o_elliptic_envelope) } t3 = flow_now_ns() @@ -229,6 +443,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_empirical_covariance: EmpiricalCovariance = empirical_covariance_fit(X_c) @@ -246,6 +461,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_extra_tree_classifier: ExtraTreeClassifier = extra_tree_classifier_fit(X_c, y_c, n_c, f_c, 3, 5, 42) @@ -256,6 +472,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_extra_tree_classifier: ptr = extra_tree_classifier_predict(fitted_extra_tree_classifier, X_c) + sink = sink + o_extra_tree_classifier[0] array_free_f32(o_extra_tree_classifier) } t3 = flow_now_ns() @@ -271,6 +488,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_extra_tree_regressor: ExtraTreeRegressor = extra_tree_regressor_fit(X_r, y_r, n_r, f_r, 5, 42) @@ -281,6 +499,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_extra_tree_regressor: ptr = extra_tree_regressor_predict(fitted_extra_tree_regressor, X_r) + sink = sink + o_extra_tree_regressor[0] array_free_f32(o_extra_tree_regressor) } t3 = flow_now_ns() @@ -296,6 +515,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_extra_trees_classifier: ExtraTreesClassifier = extra_trees_classifier_fit(X_c, y_c, 3, 10, 5, 42) @@ -306,6 +526,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_extra_trees_classifier: ptr = extra_trees_classifier_predict(fitted_extra_trees_classifier, X_c) + sink = sink + o_extra_trees_classifier[0] array_free_f32(o_extra_trees_classifier) } t3 = flow_now_ns() @@ -321,6 +542,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_extra_trees_regressor: ExtraTreesRegressor = extra_trees_regressor_fit(X_r, y_r, 10, 5, 42) @@ -331,6 +553,7 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_extra_trees_regressor: ptr = extra_trees_regressor_predict(fitted_extra_trees_regressor, X_r) + sink = sink + o_extra_trees_regressor[0] array_free_f32(o_extra_trees_regressor) } t3 = flow_now_ns() @@ -346,6 +569,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_factor_analysis: FactorAnalysis = factor_analysis_fit(X_c, 2, 100, 0.0001) @@ -356,6 +580,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_factor_analysis: Matrix = factor_analysis_transform(fitted_factor_analysis, X_c) + if o_factor_analysis.rows > 0 { + if o_factor_analysis.cols > 0 { sink = sink + o_factor_analysis.data[0] } + } matrix_free(o_factor_analysis) } t3 = flow_now_ns() @@ -371,6 +598,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_fast_ica: FastICA = fast_ica_fit(X_c, 2, 100, 0.0001, 42) @@ -381,6 +609,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_fast_ica: Matrix = fast_ica_transform(fitted_fast_ica, X_c) + if o_fast_ica.rows > 0 { + if o_fast_ica.cols > 0 { sink = sink + o_fast_ica.data[0] } + } matrix_free(o_fast_ica) } t3 = flow_now_ns() @@ -396,6 +627,7 @@ function main() -> i32 { reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { let m_feature_agglomeration: FeatureAgglomeration = feature_agglomeration_fit(X_c, 3) @@ -406,6 +638,9 @@ function main() -> i32 { t2 = flow_now_ns() for rep2 in 0 to reps { let o_feature_agglomeration: Matrix = feature_agglomeration_transform(fitted_feature_agglomeration, X_c) + if o_feature_agglomeration.rows > 0 { + if o_feature_agglomeration.cols > 0 { sink = sink + o_feature_agglomeration.data[0] } + } matrix_free(o_feature_agglomeration) } t3 = flow_now_ns() @@ -413,148 +648,6 @@ function main() -> i32 { printf("ESTIMATOR|feature_agglomeration|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- gamma_regressor (regression) ---- - t0 = flow_now_ns() - let probe_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) - t1 = flow_now_ns() - gamma_regressor_free(probe_gamma_regressor) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) - gamma_regressor_free(m_gamma_regressor) - } - t1 = flow_now_ns() - let fitted_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_gamma_regressor: ptr = gamma_regressor_predict(fitted_gamma_regressor, X_r) - array_free_f32(o_gamma_regressor) - } - t3 = flow_now_ns() - gamma_regressor_free(fitted_gamma_regressor) - printf("ESTIMATOR|gamma_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - - # ---- gaussian_mixture (unsupervised) ---- - t0 = flow_now_ns() - let probe_gaussian_mixture: GaussianMixture = gaussian_mixture_fit(X_c, 2, 100, 0.0001, 42) - t1 = flow_now_ns() - gaussian_mixture_free(probe_gaussian_mixture) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gaussian_mixture: GaussianMixture = gaussian_mixture_fit(X_c, 2, 100, 0.0001, 42) - gaussian_mixture_free(m_gaussian_mixture) - } - t1 = flow_now_ns() - printf("ESTIMATOR|gaussian_mixture|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) - fflush(null) - - # ---- gaussian_nb (classification) ---- - t0 = flow_now_ns() - let probe_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) - t1 = flow_now_ns() - gaussian_nb_free(probe_gaussian_nb) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) - gaussian_nb_free(m_gaussian_nb) - } - t1 = flow_now_ns() - let fitted_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_gaussian_nb: ptr = gaussian_nb_predict(fitted_gaussian_nb, X_c) - array_free_f32(o_gaussian_nb) - } - t3 = flow_now_ns() - gaussian_nb_free(fitted_gaussian_nb) - printf("ESTIMATOR|gaussian_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - - # ---- gaussian_process_classifier (regression) ---- - t0 = flow_now_ns() - let probe_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) - t1 = flow_now_ns() - gaussian_process_classifier_free(probe_gaussian_process_classifier) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) - gaussian_process_classifier_free(m_gaussian_process_classifier) - } - t1 = flow_now_ns() - let fitted_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_gaussian_process_classifier: ptr = gaussian_process_classifier_predict(fitted_gaussian_process_classifier, X_r) - array_free_f32(o_gaussian_process_classifier) - } - t3 = flow_now_ns() - gaussian_process_classifier_free(fitted_gaussian_process_classifier) - printf("ESTIMATOR|gaussian_process_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - - # ---- gaussian_process_regressor (regression) ---- - t0 = flow_now_ns() - let probe_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) - t1 = flow_now_ns() - gaussian_process_regressor_free(probe_gaussian_process_regressor) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) - gaussian_process_regressor_free(m_gaussian_process_regressor) - } - t1 = flow_now_ns() - let fitted_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_gaussian_process_regressor: ptr = gaussian_process_regressor_predict(fitted_gaussian_process_regressor, X_r) - array_free_f32(o_gaussian_process_regressor) - } - t3 = flow_now_ns() - gaussian_process_regressor_free(fitted_gaussian_process_regressor) - printf("ESTIMATOR|gaussian_process_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - - # ---- gradient_boosting_classifier (classification) ---- - t0 = flow_now_ns() - let probe_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) - t1 = flow_now_ns() - gradient_boosting_classifier_free(probe_gradient_boosting_classifier) - reps = 1 - if (t1 - t0) < 200000 { reps = 200 } - elif (t1 - t0) < 2000000 { reps = 20 } - t0 = flow_now_ns() - for rep in 0 to reps { - let m_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) - gradient_boosting_classifier_free(m_gradient_boosting_classifier) - } - t1 = flow_now_ns() - let fitted_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_gradient_boosting_classifier: ptr = gradient_boosting_classifier_predict(fitted_gradient_boosting_classifier, X_c) - array_free_f32(o_gradient_boosting_classifier) - } - t3 = flow_now_ns() - gradient_boosting_classifier_free(fitted_gradient_boosting_classifier) - printf("ESTIMATOR|gradient_boosting_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) - fflush(null) - for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } free(Y_label_rows as ptr) matrix_free(Y_labels) @@ -563,5 +656,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_02.flow b/benchmarks/generated/bench_estimators_02.flow index 79de920..03364b2 100644 --- a/benchmarks/generated/bench_estimators_02.flow +++ b/benchmarks/generated/bench_estimators_02.flow @@ -65,462 +65,527 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- gradient_boosting_regressor (regression) ---- + # ---- feature_union (classification, written out) ---- + let fu_obj: FeatureUnion = feature_union_init(2) + # 0 is FU_TRANSFORMER_STANDARD_SCALER and 2 is FU_TRANSFORMER_PASSTHROUGH, + # written out for the reason the column transformer above gives. + feature_union_set_transformer(fu_obj, 0, 0, 0) + feature_union_set_transformer(fu_obj, 1, 2, 0) t0 = flow_now_ns() - let probe_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) + let probe_feature_union: FeatureUnion = feature_union_fit(fu_obj, X_c) t1 = flow_now_ns() - gradient_boosting_regressor_free(probe_gradient_boosting_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) - gradient_boosting_regressor_free(m_gradient_boosting_regressor) + let m_feature_union: FeatureUnion = feature_union_fit(fu_obj, X_c) } t1 = flow_now_ns() - let fitted_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) + let fitted_feature_union: FeatureUnion = feature_union_fit(fu_obj, X_c) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_gradient_boosting_regressor: ptr = gradient_boosting_regressor_predict(fitted_gradient_boosting_regressor, X_r) - array_free_f32(o_gradient_boosting_regressor) + let o_feature_union: Matrix = feature_union_transform(fitted_feature_union, X_c) + if o_feature_union.rows > 0 { + if o_feature_union.cols > 0 { sink = sink + o_feature_union.data[0] } + } + matrix_free(o_feature_union) } t3 = flow_now_ns() - gradient_boosting_regressor_free(fitted_gradient_boosting_regressor) - printf("ESTIMATOR|gradient_boosting_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + feature_union_free(fitted_feature_union) + printf("ESTIMATOR|feature_union|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- graphical_lasso (unsupervised) ---- + # ---- gamma_regressor (regression) ---- t0 = flow_now_ns() - let probe_graphical_lasso: GraphicalLasso = graphical_lasso_fit(X_c, 1.0, 100, 0.0001) + let probe_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) t1 = flow_now_ns() - graphical_lasso_free(probe_graphical_lasso) + gamma_regressor_free(probe_gamma_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_graphical_lasso: GraphicalLasso = graphical_lasso_fit(X_c, 1.0, 100, 0.0001) - graphical_lasso_free(m_graphical_lasso) + let m_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) + gamma_regressor_free(m_gamma_regressor) } t1 = flow_now_ns() - printf("ESTIMATOR|graphical_lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_gamma_regressor: GammaRegressor = gamma_regressor_fit(X_r, y_r, 1.0, 100, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_gamma_regressor: ptr = gamma_regressor_predict(fitted_gamma_regressor, X_r) + sink = sink + o_gamma_regressor[0] + array_free_f32(o_gamma_regressor) + } + t3 = flow_now_ns() + gamma_regressor_free(fitted_gamma_regressor) + printf("ESTIMATOR|gamma_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- hdbscan (unsupervised) ---- + # ---- gaussian_mixture (unsupervised) ---- t0 = flow_now_ns() - let probe_hdbscan: HDBSCAN = hdbscan_fit(X_c, 5, 5) + let probe_gaussian_mixture: GaussianMixture = gaussian_mixture_fit(X_c, 2, 100, 0.0001, 42) t1 = flow_now_ns() - hdbscan_free(probe_hdbscan) + gaussian_mixture_free(probe_gaussian_mixture) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_hdbscan: HDBSCAN = hdbscan_fit(X_c, 5, 5) - hdbscan_free(m_hdbscan) + let m_gaussian_mixture: GaussianMixture = gaussian_mixture_fit(X_c, 2, 100, 0.0001, 42) + gaussian_mixture_free(m_gaussian_mixture) } t1 = flow_now_ns() - printf("ESTIMATOR|hdbscan|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + printf("ESTIMATOR|gaussian_mixture|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- hist_gradient_boosting_classifier (classification) ---- + # ---- gaussian_nb (classification) ---- t0 = flow_now_ns() - let probe_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) + let probe_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) t1 = flow_now_ns() - hist_gradient_boosting_classifier_free(probe_hist_gradient_boosting_classifier) + gaussian_nb_free(probe_gaussian_nb) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) - hist_gradient_boosting_classifier_free(m_hist_gradient_boosting_classifier) + let m_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) + gaussian_nb_free(m_gaussian_nb) } t1 = flow_now_ns() - let fitted_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) + let fitted_gaussian_nb: GaussianNB = gaussian_nb_fit(X_c, y_c, 3, 0.000000001) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_hist_gradient_boosting_classifier: ptr = hist_gradient_boosting_classifier_predict(fitted_hist_gradient_boosting_classifier, X_c) - array_free_f32(o_hist_gradient_boosting_classifier) + let o_gaussian_nb: ptr = gaussian_nb_predict(fitted_gaussian_nb, X_c) + sink = sink + o_gaussian_nb[0] + array_free_f32(o_gaussian_nb) } t3 = flow_now_ns() - hist_gradient_boosting_classifier_free(fitted_hist_gradient_boosting_classifier) - printf("ESTIMATOR|hist_gradient_boosting_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + gaussian_nb_free(fitted_gaussian_nb) + printf("ESTIMATOR|gaussian_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- hist_gradient_boosting_regressor (regression) ---- + # ---- gaussian_process_classifier (regression) ---- t0 = flow_now_ns() - let probe_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) + let probe_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) t1 = flow_now_ns() - hist_gradient_boosting_regressor_free(probe_hist_gradient_boosting_regressor) + gaussian_process_classifier_free(probe_gaussian_process_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) - hist_gradient_boosting_regressor_free(m_hist_gradient_boosting_regressor) + let m_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) + gaussian_process_classifier_free(m_gaussian_process_classifier) } t1 = flow_now_ns() - let fitted_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) + let fitted_gaussian_process_classifier: GaussianProcessClassifier = gaussian_process_classifier_fit(X_r, y_r, 0.1, 100) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_hist_gradient_boosting_regressor: ptr = hist_gradient_boosting_regressor_predict(fitted_hist_gradient_boosting_regressor, X_r) - array_free_f32(o_hist_gradient_boosting_regressor) + let o_gaussian_process_classifier: ptr = gaussian_process_classifier_predict(fitted_gaussian_process_classifier, X_r) + sink = sink + o_gaussian_process_classifier[0] + array_free_f32(o_gaussian_process_classifier) } t3 = flow_now_ns() - hist_gradient_boosting_regressor_free(fitted_hist_gradient_boosting_regressor) - printf("ESTIMATOR|hist_gradient_boosting_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + gaussian_process_classifier_free(fitted_gaussian_process_classifier) + printf("ESTIMATOR|gaussian_process_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- huber_regressor (regression) ---- + # ---- gaussian_process_regressor (regression) ---- t0 = flow_now_ns() - let probe_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) + let probe_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) t1 = flow_now_ns() - huber_regressor_free(probe_huber_regressor) + gaussian_process_regressor_free(probe_gaussian_process_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) - huber_regressor_free(m_huber_regressor) + let m_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) + gaussian_process_regressor_free(m_gaussian_process_regressor) } t1 = flow_now_ns() - let fitted_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) + let fitted_gaussian_process_regressor: GaussianProcessRegressor = gaussian_process_regressor_fit(X_r, y_r, 1.0, 1.0, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_huber_regressor: ptr = huber_regressor_predict(fitted_huber_regressor, X_r) - array_free_f32(o_huber_regressor) + let o_gaussian_process_regressor: ptr = gaussian_process_regressor_predict(fitted_gaussian_process_regressor, X_r) + sink = sink + o_gaussian_process_regressor[0] + array_free_f32(o_gaussian_process_regressor) } t3 = flow_now_ns() - huber_regressor_free(fitted_huber_regressor) - printf("ESTIMATOR|huber_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + gaussian_process_regressor_free(fitted_gaussian_process_regressor) + printf("ESTIMATOR|gaussian_process_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- isolation_forest (unsupervised) ---- + # ---- gaussian_random_projection (classification, written out) ---- t0 = flow_now_ns() - let probe_isolation_forest: IsolationForest = isolation_forest_fit(X_c, 10, 5, 42) + let probe_gaussian_random_projection: GaussianRandomProjection = gaussian_random_projection_fit(f_c, 2, 42) t1 = flow_now_ns() - isolation_forest_free(probe_isolation_forest) + gaussian_random_projection_free(probe_gaussian_random_projection) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_isolation_forest: IsolationForest = isolation_forest_fit(X_c, 10, 5, 42) - isolation_forest_free(m_isolation_forest) + let m_gaussian_random_projection: GaussianRandomProjection = gaussian_random_projection_fit(f_c, 2, 42) + gaussian_random_projection_free(m_gaussian_random_projection) } t1 = flow_now_ns() - printf("ESTIMATOR|isolation_forest|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_gaussian_random_projection: GaussianRandomProjection = gaussian_random_projection_fit(f_c, 2, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_gaussian_random_projection: Matrix = gaussian_random_projection_transform(fitted_gaussian_random_projection, X_c) + if o_gaussian_random_projection.rows > 0 { + if o_gaussian_random_projection.cols > 0 { sink = sink + o_gaussian_random_projection.data[0] } + } + matrix_free(o_gaussian_random_projection) + } + t3 = flow_now_ns() + gaussian_random_projection_free(fitted_gaussian_random_projection) + printf("ESTIMATOR|gaussian_random_projection|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- isomap (unsupervised) ---- + # ---- gradient_boosting_classifier (classification) ---- t0 = flow_now_ns() - let probe_isomap: Isomap = isomap_fit(X_c, 2, 5) + let probe_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) t1 = flow_now_ns() - isomap_free(probe_isomap) + gradient_boosting_classifier_free(probe_gradient_boosting_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_isomap: Isomap = isomap_fit(X_c, 2, 5) - isomap_free(m_isomap) + let m_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) + gradient_boosting_classifier_free(m_gradient_boosting_classifier) } t1 = flow_now_ns() - printf("ESTIMATOR|isomap|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_gradient_boosting_classifier: GradientBoostingClassifier = gradient_boosting_classifier_fit(X_c, y_c, 3, 10, 0.1, 5, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_gradient_boosting_classifier: ptr = gradient_boosting_classifier_predict(fitted_gradient_boosting_classifier, X_c) + sink = sink + o_gradient_boosting_classifier[0] + array_free_f32(o_gradient_boosting_classifier) + } + t3 = flow_now_ns() + gradient_boosting_classifier_free(fitted_gradient_boosting_classifier) + printf("ESTIMATOR|gradient_boosting_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- iterative_imputer (unsupervised) ---- + # ---- gradient_boosting_regressor (regression) ---- t0 = flow_now_ns() - let probe_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) + let probe_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) t1 = flow_now_ns() - iterative_imputer_free(probe_iterative_imputer) + gradient_boosting_regressor_free(probe_gradient_boosting_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) - iterative_imputer_free(m_iterative_imputer) + let m_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) + gradient_boosting_regressor_free(m_gradient_boosting_regressor) } t1 = flow_now_ns() - let fitted_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) + let fitted_gradient_boosting_regressor: GradientBoostingRegressor = gradient_boosting_regressor_fit(X_r, y_r, 10, 0.1, 5) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_iterative_imputer: Matrix = iterative_imputer_transform(fitted_iterative_imputer, X_c) - matrix_free(o_iterative_imputer) + let o_gradient_boosting_regressor: ptr = gradient_boosting_regressor_predict(fitted_gradient_boosting_regressor, X_r) + sink = sink + o_gradient_boosting_regressor[0] + array_free_f32(o_gradient_boosting_regressor) } t3 = flow_now_ns() - iterative_imputer_free(fitted_iterative_imputer) - printf("ESTIMATOR|iterative_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + gradient_boosting_regressor_free(fitted_gradient_boosting_regressor) + printf("ESTIMATOR|gradient_boosting_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- kbins_discretizer (unsupervised) ---- + # ---- graphical_lasso (unsupervised) ---- t0 = flow_now_ns() - let probe_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) + let probe_graphical_lasso: GraphicalLasso = graphical_lasso_fit(X_c, 1.0, 100, 0.0001) t1 = flow_now_ns() - kbins_discretizer_free(probe_kbins_discretizer) + graphical_lasso_free(probe_graphical_lasso) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) - kbins_discretizer_free(m_kbins_discretizer) + let m_graphical_lasso: GraphicalLasso = graphical_lasso_fit(X_c, 1.0, 100, 0.0001) + graphical_lasso_free(m_graphical_lasso) } t1 = flow_now_ns() - let fitted_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_kbins_discretizer: Matrix = kbins_discretizer_transform(fitted_kbins_discretizer, X_c) - matrix_free(o_kbins_discretizer) - } - t3 = flow_now_ns() - kbins_discretizer_free(fitted_kbins_discretizer) - printf("ESTIMATOR|kbins_discretizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|graphical_lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- kernel_density (unsupervised) ---- + # ---- hdbscan (unsupervised) ---- t0 = flow_now_ns() - let probe_kernel_density: KernelDensity = kernel_density_fit(X_c, 1.0, 0) + let probe_hdbscan: HDBSCAN = hdbscan_fit(X_c, 5, 5) t1 = flow_now_ns() - kernel_density_free(probe_kernel_density) + hdbscan_free(probe_hdbscan) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kernel_density: KernelDensity = kernel_density_fit(X_c, 1.0, 0) - kernel_density_free(m_kernel_density) + let m_hdbscan: HDBSCAN = hdbscan_fit(X_c, 5, 5) + hdbscan_free(m_hdbscan) } t1 = flow_now_ns() - printf("ESTIMATOR|kernel_density|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + printf("ESTIMATOR|hdbscan|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- kernel_pca (unsupervised) ---- + # ---- hist_gradient_boosting_classifier (classification) ---- t0 = flow_now_ns() - let probe_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) + let probe_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) t1 = flow_now_ns() - kernel_pca_free(probe_kernel_pca) + hist_gradient_boosting_classifier_free(probe_hist_gradient_boosting_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) - kernel_pca_free(m_kernel_pca) + let m_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) + hist_gradient_boosting_classifier_free(m_hist_gradient_boosting_classifier) } t1 = flow_now_ns() - let fitted_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) + let fitted_hist_gradient_boosting_classifier: HistGradientBoostingClassifier = hist_gradient_boosting_classifier_fit(X_c, y_c, n_c, 3, 10, 0.1, 5) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_kernel_pca: Matrix = kernel_pca_transform(fitted_kernel_pca, X_c) - matrix_free(o_kernel_pca) + let o_hist_gradient_boosting_classifier: ptr = hist_gradient_boosting_classifier_predict(fitted_hist_gradient_boosting_classifier, X_c) + sink = sink + o_hist_gradient_boosting_classifier[0] + array_free_f32(o_hist_gradient_boosting_classifier) } t3 = flow_now_ns() - kernel_pca_free(fitted_kernel_pca) - printf("ESTIMATOR|kernel_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + hist_gradient_boosting_classifier_free(fitted_hist_gradient_boosting_classifier) + printf("ESTIMATOR|hist_gradient_boosting_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- kernel_ridge (regression) ---- + # ---- hist_gradient_boosting_regressor (regression) ---- t0 = flow_now_ns() - let probe_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) + let probe_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) t1 = flow_now_ns() - kernel_ridge_free(probe_kernel_ridge) + hist_gradient_boosting_regressor_free(probe_hist_gradient_boosting_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) - kernel_ridge_free(m_kernel_ridge) + let m_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) + hist_gradient_boosting_regressor_free(m_hist_gradient_boosting_regressor) } t1 = flow_now_ns() - let fitted_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) + let fitted_hist_gradient_boosting_regressor: HistGradientBoostingRegressor = hist_gradient_boosting_regressor_fit(X_r, y_r, n_r, 10, 0.1, 5) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_kernel_ridge: ptr = kernel_ridge_predict(fitted_kernel_ridge, X_r) - array_free_f32(o_kernel_ridge) + let o_hist_gradient_boosting_regressor: ptr = hist_gradient_boosting_regressor_predict(fitted_hist_gradient_boosting_regressor, X_r) + sink = sink + o_hist_gradient_boosting_regressor[0] + array_free_f32(o_hist_gradient_boosting_regressor) } t3 = flow_now_ns() - kernel_ridge_free(fitted_kernel_ridge) - printf("ESTIMATOR|kernel_ridge|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + hist_gradient_boosting_regressor_free(fitted_hist_gradient_boosting_regressor) + printf("ESTIMATOR|hist_gradient_boosting_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- kernel_svc (classification) ---- + # ---- huber_regressor (regression) ---- t0 = flow_now_ns() - let probe_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) + let probe_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) t1 = flow_now_ns() - kernel_svc_free(probe_kernel_svc) + huber_regressor_free(probe_huber_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) - kernel_svc_free(m_kernel_svc) + let m_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) + huber_regressor_free(m_huber_regressor) } t1 = flow_now_ns() - let fitted_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) + let fitted_huber_regressor: HuberRegressor = huber_regressor_fit(X_r, y_r, 0.1, 100, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_kernel_svc: ptr = kernel_svc_predict(fitted_kernel_svc, X_c) - array_free_f32(o_kernel_svc) + let o_huber_regressor: ptr = huber_regressor_predict(fitted_huber_regressor, X_r) + sink = sink + o_huber_regressor[0] + array_free_f32(o_huber_regressor) } t3 = flow_now_ns() - kernel_svc_free(fitted_kernel_svc) - printf("ESTIMATOR|kernel_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + huber_regressor_free(fitted_huber_regressor) + printf("ESTIMATOR|huber_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- kernel_svc_multi (classification) ---- + # ---- isolation_forest (unsupervised) ---- t0 = flow_now_ns() - let probe_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) + let probe_isolation_forest: IsolationForest = isolation_forest_fit(X_c, 10, 5, 42) t1 = flow_now_ns() - kernel_svc_multi_free(probe_kernel_svc_multi) + isolation_forest_free(probe_isolation_forest) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) - kernel_svc_multi_free(m_kernel_svc_multi) + let m_isolation_forest: IsolationForest = isolation_forest_fit(X_c, 10, 5, 42) + isolation_forest_free(m_isolation_forest) } t1 = flow_now_ns() - let fitted_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_kernel_svc_multi: ptr = kernel_svc_multi_predict(fitted_kernel_svc_multi, X_c) - array_free_f32(o_kernel_svc_multi) - } - t3 = flow_now_ns() - kernel_svc_multi_free(fitted_kernel_svc_multi) - printf("ESTIMATOR|kernel_svc_multi|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|isolation_forest|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- kmeans (unsupervised) ---- + # ---- isomap (unsupervised) ---- t0 = flow_now_ns() - let probe_kmeans: KMeans = kmeans_fit(X_c, 3, 100, 0.0001, 42) + let probe_isomap: Isomap = isomap_fit(X_c, 2, 5) t1 = flow_now_ns() - kmeans_free(probe_kmeans) + isomap_free(probe_isomap) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kmeans: KMeans = kmeans_fit(X_c, 3, 100, 0.0001, 42) - kmeans_free(m_kmeans) + let m_isomap: Isomap = isomap_fit(X_c, 2, 5) + isomap_free(m_isomap) } t1 = flow_now_ns() - printf("ESTIMATOR|kmeans|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + printf("ESTIMATOR|isomap|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- kneighbors_transformer (unsupervised) ---- + # ---- isotonic (regression, written out) ---- t0 = flow_now_ns() - let probe_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) + let probe_isotonic: IsotonicRegression = isotonic_fit(x1d_r, y_r, n_r, true) t1 = flow_now_ns() - kneighbors_transformer_free(probe_kneighbors_transformer) + isotonic_free(probe_isotonic) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) - kneighbors_transformer_free(m_kneighbors_transformer) + let m_isotonic: IsotonicRegression = isotonic_fit(x1d_r, y_r, n_r, true) + isotonic_free(m_isotonic) } t1 = flow_now_ns() - let fitted_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) + let fitted_isotonic: IsotonicRegression = isotonic_fit(x1d_r, y_r, n_r, true) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_kneighbors_transformer: Matrix = kneighbors_transformer_transform(fitted_kneighbors_transformer, X_c) - matrix_free(o_kneighbors_transformer) + let o_isotonic: ptr = isotonic_transform(fitted_isotonic, x1d_r, n_r) + sink = sink + o_isotonic[0] + array_free_f32(o_isotonic) } t3 = flow_now_ns() - kneighbors_transformer_free(fitted_kneighbors_transformer) - printf("ESTIMATOR|kneighbors_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + isotonic_free(fitted_isotonic) + printf("ESTIMATOR|isotonic|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- knn_classifier (classification) ---- + # ---- iterative_imputer (unsupervised) ---- t0 = flow_now_ns() - let probe_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) + let probe_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) t1 = flow_now_ns() - knn_classifier_free(probe_knn_classifier) + iterative_imputer_free(probe_iterative_imputer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) - knn_classifier_free(m_knn_classifier) + let m_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) + iterative_imputer_free(m_iterative_imputer) } t1 = flow_now_ns() - let fitted_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) + let fitted_iterative_imputer: IterativeImputer = iterative_imputer_fit(X_c, 100, 0.0001, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_knn_classifier: Matrix = knn_classifier_predict(fitted_knn_classifier, X_c) - matrix_free(o_knn_classifier) + let o_iterative_imputer: Matrix = iterative_imputer_transform(fitted_iterative_imputer, X_c) + if o_iterative_imputer.rows > 0 { + if o_iterative_imputer.cols > 0 { sink = sink + o_iterative_imputer.data[0] } + } + matrix_free(o_iterative_imputer) } t3 = flow_now_ns() - knn_classifier_free(fitted_knn_classifier) - printf("ESTIMATOR|knn_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + iterative_imputer_free(fitted_iterative_imputer) + printf("ESTIMATOR|iterative_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- knn_imputer (unsupervised) ---- + # ---- kbins_discretizer (unsupervised) ---- t0 = flow_now_ns() - let probe_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) + let probe_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) t1 = flow_now_ns() - knn_imputer_free(probe_knn_imputer) + kbins_discretizer_free(probe_kbins_discretizer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) - knn_imputer_free(m_knn_imputer) + let m_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) + kbins_discretizer_free(m_kbins_discretizer) } t1 = flow_now_ns() - let fitted_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) + let fitted_kbins_discretizer: KBinsDiscretizer = kbins_discretizer_fit(X_c, 4, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_knn_imputer: Matrix = knn_imputer_transform(fitted_knn_imputer, X_c) - matrix_free(o_knn_imputer) + let o_kbins_discretizer: Matrix = kbins_discretizer_transform(fitted_kbins_discretizer, X_c) + if o_kbins_discretizer.rows > 0 { + if o_kbins_discretizer.cols > 0 { sink = sink + o_kbins_discretizer.data[0] } + } + matrix_free(o_kbins_discretizer) } t3 = flow_now_ns() - knn_imputer_free(fitted_knn_imputer) - printf("ESTIMATOR|knn_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + kbins_discretizer_free(fitted_kbins_discretizer) + printf("ESTIMATOR|kbins_discretizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- knn_regressor (regression) ---- + # ---- kernel_density (classification, written out) ---- t0 = flow_now_ns() - let probe_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) + let probe_kernel_density: KernelDensity = kernel_density_fit(X_c, 0.5, 0) t1 = flow_now_ns() - knn_regressor_free(probe_knn_regressor) + kernel_density_free(probe_kernel_density) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) - knn_regressor_free(m_knn_regressor) + let m_kernel_density: KernelDensity = kernel_density_fit(X_c, 0.5, 0) + kernel_density_free(m_kernel_density) } t1 = flow_now_ns() - let fitted_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) + let fitted_kernel_density: KernelDensity = kernel_density_fit(X_c, 0.5, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_knn_regressor: ptr = knn_regressor_predict(fitted_knn_regressor, X_r) - array_free_f32(o_knn_regressor) + let o_kernel_density: ptr = kernel_density_score_samples(fitted_kernel_density, X_c) + sink = sink + o_kernel_density[0] + array_free_f32(o_kernel_density) } t3 = flow_now_ns() - knn_regressor_free(fitted_knn_regressor) - printf("ESTIMATOR|knn_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + kernel_density_free(fitted_kernel_density) + printf("ESTIMATOR|kernel_density|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -531,5 +596,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_03.flow b/benchmarks/generated/bench_estimators_03.flow index 542372f..9086a1b 100644 --- a/benchmarks/generated/bench_estimators_03.flow +++ b/benchmarks/generated/bench_estimators_03.flow @@ -65,460 +65,544 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- label_propagation (classification) ---- + # ---- kernel_pca (unsupervised) ---- t0 = flow_now_ns() - let probe_label_propagation: LabelPropagation = label_propagation_fit(X_c, yi_c, 3, 0.1, 100, 0.0001) + let probe_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) t1 = flow_now_ns() - label_propagation_free(probe_label_propagation) + kernel_pca_free(probe_kernel_pca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_label_propagation: LabelPropagation = label_propagation_fit(X_c, yi_c, 3, 0.1, 100, 0.0001) - label_propagation_free(m_label_propagation) + let m_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) + kernel_pca_free(m_kernel_pca) } t1 = flow_now_ns() - printf("ESTIMATOR|label_propagation|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_kernel_pca: KernelPCA = kernel_pca_fit(X_c, 2, 0, 0.1, 2, 0.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_kernel_pca: Matrix = kernel_pca_transform(fitted_kernel_pca, X_c) + if o_kernel_pca.rows > 0 { + if o_kernel_pca.cols > 0 { sink = sink + o_kernel_pca.data[0] } + } + matrix_free(o_kernel_pca) + } + t3 = flow_now_ns() + kernel_pca_free(fitted_kernel_pca) + printf("ESTIMATOR|kernel_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- label_spreading (classification) ---- + # ---- kernel_ridge (regression) ---- t0 = flow_now_ns() - let probe_label_spreading: LabelSpreading = label_spreading_fit(X_c, yi_c, 3, 0.1, 1.0, 100, 0.0001) + let probe_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) t1 = flow_now_ns() - label_spreading_free(probe_label_spreading) + kernel_ridge_free(probe_kernel_ridge) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_label_spreading: LabelSpreading = label_spreading_fit(X_c, yi_c, 3, 0.1, 1.0, 100, 0.0001) - label_spreading_free(m_label_spreading) + let m_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) + kernel_ridge_free(m_kernel_ridge) } t1 = flow_now_ns() - printf("ESTIMATOR|label_spreading|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_kernel_ridge: KernelRidge = kernel_ridge_fit(X_r, y_r, 1.0, 0, 0.1, 2, 0.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_kernel_ridge: ptr = kernel_ridge_predict(fitted_kernel_ridge, X_r) + sink = sink + o_kernel_ridge[0] + array_free_f32(o_kernel_ridge) + } + t3 = flow_now_ns() + kernel_ridge_free(fitted_kernel_ridge) + printf("ESTIMATOR|kernel_ridge|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lars_cv (regression) ---- + # ---- kernel_svc (classification) ---- t0 = flow_now_ns() - let probe_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) + let probe_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) t1 = flow_now_ns() - lars_cv_free(probe_lars_cv) + kernel_svc_free(probe_kernel_svc) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) - lars_cv_free(m_lars_cv) + let m_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) + kernel_svc_free(m_kernel_svc) } t1 = flow_now_ns() - let fitted_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) + let fitted_kernel_svc: KernelSVC = kernel_svc_fit(X_c, y_c, 3, 1.0, 0.1, 100) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lars_cv: ptr = lars_cv_predict(fitted_lars_cv, X_r) - array_free_f32(o_lars_cv) + let o_kernel_svc: ptr = kernel_svc_predict(fitted_kernel_svc, X_c) + sink = sink + o_kernel_svc[0] + array_free_f32(o_kernel_svc) } t3 = flow_now_ns() - lars_cv_free(fitted_lars_cv) - printf("ESTIMATOR|lars_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + kernel_svc_free(fitted_kernel_svc) + printf("ESTIMATOR|kernel_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lars (regression) ---- + # ---- kernel_svc_multi (classification) ---- t0 = flow_now_ns() - let probe_lars: Lars = lars_fit(X_r, y_r, 2) + let probe_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) t1 = flow_now_ns() - lars_free(probe_lars) + kernel_svc_multi_free(probe_kernel_svc_multi) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lars: Lars = lars_fit(X_r, y_r, 2) - lars_free(m_lars) + let m_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) + kernel_svc_multi_free(m_kernel_svc_multi) } t1 = flow_now_ns() - let fitted_lars: Lars = lars_fit(X_r, y_r, 2) + let fitted_kernel_svc_multi: KernelSVCMulti = kernel_svc_multi_fit(X_c, y_c, 3, 0.1, 1.0, 100) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lars: ptr = lars_predict(fitted_lars, X_r) - array_free_f32(o_lars) + let o_kernel_svc_multi: ptr = kernel_svc_multi_predict(fitted_kernel_svc_multi, X_c) + sink = sink + o_kernel_svc_multi[0] + array_free_f32(o_kernel_svc_multi) } t3 = flow_now_ns() - lars_free(fitted_lars) - printf("ESTIMATOR|lars|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + kernel_svc_multi_free(fitted_kernel_svc_multi) + printf("ESTIMATOR|kernel_svc_multi|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lasso_cv (regression) ---- + # ---- kmeans (unsupervised) ---- t0 = flow_now_ns() - let probe_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) + let probe_kmeans: KMeans = kmeans_fit(X_c, 3, 100, 0.0001, 42) t1 = flow_now_ns() - lasso_cv_free(probe_lasso_cv) + kmeans_free(probe_kmeans) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) - lasso_cv_free(m_lasso_cv) + let m_kmeans: KMeans = kmeans_fit(X_c, 3, 100, 0.0001, 42) + kmeans_free(m_kmeans) } t1 = flow_now_ns() - let fitted_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_lasso_cv: ptr = lasso_cv_predict(fitted_lasso_cv, X_r) - array_free_f32(o_lasso_cv) - } - t3 = flow_now_ns() - lasso_cv_free(fitted_lasso_cv) - printf("ESTIMATOR|lasso_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|kmeans|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- lasso (regression) ---- + # ---- kneighbors_transformer (unsupervised) ---- t0 = flow_now_ns() - let probe_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) + let probe_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) t1 = flow_now_ns() - lasso_free(probe_lasso) + kneighbors_transformer_free(probe_kneighbors_transformer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) - lasso_free(m_lasso) + let m_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) + kneighbors_transformer_free(m_kneighbors_transformer) } t1 = flow_now_ns() - let fitted_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) + let fitted_kneighbors_transformer: KNeighborsTransformer = kneighbors_transformer_fit(X_c, 5, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lasso: ptr = lasso_predict(fitted_lasso, X_r) - array_free_f32(o_lasso) + let o_kneighbors_transformer: Matrix = kneighbors_transformer_transform(fitted_kneighbors_transformer, X_c) + if o_kneighbors_transformer.rows > 0 { + if o_kneighbors_transformer.cols > 0 { sink = sink + o_kneighbors_transformer.data[0] } + } + matrix_free(o_kneighbors_transformer) } t3 = flow_now_ns() - lasso_free(fitted_lasso) - printf("ESTIMATOR|lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + kneighbors_transformer_free(fitted_kneighbors_transformer) + printf("ESTIMATOR|kneighbors_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lasso_lars_cv (regression) ---- + # ---- knn_classifier (classification) ---- t0 = flow_now_ns() - let probe_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) + let probe_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) t1 = flow_now_ns() - lasso_lars_cv_free(probe_lasso_lars_cv) + knn_classifier_free(probe_knn_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) - lasso_lars_cv_free(m_lasso_lars_cv) + let m_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) + knn_classifier_free(m_knn_classifier) } t1 = flow_now_ns() - let fitted_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) + let fitted_knn_classifier: KNNClassifier = knn_classifier_fit(X_c, y_c, 3, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lasso_lars_cv: ptr = lasso_lars_cv_predict(fitted_lasso_lars_cv, X_r) - array_free_f32(o_lasso_lars_cv) + let o_knn_classifier: Matrix = knn_classifier_predict(fitted_knn_classifier, X_c) + if o_knn_classifier.rows > 0 { + if o_knn_classifier.cols > 0 { sink = sink + o_knn_classifier.data[0] } + } + matrix_free(o_knn_classifier) } t3 = flow_now_ns() - lasso_lars_cv_free(fitted_lasso_lars_cv) - printf("ESTIMATOR|lasso_lars_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + knn_classifier_free(fitted_knn_classifier) + printf("ESTIMATOR|knn_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lasso_lars (regression) ---- + # ---- knn_imputer (unsupervised) ---- t0 = flow_now_ns() - let probe_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) + let probe_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) t1 = flow_now_ns() - lasso_lars_free(probe_lasso_lars) + knn_imputer_free(probe_knn_imputer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) - lasso_lars_free(m_lasso_lars) + let m_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) + knn_imputer_free(m_knn_imputer) } t1 = flow_now_ns() - let fitted_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) + let fitted_knn_imputer: KNNImputer = knn_imputer_fit(X_c, 5, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lasso_lars: ptr = lasso_lars_predict(fitted_lasso_lars, X_r) - array_free_f32(o_lasso_lars) + let o_knn_imputer: Matrix = knn_imputer_transform(fitted_knn_imputer, X_c) + if o_knn_imputer.rows > 0 { + if o_knn_imputer.cols > 0 { sink = sink + o_knn_imputer.data[0] } + } + matrix_free(o_knn_imputer) } t3 = flow_now_ns() - lasso_lars_free(fitted_lasso_lars) - printf("ESTIMATOR|lasso_lars|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + knn_imputer_free(fitted_knn_imputer) + printf("ESTIMATOR|knn_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lasso_lars_ic (regression) ---- + # ---- knn_regressor (regression) ---- t0 = flow_now_ns() - let probe_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) + let probe_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) t1 = flow_now_ns() - lasso_lars_ic_free(probe_lasso_lars_ic) + knn_regressor_free(probe_knn_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) - lasso_lars_ic_free(m_lasso_lars_ic) + let m_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) + knn_regressor_free(m_knn_regressor) } t1 = flow_now_ns() - let fitted_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) + let fitted_knn_regressor: KNNRegressor = knn_regressor_fit(X_r, y_r, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lasso_lars_ic: ptr = lasso_lars_ic_predict(fitted_lasso_lars_ic, X_r) - array_free_f32(o_lasso_lars_ic) + let o_knn_regressor: ptr = knn_regressor_predict(fitted_knn_regressor, X_r) + sink = sink + o_knn_regressor[0] + array_free_f32(o_knn_regressor) } t3 = flow_now_ns() - lasso_lars_ic_free(fitted_lasso_lars_ic) - printf("ESTIMATOR|lasso_lars_ic|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + knn_regressor_free(fitted_knn_regressor) + printf("ESTIMATOR|knn_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lda (unsupervised) ---- + # ---- label_binarizer (classification, written out) ---- t0 = flow_now_ns() - let probe_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) + let probe_label_binarizer: LabelBinarizer = label_binarizer_fit(y_c, n_c, 0.0, 1.0) t1 = flow_now_ns() - lda_free(probe_lda) + label_binarizer_free(probe_label_binarizer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) - lda_free(m_lda) + let m_label_binarizer: LabelBinarizer = label_binarizer_fit(y_c, n_c, 0.0, 1.0) + label_binarizer_free(m_label_binarizer) } t1 = flow_now_ns() - let fitted_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) + let fitted_label_binarizer: LabelBinarizer = label_binarizer_fit(y_c, n_c, 0.0, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_lda: Matrix = lda_transform(fitted_lda, X_c) - matrix_free(o_lda) + let o_label_binarizer: Matrix = label_binarizer_transform(fitted_label_binarizer, y_c, n_c) + if o_label_binarizer.rows > 0 { + if o_label_binarizer.cols > 0 { sink = sink + o_label_binarizer.data[0] } + } + matrix_free(o_label_binarizer) } t3 = flow_now_ns() - lda_free(fitted_lda) - printf("ESTIMATOR|lda|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + label_binarizer_free(fitted_label_binarizer) + printf("ESTIMATOR|label_binarizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ledoit_wolf_estimator (unsupervised) ---- + # ---- label_encoder (classification, written out) ---- t0 = flow_now_ns() - let probe_ledoit_wolf_estimator: ShrunkCovariance = ledoit_wolf_estimator_fit(X_c) + let probe_label_encoder: LabelEncoder = label_encoder_fit(y_c, n_c) t1 = flow_now_ns() + label_encoder_free(probe_label_encoder) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ledoit_wolf_estimator: ShrunkCovariance = ledoit_wolf_estimator_fit(X_c) + let m_label_encoder: LabelEncoder = label_encoder_fit(y_c, n_c) + label_encoder_free(m_label_encoder) } t1 = flow_now_ns() - printf("ESTIMATOR|ledoit_wolf_estimator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_label_encoder: LabelEncoder = label_encoder_fit(y_c, n_c) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_label_encoder: ptr = label_encoder_transform(fitted_label_encoder, y_c, n_c) + sink = sink + o_label_encoder[0] + array_free_f32(o_label_encoder) + } + t3 = flow_now_ns() + label_encoder_free(fitted_label_encoder) + printf("ESTIMATOR|label_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- linear_regression (regression) ---- + # ---- label_propagation (classification) ---- t0 = flow_now_ns() - let probe_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) + let probe_label_propagation: LabelPropagation = label_propagation_fit(X_c, yi_c, 3, 0.1, 100, 0.0001) t1 = flow_now_ns() - linear_regression_free(probe_linear_regression) + label_propagation_free(probe_label_propagation) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) - linear_regression_free(m_linear_regression) + let m_label_propagation: LabelPropagation = label_propagation_fit(X_c, yi_c, 3, 0.1, 100, 0.0001) + label_propagation_free(m_label_propagation) } t1 = flow_now_ns() - let fitted_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_linear_regression: ptr = linear_regression_predict(fitted_linear_regression, X_r) - array_free_f32(o_linear_regression) - } - t3 = flow_now_ns() - linear_regression_free(fitted_linear_regression) - printf("ESTIMATOR|linear_regression|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|label_propagation|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- linear_svc (classification) ---- + # ---- label_spreading (classification) ---- t0 = flow_now_ns() - let probe_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) + let probe_label_spreading: LabelSpreading = label_spreading_fit(X_c, yi_c, 3, 0.1, 1.0, 100, 0.0001) t1 = flow_now_ns() - linear_svc_free(probe_linear_svc) + label_spreading_free(probe_label_spreading) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) - linear_svc_free(m_linear_svc) + let m_label_spreading: LabelSpreading = label_spreading_fit(X_c, yi_c, 3, 0.1, 1.0, 100, 0.0001) + label_spreading_free(m_label_spreading) } t1 = flow_now_ns() - let fitted_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_linear_svc: ptr = linear_svc_predict(fitted_linear_svc, X_c) - array_free_f32(o_linear_svc) - } - t3 = flow_now_ns() - linear_svc_free(fitted_linear_svc) - printf("ESTIMATOR|linear_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|label_spreading|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- linear_svc_multi (classification) ---- + # ---- lars_cv (regression) ---- t0 = flow_now_ns() - let probe_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) + let probe_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) t1 = flow_now_ns() - linear_svc_multi_free(probe_linear_svc_multi) + lars_cv_free(probe_lars_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) - linear_svc_multi_free(m_linear_svc_multi) + let m_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) + lars_cv_free(m_lars_cv) } t1 = flow_now_ns() - let fitted_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) + let fitted_lars_cv: LarsCV = lars_cv_fit(X_r, y_r, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_linear_svc_multi: ptr = linear_svc_multi_predict(fitted_linear_svc_multi, X_c) - array_free_f32(o_linear_svc_multi) + let o_lars_cv: ptr = lars_cv_predict(fitted_lars_cv, X_r) + sink = sink + o_lars_cv[0] + array_free_f32(o_lars_cv) } t3 = flow_now_ns() - linear_svc_multi_free(fitted_linear_svc_multi) - printf("ESTIMATOR|linear_svc_multi|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + lars_cv_free(fitted_lars_cv) + printf("ESTIMATOR|lars_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- linear_svr (regression) ---- + # ---- lars (regression) ---- t0 = flow_now_ns() - let probe_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) + let probe_lars: Lars = lars_fit(X_r, y_r, 2) t1 = flow_now_ns() - linear_svr_free(probe_linear_svr) + lars_free(probe_lars) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) - linear_svr_free(m_linear_svr) + let m_lars: Lars = lars_fit(X_r, y_r, 2) + lars_free(m_lars) } t1 = flow_now_ns() - let fitted_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) + let fitted_lars: Lars = lars_fit(X_r, y_r, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_linear_svr: ptr = linear_svr_predict(fitted_linear_svr, X_r) - array_free_f32(o_linear_svr) + let o_lars: ptr = lars_predict(fitted_lars, X_r) + sink = sink + o_lars[0] + array_free_f32(o_lars) } t3 = flow_now_ns() - linear_svr_free(fitted_linear_svr) - printf("ESTIMATOR|linear_svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + lars_free(fitted_lars) + printf("ESTIMATOR|lars|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- lle (unsupervised) ---- + # ---- lasso_cv (regression) ---- t0 = flow_now_ns() - let probe_lle: LLE = lle_fit(X_c, 2, 5) + let probe_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) t1 = flow_now_ns() - lle_free(probe_lle) + lasso_cv_free(probe_lasso_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_lle: LLE = lle_fit(X_c, 2, 5) - lle_free(m_lle) + let m_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) + lasso_cv_free(m_lasso_cv) } t1 = flow_now_ns() - printf("ESTIMATOR|lle|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_lasso_cv: LassoCV = lasso_cv_fit(X_r, y_r, 10) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_lasso_cv: ptr = lasso_cv_predict(fitted_lasso_cv, X_r) + sink = sink + o_lasso_cv[0] + array_free_f32(o_lasso_cv) + } + t3 = flow_now_ns() + lasso_cv_free(fitted_lasso_cv) + printf("ESTIMATOR|lasso_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- local_outlier_factor (unsupervised) ---- + # ---- lasso (regression) ---- t0 = flow_now_ns() - let probe_local_outlier_factor: LocalOutlierFactor = local_outlier_factor_fit(X_c, 5) + let probe_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) t1 = flow_now_ns() - local_outlier_factor_free(probe_local_outlier_factor) + lasso_free(probe_lasso) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_local_outlier_factor: LocalOutlierFactor = local_outlier_factor_fit(X_c, 5) - local_outlier_factor_free(m_local_outlier_factor) + let m_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) + lasso_free(m_lasso) } t1 = flow_now_ns() - printf("ESTIMATOR|local_outlier_factor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_lasso: Lasso = lasso_fit(X_r, y_r, 1.0, 50, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_lasso: ptr = lasso_predict(fitted_lasso, X_r) + sink = sink + o_lasso[0] + array_free_f32(o_lasso) + } + t3 = flow_now_ns() + lasso_free(fitted_lasso) + printf("ESTIMATOR|lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- logistic_regression_cv (classification) ---- + # ---- lasso_lars_cv (regression) ---- t0 = flow_now_ns() - let probe_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) + let probe_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) t1 = flow_now_ns() - logistic_regression_cv_free(probe_logistic_regression_cv) + lasso_lars_cv_free(probe_lasso_lars_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) - logistic_regression_cv_free(m_logistic_regression_cv) + let m_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) + lasso_lars_cv_free(m_lasso_lars_cv) } t1 = flow_now_ns() - let fitted_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) + let fitted_lasso_lars_cv: LassoLarsCV = lasso_lars_cv_fit(X_r, y_r, 10) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_logistic_regression_cv: ptr = logistic_regression_cv_predict(fitted_logistic_regression_cv, X_c) - array_free_f32(o_logistic_regression_cv) + let o_lasso_lars_cv: ptr = lasso_lars_cv_predict(fitted_lasso_lars_cv, X_r) + sink = sink + o_lasso_lars_cv[0] + array_free_f32(o_lasso_lars_cv) } t3 = flow_now_ns() - logistic_regression_cv_free(fitted_logistic_regression_cv) - printf("ESTIMATOR|logistic_regression_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + lasso_lars_cv_free(fitted_lasso_lars_cv) + printf("ESTIMATOR|lasso_lars_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- logistic_regression (classification) ---- + # ---- lasso_lars (regression) ---- t0 = flow_now_ns() - let probe_logistic_regression: LogisticRegression = logistic_regression_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let probe_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) t1 = flow_now_ns() - logistic_regression_free(probe_logistic_regression) + lasso_lars_free(probe_lasso_lars) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_logistic_regression: LogisticRegression = logistic_regression_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) - logistic_regression_free(m_logistic_regression) + let m_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) + lasso_lars_free(m_lasso_lars) } t1 = flow_now_ns() - printf("ESTIMATOR|logistic_regression|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_lasso_lars: LassoLars = lasso_lars_fit(X_r, y_r, 1.0, 100) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_lasso_lars: ptr = lasso_lars_predict(fitted_lasso_lars, X_r) + sink = sink + o_lasso_lars[0] + array_free_f32(o_lasso_lars) + } + t3 = flow_now_ns() + lasso_lars_free(fitted_lasso_lars) + printf("ESTIMATOR|lasso_lars|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- maxabs_scaler (unsupervised) ---- + # ---- lasso_lars_ic (regression) ---- t0 = flow_now_ns() - let probe_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) + let probe_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) t1 = flow_now_ns() - maxabs_scaler_free(probe_maxabs_scaler) + lasso_lars_ic_free(probe_lasso_lars_ic) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) - maxabs_scaler_free(m_maxabs_scaler) + let m_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) + lasso_lars_ic_free(m_lasso_lars_ic) } t1 = flow_now_ns() - let fitted_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) + let fitted_lasso_lars_ic: LassoLarsIC = lasso_lars_ic_fit(X_r, y_r, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_maxabs_scaler: Matrix = maxabs_scaler_transform(fitted_maxabs_scaler, X_c) - matrix_free(o_maxabs_scaler) + let o_lasso_lars_ic: ptr = lasso_lars_ic_predict(fitted_lasso_lars_ic, X_r) + sink = sink + o_lasso_lars_ic[0] + array_free_f32(o_lasso_lars_ic) } t3 = flow_now_ns() - maxabs_scaler_free(fitted_maxabs_scaler) - printf("ESTIMATOR|maxabs_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + lasso_lars_ic_free(fitted_lasso_lars_ic) + printf("ESTIMATOR|lasso_lars_ic|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -529,5 +613,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_04.flow b/benchmarks/generated/bench_estimators_04.flow index 132eb6e..dca7f52 100644 --- a/benchmarks/generated/bench_estimators_04.flow +++ b/benchmarks/generated/bench_estimators_04.flow @@ -65,480 +65,501 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- mds (unsupervised) ---- + # ---- lda (unsupervised) ---- t0 = flow_now_ns() - let probe_mds: MDS = mds_fit(X_c, 2, 100, 0.0001, 42) + let probe_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) t1 = flow_now_ns() - mds_free(probe_mds) + lda_free(probe_lda) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_mds: MDS = mds_fit(X_c, 2, 100, 0.0001, 42) - mds_free(m_mds) + let m_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) + lda_free(m_lda) } t1 = flow_now_ns() - printf("ESTIMATOR|mds|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_lda: LatentDirichletAllocation = lda_fit(X_c, 3, 50, 1.0, 0.1, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_lda: Matrix = lda_transform(fitted_lda, X_c) + if o_lda.rows > 0 { + if o_lda.cols > 0 { sink = sink + o_lda.data[0] } + } + matrix_free(o_lda) + } + t3 = flow_now_ns() + lda_free(fitted_lda) + printf("ESTIMATOR|lda|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- mean_shift (unsupervised) ---- + # ---- ledoit_wolf_estimator (unsupervised) ---- t0 = flow_now_ns() - let probe_mean_shift: MeanShift = mean_shift_fit(X_c, 1.0, 100) + let probe_ledoit_wolf_estimator: ShrunkCovariance = ledoit_wolf_estimator_fit(X_c) t1 = flow_now_ns() - mean_shift_free(probe_mean_shift) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_mean_shift: MeanShift = mean_shift_fit(X_c, 1.0, 100) - mean_shift_free(m_mean_shift) + let m_ledoit_wolf_estimator: ShrunkCovariance = ledoit_wolf_estimator_fit(X_c) } t1 = flow_now_ns() - printf("ESTIMATOR|mean_shift|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + printf("ESTIMATOR|ledoit_wolf_estimator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- min_cov_det (unsupervised) ---- + # ---- linear_regression (regression) ---- t0 = flow_now_ns() - let probe_min_cov_det: MinCovDet = min_cov_det_fit(X_c, 0.75, 100, 42) + let probe_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) t1 = flow_now_ns() - min_cov_det_free(probe_min_cov_det) + linear_regression_free(probe_linear_regression) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_min_cov_det: MinCovDet = min_cov_det_fit(X_c, 0.75, 100, 42) - min_cov_det_free(m_min_cov_det) + let m_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) + linear_regression_free(m_linear_regression) } t1 = flow_now_ns() - printf("ESTIMATOR|min_cov_det|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_linear_regression: LinearRegression = linear_regression_fit(X_r, y_r, penalty_none()) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_linear_regression: ptr = linear_regression_predict(fitted_linear_regression, X_r) + sink = sink + o_linear_regression[0] + array_free_f32(o_linear_regression) + } + t3 = flow_now_ns() + linear_regression_free(fitted_linear_regression) + printf("ESTIMATOR|linear_regression|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- minibatch_dictionary_learning (unsupervised) ---- + # ---- linear_svc (classification) ---- t0 = flow_now_ns() - let probe_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) + let probe_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) t1 = flow_now_ns() - minibatch_dictionary_learning_free(probe_minibatch_dictionary_learning) + linear_svc_free(probe_linear_svc) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) - minibatch_dictionary_learning_free(m_minibatch_dictionary_learning) + let m_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) + linear_svc_free(m_linear_svc) } t1 = flow_now_ns() - let fitted_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) + let fitted_linear_svc: LinearSVC = linear_svc_fit(X_c, y_c, 3, 1.0, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_minibatch_dictionary_learning: Matrix = minibatch_dictionary_learning_transform(fitted_minibatch_dictionary_learning, X_c) - matrix_free(o_minibatch_dictionary_learning) + let o_linear_svc: ptr = linear_svc_predict(fitted_linear_svc, X_c) + sink = sink + o_linear_svc[0] + array_free_f32(o_linear_svc) } t3 = flow_now_ns() - minibatch_dictionary_learning_free(fitted_minibatch_dictionary_learning) - printf("ESTIMATOR|minibatch_dictionary_learning|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + linear_svc_free(fitted_linear_svc) + printf("ESTIMATOR|linear_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- minibatch_kmeans (unsupervised) ---- + # ---- linear_svc_multi (classification) ---- t0 = flow_now_ns() - let probe_minibatch_kmeans: MiniBatchKMeans = minibatch_kmeans_fit(X_c, 3, 32, 100, 0.0001, 42) + let probe_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) t1 = flow_now_ns() - minibatch_kmeans_free(probe_minibatch_kmeans) + linear_svc_multi_free(probe_linear_svc_multi) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_minibatch_kmeans: MiniBatchKMeans = minibatch_kmeans_fit(X_c, 3, 32, 100, 0.0001, 42) - minibatch_kmeans_free(m_minibatch_kmeans) + let m_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) + linear_svc_multi_free(m_linear_svc_multi) } t1 = flow_now_ns() - printf("ESTIMATOR|minibatch_kmeans|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_linear_svc_multi: LinearSVCMulti = linear_svc_multi_fit(X_c, y_c, 3, 1.0, 50) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_linear_svc_multi: ptr = linear_svc_multi_predict(fitted_linear_svc_multi, X_c) + sink = sink + o_linear_svc_multi[0] + array_free_f32(o_linear_svc_multi) + } + t3 = flow_now_ns() + linear_svc_multi_free(fitted_linear_svc_multi) + printf("ESTIMATOR|linear_svc_multi|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- minibatch_nmf (unsupervised) ---- + # ---- linear_svr (regression) ---- t0 = flow_now_ns() - let probe_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) + let probe_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) t1 = flow_now_ns() - minibatch_nmf_free(probe_minibatch_nmf) + linear_svr_free(probe_linear_svr) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) - minibatch_nmf_free(m_minibatch_nmf) + let m_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) + linear_svr_free(m_linear_svr) } t1 = flow_now_ns() - let fitted_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) + let fitted_linear_svr: LinearSVR = linear_svr_fit(X_r, y_r, 1.0, 0.1, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_minibatch_nmf: Matrix = minibatch_nmf_transform(fitted_minibatch_nmf, X_c) - matrix_free(o_minibatch_nmf) + let o_linear_svr: ptr = linear_svr_predict(fitted_linear_svr, X_r) + sink = sink + o_linear_svr[0] + array_free_f32(o_linear_svr) } t3 = flow_now_ns() - minibatch_nmf_free(fitted_minibatch_nmf) - printf("ESTIMATOR|minibatch_nmf|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + linear_svr_free(fitted_linear_svr) + printf("ESTIMATOR|linear_svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- minibatch_sparse_pca (unsupervised) ---- + # ---- lle (unsupervised) ---- t0 = flow_now_ns() - let probe_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) + let probe_lle: LLE = lle_fit(X_c, 2, 5) t1 = flow_now_ns() - minibatch_sparse_pca_free(probe_minibatch_sparse_pca) + lle_free(probe_lle) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) - minibatch_sparse_pca_free(m_minibatch_sparse_pca) + let m_lle: LLE = lle_fit(X_c, 2, 5) + lle_free(m_lle) } t1 = flow_now_ns() - let fitted_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_minibatch_sparse_pca: Matrix = minibatch_sparse_pca_transform(fitted_minibatch_sparse_pca, X_c) - matrix_free(o_minibatch_sparse_pca) - } - t3 = flow_now_ns() - minibatch_sparse_pca_free(fitted_minibatch_sparse_pca) - printf("ESTIMATOR|minibatch_sparse_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|lle|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- minmax_scaler (unsupervised) ---- + # ---- local_outlier_factor (unsupervised) ---- t0 = flow_now_ns() - let probe_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) + let probe_local_outlier_factor: LocalOutlierFactor = local_outlier_factor_fit(X_c, 5) t1 = flow_now_ns() - minmax_scaler_free(probe_minmax_scaler) + local_outlier_factor_free(probe_local_outlier_factor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) - minmax_scaler_free(m_minmax_scaler) + let m_local_outlier_factor: LocalOutlierFactor = local_outlier_factor_fit(X_c, 5) + local_outlier_factor_free(m_local_outlier_factor) } t1 = flow_now_ns() - let fitted_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_minmax_scaler: Matrix = minmax_scaler_transform(fitted_minmax_scaler, X_c) - matrix_free(o_minmax_scaler) - } - t3 = flow_now_ns() - minmax_scaler_free(fitted_minmax_scaler) - printf("ESTIMATOR|minmax_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|local_outlier_factor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- missing_indicator (unsupervised) ---- + # ---- logistic_regression_cv (classification) ---- t0 = flow_now_ns() - let probe_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) + let probe_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) t1 = flow_now_ns() - missing_indicator_free(probe_missing_indicator) + logistic_regression_cv_free(probe_logistic_regression_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) - missing_indicator_free(m_missing_indicator) + let m_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) + logistic_regression_cv_free(m_logistic_regression_cv) } t1 = flow_now_ns() - let fitted_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) + let fitted_logistic_regression_cv: LogisticRegressionCV = logistic_regression_cv_fit(X_c, y_c, 3, 5) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_missing_indicator: Matrix = missing_indicator_transform(fitted_missing_indicator, X_c) - matrix_free(o_missing_indicator) + let o_logistic_regression_cv: ptr = logistic_regression_cv_predict(fitted_logistic_regression_cv, X_c) + sink = sink + o_logistic_regression_cv[0] + array_free_f32(o_logistic_regression_cv) } t3 = flow_now_ns() - missing_indicator_free(fitted_missing_indicator) - printf("ESTIMATOR|missing_indicator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + logistic_regression_cv_free(fitted_logistic_regression_cv) + printf("ESTIMATOR|logistic_regression_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- mlp_classifier (classification) ---- - let hidden_sizes_mlp_classifier: array = [8] + # ---- logistic_regression (classification) ---- t0 = flow_now_ns() - let probe_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) + let probe_logistic_regression: LogisticRegression = logistic_regression_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t1 = flow_now_ns() - mlp_classifier_free(probe_mlp_classifier) + logistic_regression_free(probe_logistic_regression) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) - mlp_classifier_free(m_mlp_classifier) + let m_logistic_regression: LogisticRegression = logistic_regression_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + logistic_regression_free(m_logistic_regression) } t1 = flow_now_ns() - let fitted_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_mlp_classifier: Matrix = mlp_classifier_predict(fitted_mlp_classifier, X_c) - matrix_free(o_mlp_classifier) - } - t3 = flow_now_ns() - mlp_classifier_free(fitted_mlp_classifier) - printf("ESTIMATOR|mlp_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|logistic_regression|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- mlp_regressor (regression) ---- - let hidden_sizes_mlp_regressor: array = [8] + # ---- maxabs_scaler (unsupervised) ---- t0 = flow_now_ns() - let probe_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) + let probe_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) t1 = flow_now_ns() - mlp_regressor_free(probe_mlp_regressor) + maxabs_scaler_free(probe_maxabs_scaler) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) - mlp_regressor_free(m_mlp_regressor) + let m_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) + maxabs_scaler_free(m_maxabs_scaler) } t1 = flow_now_ns() - let fitted_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) + let fitted_maxabs_scaler: MaxAbsScaler = maxabs_scaler_fit(X_c) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_mlp_regressor: ptr = mlp_regressor_predict(fitted_mlp_regressor, X_r) - array_free_f32(o_mlp_regressor) + let o_maxabs_scaler: Matrix = maxabs_scaler_transform(fitted_maxabs_scaler, X_c) + if o_maxabs_scaler.rows > 0 { + if o_maxabs_scaler.cols > 0 { sink = sink + o_maxabs_scaler.data[0] } + } + matrix_free(o_maxabs_scaler) } t3 = flow_now_ns() - mlp_regressor_free(fitted_mlp_regressor) - printf("ESTIMATOR|mlp_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + maxabs_scaler_free(fitted_maxabs_scaler) + printf("ESTIMATOR|maxabs_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- multi_output_classifier (multioutput_class) ---- + # ---- mds (unsupervised) ---- t0 = flow_now_ns() - let probe_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) + let probe_mds: MDS = mds_fit(X_c, 2, 100, 0.0001, 42) t1 = flow_now_ns() - multi_output_classifier_free(probe_multi_output_classifier) + mds_free(probe_mds) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) - multi_output_classifier_free(m_multi_output_classifier) + let m_mds: MDS = mds_fit(X_c, 2, 100, 0.0001, 42) + mds_free(m_mds) } t1 = flow_now_ns() - let fitted_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_multi_output_classifier: Matrix = multi_output_classifier_predict(fitted_multi_output_classifier, X_c) - matrix_free(o_multi_output_classifier) - } - t3 = flow_now_ns() - multi_output_classifier_free(fitted_multi_output_classifier) - printf("ESTIMATOR|multi_output_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|mds|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- multi_output_regressor (multioutput) ---- + # ---- mean_shift (unsupervised) ---- t0 = flow_now_ns() - let probe_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) + let probe_mean_shift: MeanShift = mean_shift_fit(X_c, 1.0, 100) t1 = flow_now_ns() - multi_output_regressor_free(probe_multi_output_regressor) + mean_shift_free(probe_mean_shift) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) - multi_output_regressor_free(m_multi_output_regressor) + let m_mean_shift: MeanShift = mean_shift_fit(X_c, 1.0, 100) + mean_shift_free(m_mean_shift) } t1 = flow_now_ns() - let fitted_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_multi_output_regressor: Matrix = multi_output_regressor_predict(fitted_multi_output_regressor, X_r) - matrix_free(o_multi_output_regressor) - } - t3 = flow_now_ns() - multi_output_regressor_free(fitted_multi_output_regressor) - printf("ESTIMATOR|multi_output_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|mean_shift|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- multiclass_logistic (classification) ---- + # ---- min_cov_det (unsupervised) ---- t0 = flow_now_ns() - let probe_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let probe_min_cov_det: MinCovDet = min_cov_det_fit(X_c, 0.75, 100, 42) t1 = flow_now_ns() - multiclass_logistic_free(probe_multiclass_logistic) + min_cov_det_free(probe_min_cov_det) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) - multiclass_logistic_free(m_multiclass_logistic) + let m_min_cov_det: MinCovDet = min_cov_det_fit(X_c, 0.75, 100, 42) + min_cov_det_free(m_min_cov_det) } t1 = flow_now_ns() - let fitted_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_multiclass_logistic: Matrix = multiclass_logistic_predict(fitted_multiclass_logistic, X_c) - matrix_free(o_multiclass_logistic) - } - t3 = flow_now_ns() - multiclass_logistic_free(fitted_multiclass_logistic) - printf("ESTIMATOR|multiclass_logistic|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|min_cov_det|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- multinomial_nb (classification) ---- + # ---- minibatch_dictionary_learning (unsupervised) ---- t0 = flow_now_ns() - let probe_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) + let probe_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) t1 = flow_now_ns() - multinomial_nb_free(probe_multinomial_nb) + minibatch_dictionary_learning_free(probe_minibatch_dictionary_learning) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) - multinomial_nb_free(m_multinomial_nb) + let m_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) + minibatch_dictionary_learning_free(m_minibatch_dictionary_learning) } t1 = flow_now_ns() - let fitted_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) + let fitted_minibatch_dictionary_learning: MiniBatchDictionaryLearning = minibatch_dictionary_learning_fit(X_c, 2, 50, 32, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_multinomial_nb: ptr = multinomial_nb_predict(fitted_multinomial_nb, X_c) - array_free_f32(o_multinomial_nb) + let o_minibatch_dictionary_learning: Matrix = minibatch_dictionary_learning_transform(fitted_minibatch_dictionary_learning, X_c) + if o_minibatch_dictionary_learning.rows > 0 { + if o_minibatch_dictionary_learning.cols > 0 { sink = sink + o_minibatch_dictionary_learning.data[0] } + } + matrix_free(o_minibatch_dictionary_learning) } t3 = flow_now_ns() - multinomial_nb_free(fitted_multinomial_nb) - printf("ESTIMATOR|multinomial_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + minibatch_dictionary_learning_free(fitted_minibatch_dictionary_learning) + printf("ESTIMATOR|minibatch_dictionary_learning|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- multitask_elastic_net_cv (multioutput) ---- + # ---- minibatch_kmeans (unsupervised) ---- t0 = flow_now_ns() - let probe_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) + let probe_minibatch_kmeans: MiniBatchKMeans = minibatch_kmeans_fit(X_c, 3, 32, 100, 0.0001, 42) t1 = flow_now_ns() - multitask_elastic_net_cv_free(probe_multitask_elastic_net_cv) + minibatch_kmeans_free(probe_minibatch_kmeans) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) - multitask_elastic_net_cv_free(m_multitask_elastic_net_cv) + let m_minibatch_kmeans: MiniBatchKMeans = minibatch_kmeans_fit(X_c, 3, 32, 100, 0.0001, 42) + minibatch_kmeans_free(m_minibatch_kmeans) } t1 = flow_now_ns() - let fitted_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_multitask_elastic_net_cv: Matrix = multitask_elastic_net_cv_predict(fitted_multitask_elastic_net_cv, X_r) - matrix_free(o_multitask_elastic_net_cv) - } - t3 = flow_now_ns() - multitask_elastic_net_cv_free(fitted_multitask_elastic_net_cv) - printf("ESTIMATOR|multitask_elastic_net_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|minibatch_kmeans|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- multitask_elastic_net (multioutput) ---- + # ---- minibatch_nmf (unsupervised) ---- t0 = flow_now_ns() - let probe_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) + let probe_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) t1 = flow_now_ns() - multitask_elastic_net_free(probe_multitask_elastic_net) + minibatch_nmf_free(probe_minibatch_nmf) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) - multitask_elastic_net_free(m_multitask_elastic_net) + let m_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) + minibatch_nmf_free(m_minibatch_nmf) } t1 = flow_now_ns() - let fitted_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) + let fitted_minibatch_nmf: MiniBatchNMF = minibatch_nmf_fit(X_c, 2, 100, 32, 0.0001, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_multitask_elastic_net: Matrix = multitask_elastic_net_predict(fitted_multitask_elastic_net, X_r) - matrix_free(o_multitask_elastic_net) + let o_minibatch_nmf: Matrix = minibatch_nmf_transform(fitted_minibatch_nmf, X_c) + if o_minibatch_nmf.rows > 0 { + if o_minibatch_nmf.cols > 0 { sink = sink + o_minibatch_nmf.data[0] } + } + matrix_free(o_minibatch_nmf) } t3 = flow_now_ns() - multitask_elastic_net_free(fitted_multitask_elastic_net) - printf("ESTIMATOR|multitask_elastic_net|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + minibatch_nmf_free(fitted_minibatch_nmf) + printf("ESTIMATOR|minibatch_nmf|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- multitask_lasso_cv (multioutput) ---- + # ---- minibatch_sparse_pca (unsupervised) ---- t0 = flow_now_ns() - let probe_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) + let probe_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) t1 = flow_now_ns() - multitask_lasso_cv_free(probe_multitask_lasso_cv) + minibatch_sparse_pca_free(probe_minibatch_sparse_pca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) - multitask_lasso_cv_free(m_multitask_lasso_cv) + let m_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) + minibatch_sparse_pca_free(m_minibatch_sparse_pca) } t1 = flow_now_ns() - let fitted_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) + let fitted_minibatch_sparse_pca: MiniBatchSparsePCA = minibatch_sparse_pca_fit(X_c, 2, 1.0, 100, 32, 0.0001, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_multitask_lasso_cv: Matrix = multitask_lasso_cv_predict(fitted_multitask_lasso_cv, X_r) - matrix_free(o_multitask_lasso_cv) + let o_minibatch_sparse_pca: Matrix = minibatch_sparse_pca_transform(fitted_minibatch_sparse_pca, X_c) + if o_minibatch_sparse_pca.rows > 0 { + if o_minibatch_sparse_pca.cols > 0 { sink = sink + o_minibatch_sparse_pca.data[0] } + } + matrix_free(o_minibatch_sparse_pca) } t3 = flow_now_ns() - multitask_lasso_cv_free(fitted_multitask_lasso_cv) - printf("ESTIMATOR|multitask_lasso_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + minibatch_sparse_pca_free(fitted_minibatch_sparse_pca) + printf("ESTIMATOR|minibatch_sparse_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- multitask_lasso (multioutput) ---- + # ---- minmax_scaler (unsupervised) ---- t0 = flow_now_ns() - let probe_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) + let probe_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) t1 = flow_now_ns() - multitask_lasso_free(probe_multitask_lasso) + minmax_scaler_free(probe_minmax_scaler) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) - multitask_lasso_free(m_multitask_lasso) + let m_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) + minmax_scaler_free(m_minmax_scaler) } t1 = flow_now_ns() - let fitted_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) + let fitted_minmax_scaler: MinMaxScaler = minmax_scaler_fit(X_c) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_multitask_lasso: Matrix = multitask_lasso_predict(fitted_multitask_lasso, X_r) - matrix_free(o_multitask_lasso) + let o_minmax_scaler: Matrix = minmax_scaler_transform(fitted_minmax_scaler, X_c) + if o_minmax_scaler.rows > 0 { + if o_minmax_scaler.cols > 0 { sink = sink + o_minmax_scaler.data[0] } + } + matrix_free(o_minmax_scaler) } t3 = flow_now_ns() - multitask_lasso_free(fitted_multitask_lasso) - printf("ESTIMATOR|multitask_lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + minmax_scaler_free(fitted_minmax_scaler) + printf("ESTIMATOR|minmax_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nca (regression) ---- + # ---- missing_indicator (unsupervised) ---- t0 = flow_now_ns() - let probe_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) + let probe_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) t1 = flow_now_ns() - nca_free(probe_nca) + missing_indicator_free(probe_missing_indicator) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) - nca_free(m_nca) + let m_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) + missing_indicator_free(m_missing_indicator) } t1 = flow_now_ns() - let fitted_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) + let fitted_missing_indicator: MissingIndicator = missing_indicator_fit(X_c, 0.0, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nca: Matrix = nca_transform(fitted_nca, X_r) - matrix_free(o_nca) + let o_missing_indicator: Matrix = missing_indicator_transform(fitted_missing_indicator, X_c) + if o_missing_indicator.rows > 0 { + if o_missing_indicator.cols > 0 { sink = sink + o_missing_indicator.data[0] } + } + matrix_free(o_missing_indicator) } t3 = flow_now_ns() - nca_free(fitted_nca) - printf("ESTIMATOR|nca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + missing_indicator_free(fitted_missing_indicator) + printf("ESTIMATOR|missing_indicator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -549,5 +570,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_05.flow b/benchmarks/generated/bench_estimators_05.flow index 37b3de4..ae3c9d1 100644 --- a/benchmarks/generated/bench_estimators_05.flow +++ b/benchmarks/generated/bench_estimators_05.flow @@ -65,476 +65,577 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- nearest_centroid (classification) ---- + # ---- mlp_classifier (classification) ---- + let hidden_sizes_mlp_classifier: array = [8] t0 = flow_now_ns() - let probe_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) + let probe_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) t1 = flow_now_ns() - nearest_centroid_free(probe_nearest_centroid) + mlp_classifier_free(probe_mlp_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) - nearest_centroid_free(m_nearest_centroid) + let m_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) + mlp_classifier_free(m_mlp_classifier) } t1 = flow_now_ns() - let fitted_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) + let fitted_mlp_classifier: MLPClassifier = mlp_classifier_fit(X_c, y_c, 3, hidden_sizes_mlp_classifier, 1, 0, 50, 0.01, 0.9, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nearest_centroid: ptr = nearest_centroid_predict(fitted_nearest_centroid, X_c) - array_free_f32(o_nearest_centroid) + let o_mlp_classifier: Matrix = mlp_classifier_predict(fitted_mlp_classifier, X_c) + if o_mlp_classifier.rows > 0 { + if o_mlp_classifier.cols > 0 { sink = sink + o_mlp_classifier.data[0] } + } + matrix_free(o_mlp_classifier) } t3 = flow_now_ns() - nearest_centroid_free(fitted_nearest_centroid) - printf("ESTIMATOR|nearest_centroid|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + mlp_classifier_free(fitted_mlp_classifier) + printf("ESTIMATOR|mlp_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nearest_neighbors (unsupervised) ---- + # ---- mlp_regressor (regression) ---- + let hidden_sizes_mlp_regressor: array = [8] t0 = flow_now_ns() - let probe_nearest_neighbors: NearestNeighbors = nearest_neighbors_fit(X_c, 5) + let probe_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) t1 = flow_now_ns() - nearest_neighbors_free(probe_nearest_neighbors) + mlp_regressor_free(probe_mlp_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nearest_neighbors: NearestNeighbors = nearest_neighbors_fit(X_c, 5) - nearest_neighbors_free(m_nearest_neighbors) + let m_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) + mlp_regressor_free(m_mlp_regressor) } t1 = flow_now_ns() - printf("ESTIMATOR|nearest_neighbors|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_mlp_regressor: MLPRegressor = mlp_regressor_fit(X_r, y_r, hidden_sizes_mlp_regressor, 1, 0, 50, 0.01, 0.9, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_mlp_regressor: ptr = mlp_regressor_predict(fitted_mlp_regressor, X_r) + sink = sink + o_mlp_regressor[0] + array_free_f32(o_mlp_regressor) + } + t3 = flow_now_ns() + mlp_regressor_free(fitted_mlp_regressor) + printf("ESTIMATOR|mlp_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nmf (unsupervised) ---- + # ---- multi_output_classifier (multioutput_class) ---- t0 = flow_now_ns() - let probe_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) + let probe_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) t1 = flow_now_ns() - nmf_free(probe_nmf) + multi_output_classifier_free(probe_multi_output_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) - nmf_free(m_nmf) + let m_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) + multi_output_classifier_free(m_multi_output_classifier) } t1 = flow_now_ns() - let fitted_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) + let fitted_multi_output_classifier: MultiOutputClassifier = multi_output_classifier_fit(X_c, Y_labels, 2, 1, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nmf: Matrix = nmf_transform(fitted_nmf, X_c) - matrix_free(o_nmf) + let o_multi_output_classifier: Matrix = multi_output_classifier_predict(fitted_multi_output_classifier, X_c) + if o_multi_output_classifier.rows > 0 { + if o_multi_output_classifier.cols > 0 { sink = sink + o_multi_output_classifier.data[0] } + } + matrix_free(o_multi_output_classifier) } t3 = flow_now_ns() - nmf_free(fitted_nmf) - printf("ESTIMATOR|nmf|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multi_output_classifier_free(fitted_multi_output_classifier) + printf("ESTIMATOR|multi_output_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nu_svc (regression) ---- + # ---- multi_output_regressor (multioutput) ---- t0 = flow_now_ns() - let probe_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) + let probe_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) t1 = flow_now_ns() - nu_svc_free(probe_nu_svc) + multi_output_regressor_free(probe_multi_output_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) - nu_svc_free(m_nu_svc) + let m_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) + multi_output_regressor_free(m_multi_output_regressor) } t1 = flow_now_ns() - let fitted_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) + let fitted_multi_output_regressor: MultiOutputRegressor = multi_output_regressor_fit(X_r, Y_multi, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nu_svc: ptr = nu_svc_predict(fitted_nu_svc, X_r) - array_free_f32(o_nu_svc) + let o_multi_output_regressor: Matrix = multi_output_regressor_predict(fitted_multi_output_regressor, X_r) + if o_multi_output_regressor.rows > 0 { + if o_multi_output_regressor.cols > 0 { sink = sink + o_multi_output_regressor.data[0] } + } + matrix_free(o_multi_output_regressor) } t3 = flow_now_ns() - nu_svc_free(fitted_nu_svc) - printf("ESTIMATOR|nu_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multi_output_regressor_free(fitted_multi_output_regressor) + printf("ESTIMATOR|multi_output_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nu_svr (regression) ---- + # ---- multiclass_logistic (classification) ---- t0 = flow_now_ns() - let probe_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) + let probe_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t1 = flow_now_ns() - nu_svr_free(probe_nu_svr) + multiclass_logistic_free(probe_multiclass_logistic) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) - nu_svr_free(m_nu_svr) + let m_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + multiclass_logistic_free(m_multiclass_logistic) } t1 = flow_now_ns() - let fitted_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) + let fitted_multiclass_logistic: MultiClassLogisticRegression = multiclass_logistic_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nu_svr: ptr = nu_svr_predict(fitted_nu_svr, X_r) - array_free_f32(o_nu_svr) + let o_multiclass_logistic: Matrix = multiclass_logistic_predict(fitted_multiclass_logistic, X_c) + if o_multiclass_logistic.rows > 0 { + if o_multiclass_logistic.cols > 0 { sink = sink + o_multiclass_logistic.data[0] } + } + matrix_free(o_multiclass_logistic) } t3 = flow_now_ns() - nu_svr_free(fitted_nu_svr) - printf("ESTIMATOR|nu_svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multiclass_logistic_free(fitted_multiclass_logistic) + printf("ESTIMATOR|multiclass_logistic|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- nystroem (unsupervised) ---- + # ---- multilabel_binarizer (multioutput_class, written out) ---- + let mlb_counts: ptr = malloc((n_c as i64) * 4) as ptr + for i in 0 to n_c { mlb_counts[i] = 2 } t0 = flow_now_ns() - let probe_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) + let probe_multilabel_binarizer: MultiLabelBinarizer = multilabel_binarizer_fit(Y_label_rows, n_c, mlb_counts, 3) t1 = flow_now_ns() - nystroem_free(probe_nystroem) + multilabel_binarizer_free(probe_multilabel_binarizer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) - nystroem_free(m_nystroem) + let m_multilabel_binarizer: MultiLabelBinarizer = multilabel_binarizer_fit(Y_label_rows, n_c, mlb_counts, 3) + multilabel_binarizer_free(m_multilabel_binarizer) } t1 = flow_now_ns() - let fitted_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) + let fitted_multilabel_binarizer: MultiLabelBinarizer = multilabel_binarizer_fit(Y_label_rows, n_c, mlb_counts, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_nystroem: Matrix = nystroem_transform(fitted_nystroem, X_c) - matrix_free(o_nystroem) + let o_multilabel_binarizer: Matrix = multilabel_binarizer_transform(fitted_multilabel_binarizer, Y_label_rows, n_c, mlb_counts) + if o_multilabel_binarizer.rows > 0 { + if o_multilabel_binarizer.cols > 0 { sink = sink + o_multilabel_binarizer.data[0] } + } + matrix_free(o_multilabel_binarizer) } t3 = flow_now_ns() - nystroem_free(fitted_nystroem) - printf("ESTIMATOR|nystroem|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multilabel_binarizer_free(fitted_multilabel_binarizer) + printf("ESTIMATOR|multilabel_binarizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- oas_estimator (unsupervised) ---- + # ---- multinomial_nb (classification) ---- t0 = flow_now_ns() - let probe_oas_estimator: ShrunkCovariance = oas_estimator_fit(X_c) + let probe_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) t1 = flow_now_ns() + multinomial_nb_free(probe_multinomial_nb) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_oas_estimator: ShrunkCovariance = oas_estimator_fit(X_c) + let m_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) + multinomial_nb_free(m_multinomial_nb) } t1 = flow_now_ns() - printf("ESTIMATOR|oas_estimator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_multinomial_nb: MultinomialNB = multinomial_nb_fit(X_c, y_c, 3, 1.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_multinomial_nb: ptr = multinomial_nb_predict(fitted_multinomial_nb, X_c) + sink = sink + o_multinomial_nb[0] + array_free_f32(o_multinomial_nb) + } + t3 = flow_now_ns() + multinomial_nb_free(fitted_multinomial_nb) + printf("ESTIMATOR|multinomial_nb|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- omp_cv (regression) ---- + # ---- multitask_elastic_net_cv (multioutput) ---- t0 = flow_now_ns() - let probe_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) + let probe_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) t1 = flow_now_ns() - omp_cv_free(probe_omp_cv) + multitask_elastic_net_cv_free(probe_multitask_elastic_net_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) - omp_cv_free(m_omp_cv) + let m_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) + multitask_elastic_net_cv_free(m_multitask_elastic_net_cv) } t1 = flow_now_ns() - let fitted_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) + let fitted_multitask_elastic_net_cv: MultiTaskElasticNetCV = multitask_elastic_net_cv_fit(X_r, Y_multi, 10) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_omp_cv: ptr = omp_cv_predict(fitted_omp_cv, X_r) - array_free_f32(o_omp_cv) + let o_multitask_elastic_net_cv: Matrix = multitask_elastic_net_cv_predict(fitted_multitask_elastic_net_cv, X_r) + if o_multitask_elastic_net_cv.rows > 0 { + if o_multitask_elastic_net_cv.cols > 0 { sink = sink + o_multitask_elastic_net_cv.data[0] } + } + matrix_free(o_multitask_elastic_net_cv) } t3 = flow_now_ns() - omp_cv_free(fitted_omp_cv) - printf("ESTIMATOR|omp_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multitask_elastic_net_cv_free(fitted_multitask_elastic_net_cv) + printf("ESTIMATOR|multitask_elastic_net_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- one_class_svm (unsupervised) ---- + # ---- multitask_elastic_net (multioutput) ---- t0 = flow_now_ns() - let probe_one_class_svm: OneClassSVM = one_class_svm_fit(X_c, 0.5, 0.1, 100) + let probe_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) t1 = flow_now_ns() - one_class_svm_free(probe_one_class_svm) + multitask_elastic_net_free(probe_multitask_elastic_net) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_one_class_svm: OneClassSVM = one_class_svm_fit(X_c, 0.5, 0.1, 100) - one_class_svm_free(m_one_class_svm) + let m_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) + multitask_elastic_net_free(m_multitask_elastic_net) } t1 = flow_now_ns() - printf("ESTIMATOR|one_class_svm|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_multitask_elastic_net: MultiTaskElasticNet = multitask_elastic_net_fit(X_r, Y_multi, 1.0, 0.5, 50, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_multitask_elastic_net: Matrix = multitask_elastic_net_predict(fitted_multitask_elastic_net, X_r) + if o_multitask_elastic_net.rows > 0 { + if o_multitask_elastic_net.cols > 0 { sink = sink + o_multitask_elastic_net.data[0] } + } + matrix_free(o_multitask_elastic_net) + } + t3 = flow_now_ns() + multitask_elastic_net_free(fitted_multitask_elastic_net) + printf("ESTIMATOR|multitask_elastic_net|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- one_vs_one (classification) ---- + # ---- multitask_lasso_cv (multioutput) ---- t0 = flow_now_ns() - let probe_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let probe_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) t1 = flow_now_ns() - one_vs_one_free(probe_one_vs_one) + multitask_lasso_cv_free(probe_multitask_lasso_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) - one_vs_one_free(m_one_vs_one) + let m_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) + multitask_lasso_cv_free(m_multitask_lasso_cv) } t1 = flow_now_ns() - let fitted_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let fitted_multitask_lasso_cv: MultiTaskLassoCV = multitask_lasso_cv_fit(X_r, Y_multi, 10) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_one_vs_one: ptr = one_vs_one_predict(fitted_one_vs_one, X_c) - array_free_f32(o_one_vs_one) + let o_multitask_lasso_cv: Matrix = multitask_lasso_cv_predict(fitted_multitask_lasso_cv, X_r) + if o_multitask_lasso_cv.rows > 0 { + if o_multitask_lasso_cv.cols > 0 { sink = sink + o_multitask_lasso_cv.data[0] } + } + matrix_free(o_multitask_lasso_cv) } t3 = flow_now_ns() - one_vs_one_free(fitted_one_vs_one) - printf("ESTIMATOR|one_vs_one|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multitask_lasso_cv_free(fitted_multitask_lasso_cv) + printf("ESTIMATOR|multitask_lasso_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- one_vs_rest (classification) ---- + # ---- multitask_lasso (multioutput) ---- t0 = flow_now_ns() - let probe_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let probe_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) t1 = flow_now_ns() - one_vs_rest_free(probe_one_vs_rest) + multitask_lasso_free(probe_multitask_lasso) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) - one_vs_rest_free(m_one_vs_rest) + let m_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) + multitask_lasso_free(m_multitask_lasso) } t1 = flow_now_ns() - let fitted_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + let fitted_multitask_lasso: MultiTaskLasso = multitask_lasso_fit(X_r, Y_multi, 1.0, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_one_vs_rest: ptr = one_vs_rest_predict(fitted_one_vs_rest, X_c) - array_free_f32(o_one_vs_rest) + let o_multitask_lasso: Matrix = multitask_lasso_predict(fitted_multitask_lasso, X_r) + if o_multitask_lasso.rows > 0 { + if o_multitask_lasso.cols > 0 { sink = sink + o_multitask_lasso.data[0] } + } + matrix_free(o_multitask_lasso) } t3 = flow_now_ns() - one_vs_rest_free(fitted_one_vs_rest) - printf("ESTIMATOR|one_vs_rest|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + multitask_lasso_free(fitted_multitask_lasso) + printf("ESTIMATOR|multitask_lasso|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- onehot_encoder (unsupervised) ---- + # ---- nca (regression) ---- t0 = flow_now_ns() - let probe_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) + let probe_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) t1 = flow_now_ns() - onehot_encoder_free(probe_onehot_encoder) + nca_free(probe_nca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) - onehot_encoder_free(m_onehot_encoder) + let m_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) + nca_free(m_nca) } t1 = flow_now_ns() - let fitted_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) + let fitted_nca: NeighborhoodComponentsAnalysis = nca_fit(X_r, y_r, 2, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_onehot_encoder: Matrix = onehot_encoder_transform(fitted_onehot_encoder, X_c) - matrix_free(o_onehot_encoder) + let o_nca: Matrix = nca_transform(fitted_nca, X_r) + if o_nca.rows > 0 { + if o_nca.cols > 0 { sink = sink + o_nca.data[0] } + } + matrix_free(o_nca) } t3 = flow_now_ns() - onehot_encoder_free(fitted_onehot_encoder) - printf("ESTIMATOR|onehot_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nca_free(fitted_nca) + printf("ESTIMATOR|nca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- optics (unsupervised) ---- + # ---- nearest_centroid (classification) ---- t0 = flow_now_ns() - let probe_optics: OPTICS = optics_fit(X_c, 0.5, 5) + let probe_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) t1 = flow_now_ns() - optics_free(probe_optics) + nearest_centroid_free(probe_nearest_centroid) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_optics: OPTICS = optics_fit(X_c, 0.5, 5) - optics_free(m_optics) + let m_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) + nearest_centroid_free(m_nearest_centroid) } t1 = flow_now_ns() - printf("ESTIMATOR|optics|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_nearest_centroid: NearestCentroid = nearest_centroid_fit(X_c, y_c, 3) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_nearest_centroid: ptr = nearest_centroid_predict(fitted_nearest_centroid, X_c) + sink = sink + o_nearest_centroid[0] + array_free_f32(o_nearest_centroid) + } + t3 = flow_now_ns() + nearest_centroid_free(fitted_nearest_centroid) + printf("ESTIMATOR|nearest_centroid|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ordinal_encoder (unsupervised) ---- + # ---- nearest_neighbors (classification, written out) ---- t0 = flow_now_ns() - let probe_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) + let probe_nearest_neighbors: NearestNeighbors = nearest_neighbors_fit(X_c, 5) t1 = flow_now_ns() - ordinal_encoder_free(probe_ordinal_encoder) + nearest_neighbors_free(probe_nearest_neighbors) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) - ordinal_encoder_free(m_ordinal_encoder) + let m_nearest_neighbors: NearestNeighbors = nearest_neighbors_fit(X_c, 5) + nearest_neighbors_free(m_nearest_neighbors) } t1 = flow_now_ns() - let fitted_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) + let fitted_nearest_neighbors: NearestNeighbors = nearest_neighbors_fit(X_c, 5) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ordinal_encoder: Matrix = ordinal_encoder_transform(fitted_ordinal_encoder, X_c) - matrix_free(o_ordinal_encoder) + let o_nearest_neighbors: ptr = nearest_neighbors_kneighbors(fitted_nearest_neighbors, X_c) + nearest_neighbors_free_results(o_nearest_neighbors, n_c) } t3 = flow_now_ns() - ordinal_encoder_free(fitted_ordinal_encoder) - printf("ESTIMATOR|ordinal_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nearest_neighbors_free(fitted_nearest_neighbors) + printf("ESTIMATOR|nearest_neighbors|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- orthogonal_matching_pursuit (regression) ---- + # ---- nmf (unsupervised) ---- t0 = flow_now_ns() - let probe_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) + let probe_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) t1 = flow_now_ns() - orthogonal_matching_pursuit_free(probe_orthogonal_matching_pursuit) + nmf_free(probe_nmf) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) - orthogonal_matching_pursuit_free(m_orthogonal_matching_pursuit) + let m_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) + nmf_free(m_nmf) } t1 = flow_now_ns() - let fitted_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) + let fitted_nmf: NMF = nmf_fit(X_c, 2, 100, 0.0001, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_orthogonal_matching_pursuit: ptr = orthogonal_matching_pursuit_predict(fitted_orthogonal_matching_pursuit, X_r) - array_free_f32(o_orthogonal_matching_pursuit) + let o_nmf: Matrix = nmf_transform(fitted_nmf, X_c) + if o_nmf.rows > 0 { + if o_nmf.cols > 0 { sink = sink + o_nmf.data[0] } + } + matrix_free(o_nmf) } t3 = flow_now_ns() - orthogonal_matching_pursuit_free(fitted_orthogonal_matching_pursuit) - printf("ESTIMATOR|orthogonal_matching_pursuit|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nmf_free(fitted_nmf) + printf("ESTIMATOR|nmf|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- output_code (classification) ---- + # ---- nu_svc (regression) ---- t0 = flow_now_ns() - let probe_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) + let probe_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) t1 = flow_now_ns() - output_code_free(probe_output_code) + nu_svc_free(probe_nu_svc) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) - output_code_free(m_output_code) + let m_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) + nu_svc_free(m_nu_svc) } t1 = flow_now_ns() - let fitted_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) + let fitted_nu_svc: NuSVC = nu_svc_fit(X_r, y_r, 0.5, 0.1, 100, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_output_code: ptr = output_code_predict(fitted_output_code, X_c) - array_free_f32(o_output_code) + let o_nu_svc: ptr = nu_svc_predict(fitted_nu_svc, X_r) + sink = sink + o_nu_svc[0] + array_free_f32(o_nu_svc) } t3 = flow_now_ns() - output_code_free(fitted_output_code) - printf("ESTIMATOR|output_code|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nu_svc_free(fitted_nu_svc) + printf("ESTIMATOR|nu_svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- passive_aggressive_classifier (regression) ---- + # ---- nu_svr (regression) ---- t0 = flow_now_ns() - let probe_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) + let probe_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) t1 = flow_now_ns() - passive_aggressive_classifier_free(probe_passive_aggressive_classifier) + nu_svr_free(probe_nu_svr) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) - passive_aggressive_classifier_free(m_passive_aggressive_classifier) + let m_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) + nu_svr_free(m_nu_svr) } t1 = flow_now_ns() - let fitted_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) + let fitted_nu_svr: NuSVR = nu_svr_fit(X_r, y_r, 0.5, 1.0, 0.1, 100) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_passive_aggressive_classifier: ptr = passive_aggressive_classifier_predict(fitted_passive_aggressive_classifier, X_r) - array_free_f32(o_passive_aggressive_classifier) + let o_nu_svr: ptr = nu_svr_predict(fitted_nu_svr, X_r) + sink = sink + o_nu_svr[0] + array_free_f32(o_nu_svr) } t3 = flow_now_ns() - passive_aggressive_classifier_free(fitted_passive_aggressive_classifier) - printf("ESTIMATOR|passive_aggressive_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nu_svr_free(fitted_nu_svr) + printf("ESTIMATOR|nu_svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- passive_aggressive_regressor (regression) ---- + # ---- nystroem (unsupervised) ---- t0 = flow_now_ns() - let probe_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) + let probe_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) t1 = flow_now_ns() - passive_aggressive_regressor_free(probe_passive_aggressive_regressor) + nystroem_free(probe_nystroem) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) - passive_aggressive_regressor_free(m_passive_aggressive_regressor) + let m_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) + nystroem_free(m_nystroem) } t1 = flow_now_ns() - let fitted_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) + let fitted_nystroem: Nystroem = nystroem_fit(X_c, 2, 0.1, 0, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_passive_aggressive_regressor: ptr = passive_aggressive_regressor_predict(fitted_passive_aggressive_regressor, X_r) - array_free_f32(o_passive_aggressive_regressor) + let o_nystroem: Matrix = nystroem_transform(fitted_nystroem, X_c) + if o_nystroem.rows > 0 { + if o_nystroem.cols > 0 { sink = sink + o_nystroem.data[0] } + } + matrix_free(o_nystroem) } t3 = flow_now_ns() - passive_aggressive_regressor_free(fitted_passive_aggressive_regressor) - printf("ESTIMATOR|passive_aggressive_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + nystroem_free(fitted_nystroem) + printf("ESTIMATOR|nystroem|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- pca (unsupervised) ---- + # ---- oas_estimator (unsupervised) ---- t0 = flow_now_ns() - let probe_pca: PCA = pca_fit(X_c, 2) + let probe_oas_estimator: ShrunkCovariance = oas_estimator_fit(X_c) t1 = flow_now_ns() - pca_free(probe_pca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_pca: PCA = pca_fit(X_c, 2) - pca_free(m_pca) + let m_oas_estimator: ShrunkCovariance = oas_estimator_fit(X_c) } t1 = flow_now_ns() - let fitted_pca: PCA = pca_fit(X_c, 2) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_pca: Matrix = pca_transform(fitted_pca, X_c) - matrix_free(o_pca) - } - t3 = flow_now_ns() - pca_free(fitted_pca) - printf("ESTIMATOR|pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|oas_estimator|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- perceptron (regression) ---- + # ---- omp_cv (regression) ---- t0 = flow_now_ns() - let probe_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) + let probe_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) t1 = flow_now_ns() - perceptron_free(probe_perceptron) + omp_cv_free(probe_omp_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) - perceptron_free(m_perceptron) + let m_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) + omp_cv_free(m_omp_cv) } t1 = flow_now_ns() - let fitted_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) + let fitted_omp_cv: OrthogonalMatchingPursuitCV = omp_cv_fit(X_r, y_r, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_perceptron: ptr = perceptron_predict(fitted_perceptron, X_r) - array_free_f32(o_perceptron) + let o_omp_cv: ptr = omp_cv_predict(fitted_omp_cv, X_r) + sink = sink + o_omp_cv[0] + array_free_f32(o_omp_cv) } t3 = flow_now_ns() - perceptron_free(fitted_perceptron) - printf("ESTIMATOR|perceptron|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + omp_cv_free(fitted_omp_cv) + printf("ESTIMATOR|omp_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -545,5 +646,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_06.flow b/benchmarks/generated/bench_estimators_06.flow index 043cadb..cf91890 100644 --- a/benchmarks/generated/bench_estimators_06.flow +++ b/benchmarks/generated/bench_estimators_06.flow @@ -65,494 +65,564 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- pls_canonical (multioutput) ---- + # ---- one_class_svm (unsupervised) ---- t0 = flow_now_ns() - let probe_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) + let probe_one_class_svm: OneClassSVM = one_class_svm_fit(X_c, 0.5, 0.1, 100) t1 = flow_now_ns() - pls_canonical_free(probe_pls_canonical) + one_class_svm_free(probe_one_class_svm) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) - pls_canonical_free(m_pls_canonical) + let m_one_class_svm: OneClassSVM = one_class_svm_fit(X_c, 0.5, 0.1, 100) + one_class_svm_free(m_one_class_svm) } t1 = flow_now_ns() - let fitted_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_pls_canonical: Matrix = pls_canonical_transform(fitted_pls_canonical, X_r) - matrix_free(o_pls_canonical) - } - t3 = flow_now_ns() - pls_canonical_free(fitted_pls_canonical) - printf("ESTIMATOR|pls_canonical|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|one_class_svm|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- pls (multioutput) ---- + # ---- one_vs_one (classification) ---- t0 = flow_now_ns() - let probe_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) + let probe_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t1 = flow_now_ns() - pls_free(probe_pls) + one_vs_one_free(probe_one_vs_one) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) - pls_free(m_pls) + let m_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + one_vs_one_free(m_one_vs_one) } t1 = flow_now_ns() - let fitted_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) + let fitted_one_vs_one: OneVsOneClassifier = one_vs_one_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_pls: Matrix = pls_predict(fitted_pls, X_r) - matrix_free(o_pls) + let o_one_vs_one: ptr = one_vs_one_predict(fitted_one_vs_one, X_c) + sink = sink + o_one_vs_one[0] + array_free_f32(o_one_vs_one) } t3 = flow_now_ns() - pls_free(fitted_pls) - printf("ESTIMATOR|pls|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + one_vs_one_free(fitted_one_vs_one) + printf("ESTIMATOR|one_vs_one|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- pls_svd (multioutput) ---- + # ---- one_vs_rest (classification) ---- t0 = flow_now_ns() - let probe_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) + let probe_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t1 = flow_now_ns() - pls_svd_free(probe_pls_svd) + one_vs_rest_free(probe_one_vs_rest) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) - pls_svd_free(m_pls_svd) + let m_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) + one_vs_rest_free(m_one_vs_rest) } t1 = flow_now_ns() - let fitted_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) + let fitted_one_vs_rest: OneVsRestClassifier = one_vs_rest_fit(X_c, y_c, 3, 50, 0.01, penalty_none()) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_pls_svd: Matrix = pls_svd_transform(fitted_pls_svd, X_r) - matrix_free(o_pls_svd) + let o_one_vs_rest: ptr = one_vs_rest_predict(fitted_one_vs_rest, X_c) + sink = sink + o_one_vs_rest[0] + array_free_f32(o_one_vs_rest) } t3 = flow_now_ns() - pls_svd_free(fitted_pls_svd) - printf("ESTIMATOR|pls_svd|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + one_vs_rest_free(fitted_one_vs_rest) + printf("ESTIMATOR|one_vs_rest|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- poisson_regressor (regression) ---- + # ---- onehot_encoder (unsupervised) ---- t0 = flow_now_ns() - let probe_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) + let probe_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) t1 = flow_now_ns() - poisson_regressor_free(probe_poisson_regressor) + onehot_encoder_free(probe_onehot_encoder) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) - poisson_regressor_free(m_poisson_regressor) + let m_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) + onehot_encoder_free(m_onehot_encoder) } t1 = flow_now_ns() - let fitted_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) + let fitted_onehot_encoder: OneHotEncoder = onehot_encoder_fit(X_c, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_poisson_regressor: ptr = poisson_regressor_predict(fitted_poisson_regressor, X_r) - array_free_f32(o_poisson_regressor) + let o_onehot_encoder: Matrix = onehot_encoder_transform(fitted_onehot_encoder, X_c) + if o_onehot_encoder.rows > 0 { + if o_onehot_encoder.cols > 0 { sink = sink + o_onehot_encoder.data[0] } + } + matrix_free(o_onehot_encoder) } t3 = flow_now_ns() - poisson_regressor_free(fitted_poisson_regressor) - printf("ESTIMATOR|poisson_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + onehot_encoder_free(fitted_onehot_encoder) + printf("ESTIMATOR|onehot_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- power_transformer (unsupervised) ---- + # ---- optics (unsupervised) ---- t0 = flow_now_ns() - let probe_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) + let probe_optics: OPTICS = optics_fit(X_c, 0.5, 5) t1 = flow_now_ns() - power_transformer_free(probe_power_transformer) + optics_free(probe_optics) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) - power_transformer_free(m_power_transformer) + let m_optics: OPTICS = optics_fit(X_c, 0.5, 5) + optics_free(m_optics) } t1 = flow_now_ns() - let fitted_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_power_transformer: Matrix = power_transformer_transform(fitted_power_transformer, X_c) - matrix_free(o_power_transformer) - } - t3 = flow_now_ns() - power_transformer_free(fitted_power_transformer) - printf("ESTIMATOR|power_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|optics|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- qda (classification) ---- + # ---- ordinal_encoder (unsupervised) ---- t0 = flow_now_ns() - let probe_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) + let probe_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) t1 = flow_now_ns() - qda_free(probe_qda) + ordinal_encoder_free(probe_ordinal_encoder) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) - qda_free(m_qda) + let m_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) + ordinal_encoder_free(m_ordinal_encoder) } t1 = flow_now_ns() - let fitted_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) + let fitted_ordinal_encoder: OrdinalEncoder = ordinal_encoder_fit(X_c, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_qda: ptr = qda_predict(fitted_qda, X_c) - array_free_f32(o_qda) + let o_ordinal_encoder: Matrix = ordinal_encoder_transform(fitted_ordinal_encoder, X_c) + if o_ordinal_encoder.rows > 0 { + if o_ordinal_encoder.cols > 0 { sink = sink + o_ordinal_encoder.data[0] } + } + matrix_free(o_ordinal_encoder) } t3 = flow_now_ns() - qda_free(fitted_qda) - printf("ESTIMATOR|qda|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + ordinal_encoder_free(fitted_ordinal_encoder) + printf("ESTIMATOR|ordinal_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- quantile_regressor (regression) ---- + # ---- orthogonal_matching_pursuit (regression) ---- t0 = flow_now_ns() - let probe_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) + let probe_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) t1 = flow_now_ns() - quantile_regressor_free(probe_quantile_regressor) + orthogonal_matching_pursuit_free(probe_orthogonal_matching_pursuit) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) - quantile_regressor_free(m_quantile_regressor) + let m_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) + orthogonal_matching_pursuit_free(m_orthogonal_matching_pursuit) } t1 = flow_now_ns() - let fitted_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) + let fitted_orthogonal_matching_pursuit: OrthogonalMatchingPursuit = orthogonal_matching_pursuit_fit(X_r, y_r, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_quantile_regressor: ptr = quantile_regressor_predict(fitted_quantile_regressor, X_r) - array_free_f32(o_quantile_regressor) + let o_orthogonal_matching_pursuit: ptr = orthogonal_matching_pursuit_predict(fitted_orthogonal_matching_pursuit, X_r) + sink = sink + o_orthogonal_matching_pursuit[0] + array_free_f32(o_orthogonal_matching_pursuit) } t3 = flow_now_ns() - quantile_regressor_free(fitted_quantile_regressor) - printf("ESTIMATOR|quantile_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + orthogonal_matching_pursuit_free(fitted_orthogonal_matching_pursuit) + printf("ESTIMATOR|orthogonal_matching_pursuit|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- quantile_transformer (unsupervised) ---- + # ---- output_code (classification) ---- t0 = flow_now_ns() - let probe_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) + let probe_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) t1 = flow_now_ns() - quantile_transformer_free(probe_quantile_transformer) + output_code_free(probe_output_code) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) - quantile_transformer_free(m_quantile_transformer) + let m_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) + output_code_free(m_output_code) } t1 = flow_now_ns() - let fitted_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) + let fitted_output_code: OutputCodeClassifier = output_code_fit(X_c, y_c, 3, 4, 50, 0.01, penalty_none()) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_quantile_transformer: Matrix = quantile_transformer_transform(fitted_quantile_transformer, X_c) - matrix_free(o_quantile_transformer) + let o_output_code: ptr = output_code_predict(fitted_output_code, X_c) + sink = sink + o_output_code[0] + array_free_f32(o_output_code) } t3 = flow_now_ns() - quantile_transformer_free(fitted_quantile_transformer) - printf("ESTIMATOR|quantile_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + output_code_free(fitted_output_code) + printf("ESTIMATOR|output_code|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- radius_neighbors_classifier (classification) ---- + # ---- passive_aggressive_classifier (regression) ---- t0 = flow_now_ns() - let probe_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) + let probe_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) t1 = flow_now_ns() - radius_neighbors_classifier_free(probe_radius_neighbors_classifier) + passive_aggressive_classifier_free(probe_passive_aggressive_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) - radius_neighbors_classifier_free(m_radius_neighbors_classifier) + let m_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) + passive_aggressive_classifier_free(m_passive_aggressive_classifier) } t1 = flow_now_ns() - let fitted_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) + let fitted_passive_aggressive_classifier: PassiveAggressiveClassifier = passive_aggressive_classifier_fit(X_r, y_r, 50, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_radius_neighbors_classifier: ptr = radius_neighbors_classifier_predict(fitted_radius_neighbors_classifier, X_c) - array_free_f32(o_radius_neighbors_classifier) + let o_passive_aggressive_classifier: ptr = passive_aggressive_classifier_predict(fitted_passive_aggressive_classifier, X_r) + sink = sink + o_passive_aggressive_classifier[0] + array_free_f32(o_passive_aggressive_classifier) } t3 = flow_now_ns() - radius_neighbors_classifier_free(fitted_radius_neighbors_classifier) - printf("ESTIMATOR|radius_neighbors_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + passive_aggressive_classifier_free(fitted_passive_aggressive_classifier) + printf("ESTIMATOR|passive_aggressive_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- radius_neighbors_regressor (regression) ---- + # ---- passive_aggressive_regressor (regression) ---- t0 = flow_now_ns() - let probe_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) + let probe_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) t1 = flow_now_ns() - radius_neighbors_regressor_free(probe_radius_neighbors_regressor) + passive_aggressive_regressor_free(probe_passive_aggressive_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) - radius_neighbors_regressor_free(m_radius_neighbors_regressor) + let m_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) + passive_aggressive_regressor_free(m_passive_aggressive_regressor) } t1 = flow_now_ns() - let fitted_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) + let fitted_passive_aggressive_regressor: PassiveAggressiveRegressor = passive_aggressive_regressor_fit(X_r, y_r, 1.0, 100) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_radius_neighbors_regressor: ptr = radius_neighbors_regressor_predict(fitted_radius_neighbors_regressor, X_r) - array_free_f32(o_radius_neighbors_regressor) + let o_passive_aggressive_regressor: ptr = passive_aggressive_regressor_predict(fitted_passive_aggressive_regressor, X_r) + sink = sink + o_passive_aggressive_regressor[0] + array_free_f32(o_passive_aggressive_regressor) } t3 = flow_now_ns() - radius_neighbors_regressor_free(fitted_radius_neighbors_regressor) - printf("ESTIMATOR|radius_neighbors_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + passive_aggressive_regressor_free(fitted_passive_aggressive_regressor) + printf("ESTIMATOR|passive_aggressive_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- radius_neighbors_transformer (unsupervised) ---- + # ---- pca (unsupervised) ---- t0 = flow_now_ns() - let probe_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) + let probe_pca: PCA = pca_fit(X_c, 2) t1 = flow_now_ns() - radius_neighbors_transformer_free(probe_radius_neighbors_transformer) + pca_free(probe_pca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) - radius_neighbors_transformer_free(m_radius_neighbors_transformer) + let m_pca: PCA = pca_fit(X_c, 2) + pca_free(m_pca) } t1 = flow_now_ns() - let fitted_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) + let fitted_pca: PCA = pca_fit(X_c, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_radius_neighbors_transformer: Matrix = radius_neighbors_transformer_transform(fitted_radius_neighbors_transformer, X_c) - matrix_free(o_radius_neighbors_transformer) + let o_pca: Matrix = pca_transform(fitted_pca, X_c) + if o_pca.rows > 0 { + if o_pca.cols > 0 { sink = sink + o_pca.data[0] } + } + matrix_free(o_pca) } t3 = flow_now_ns() - radius_neighbors_transformer_free(fitted_radius_neighbors_transformer) - printf("ESTIMATOR|radius_neighbors_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + pca_free(fitted_pca) + printf("ESTIMATOR|pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- random_forest_classifier (classification) ---- + # ---- perceptron (regression) ---- t0 = flow_now_ns() - let probe_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) + let probe_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) t1 = flow_now_ns() - random_forest_classifier_free(probe_random_forest_classifier) + perceptron_free(probe_perceptron) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) - random_forest_classifier_free(m_random_forest_classifier) + let m_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) + perceptron_free(m_perceptron) } t1 = flow_now_ns() - let fitted_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) + let fitted_perceptron: Perceptron = perceptron_fit(X_r, y_r, 50, 0.01, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_random_forest_classifier: ptr = random_forest_classifier_predict(fitted_random_forest_classifier, X_c) - array_free_f32(o_random_forest_classifier) + let o_perceptron: ptr = perceptron_predict(fitted_perceptron, X_r) + sink = sink + o_perceptron[0] + array_free_f32(o_perceptron) } t3 = flow_now_ns() - random_forest_classifier_free(fitted_random_forest_classifier) - printf("ESTIMATOR|random_forest_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + perceptron_free(fitted_perceptron) + printf("ESTIMATOR|perceptron|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- random_forest_regressor (regression) ---- + # ---- pipeline (classification, written out) ---- + let pipe_steps: array = [ + step_standard_scaler("scaler"), + step_logistic_regression("classifier", 3, 50, 0.5, penalty_none()) + ] + let pipe_obj: Pipeline = pipeline_new(pipe_steps, 2) t0 = flow_now_ns() - let probe_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) + let probe_pipeline: Pipeline = pipeline_fit(pipe_obj, X_c, y_c) t1 = flow_now_ns() - random_forest_regressor_free(probe_random_forest_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) - random_forest_regressor_free(m_random_forest_regressor) + let m_pipeline: Pipeline = pipeline_fit(pipe_obj, X_c, y_c) } t1 = flow_now_ns() - let fitted_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) + let fitted_pipeline: Pipeline = pipeline_fit(pipe_obj, X_c, y_c) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_random_forest_regressor: ptr = random_forest_regressor_predict(fitted_random_forest_regressor, X_r) - array_free_f32(o_random_forest_regressor) + let o_pipeline: ptr = pipeline_predict(fitted_pipeline, X_c) + sink = sink + o_pipeline[0] + array_free_f32(o_pipeline) } t3 = flow_now_ns() - random_forest_regressor_free(fitted_random_forest_regressor) - printf("ESTIMATOR|random_forest_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + pipeline_free(fitted_pipeline) + printf("ESTIMATOR|pipeline|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- random_trees_embedding (unsupervised) ---- + # ---- pls_canonical (multioutput) ---- t0 = flow_now_ns() - let probe_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) + let probe_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) t1 = flow_now_ns() - random_trees_embedding_free(probe_random_trees_embedding) + pls_canonical_free(probe_pls_canonical) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) - random_trees_embedding_free(m_random_trees_embedding) + let m_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) + pls_canonical_free(m_pls_canonical) } t1 = flow_now_ns() - let fitted_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) + let fitted_pls_canonical: PLSCanonical = pls_canonical_fit(X_r, Y_multi, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_random_trees_embedding: Matrix = random_trees_embedding_transform(fitted_random_trees_embedding, X_c) - matrix_free(o_random_trees_embedding) + let o_pls_canonical: Matrix = pls_canonical_transform(fitted_pls_canonical, X_r) + if o_pls_canonical.rows > 0 { + if o_pls_canonical.cols > 0 { sink = sink + o_pls_canonical.data[0] } + } + matrix_free(o_pls_canonical) } t3 = flow_now_ns() - random_trees_embedding_free(fitted_random_trees_embedding) - printf("ESTIMATOR|random_trees_embedding|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + pls_canonical_free(fitted_pls_canonical) + printf("ESTIMATOR|pls_canonical|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ransac_regressor (regression) ---- + # ---- pls (multioutput) ---- t0 = flow_now_ns() - let probe_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) + let probe_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) t1 = flow_now_ns() - ransac_regressor_free(probe_ransac_regressor) + pls_free(probe_pls) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) - ransac_regressor_free(m_ransac_regressor) + let m_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) + pls_free(m_pls) } t1 = flow_now_ns() - let fitted_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) + let fitted_pls: PLSRegression = pls_fit(X_r, Y_multi, 2, 100, 0.0001) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ransac_regressor: ptr = ransac_regressor_predict(fitted_ransac_regressor, X_r) - array_free_f32(o_ransac_regressor) + let o_pls: Matrix = pls_predict(fitted_pls, X_r) + if o_pls.rows > 0 { + if o_pls.cols > 0 { sink = sink + o_pls.data[0] } + } + matrix_free(o_pls) } t3 = flow_now_ns() - ransac_regressor_free(fitted_ransac_regressor) - printf("ESTIMATOR|ransac_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + pls_free(fitted_pls) + printf("ESTIMATOR|pls|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- regressor_chain (multioutput_class) ---- + # ---- pls_svd (multioutput) ---- t0 = flow_now_ns() - let probe_regressor_chain: RegressorChain = regressor_chain_fit(X_c, Y_label_rows, n_c, f_c, 2) + let probe_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) t1 = flow_now_ns() - regressor_chain_free(probe_regressor_chain) + pls_svd_free(probe_pls_svd) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_regressor_chain: RegressorChain = regressor_chain_fit(X_c, Y_label_rows, n_c, f_c, 2) - regressor_chain_free(m_regressor_chain) + let m_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) + pls_svd_free(m_pls_svd) } t1 = flow_now_ns() - printf("ESTIMATOR|regressor_chain|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_pls_svd: PLSSVD = pls_svd_fit(X_r, Y_multi, 2) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_pls_svd: Matrix = pls_svd_transform(fitted_pls_svd, X_r) + if o_pls_svd.rows > 0 { + if o_pls_svd.cols > 0 { sink = sink + o_pls_svd.data[0] } + } + matrix_free(o_pls_svd) + } + t3 = flow_now_ns() + pls_svd_free(fitted_pls_svd) + printf("ESTIMATOR|pls_svd|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- rfe (regression) ---- + # ---- poisson_regressor (regression) ---- t0 = flow_now_ns() - let probe_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) + let probe_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) t1 = flow_now_ns() - rfe_free(probe_rfe) + poisson_regressor_free(probe_poisson_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) - rfe_free(m_rfe) + let m_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) + poisson_regressor_free(m_poisson_regressor) } t1 = flow_now_ns() - let fitted_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) + let fitted_poisson_regressor: PoissonRegressor = poisson_regressor_fit(X_r, y_r, 1.0, 100, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_rfe: Matrix = rfe_transform(fitted_rfe, X_r) - matrix_free(o_rfe) + let o_poisson_regressor: ptr = poisson_regressor_predict(fitted_poisson_regressor, X_r) + sink = sink + o_poisson_regressor[0] + array_free_f32(o_poisson_regressor) } t3 = flow_now_ns() - rfe_free(fitted_rfe) - printf("ESTIMATOR|rfe|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + poisson_regressor_free(fitted_poisson_regressor) + printf("ESTIMATOR|poisson_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- rfecv (regression) ---- + # ---- polynomial_count_sketch (classification, written out) ---- t0 = flow_now_ns() - let probe_rfecv: RFECV = rfecv_fit(X_r, y_r, n_r, f_r, 3, null) + let probe_polynomial_count_sketch: PolynomialCountSketch = polynomial_count_sketch_fit(f_c, 2, 2) t1 = flow_now_ns() - rfecv_free(probe_rfecv) + polynomial_count_sketch_free(probe_polynomial_count_sketch) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_rfecv: RFECV = rfecv_fit(X_r, y_r, n_r, f_r, 3, null) - rfecv_free(m_rfecv) + let m_polynomial_count_sketch: PolynomialCountSketch = polynomial_count_sketch_fit(f_c, 2, 2) + polynomial_count_sketch_free(m_polynomial_count_sketch) } t1 = flow_now_ns() - printf("ESTIMATOR|rfecv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_polynomial_count_sketch: PolynomialCountSketch = polynomial_count_sketch_fit(f_c, 2, 2) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_polynomial_count_sketch: Matrix = polynomial_count_sketch_transform(fitted_polynomial_count_sketch, X_c) + if o_polynomial_count_sketch.rows > 0 { + if o_polynomial_count_sketch.cols > 0 { sink = sink + o_polynomial_count_sketch.data[0] } + } + matrix_free(o_polynomial_count_sketch) + } + t3 = flow_now_ns() + polynomial_count_sketch_free(fitted_polynomial_count_sketch) + printf("ESTIMATOR|polynomial_count_sketch|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ridge_classifier_cv (classification) ---- + # ---- polynomial_features (classification, written out) ---- t0 = flow_now_ns() - let probe_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) + let probe_polynomial_features: PolynomialFeatures = polynomial_features_fit(f_c, 2, false, true) t1 = flow_now_ns() - ridge_classifier_cv_free(probe_ridge_classifier_cv) + polynomial_features_free(probe_polynomial_features) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) - ridge_classifier_cv_free(m_ridge_classifier_cv) + let m_polynomial_features: PolynomialFeatures = polynomial_features_fit(f_c, 2, false, true) + polynomial_features_free(m_polynomial_features) } t1 = flow_now_ns() - let fitted_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) + let fitted_polynomial_features: PolynomialFeatures = polynomial_features_fit(f_c, 2, false, true) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ridge_classifier_cv: ptr = ridge_classifier_cv_predict(fitted_ridge_classifier_cv, X_c) - array_free_f32(o_ridge_classifier_cv) + let o_polynomial_features: Matrix = polynomial_features_transform(fitted_polynomial_features, X_c) + if o_polynomial_features.rows > 0 { + if o_polynomial_features.cols > 0 { sink = sink + o_polynomial_features.data[0] } + } + matrix_free(o_polynomial_features) } t3 = flow_now_ns() - ridge_classifier_cv_free(fitted_ridge_classifier_cv) - printf("ESTIMATOR|ridge_classifier_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + polynomial_features_free(fitted_polynomial_features) + printf("ESTIMATOR|polynomial_features|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ridge_classifier (regression) ---- + # ---- power_transformer (unsupervised) ---- t0 = flow_now_ns() - let probe_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) + let probe_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) t1 = flow_now_ns() - ridge_classifier_free(probe_ridge_classifier) + power_transformer_free(probe_power_transformer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) - ridge_classifier_free(m_ridge_classifier) + let m_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) + power_transformer_free(m_power_transformer) } t1 = flow_now_ns() - let fitted_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) + let fitted_power_transformer: PowerTransformer = power_transformer_fit(X_c, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ridge_classifier: ptr = ridge_classifier_predict(fitted_ridge_classifier, X_r) - array_free_f32(o_ridge_classifier) + let o_power_transformer: Matrix = power_transformer_transform(fitted_power_transformer, X_c) + if o_power_transformer.rows > 0 { + if o_power_transformer.cols > 0 { sink = sink + o_power_transformer.data[0] } + } + matrix_free(o_power_transformer) } t3 = flow_now_ns() - ridge_classifier_free(fitted_ridge_classifier) - printf("ESTIMATOR|ridge_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + power_transformer_free(fitted_power_transformer) + printf("ESTIMATOR|power_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -563,5 +633,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_07.flow b/benchmarks/generated/bench_estimators_07.flow index 17e83a2..d96f2eb 100644 --- a/benchmarks/generated/bench_estimators_07.flow +++ b/benchmarks/generated/bench_estimators_07.flow @@ -65,470 +65,557 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- ridge_cv (regression) ---- + # ---- qda (classification) ---- t0 = flow_now_ns() - let probe_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) + let probe_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) t1 = flow_now_ns() - ridge_cv_free(probe_ridge_cv) + qda_free(probe_qda) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) - ridge_cv_free(m_ridge_cv) + let m_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) + qda_free(m_qda) } t1 = flow_now_ns() - let fitted_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) + let fitted_qda: QuadraticDiscriminantAnalysis = qda_fit(X_c, y_c, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ridge_cv: ptr = ridge_cv_predict(fitted_ridge_cv, X_r) - array_free_f32(o_ridge_cv) + let o_qda: ptr = qda_predict(fitted_qda, X_c) + sink = sink + o_qda[0] + array_free_f32(o_qda) } t3 = flow_now_ns() - ridge_cv_free(fitted_ridge_cv) - printf("ESTIMATOR|ridge_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + qda_free(fitted_qda) + printf("ESTIMATOR|qda|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- ridge (regression) ---- + # ---- quantile_regressor (regression) ---- t0 = flow_now_ns() - let probe_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) + let probe_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) t1 = flow_now_ns() - ridge_free(probe_ridge) + quantile_regressor_free(probe_quantile_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) - ridge_free(m_ridge) + let m_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) + quantile_regressor_free(m_quantile_regressor) } t1 = flow_now_ns() - let fitted_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) + let fitted_quantile_regressor: QuantileRegressor = quantile_regressor_fit(X_r, y_r, 0.5, 1.0, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_ridge: ptr = ridge_predict(fitted_ridge, X_r) - array_free_f32(o_ridge) + let o_quantile_regressor: ptr = quantile_regressor_predict(fitted_quantile_regressor, X_r) + sink = sink + o_quantile_regressor[0] + array_free_f32(o_quantile_regressor) } t3 = flow_now_ns() - ridge_free(fitted_ridge) - printf("ESTIMATOR|ridge|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + quantile_regressor_free(fitted_quantile_regressor) + printf("ESTIMATOR|quantile_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- robust_scaler (unsupervised) ---- + # ---- quantile_transformer (unsupervised) ---- t0 = flow_now_ns() - let probe_robust_scaler: RobustScaler = robust_scaler_fit(X_c) + let probe_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) t1 = flow_now_ns() - robust_scaler_free(probe_robust_scaler) + quantile_transformer_free(probe_quantile_transformer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_robust_scaler: RobustScaler = robust_scaler_fit(X_c) - robust_scaler_free(m_robust_scaler) + let m_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) + quantile_transformer_free(m_quantile_transformer) } t1 = flow_now_ns() - let fitted_robust_scaler: RobustScaler = robust_scaler_fit(X_c) + let fitted_quantile_transformer: QuantileTransformer = quantile_transformer_fit(X_c, 10, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_robust_scaler: Matrix = robust_scaler_transform(fitted_robust_scaler, X_c) - matrix_free(o_robust_scaler) + let o_quantile_transformer: Matrix = quantile_transformer_transform(fitted_quantile_transformer, X_c) + if o_quantile_transformer.rows > 0 { + if o_quantile_transformer.cols > 0 { sink = sink + o_quantile_transformer.data[0] } + } + matrix_free(o_quantile_transformer) } t3 = flow_now_ns() - robust_scaler_free(fitted_robust_scaler) - printf("ESTIMATOR|robust_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + quantile_transformer_free(fitted_quantile_transformer) + printf("ESTIMATOR|quantile_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- select_fdr (regression) ---- + # ---- radius_neighbors_classifier (classification) ---- t0 = flow_now_ns() - let probe_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) + let probe_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) t1 = flow_now_ns() - select_fdr_free(probe_select_fdr) + radius_neighbors_classifier_free(probe_radius_neighbors_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) - select_fdr_free(m_select_fdr) + let m_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) + radius_neighbors_classifier_free(m_radius_neighbors_classifier) } t1 = flow_now_ns() - let fitted_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) + let fitted_radius_neighbors_classifier: RadiusNeighborsClassifier = radius_neighbors_classifier_fit(X_c, y_c, 1.0, 3, 0.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_select_fdr: Matrix = select_fdr_transform(fitted_select_fdr, X_r) - matrix_free(o_select_fdr) + let o_radius_neighbors_classifier: ptr = radius_neighbors_classifier_predict(fitted_radius_neighbors_classifier, X_c) + sink = sink + o_radius_neighbors_classifier[0] + array_free_f32(o_radius_neighbors_classifier) } t3 = flow_now_ns() - select_fdr_free(fitted_select_fdr) - printf("ESTIMATOR|select_fdr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + radius_neighbors_classifier_free(fitted_radius_neighbors_classifier) + printf("ESTIMATOR|radius_neighbors_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- select_fpr (regression) ---- + # ---- radius_neighbors_regressor (regression) ---- t0 = flow_now_ns() - let probe_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) + let probe_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) t1 = flow_now_ns() - select_fpr_free(probe_select_fpr) + radius_neighbors_regressor_free(probe_radius_neighbors_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) - select_fpr_free(m_select_fpr) + let m_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) + radius_neighbors_regressor_free(m_radius_neighbors_regressor) } t1 = flow_now_ns() - let fitted_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) + let fitted_radius_neighbors_regressor: RadiusNeighborsRegressor = radius_neighbors_regressor_fit(X_r, y_r, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_select_fpr: Matrix = select_fpr_transform(fitted_select_fpr, X_r) - matrix_free(o_select_fpr) + let o_radius_neighbors_regressor: ptr = radius_neighbors_regressor_predict(fitted_radius_neighbors_regressor, X_r) + sink = sink + o_radius_neighbors_regressor[0] + array_free_f32(o_radius_neighbors_regressor) } t3 = flow_now_ns() - select_fpr_free(fitted_select_fpr) - printf("ESTIMATOR|select_fpr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + radius_neighbors_regressor_free(fitted_radius_neighbors_regressor) + printf("ESTIMATOR|radius_neighbors_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- select_fwe (regression) ---- + # ---- radius_neighbors_transformer (unsupervised) ---- t0 = flow_now_ns() - let probe_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) + let probe_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) t1 = flow_now_ns() - select_fwe_free(probe_select_fwe) + radius_neighbors_transformer_free(probe_radius_neighbors_transformer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) - select_fwe_free(m_select_fwe) + let m_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) + radius_neighbors_transformer_free(m_radius_neighbors_transformer) } t1 = flow_now_ns() - let fitted_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) + let fitted_radius_neighbors_transformer: RadiusNeighborsTransformer = radius_neighbors_transformer_fit(X_c, 1.0, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_select_fwe: Matrix = select_fwe_transform(fitted_select_fwe, X_r) - matrix_free(o_select_fwe) + let o_radius_neighbors_transformer: Matrix = radius_neighbors_transformer_transform(fitted_radius_neighbors_transformer, X_c) + if o_radius_neighbors_transformer.rows > 0 { + if o_radius_neighbors_transformer.cols > 0 { sink = sink + o_radius_neighbors_transformer.data[0] } + } + matrix_free(o_radius_neighbors_transformer) } t3 = flow_now_ns() - select_fwe_free(fitted_select_fwe) - printf("ESTIMATOR|select_fwe|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + radius_neighbors_transformer_free(fitted_radius_neighbors_transformer) + printf("ESTIMATOR|radius_neighbors_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- select_k_best (regression) ---- + # ---- random_forest_classifier (classification) ---- t0 = flow_now_ns() - let probe_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) + let probe_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) t1 = flow_now_ns() - select_k_best_free(probe_select_k_best) + random_forest_classifier_free(probe_random_forest_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) - select_k_best_free(m_select_k_best) + let m_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) + random_forest_classifier_free(m_random_forest_classifier) } t1 = flow_now_ns() - let fitted_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) + let fitted_random_forest_classifier: RandomForestClassifier = random_forest_classifier_fit(X_c, y_c, 3, 10, 5, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_select_k_best: Matrix = select_k_best_transform(fitted_select_k_best, X_r) - matrix_free(o_select_k_best) + let o_random_forest_classifier: ptr = random_forest_classifier_predict(fitted_random_forest_classifier, X_c) + sink = sink + o_random_forest_classifier[0] + array_free_f32(o_random_forest_classifier) } t3 = flow_now_ns() - select_k_best_free(fitted_select_k_best) - printf("ESTIMATOR|select_k_best|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + random_forest_classifier_free(fitted_random_forest_classifier) + printf("ESTIMATOR|random_forest_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- select_percentile (regression) ---- + # ---- random_forest_regressor (regression) ---- t0 = flow_now_ns() - let probe_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) + let probe_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) t1 = flow_now_ns() - select_percentile_free(probe_select_percentile) + random_forest_regressor_free(probe_random_forest_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) - select_percentile_free(m_select_percentile) + let m_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) + random_forest_regressor_free(m_random_forest_regressor) } t1 = flow_now_ns() - let fitted_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) + let fitted_random_forest_regressor: RandomForestRegressor = random_forest_regressor_fit(X_r, y_r, 10, 5, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_select_percentile: Matrix = select_percentile_transform(fitted_select_percentile, X_r) - matrix_free(o_select_percentile) + let o_random_forest_regressor: ptr = random_forest_regressor_predict(fitted_random_forest_regressor, X_r) + sink = sink + o_random_forest_regressor[0] + array_free_f32(o_random_forest_regressor) } t3 = flow_now_ns() - select_percentile_free(fitted_select_percentile) - printf("ESTIMATOR|select_percentile|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + random_forest_regressor_free(fitted_random_forest_regressor) + printf("ESTIMATOR|random_forest_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- sequential_feature_selector (regression) ---- + # ---- random_trees_embedding (unsupervised) ---- t0 = flow_now_ns() - let probe_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) + let probe_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) t1 = flow_now_ns() - sequential_feature_selector_free(probe_sequential_feature_selector) + random_trees_embedding_free(probe_random_trees_embedding) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) - sequential_feature_selector_free(m_sequential_feature_selector) + let m_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) + random_trees_embedding_free(m_random_trees_embedding) } t1 = flow_now_ns() - let fitted_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) + let fitted_random_trees_embedding: RandomTreesEmbedding = random_trees_embedding_fit(X_c, 10, 5, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_sequential_feature_selector: Matrix = sequential_feature_selector_transform(fitted_sequential_feature_selector, X_r) - matrix_free(o_sequential_feature_selector) + let o_random_trees_embedding: Matrix = random_trees_embedding_transform(fitted_random_trees_embedding, X_c) + if o_random_trees_embedding.rows > 0 { + if o_random_trees_embedding.cols > 0 { sink = sink + o_random_trees_embedding.data[0] } + } + matrix_free(o_random_trees_embedding) } t3 = flow_now_ns() - sequential_feature_selector_free(fitted_sequential_feature_selector) - printf("ESTIMATOR|sequential_feature_selector|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + random_trees_embedding_free(fitted_random_trees_embedding) + printf("ESTIMATOR|random_trees_embedding|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- sgd_classifier (classification) ---- + # ---- ransac_regressor (regression) ---- t0 = flow_now_ns() - let probe_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) + let probe_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) t1 = flow_now_ns() - sgd_classifier_free(probe_sgd_classifier) + ransac_regressor_free(probe_ransac_regressor) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) - sgd_classifier_free(m_sgd_classifier) + let m_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) + ransac_regressor_free(m_ransac_regressor) } t1 = flow_now_ns() - let fitted_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) + let fitted_ransac_regressor: RANSACRegressor = ransac_regressor_fit(X_r, y_r, 5, 10, 1.0, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_sgd_classifier: ptr = sgd_classifier_predict(fitted_sgd_classifier, X_c) - array_free_f32(o_sgd_classifier) + let o_ransac_regressor: ptr = ransac_regressor_predict(fitted_ransac_regressor, X_r) + sink = sink + o_ransac_regressor[0] + array_free_f32(o_ransac_regressor) } t3 = flow_now_ns() - sgd_classifier_free(fitted_sgd_classifier) - printf("ESTIMATOR|sgd_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + ransac_regressor_free(fitted_ransac_regressor) + printf("ESTIMATOR|ransac_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- sgd_one_class_svm (unsupervised) ---- + # ---- rbf_sampler (classification, written out) ---- t0 = flow_now_ns() - let probe_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) + let probe_rbf_sampler: RBFSampler = rbf_sampler_fit(f_c, 0.1, 2, 42) t1 = flow_now_ns() - sgd_one_class_svm_free(probe_sgd_one_class_svm) + rbf_sampler_free(probe_rbf_sampler) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) - sgd_one_class_svm_free(m_sgd_one_class_svm) + let m_rbf_sampler: RBFSampler = rbf_sampler_fit(f_c, 0.1, 2, 42) + rbf_sampler_free(m_rbf_sampler) } t1 = flow_now_ns() - let fitted_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) + let fitted_rbf_sampler: RBFSampler = rbf_sampler_fit(f_c, 0.1, 2, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_sgd_one_class_svm: ptr = sgd_one_class_svm_predict(fitted_sgd_one_class_svm, X_c) - array_free_f32(o_sgd_one_class_svm) + let o_rbf_sampler: Matrix = rbf_sampler_transform(fitted_rbf_sampler, X_c) + if o_rbf_sampler.rows > 0 { + if o_rbf_sampler.cols > 0 { sink = sink + o_rbf_sampler.data[0] } + } + matrix_free(o_rbf_sampler) } t3 = flow_now_ns() - sgd_one_class_svm_free(fitted_sgd_one_class_svm) - printf("ESTIMATOR|sgd_one_class_svm|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + rbf_sampler_free(fitted_rbf_sampler) + printf("ESTIMATOR|rbf_sampler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- sgd_regressor (regression) ---- + # ---- regressor_chain (multioutput_class) ---- t0 = flow_now_ns() - let probe_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) + let probe_regressor_chain: RegressorChain = regressor_chain_fit(X_c, Y_label_rows, n_c, f_c, 2) t1 = flow_now_ns() - sgd_regressor_free(probe_sgd_regressor) + regressor_chain_free(probe_regressor_chain) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) - sgd_regressor_free(m_sgd_regressor) + let m_regressor_chain: RegressorChain = regressor_chain_fit(X_c, Y_label_rows, n_c, f_c, 2) + regressor_chain_free(m_regressor_chain) } t1 = flow_now_ns() - let fitted_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_sgd_regressor: ptr = sgd_regressor_predict(fitted_sgd_regressor, X_r) - array_free_f32(o_sgd_regressor) - } - t3 = flow_now_ns() - sgd_regressor_free(fitted_sgd_regressor) - printf("ESTIMATOR|sgd_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|regressor_chain|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- shrunk_covariance (unsupervised) ---- + # ---- rfe (regression) ---- t0 = flow_now_ns() - let probe_shrunk_covariance: ShrunkCovariance = shrunk_covariance_fit(X_c, 0.1) + let probe_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) t1 = flow_now_ns() - shrunk_covariance_free(probe_shrunk_covariance) + rfe_free(probe_rfe) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_shrunk_covariance: ShrunkCovariance = shrunk_covariance_fit(X_c, 0.1) - shrunk_covariance_free(m_shrunk_covariance) + let m_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) + rfe_free(m_rfe) } t1 = flow_now_ns() - printf("ESTIMATOR|shrunk_covariance|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_rfe: RFE = rfe_fit(X_r, y_r, 2, null, n_r, f_r) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_rfe: Matrix = rfe_transform(fitted_rfe, X_r) + if o_rfe.rows > 0 { + if o_rfe.cols > 0 { sink = sink + o_rfe.data[0] } + } + matrix_free(o_rfe) + } + t3 = flow_now_ns() + rfe_free(fitted_rfe) + printf("ESTIMATOR|rfe|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- simple_imputer (unsupervised) ---- + # ---- rfecv (regression) ---- t0 = flow_now_ns() - let probe_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) + let probe_rfecv: RFECV = rfecv_fit(X_r, y_r, n_r, f_r, 3, null) t1 = flow_now_ns() - simple_imputer_free(probe_simple_imputer) + rfecv_free(probe_rfecv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) - simple_imputer_free(m_simple_imputer) + let m_rfecv: RFECV = rfecv_fit(X_r, y_r, n_r, f_r, 3, null) + rfecv_free(m_rfecv) } t1 = flow_now_ns() - let fitted_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) - t2 = flow_now_ns() - for rep2 in 0 to reps { - let o_simple_imputer: Matrix = simple_imputer_transform(fitted_simple_imputer, X_c) - matrix_free(o_simple_imputer) - } - t3 = flow_now_ns() - simple_imputer_free(fitted_simple_imputer) - printf("ESTIMATOR|simple_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + printf("ESTIMATOR|rfecv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- sparse_coder (unsupervised) ---- + # ---- ridge_classifier_cv (classification) ---- t0 = flow_now_ns() - let probe_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) + let probe_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) t1 = flow_now_ns() - sparse_coder_free(probe_sparse_coder) + ridge_classifier_cv_free(probe_ridge_classifier_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) - sparse_coder_free(m_sparse_coder) + let m_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) + ridge_classifier_cv_free(m_ridge_classifier_cv) } t1 = flow_now_ns() - let fitted_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) + let fitted_ridge_classifier_cv: RidgeClassifierCV = ridge_classifier_cv_fit(X_c, y_c, 3, 10) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_sparse_coder: Matrix = sparse_coder_transform(fitted_sparse_coder, X_c) - matrix_free(o_sparse_coder) + let o_ridge_classifier_cv: ptr = ridge_classifier_cv_predict(fitted_ridge_classifier_cv, X_c) + sink = sink + o_ridge_classifier_cv[0] + array_free_f32(o_ridge_classifier_cv) } t3 = flow_now_ns() - sparse_coder_free(fitted_sparse_coder) - printf("ESTIMATOR|sparse_coder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + ridge_classifier_cv_free(fitted_ridge_classifier_cv) + printf("ESTIMATOR|ridge_classifier_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- sparse_pca (unsupervised) ---- + # ---- ridge_classifier (regression) ---- t0 = flow_now_ns() - let probe_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) + let probe_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) t1 = flow_now_ns() - sparse_pca_free(probe_sparse_pca) + ridge_classifier_free(probe_ridge_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) - sparse_pca_free(m_sparse_pca) + let m_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) + ridge_classifier_free(m_ridge_classifier) } t1 = flow_now_ns() - let fitted_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) + let fitted_ridge_classifier: RidgeClassifier = ridge_classifier_fit(X_r, y_r, 1.0, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_sparse_pca: Matrix = sparse_pca_transform(fitted_sparse_pca, X_c) - matrix_free(o_sparse_pca) + let o_ridge_classifier: ptr = ridge_classifier_predict(fitted_ridge_classifier, X_r) + sink = sink + o_ridge_classifier[0] + array_free_f32(o_ridge_classifier) } t3 = flow_now_ns() - sparse_pca_free(fitted_sparse_pca) - printf("ESTIMATOR|sparse_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + ridge_classifier_free(fitted_ridge_classifier) + printf("ESTIMATOR|ridge_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- spectral_biclustering (unsupervised) ---- + # ---- ridge_cv (regression) ---- t0 = flow_now_ns() - let probe_spectral_biclustering: SpectralBiclustering = spectral_biclustering_fit(X_c, 2, 2) + let probe_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) t1 = flow_now_ns() - spectral_biclustering_free(probe_spectral_biclustering) + ridge_cv_free(probe_ridge_cv) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_spectral_biclustering: SpectralBiclustering = spectral_biclustering_fit(X_c, 2, 2) - spectral_biclustering_free(m_spectral_biclustering) + let m_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) + ridge_cv_free(m_ridge_cv) } t1 = flow_now_ns() - printf("ESTIMATOR|spectral_biclustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_ridge_cv: RidgeCV = ridge_cv_fit(X_r, y_r, 10) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_ridge_cv: ptr = ridge_cv_predict(fitted_ridge_cv, X_r) + sink = sink + o_ridge_cv[0] + array_free_f32(o_ridge_cv) + } + t3 = flow_now_ns() + ridge_cv_free(fitted_ridge_cv) + printf("ESTIMATOR|ridge_cv|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- spectral_clustering (unsupervised) ---- + # ---- ridge (regression) ---- t0 = flow_now_ns() - let probe_spectral_clustering: SpectralClustering = spectral_clustering_fit(X_c, 3, 0.1, 42) + let probe_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) t1 = flow_now_ns() - spectral_clustering_free(probe_spectral_clustering) + ridge_free(probe_ridge) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_spectral_clustering: SpectralClustering = spectral_clustering_fit(X_c, 3, 0.1, 42) - spectral_clustering_free(m_spectral_clustering) + let m_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) + ridge_free(m_ridge) } t1 = flow_now_ns() - printf("ESTIMATOR|spectral_clustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_ridge: Ridge = ridge_fit(X_r, y_r, 1.0, 50, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_ridge: ptr = ridge_predict(fitted_ridge, X_r) + sink = sink + o_ridge[0] + array_free_f32(o_ridge) + } + t3 = flow_now_ns() + ridge_free(fitted_ridge) + printf("ESTIMATOR|ridge|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- spectral_coclustering (unsupervised) ---- + # ---- robust_scaler (unsupervised) ---- t0 = flow_now_ns() - let probe_spectral_coclustering: SpectralCoclustering = spectral_coclustering_fit(X_c, 3) + let probe_robust_scaler: RobustScaler = robust_scaler_fit(X_c) t1 = flow_now_ns() - spectral_coclustering_free(probe_spectral_coclustering) + robust_scaler_free(probe_robust_scaler) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_spectral_coclustering: SpectralCoclustering = spectral_coclustering_fit(X_c, 3) - spectral_coclustering_free(m_spectral_coclustering) + let m_robust_scaler: RobustScaler = robust_scaler_fit(X_c) + robust_scaler_free(m_robust_scaler) } t1 = flow_now_ns() - printf("ESTIMATOR|spectral_coclustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_robust_scaler: RobustScaler = robust_scaler_fit(X_c) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_robust_scaler: Matrix = robust_scaler_transform(fitted_robust_scaler, X_c) + if o_robust_scaler.rows > 0 { + if o_robust_scaler.cols > 0 { sink = sink + o_robust_scaler.data[0] } + } + matrix_free(o_robust_scaler) + } + t3 = flow_now_ns() + robust_scaler_free(fitted_robust_scaler) + printf("ESTIMATOR|robust_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- spectral_embedding (unsupervised) ---- + # ---- select_fdr (regression) ---- t0 = flow_now_ns() - let probe_spectral_embedding: SpectralEmbedding = spectral_embedding_fit(X_c, 2, 0.1) + let probe_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) t1 = flow_now_ns() - spectral_embedding_free(probe_spectral_embedding) + select_fdr_free(probe_select_fdr) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_spectral_embedding: SpectralEmbedding = spectral_embedding_fit(X_c, 2, 0.1) - spectral_embedding_free(m_spectral_embedding) + let m_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) + select_fdr_free(m_select_fdr) } t1 = flow_now_ns() - printf("ESTIMATOR|spectral_embedding|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_select_fdr: SelectFdr = select_fdr_fit(X_r, y_r, 1.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_select_fdr: Matrix = select_fdr_transform(fitted_select_fdr, X_r) + if o_select_fdr.rows > 0 { + if o_select_fdr.cols > 0 { sink = sink + o_select_fdr.data[0] } + } + matrix_free(o_select_fdr) + } + t3 = flow_now_ns() + select_fdr_free(fitted_select_fdr) + printf("ESTIMATOR|select_fdr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -539,5 +626,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_08.flow b/benchmarks/generated/bench_estimators_08.flow index a654f22..7c4579c 100644 --- a/benchmarks/generated/bench_estimators_08.flow +++ b/benchmarks/generated/bench_estimators_08.flow @@ -65,302 +65,540 @@ function main() -> i32 { Y_rows[i] = row } + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + let mut t0: i64 = 0 let mut t1: i64 = 0 let mut t2: i64 = 0 let mut t3: i64 = 0 let mut reps: i32 = 1 + let mut sink: f32 = 0.0 - # ---- spline_transformer (unsupervised) ---- + # ---- select_fpr (regression) ---- t0 = flow_now_ns() - let probe_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) + let probe_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) t1 = flow_now_ns() - spline_transformer_free(probe_spline_transformer) + select_fpr_free(probe_select_fpr) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) - spline_transformer_free(m_spline_transformer) + let m_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) + select_fpr_free(m_select_fpr) } t1 = flow_now_ns() - let fitted_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) + let fitted_select_fpr: SelectFpr = select_fpr_fit(X_r, y_r, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_spline_transformer: Matrix = spline_transformer_transform(fitted_spline_transformer, X_c) - matrix_free(o_spline_transformer) + let o_select_fpr: Matrix = select_fpr_transform(fitted_select_fpr, X_r) + if o_select_fpr.rows > 0 { + if o_select_fpr.cols > 0 { sink = sink + o_select_fpr.data[0] } + } + matrix_free(o_select_fpr) } t3 = flow_now_ns() - spline_transformer_free(fitted_spline_transformer) - printf("ESTIMATOR|spline_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + select_fpr_free(fitted_select_fpr) + printf("ESTIMATOR|select_fpr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- select_from_model (classification, written out) ---- + t0 = flow_now_ns() + let probe_select_from_model: SelectFromModel = select_from_model_fit(w_f, f_c, 0.5) + t1 = flow_now_ns() + select_from_model_free(probe_select_from_model) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_select_from_model: SelectFromModel = select_from_model_fit(w_f, f_c, 0.5) + select_from_model_free(m_select_from_model) + } + t1 = flow_now_ns() + let fitted_select_from_model: SelectFromModel = select_from_model_fit(w_f, f_c, 0.5) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_select_from_model: Matrix = select_from_model_transform(fitted_select_from_model, X_c) + if o_select_from_model.rows > 0 { + if o_select_from_model.cols > 0 { sink = sink + o_select_from_model.data[0] } + } + matrix_free(o_select_from_model) + } + t3 = flow_now_ns() + select_from_model_free(fitted_select_from_model) + printf("ESTIMATOR|select_from_model|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- stacking_regressor (regression) ---- + # ---- select_fwe (regression) ---- t0 = flow_now_ns() - let probe_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) + let probe_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) t1 = flow_now_ns() - stacking_regressor_free(probe_stacking_regressor) + select_fwe_free(probe_select_fwe) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) - stacking_regressor_free(m_stacking_regressor) + let m_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) + select_fwe_free(m_select_fwe) } t1 = flow_now_ns() - let fitted_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) + let fitted_select_fwe: SelectFwe = select_fwe_fit(X_r, y_r, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_stacking_regressor: ptr = stacking_regressor_predict(fitted_stacking_regressor, X_r) - array_free_f32(o_stacking_regressor) + let o_select_fwe: Matrix = select_fwe_transform(fitted_select_fwe, X_r) + if o_select_fwe.rows > 0 { + if o_select_fwe.cols > 0 { sink = sink + o_select_fwe.data[0] } + } + matrix_free(o_select_fwe) } t3 = flow_now_ns() - stacking_regressor_free(fitted_stacking_regressor) - printf("ESTIMATOR|stacking_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + select_fwe_free(fitted_select_fwe) + printf("ESTIMATOR|select_fwe|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- standard_scaler (unsupervised) ---- + # ---- select_k_best (regression) ---- t0 = flow_now_ns() - let probe_standard_scaler: StandardScaler = standard_scaler_fit(X_c) + let probe_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) t1 = flow_now_ns() - standard_scaler_free(probe_standard_scaler) + select_k_best_free(probe_select_k_best) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_standard_scaler: StandardScaler = standard_scaler_fit(X_c) - standard_scaler_free(m_standard_scaler) + let m_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) + select_k_best_free(m_select_k_best) } t1 = flow_now_ns() - let fitted_standard_scaler: StandardScaler = standard_scaler_fit(X_c) + let fitted_select_k_best: SelectKBest = select_k_best_fit(X_r, y_r, 3, 0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_standard_scaler: Matrix = standard_scaler_transform(fitted_standard_scaler, X_c) - matrix_free(o_standard_scaler) + let o_select_k_best: Matrix = select_k_best_transform(fitted_select_k_best, X_r) + if o_select_k_best.rows > 0 { + if o_select_k_best.cols > 0 { sink = sink + o_select_k_best.data[0] } + } + matrix_free(o_select_k_best) } t3 = flow_now_ns() - standard_scaler_free(fitted_standard_scaler) - printf("ESTIMATOR|standard_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + select_k_best_free(fitted_select_k_best) + printf("ESTIMATOR|select_k_best|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- svc (classification) ---- + # ---- select_percentile (regression) ---- t0 = flow_now_ns() - let probe_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) + let probe_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) t1 = flow_now_ns() - svc_free(probe_svc) + select_percentile_free(probe_select_percentile) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) - svc_free(m_svc) + let m_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) + select_percentile_free(m_select_percentile) } t1 = flow_now_ns() - let fitted_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) + let fitted_select_percentile: SelectPercentile = select_percentile_fit(X_r, y_r, 50) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_svc: ptr = svc_predict(fitted_svc, X_c) - array_free_f32(o_svc) + let o_select_percentile: Matrix = select_percentile_transform(fitted_select_percentile, X_r) + if o_select_percentile.rows > 0 { + if o_select_percentile.cols > 0 { sink = sink + o_select_percentile.data[0] } + } + matrix_free(o_select_percentile) } t3 = flow_now_ns() - svc_free(fitted_svc) - printf("ESTIMATOR|svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + select_percentile_free(fitted_select_percentile) + printf("ESTIMATOR|select_percentile|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- svr (regression) ---- + # ---- sequential_feature_selector (regression) ---- t0 = flow_now_ns() - let probe_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) + let probe_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) t1 = flow_now_ns() - svr_free(probe_svr) + sequential_feature_selector_free(probe_sequential_feature_selector) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) - svr_free(m_svr) + let m_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) + sequential_feature_selector_free(m_sequential_feature_selector) } t1 = flow_now_ns() - let fitted_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) + let fitted_sequential_feature_selector: SequentialFeatureSelector = sequential_feature_selector_fit(X_r, y_r, 2, 0, n_r, f_r) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_svr: ptr = svr_predict(fitted_svr, X_r) - array_free_f32(o_svr) + let o_sequential_feature_selector: Matrix = sequential_feature_selector_transform(fitted_sequential_feature_selector, X_r) + if o_sequential_feature_selector.rows > 0 { + if o_sequential_feature_selector.cols > 0 { sink = sink + o_sequential_feature_selector.data[0] } + } + matrix_free(o_sequential_feature_selector) } t3 = flow_now_ns() - svr_free(fitted_svr) - printf("ESTIMATOR|svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + sequential_feature_selector_free(fitted_sequential_feature_selector) + printf("ESTIMATOR|sequential_feature_selector|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- target_encoder (regression) ---- + # ---- sgd_classifier (classification) ---- t0 = flow_now_ns() - let probe_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) + let probe_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) t1 = flow_now_ns() - target_encoder_free(probe_target_encoder) + sgd_classifier_free(probe_sgd_classifier) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) - target_encoder_free(m_target_encoder) + let m_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) + sgd_classifier_free(m_sgd_classifier) } t1 = flow_now_ns() - let fitted_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) + let fitted_sgd_classifier: SGDClassifier = sgd_classifier_fit(X_c, y_c, 3, 0, 1.0, 50, 0.01) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_target_encoder: Matrix = target_encoder_transform(fitted_target_encoder, X_r) - matrix_free(o_target_encoder) + let o_sgd_classifier: ptr = sgd_classifier_predict(fitted_sgd_classifier, X_c) + sink = sink + o_sgd_classifier[0] + array_free_f32(o_sgd_classifier) } t3 = flow_now_ns() - target_encoder_free(fitted_target_encoder) - printf("ESTIMATOR|target_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + sgd_classifier_free(fitted_sgd_classifier) + printf("ESTIMATOR|sgd_classifier|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- sgd_one_class_svm (unsupervised) ---- + t0 = flow_now_ns() + let probe_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) + t1 = flow_now_ns() + sgd_one_class_svm_free(probe_sgd_one_class_svm) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) + sgd_one_class_svm_free(m_sgd_one_class_svm) + } + t1 = flow_now_ns() + let fitted_sgd_one_class_svm: SGDOneClassSVM = sgd_one_class_svm_fit(X_c, 0.5, 50, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_sgd_one_class_svm: ptr = sgd_one_class_svm_predict(fitted_sgd_one_class_svm, X_c) + sink = sink + o_sgd_one_class_svm[0] + array_free_f32(o_sgd_one_class_svm) + } + t3 = flow_now_ns() + sgd_one_class_svm_free(fitted_sgd_one_class_svm) + printf("ESTIMATOR|sgd_one_class_svm|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- sgd_regressor (regression) ---- + t0 = flow_now_ns() + let probe_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) + t1 = flow_now_ns() + sgd_regressor_free(probe_sgd_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) + sgd_regressor_free(m_sgd_regressor) + } + t1 = flow_now_ns() + let fitted_sgd_regressor: SGDRegressor = sgd_regressor_fit(X_r, y_r, 0, 1.0, 0.1, 50, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_sgd_regressor: ptr = sgd_regressor_predict(fitted_sgd_regressor, X_r) + sink = sink + o_sgd_regressor[0] + array_free_f32(o_sgd_regressor) + } + t3 = flow_now_ns() + sgd_regressor_free(fitted_sgd_regressor) + printf("ESTIMATOR|sgd_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- shrunk_covariance (unsupervised) ---- + t0 = flow_now_ns() + let probe_shrunk_covariance: ShrunkCovariance = shrunk_covariance_fit(X_c, 0.1) + t1 = flow_now_ns() + shrunk_covariance_free(probe_shrunk_covariance) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_shrunk_covariance: ShrunkCovariance = shrunk_covariance_fit(X_c, 0.1) + shrunk_covariance_free(m_shrunk_covariance) + } + t1 = flow_now_ns() + printf("ESTIMATOR|shrunk_covariance|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- theil_sen_regressor (regression) ---- + # ---- simple_imputer (unsupervised) ---- t0 = flow_now_ns() - let probe_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) + let probe_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) t1 = flow_now_ns() - theil_sen_regressor_free(probe_theil_sen_regressor) + simple_imputer_free(probe_simple_imputer) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) - theil_sen_regressor_free(m_theil_sen_regressor) + let m_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) + simple_imputer_free(m_simple_imputer) } t1 = flow_now_ns() - let fitted_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) + let fitted_simple_imputer: SimpleImputer = simple_imputer_fit(X_c, 0, 1.0) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_theil_sen_regressor: ptr = theil_sen_regressor_predict(fitted_theil_sen_regressor, X_r) - array_free_f32(o_theil_sen_regressor) + let o_simple_imputer: Matrix = simple_imputer_transform(fitted_simple_imputer, X_c) + if o_simple_imputer.rows > 0 { + if o_simple_imputer.cols > 0 { sink = sink + o_simple_imputer.data[0] } + } + matrix_free(o_simple_imputer) } t3 = flow_now_ns() - theil_sen_regressor_free(fitted_theil_sen_regressor) - printf("ESTIMATOR|theil_sen_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + simple_imputer_free(fitted_simple_imputer) + printf("ESTIMATOR|simple_imputer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- transformed_target_regressor (regression) ---- + # ---- skewed_chi2_sampler (classification, written out) ---- t0 = flow_now_ns() - let probe_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) + let probe_skewed_chi2_sampler: SkewedChi2Sampler = skewed_chi2_sampler_fit(f_c, 1.0, 2, 42) t1 = flow_now_ns() - transformed_target_regressor_free(probe_transformed_target_regressor) + skewed_chi2_sampler_free(probe_skewed_chi2_sampler) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) - transformed_target_regressor_free(m_transformed_target_regressor) + let m_skewed_chi2_sampler: SkewedChi2Sampler = skewed_chi2_sampler_fit(f_c, 1.0, 2, 42) + skewed_chi2_sampler_free(m_skewed_chi2_sampler) } t1 = flow_now_ns() - let fitted_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) + let fitted_skewed_chi2_sampler: SkewedChi2Sampler = skewed_chi2_sampler_fit(f_c, 1.0, 2, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_transformed_target_regressor: ptr = transformed_target_regressor_predict(fitted_transformed_target_regressor, X_r) - array_free_f32(o_transformed_target_regressor) + let o_skewed_chi2_sampler: Matrix = skewed_chi2_sampler_transform(fitted_skewed_chi2_sampler, X_c) + if o_skewed_chi2_sampler.rows > 0 { + if o_skewed_chi2_sampler.cols > 0 { sink = sink + o_skewed_chi2_sampler.data[0] } + } + matrix_free(o_skewed_chi2_sampler) } t3 = flow_now_ns() - transformed_target_regressor_free(fitted_transformed_target_regressor) - printf("ESTIMATOR|transformed_target_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + skewed_chi2_sampler_free(fitted_skewed_chi2_sampler) + printf("ESTIMATOR|skewed_chi2_sampler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- truncated_svd (unsupervised) ---- + # ---- sparse_coder (unsupervised) ---- t0 = flow_now_ns() - let probe_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) + let probe_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) t1 = flow_now_ns() - truncated_svd_free(probe_truncated_svd) + sparse_coder_free(probe_sparse_coder) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) - truncated_svd_free(m_truncated_svd) + let m_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) + sparse_coder_free(m_sparse_coder) } t1 = flow_now_ns() - let fitted_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) + let fitted_sparse_coder: SparseCoder = sparse_coder_fit(X_c, 3) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_truncated_svd: Matrix = truncated_svd_transform(fitted_truncated_svd, X_c) - matrix_free(o_truncated_svd) + let o_sparse_coder: Matrix = sparse_coder_transform(fitted_sparse_coder, X_c) + if o_sparse_coder.rows > 0 { + if o_sparse_coder.cols > 0 { sink = sink + o_sparse_coder.data[0] } + } + matrix_free(o_sparse_coder) } t3 = flow_now_ns() - truncated_svd_free(fitted_truncated_svd) - printf("ESTIMATOR|truncated_svd|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + sparse_coder_free(fitted_sparse_coder) + printf("ESTIMATOR|sparse_coder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- tsne (unsupervised) ---- + # ---- sparse_pca (unsupervised) ---- t0 = flow_now_ns() - let probe_tsne: TSNE = tsne_fit(X_c, 2, 5.0, 0.1, 50, 42) + let probe_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) t1 = flow_now_ns() - tsne_free(probe_tsne) + sparse_pca_free(probe_sparse_pca) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_tsne: TSNE = tsne_fit(X_c, 2, 5.0, 0.1, 50, 42) - tsne_free(m_tsne) + let m_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) + sparse_pca_free(m_sparse_pca) } t1 = flow_now_ns() - printf("ESTIMATOR|tsne|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + let fitted_sparse_pca: SparsePCA = sparse_pca_fit(X_c, 2, 1.0, 100, 0.0001, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_sparse_pca: Matrix = sparse_pca_transform(fitted_sparse_pca, X_c) + if o_sparse_pca.rows > 0 { + if o_sparse_pca.cols > 0 { sink = sink + o_sparse_pca.data[0] } + } + matrix_free(o_sparse_pca) + } + t3 = flow_now_ns() + sparse_pca_free(fitted_sparse_pca) + printf("ESTIMATOR|sparse_pca|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) - # ---- tweedie_regressor (regression) ---- + # ---- sparse_random_projection (classification, written out) ---- t0 = flow_now_ns() - let probe_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) + let probe_sparse_random_projection: SparseRandomProjection = sparse_random_projection_fit(f_c, 2, 0.3, 42) t1 = flow_now_ns() - tweedie_regressor_free(probe_tweedie_regressor) + sparse_random_projection_free(probe_sparse_random_projection) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) - tweedie_regressor_free(m_tweedie_regressor) + let m_sparse_random_projection: SparseRandomProjection = sparse_random_projection_fit(f_c, 2, 0.3, 42) + sparse_random_projection_free(m_sparse_random_projection) } t1 = flow_now_ns() - let fitted_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) + let fitted_sparse_random_projection: SparseRandomProjection = sparse_random_projection_fit(f_c, 2, 0.3, 42) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_tweedie_regressor: ptr = tweedie_regressor_predict(fitted_tweedie_regressor, X_r) - array_free_f32(o_tweedie_regressor) + let o_sparse_random_projection: Matrix = sparse_random_projection_transform(fitted_sparse_random_projection, X_c) + if o_sparse_random_projection.rows > 0 { + if o_sparse_random_projection.cols > 0 { sink = sink + o_sparse_random_projection.data[0] } + } + matrix_free(o_sparse_random_projection) } t3 = flow_now_ns() - tweedie_regressor_free(fitted_tweedie_regressor) - printf("ESTIMATOR|tweedie_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + sparse_random_projection_free(fitted_sparse_random_projection) + printf("ESTIMATOR|sparse_random_projection|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- spectral_biclustering (unsupervised) ---- + t0 = flow_now_ns() + let probe_spectral_biclustering: SpectralBiclustering = spectral_biclustering_fit(X_c, 2, 2) + t1 = flow_now_ns() + spectral_biclustering_free(probe_spectral_biclustering) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_spectral_biclustering: SpectralBiclustering = spectral_biclustering_fit(X_c, 2, 2) + spectral_biclustering_free(m_spectral_biclustering) + } + t1 = flow_now_ns() + printf("ESTIMATOR|spectral_biclustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + fflush(null) + + # ---- spectral_clustering (unsupervised) ---- + t0 = flow_now_ns() + let probe_spectral_clustering: SpectralClustering = spectral_clustering_fit(X_c, 3, 0.1, 42) + t1 = flow_now_ns() + spectral_clustering_free(probe_spectral_clustering) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_spectral_clustering: SpectralClustering = spectral_clustering_fit(X_c, 3, 0.1, 42) + spectral_clustering_free(m_spectral_clustering) + } + t1 = flow_now_ns() + printf("ESTIMATOR|spectral_clustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) fflush(null) - # ---- variance_threshold (unsupervised) ---- + # ---- spectral_coclustering (unsupervised) ---- t0 = flow_now_ns() - let probe_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) + let probe_spectral_coclustering: SpectralCoclustering = spectral_coclustering_fit(X_c, 3) t1 = flow_now_ns() - variance_threshold_free(probe_variance_threshold) + spectral_coclustering_free(probe_spectral_coclustering) reps = 1 if (t1 - t0) < 200000 { reps = 200 } elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } t0 = flow_now_ns() for rep in 0 to reps { - let m_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) - variance_threshold_free(m_variance_threshold) + let m_spectral_coclustering: SpectralCoclustering = spectral_coclustering_fit(X_c, 3) + spectral_coclustering_free(m_spectral_coclustering) } t1 = flow_now_ns() - let fitted_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) + printf("ESTIMATOR|spectral_coclustering|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + fflush(null) + + # ---- spectral_embedding (unsupervised) ---- + t0 = flow_now_ns() + let probe_spectral_embedding: SpectralEmbedding = spectral_embedding_fit(X_c, 2, 0.1) + t1 = flow_now_ns() + spectral_embedding_free(probe_spectral_embedding) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_spectral_embedding: SpectralEmbedding = spectral_embedding_fit(X_c, 2, 0.1) + spectral_embedding_free(m_spectral_embedding) + } + t1 = flow_now_ns() + printf("ESTIMATOR|spectral_embedding|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + fflush(null) + + # ---- spline_transformer (unsupervised) ---- + t0 = flow_now_ns() + let probe_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) + t1 = flow_now_ns() + spline_transformer_free(probe_spline_transformer) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) + spline_transformer_free(m_spline_transformer) + } + t1 = flow_now_ns() + let fitted_spline_transformer: SplineTransformer = spline_transformer_fit(X_c, 4, 2) t2 = flow_now_ns() for rep2 in 0 to reps { - let o_variance_threshold: Matrix = variance_threshold_transform(fitted_variance_threshold, X_c) - matrix_free(o_variance_threshold) + let o_spline_transformer: Matrix = spline_transformer_transform(fitted_spline_transformer, X_c) + if o_spline_transformer.rows > 0 { + if o_spline_transformer.cols > 0 { sink = sink + o_spline_transformer.data[0] } + } + matrix_free(o_spline_transformer) } t3 = flow_now_ns() - variance_threshold_free(fitted_variance_threshold) - printf("ESTIMATOR|variance_threshold|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + spline_transformer_free(fitted_spline_transformer) + printf("ESTIMATOR|spline_transformer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) fflush(null) for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } @@ -371,5 +609,10 @@ function main() -> i32 { matrix_free(Y_multi) free(yi_c as ptr) free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) return 0 } diff --git a/benchmarks/generated/bench_estimators_09.flow b/benchmarks/generated/bench_estimators_09.flow new file mode 100644 index 0000000..fe79af1 --- /dev/null +++ b/benchmarks/generated/bench_estimators_09.flow @@ -0,0 +1,442 @@ +# Generated by benchmarks/generate_estimator_bench.py. Do not edit. +# +# One timing block per Flow estimator that the registry marks runnable. +# Regenerate with: +# python benchmarks/estimator_coverage.py +# python benchmarks/generate_estimator_bench.py + +import "lib/scikit/scikit.flow" + +extern { + function printf(fmt: string, ...) -> i32 + function flow_now_ns() -> i64 + function malloc(size: i64) -> ptr + function free(p: ptr) -> void + function fflush(stream: ptr) -> i32 +} + +function ms_between(start: i64, finish: i64) -> f32 { + return ((((finish - start) as f64) / 1000000.0) as f32) +} + +function main() -> i32 { + let iris: Dataset = load_iris() + let diabetes: Dataset = load_diabetes() + let X_c: Matrix = iris.X + let y_c: ptr = iris.y + let n_c: i32 = X_c.rows + let f_c: i32 = X_c.cols + let X_r: Matrix = diabetes.X + let y_r: ptr = diabetes.y + let n_r: i32 = X_r.rows + let f_r: i32 = X_r.cols + + let yi_c: ptr = malloc((n_c as i64) * 4) as ptr + for i in 0 to n_c { yi_c[i] = y_c[i] as i32 } + let yi_r: ptr = malloc((n_r as i64) * 4) as ptr + for i in 0 to n_r { yi_r[i] = y_r[i] as i32 } + + # A two-column target for the cross-decomposition estimators. + let Y_multi: Matrix = matrix_new(n_r, 2) + for i in 0 to n_r { + matrix_set(Y_multi, i, 0, y_r[i]) + matrix_set(Y_multi, i, 1, y_r[i] * 0.5) + } + + # Label-valued targets for the multi-output classifiers. + let Y_labels: Matrix = matrix_new(n_c, 2) + let Y_label_rows: ptr > = malloc((n_c as i64) * 8) as ptr > + for i in 0 to n_c { + let a: f32 = y_c[i] + let b: f32 = ((((y_c[i] as i32) + 1) % 3) as f32) + matrix_set(Y_labels, i, 0, a) + matrix_set(Y_labels, i, 1, b) + let lrow: ptr = array_new_f32(2) + lrow[0] = a + lrow[1] = b + Y_label_rows[i] = lrow + } + + let Y_rows: ptr > = malloc((n_r as i64) * 8) as ptr > + for i in 0 to n_r { + let row: ptr = array_new_f32(2) + row[0] = y_r[i] + row[1] = y_r[i] * 0.5 + Y_rows[i] = row + } + + # A one-dimensional x for the isotonic row, which regresses against a + # single ordered variable rather than a design. + let x1d_r: ptr = array_new_f32(n_r) + for i in 0 to n_r { x1d_r[i] = matrix_at(X_r, i, 0) } + + # Per-feature importances for the selector row, which takes the weights a + # fitted model would hand it rather than a design. + let w_f: ptr = array_new_f32(f_c) + for i in 0 to f_c { w_f[i] = 1.0 / ((i + 1) as f32) } + + let mut t0: i64 = 0 + let mut t1: i64 = 0 + let mut t2: i64 = 0 + let mut t3: i64 = 0 + let mut reps: i32 = 1 + let mut sink: f32 = 0.0 + + # ---- stacking_regressor (regression) ---- + t0 = flow_now_ns() + let probe_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) + t1 = flow_now_ns() + stacking_regressor_free(probe_stacking_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) + stacking_regressor_free(m_stacking_regressor) + } + t1 = flow_now_ns() + let fitted_stacking_regressor: StackingRegressor = stacking_regressor_fit(X_r, y_r, 3, 5, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_stacking_regressor: ptr = stacking_regressor_predict(fitted_stacking_regressor, X_r) + sink = sink + o_stacking_regressor[0] + array_free_f32(o_stacking_regressor) + } + t3 = flow_now_ns() + stacking_regressor_free(fitted_stacking_regressor) + printf("ESTIMATOR|stacking_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- standard_scaler (unsupervised) ---- + t0 = flow_now_ns() + let probe_standard_scaler: StandardScaler = standard_scaler_fit(X_c) + t1 = flow_now_ns() + standard_scaler_free(probe_standard_scaler) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_standard_scaler: StandardScaler = standard_scaler_fit(X_c) + standard_scaler_free(m_standard_scaler) + } + t1 = flow_now_ns() + let fitted_standard_scaler: StandardScaler = standard_scaler_fit(X_c) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_standard_scaler: Matrix = standard_scaler_transform(fitted_standard_scaler, X_c) + if o_standard_scaler.rows > 0 { + if o_standard_scaler.cols > 0 { sink = sink + o_standard_scaler.data[0] } + } + matrix_free(o_standard_scaler) + } + t3 = flow_now_ns() + standard_scaler_free(fitted_standard_scaler) + printf("ESTIMATOR|standard_scaler|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- svc (classification) ---- + t0 = flow_now_ns() + let probe_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) + t1 = flow_now_ns() + svc_free(probe_svc) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) + svc_free(m_svc) + } + t1 = flow_now_ns() + let fitted_svc: SVC = svc_fit(X_c, y_c, 3, 1.0, 0, 0.1, 2, 0.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_svc: ptr = svc_predict(fitted_svc, X_c) + sink = sink + o_svc[0] + array_free_f32(o_svc) + } + t3 = flow_now_ns() + svc_free(fitted_svc) + printf("ESTIMATOR|svc|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- svr (regression) ---- + t0 = flow_now_ns() + let probe_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) + t1 = flow_now_ns() + svr_free(probe_svr) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) + svr_free(m_svr) + } + t1 = flow_now_ns() + let fitted_svr: SVR = svr_fit(X_r, y_r, 1.0, 0.1, 0, 0.1, 2, 0.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_svr: ptr = svr_predict(fitted_svr, X_r) + sink = sink + o_svr[0] + array_free_f32(o_svr) + } + t3 = flow_now_ns() + svr_free(fitted_svr) + printf("ESTIMATOR|svr|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- target_encoder (regression) ---- + t0 = flow_now_ns() + let probe_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) + t1 = flow_now_ns() + target_encoder_free(probe_target_encoder) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) + target_encoder_free(m_target_encoder) + } + t1 = flow_now_ns() + let fitted_target_encoder: TargetEncoder = target_encoder_fit(X_r, y_r, n_r, 1.0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_target_encoder: Matrix = target_encoder_transform(fitted_target_encoder, X_r) + if o_target_encoder.rows > 0 { + if o_target_encoder.cols > 0 { sink = sink + o_target_encoder.data[0] } + } + matrix_free(o_target_encoder) + } + t3 = flow_now_ns() + target_encoder_free(fitted_target_encoder) + printf("ESTIMATOR|target_encoder|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- tfidf_vectorizer (classification, written out) ---- + let tfidf_vectorizer_docs: array = [ + "the quick brown fox jumps over the lazy dog", + "a lazy dog sleeps in the warm sun", + "quick brown foxes are rare in the city", + "the dog and the fox share a field", + "warm sun and a cold river run together", + "a field of brown grass in the sun", + "the city river runs past the old field", + "old dogs sleep through a quick storm", + "a storm over the city wakes the dog", + "foxes hunt in the cold river valley", + "the valley holds a warm field of grass", + "grass grows where the river meets the sun", + "a rare fox crosses the old stone bridge", + "the stone bridge over the cold river", + "dogs and foxes keep their distance here", + "here the field the river and the city meet" + ] + t0 = flow_now_ns() + let probe_tfidf_vectorizer: TfidfVectorizer = tfidf_vectorizer_fit(tfidf_vectorizer_docs, 16, 50) + t1 = flow_now_ns() + tfidf_vectorizer_free(probe_tfidf_vectorizer) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_tfidf_vectorizer: TfidfVectorizer = tfidf_vectorizer_fit(tfidf_vectorizer_docs, 16, 50) + tfidf_vectorizer_free(m_tfidf_vectorizer) + } + t1 = flow_now_ns() + let fitted_tfidf_vectorizer: TfidfVectorizer = tfidf_vectorizer_fit(tfidf_vectorizer_docs, 16, 50) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_tfidf_vectorizer: Matrix = tfidf_vectorizer_transform(fitted_tfidf_vectorizer, tfidf_vectorizer_docs, 16) + if o_tfidf_vectorizer.rows > 0 { + if o_tfidf_vectorizer.cols > 0 { sink = sink + o_tfidf_vectorizer.data[0] } + } + matrix_free(o_tfidf_vectorizer) + } + t3 = flow_now_ns() + tfidf_vectorizer_free(fitted_tfidf_vectorizer) + printf("ESTIMATOR|tfidf_vectorizer|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- theil_sen_regressor (regression) ---- + t0 = flow_now_ns() + let probe_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) + t1 = flow_now_ns() + theil_sen_regressor_free(probe_theil_sen_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) + theil_sen_regressor_free(m_theil_sen_regressor) + } + t1 = flow_now_ns() + let fitted_theil_sen_regressor: TheilSenRegressor = theil_sen_regressor_fit(X_r, y_r, 10, 100, 42) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_theil_sen_regressor: ptr = theil_sen_regressor_predict(fitted_theil_sen_regressor, X_r) + sink = sink + o_theil_sen_regressor[0] + array_free_f32(o_theil_sen_regressor) + } + t3 = flow_now_ns() + theil_sen_regressor_free(fitted_theil_sen_regressor) + printf("ESTIMATOR|theil_sen_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- transformed_target_regressor (regression) ---- + t0 = flow_now_ns() + let probe_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) + t1 = flow_now_ns() + transformed_target_regressor_free(probe_transformed_target_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) + transformed_target_regressor_free(m_transformed_target_regressor) + } + t1 = flow_now_ns() + let fitted_transformed_target_regressor: TransformedTargetRegressor = transformed_target_regressor_fit(X_r, y_r, 0) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_transformed_target_regressor: ptr = transformed_target_regressor_predict(fitted_transformed_target_regressor, X_r) + sink = sink + o_transformed_target_regressor[0] + array_free_f32(o_transformed_target_regressor) + } + t3 = flow_now_ns() + transformed_target_regressor_free(fitted_transformed_target_regressor) + printf("ESTIMATOR|transformed_target_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- truncated_svd (unsupervised) ---- + t0 = flow_now_ns() + let probe_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) + t1 = flow_now_ns() + truncated_svd_free(probe_truncated_svd) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) + truncated_svd_free(m_truncated_svd) + } + t1 = flow_now_ns() + let fitted_truncated_svd: TruncatedSVD = truncated_svd_fit(X_c, 2) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_truncated_svd: Matrix = truncated_svd_transform(fitted_truncated_svd, X_c) + if o_truncated_svd.rows > 0 { + if o_truncated_svd.cols > 0 { sink = sink + o_truncated_svd.data[0] } + } + matrix_free(o_truncated_svd) + } + t3 = flow_now_ns() + truncated_svd_free(fitted_truncated_svd) + printf("ESTIMATOR|truncated_svd|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- tsne (unsupervised) ---- + t0 = flow_now_ns() + let probe_tsne: TSNE = tsne_fit(X_c, 2, 5.0, 0.1, 50, 42) + t1 = flow_now_ns() + tsne_free(probe_tsne) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_tsne: TSNE = tsne_fit(X_c, 2, 5.0, 0.1, 50, 42) + tsne_free(m_tsne) + } + t1 = flow_now_ns() + printf("ESTIMATOR|tsne|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), 0.0, reps) + fflush(null) + + # ---- tweedie_regressor (regression) ---- + t0 = flow_now_ns() + let probe_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) + t1 = flow_now_ns() + tweedie_regressor_free(probe_tweedie_regressor) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) + tweedie_regressor_free(m_tweedie_regressor) + } + t1 = flow_now_ns() + let fitted_tweedie_regressor: TweedieRegressor = tweedie_regressor_fit(X_r, y_r, 1.0, 1.5, 100, 0.01) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_tweedie_regressor: ptr = tweedie_regressor_predict(fitted_tweedie_regressor, X_r) + sink = sink + o_tweedie_regressor[0] + array_free_f32(o_tweedie_regressor) + } + t3 = flow_now_ns() + tweedie_regressor_free(fitted_tweedie_regressor) + printf("ESTIMATOR|tweedie_regressor|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + # ---- variance_threshold (unsupervised) ---- + t0 = flow_now_ns() + let probe_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) + t1 = flow_now_ns() + variance_threshold_free(probe_variance_threshold) + reps = 1 + if (t1 - t0) < 200000 { reps = 200 } + elif (t1 - t0) < 2000000 { reps = 20 } + elif (t1 - t0) < 20000000 { reps = 5 } + t0 = flow_now_ns() + for rep in 0 to reps { + let m_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) + variance_threshold_free(m_variance_threshold) + } + t1 = flow_now_ns() + let fitted_variance_threshold: VarianceThreshold = variance_threshold_fit(X_c, 0.5) + t2 = flow_now_ns() + for rep2 in 0 to reps { + let o_variance_threshold: Matrix = variance_threshold_transform(fitted_variance_threshold, X_c) + if o_variance_threshold.rows > 0 { + if o_variance_threshold.cols > 0 { sink = sink + o_variance_threshold.data[0] } + } + matrix_free(o_variance_threshold) + } + t3 = flow_now_ns() + variance_threshold_free(fitted_variance_threshold) + printf("ESTIMATOR|variance_threshold|%.9f|%.9f|%d|ok\n", ms_between(t0, t1) / (reps as f32), ms_between(t2, t3) / (reps as f32), reps) + fflush(null) + + for i in 0 to n_c { array_free_f32(Y_label_rows[i]) } + free(Y_label_rows as ptr) + matrix_free(Y_labels) + for i in 0 to n_r { array_free_f32(Y_rows[i]) } + free(Y_rows as ptr) + matrix_free(Y_multi) + free(yi_c as ptr) + free(yi_r as ptr) + array_free_f32(x1d_r) + array_free_f32(w_f) + # The sink is printed so the work above cannot be optimized away. The + # parser matches ESTIMATOR lines only, so this one is ignored. + printf("SINK|%.9f\n", sink) + return 0 +} diff --git a/benchmarks/run_estimator_bench.py b/benchmarks/run_estimator_bench.py new file mode 100755 index 0000000..ab9e43a --- /dev/null +++ b/benchmarks/run_estimator_bench.py @@ -0,0 +1,121 @@ +#!/usr/bin/env python3 +"""Compile and run the generated estimator benchmarks, and collect their lines. + +The wide matrix used to be driven by a shell loop typed out by hand, which is +how a chunk that died mid-run once passed for a chunk that had no rows. This +does the same work with the failures visible: every chunk's exit status is +checked, and a chunk that dies takes its own row count down with it in the +summary rather than disappearing. + +Each chunk prints one `ESTIMATOR|name|fit_ms|pred_ms|reps|ok` line per +estimator and flushes it, so a crash costs only the rows after it. +compare_estimators.py keeps the fastest observation per estimator, so several +rounds of the same chunk are appended to one file and joined there. + +The first round compiles through `flow run`. Later rounds re-exec the binary +that left behind, because compiling the library at -O3 costs far more than +running the timings. +""" +from __future__ import annotations + +import argparse +import os +import platform +import re +import shutil +import subprocess +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +GENERATED = ROOT / "benchmarks" / "generated" +LINE = re.compile(r"^ESTIMATOR\|([a-z_0-9]+)\|") + +MAC_LDFLAGS = "-framework Accelerate lib/scikit/flow_time.c lib/scikit/flow_parallel.c" +LINUX_LDFLAGS = "-lm -lopenblas lib/scikit/flow_time.c lib/scikit/flow_parallel.c" + + +def default_ldflags() -> str: + return MAC_LDFLAGS if platform.system() == "Darwin" else LINUX_LDFLAGS + + +def flow_binary() -> str: + explicit = os.environ.get("FLOW_BIN") + if explicit: + return explicit + found = shutil.which("flow") + if not found: + raise SystemExit("no flow binary on PATH and FLOW_BIN is unset") + return found + + +def built_binary(flow_bin: str, stem: str) -> Path: + # flow run writes its C and its executable into one build directory next to + # the compiler, whatever directory the source came from. + return Path(flow_bin).resolve().parent / "build" / stem + + +def run(cmd: list[str], env: dict[str, str], cwd: Path, timeout: int) -> tuple[int, str]: + try: + proc = subprocess.run( + cmd, cwd=cwd, env=env, capture_output=True, text=True, timeout=timeout + ) + except subprocess.TimeoutExpired as exc: + return 124, (exc.stdout or "") if isinstance(exc.stdout, str) else "" + return proc.returncode, proc.stdout + proc.stderr + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--rounds", type=int, default=3) + ap.add_argument("--out", type=Path, default=ROOT / "benchmarks" / "estimator_flow_raw.txt") + ap.add_argument("--outdir", type=Path, default=GENERATED) + ap.add_argument("--opt", default="3") + ap.add_argument("--timeout", type=int, default=1800) + ap.add_argument("--only", help="one chunk stem, for bisecting a failure") + args = ap.parse_args() + + chunks = sorted(args.outdir.glob("bench_estimators_*.flow")) + if args.only: + chunks = [c for c in chunks if c.stem == args.only] + if not chunks: + raise SystemExit(f"no generated benchmarks in {args.outdir}") + + flow_bin = flow_binary() + env = dict(os.environ) + env["FLOW_HOST"] = env.get("FLOW_HOST", "python") + env["FLOW_OPT_LEVEL"] = args.opt + env["FLOW_LDFLAGS"] = env.get("FLOW_LDFLAGS") or default_ldflags() + + collected: list[str] = [] + seen: set[str] = set() + failures: list[str] = [] + + for round_index in range(args.rounds): + for chunk in chunks: + binary = built_binary(flow_bin, chunk.stem) + if round_index == 0 or not binary.exists(): + cmd = [flow_bin, "run", str(chunk)] + else: + cmd = [str(binary)] + code, output = run(cmd, env, ROOT, args.timeout) + rows = [l for l in output.splitlines() if LINE.match(l.strip())] + collected.extend(rows) + for line in rows: + seen.add(line.split("|")[1]) + status = "ok" if code == 0 else f"exit={code}" + if code != 0: + failures.append(f"round {round_index} {chunk.stem} {status}") + print(f"round {round_index} {chunk.stem}: {len(rows)} rows {status}", flush=True) + + args.out.write_text("\n".join(collected) + "\n") + print(f"{len(seen)} estimators, {len(collected)} lines -> {args.out}") + if failures: + print("failures:") + for f in failures: + print(f" {f}") + return 1 if failures else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/benchmarks/validate_published_claims.py b/benchmarks/validate_published_claims.py index 9044d16..e661a1f 100644 --- a/benchmarks/validate_published_claims.py +++ b/benchmarks/validate_published_claims.py @@ -27,6 +27,7 @@ ROOT = Path(__file__).resolve().parents[1] RESULT = ROOT / "benchmarks" / "headline_result_v2.json" ARCH = ROOT / "benchmarks" / "architecture_performance_map.json" +ESTIMATORS = ROOT / "benchmarks" / "estimator_comparison.json" # (path, regex with one group per count, builder for the replacement text) TARGETS = [ @@ -88,6 +89,22 @@ ] +# And the same drift from the wide matrix. The README quotes how many of the +# ranked rows favour Flow, and that count moves whenever CI re-measures the +# 192 rows it times. +ESTIMATOR_TARGETS = [ + ( + ROOT / "README.md", + re.compile(r"\*\*Flow is faster on (\d+) of the (\d+) ranked rows\.\*\*"), + lambda c: f"**Flow is faster on {c['flow_wins']} of the {c['compared']} ranked rows.**", + ), +] + + +def estimator_counts() -> dict: + return json.loads(ESTIMATORS.read_text())["counts"] + + def substrate_means() -> dict[str, float]: groups = json.loads(ARCH.read_text())["speedup_by_execution_substrate"] return {g["execution_class"]: float(g["mean_flow_speedup"]) for g in groups} @@ -100,10 +117,13 @@ def main() -> int: counts = json.loads(RESULT.read_text())["counts"] means = substrate_means() + est = estimator_counts() drifted: list[str] = [] for path, pattern, build in TARGETS + [ (p, r, (lambda b: lambda _c: b(means))(build)) for p, r, build in SUBSTRATE_TARGETS + ] + [ + (p, r, (lambda b: lambda _c: b(est))(build)) for p, r, build in ESTIMATOR_TARGETS ]: text = path.read_text() match = pattern.search(text) diff --git a/docs/benchmarks.html b/docs/benchmarks.html index f7754de..cf2a688 100644 --- a/docs/benchmarks.html +++ b/docs/benchmarks.html @@ -1,4 +1,4 @@ -Benchmarks, flow-scikit
flow~scikit

canonical v2 / parity + disparity benchmark

Eligibility never means identity.

All 19 canonical rows are measured and currently eligible for comparison, but numerical, semantic and runtime disparities remain first-class evidence. This page renders the committed benchmark and disparity artifacts directly so differences cannot disappear merely because a row passes its contract.

Flow wins...

End-to-end fit + predict comparisons won by Flow.

sklearn wins...

End-to-end comparisons won by scikit-learn.

parity eligible...

Rows admitted to the competitive denominator.

substantive disparities...

Rows whose fitted state, score, configuration or semantics genuinely diverge, above float-noise floors. Runtime differences are tracked per row but not counted here.

TIMING_UNIT|msend-to-endseed=4280/20 persisted split2% practical tie thresholddisparity retained after eligibility
KMeans note: Digits KMeans is eligible under the same declared contract as every other clustering row. Its seeded k-means++ initialization now matches scikit-learn's, so the strict diagnostic and the final eligibility decision agree. The convergence statistic, the point at which inertia is reported, empty-cluster relocation and the n_init selection rule still differ and stay visible in the disparity artifact.

runtime overview

The plots are generated from the canonical JSON.

Each runtime plot shows end-to-end fit + predict time on a log scale. The plots use the same rows as the table below and therefore update whenever the frozen canonical result changes.

All 19 speed ratios

scikit-learn total time divided by Flow total time. The vertical 1× line separates Flow wins from scikit-learn wins.

Iris total runtime

scikit-learnFlow

Digits total runtime

scikit-learnFlow

Diabetes total runtime

scikit-learnFlow

persistent disparity

Passing parity does not erase the gap.

The disparity plot normalizes each row's principal numerical difference against its effective tolerance where a tolerance is available. A value near 1 means the row is close to the acceptance boundary. Semantic/configuration differences are tracked in the same artifact and remain visible in the table.

Numerical disparity relative to tolerance

The dashed line is the acceptance boundary. Values can remain non-zero even for eligible rows.

all canonical rows

No selected-win table.

Every row is shown below. Speedup is sklearn_ms / flow_ms; values above 1× favor Flow. Strict diagnostic status is kept separate from final eligibility.

AlgorithmDatasetFinal parityStrict diagnosticWinnerscore |Δ|sklearn msFlow msspeedup

larger data

The canonical rows all fit in 1797 samples.

A separate matrix runs five estimators at 100, 1000 and 10000 rows against 8 and 32 features. One run of it does not settle a row: at 100 and 1000 samples a fit finishes in well under a millisecond, and the CI runner moves that by more than the difference being measured. Lasso at 1000 rows and 32 features was recorded at 3.83x and at 0.92x on code that differs in nothing touching Lasso. The table is therefore the spread across consecutive runs rather than one run's number, sorted with the narrowest margins first.

Algorithmsamplesfeaturesruns wonmedianrange

This matrix is reported without gating the build. What does gate is benchmarks/scaled_flow_baseline.json, a Flow-against-itself comparison refreshed from a CI artifact.

the rest of the library

The library is 203 estimators.

The canonical rows above race twelve estimators under a parity contract. lib/scikit exports 203, and a statement about Flow against scikit-learn covers six percent of it while the rest go unmeasured. A separate registry maps every exported fit to its scikit-learn counterpart and times both sides on the same data.

raced...

Ranked against a named scikit-learn class.

Flow faster...

Of the rows that produce a ratio.

own harness needed...

Takes a pipeline, a vectorizer input or a list of fitted models first.

no counterpart...

Flow implements it and scikit-learn has no equivalent.

Read this for what it is. These rows carry no parity contract, no declared tolerances and no disparity report. Each library runs its own defaults over the same data, which answers whether an implementation is in the same performance league and says nothing about whether it computes the same thing. The canonical rows above are where numerical equivalence is established. These timings also come from a developer machine rather than from CI, and the machine was not idle.

The widest ratios say more about the defaults than about the code. Each side runs its own. A row can differ by three orders of magnitude simply because one library does far more work at its defaults. That is a difference in the job, and the ratio does not measure how fast either one is at the same job. Iris also carries no missing values, which leaves the imputers nothing to impute on the Flow side while scikit-learn still runs its full round robin. The narrow rows at the top of the table are the informative ones.

Flow estimatorscikit-learnFlow msscikit-learn msspeedup

methodology

Correctness, disparity and timing are separate dimensions.

The benchmark consumes the same persisted train/test indices in Python and Flow. Python uses high-resolution adaptive timing and the canonical runner aggregates repeated process measurements with medians and IQR. Flow timings are emitted in milliseconds and aggregated by the same runner.

Supervised rows compare predictive metrics under declared tolerances. PCA additionally checks explained variance, singular values, reconstruction error and sign-aligned components. KMeans uses permutation-invariant clustering quality and inertia. The persistent disparity artifact preserves raw numerical gaps and known semantic/configuration differences even after the estimator-specific eligibility contract succeeds.

historical deployment evidence

Footprint and startup remain separate experiments.

The repository also contains a historical deployment comparison recording a roughly 1.4 MB Flow native executable and a roughly 65× cold-start advantage (33 ms versus 2160 ms). Those figures come from a different deployment experiment and are intentionally not mixed into the canonical estimator timing denominator.

trajectory

Flow versus Python, across freezes.

Each row's speedup at the previous freeze and at the latest one. A speedup can move because Flow changed or because scikit-learn's side changed on that runner. When a row moves by more than 10%, the last column names which side's own time moved more, from the committed absolute timings.

reproduce

Read the source artifacts.

Canonical result ↗ Disparity report ↗