Skip to content

Fix DynamicDML covariance orientation and OrthoForest marginal standard errors - #1077

Open
kbattocchi wants to merge 2 commits into
kebatt/numpy-reduction-fixesfrom
kebatt/multidimensional-axis-order-fixes
Open

kbattocchi wants to merge 2 commits into
kebatt/numpy-reduction-fixesfrom
kebatt/multidimensional-axis-order-fixes

Conversation

@kbattocchi

@kbattocchi kbattocchi commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

Stacked follow-up to #1069. Contains two independently confirmed inference fixes after auditing the codebase for other issues with multidimensional arrays; point-estimation logic is unchanged but we had bugs in our confidence intervals for DynamicDML and for OrthoForest's marginal effects when using an explicit treatment featurizer.

  • DynamicDML: correct the cross-period Jacobian blocks so rows index current-period score coordinates and columns index future-period coefficient coordinates. The previous outer-product order transposed nonsymmetric off-diagonal blocks and therefore corrupted the sandwich covariance. Added comments to clarify the period/treatment/feature covariance ordering in the implementation.
  • OrthoForest BLB inference: take the square root of the projected covariance before returning transformed marginal-effect standard errors. This bug was just variance being reported as standard error, not an axis transpose.

Regression coverage

  • DynamicDML: independent numerical derivatives of stacked panel scores, independently solved point coefficients, and public effect-inference standard errors. Covers one/three periods, scalar/two-coordinate treatments, no/two heterogeneity features, and vector/single-column/two-output outcomes. Nonsymmetric cross-period coupling exposes the original bug. This estimating-equation/finite-difference reference reuses fitted nuisance residuals to isolate final-stage correctness. Each independent panel unit contributes one influence vector with periods kept together, giving coefficient sampling covariance rather than outcome variance.
  • OrthoForest: fixed coefficient covariance with correlated errors; analytical derivatives and coefficient-error factors give independent expected standard deviations for identity, linear, and quadratic treatment transformations. Checks point estimates, standard errors, variances, confidence intervals, identity invariance, and scalar/vector shapes.
  • With an explicit identity treatment featurizer, compare marginal-effect inference for each treatment coordinate against effect inference for a one-unit change in that coordinate, using the same fitted inference object, inputs, and confidence level (alpha=0.1). Check point estimates, standard errors, and both confidence-interval endpoints through effect_inference and effect_interval. Preserve scalar/column outcome shapes. Both inference calculations run unmodified against the same fixed estimator fixture.

Validation

  • Before the implementation fixes: 24 failed, 18 passed across the original new regressions (six DynamicDML covariance cases and all 18 OrthoForest cases failed).
  • After the fixes: 42 passed across those regressions.

@kbattocchi
kbattocchi added this pull request to stack #1070 October 20, 2026 21:28
@kbattocchi
kbattocchi force-pushed the kebatt/multidimensional-axis-order-fixes branch 4 times, most recently from 9fe7557 to 1a9ff87 Compare October 27, 2026 13:10
@kbattocchi
kbattocchi requested review from fverac and a balanced review from Copilot October 27, 2026 13:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
What changed in this PR

Fixes two inference bugs affecting reported uncertainty (not point estimates) in DynamicDML and OrthoForest, and adds regression tests to prevent recurrence.

Changes:

  • Correct DynamicDML cross-period Jacobian block orientation used in sandwich covariance calculation.
  • Fix OrthoForest BLB marginal-effect inference to return standard errors (sqrt of projected variance).
  • Add focused regression tests covering multidimensional shape cases and transformation/Jacobian paths.
File Description
econml/​orf/​_ortho_forest.py Fixes BLB marginal-effect scaling to report standard errors by taking sqrt of projected variance.
econml/​panel/​dml/​_dml.py Corrects Jacobian block outer-product order for DynamicDML covariance and clarifies ordering via comments/docstring.
econml/​tests/​test_orf.py Adds regression tests validating BLB transformed marginal-effect stderr/CI behavior across shapes and transforms.
econml/​tests/​test_dynamic_dml.py Adds regression test comparing DynamicDML covariance/effect inference vs. independent finite-difference score Jacobian reference.

Review effort: Lite

scales[tuple(me_index)] = final
projected_var = np.einsum('mj, mjk, mk -> m', jac, cov, jac)
# Inference needs the standard deviation of the projected coefficient uncertainty.
scales[tuple(me_index)] = np.sqrt(projected_var)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that roundoff can make a theoretically nonnegative projected variance slightly negative. I'd defer clipping here: unconditional maximum(..., 0) would also hide materially invalid covariance estimates, and the existing effect and constant-marginal inference paths take square roots without clipping. Numerical hardening should use a consistent, scale-aware policy across those paths, with near-singular regression cases. This fix stays focused on converting variance to standard error.

Comment thread econml/tests/test_orf.py
from sklearn.linear_model import Lasso, LogisticRegression, LogisticRegressionCV
from sklearn.multioutput import MultiOutputRegressor
from sklearn.preprocessing import FunctionTransformer
from econml.orf import DMLOrthoForest, DROrthoForest

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direct use of BLBInference is deliberate here. The comment beside factor explains that the stub supplies known coefficient covariance to isolate inference propagation from forest fitting. The inference methods themselves are real, and the test checks an independent factor-based standard-deviation reference plus agreement between identity-transform marginal and finite-contrast inference. A fitted-forest-only test would lose that controlled reference, so I'd retain this focused component test alongside the existing estimator tests.

Differentiate panel scores with score coordinates on rows and future coefficient coordinates on columns. Check the fitted covariance against numerical score derivatives, preserve point estimates, and verify public effect standard errors across treatment, feature, and outcome shapes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 818b2f7a-7f36-4a43-b9d0-3e9af6470ee5
Signed-off-by: Keith Battocchi <kebatt@microsoft.com>
Take the square root of projected covariance before constructing marginal inference results. Regress identity invariance, linear and quadratic transformations, correlated coefficient errors, scalar/vector shapes, and confidence intervals.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 818b2f7a-7f36-4a43-b9d0-3e9af6470ee5
Signed-off-by: Keith Battocchi <kebatt@microsoft.com>
@kbattocchi
kbattocchi force-pushed the kebatt/multidimensional-axis-order-fixes branch from 0ac6961 to 7a2c9d3 Compare October 27, 2026 19:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants