[BUG] *WLS Asymmetrical State Estimation diverges and applies inconsistent Phase/Line scaling (Regression since >= 1.13.147)*

[davidcano90]

Describe the bug

Description:
We have identified a critical mathematical regression strictly confined to the State Estimator (WLS) module when working with asymmetrical networks (is_symmetrical=False). Since the topological refactoring introduced in version 1.13.147 ("Merge links into nodes"), the WLS solver diverges drastically and outputs heavily distorted/scaled node voltages (e.g., mixing Phase-to-Neutral and Line-to-Line magnitude scalings).

Crucial Isolation of the Bug:
We have empirically proven that the underlying physical topology and components are processed correctly in 1.13.158. If we feed the exact same asymmetrical grid data but execute a standard Load Flow calculation (bypassing WLS sensors), versions 1.13.146 and 1.13.158 yield mathematically identical and correct results ($\approx 230$ V on LV nodes). The severe distortion only happens when active asymmetrical voltage measurements are ingested and processed by the WLS solver.

Input Data Validity

• I checked the validity of input data using the helper function (examples here).

Data Dump

No response

To Reproduce

Steps to Reproduce:

1. Configure an asymmetrical distribution grid with a standard MV/LV transformer (e.g., 10kV / 400V, wye_n / wye_n).
2. Ensure the transformer neutral is explicitly grounded (using r_grounding_from/to = 0.0), as required by recent PGM versions to avoid SparseMatrixError.
3. Provide valid asymmetrical asym_voltage_sensor measurements for the LV nodes using Phase-to-Neutral magnitudes (e.g., 230.94 V / 1.0 p.u.).
4. Execute the State Estimation (WLS).

Expected behavior

Expected Behavior (PGM <= 1.13.146):
The State Estimator converges successfully, and the resulting estimated voltages in the nodes_se dataset are physically correct ($\approx 230.94$ V Phase-to-Neutral on the LV side, and $\approx 5.7$ kV on the MV side).

Actual Behavior (PGM >= 1.13.147 up to 1.13.158):
The WLS solver diverges and the raw nodes_se["u"] output array becomes heavily fragmented and corrupted:

• MV nodes spike from an expected $\approx 5.7$ kV up to $\approx 9.5$ kV (an exact deviation of $\approx \sqrt{3}$).
• Some LV nodes are returned erroneously scaled by $\sqrt{3}$ (e.g., returning $\approx 382$ V instead of $\approx 220$ V).
• Other LV nodes on the same grid remain close to $\approx 230$ V.

Mathematical Evidence:
Comparison of Max Voltages across the exact same grid topology:

• Load Flow only (No WLS sensors):
  • PGM 1.13.146 Max Voltage: 5773.50 V
  • PGM 1.13.158 Max Voltage: 5773.50 V (Zero Divergence. Topology is intact).
• State Estimation (With active WLS sensors):
  • PGM 1.13.146 Max Voltage: 5773.56 V (Correct)
  • PGM 1.13.158 Max Voltage: 9521.10 V (Diverged. Massive amount of nodes across the grid diverge significantly from the expected physical baseline).

This strongly indicates that the WLS solver backend in C++ is struggling to correctly map, scale, or de-normalize the objective function arrays (mixing $U_{LL}$ and $U_{LN}$ base expectations) during the asymmetrical state estimation process since the topological node-merging update.

Screenshots

No response

Additional context

Environment:

• OS: Linux
• Python: 3.12
• power-grid-model version: 1.13.147 through 1.13.158 (Bug is completely absent in 1.13.146)

Community Meeting

• I wish to discuss this bug in the bi-monthly PGM community meeting. Please join the PGM community meeting.

[TonyXiang8787]

Dear @davidcano90,

Thanks for reporting this bug. I tried to use your instruction to reproduce the bug. But it seems to be working for the latest version. See below my input and result data:

Input

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "u_rated": 10000},
      {"id": 2, "u_rated": 400}
    ],
    "source": [
      {"id": 3, "node": 1, "status": 1, "u_ref": 1}
    ],
    "transformer": [
      {"id": 4, "from_node": 1, "to_node": 2, "from_status": 1, "to_status": 1, "u1": 10000, "u2": 400, "sn": 100e3, "uk": 0.05, "pk": 0, "i0": 0, "p0": 0, "winding_from": 1, "winding_to": 1, "clock": 12, "tap_side": 0, "tap_pos": 0, "tap_min": -3, "tap_max": 3, "tap_size": 100, "r_grounding_to": 0, "x_grounding_to": 0, "r_grounding_from": 0, "x_grounding_from": 0}
    ],
    "asym_voltage_sensor": [
      {"id": 5, "measured_object": 2, "u_sigma": 10, "u_measured": [230.94010767585033, 230.94010767585033, 230.94010767585033]}
    ]
  }
}

Result

{
  "version": "1.0",
  "type": "asym_output",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 1, "energized": 1, "u_pu": [1, 1, 1], "u": [5773.5026918962576, 5773.5026918962576, 5773.5026918962576], "u_angle": [0, -2.0943951023931957, 2.0943951023931957], "p": [0, 0, 0], "q": [0, 0, 0]},
      {"id": 2, "energized": 1, "u_pu": [1, 1, 1], "u": [230.9401076758503, 230.9401076758503, 230.9401076758503], "u_angle": [0, -2.0943951023931957, 2.0943951023931957], "p": [0, 0, 0], "q": [0, 0, 0]}
    ],
    "transformer": [
      {"id": 4, "energized": 1, "loading": 0, "p_from": [0, -0, 0], "q_from": [0, 0, 0], "i_from": [0, 0, 0], "s_from": [0, 0, 0], "p_to": [0, -0, 0], "q_to": [0, 0, 0], "i_to": [0, 0, 0], "s_to": [0, 0, 0]}
    ],
    "source": [
      {"id": 3, "energized": 1, "p": [0, -0, 0], "q": [0, 0, 0], "i": [0, 0, 0], "s": [0, 0, 0], "pf": [0, 0, 0]}
    ],
    "asym_voltage_sensor": [
      {"id": 5, "energized": 1, "u_residual": [0, 0, 0]}
    ]
  }
}

The result is expected with correct voltage at MV and LV side. Could you please provide a (minimal) reproducible dataset for this bug?

[davidcano90]

Hello @TonyXiang8787 ,
thank you for your reply. how would you prefer to have the dataset?
For now, I've included the models and their respective results in the attached spreadsheets. The suffix identifies the version used to run the analysis. If you'd prefer a different format, please don't hesitate to let me know.

Thank you very much in advance for your help.

SE_PGM_WLS_controller_id_1_v158.xlsx
SE_PGM_WLS_controller_id_1_v146.xlsx

[TonyXiang8787]

Hello @TonyXiang8787 , thank you for your reply. how would you prefer to have the dataset? For now, I've included the models and their respective results in the attached spreadsheets. The suffix identifies the version used to run the analysis. If you'd prefer a different format, please don't hesitate to let me know.

Thank you very much in advance for your help.

SE_PGM_WLS_controller_id_1_v158.xlsx SE_PGM_WLS_controller_id_1_v146.xlsx

Hi @davidcano90,

You can dump the input dataset in json using our serialization function. See example here:

https://power-grid-model.readthedocs.io/en/stable/examples/Serialization%20Example.html

If would be nice if you can reduce the dataset to minimal which can still reproduce the error.

[davidcano90]

Hi @TonyXiang8787 ,

Thank you! I attach the input and output files for both versions. The file naming follows: v1_13_{146,148}_{input,output}.txt.

Let me know if further information is requiered

v1_13_158_output.txt
v1_13_158_input.txt
v1_13_146_output.txt
v1_13_146_input.txt

[TonyXiang8787]

Hi @TonyXiang8787 ,

Thank you! I attach the input and output files for both versions. The file naming follows: v1_13_{146,148}_{input,output}.txt.

Let me know if further information is requiered

v1_13_158_output.txt v1_13_158_input.txt v1_13_146_output.txt v1_13_146_input.txt

Hi @davidcano90,

Thanks for the data dump. We are going to investigate what happens. To be followed.

Just some reminder by quickly scanning your data. You have many power sensors with:

"measured_terminal_type": 9

This is a node power injection sensor, which is already deprecated and will be removed in the coming v2 major version release. See #1334 for detail.

I am not sure if we have done something wrong to break this before moving to v2. We will look into it.

[TonyXiang8787]

Hi @TonyXiang8787 ,
Thank you! I attach the input and output files for both versions. The file naming follows: v1_13_{146,148}_{input,output}.txt.
Let me know if further information is requiered
v1_13_158_output.txt v1_13_158_input.txt v1_13_146_output.txt v1_13_146_input.txt

Hi @davidcano90,

Thanks for the data dump. We are going to investigate what happens. To be followed.

Just some reminder by quickly scanning your data. You have many power sensors with:

"measured_terminal_type": 9

This is a node power injection sensor, which is already deprecated and will be removed in the coming v2 major version release. See #1334 for detail.

I am not sure if we have done something wrong to break this before moving to v2. We will look into it.

Hi @davidcano90,

After some experimenting, I have managed to reproduce the error with a minimal dataset. See below. The same dataset works for version 1.13.146.

It seems that there will be an error when the power sensor is placed at the transformer, and the node is connecting to another node via a link.

The calculation fails for both symmetric and asymmetric state estimation.

Why this could happen is to be investigated.

{
  "version": "1.0",
  "type": "input",
  "is_batch": false,
  "attributes": {},
  "data": {
    "node": [
      {"id": 30, "u_rated": 400},
      {"id": 90, "u_rated": 10000},
      {"id": 112, "u_rated": 400}
    ],
    "source": [
      {"id": 207, "node": 90, "status": 1, "u_ref": 1, "u_ref_angle": 0}
    ],
    "transformer": [
      {"id": 208, "from_node": 90, "to_node": 112, "from_status": 1, "to_status": 1, "u1": 10000, "u2": 400, "sn": 630000, "uk": 0.040000000000000001, "pk": 6800.2200000000003, "i0": 0.0018730158730158729, "p0": 1180, "winding_from": 1, "winding_to": 1, "clock": 0, "tap_side": 1, "tap_pos": 5, "tap_min": 1, "tap_max": 9, "tap_nom": 5, "tap_size": 10, "r_grounding_from": 0, "x_grounding_from": 0, "r_grounding_to": 0, "x_grounding_to": 0}
    ],
    "link": [
      {"id": 229, "from_node": 30, "to_node": 112, "from_status": 1, "to_status": 1}
    ],
    "asym_voltage_sensor": [
      {"id": 329, "measured_object": 30, "u_sigma": 2.3094000000000001, "u_measured": [230.94011, 230.94011, 230.94011]}

    ],
    "asym_power_sensor": [
       {"id": 466, "measured_object": 208, "measured_terminal_type": 1, "p_measured": [0, 0, 0], "q_measured": [0, 0, 0], "p_sigma": [1000, 1000, 1000], "q_sigma": [1000, 1000, 1000]}
    ]
  }
}

[davidcano90]

Interesting. Thanks for the information. I'll keep an eye on how this bug develops.

[davidcano90]

Hi @TonyXiang8787,

I've noticed that release .159 appears to address the issue described. I've run the state estimation both on my dataset (which is a bit more complex) and on the minimalist dataset you posted earlier, and it allows the error to be reproduced. However, the error is still present in the new version and it does not appear on the .146 version. I am reporting this in case you are not already aware of it. If you are already aware of this issue, please disregard this message.

Of course, thank you very much for your help!

Here the error message that I get:

power_grid_model._core.errors.SparseMatrixError: Sparse matrix error, possibly singular matrix!
If you get this error from state estimation, it might mean the system is not fully observable, i.e. not enough measurements.
It might also mean that you are running into a corner case where PGM cannot resolve yet.
See https://github.com/PowerGridModel/power-grid-model/issues/864.
Try validate_input_data() or validate_batch_data() to validate your data.

I share again the results with the large dataset:
v1_13_159_input.txt
v1_13_159_output.txt

Kind Regards,

David

[figueroa1395]

Hello @davidcano90,

We are indeed aware of this. Release 1.13.159 included the test case to help find this issue. We have a pretty good idea of the issue and the fix, but we are confirming that everything is correct before we merge it. We'll let you know once it's out :).

In any case, thanks for the heads up and don't hesitate to let us know if you see something else strange. We'll keep you posted.

Regards,
Santiago.

[figueroa1395]

Hello @davidcano90,

We have a bugfix on the way (see this if you are interested in the details). Right now, CI is finishing its job, but we expect this to be merged with main in a few hours. This means that v1.13.162 will most likely be the tag with the fix.

In the meantime, you can try your local off of the bugfix PR if you'd like. Once it's merged, I'll ping you again for a final confirmation.

[mgovers]

Hi @davidcano90,

We have found, isolated and fixed the root cause in #1576 and #1580 . See also the repro case in #1571 and extended in #1567 . This will probably be automatically merged and released within the coming few hours. This issue will be automatically closed once the first part is done, but it may still take some time until the fix becomes available, starting with release v1.13.162. I hope that this resolves everything on your end.

Thank you once more for finding and reporting the bug, as well as your patience along the way.