Stop printing both strings in full when Should-BeString fails on a big string

[nohwnd]

Fix #2951

Comparing whole generated files against an expected copy is a real use for Should-BeString, and failing one printed both strings in their entirety with a caret under the first differing character. For the 10 000 line file in the issue that is a hundred thousand lines of output, and a character offset into it that nobody can act on. Should -BeExactly in v5 at least truncated.

Before, on the issue's example: both strings printed in full.

After:

Expected strings to be the same, but they were different.
String lengths are both 109999.
Expected 10000 line(s), actual 10000 line(s).
Lines differ at line 5433.

  5431 |   Line 05431
  5432 |   Line 05432
  5433 | - Line 99999
       | + Line 05433
  5434 |   Line 05434
  5435 |   Line 05435

What changed

Strings small enough to read are left exactly as they were, the full text with a caret is the most precise thing we can show. Only bigger ones get a compact view:

Input	Output
more than 10 lines	the differing line, two lines of context either side
more than 120 characters on one line	excerpt around the difference with ellipses, like v5
anything smaller	unchanged

Only the differing lines are passed through Expand-SpecialCharacters, so a trailing space or a stray CR is visible without turning every other line into escape codes.

Line endings

Splitting on line endings throws away the difference when the difference is the line ending. The first version of this printed two identical looking blocks and claimed line 1 differed, which is worse than no diff at all. It now detects that every line matches and names the endings instead:

Every line is the same, only the line endings differ.
Expected: 19 LF
But was:  19 CRLF
Use -NormalizeLineEnding to ignore this.

Not done here

No diff library. One difference in a big string needs "which line, plus context", which is a first-differing-line scan. A real diff engine earns its keep when there are many scattered differences and you need hunks, which is the snapshot testing case, and that deserves its own justification rather than being folded in here.

Should-BeString -Output Hex from #2562 is also still open.

Tests

Three added: the differing-line-with-context case including an assertion that the whole string is not printed, the line-endings-only case, and the long single line truncation. The existing message-shape test passes untouched, and tst/functions/assert is at 1146 passed / 7 failed, identical to main's baseline on the same machine.

🤖

[nohwnd]

More examples of what this prints

Ran the branch (5c5d19c) against a wider set of inputs so we can look at the actual output before deciding. Everything below is copy-pasted from a real failure message, nothing is hand written.

Small strings, unchanged

10 lines or fewer and under 120 characters still print in full with the caret, exactly like on main.

Expected strings to be the same, but they were different.
String lengths are both 11.
Strings differ at index 8.
Expected: 'hello world'
But was:  'hello wonld'
           --------^

Expected strings to be the same, but they were different.
String lengths are both 70.
Strings differ at index 26.
Expected: 'Line 1␊Line 2␊Line 3␊Line 4␊Line 5␊Line 6␊Line 7␊Line 8␊Line 9␊Line 10'
But was:  'Line 1␊Line 2␊Line 3␊Line X␊Line 5␊Line 6␊Line 7␊Line 8␊Line 9␊Line 10'
           --------------------------^

One more line and it switches to the compact view:

Expected strings to be the same, but they were different.
String lengths are both 78.
Expected 11 line(s), actual 11 line(s).
Lines differ at line 6.

  4 |   Line 4
  5 |   Line 5
  6 | - Line 6
    | + Line X
  7 |   Line 7
  8 |   Line 8

Realistic files

A JSON document where one value differs:

Expected strings to be the same, but they were different.
String lengths are both 221.
Expected 12 line(s), actual 12 line(s).
Lines differ at line 3.

  1 |   {
  2 |     "name": "Pester",
  3 | -   "version": "6.1.0",
    | +   "version": "6.1.1",
  4 |     "authors": [
  5 |       "Jakub",

The same document reindented from 2 spaces to 4:

Expected strings to be the same, but they were different.
Expected length: 221
Actual length:   247
Expected 12 line(s), actual 12 line(s).
Lines differ at line 2.

  1 |   {
  2 | -   "name": "Pester",
    | +     "name": "Pester",
  3 |     "version": "6.1.0",
  4 |     "authors": [

A line inserted at line 6 of 20:

Expected strings to be the same, but they were different.
Expected length: 150
Actual length:   159
Expected 20 line(s), actual 21 line(s).
Lines differ at line 6.

  4 |   Line 4
  5 |   Line 5
  6 | - Line 6
    | + INSERTED
  7 |   Line 7
  8 |   Line 8

A line deleted at line 6 of 20:

Expected strings to be the same, but they were different.
Expected length: 150
Actual length:   143
Expected 20 line(s), actual 19 line(s).
Lines differ at line 6.

  4 |   Line 4
  5 |   Line 5
  6 | - Line 6
    | + Line 7
  7 |   Line 7
  8 |   Line 8

A trailing space on line 7, which is invisible in the source but shows up because only the differing lines are escaped:

Expected strings to be the same, but they were different.
Expected length: 110
Actual length:   111
Expected 15 line(s), actual 15 line(s).
Lines differ at line 7.

  5 |   Line 5
  6 |   Line 6
  7 | - Line 7
    | + Line 7 
  8 |   Line 8
  9 |   Line 9

Line endings

All 20 lines LF against all 20 CRLF:

Expected strings to be the same, but they were different.
Expected length: 150
Actual length:   169
Expected 20 line(s), actual 20 line(s).
Every line is the same, only the line endings differ.
Expected: 19 LF
But was:  19 CRLF
Use -NormalizeLineEnding to ignore this.

Only one of the 20 line endings is CRLF:

Expected strings to be the same, but they were different.
Expected length: 150
Actual length:   151
Expected 20 line(s), actual 20 line(s).
Every line is the same, only the line endings differ.
Expected: 19 LF
But was:  1 CRLF, 18 LF
Use -NormalizeLineEnding to ignore this.

Three lines, so still small, so it keeps the old rendering, which is readable here anyway:

Expected strings to be the same, but they were different.
Expected length: 5
Actual length:   7
Strings differ at index 1.
Expected: 'a␊b␊c'
But was:  'a␍␊b␍␊c'
           -^

Long single lines

401 characters, difference in the middle:

Expected strings to be the same, but they were different.
String lengths are both 401.
Strings differ at index 200.
Expected: '...xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxByyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
But was:  '...xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxAyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
           -------------------------------------------^

Difference at index 0, so no leading ellipsis and the caret sits at the start:

Expected: 'Byyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
But was:  'Ayyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
           ^

Expected is a prefix of actual, so the caret points just past the end of the expected excerpt:

Expected strings to be the same, but they were different.
Expected length: 300
Actual length:   350
Strings differ at index 300.
Expected: '...yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
But was:  '...yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz...'
           -------------------------------------------^

Escaping is length preserving ([Pester.Formatter]::EscapeControlChars maps one control character to one symbol), so 60 tabs before the difference do not push the caret off:

Expected: '...␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉Byyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
But was:  '...␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉␉Ayyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy...'
           -------------------------------------------^

Inside a real run

Output.Verbosity = Detailed, 40 line file, one line differs. The blank line after the header gets eaten by the output writer, the rest keeps its alignment under the indent:

Describing generated file
  [-] matches the golden copy 70ms
   Expected strings to be the same, but they were different.
   String lengths are both 319.
   Expected 40 line(s), actual 40 line(s).
   Lines differ at line 20.
     18 |   Line 18
     19 |   Line 19
     20 | - Line 20
        | + Line 99
     21 |   Line 21
     22 |   Line 22
   at $actual | Should-BeString $expected, render.Tests.ps1:5

🤖

[nohwnd]

What the examples show that I am not happy with yet

Six things came out of running the wider set. The first three produce output that is wrong or useless, the last three are judgement calls. The Expected strings to be the same, but they were different. first line is cut from the blocks below.

1. The context lines always come from Expected, so they go blank when Expected is shorter

Expected has 12 lines, actual has 15:

Expected 12 line(s), actual 15 line(s).
Lines differ at line 13.

  11 |   Line 11
  12 |   Line 12
  13 | - 
     | + Line 13
  14 |   
  15 |   

Lines 14 and 15 of the actual are Line 14 and Line 15, and the message prints them as blank, because the context branch renders $e only. Reading this you would think the actual ends with two empty lines. Same thing with an empty expected against a 20 line actual, it prints 2 | and 3 | empty.

Fix is either to fall back to the actual line when expected ran out, or to stop printing context past the end of the shorter side.

2. A difference that is only a trailing newline prints two blank lines

Expected length: 87
Actual length:   86
Expected 13 line(s), actual 12 line(s).
Lines differ at line 13.

  11 |   Line 11
  12 |   Line 12
  13 | - 
     | + 

Both sides render as nothing. This is the same class of problem as the line-endings case that this PR already handles, the split threw away the only difference there is. A file ending with or without a final newline is a common real difference, so it deserves its own sentence the way the line endings got one, something like "The strings are the same, actual is missing the final newline."

3. The differing line itself is not truncated

The whole point of the PR is not to print a huge string, and the line branch happily prints a 500 character line twice. Real output, with the two long lines shortened by me so this comment stays readable, they are printed in full:

String lengths are both 574.
Expected 11 line(s), actual 11 line(s).
Lines differ at line 6.

  4 |   Line 4
  5 |   Line 5
  6 | - v=aaaaaaaa[496 more a]a
    | + v=aaaaaaaa[495 more a]b
  7 |   Line 7
  8 |   Line 8

A minified JS bundle, a base64 blob or a long connection string in an otherwise short config file hits this. The 120 character window from the single line branch should apply to the differing line here too, with the caret, otherwise we solved the tall case and left the wide case.

4. Mixed line endings do not say which line

Every line is the same, only the line endings differ.
Expected: 19 LF
But was:  1 CRLF, 18 LF

Correct and still not actionable. When it is 1 of 19 the useful part is which one. When it is all of them the counts are enough. Worth naming the first line whose ending differs when the endings are mixed.

5. Only the first differing line is reported, so an insert reads as an edit

A line inserted at position 6 of 20:

Expected length: 150
Actual length:   159
Expected 20 line(s), actual 21 line(s).
Lines differ at line 6.

  4 |   Line 4
  5 |   Line 5
  6 | - Line 6
    | + INSERTED
  7 |   Line 7
  8 |   Line 8

Technically true, and it hides that everything below line 6 is shifted by one and otherwise fine. The line counts in the header are the only hint. I do not want a diff library here (see the PR description), but a shift is cheap to detect: if expected[i] equals actual[i + k] for a small k, say so. Question is whether that is worth the code, or whether the line count difference in the header is enough of a hint.

6. The 10 line and 120 character thresholds are a hard cliff

10 lines gives the full string with a caret, 11 lines gives the compact block. Both are readable, but the same test can flip between two completely different message shapes because someone added a line. Alternative is to always use the line view for anything with more than one line and keep the caret view only for single line strings. That is a bigger behavior change and would need the existing message shape tests updated, so I left it, but it is the version I would defend more easily.

🤖

[nohwnd]

we will instead embed a diffing lib to avoid going into a diffing lib authoring bussiness.