Bug report

Bug description:

The hz encoder writes the ~~ escape for a ~ as two separate bytes and consumes the character after the first one. If the output buffer becomes full between the two, the encoder is called again at the next character and the second ~ is never written. The result is a bare ~ that the decoder combines with the byte after it, so the data decodes to different text or fails to decode. It needs text that mixes ASCII and Chinese characters (plain ASCII or plain Chinese never fills the initial buffer) with a ~ at the wrong offset.

text = "a中" * 7 + "xx" + "~{"
data = text.encode("hz")
print(data[-8:])                  # b'VP~}xx~{', should end in b'xx~~{'
print(data.decode("hz") == text)  # False

With "xxx" instead of "xx" the output is correct. The two-step write dates from gh-74189.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-160518: Fix hz encoder dropping half of the ~~ escape #160519