[absolutely-vivid]

Google's homepage returns utf8-encoded data, which was making the test fail due to the use of httpc.send instead of httpc.send_bits.

Google's homepage returns non-utf8-encoded data, which was making the test fail due to the use of httpc.send instead of httpc.send_bits.

[lpil]

Gleam strings are unicode strings and unicode bodies are expected to be UTF8 encoded, so I don't understand the change here. Could you expand on what behaviour you were seeing?

[absolutely-vivid]

Hmm, that makes sense. I re-ran the failing test, and this is the output

   Compiled in 0.26s
    Running gleam_httpc_test.main

let assert test/gleam_httpc_test.gleam:108
 test: gleam_httpc_test.ipv6_test
 code: let assert Ok(resp) = httpc.send(req)
value: Error(InvalidUtf8Response)
 info: Pattern match failed, no pattern matched the value.

0 tests, 1 failures

I think what's actually happening is that ipv6.google.com is returning non-unicode data, and I misinterpreted the InvalidUtf8Response as the opposite of what it actually meant. Oops 😅 That would certainly explain why the fix was to use send_bits.

I'll edit the issue and change the commit message.

[lpil]

Could it be something specific to your network? The HTML returned has says it is UTF8 with <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">, and it seems to be valid to me:

if curl -fsS https://ipv6.google.com | iconv -f UTF-8 -t UTF-8 >/dev/null; then
    echo valid
else
    echo not valid
fi
# valid

[jtdowney]

I wonder if something is causing compressed data to be returned. I wouldn't expect that to be network-dependent. FWIW, I see only valid UTF-8 data from a curl on my network/machine.

[absolutely-vivid]

That's all really interesting! I did some digging and found some fun results.

• The iconv test prints not valid for me
• On my local machine running macos with curl 8.21.0, google returns a content type of content-type: text/html; charset=ISO-8859-1, which is the latin-1 encoding, and file agrees that that's the encoding.
• Using a remote machine (completely different network, running debian with curl 7.88.1) gets the same ISO-8859-1 charset in the content type.
  • but the iconv test prints valid on the remote machine
• Explicitly setting the user-agent header to a browser header returns a content-type using a UTF-8 charset on both machines.
• A friend of mine, on a different network with a different operating system, is experiencing completely unrelated weirdness with google's responses

All I can think is that different google servers return slightly different responses. All that tracks with the weird network things google seems to love 😅

[lpil]

@jtdowney I was imagining a corporate proxy returning a different page :)

I think it would be good to replace these external services with ones run locally, to give us full control.

[jtdowney]

Since it is over TLS, it would have to be an SSL-busting proxy, but they are certainly a thing, just somewhat rarer. My thinking was that Google was doing something different if they thought the end device was a mobile phone because it was on an IPv6 native mobile network. Google works in mysterious ways.

I think with #64, #57 can be updated to really take advantage of local services, and the external services could be moved to an integration suite or just dropped.