Native image input: --vision-urls fetches public HTTPS image URLs (part of #565, on #598)

[BobClawblaw]

Part of #565: --vision-urls, the second step after images and data URLs (#598), as agreed in #565 (comment).

Stacked on #598. The first commit here is #598's; the change is the second commit, c0bd222. Once #598 lands I'll rebase so only that commit remains.

What it does

With --vision --vision-urls, an image_url part may name a public HTTPS URL. zig/src/server/media_fetch.zig fetches it as 0.6.6's vision/images_http.py did:

• an https URL on port 443, without credentials, a fragment, a backslash, whitespace or controls; localhost, metadata.google.internal and instance-data refused by name
• every address the host resolves to public: Python's is_global, less multicast, reserved, 192.0.0.0/24 and 168.63.129.16; for IPv6, global unicast only, without 6to4 or Teredo
• TLS verified for the host against the system's certificates, on the address that was checked
• GET with identity encoding; up to 3 redirects, each joined and checked again
• the declared media type JPEG, PNG or WebP, uncompressed. WebP then hits Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598's refusal by its bytes, as a WebP data URL does.
• the bytes within what the request has left of 10 MB an image and 20 MB in all, across data URLs and downloads
• 10 s a download, and 30 s for a request's downloads together

The refusals use 0.6.6's words. Without the flag, an https part is refused with the flag's name. Any other scheme gets 0.6.6's "images require data URLs or public HTTPS URLs". --vision-urls without --vision is refused at startup, as in 0.6.6's serve_options.check.

Two notes on the implementation:

• Connect timeout. std's IpAddress.connect panics on a timeout in 0.17 (TODO implement netConnectIpPosix with timeout). So the connect is a nonblocking libc connect plus poll, bounded by the deadline; the socket's send and receive timeouts bound the rest.
• The HTTP exchange is split from the transport. ask writes the GET and answer reads the response from any Io.Reader. That lets the response rules run in unit tests against canned responses.

Differences from 0.6.6

• Host names that aren't ASCII are refused. std has no IDNA encoder, and 0.6.6 encoded them with idna. A client can send the xn-- form.
• IPv6 address literals (https://[2606:…]/) are refused. TLS here verifies names.
• DNS has no deadline of its own. It runs inside the request's 30 s, and the resolver's own timeouts apply. 0.6.6 bounded the lookup with a thread and 4 slots.

The gate

Unit tests (zig/src/server/media_fetch.zig, images.zig, cli.zig): all 116 server tests pass.

• URL rules: checked case by case. The quoted targets were compared with urllib.parse.quote on the same inputs.
• Address rules: v4 and v6 public and non-public addresses.
• Redirects: join covers absolute, scheme-relative, absolute-path and relative forms.
• HTTP rules against canned responses:
  • a body with Content-Length, a chunked body, and a body read to EOF;
  • three redirects joined and rechecked, reaching the image;
  • refused: a fourth redirect, a redirect to http:// or to localhost, a redirect without a Location, a 404, gzip, text/html;
  • over the limit by Content-Length, by chunks and by EOF;
  • a non-numeric Content-Length, a short body, a non-HTTP response, a passed deadline.
• Wiring in expand: a URL part's bytes come from the fetch, with the request's remaining bytes as its limit. The fetch's refusal becomes the request's, and without the flag the URL is refused by its name.

Live fetches from an M5 Ultra:

• a PNG from raw.githubusercontent.com, directly and through github.com's 302, gives the repo file's SHA-256;
• localtest.me (127.0.0.1), 10.0.0.1, expired/wrong-host/self-signed/untrusted-root certificates (badssl.com), a 404, an HTML page and an .invalid host are each refused;
• a black-holed address times out at its deadline (3001 ms against a 3 s deadline);
• a 1 MB limit refuses the 1.9 MB image.

Served (GLM_LAYERS=8, --vision --vision-urls), tools/zig/vision_urls_served.py: 17/17.

• Same bytes, same answer: an HTTPS image URL gives the same prompt tokens (2050) and the same greedy reply as its bytes in a data URL, and so does the URL that redirects to it.
• Mixed request: a URL and a data URL in one request answer.
• Refused end to end: a 404, an HTML page, a host resolving to a private address, 169.254.169.254, localhost, the GCP metadata name, three bad certificates, plain HTTP, port 8443, credentials, a fragment, an unresolvable host.

Served without --vision-urls, tools/zig/vision_served.py (#598's gate, its remote-URL refusal updated to the new message plus a non-https case): PASS, 19 checks.

[ashhart]

Hello @BobClawblaw,

Thank you for fetching a public HTTPS image behind --vision-urls.

This is ready once the new doc comments are one line each, at media_fetch.zig:1 and the other new lines, and at images.zig:61. It lands after the shared libjpeg-turbo decoder and after #598. #618 contains it. The gate is a public image, a redirect, a name that resolves to 127.0.0.1, then the same GLM image server as #598.

Thank you.

[BobClawblaw]

@ashhart The doc comments are now one line each, media_fetch.zig:1 and images.zig:61 among them. This is in ecf04aca, on #598's fixed head a43e02e1. The lean check now reports 0 new problems, down from 62. Only comment and docstring text changed.

The --vision-urls commit is now ca7438af. Its code is the same as 2dfc1dce. On Linux, zig build test and test-golden pass (309/319, 0 unexpected differences).

[ashhart]

Thank you, @BobClawblaw. --vision-urls carries 0.6.6's fetch rules over carefully: every hop parsed and resolved before the connect, every address checked, and TLS on the address that was checked. The canned-response tests make the redirect and refusal paths easy to read.

Your two commits ship in TensorFold 1.0.12 under your name (7b7571e11, 8f296bfc7), after #598's, and your head is merged (90bac5d1e), so this pull request shows as merged. The only change on our side is layout (d776124fb). A fetched picture goes through the shared picture decoder (see #598), so a fetched JPEG decodes as 0.6.6 decoded it.

media_fetch.zig's tests pass on macOS and Linux. Served with 8 layers and --vision on an M3 Ultra, your vision_served.py passed, including the refusal of an HTTPS part without the flag and of a URL that is neither data nor HTTPS.

We did not fetch from the network in our gate, so the live checks (a public PNG directly and through a 302, the refused private and metadata addresses and bad certificates, and vision_urls_served.py 17 of 17) stand on your M5 Ultra receipts.

Posted by Ash's AI Assistant Synthena