feat(models): add Google Cloud Translation v2 and v3 as LID models

[malteos]

Closes #11.

Why

Adds Google Cloud Translation's language detection as two models, one per API edition, so the commercial detector can be compared against the open models on the leaderboard:

model_id	Edition	Auth
GoogleTranslate-v2	Basic (v2) detect	API key
GoogleTranslate-v3	Advanced (v3) detectLanguage	Application Default Credentials

Builds on #17: both wrappers return the API's confidence as the prediction score.

How the editions differ

Per the official reference, three differences matter for detection:

• Auth. Basic accepts API keys and service accounts. Advanced "requires service account authentication that's integrated with IAM roles" and "does not support API keys". Trying anyway fails on both transports:

  401 API keys are not supported by this API. Expected OAuth2 access token
  or other authentication credentials that assert a principal.
  [reason: "CREDENTIALS_MISSING"]
  

• Batching. v2's q is repeatable, so one request detects a whole list. v3's content is a single string, so v3 needs one request per text.

• Confidence. v2 documents both isReliable and confidence as deprecated and recommends "not basing any decisions or thresholds" on them. v3 carries no such notice. The README score table records this.

Do they agree?

On 300 samples of commonlid_nano, the two editions return the same prediction for 297 (99.0%) and identical confidences to 1e-6 for 298. Every disagreement is the same labelling difference, not a detection one: for Tagalog, v2 returns fil and v3 returns tl, with identical confidence.

Those codes conform to two distinct ISO 639-3 languages, Filipino fil and Tagalog tgl, and the benchmark's gold labels use both. This is deliberately not remapped for Google alone: GlotLID, OpenLID-v2, cld3 and funlangid also predict fil and get none of the tgl samples right.

Rename: GoogleTranslate to GoogleTranslate-v2

An earlier revision of this PR registered the v2 model as GoogleTranslate. It is renamed so the edition is explicit. The id is persisted in result paths, summaries and cache keys, so the one published result is migrated rather than recomputed in https://huggingface.co/datasets/commoncrawl/commonlid-results/discussions/7. There, predictions.jsonl is byte-identical and summary.json differs only in model_id. Merge that PR together with this one.

Credentials

Both models read only the process environment; neither loads .env files.

• GoogleTranslate-v2 needs GOOGLE_TRANSLATE_API_KEY.
• GoogleTranslate-v3 needs Application Default Credentials, via gcloud auth application-default login or GOOGLE_APPLICATION_CREDENTIALS. Its project id resolves from GOOGLE_TRANSLATE_PROJECT_ID, then GOOGLE_CLOUD_PROJECT, then the ADC project, then the ADC quota project.

Missing credentials fail at load() with a message naming what to set.

Cost

Detection is billed per character. v2 detection costs $20 per million characters, with the first 500K each month free. Measured on the cleaned text that is actually sent, since predict() applies the OpenLID normer first:

dataset	samples	billable chars	cost
commonlid_nano	1,507	305,431	~$6
commonlid	373,230	76,713,413	~$1,534

The character counts are the same for both editions, but v3 sends one request per text. commonlid_nano has been evaluated with v2 and published. No full commonlid run has been done, and none should be without agreeing the spend first. Re-runs are free because the evaluator's PredictionCache covers them.

Implementation notes

• Both wrappers short-circuit blank text before calling the API, since a rejected request is still billed.
• Both retry rate-limit and transient errors with bounded backoff. A dead endpoint or bad credentials raise rather than being scored as abstention.
• Detected BCP-47 codes are cut at the first - before the existing ISO 639-3 conformance pass, so zh-CN becomes zho.
• discover_supported_languages() asks the API. v2 yields 187 conformed codes from 195 raw, v3 188 from 196, with nothing dropped by conformance.
• v2 re-splits each batch to stay under the per-request segment and size caps, and sends the chunks concurrently. The client library ignores client_options={"api_key": ...} and falls through to ADC, so the key is passed as google.auth.api_key.Credentials.
• v3 fans a batch out over a thread pool of 16, well under the project quota of 6,000 requests per minute, and clips text at v3's 30,000-code-point limit.
• New optional extra commonlid[google-translate] covers both. It is mirrored into dev so the tests exercise the real client, and added to all. New make install-google-translate.
• The v2 wrapper imports google.cloud.translate_v2 as a module. It ships no py.typed, so once v3's typed import is present, mypy otherwise reports it as a missing attribute.

Testing

make check passes: 295 tests, 95.8% coverage, ruff and mypy strict clean.

The model tests never touch the network. They swap in fake clients or monkeypatch the library out of sys.modules. They cover missing extras and credentials, v3's project-id resolution order, code conformance, confidence surviving an unmappable code, blank-text short-circuiting with zero API calls, v2 chunking, order preservation across thread pools, and the retry and give-up paths.

Verified against the live API with both editions:

{"text": "Joem, nag-iipon muna bago buntisin ang GF", "pred": "fil", "score": 1.0, "model": "GoogleTranslate-v2"}
{"text": "Joem, nag-iipon muna bago buntisin ang GF", "pred": "tgl", "score": 1.0, "model": "GoogleTranslate-v3"}

[malteos]

The CommonLID nano results are already on the leaderboard: https://huggingface.co/spaces/commoncrawl/commonlid

[laurieburchell]

lgtm

[pjox]

Maybe we should put an explicit reference in the table to the fact that it is V2 and not V3? I'm having trouble finding the list of languages by version, but my understanding was that V3 had more coverage. It's sad we cannot use it, but we should make it explicit which version it is

[malteos]

I think v2 just refers to the API version. I couldn't find any information on the actual model.

[malteos]

I've run a quick test:

The comparison, on 300 samples of commonlid_nano:

┌──────────────────────────────┬───────────────────┐
│                              │                   │
├──────────────────────────────┼───────────────────┤
│ Identical predictions        │ 297 / 300 (99.0%) │
├──────────────────────────────┼───────────────────┤
│ Identical confidence to 1e-6 │ 298 / 300         │
├──────────────────────────────┼───────────────────┤
│ Accuracy vs gold, v2         │ 75.67%            │
├──────────────────────────────┼───────────────────┤
│ Accuracy vs gold, v3         │ 76.67%            │
└──────────────────────────────┴───────────────────┘

Every disagreement is the same thing, and it is a labelling difference rather than a detection one. For Tagalog, v2 returns fil and v3 returns tl, with byte-identical confidence on the same input. Those conform to fil and tgl. CommonLID's gold label is tgl, so v3 wins those three purely on code vocabulary.

v2 raw: [('fil', 1), ('fil', 0.9796053171157837)]
v3 raw: [('tl', 1.0)], [('tl', 0.9796053171157837)]

That is strong evidence the same detector sits behind both, which the model field in v3 hinted at but Google never states.

How v3 differs in the wrapper. It authenticates with Application Default Credentials, since v3 rejects API keys. The project id resolves from GOOGLE_TRANSLATE_PROJECT_ID, then GOOGLE_CLOUD_PROJECT, then ADC. Your user ADC reports no project of its own, only the quota project commonlid, so the wrapper falls through to that. And because v3 takes one text per call, a batch becomes a thread pool rather than a single request, which makes v3 far more expensive in requests for a full run.