Add augmented translation for incoming messages and replies

[senadaruc]

Summary

Adds Settings → Translation (off by default). It translates chats in apps the user picks: WhatsApp, Teams, Telegram, Slack, Outlook, Mail and Messages are suggested.

• Incoming messages. Every 2 s, and right after scrolling, TranslationCoordinator does the following:
  • Captures the frontmost chat window with the new WindowScreenshotService.captureActiveWindow.
  • Reads it with Vision OCR, using language hints via the new optional ScreenTextExtractor.recognitionLanguages.
  • Groups the text into messages with MessageBlockGrouper: only the conversation area above and right of the reply field is read, and timestamps and low-confidence lines are dropped.
  • Draws each translation under its message in a click-through overlay (TranslationOverlayController).
  • Skips unchanged windows using a pixel fingerprint.
• Replies. When the user writes in their reading language in a chat whose language is known (the language of the newest incoming message, tracked per window title), the reply is translated after a 700 ms pause and shown in a card under the field. ⌃⌥T (recordable) replaces the draft through select-all and paste (SuggestionInserter.replaceFieldText), with the clipboard restored. It never sends. The shortcut tap (TranslationHotkeyTap) exists only while the card is visible.
• Engines. Apple Translation (TranslationSession(installedSource:target:), macOS 26+, on-device) handles its roughly 20 languages, at about 22 ms per message once warm on an M-series Mac. For pairs it lacks, such as Macedonian, the selected Open Source model translates with a few-shot prompt (FewShotTranslationPrompt), because the bundled models are base models. Results are cached in memory only (TranslationCache).
• TranslationLanguagePolicy is deliberately conservative: at least 12 letters, at least 0.8 confidence, and no links or numbers. It recognizes Macedonian by its alphabet, because Natural Language reports Macedonian as Bulgarian.

Translation respects Cotabby's own gates (globally on, not paused, app not disabled, Low Power Mode), never reads secure fields, and stores, logs or sends nothing.

Validation

xcodebuild test ... CODE_SIGNING_ALLOWED=NO
Executed 2663 tests, with 14 tests skipped and 1 failure

The failure is AXTextGeometryResolverTests.test_resolveCaretRect_returnsRealGeometry_forNativeTextField, which reads live Accessibility geometry and also fails on unmodified main on this machine.

25 new tests cover the language policy (Turkish, Macedonian vs Russian, short text, links, numbers), the cache, few-shot prompt building and output trimming, message grouping on synthetic OCR boxes (wrapped lines, chat list and draft excluded, timestamps dropped), conversation-language tracking, engine routing with fake engines, and preference persistence.

Checked by hand on this Mac: Apple Translation tr→en and en→tr outside the app (375 ms for the first call, 22 ms after).

Draft: not yet exercised end to end in a real chat. Remaining work: tune the grouper on recorded WhatsApp and Teams OCR output, measure Release latency and CPU, and check overlay placement on multiple displays.

Risk / rollout notes

• New Settings category (translation) and new UserDefaults keys (cotabbyTranslation*).
• Reads chat windows through Screen Recording while enabled; off by default, and the Privacy section describes the behavior.
• The local-model path shares the llama runtime lock with autocomplete.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}