Skip to content

fix(connectors): validate and repair the knowledge-base connector fleet - #6757

Merged
waleedlatif1 merged 9 commits into
stagingfrom
chore/connector-validation-audit
Aug 17, 2026
Merged

fix(connectors): validate and repair the knowledge-base connector fleet#6757
waleedlatif1 merged 9 commits into
stagingfrom
chore/connector-validation-audit

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

Five review passes over all 61 knowledge-base connectors, validated against each provider's live documentation or machine-readable spec. Passes 3 through 5 reviewed the previous pass's edits, which is where most of what follows was caught — including regressions the audit itself introduced.

The dominant bug class

shouldReconcileDeletions hard-deletes any stored document absent from a listing that did not set syncContext.listingCapped. So any path that turns a failure into an empty or short listing is a silent mass-delete. Most fixes here are instances of that one shape:

  • confluence — the CQL path paginated with start and read totalSize; neither exists on /content/search. hasMore could never be true, so it only ever returned the first page, and that truncated listing fed deletion reconciliation on every sync.
  • gmail — a failed labels.list was swallowed, producing a query matching nothing. An empty page arriving with a nextPageToken also ended the listing early, producing a short-but-complete-looking listing that reconciliation acted on. Separately, the labels selector stores label ids while validateConfig compared against label names, so any user label was rejected at save time and could never be configured at all (system labels, where id equals name, worked).
  • google-calendarlistingCapped was set nowhere, and maxEvents defaults to 500. Any calendar with more than 500 events in the configured window listed only the first ~500 and reconciliation hard-deleted the rest — on the default configuration, with nothing for the user to misconfigure.
  • wordpress and zendesk — both capped their listings while setting listingCapped nowhere, so every stored document beyond the cap was hard-deleted on every run. wordpress additionally paged by pure offset, which re-numbers mid-run when a post is published, skipping posts.
  • zoomformatDate used local-time getters against an API documented in UTC, so on any non-UTC host the window boundary shifted by a day and that day's recordings could fall outside every window and never be listed. Separately, both bounds are inclusive but the span was computed as to - WINDOW_DAYS, producing 31-day windows that overlapped by a day at each seam — seam-day recordings were listed twice and consumed the budget twice.
  • salesforce — a hardcoded LIMIT bounded the entire result set with no nextRecordsUrl, so any org larger than that limit had the remainder hard-deleted every sync.
  • githublistingCapped was set nowhere, so a repo truncated by the Trees API's own 100,000-entry / 7 MB limit, or by a maxFiles cap, produced a short listing and reconciliation hard-deleted the stored documents outside it.
  • sharepoint — the cap gate missed in-page truncation. Now documents.length < stubs.length, mutation-verified.
  • discord — a non-numeric maxMessages produced NaN, the loop never ran, and the empty listing hard-deleted the stored document.

Other production faults repaired

  • salesforceAPI_VERSION was 'v62.0' substituted into a {version} placeholder that already carries the v, yielding /services/data/vv62.0/. Every REST call 404'd.
  • notion — block-fetch failures were swallowed and the walk never recursed, so pages with tables, toggles, callouts, columns or nested lists were indexed empty.
  • airtable — the content hash included attachment URLs, which expire every two hours, so attachment-bearing records re-indexed on every sync forever.
  • google-docs / google-driveincompleteSearch was read in code but never named in the Drive fields mask, so the partial-corpus guard was dead code.
  • jiralistingCapped was set nowhere, so any source with maxIssues below its real issue count listed only the top N by updated DESC and hard-deleted everything that fell out of that window, every sync. (Correcting an earlier draft of this description: the /search/jql migration is not in this diff — staging already used that endpoint. Only PAGE_SIZE changed.)
  • github — a 30-item batch size capped listings at 15,000 files, so larger repos hit the page ceiling and had reconciliation blocked permanently.

Cap-at-exhaustion, and the deletion wave it implies

The inverse error: setting listingCapped when the cap lands exactly on source exhaustion permanently blocks reconciliation, so documents deleted upstream are stranded in the knowledge base forever. Fixed across gmail, ashby, azure-devops, grain, granola, greenhouse, incidentio and others.

This has a visible consequence on the first sync. Those connectors have been over-flagging, so deletion reconciliation has been suppressed on them — possibly since setup. Once the gate is correct, the documents that were stranded become eligible for removal and will reconcile away. That is the fix working as intended, and it is two-phase (tombstone, then delete only on a second consecutive absence, behind the suspect-listing guard), but it will read as a deletion wave.

Two connectors were worse than over-flagging: hubspot and intercom never set listingCapped at all, so every capped sync hard-deleted every stored document beyond the cap, on every run. Both now set it.

Pagination

"Last-page precision" (shrinking page size to the remaining budget) is only valid when the cursor is position-independent. Over an opaque cursor token no provider documents that a cursor survives a changed page size, and the cap was already enforced server-side — so it bought nothing and risked a mid-sync iterator error. Reverted on airtable, asana, ashby and confluence.

To be precise about scope, since an earlier draft of this description overstated it: no taper was ever committed for greenhouse or intercom. Both were audited as page-number APIs (Harvest documents page as "the n-th chunk of per_page objects", and Intercom's /articles is the same shape), and both already held their page size constant on staging. The only change there is a comment recording why it must stay constant — there was nothing to revert.

Regressions this audit introduced and fixed within it

Net zero versus staging. Listed because the record should be honest about which findings were pre-existing and which were our own — a later pass reviewed each earlier pass's edits, which is the only reason these did not ship.

  • fireflies — an early pass added .catch(() => null) on the JSON parse plus data?.data ?? {}, turning an unparseable 2xx into a confident empty listing. Staging returned data.data bare and threw loudly, so this was never a production fault.
  • github — an early pass narrowed matchesExtension in a way that dropped dotfiles, and separately added a utf-8 blob branch justified by a misattributed quote. Both reverted; net versus staging is TSDoc only.
  • airtable, asana, ashby — page-size tapering over opaque cursors, added then reverted. Staging always sent a constant page size.
  • greenhouse — a 403 classified as transient, appending :partial to a hash that then never converged.
  • google-meet — a 404 → break in the new fetchParticipants; staging had no such function.
  • google-docs — the response byte cap was briefly 800 MB before landing at 100 MB.
  • hubspot — a loose HTML-detection regex, before the shared anchored helper; staging did no detection at all.
  • typeform — an undocumented started enum member; staging omitted the parameter entirely.
  • secure-fetch — retry headers attached enumerably, so the logger's own-enumerable copy wrote upstream Set-Cookie values into sync logs. Staging attached no headers at all.

Security

SecureFetchHeaders declares private setCookies: string[], but TypeScript private is compile-time only — at runtime it is an own enumerable property. @sim/logger copies own enumerable properties into the formatted line and the retry loop logs { error } on every failed attempt, so newly-attached response headers wrote upstream Set-Cookie values into connector-sync logs. Now attached non-enumerably. Scope is the dev/colorize branch only — production reduces errors to .message.

SFTP now requires a host-key fingerprint on all sources, failing closed before the credential is sent (verified by mutation: 7 tests go red). DocuSign's OAuth host is allowlisted to demo/production so a misconfigured value cannot redirect the flow and its bearer token to another origin.

Fabricated doc citations

Eight TSDoc comments quoted provider sentences that do not exist (github ×2, jira, jsm, linear, google-meet, dropbox, youtube) — two of them introduced by this audit. One inverted the usual failure mode: google-meet's code was correct and the comment invented a sentence to justify it, which nearly got a correct fallback reverted. Worth noting that two of three "unverifiable" flags carried into the final pass turned out to be fully documented — re-checking beat reverting on suspicion.

Removed

Evernote, entirely — its auth flow was non-functional, so this is removal rather than repair. Verified complete: the only surviving references are prose in agent-tooling markdown; both icon registries are de-registered. No knowledge-base source, no deployed workflow, and no executed workflow references it.

Customer impact

One-time re-index, deliberate. externalId is unchanged in all three, so these are in-place updates — no deletes, no re-creates, no re-auth:

Connector Why
notion tables, toggles and callouts were previously indexed empty
google-docs tabbed documents were previously unread
hubspot raw HTML now stripped

Other behaviour changes worth watching on first sync:

  • airtable — records with attachment/collaborator/barcode/button cells re-index once. This ends permanent churn rather than causing it.
  • asana — archived projects are now excluded; tasks living only under archived projects reconcile away.
  • github — file ceiling rises 15,000 → 100,000. Repos previously hitting it had deletions permanently suppressed and will now reconcile normally. An add wave, not a delete wave.
  • google-drivegetDocument now throws instead of returning null, so persistent content-read failures become visible failed rows. Expect a one-time rise in reported failures; this is the intended invariant, not a defect.
  • confluence — sources whose listing was previously truncated to a single page will index their full corpus once.
  • monday — consolidating MONDAY_API_VERSION moves four call sites (both selector routes, and webhook create/delete) from an inline 2024-10 pin to 2026-04. To be precise: 2024-10 is in Maintenance, not deprecated, so it was still being served as declared — this is not a deprecation fix. What was genuinely wrong is the divergence: the selector routes listed boards at one schema version while the tools operating on those boards ran at another, inside a single user-visible flow. The consolidation is right, but it jumps roughly five version bumps at those four sites, and monday fails silently here (an unrecognized version resolves to Current rather than erroring). Worth a reviewer's eye rather than being read as a pure refactor.
  • reddit, microsoft-teams, outlook — smaller deliberate one-time re-indexes (edit detection, threaded replies, and diverged conversation dates respectively).
  • s3 — drops rtf from the default extension list and excludes keys containing ./.. path segments. Both are intentional scope filters, so they correctly do not set listingCapped and the affected documents reconcile away.

Operational notes for the first post-deploy sync

Two things will look alarming in logs and are not incidents.

notion's re-index will not finish in one task window. The corpus must re-hydrate against a documented ~3 req/s average while the engine hydrates 5 documents concurrently (SYNC_BATCH_SIZE = 5) against a 30-minute budget (maxDuration: 1800). Expect the run to be killed mid-sync with heavy 429 backoff first. It is not destructive and needs no intervention: batches commit as they go, a page that exhausts its retries rejects and is therefore excluded from deletion reconciliation, and a re-hydrated page classifies unchanged next run. It should converge over several scheduled syncs.

jira's deletion reconciliation will run for the first time on mid-sized projects. PAGE_SIZE 50 → 100 raises the effective listing ceiling from 25,000 to 50,000 (MAX_PAGES = 500). A project in the 25,001–50,000 band previously always hit the ceiling, which set listingTruncated — an unconditional, non-overridable block on reconciliation — so stale rows have accumulated since setup. Those projects now list completely and will tombstone that backlog on the first full sync. This is the correct outcome and it is two-phase behind the suspect-listing guard, but note the honest framing: raising the page size enables deletion for that band rather than fixing a deletion bug. A project larger than 50,000 truncates again and still fails safe.

Final validation sweep

A last read-only /validate-connector pass ran over all 59 changed connectors plus the shared files, verifying every quoted claim against provider specs. It found six real defects, all now fixed:

  • ashby, azure-devopsgetDocument returned null on an API-shape fault. Both connectors set contentDeferred, so these paths were live: a fulfilled null records no failure and no log, and the document silently vanishes.
  • dropbox — 409 covers Dropbox's entire LookupError union, and restricted_content / locked both mean the file still exists. Only not_found is absence now.
  • docusignfetchFormValues swallowed every non-OK status, baking a permanently incomplete document (the hash is metadata-only, so the next sync sees it as unchanged forever).
  • typeformall sent response_type=started,partial,completed, but Typeform documents only partial and completed. An unknown enum member risks a 400 that fails the whole sync, and staging omitted the parameter entirely — so this shipped as a regression. Now sends the widest documented set, which is strictly wider than staging's effective behaviour.
  • github — a utf-8 blob branch was added on a misattributed quote. That sentence describes the encoding request parameter of Create a blob; the GET response is documented as always base64. Branch removed, and two comments corrected that were hiding a real drop (>1 MB files under vnd.github+json 403 rather than returning encoding: none).

The sweep also confirmed the htmlToPlainText blast radius independently: exactly microsoft-teams and mintlify both decode HTML and hash the result, verified by intersecting the htmlToPlainText importers against the computeContentHash callers.

The corollary matters more than the re-index does: for the other ~22 connectors that call htmlToPlainText, the hash is metadata-keyed, so the entity-decode fix is forward-only — already-indexed documents keep their literal &#8217; text until the upstream record's own timestamp changes. The same is true of the Jira ADF extraction improvements, since both jira and jsm hash on metadata. Correcting the existing corpus would need a deliberate rehydrate, not this deploy.

hubspot additionally moved to a shared anchored HTML-detection helper. The loose pattern matched angle-bracketed prose (Reply from John <john@acme.com>), and htmlToPlainText deletes the span and collapses line structure — so a false positive loses data rather than passing it through. Worth fixing now rather than later precisely because the hubspot:v2: bump rewrites every affected document once.

Adversarial re-validation

A final pass re-ran /validate-connector per connector against the current branch state and put every finding in front of an independent agent whose job was to refute it. Zero blockers survived. One major and a set of minors did; two were fixed here.

  • azure-devops (major) — the 20,000-item WIQL ceiling was detected by probing for a matching item with an id beyond the largest returned. The probe is unsound: buildWiql orders by ChangedDate DESC while work-item ids are assigned in creation order, so the highest-id match is almost always inside the returned window. The probe therefore came back empty for genuinely truncated projects, left the listing unflagged, and let deletion reconciliation remove every indexed item outside it — on the default, non-incremental sync path. Now flagged unconditionally. The trade is that a project sitting exactly at the ceiling stops reconciling deletions until a full resync; the alternative was silent data loss on every scheduled sync. Proper pagination (order by [System.Id] ASC, loop on idAfter) is left as a seam with the parameter and a TSDoc pointing at it.
  • docusignfetchFormValues threw on a non-404 status and then caught its own throw, returning []. An earlier fix in this PR was a no-op; the catch now rethrows.

Honest caveat on that pass: it hit a session limit partway through, so roughly the second half of the fleet alphabetically had its refutation stage killed. Those connectors' findings were dropped rather than adjudicated, and the pass reports them as clean — they are unadjudicated, not verified. The findings themselves are recorded and none was a blocker in the first-half sample.

The one item I'd want a decision on before merge

sentry — the endpoint migration narrows scope and will delete issues. Moving from the deprecated /projects/{org}/{project}/issues/ to /organizations/{org}/issues/?project= is correct on its own terms, and it fixes a genuine bug alongside it (staging sorted on Sentry's default date, a mutable key, so under cursor paging an issue whose lastSeen advanced mid-sync jumped the cursor, was skipped, and was hard-deleted as absent).

But the organization endpoint always resolves a date range, and with no statsPeriod/start/end it defaults to roughly 90 days. Issues last seen before that window are absent from the listing and reconcile away. That is a scope narrowing this PR introduces, not one it inherits — the in-code comment currently rationalizes it as the pre-existing "aged out of the query window" semantic, which understates it.

I did not set an explicit window because I could not verify Sentry's maximum accepted value, and a wrong one 400s the endpoint. There are no live Sentry sources, so nothing is at risk today. Options are: accept it as intended scope, pin an explicit widest-supported window, or set listingCapped when the window is the limiting factor. Worth an owner's call rather than my guess.

Known gaps

All 61 connectors have been through the final pass, verified against the four invariants — cap gating, getDocument absence semantics, contentHash parity, and pagination shape.

isScopeSatisfiedBy (lib/oauth/utils.ts) was the one shared-OAuth item flagged as needing scrutiny — it treats a granted non-readonly scope as satisfying its .readonly variant by string suffix, which would be unsound if any provider had a foo.readonly scope where foo is not a superset. Checked and clear. Exactly three .readonly requiredScopes exist fleet-wide, all Google, and all three appear verbatim in their provider scope lists, so exact match already satisfies them. The suffix rule fires for only one: ediscovery (bare) is grantable and is a genuine documented superset of ediscovery.readonly. The other two cannot trigger it — bare meetings.space and forms.responses are not real scopes and appear in no provider list. No false-positive path exists.

Issues found and deliberately not fixed here, each with a reason:

  1. outlook routes a correctness-critical value between listDocuments and getDocument through an untyped syncContext key that connectors/types.ts documents as a cache. Correct today — the engine has exactly one getDocument caller — but the fallback fails silently and permanently if that ever changes.
  2. "Full resync" is destructive on capped sources — it overrides listingCapped and skips the tombstone grace period, hard-deleting hidden documents in one pass. Correct semantics; deserves a UI warning.
  3. gong maps a 404 to an empty page unconditionally, which can hard-delete a source over two consecutive syncs. The fix requires matching a documented error string that could not be verified — Gong's API reference is not publicly fetchable — and guessing at it would be worse than the known behaviour.
  4. rootly swallows timeline-fetch failures into an empty array, baking a permanently timeline-less document (the hash is updated_at-keyed, so it never self-repairs). Pagerduty solved this class with an explicit failure flag; rootly is the remaining instance.
  5. gitlab and notion flag listingCapped when a count merely reaches the cap without confirming more remained. Errs safe — over-suppression, never over-deletion — and pre-existing.
  6. servicenow sorts by a mutable key under offset pagination, so a record updated mid-sync can slide past the window and read as deleted. Sentry's equivalent was fixed here by switching to an immutable sort key.
  7. youtube builds contentHash from different fields on the stub and hydrated paths. Tracing the engine shows it settles to docsUnchanged rather than re-indexing, so the cost is a wasted API call per video per sync — quota burn, not churn.

Verification

29 audits (check:audits), 24/24 type-check, 1,332 tests, CodeQL clean, biome clean.

@vercel

vercel Bot commented Aug 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview Aug 17, 2026 6:34am

Request Review

@cursor

cursor Bot commented Aug 16, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Wide connector sync and deletion-reconciliation behavior changes can cause one-time re-indexing, new failed sync rows, or reconciliation of previously capped sources; Evernote removal breaks any workflows still using that block.

Overview
This PR removes the Evernote integration end-to-end (workflow block, tool API routes, Thrift client, icons, and integration docs) and updates knowledge-base connector docs to 60 built-in connectors (Evernote and SFTP host-key fingerprint wording).

The bulk of the change is connector sync correctness around listings, caps, and hydration. Connectors now set listingCapped only when a cap or filter knowingly hides more data, so deletion reconciliation does not mass-tombstone documents on transient failures or truncated listings. getDocument paths rethrow non-404 errors (Asana, Ashby, Box, Azure DevOps files) so failed hydrations stay visible instead of looking like missing content. Pagination fixes keep page size constant when reusing opaque cursors (Airtable, Asana, Ashby) and parse CQL search cursors correctly in Confluence.

Per-connector behavior changes include: Airtable stable cell hashing and table-ID deep links; Asana batched project walks with a per-call request budget; Ashby feedback select-label rendering and tighter cap flags; Azure DevOps wiki/repo filter staleness and file metadata response shapes; Confluence labels from include-labels on single-item GETs; Box root-folder access failures and representation polling errors surfacing. DocuSign resolves userinfo via getDocusignOAuthUrl. Monday selector routes share MONDAY_API_URL / mondayHeaders.

Docs-only tweaks: Ashby Temporary employment type, minor MDX formatting, LogRocket output table spacing.

Reviewed by Cursor Bugbot for commit 6f37771. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 6f37771. Configure here.

Comment thread apps/sim/connectors/confluence/confluence.ts
Comment thread apps/sim/lib/auth/connectors/providers.ts
@waleedlatif1

waleedlatif1 commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator Author

Addressing Bugbot's one substantive point: Evernote removal does not break any live workflow. Removal was chosen over repair because the integration's auth flow was non-functional — it could not be connected at all.

The rest of Bugbot's "High Risk" summary is accurate and is enumerated under Customer impact in the description.

Note the bot reviewed the pre-rebase commit; the branch has since been rebased onto current staging (conflicts were confined to the three generated tool artifacts, resolved by regeneration and verified idempotent).

@greptile-apps review — this exceeded the 100-file limit, so no review was produced. Highest-value areas: listingCapped gating in apps/sim/connectors/*/ (a listing that omits documents without setting it causes hard deletion), and getDocument returning null for anything other than documented absence.

@greptile-apps

greptile-apps Bot commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR comprehensively repairs knowledge-base connector pagination, cap detection, content extraction, reconciliation safety, and provider-specific API behavior while removing the nonfunctional Evernote integration.

  • Corrects incomplete-listing paths that could cause erroneous deletion reconciliation.
  • Improves connector content hydration, hashing, pagination, and error handling across the provider fleet.
  • Adds security hardening for SFTP host verification, DocuSign OAuth hosts, and retry-header logging.
  • Removes Evernote connector, block, tool, API, registry, icon, and documentation entries.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains from the previous review thread; the Jira ADF helpers now use unknown inputs and safely narrow malformed values before property access.

Important Files Changed

Filename Overview
apps/sim/tools/jira/utils.ts The prior ADF typing concern is resolved by accepting unknown and narrowing before reading type, without changing malformed-node behavior.
apps/sim/connectors/asana/asana.ts Updates multi-project pagination and cap accounting while preserving constant provider page sizes.
apps/sim/connectors/azure-devops/azure-devops.ts Fails safe at the WIQL result ceiling by suppressing deletion reconciliation when completeness cannot be established.
apps/sim/connectors/airtable/airtable.ts Stabilizes content hashing while preserving AI-generated values and recursively rendering nested arrays.
apps/sim/lib/knowledge/documents/secure-fetch.server.ts Prevents attached response headers, including cookies, from being copied into development retry logs.

Reviews (5): Last reviewed commit: "fix(connectors): restore airtable AI Tex..." | Re-trigger Greptile

Comment thread apps/sim/tools/jira/utils.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile-apps review — this exceeds the 100-file limit, so no automatic review is produced.

@cursor review

Both bots last reviewed 6f37771; the branch is now at 95d6fc9346 and the changes since have had no review. Your three earlier findings are all fixed and resolved (confluence opaque-cursor page size, monday API version split, jira ADF any).

What is new since then, and where scrutiny is most valuable:

getDocument returning null for something that is not documented absence. This is the defect class the whole PR is about: the engine hydrates under Promise.allSettled, and only a rejected promise increments docsFailed/failedExternalIds. A fulfilled null on an add hits neither branch, so the document silently disappears with no counter and no log. Fixed in ashby and azure-devops (both set contentDeferred, so those paths are live), dropbox (409 covers the whole LookupError union — restricted_content and locked mean the file still exists), and docusign (fetchFormValues swallowed every non-OK, baking a permanently incomplete document because the hash is metadata-only).

typeformall was sending response_type=started,partial,completed, but Typeform documents only partial and completed. Staging omitted the parameter entirely, so an unknown enum member risking a 400 was a regression introduced here. Now sends the widest documented set.

github — removed a utf-8 blob branch justified by a misattributed quote: that sentence describes the encoding request parameter of Create a blob, while the GET response is documented as always base64.

connectors/utils.ts — new exported looksLikeHtml, now used by hubspot and granola. Worth checking the anchoring: htmlToPlainText strips tags and collapses all whitespace, so a false positive deletes the bracketed span and flattens line structure rather than passing the value through.

Full context, including the two operational notes for the first post-deploy sync (notion exceeding the 30-minute task budget, and jira's deletion reconciliation running for the first time), is in the description.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile-apps review

Three behavioural fixes landed since the last bot pass, all found by an adversarial re-validation of the final branch state:

azure-devops (major). The 20,000-item WIQL ceiling was detected by probing for a matching item with an id beyond the largest returned. That probe is unsound — buildWiql orders by [System.ChangedDate] DESC while work-item ids are assigned in creation order, so the highest-id match is almost always inside the returned window. The probe came back empty for genuinely truncated projects, left listingCapped unset, and let deletion reconciliation remove every indexed item outside the window. This is the default sync path (syncMode defaults to full, so reconciliation is enabled). Now flagged unconditionally.

docusign. fetchFormValues threw on a non-404 status and then caught its own throw, returning [] — an earlier fix in this PR was a no-op. The catch now rethrows.

incidentio. The :partial content-hash marker was applied on any non-OK, including 403 (a key without incident_updates permission returns it for every incident, forever) and 404. That hash never matches the listing stub, so the incident re-hydrated on every sync without converging. Only transient failures mark content incomplete now, matching how greenhouse treats the same class.

Worth the most scrutiny: the azure-devops trade-off. Flagging unconditionally means a project sitting exactly at 20,000 matching items stops reconciling deletions until a full resync. I took that over silent data loss on every scheduled sync. Proper pagination (order by [System.Id] ASC, loop on the existing unused idAfter parameter) is left as a documented seam rather than attempted at a ship gate.

@waleedlatif1
waleedlatif1 force-pushed the chore/connector-validation-audit branch 2 times, most recently from 8971389 to 29a10df Compare August 17, 2026 06:20
Audits every KB connector against its provider's live API documentation and
fixes what the audit found. The dominant defect class is deletion
reconciliation: the sync engine hard-deletes any stored document absent from a
"full" listing, and most connectors had a path where a truncated or errored
listing failed to set `syncContext.listingCapped`.

Highest-impact fixes:

- linear: `getDocument` was dead code. The query declared `$id: ID!` where the
  schema is `issue(id: String!)`, so every call failed variable validation.
- salesforce: `v62.0` was substituted into a `{version}` template that already
  contains the `v`, so every REST call 404'd. SOQL `LIMIT` was also used as a
  page size, silently capping every sync at 200 records.
- notion: only the first level of blocks was fetched, so tables indexed empty.
- microsoft-teams: `/messages` returns messages without replies, so no threaded
  content was ever indexed.
- gmail: an empty page discarded `nextPageToken`, which reads as a complete
  empty listing and hard-deletes every stored thread.
- confluence: the CQL path paginated with `start` and `totalSize`, neither of
  which exists on that endpoint, so label-filtered syncs stopped after one page.
- box: `supportsRefreshTokenRotation` was unset, so Box's rotated refresh token
  was discarded and every credential died on its second refresh.

Removes the Evernote integration entirely: the classic EDAM API is deprecated,
its sandbox is decommissioned, and developer tokens are no longer obtainable.

Makes SFTP host-key verification mandatory, and adds an attendee-PII opt-out to
google-calendar and google-meet (default on, so existing sources are unchanged).

Bumps the contentHash namespace for notion, google-docs and hubspot so existing
documents re-hydrate once and actually receive the content fixes above.
Ship-gate pass over the connector audit. Every finding was re-verified
against the provider's live documentation or machine-readable spec before
being acted on; several pass-3 edits were reverted rather than extended.

Correctness fixes:
- fireflies: a 2xx with an unparseable body returned an empty listing
  instead of throwing. fireflies runs a full sync every time, so a fault
  persisting across two syncs would have tombstoned all indexed docs.
- linear: same shape via `data.issues || {}` on a non-nullable connection.
- greenhouse: a 403 from a key without scorecard permission was treated as
  transient, appending `:partial` to the hash. That never matches the list
  stub, forcing full re-hydration of every candidate on every sync forever.
- google-meet: `fetchParticipants` carried a 404 swallow copied from its
  transcript siblings, freezing every speaker as "Unknown".
- airtable, asana, ashby: reverted page-size tapers applied over opaque
  cursor tokens. The cap was already enforced server-side.
- google-docs: response byte cap resolved to 800MB and could never fire.
- google-forms, google-vault, notion, sharepoint, dropbox: `getDocument`
  now throws on transient failure instead of returning null, which the
  engine reads as absence.

Security:
- Retry headers are attached non-enumerably. TypeScript `private` is
  compile-time only, so `SecureFetchHeaders.setCookies` was an own
  enumerable property that the logger serialized into sync logs.

Docs and dead code:
- Corrected six fabricated doc citations (github, jira, jsm, linear,
  google-meet, dropbox) and removed the Evernote integration entirely.
…ead of returning null

A null from getDocument reads as documented absence, so on an add the
document is dropped with neither a failure counter nor a log. Both
listDocuments paths already throw on the same missing config.
… API version

The CQL search endpoint paginates by opaque cursor, and Atlassian does not
document that a cursor issued against one limit survives a request asking
for a different one. Narrowing limit to the remaining budget was the same
pattern reverted on airtable, asana, and ashby. The page size is now
constant and the cap is applied by trimming the returned page, which keeps
the cap exact without varying the request.

Monday OAuth getUserInfo hardcoded API-Version 2024-10 while every other
monday surface reads MONDAY_API_VERSION, defeating the single-source pin.
Findings from a read-only /validate-connector pass over all 59 changed
connectors, verified against provider specs before acting.

Silent-drop fixes (a fulfilled null from getDocument records no failure and
no log, so the document vanishes):
- ashby: candidate.info returning success with an unusable payload. Ashby
  sets contentDeferred, so this path is live.
- azure-devops: an unresolvable branch, likewise live.
- dropbox: 409 covers the whole LookupError union, and restricted_content
  and locked both mean the file still exists. Only not_found is absence.
- docusign: fetchFormValues swallowed every non-OK status, baking a
  permanently incomplete document since the hash is metadata-only.

typeform: 'all' sent response_type=started,partial,completed, but Typeform
documents only partial and completed. An unknown enum member risks a 400
that fails the whole sync, and staging omitted the parameter entirely, so
this shipped as a regression. Now requests the widest documented set.

github: removes a utf-8 blob branch justified by a misattributed quote —
that sentence describes the encoding REQUEST parameter of Create a blob;
the GET response is documented as always base64. Also corrects two comments
that hid a real drop: >1 MB files under vnd.github+json 403 rather than
returning encoding: none.

hubspot: routes HTML detection through a shared anchored helper. The loose
pattern matched angle-bracketed prose such as an email address, and
htmlToPlainText deletes the span and collapses line structure. This matters
now because the hubspot:v2: bump rewrites every live document once.

youtube: drops an invented channel-ID format quote.
A 403 from a key without incident_updates permission, or a 404, returns on
every sync. Marking those incomplete appended :partial to a hash that then
never matched the listing stub, so the incident re-hydrated forever without
converging. Only a transient failure may mark content incomplete now, which
matches how greenhouse already treats the same class.
…g docusign form-data failures

azure-devops: the 20,000-item WIQL ceiling was probed by asking for a matching
item with an id beyond the largest returned. That probe is unsound — buildWiql
orders by ChangedDate DESC while ids are assigned in creation order, so the
highest-id match is almost always inside the returned window. The probe came
back empty for genuinely truncated projects, left the listing unflagged, and
let deletion reconciliation remove every indexed item outside it. Flag
unconditionally instead; the cost is a project sitting exactly at the ceiling
not reconciling deletions until a full resync.

docusign: fetchFormValues threw on a non-404 status and then caught its own
throw, returning []. The earlier fix was a no-op. The catch now rethrows, so a
transient failure produces a failed row instead of a permanently incomplete
document under a metadata-only hash.
…ssues

airtable: staging rendered object cells with a JSON.stringify fallback, so AI
Text values and nested lookup arrays reached the index. This branch replaced
that with a fixed key-probe list to stop attachment-URL hash churn, but the
probe list has no fallback — aiText ({state,isStale,value}) and nested arrays
rendered to the empty string and vanished from every document, and the
content-derived hash never moved when the text regenerated. Read `value` last,
after the existing probes, and recurse on nested arrays. The generated text is
stable rather than an expiring signed URL, so this does not reintroduce churn.

sentry: maxIssues now feeds the request limit, and Sentry rejects a non-integer.
validateConfig accepts a fractional entry, so a config that saved cleanly would
fail every sync at listing time. Staging was immune only because it sent a
hardcoded page size.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile-apps review

@cursor review

Both bots are several commits stale — Cursor last reviewed 6f37771, Greptile 8d8f343; the branch is now at e4aaec8. Everything since came out of a per-connector re-validation pass in which every finding was put in front of an independent agent tasked with refuting it against three sources: the origin/staging version of the file, the branch version, and the provider's own documentation. Zero blockers survived. These are the changes worth your attention:

airtable — a content-loss regression this branch introduced. Staging rendered object cells with a JSON.stringify fallback, so AI Text values and nested lookup arrays reached the index. This branch replaced that with a fixed key-probe list (to stop attachment-URL hash churn, which was a real fix), but the probe list had no fallback — aiText cells ({state, isStale, value}) and nested arrays rendered to the empty string and vanished from every document, and the content-derived hash never moved when the text regenerated. Now reads value last, after the existing probes, and recurses on nested arrays. The generated text is stable rather than an expiring signed URL, so this does not reintroduce the churn the probe list exists to prevent.

azure-devops — silent deletion past the WIQL ceiling. The 20,000-item truncation guard probed for a matching item with an id beyond the largest returned. That probe is unsound: buildWiql orders by [System.ChangedDate] DESC while work-item ids are assigned in creation order, so the highest-id match is almost always already inside the returned window. The probe came back empty for genuinely truncated projects, left listingCapped unset, and let deletion reconciliation remove every indexed item outside the window — on the default sync path, since syncMode defaults to full. Now flagged unconditionally. The trade-off worth reviewing: a project sitting exactly at 20,000 stops reconciling deletions until a full resync. Proper id-keyset pagination is left as a documented seam on the existing unused idAfter parameter.

docusignfetchFormValues threw on a non-404 status and then caught its own throw, returning []. An earlier fix in this PR was a no-op; the catch now rethrows.

incidentio — the :partial hash marker fired on any non-OK, including 403 (a key without incident_updates permission returns it for every incident, forever) and 404, so the hash never matched the listing stub and the incident re-hydrated on every sync without converging. Only transient failures mark content incomplete now.

sentrymaxIssues now feeds the request limit, and Sentry rejects a non-integer. validateConfig accepts a fractional entry, so a config that saved cleanly would fail every sync at listing time. Floored at parse.

lib/knowledge/documents/utils.ts — resolved against #6770, which removed non-TSDoc inline comments from this file. Kept our functional changes (403 retryable only with positive rate-limit evidence, headers passed into isRetryableError) while honoring that comment cleanup rather than reverting it.

Generated tool artifacts were regenerated against current staging and verified idempotent (4,878 tools, 280 doc entries, zero diff on a second run).

@waleedlatif1
waleedlatif1 merged commit 31521d6 into staging Aug 17, 2026
29 checks passed
@waleedlatif1
waleedlatif1 deleted the chore/connector-validation-audit branch August 17, 2026 07:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant