fix: list_public_tokens_tool asks the model to copy the token exactly and check it

CHANGELOG.md

@@ -14,6 +14,7 @@
   - **Billing:** the tool told the model each item "may be billed", and a duplicate result said a second one "would only add a charge". The API is free during Public Preview; the $0.02 per item applies to Feedback Agent at General Availability. The duplicate check is now explained as avoiding filing the same problem twice.
   - **Categories:** adds `routing_issue`, the general routing category the API docs use.
   - **Location:** `coordinates` is now optional, so feedback that isn't about a place, such as most feedback about the app, no longer needs a made-up location. The API still requires lat/lon, so the tool sends 0, 0, as the Feedback API team recommends until location becomes optional there. That feedback skips the duplicate check, since every report without a place sits at 0, 0.
+- **`list_public_tokens_tool`: asks the model to copy the token exactly and check the copy (#332).** A dashboard ChatGPT built for For Five failed with "invalid Mapbox access token": the tool returned the right token, but the model retyped it into the page and changed a character, which is easy with about 90 characters of base64. The result now says to copy the token exactly, never retyped, and, when the agent wrote it into a file and can run commands, to check the copy from that file: `grep` the `pk.` token out of the file and call `GET /tokens/v2`, which returns `TokenValid`, or `TokenInvalid` when a character changed. Checked against the live API with a valid token and one with a single character changed.
 
 ### Maintenance
 

src/tools/list-public-tokens-tool/ListPublicTokensTool.ts

@@ -25,6 +25,20 @@ const GUIDANCE =
   `if none matches, ask the user to create one at ${CREATE_TOKEN_URL} (add the site's URL under URL ` +
   'restrictions) and use that instead. The default token is fine for local prototypes.';
 
+/**
+ * Models often retype a long token instead of copying it, and one changed
+ * character makes it invalid (seen in a ChatGPT-built dashboard whose map
+ * failed with "invalid Mapbox access token"). Agents that write files can
+ * usually run commands, so this asks them to check the copy from the file.
+ */
+const COPY_CHECK =
+  'Copy the token exactly as shown, never retyped: it is long, and one changed character makes ' +
+  'Mapbox reject it with "invalid Mapbox access token". If you write it into a file and can run ' +
+  'commands, check the copy from the file instead of retyping it, for example:\n' +
+  "TOKEN=$(grep -o 'pk\\.[A-Za-z0-9._-]*' index.html | head -1); " +
+  'curl -s "https://api.mapbox.com/tokens/v2?access_token=$TOKEN"\n' +
+  '"code":"TokenValid" means the copy is right; "TokenInvalid" means a character changed, so copy it again.';
+
 /** What the Tokens API returns for each token; only pk entries are kept. */
 interface ApiToken {
   usage?: string;
@@ -243,7 +257,7 @@ export class ListPublicTokensTool extends MapboxApiBasedTool<
         ? `Public tokens (${output.tokens.length}):\n${lines.join('\n')}`
         : undefined,
       output.recommended
-        ? `Recommended: ${output.recommended.token}\n${output.recommended.reason}`
+        ? `Recommended: ${output.recommended.token}\n${output.recommended.reason}\n\n${COPY_CHECK}`
         : output.tokens.length > 0
           ? 'Every public token on this account is restricted to other URLs, so none is recommended.'
           : 'This account has no public tokens.',

test/tools/list-public-tokens-tool/ListPublicTokensTool.test.ts

@@ -98,6 +98,15 @@ describe('ListPublicTokensTool', () => {
       'account.mapbox.com/access-tokens/create'
     );
     expect(textOf(result)).toContain('Recommended: pk.default.sig');
+    // Models retype long tokens; the text asks them to copy it exactly and
+    // check the copy from the file they wrote it into.
+    expect(textOf(result)).toContain('never retyped');
+    expect(textOf(result)).toContain(
+      "grep -o 'pk\\.[A-Za-z0-9._-]*' index.html"
+    );
+    expect(textOf(result)).toContain(
+      'https://api.mapbox.com/tokens/v2?access_token=$TOKEN'
+    );
   });
 
   it('recommends a token restricted to the site_url', async () => {