关注刷新和 MCP 的 get_follows 改成按关注分组读名单

docs/AGENT.md

@@ -92,7 +92,7 @@ Every other `get` (`/bangumi/`, `/festival/`, `/list/` without `bvid`, YouTube)
 
 - `favorites` without `folderId`: `{ folders: [{ id, title, count, private?, included?, synced? }] }`, 稍后再看 first with id `"toview"`, then the user's own folders (Bilibili's folder list, read live). `count` is Bilibili's; `private` is Bilibili's 私密 (the lowest bit of the folder's `attr`); `included` means ticked in the 分拣台's 收藏夹设置; `synced`, on a ticked folder with a local copy (`triage_snapshot_<id>`), is how many videos the copy holds, and is absent while the folder has none yet. With `folderId`: `{ folderId, source, total, page, hasMore, items }`, 40 a page. A folder ticked in the 分拣台 is read from its local copy (`source: "local"`, as of its last read, no Bilibili request) without the videos 取消收藏 since that read (the copy keeps those until its next read; `triage_decisions_<id>` names them); `toview` is read whole from Bilibili in one request; any other folder is one Bilibili page (`source: "bilibili"`). `query`: words that must all appear, case-insensitive, matched on title, author, tags, note and summary for the local copy and `toview`; for other folders Bilibili's own favorites search gets it as `keyword` (titles only, and `total` is then left out). An item: `{ bvid, url, title, author, duration, favTime, invalid?, progress?, tags?, note?, verdict?, summary?, kept? }`: `tags` are the user's 分拣 tags (and 摘读记录 tags), `note` the 备注, `verdict` `keep` / `drop` / `unsure` (`MdgTagCore.verdictOf`), `summary` the 细看's one-liner, `kept` the 保留 mark, `progress` seconds watched in 稍后再看 (-1 finished).
 - `favorites` with `all: true` asks Bilibili nothing: `{ all: true, source: "local", unsynced?, total, page, hasMore, items }`, the local copies of every ticked folder as one list (`folderId` limits it to one), 300 a page, each video once (`TriageCore.mergeFolderItems`, the 分拣台's 所有收藏夹; 取消收藏 ones left out as above). An item is the one above plus `folders` (the ids of every ticked folder whose copy holds it) and `stage` (`none`, `coarse`: a 粗看 verdict, `fine`: a 细看 one). Filters, all applied together: `verdict` (`none` / `keep` / `drop` / `unsure`), `stage` (`none` / `coarse` / `fine`), `tag` (one tag name the video carries), `invalid` (`true`: only videos Bilibili marks 已失效, `false`: only the others), `query` as for a local copy; a wrong `verdict`, `stage` or `folderId` is `BAD_PARAMS`. `unsynced` lists the ticked folders with no copy yet, which the list therefore lacks: `sync` reads them.
-- `follows`: `{ source, total, page, hasMore, items: [{ mid, name, sign, groups, tags? }] }`, 50 a page. With a synced 关注 list (the 分拣台's 关注) it is read from there (`source: "local"`, with the user's UP 标签 as `tags`, and `query` matched on name, sign, groups and tags); else one Bilibili page of the followings plus the group names (`source: "bilibili"`, `query` is not applied and `queryIgnored: true`). `groups` are 关注分组 names, 特别关注 and 默认分组 included.
+- `follows`: `{ source, total, page, hasMore, items: [{ mid, name, sign, groups, tags? }] }`, 50 a page. With a synced 关注 list (the 分拣台's 关注) it is read from there (`source: "local"`, with the user's UP 标签 as `tags`, and `query` matched on name, sign, groups and tags); else Bilibili's whole list, read by 关注分组 as the 分拣台's 刷新 reads it (`followReadGroups` in `triage/follow-bg.js`: 500 a request, about 10 requests for 4000 follows, checked against the followings' total and kept 3 minutes so paging reads it once), in Bilibili's group order (`source: "bilibili"`, `query` matched on name, sign and groups); a read short of the total falls back to one page of the followings (`query` not applied, `queryIgnored: true`) and is not kept. `groups` are 关注分组 names, 特别关注 and 默认分组 included.
 - `history`: `{ items: [{ bvid?, url, title, author, viewAt, progress, duration, kind }], cursor }`, 30 a call, newest first. `cursor` is opaque; pass it back for the next page, `""` at the end. `kind` is Bilibili's business (`archive` for a video, also `pgc`, `live`, `article`…); `progress` is seconds, -1 finished; `viewAt` seconds since the epoch.
 
 `collection` and `collectionWrite` (`extension/agent-collections.js`) work on any site whose adapter has `collections` (§5): Bilibili and YouTube. `kind` is one of the adapter's kinds, else `UNSUPPORTED_KIND`; Bilibili has `watch_later` (稍后再看, `add` and `remove`; its favorites, follows and history stay in `favorites`, `follows`, `history`), through `triage/triage-bg.js`'s own 稍后再看 requests in the service worker; YouTube has `watch_later`, `playlists`, `playlist` (with `id`), `likes`, `subscriptions`, `history`, and writes to `watch_later` and `playlist` only (`add`, `remove`). Calls run one at a time, a second apart, and stop with `TIMEOUT` while still queued at the deadline.

extension/agent-mydata.js

@@ -1,6 +1,6 @@
 // MoonDigest's agent interface, docs/AGENT.md §2: the user's own Bilibili data (favorites, 稍后再看, follows, watch
 // history) for the MCP server, read only. Only GETs, through triage-bg.js's B站 channel (triageBiliGet), one call at a
-// time, one page per call (稍后再看 and the 分拣台's local copies are read whole, then paged).
+// time, one page per call (稍后再看, the follow list and the 分拣台's local copies are read whole, then paged).
 // The worker loads it after triage-bg.js and follow-core.js, whose helpers it uses.
 (() => {
   const K = MdgLimits.KEYS;
@@ -128,20 +128,67 @@
     groups: FollowCore.groupNames(FollowCore.groupsOf(String(mid), D), D)
   });
 
-  // The 关注 the 分拣台 synced (with UP 标签; query searches name, sign, groups and tags), else B站's list a page at a time.
+  // B站's whole follow list, read by group as the 分拣台's 刷新 reads it (follow-bg.js followReadGroups), and followings'
+  // total to check it is whole: its requests GAP_MS apart, through this file's channel (B站 risk control is THROTTLED at
+  // once, as for every other call). A whole list is kept FOLLOWS_KEEP_MS, so paging through it reads B站 once; one short of
+  // the total is not, the next call reads again.
+  // ponytail: one request a group with members, plus one per 500 more; past ~45 groups a call outruns agent-bridge.js's budget
+  const FOLLOWS_KEEP_MS = 3 * 60 * 1000;
+  let followsRead = null; // { at, promise }
+  function biliFollows() {
+    if (!followsRead || Date.now() - followsRead.at > FOLLOWS_KEEP_MS) {
+      let last = 0;
+      const spaced = async (url) => {
+        const wait = last + GAP_MS - Date.now();
+        if (wait > 0) await new Promise((r) => setTimeout(r, wait));
+        try {
+          return await triageBiliGet(url);
+        } finally {
+          last = Date.now();
+        }
+      };
+      const promise = (async () => {
+        const got = await followReadGroups((call) => call(), undefined, spaced);
+        const first = await spaced(`${API}/x/relation/followings?vmid=${await triageMid()}&pn=1&ps=${FOLLOW_PAGE}&order=desc`);
+        return { ...got, total: Number(first?.total) || 0 };
+      })();
+      const read = { at: Date.now(), promise };
+      const drop = () => followsRead === read && (followsRead = null);
+      read.promise.then((r) => r.fresh.list.length !== r.total && drop(), drop);
+      followsRead = read;
+    }
+    return followsRead.promise;
+  }
+
+  // The 关注 the 分拣台 synced (with UP 标签), else B站's list read whole; query searches name, sign, groups (and tags).
   async function follows({ page, query }) {
     const pn = pageOf(page);
     const q = typeof query === "string" ? query.trim() : "";
     const D = await MdgLimits.followGet(["list", "people", "groups", "tags", "tag_map"]);
+    let rows;
+    let source = "local";
     if (D.list?.list?.length && !D.list.partial) {
       const names = new Map((D.tags || []).map((t) => [t.id, t.name]));
-      const rows = D.list.list.map((mid) => {
+      rows = D.list.list.map((mid) => {
         const tags = FollowCore.liveTags(mid, { map: D.tag_map, tags: D.tags || [] }).map((t) => names.get(t)).filter(Boolean);
         return { ...upRow(mid, D.people?.[mid], D), ...(tags.length && { tags }) };
       });
-      const hit = q ? rows.filter((r) => matches(q, [r.name, r.sign, ...r.groups, ...(r.tags || [])])) : rows;
-      return { source: "local", ...slice(hit, pn, FOLLOW_PAGE) };
+    } else {
+      const { fresh, people, tags, total } = await biliFollows();
+      // Not the whole list (a group read short, or a follow meanwhile): one page of followings, as before, rather than
+      // a list missing people. The 分拣台's 刷新 falls back the same way.
+      if (fresh.list.length !== total) return followingsPage(pn, q);
+      rows = fresh.list.map((mid) => upRow(mid, people[mid], { list: fresh, groups: groupsOf(tags) }));
+      source = "bilibili";
     }
+    const hit = q ? rows.filter((r) => matches(q, [r.name, r.sign, ...r.groups, ...(r.tags || [])])) : rows;
+    return { source, ...slice(hit, pn, FOLLOW_PAGE) };
+  }
+
+  const groupsOf = (tags) => (Array.isArray(tags) ? tags : []).filter((g) => g.tagid > 0).map((g) => ({ id: g.tagid, name: String(g.name || "") }));
+
+  // One page of B站's followings with the group names; query not applied.
+  async function followingsPage(pn, q) {
     const mid = await triageMid();
     const data = await triageBiliGet(`${API}/x/relation/followings?vmid=${mid}&pn=${pn}&ps=${FOLLOW_PAGE}&order=desc`);
     const tags = await triageBiliGet(`${API}/x/relation/tags`);
@@ -151,7 +198,7 @@
         special: Object.fromEntries(list.filter((x) => x.special === 1 || (x.tag || []).includes(TriageUi.FOLLOW_SPECIAL)).map((x) => [String(x.mid), 1])),
         groups: Object.fromEntries(list.map((x) => [String(x.mid), (x.tag || []).filter((t) => t > 0)]))
       },
-      groups: (Array.isArray(tags) ? tags : []).filter((g) => g.tagid > 0).map((g) => ({ id: g.tagid, name: String(g.name || "") }))
+      groups: groupsOf(tags)
     };
     const total = Number(data?.total) || 0;
     return {

extension/agent-mydata.selftest.js

@@ -11,7 +11,7 @@ const gaps = [];
 const fastGap = (fn, ms) => (ms > 0 && ms <= 1000 ? (gaps.push(ms), setImmediate(fn)) : setTimeout(fn, ms));
 const ctx = vm.createContext({ TextEncoder, URL, URLSearchParams, console, setTimeout: fastGap, clearTimeout, setInterval, clearInterval, AbortController });
 const order = [...fs.readFileSync(path.join(__dirname, "background.js"), "utf8").matchAll(/importScripts\(([^)]*)\)/g)].flatMap((m) => m[1].match(/"[^"]+"/g).map((f) => JSON.parse(f)));
-assert.ok(order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-core.js"), "agent-mydata.js loads after the triage helpers it uses");
+assert.ok(order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-core.js") && order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-bg.js"), "agent-mydata.js loads after the triage helpers it uses");
 let M = null;
 for (const file of order) {
   if (file === "agent-mydata.js") ctx.MdgAgentBridge = { addMethods: (m) => (M ??= m) };
@@ -59,7 +59,7 @@ ctx.fetch = async (url, init = {}) => {
 };
 const READS = new Set([
   "/x/web-interface/nav", "/x/v3/fav/folder/created/list", "/x/v2/history/toview/web", "/x/v2/history/toview",
-  "/x/v3/fav/resource/list", "/x/relation/followings", "/x/relation/tags", "/x/web-interface/history/cursor"
+  "/x/v3/fav/resource/list", "/x/relation/tag", "/x/relation/followings", "/x/relation/tags", "/x/web-interface/history/cursor"
 ]);
 const ok = (data) => () => json({ code: 0, data });
 const baseRoutes = () => ({
@@ -71,11 +71,13 @@ const baseRoutes = () => ({
     { type: 2, id: 77, bvid: "BV1remote001", title: "远程视频", upper: { name: "UPr", mid: 8 }, duration: 61, fav_time: 1700000100, attr: 0, intro: "" },
     { type: 12, id: 1, title: "音频不列" }
   ] })(),
+  // A group's members: 2 is in 特别关注 and in none of their own groups, so in 默认分组 too (listed twice, counted once).
+  "/x/relation/tag": (u) => ok({ "-10": [{ mid: 2, uname: "乙", sign: "", tag: null, special: 1 }], 0: [{ mid: 2, uname: "乙", sign: "", tag: null, special: 1 }], 5: [{ mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 }] }[u.searchParams.get("tagid")] || [])(),
   "/x/relation/followings": ok({ total: 2, list: [
     { mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 },
     { mid: 2, uname: "乙", sign: "", tag: null, special: 1 }
   ] }),
-  "/x/relation/tags": ok([{ tagid: 0, name: "默认分组" }, { tagid: 5, name: "编程" }, { tagid: -10, name: "特别关注" }]),
+  "/x/relation/tags": ok([{ tagid: 0, name: "默认分组", count: 1 }, { tagid: 5, name: "编程", count: 1 }, { tagid: -10, name: "特别关注", count: 1 }, { tagid: 6, name: "空", count: 0 }]),
   "/x/web-interface/history/cursor": ok({ cursor: { max: 999, view_at: 1700000500, business: "archive" }, list: [
     { title: "看过的", author_name: "UPh", view_at: 1700000600, progress: -1, duration: 300, history: { business: "archive", bvid: "BV1hist00001" } },
     { title: "直播", author_name: "主播", view_at: 1700000550, progress: 0, duration: 0, uri: "https://live.bilibili.com/1", history: { business: "live" } }
@@ -183,16 +185,28 @@ check("follows: the synced list with groups and UP 标签, query on it", async (
   assert.deepStrictEqual((await M.follows({ query: "硬核" })).data.items.map((u) => u.mid), ["1"]);
 });
 
-check("follows without a synced list: one B站 page and the group names; query ignored", async () => {
+check("follows without a synced list: groups short of followings' total fall back to one followings page, not kept", async () => {
+  routes["/x/relation/followings"] = ok({ total: 3, list: [{ mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 }] });
   const r = await M.follows({ query: "x" });
-  assert.deepStrictEqual(plain(r.data), { source: "bilibili", queryIgnored: true, total: 2, page: 1, hasMore: false, items: [
-    { mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] },
-    { mid: "2", name: "乙", sign: "", groups: ["特别关注", "默认分组"] }
+  assert.deepStrictEqual(plain(r.data), { source: "bilibili", queryIgnored: true, total: 3, page: 1, hasMore: false, items: [{ mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] }] });
+  // not kept: the next check (followings whole again) reads the groups again
+});
+
+check("follows without a synced list: B站's list read whole by group, as the 分拣台 reads it; query works", async () => {
+  const r = await M.follows({});
+  // In B站's group order: no follow time comes with a group's members.
+  assert.deepStrictEqual(plain(r.data), { source: "bilibili", total: 2, page: 1, hasMore: false, items: [
+    { mid: "2", name: "乙", sign: "", groups: ["特别关注", "默认分组"] },
+    { mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] }
   ] });
-  assert.deepStrictEqual(fetched.map((f) => new URL(f.url).pathname), ["/x/web-interface/nav", "/x/relation/followings", "/x/relation/tags"]);
-  assert.strictEqual(new URL(fetched[1].url).searchParams.get("ps"), "50");
+  assert.deepStrictEqual(fetched.map((f) => { const u = new URL(f.url); return u.pathname + (u.searchParams.has("tagid") ? `#${u.searchParams.get("tagid")}` : ""); }), ["/x/relation/tags", "/x/relation/tag#0", "/x/relation/tag#5", "/x/relation/tag#-10", "/x/web-interface/nav", "/x/relation/followings"], "an empty group is not read; followings' total checks the list is whole");
+  assert.strictEqual(new URL(fetched[1].url).searchParams.get("ps"), "500");
+  fetched.length = 0;
+  assert.deepStrictEqual(plain((await M.follows({ query: "rust" })).data.items.map((u) => u.mid)), ["1"]);
+  assert.strictEqual(fetched.length, 0, "a second call within minutes pages the list already read");
 });
 
+
 check("history: one page, cursor round trip", async () => {
   const r = await M.history({});
   assert.deepStrictEqual(plain(r.data), { cursor: "999:1700000500:archive", items: [