diff --git a/docs/AGENT.md b/docs/AGENT.md index 4daa3f01..3331d350 100644 --- a/docs/AGENT.md +++ b/docs/AGENT.md @@ -92,7 +92,7 @@ Every other `get` (`/bangumi/`, `/festival/`, `/list/` without `bvid`, YouTube) - `favorites` without `folderId`: `{ folders: [{ id, title, count, private?, included?, synced? }] }`, 稍后再看 first with id `"toview"`, then the user's own folders (Bilibili's folder list, read live). `count` is Bilibili's; `private` is Bilibili's 私密 (the lowest bit of the folder's `attr`); `included` means ticked in the 分拣台's 收藏夹设置; `synced`, on a ticked folder with a local copy (`triage_snapshot_`), is how many videos the copy holds, and is absent while the folder has none yet. With `folderId`: `{ folderId, source, total, page, hasMore, items }`, 40 a page. A folder ticked in the 分拣台 is read from its local copy (`source: "local"`, as of its last read, no Bilibili request) without the videos 取消收藏 since that read (the copy keeps those until its next read; `triage_decisions_` names them); `toview` is read whole from Bilibili in one request; any other folder is one Bilibili page (`source: "bilibili"`). `query`: words that must all appear, case-insensitive, matched on title, author, tags, note and summary for the local copy and `toview`; for other folders Bilibili's own favorites search gets it as `keyword` (titles only, and `total` is then left out). An item: `{ bvid, url, title, author, duration, favTime, invalid?, progress?, tags?, note?, verdict?, summary?, kept? }`: `tags` are the user's 分拣 tags (and 摘读记录 tags), `note` the 备注, `verdict` `keep` / `drop` / `unsure` (`MdgTagCore.verdictOf`), `summary` the 细看's one-liner, `kept` the 保留 mark, `progress` seconds watched in 稍后再看 (-1 finished). - `favorites` with `all: true` asks Bilibili nothing: `{ all: true, source: "local", unsynced?, total, page, hasMore, items }`, the local copies of every ticked folder as one list (`folderId` limits it to one), 300 a page, each video once (`TriageCore.mergeFolderItems`, the 分拣台's 所有收藏夹; 取消收藏 ones left out as above). An item is the one above plus `folders` (the ids of every ticked folder whose copy holds it) and `stage` (`none`, `coarse`: a 粗看 verdict, `fine`: a 细看 one). Filters, all applied together: `verdict` (`none` / `keep` / `drop` / `unsure`), `stage` (`none` / `coarse` / `fine`), `tag` (one tag name the video carries), `invalid` (`true`: only videos Bilibili marks 已失效, `false`: only the others), `query` as for a local copy; a wrong `verdict`, `stage` or `folderId` is `BAD_PARAMS`. `unsynced` lists the ticked folders with no copy yet, which the list therefore lacks: `sync` reads them. -- `follows`: `{ source, total, page, hasMore, items: [{ mid, name, sign, groups, tags? }] }`, 50 a page. With a synced 关注 list (the 分拣台's 关注) it is read from there (`source: "local"`, with the user's UP 标签 as `tags`, and `query` matched on name, sign, groups and tags); else one Bilibili page of the followings plus the group names (`source: "bilibili"`, `query` is not applied and `queryIgnored: true`). `groups` are 关注分组 names, 特别关注 and 默认分组 included. +- `follows`: `{ source, total, page, hasMore, items: [{ mid, name, sign, groups, tags? }] }`, 50 a page. With a synced 关注 list (the 分拣台's 关注) it is read from there (`source: "local"`, with the user's UP 标签 as `tags`, and `query` matched on name, sign, groups and tags); else Bilibili's whole list, read by 关注分组 as the 分拣台's 刷新 reads it (`followReadGroups` in `triage/follow-bg.js`: 500 a request, about 10 requests for 4000 follows, checked against the followings' total and kept 3 minutes so paging reads it once), in Bilibili's group order (`source: "bilibili"`, `query` matched on name, sign and groups); a read short of the total falls back to one page of the followings (`query` not applied, `queryIgnored: true`) and is not kept. `groups` are 关注分组 names, 特别关注 and 默认分组 included. - `history`: `{ items: [{ bvid?, url, title, author, viewAt, progress, duration, kind }], cursor }`, 30 a call, newest first. `cursor` is opaque; pass it back for the next page, `""` at the end. `kind` is Bilibili's business (`archive` for a video, also `pgc`, `live`, `article`…); `progress` is seconds, -1 finished; `viewAt` seconds since the epoch. `collection` and `collectionWrite` (`extension/agent-collections.js`) work on any site whose adapter has `collections` (§5): Bilibili and YouTube. `kind` is one of the adapter's kinds, else `UNSUPPORTED_KIND`; Bilibili has `watch_later` (稍后再看, `add` and `remove`; its favorites, follows and history stay in `favorites`, `follows`, `history`), through `triage/triage-bg.js`'s own 稍后再看 requests in the service worker; YouTube has `watch_later`, `playlists`, `playlist` (with `id`), `likes`, `subscriptions`, `history`, and writes to `watch_later` and `playlist` only (`add`, `remove`). Calls run one at a time, a second apart, and stop with `TIMEOUT` while still queued at the deadline. diff --git a/extension/agent-mydata.js b/extension/agent-mydata.js index a952f550..1ca861c1 100644 --- a/extension/agent-mydata.js +++ b/extension/agent-mydata.js @@ -1,6 +1,6 @@ // MoonDigest's agent interface, docs/AGENT.md §2: the user's own Bilibili data (favorites, 稍后再看, follows, watch // history) for the MCP server, read only. Only GETs, through triage-bg.js's B站 channel (triageBiliGet), one call at a -// time, one page per call (稍后再看 and the 分拣台's local copies are read whole, then paged). +// time, one page per call (稍后再看, the follow list and the 分拣台's local copies are read whole, then paged). // The worker loads it after triage-bg.js and follow-core.js, whose helpers it uses. (() => { const K = MdgLimits.KEYS; @@ -128,20 +128,67 @@ groups: FollowCore.groupNames(FollowCore.groupsOf(String(mid), D), D) }); - // The 关注 the 分拣台 synced (with UP 标签; query searches name, sign, groups and tags), else B站's list a page at a time. + // B站's whole follow list, read by group as the 分拣台's 刷新 reads it (follow-bg.js followReadGroups), and followings' + // total to check it is whole: its requests GAP_MS apart, through this file's channel (B站 risk control is THROTTLED at + // once, as for every other call). A whole list is kept FOLLOWS_KEEP_MS, so paging through it reads B站 once; one short of + // the total is not, the next call reads again. + // ponytail: one request a group with members, plus one per 500 more; past ~45 groups a call outruns agent-bridge.js's budget + const FOLLOWS_KEEP_MS = 3 * 60 * 1000; + let followsRead = null; // { at, promise } + function biliFollows() { + if (!followsRead || Date.now() - followsRead.at > FOLLOWS_KEEP_MS) { + let last = 0; + const spaced = async (url) => { + const wait = last + GAP_MS - Date.now(); + if (wait > 0) await new Promise((r) => setTimeout(r, wait)); + try { + return await triageBiliGet(url); + } finally { + last = Date.now(); + } + }; + const promise = (async () => { + const got = await followReadGroups((call) => call(), undefined, spaced); + const first = await spaced(`${API}/x/relation/followings?vmid=${await triageMid()}&pn=1&ps=${FOLLOW_PAGE}&order=desc`); + return { ...got, total: Number(first?.total) || 0 }; + })(); + const read = { at: Date.now(), promise }; + const drop = () => followsRead === read && (followsRead = null); + read.promise.then((r) => r.fresh.list.length !== r.total && drop(), drop); + followsRead = read; + } + return followsRead.promise; + } + + // The 关注 the 分拣台 synced (with UP 标签), else B站's list read whole; query searches name, sign, groups (and tags). async function follows({ page, query }) { const pn = pageOf(page); const q = typeof query === "string" ? query.trim() : ""; const D = await MdgLimits.followGet(["list", "people", "groups", "tags", "tag_map"]); + let rows; + let source = "local"; if (D.list?.list?.length && !D.list.partial) { const names = new Map((D.tags || []).map((t) => [t.id, t.name])); - const rows = D.list.list.map((mid) => { + rows = D.list.list.map((mid) => { const tags = FollowCore.liveTags(mid, { map: D.tag_map, tags: D.tags || [] }).map((t) => names.get(t)).filter(Boolean); return { ...upRow(mid, D.people?.[mid], D), ...(tags.length && { tags }) }; }); - const hit = q ? rows.filter((r) => matches(q, [r.name, r.sign, ...r.groups, ...(r.tags || [])])) : rows; - return { source: "local", ...slice(hit, pn, FOLLOW_PAGE) }; + } else { + const { fresh, people, tags, total } = await biliFollows(); + // Not the whole list (a group read short, or a follow meanwhile): one page of followings, as before, rather than + // a list missing people. The 分拣台's 刷新 falls back the same way. + if (fresh.list.length !== total) return followingsPage(pn, q); + rows = fresh.list.map((mid) => upRow(mid, people[mid], { list: fresh, groups: groupsOf(tags) })); + source = "bilibili"; } + const hit = q ? rows.filter((r) => matches(q, [r.name, r.sign, ...r.groups, ...(r.tags || [])])) : rows; + return { source, ...slice(hit, pn, FOLLOW_PAGE) }; + } + + const groupsOf = (tags) => (Array.isArray(tags) ? tags : []).filter((g) => g.tagid > 0).map((g) => ({ id: g.tagid, name: String(g.name || "") })); + + // One page of B站's followings with the group names; query not applied. + async function followingsPage(pn, q) { const mid = await triageMid(); const data = await triageBiliGet(`${API}/x/relation/followings?vmid=${mid}&pn=${pn}&ps=${FOLLOW_PAGE}&order=desc`); const tags = await triageBiliGet(`${API}/x/relation/tags`); @@ -151,7 +198,7 @@ special: Object.fromEntries(list.filter((x) => x.special === 1 || (x.tag || []).includes(TriageUi.FOLLOW_SPECIAL)).map((x) => [String(x.mid), 1])), groups: Object.fromEntries(list.map((x) => [String(x.mid), (x.tag || []).filter((t) => t > 0)])) }, - groups: (Array.isArray(tags) ? tags : []).filter((g) => g.tagid > 0).map((g) => ({ id: g.tagid, name: String(g.name || "") })) + groups: groupsOf(tags) }; const total = Number(data?.total) || 0; return { diff --git a/extension/agent-mydata.selftest.js b/extension/agent-mydata.selftest.js index c1e49243..2ec3a092 100644 --- a/extension/agent-mydata.selftest.js +++ b/extension/agent-mydata.selftest.js @@ -11,7 +11,7 @@ const gaps = []; const fastGap = (fn, ms) => (ms > 0 && ms <= 1000 ? (gaps.push(ms), setImmediate(fn)) : setTimeout(fn, ms)); const ctx = vm.createContext({ TextEncoder, URL, URLSearchParams, console, setTimeout: fastGap, clearTimeout, setInterval, clearInterval, AbortController }); const order = [...fs.readFileSync(path.join(__dirname, "background.js"), "utf8").matchAll(/importScripts\(([^)]*)\)/g)].flatMap((m) => m[1].match(/"[^"]+"/g).map((f) => JSON.parse(f))); -assert.ok(order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-core.js"), "agent-mydata.js loads after the triage helpers it uses"); +assert.ok(order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-core.js") && order.indexOf("agent-mydata.js") > order.indexOf("triage/follow-bg.js"), "agent-mydata.js loads after the triage helpers it uses"); let M = null; for (const file of order) { if (file === "agent-mydata.js") ctx.MdgAgentBridge = { addMethods: (m) => (M ??= m) }; @@ -59,7 +59,7 @@ ctx.fetch = async (url, init = {}) => { }; const READS = new Set([ "/x/web-interface/nav", "/x/v3/fav/folder/created/list", "/x/v2/history/toview/web", "/x/v2/history/toview", - "/x/v3/fav/resource/list", "/x/relation/followings", "/x/relation/tags", "/x/web-interface/history/cursor" + "/x/v3/fav/resource/list", "/x/relation/tag", "/x/relation/followings", "/x/relation/tags", "/x/web-interface/history/cursor" ]); const ok = (data) => () => json({ code: 0, data }); const baseRoutes = () => ({ @@ -71,11 +71,13 @@ const baseRoutes = () => ({ { type: 2, id: 77, bvid: "BV1remote001", title: "远程视频", upper: { name: "UPr", mid: 8 }, duration: 61, fav_time: 1700000100, attr: 0, intro: "" }, { type: 12, id: 1, title: "音频不列" } ] })(), + // A group's members: 2 is in 特别关注 and in none of their own groups, so in 默认分组 too (listed twice, counted once). + "/x/relation/tag": (u) => ok({ "-10": [{ mid: 2, uname: "乙", sign: "", tag: null, special: 1 }], 0: [{ mid: 2, uname: "乙", sign: "", tag: null, special: 1 }], 5: [{ mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 }] }[u.searchParams.get("tagid")] || [])(), "/x/relation/followings": ok({ total: 2, list: [ { mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 }, { mid: 2, uname: "乙", sign: "", tag: null, special: 1 } ] }), - "/x/relation/tags": ok([{ tagid: 0, name: "默认分组" }, { tagid: 5, name: "编程" }, { tagid: -10, name: "特别关注" }]), + "/x/relation/tags": ok([{ tagid: 0, name: "默认分组", count: 1 }, { tagid: 5, name: "编程", count: 1 }, { tagid: -10, name: "特别关注", count: 1 }, { tagid: 6, name: "空", count: 0 }]), "/x/web-interface/history/cursor": ok({ cursor: { max: 999, view_at: 1700000500, business: "archive" }, list: [ { title: "看过的", author_name: "UPh", view_at: 1700000600, progress: -1, duration: 300, history: { business: "archive", bvid: "BV1hist00001" } }, { title: "直播", author_name: "主播", view_at: 1700000550, progress: 0, duration: 0, uri: "https://live.bilibili.com/1", history: { business: "live" } } @@ -183,16 +185,28 @@ check("follows: the synced list with groups and UP 标签, query on it", async ( assert.deepStrictEqual((await M.follows({ query: "硬核" })).data.items.map((u) => u.mid), ["1"]); }); -check("follows without a synced list: one B站 page and the group names; query ignored", async () => { +check("follows without a synced list: groups short of followings' total fall back to one followings page, not kept", async () => { + routes["/x/relation/followings"] = ok({ total: 3, list: [{ mid: 1, uname: "甲", sign: "写 Rust", tag: [5], special: 0 }] }); const r = await M.follows({ query: "x" }); - assert.deepStrictEqual(plain(r.data), { source: "bilibili", queryIgnored: true, total: 2, page: 1, hasMore: false, items: [ - { mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] }, - { mid: "2", name: "乙", sign: "", groups: ["特别关注", "默认分组"] } + assert.deepStrictEqual(plain(r.data), { source: "bilibili", queryIgnored: true, total: 3, page: 1, hasMore: false, items: [{ mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] }] }); + // not kept: the next check (followings whole again) reads the groups again +}); + +check("follows without a synced list: B站's list read whole by group, as the 分拣台 reads it; query works", async () => { + const r = await M.follows({}); + // In B站's group order: no follow time comes with a group's members. + assert.deepStrictEqual(plain(r.data), { source: "bilibili", total: 2, page: 1, hasMore: false, items: [ + { mid: "2", name: "乙", sign: "", groups: ["特别关注", "默认分组"] }, + { mid: "1", name: "甲", sign: "写 Rust", groups: ["编程"] } ] }); - assert.deepStrictEqual(fetched.map((f) => new URL(f.url).pathname), ["/x/web-interface/nav", "/x/relation/followings", "/x/relation/tags"]); - assert.strictEqual(new URL(fetched[1].url).searchParams.get("ps"), "50"); + assert.deepStrictEqual(fetched.map((f) => { const u = new URL(f.url); return u.pathname + (u.searchParams.has("tagid") ? `#${u.searchParams.get("tagid")}` : ""); }), ["/x/relation/tags", "/x/relation/tag#0", "/x/relation/tag#5", "/x/relation/tag#-10", "/x/web-interface/nav", "/x/relation/followings"], "an empty group is not read; followings' total checks the list is whole"); + assert.strictEqual(new URL(fetched[1].url).searchParams.get("ps"), "500"); + fetched.length = 0; + assert.deepStrictEqual(plain((await M.follows({ query: "rust" })).data.items.map((u) => u.mid)), ["1"]); + assert.strictEqual(fetched.length, 0, "a second call within minutes pages the list already read"); }); + check("history: one page, cursor round trip", async () => { const r = await M.history({}); assert.deepStrictEqual(plain(r.data), { cursor: "999:1700000500:archive", items: [ diff --git a/extension/triage/follow-bg.js b/extension/triage/follow-bg.js index 35be0117..92f5f691 100644 --- a/extension/triage/follow-bg.js +++ b/extension/triage/follow-bg.js @@ -18,6 +18,7 @@ // // 同步分三步，每步做完记进 follow_jobs.cursor；跑着时开 1 分钟的 follow-keepalive 闹钟，worker 被杀后由它叫醒、从游标接着跑： // 1 list：拉我的全部关注 → follow_list / follow_people；和上次比，消失的人进 follow_unfollowed（source "bili"），只在拉全时判断。 +// 已有完整名单时按分组读（followListByGroups），人数对不上再逐页读 followings。 // 第一次同步每拉一页就先写进 follow_list（partial: true），界面不用等拉完。 // 2 feed：视频动态往回翻过 followSlowDays 天 → follow_last（谁最近发过、最近 3 个标题），翻完才写，翻到一半的每页存进 follow_last_wip，界面边翻边读。6 小时内翻过就跳过。 // 3 arc：动态里没出现的人逐个 arc/search（约 1 秒 1 个）→ follow_content；已有记录且不早于 follow_last.since 的跳过。 @@ -318,6 +319,67 @@ const followSyncStep = (event, fields) => // ===== 同步 ===== +const followFollowingsUrl = (vmid, pn) => `${FOLLOW_API}/x/relation/followings?${followQuery({ vmid, pn, ps: 50, order: "desc" })}`; + +// One followed UP 主 (an item of followings or of a group's members) into fresh and people; seen ones are left as they are. +function followAddPerson(fresh, people, x) { + const mid = String(x.mid); + if (people[mid]) return; + fresh.list.push(mid); + if (x.mtime) fresh.followTime[mid] = Number(x.mtime); + if (x.special === 1 || (x.tag || []).includes(FOLLOW_SPECIAL)) fresh.special[mid] = 1; + const groups = (x.tag || []).filter((t) => t > 0); + if (groups.length) fresh.groups[mid] = groups; + people[mid] = { mid, name: x.uname || "", face: x.face || "", sign: String(x.sign || "").slice(0, 60), ov: x.official_verify?.desc || "" }; +} + +// 按分组读全部关注：x/relation/tag 一页给 500 人，每个分组读到不满一页为止，x/relation/tags 说没人的分组不读。分进自己分组的人 +// 不在默认分组里，所有分组合起来就是全部关注（2026-11-12 实测），按 mid 去重。4000 人约 10 个请求，逐页读 followings 要 80 个。 +// 分组成员不带关注时间（fresh.followTime 为空）。读全没有，由调用方拿 followings 的总数核对（followListByGroups，MCP 的 follows）。 +// 同步的第 1 步和 MCP 的 get_follows（agent-mydata.js）都用它。get 是调用方的请求包装（同步的 ctx.get 能被停止打断）， +// onPage(已读人数) 每读一页调一次，getData(url) 发请求：同步走关注的队列（限流时整队暂停 90 秒重试），MCP 传自己的（限流直接报 THROTTLED）。 +const FOLLOW_GROUP_PS = 500; +async function followReadGroups(get, onPage = async () => {}, getData = followGetData) { + const fresh = { list: [], followTime: {}, special: {}, groups: {} }; + const people = {}; + const tags = await get(() => getData(`${FOLLOW_API}/x/relation/tags`)); + for (const g of Array.isArray(tags) ? tags : []) { + if (!(Number(g.count) > 0)) continue; + // ponytail: 20 页 = 10000 人，超过 B站 5000 人的关注上限 + for (let pn = 1; pn <= 20; pn++) { + const url = `${FOLLOW_API}/x/relation/tag?${followQuery({ tagid: g.tagid, pn, ps: FOLLOW_GROUP_PS })}`; + // 有人的分组第 1 页回空，可能是风控：等一下再问一次（followAgain）。 + const page = (await get(() => followAgain(() => getData(url), (p) => pn === 1 && !p?.length))) || []; + for (const x of page) followAddPerson(fresh, people, x); + await onPage(fresh.list.length); + if (page.length < FOLLOW_GROUP_PS) break; + } + } + return { fresh, people, tags }; +} + +// 已有完整名单时第 1 步按分组读（followReadGroups）。关注时间：新关注的人排在 followings 最前面，从第 1 页读到某页没有新人为止， +// 读到的人都用 followings 的时间（取关后又关注的人排在前面，时间也更新）；没读到的沿用存着的。合起来的人数和 followings 的总数 +// 对不上（读的时候关注有变化，或分组接口少给了），返回 null，由调用方逐页整份重读。 +async function followListByGroups(ctx, vmid, stored) { + const { fresh, people, tags } = await followReadGroups(ctx.get, (done) => ctx.progress({ done })); + const old = new Set(stored.list || []); + let total = 0; + for (let pn = 1; pn <= 100; pn++) { + const j = await ctx.get(() => followAgain(() => followGetJson(followFollowingsUrl(vmid, pn)), (j) => pn === 1 && j.code === 0 && !j.data?.list?.length)); + if (j.code !== 0) throw followCodeError(j, "读取关注列表失败："); + const page = j.data?.list || []; + if (pn === 1) total = Number(j.data?.total) || 0; + for (const x of page) if (x.mtime) fresh.followTime[String(x.mid)] = Number(x.mtime); + if (!page.some((x) => !old.has(String(x.mid)))) break; + } + if (fresh.list.length !== total) return null; + for (const m of fresh.list) if (!fresh.followTime[m] && stored.followTime?.[m]) fresh.followTime[m] = stored.followTime[m]; + fresh.list.sort((a, b) => (fresh.followTime[b] || 0) - (fresh.followTime[a] || 0)); + Object.assign(fresh, { complete: true, total }); + return { fresh, people, tags }; +} + // The sync's run (MdgCommon.latest): followStart starts the next, followStop ends it; followRun is the sync going on. const followLife = MdgCommon.latest(); let followRun = null; // { run, promise } @@ -330,33 +392,24 @@ async function followSyncJob(ctx) { await ctx.progress({ phase: "list", step: "", done: 0, total: 0, cursor: c }); const nav = (await get(() => followGetJson(`${FOLLOW_API}/x/web-interface/nav`))).data || {}; if (!nav.isLogin || !nav.mid) throw triageNotLoggedIn(); - const fresh = { list: [], followTime: {}, special: {}, groups: {} }; - const people = {}; - let total = 0; + const stored = (await MdgLimits.followGet(["list"])).list; + const showPages = !stored || stored.partial; + const byGroups = showPages ? null : await followListByGroups(ctx, nav.mid, stored); + const fresh = byGroups?.fresh || { list: [], followTime: {}, special: {}, groups: {} }; + const people = byGroups?.people || {}; + let total = byGroups?.fresh.total || 0; // A total that moves between pages (a follow or unfollow on B站 meanwhile) shifts the pages: someone may fall // between two of them, so the read is not complete and nobody missing from it counts as unfollowed. let shifted = false; - const stored = (await MdgLimits.followGet(["list"])).list; - const showPages = !stored || stored.partial; // ponytail: 100 页 = 5000 人，B站 关注上限也是这个数 - for (let pn = 1; pn <= 100; pn++) { - const url = `${FOLLOW_API}/x/relation/followings?${followQuery({ vmid: nav.mid, pn, ps: 50, order: "desc" })}`; - const j = await get(() => followAgain(() => followGetJson(url), (j) => pn === 1 && j.code === 0 && !j.data?.list?.length)); + for (let pn = 1; !byGroups && pn <= 100; pn++) { + const j = await get(() => followAgain(() => followGetJson(followFollowingsUrl(nav.mid, pn)), (j) => pn === 1 && j.code === 0 && !j.data?.list?.length)); if (j.code !== 0) throw followCodeError(j, "读取关注列表失败："); const page = j.data?.list || []; const pageTotal = Number(j.data?.total) || 0; if (pn > 1 && pageTotal !== total) shifted = true; total = pageTotal; - for (const x of page) { - const mid = String(x.mid); - if (people[mid]) continue; - fresh.list.push(mid); - if (x.mtime) fresh.followTime[mid] = Number(x.mtime); - if (x.special === 1 || (x.tag || []).includes(FOLLOW_SPECIAL)) fresh.special[mid] = 1; - const groups = (x.tag || []).filter((t) => t > 0); - if (groups.length) fresh.groups[mid] = groups; - people[mid] = { mid, name: x.uname || "", face: x.face || "", sign: String(x.sign || "").slice(0, 60), ov: x.official_verify?.desc || "" }; - } + for (const x of page) followAddPerson(fresh, people, x); if (showPages) { await followUpdate(["list", "people", "unfollowed", "tags"], (s) => { const live = new Set((s.tags || []).map((t) => t.id)); @@ -372,7 +425,7 @@ async function followSyncJob(ctx) { fresh.complete = !shifted && fresh.list.length >= total; fresh.total = total; // B站 关注分组 names. 0 默认分组 and -10 特别关注 are B站's fixed ones; the page names them itself. - const tags = await get(() => followGetData(`${FOLLOW_API}/x/relation/tags`)); + const tags = byGroups ? byGroups.tags : await get(() => followGetData(`${FOLLOW_API}/x/relation/tags`)); const groups = (Array.isArray(tags) ? tags : []).filter((g) => g.tagid > 0).map((g) => ({ id: g.tagid, name: String(g.name || ""), count: Number(g.count) || 0 })); let changes = null; await followUpdate(["list", "people", "unfollowed", "tag_map", "tags"], (s) => { diff --git a/extension/triage/follow-bg.selftest.js b/extension/triage/follow-bg.selftest.js index a0dea153..d936e755 100644 --- a/extension/triage/follow-bg.selftest.js +++ b/extension/triage/follow-bg.selftest.js @@ -48,7 +48,7 @@ t.fetch = async (url, init = {}) => { } const u = new URL(url); calls.push(u.pathname.replace(/^\/x\//, "") + (u.searchParams.get("mid") ? `#${u.searchParams.get("mid")}` : u.searchParams.has("pn") ? `#${u.searchParams.get("pn")}` : u.searchParams.has("offset") ? `#${u.searchParams.get("offset") || "0"}` : "")); - const r = routes[u.pathname]; + const r = routes[u.pathname] || (u.pathname === "/x/relation/tag" && routes["/x/relation/followings"] && groupMembers); if (!r) throw new Error(`no route ${u.pathname}`); return r(u); }; @@ -58,6 +58,23 @@ const person = (mid, extra = {}) => ({ mid: Number(mid), uname: `UP${mid}`, face const followings = (pages, total) => (u) => json({ code: 0, data: { total, list: pages[Number(u.searchParams.get("pn")) - 1] || [] } }); // x/relation/stats: { mid: { mid, follower } } for the mids asked; follower(mid) undefined leaves that mid out. const statsOf = (follower) => (u) => json({ code: 0, data: Object.fromEntries(u.searchParams.get("mids").split(",").filter((m) => follower(m) !== undefined).map((m) => [m, { mid: Number(m), follower: follower(m) }])) }); +// A group's members as B站's x/relation/tag gives them, from the test's followings: 0 默认分组 is everyone in none of their +// own groups, -10 特别关注 the special ones; no mtime, ps per page. +const groupMembers = async (u) => { + const all = []; + for (let pn = 1; pn <= 100; pn++) { + const l = (await (await routes["/x/relation/followings"](new URL(`https://api.bilibili.com/x/relation/followings?pn=${pn}`))).json()).data?.list || []; + if (!l.length) break; + all.push(...l); + } + const id = Number(u.searchParams.get("tagid")); + const ps = Number(u.searchParams.get("ps")); + const pn = Number(u.searchParams.get("pn")); + const tags = (x) => x.tag || []; + const inIt = (x) => (id === 0 ? !tags(x).some((t) => t > 0) : id === -10 ? x.special === 1 || tags(x).includes(-10) : tags(x).includes(id)); + const members = all.filter(inIt).map(({ mtime, ...x }) => x); + return json({ code: 0, data: members.slice((pn - 1) * ps, pn * ps) }); +}; const dyn = (mid, bvid, at) => ({ modules: { module_author: { mid, name: `UP${mid}`, face: "", pub_ts: at }, module_dynamic: { major: { archive: { bvid, aid: 1, title: `T-${bvid}`, cover: "http://c", duration_text: "1:02:03", stat: { play: "1.5万" } } } } } }); @@ -170,13 +187,16 @@ const runSync = async () => { const j = local.follow_jobs; assert.strictEqual(j.state, "done", j.error); assert.ok(j.finishedAt && !j.error, j.error); - assert.deepStrictEqual(plain(local.follow_list.list), ["1", "2", "3", "4", "5", "6"]); + // A stored list is read by group: newest follow first, the follow times of the new ones from followings. + assert.deepStrictEqual(plain(local.follow_list.list), ["6", "5", "4", "3", "2", "1"]); + assert.deepStrictEqual(plain(local.follow_list.followTime), { 1: 101, 2: 102, 3: 103, 4: 104, 5: 105, 6: 106 }); + assert.deepStrictEqual(calls.filter((c) => c.startsWith("relation/tag#") || c.startsWith("relation/followings")), ["relation/tag#1", "relation/tag#1", "relation/tag#1", "relation/followings#1", "relation/followings#2", "relation/followings#3"]); assert.deepStrictEqual(plain(local.follow_list.special), { 2: 1 }); assert.deepStrictEqual(plain(local.follow_list.groups), { 2: [7] }); assert.deepStrictEqual(plain(local.follow_groups), [{ id: 7, name: "数码", count: 1 }], "B站 分组 names stored, 默认分组 and 特别关注 left out"); assert.strictEqual(calls.filter((c) => c === "relation/tags").length, 1, "group names fetched once per sync"); assert.strictEqual(local.follow_unfollowed[9].source, "bili"); - assert.deepStrictEqual([plain(local.follow_jobs.changes.added), plain(local.follow_jobs.changes.removed)], [["2", "3", "4", "5", "6"], ["9"]], "the sync's changes reach follow_jobs"); + assert.deepStrictEqual([plain(local.follow_jobs.changes.added), plain(local.follow_jobs.changes.removed)], [["6", "5", "4", "3", "2"], ["9"]], "the sync's changes reach follow_jobs"); assert.deepStrictEqual(plain(local.follow_unfollowed[9].tagIds), ["t1"]); assert.ok(!local.follow_tag_map[9] && local.follow_tag_map[1], "the vanished account's tags moved into the record"); assert.strictEqual(local.follow_people[2].name, "UP2"); @@ -185,7 +205,7 @@ const runSync = async () => { assert.deepStrictEqual(Object.keys(local.follow_last.map).sort(), ["2", "3"]); assert.strictEqual(local.follow_last.since, old); // arc/search only for followings missing from the feed and without a fresh record: 1, 5, 6 - assert.deepStrictEqual(calls.filter((c) => c.startsWith("space")), ["space/wbi/arc/search#1", "space/wbi/arc/search#5", "space/wbi/arc/search#6"]); + assert.deepStrictEqual(calls.filter((c) => c.startsWith("space")), ["space/wbi/arc/search#6", "space/wbi/arc/search#5", "space/wbi/arc/search#1"]); assert.strictEqual(local.follow_content[1].count, 3); assert.strictEqual(local.follow_content[5].code, -404, "a gone account is recorded"); assert.ok(!local.follow_content[6], "a temporary error is not recorded"); @@ -197,7 +217,7 @@ const runSync = async () => { routes["/x/space/wbi/arc/search"] = () => arcOk(2); await runSync(); assert.ok(!calls.some((c) => c.startsWith("polymer")), "fresh feed skipped"); - assert.deepStrictEqual(calls.filter((c) => c.startsWith("space")), ["space/wbi/arc/search#1", "space/wbi/arc/search#6"]); + assert.deepStrictEqual(calls.filter((c) => c.startsWith("space")), ["space/wbi/arc/search#6", "space/wbi/arc/search#1"]); } // ---------- logged out mid-sync: B站's -101 stops it as NOT_LOGGED_IN in the one sentence, never as 暂时没查到 ---------- @@ -371,10 +391,41 @@ const runSync = async () => { assert.ok(!local.follow_list.partial && local.follow_list.complete, "the full read replaces it"); assert.strictEqual(local.follow_jobs.changes, null, "a first sync shows no changes"); - // a later sync keeps the stored list until the full read - sizes.length = 0; + // a later sync reads by group: one page of members, followings page 1 for the total, nothing written midway + routes["/x/relation/tags"] = () => json({ code: 0, data: [{ tagid: -10, count: 0 }, { tagid: 0, count: 3 }] }); + routes["/x/relation/followings"] = ok; + local.follow_list.followTime[3] = 7; // 3 is not on followings page 1: its stored time stays + const writes = []; + const set = t.chrome.storage.local.set; + t.chrome.storage.local.set = async (o) => ("follow_list" in o && writes.push(o.follow_list.list.length), set(o)); + calls = []; + await runSync(); + t.chrome.storage.local.set = set; + assert.ok(!local.follow_jobs.error, local.follow_jobs.error); + assert.deepStrictEqual(calls.filter((c) => c.startsWith("relation/")), ["relation/tags", "relation/tag#1", "relation/followings#1"], "an empty group is not read; page 1 has nobody new"); + assert.deepStrictEqual(writes, [3], "the list is written once, when read whole"); + assert.deepStrictEqual(plain(local.follow_list.followTime), { 1: 101, 2: 102, 3: 7 }, "times from followings page 1, else the stored one"); + assert.deepStrictEqual(plain(local.follow_list.list), ["2", "1", "3"]); + + // A group's page 1 empty although it has members (risk control): asked again, not taken as nobody. + let empties = 1; + const members = groupMembers; + routes["/x/relation/tag"] = (u) => (empties-- > 0 ? json({ code: 0, data: [] }) : members(u)); + calls = []; + await runSync(); + delete routes["/x/relation/tag"]; + assert.deepStrictEqual(calls.filter((c) => c.startsWith("relation/tag#")), ["relation/tag#1", "relation/tag#1"]); + assert.strictEqual(calls.filter((c) => c.startsWith("relation/followings")).length, 1, "the second answer is whole: no page-by-page read"); + assert.ok(local.follow_list.complete); + + // members and followings disagree on the count (a follow meanwhile): read followings page by page + routes["/x/relation/tag"] = (u) => json({ code: 0, data: u.searchParams.get("tagid") === "0" ? [person(1), person(2)] : [] }); + calls = []; await runSync(); - assert.deepStrictEqual(sizes.map(([l]) => l.length), [3, 3]); + delete routes["/x/relation/tag"]; + assert.ok(!local.follow_jobs.error, local.follow_jobs.error); + assert.deepStrictEqual(calls.filter((c) => c.startsWith("relation/followings")), ["relation/followings#1", "relation/followings#1", "relation/followings#2"]); + assert.deepStrictEqual(plain(local.follow_list.list), ["1", "2", "3"], "nobody lost: the page-by-page read is the list"); } // ---------- partial followings: nobody marked gone ---------- diff --git a/mcp/README.md b/mcp/README.md index 9bafad94..61058863 100644 --- a/mcp/README.md +++ b/mcp/README.md @@ -29,7 +29,7 @@ MCP 客户端 ──stdio──> moondigest-mcp ──ws://127.0.0.1:47823── - 不带参数：收藏夹列表和读取进度。 - `include` / `exclude`：勾选、取消勾选收藏夹。取消勾选的后果：下次打开分拣台时，这个收藏夹的本地副本和取消收藏记录会被删除，不在其他勾选收藏夹里的视频进「已出分拣范围」；AI 判断、保留、备注和标签都留着。取消勾选有撤销记录：`undo` 会重新勾选并放回取消收藏记录，副本要再同步一次。 - `start: true`：在扩展后台读取勾选的收藏夹（或 `folders` 指定的几个），不需要打开分拣台。调用立刻返回，再不带参数调用一次看进度。视频没有增减的收藏夹只花一次请求。B站 限流时读取停下，结果里写明。扩展后台被浏览器重启后，没读完的不会接着读，要再 `start` 一次。 -- `get_follows({ page?, query?, raw? })`：关注的 UP 主，每页 50 个，带关注分组和你的 UP 标签。分拣台的「关注」刷新过就读本地的关注列表，`query` 能用；没刷新过就从 B站 读一页，`query` 不生效，结果里会写明。 +- `get_follows({ page?, query?, raw? })`：关注的 UP 主，每页 50 个，带关注分组和你的 UP 标签。分拣台的「关注」刷新过就读本地的关注列表；没刷新过就按关注分组从 B站 读整份名单（和分拣台刷新同一种读法，3 分钟内翻页不再读 B站）。两种情况 `query` 都能用；整份名单没读全时退回读 B站 一页，这时 `query` 不生效，结果里会写明。 - `get_history({ cursor?, raw? })`：观看历史，从新到旧，每次 30 条，把返回的 `cursor` 传回去读更早的。 `get_favorites`、`get_follows`、`get_history` 同一时间只跑一个，两次之间隔 1 秒。 @@ -169,7 +169,7 @@ Read your own Bilibili data: - No parameters: the folder list and the progress of a read. - `include` / `exclude`: tick and untick folders. What unticking costs: the next time the 分拣台 is open, that folder's local copy and its 取消收藏 records are removed, and its videos that are in no other ticked folder go to 已出分拣范围; AI verdicts, 保留, notes and tags stay. An untick is recorded: `undo` ticks the folder again and puts its 取消收藏 records back; its copy needs another sync. - `start: true`: reads the ticked folders (or those in `folders`) in the extension's background, without the 分拣台 open. The call answers at once; call again with no parameters for the progress. A folder whose videos did not change costs one request. Rate limiting stops the read, and the result says so. If the browser restarts the extension's background, the rest is not read: `start` again. -- `get_follows({ page?, query?, raw? })`: the UP 主 you follow, 50 a page, with follow groups and your UP tags. Read from the 分拣台's follow list when it has one (then `query` works); otherwise one page from Bilibili, where `query` is not applied and the result says so. +- `get_follows({ page?, query?, raw? })`: the UP 主 you follow, 50 a page, with follow groups and your UP tags. Read from the 分拣台's follow list when it has one; otherwise Bilibili's whole list, read by follow group as the 分拣台's refresh reads it (paging within 3 minutes does not read Bilibili again). `query` works either way, except when the whole list comes back short: then it is one page from Bilibili, `query` is not applied and the result says so. - `get_history({ cursor?, raw? })`: watch history, newest first, 30 a call. Pass the returned `cursor` back for older items. `get_favorites`, `get_follows` and `get_history` run one at a time, a second apart. diff --git a/mcp/server.js b/mcp/server.js index 465b0e70..1ca6be45 100755 --- a/mcp/server.js +++ b/mcp/server.js @@ -434,7 +434,7 @@ const TOOLS = [ }, { name: "get_follows", - description: "Read-only: the UP 主 the user follows on Bilibili, with their follow groups and the user's MoonDigest UP tags, 50 a page. query searches name, sign, groups and tags once MoonDigest has synced the follow list (otherwise it is ignored and the result says so). Page sparingly.", + description: "Read-only: the UP 主 the user follows on Bilibili, with their follow groups and the user's MoonDigest UP tags, 50 a page. query searches name, sign, groups and, once MoonDigest has synced the follow list, tags. Page sparingly.", inputSchema: { type: "object", properties: {