Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ package's README for the exact SDK calls to wire in per the official docs.

*(Required disclosure per hackathon rules — keep this list current.)*

- `@qvac/sdk` — Tether QVAC SDK, on-device inference (scam detection, TranslatePsy, VisionPsy).
- `@qvac/sdk` `0.19.0`: Tether QVAC SDK, on-device inference (scam detection, TranslatePsy, VisionPsy). Classification uses Llama 3.2 1B Instruct Q4_0 locally.
- WDK — self-custodial wallet toolkit, local signing.
- Hyperswarm (Pears Stack) — P2P discovery/sync for the risk-address list, stretch goal.
- No cloud inference APIs, no analytics SDKs, no third-party trackers.
Expand Down
38 changes: 27 additions & 11 deletions docs/performance-log.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,44 @@
# Performance log (Track 02 required deliverable)

Fill this in once `adapters-qvac` is wired to the real SDK. Required by the
QVAC Psy track: "a structured performance log including model load, prompts,
token count, TTFT, and throughput."
Required by the QVAC Psy track: a structured performance log including model
load, prompts, token count, TTFT, and throughput.

CI does not download or run the QVAC model. Fill the run table on the
submission device after `createQvacCompletionClient()` has loaded
`LLAMA_3_2_1B_INST_Q4_0`.

## Test hardware

- Device: _TODO_
- Chip / RAM: _TODO_
- OS: _TODO_
- Device: _fill on submission device_
- Chip / RAM: _fill on submission device_
- OS: _fill on submission device_

## Models used

| Model | Quantization | Purpose |
|---|---|---|
| _TODO_ | _TODO_ | scam-pattern classification |
| TranslatePsy (_variant TODO_) | _TODO_ | chat-context translation |
| VisionPsy (_variant TODO_, if stretch goal lands) | _TODO_ | screenshot OCR |
| Llama 3.2 1B Instruct (`LLAMA_3_2_1B_INST_Q4_0`) | Q4_0 | scam-pattern classification (second pass after heuristics) |
| TranslatePsy (_variant pending issue #3_) | _pending_ | chat-context translation |
| VisionPsy (_variant pending stretch_) | _pending_ | screenshot OCR |

## Sample classification prompt

The adapter sends this shape (categories come from the seed dataset):

```
Classify this crypto-chat for known scam playbooks.
Allowed categories: pig-butchering, fake-support, ...
Reply with JSON only, no markdown: {"matches":[{"category":"<allowed category>","confidence":0.0,"evidence":"<short quote>"}]}
If nothing matches: {"matches":[]}
Chat:
<pasted chat text>
```

## Runs

Add one row per representative prompt/run. Keep raw numbers, not rounded
summaries — judges asked for reproducible evidence.
summaries. Judges asked for reproducible evidence.

| Timestamp | Model | Prompt (truncated) | Input tokens | Output tokens | Model load time (ms) | TTFT (ms) | Throughput (tok/s) |
|---|---|---|---|---|---|---|---|
| _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ | _TODO_ |
| _pending on-device run_ | Llama 3.2 1B Instruct Q4_0 | Classify this crypto-chat... / guaranteed 30% weekly returns | _pending_ | _pending_ | _pending_ | _pending_ | _pending_ |
44 changes: 29 additions & 15 deletions packages/adapters-qvac/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,23 +5,37 @@ The only package allowed to import `@qvac/sdk`. Implements three ports from

| Port | File | Status |
|---|---|---|
| `ScamDetectionPort` | `QvacScamDetectionAdapter.ts` | Heuristic pre-filter works today; QVAC model call is a `TODO(sdk-integration)`. |
| `TranslationPort` | `QvacTranslateAdapter.ts` | Stub — **required for submission**, this is the Track 02 load-bearing piece. |
| `OcrPort` | `QvacVisionAdapter.ts` | Stub — stretch goal, pick up last. |
| `ScamDetectionPort` | `QvacScamDetectionAdapter.ts` | Heuristic pre-filter plus optional on-device QVAC completion pass. |
| `TranslationPort` | `QvacTranslateAdapter.ts` | Stub. Required for submission; this is the Track 02 load-bearing piece. |
| `OcrPort` | `QvacVisionAdapter.ts` | Stub. Stretch goal, pick up last. |

## Wiring the real SDK
Pinned SDK: `@qvac/sdk` `0.19.0`. Classification model: Llama 3.2 1B Instruct Q4_0 (`LLAMA_3_2_1B_INST_Q4_0`).

1. Read the model/hardware docs at [qvac.tether.io](https://qvac.tether.io) and
pick target models for scam-text classification and TranslatePsy.
2. Add `@qvac/sdk` as a dependency here with the exact version pinned.
3. Replace each `TODO(sdk-integration)` with the real model-load + inference
call. Keep the constructor signatures unchanged so `packages/core` and
`apps/web` don't need to change.
4. Record model load time, prompts used, token counts, TTFT, and throughput in
`docs/performance-log.md` — required Track 02 deliverable.
## How the two passes work

1. Keyword heuristics over `scam-patterns.json` (confidence capped at 0.6).
2. If a `QvacCompletionClient` is provided, and the text is a heuristic hit or a longer paste, `completion()` classifies paraphrases and may raise confidence. It never lowers a heuristic hit. Model failures fall back to heuristics.

`apps/web` keeps constructing `new QvacScamDetectionAdapter()` (heuristics only). Vite cannot load `@qvac/sdk` (Node / Bare / Expo). Wire the runtime from a supported host:

```ts
import { QvacScamDetectionAdapter } from "@custos/adapters-qvac";
import { createQvacCompletionClient } from "@custos/adapters-qvac/runtime";

const { client, dispose } = await createQvacCompletionClient();
const scamDetection = new QvacScamDetectionAdapter(undefined, client);
```

Unit tests inject a fake client. CI does not download a model.

## Performance log

Record load time, prompt, token counts, TTFT, and throughput in
`docs/performance-log.md` on the submission device. The first model row is
already listed; fill the run numbers when you execute `createQvacCompletionClient`
on QVAC-capable hardware.

## Local dev without a device

Until the SDK is wired, `packages/core` tests use hand-written fakes of these
ports (see `packages/core/src/**/__tests__`) — you don't need a QVAC-capable
device to work on `core` or `apps/web` wiring.
`packages/core` tests use hand-written fakes of these ports. You do not need a
QVAC-capable device to work on `core` or `apps/web` wiring.
13 changes: 12 additions & 1 deletion packages/adapters-qvac/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,25 @@
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
},
"./runtime": {
"types": "./dist/createQvacCompletionClient.d.ts",
"import": "./dist/createQvacCompletionClient.js"
}
},
"scripts": {
"build": "tsc -p tsconfig.json",
"typecheck": "tsc -p tsconfig.json --noEmit",
"test": "vitest run"
},
"dependencies": {
"@custos/core": "workspace:*",
"@custos/shared": "workspace:*"
"@custos/shared": "workspace:*",
"@qvac/sdk": "0.19.0"
},
"devDependencies": {
"typescript": "^5.6.3",
Expand Down
158 changes: 139 additions & 19 deletions packages/adapters-qvac/src/QvacScamDetectionAdapter.ts
Original file line number Diff line number Diff line change
@@ -1,42 +1,162 @@
import type { ScamDetectionPort } from "@custos/core";
import type { ScamMatch, ScamPattern } from "@custos/core";
import type { ScamCategory, ScamDetectionPort, ScamMatch, ScamPattern } from "@custos/core";
import { seedPatterns } from "@custos/shared";

/** Heuristic hits are a cheap signal; the model may raise this, never lower it. */
const HEURISTIC_CONFIDENCE = 0.6;

/** Below this length, text with no heuristic hit is treated as obviously clean. */
const MIN_MODEL_CHARS = 40;

export interface QvacCompletionClient {
complete(prompt: string): Promise<string>;
}

interface ModelMatch {
readonly category: ScamCategory;
readonly confidence: number;
readonly evidence: string;
}

/**
* TODO(sdk-integration): this adapter currently only runs the cheap
* keyword-heuristic pre-filter over the seed dataset. Wire the actual
* @qvac/sdk local-model call here to (a) raise recall beyond exact keyword
* matches and (b) produce the nuanced confidence score the heuristic can't.
* See qvac.tether.io for the model-loading and inference API — this is the
* only file in the codebase allowed to import @qvac/sdk directly.
* ScamDetectionPort over a cheap keyword pre-filter plus an optional on-device
* QVAC completion pass. The port does not change. `@qvac/sdk` is imported only
* from `createQvacCompletionClient`, so unit tests never load the runtime.
*
* Keep the heuristic pass even after the model is wired in: it's a cheap
* pre-filter that avoids running the LLM on obviously-clean text, which
* matters for on-device latency/battery.
* Heuristics still run first. The model is skipped on empty/short clean text,
* used to raise confidence on heuristic hits, and used to catch paraphrases
* the keyword list misses. A model failure falls back to heuristics.
*/
export class QvacScamDetectionAdapter implements ScamDetectionPort {
constructor(private readonly patterns: readonly ScamPattern[] = seedPatterns as unknown as ScamPattern[]) {}
constructor(
private readonly patterns: readonly ScamPattern[] = seedPatterns as unknown as ScamPattern[],
private readonly completion?: QvacCompletionClient,
) {}

async analyzeText(text: string): Promise<readonly ScamMatch[]> {
const heuristicMatches = this.heuristicMatches(text);
if (!this.shouldRunModel(text, heuristicMatches.length > 0)) {
return heuristicMatches;
}

try {
const raw = await this.completion!.complete(buildClassifyPrompt(text, this.patterns));
const modelMatches = parseModelMatches(raw, this.patterns);
return mergeMatches(heuristicMatches, modelMatches, this.patterns);
} catch {
return heuristicMatches;
}
}

private heuristicMatches(text: string): ScamMatch[] {
const lower = text.toLowerCase();
const matches: ScamMatch[] = [];

for (const pattern of this.patterns) {
const hit = pattern.heuristics.find((h) => lower.includes(h.toLowerCase()));
if (hit) {
matches.push({
pattern,
// Heuristic-only confidence, deliberately capped below what a real
// model pass should produce — replace once the QVAC model call lands.
confidence: 0.6,
confidence: HEURISTIC_CONFIDENCE,
evidenceSnippet: hit,
});
}
}
return matches;
}

private shouldRunModel(text: string, hasHeuristicHit: boolean): boolean {
if (!this.completion) return false;
const trimmed = text.trim();
if (trimmed.length === 0) return false;
if (hasHeuristicHit) return true;
return trimmed.length >= MIN_MODEL_CHARS;
}
}

// TODO(sdk-integration): await this.qvacModel.classify(text) and merge/
// replace the heuristic matches with model-scored ones.
export function buildClassifyPrompt(text: string, patterns: readonly ScamPattern[]): string {
const categories = [...new Set(patterns.map((p) => p.category))].join(", ");
return [
"Classify this crypto-chat for known scam playbooks.",
`Allowed categories: ${categories}`,
'Reply with JSON only, no markdown: {"matches":[{"category":"<allowed category>","confidence":0.0,"evidence":"<short quote>"}]}',
'If nothing matches: {"matches":[]}',
"Chat:",
text,
].join("\n");
}

return matches;
function parseModelMatches(raw: string, patterns: readonly ScamPattern[]): ModelMatch[] {
const json = extractJsonObject(raw);
if (!json) return [];
let parsed: unknown;
try {
parsed = JSON.parse(json);
} catch {
return [];
}
if (typeof parsed !== "object" || parsed === null || !("matches" in parsed)) return [];
const matches = (parsed as { matches: unknown }).matches;
if (!Array.isArray(matches)) return [];

const allowed = new Set(patterns.map((p) => p.category));
const result: ModelMatch[] = [];
for (const item of matches) {
if (typeof item !== "object" || item === null) continue;
const category = (item as { category?: unknown }).category;
const confidence = (item as { confidence?: unknown }).confidence;
const evidence = (item as { evidence?: unknown }).evidence;
if (typeof category !== "string" || !allowed.has(category as ScamCategory)) continue;
if (typeof confidence !== "number" || !Number.isFinite(confidence) || confidence <= 0) continue;
result.push({
category: category as ScamCategory,
confidence: Math.min(1, confidence),
evidence: typeof evidence === "string" ? evidence : "",
});
}
return result;
}

function extractJsonObject(raw: string): string | undefined {
const start = raw.indexOf("{");
const end = raw.lastIndexOf("}");
if (start < 0 || end <= start) return undefined;
return raw.slice(start, end + 1);
}

function mergeMatches(
heuristic: readonly ScamMatch[],
model: readonly ModelMatch[],
patterns: readonly ScamPattern[],
): ScamMatch[] {
const byCategory = new Map<ScamCategory, ScamMatch>();
for (const match of heuristic) {
const prev = byCategory.get(match.pattern.category);
if (!prev || match.confidence > prev.confidence) {
byCategory.set(match.pattern.category, match);
}
}

for (const incoming of model) {
const pattern =
byCategory.get(incoming.category)?.pattern ??
patterns.find((p) => p.category === incoming.category);
if (!pattern) continue;
const prev = byCategory.get(incoming.category);
if (!prev) {
byCategory.set(incoming.category, {
pattern,
confidence: incoming.confidence,
evidenceSnippet: incoming.evidence || undefined,
});
continue;
}
if (incoming.confidence > prev.confidence) {
byCategory.set(incoming.category, {
pattern: prev.pattern,
confidence: incoming.confidence,
evidenceSnippet: incoming.evidence || prev.evidenceSnippet,
});
}
}

return [...byCategory.values()];
}
Loading
Loading