English | 简体中文 | 日本語

NewsDIY

A news aggregation platform with AI-powered source discovery, forked from NewsNow

This project is a fork of ourongxing/newsnow. Built on top of its elegant news reading experience, NewsDIY adds AI-powered source discovery, custom source management, email login, and AI content analysis.

What's New

AI-Powered Source Discovery

Add news sources through a conversational AI interface that automatically navigates websites:

1. Describe what you need: "Find the admissions notices from XX University"
2. AI analyzes intent, generates optimized search queries
3. Playwright-controlled browser searches Bing and navigates automatically
4. page-agent indexes DOM elements for reliable clicking
5. AI evaluates each page, extracts article lists when the target is found

Custom Source Management

• AI chat mode (auto-search official websites) and manual mode (paste URL directly)
• Per-user isolation with cross-device sync after login
• Multiple scraping strategies: Jina Reader API / Playwright / native HTTP

AI Content Analysis

• Click any news item to see AI-generated summaries
• Powered by SiliconFlow API (default: MiniMax-M2.5)
• Results cached to minimize token usage

Email Login

• SMTP-based verification code login alongside GitHub OAuth

Tech Stack

Layer	Technology
Frontend	React 19 + TanStack Router + React Query + Jotai + UnoCSS
Backend	Nitro Server (vite-plugin-with-nitro)
Database	SQLite (better-sqlite3) / Cloudflare D1
AI	SiliconFlow API + page-agent + Playwright
Search	Tavily API / SearXNG
Build	Vite + SWC + pnpm

Quick Start

Requirements

• Node.js >= 20
• pnpm

Setup

corepack enable
pnpm i
cp .env.server.sample .env.server
# Edit .env.server with your API keys
pnpm dev

Configuration

See .env.server.sample for all available options. Minimum: set JWT_SECRET and INIT_TABLE=true.

Feature	Required Env Vars
GitHub Login	G_CLIENT_ID, G_CLIENT_SECRET
AI Analysis	SILICONFLOW_API_KEY
AI Source Discovery	SILICONFLOW_API_KEY + TAVILY_API_KEY
Web Scraping	JINA_API_KEY or install Playwright
Email Login	SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASS, SMTP_FROM

Docker

docker compose up

How AI Navigation Works

User describes what they need
    ↓
Intent Analysis (LLM) → search query + section keywords
    ↓
Playwright launches browser → Bing search
    ↓
┌─ Navigation Loop (max 8 steps) ──────────────┐
│  page-agent injection → DOM tree to text → LLM │
│  LLM returns action (click/scroll/back/done)   │
│  Execute → wait for load → next iteration       │
└─────────────────────────────────────────────────┘
    ↓
Target page found → extract article list → return to frontend

Key technical details:

• page-agent: Alibaba's open-source DOM indexing library. Converts interactive elements to [1] Link text [2] Button format for LLM consumption
• Anti-detection: playwright-extra + StealthPlugin + headed browser + system Chrome
• Loop detection: Tracks DOM snapshot changes; forces strategy switch after repeated ineffective scrolling
• Search suggestions: Recognizes "related searches" and "people also search for" sections for better navigation

Credits

• ourongxing/newsnow - Original project
• anthropics/page-agent - DOM indexing library
• SiliconFlow - AI model API
• Tavily - Search API

License

MIT © ourongxing

Extended by YIRC99