GreyRaphael/dsh-web-fetch-moli

中文 · GitHub

A DeepSeek Harness (DSH) plugin providing a Moli backend for the web_fetch tool. Built on Moli (an ultra-lightweight Rust headless browser), it renders dynamic micro-frontends, single-page applications (SPAs), and static sites with ~60MB memory footprint (>90% reduction compared to ~1GB for standard Chromium/Playwright), denoises pages via LinkeDOM + Readability + mdream, and returns clean, LLM-optimized GitHub Flavored Markdown.

Key Highlights

• Ultra-Lightweight Footprint — Consumes only ~50–60 MB RAM per local daemon, allowing high concurrency on resource-constrained servers without memory bloat.
• Enterprise Micro-Frontend & Dynamic SPA Compatibility — Bypasses Content Security Policy (Page.setBypassCSP), ignores certificate errors, and bypasses Service Worker caching. Drives native Moli layout tree computation via CDP Micro-Clip layout materialization (Page.captureScreenshot 1x1 micro-viewport) and triggers W3C standard scrollIntoView({ block: 'nearest' }) without any user-script monkey-patching, cleanly rendering complex micro-frontends (Alibaba Alfa, qiankun, single-spa) where other lightweight engines fail.
• Two Flexible Backends:
  • local (default): Automatically manages a local moli serve daemon over CDP, providing full SPA and micro-frontend execution with ~60MB memory footprint.
  • cdp: Connects to an existing remote Moli or Chromium CDP service over Chrome DevTools Protocol.
• High-Performance Two-Stage Denoise Pipeline — LinkeDOM + Mozilla Readability extracts primary content and eliminates noise (ads, nav bars, footers, forms), followed by mdream (Rust native with pure JS fallback) for high-fidelity, LLM-optimized Markdown conversion. Inline base64 images are elided to compact size placeholders.
• Cloudflare Challenge Resilience — Detects cf-mitigated: challenge interstitials and waits in the same page/context for natural clearance without artificial bot behavior.

Performance & Architecture Comparison

Feature / Metric	Chromium / Playwright	Lightpanda (Zig)	Obscura (Rust)	Moli (Rust) + DSH Plugin
Memory Footprint	~800 MB – 1.2 GB	~40 MB	~50 MB	~50 – 60 MB
Micro-Frontend Sandbox	✅ Full	❌ Crashes on iframe sandbox (contentWindow.bind)	❌ Fails on dynamic style-loader	✅ 100% (CDP + CSP bypass)
Infinite Scroll / Sentinels	✅ Pixel layout loop	❌ Geometry incomplete	❌ Geometry incomplete	✅ Native Layout Materialization (Micro-Clip CDP)
Execution Protocol	Heavy CDP	Lightweight CDP	Custom / CDP	Lightweight Native CDP

How it works

web_fetch (tool-web)
   └─ ctx.web.fetchProvider = moli
        ├─ local: resolves moli binary → manages `moli serve` daemon → connectOverCDP
        │           └─ setupPageHooks: Page.setBypassCSP + ignoreCertErrors + bypassServiceWorker
        ├─ cdp:   connectOverCDP(remoteEndpoint)
        ├─ page.goto → Micro-Clip layout materialization + native scroll loop → settle (networkidle) → page.content()
        ├─ denoise: LinkeDOM (DOM) → Readability (article extract) → mdream (Markdown)
        └─ Markdown (or raw HTML when denoise is disabled)

Requirements

• DeepSeek Harness (dsh web), Node.js ≥ 22 (the plugin drives CDP over the native global WebSocket, which Node 20 does not provide).
• Zero manual setup: the plugin features automatic download on first use. If no moli binary is detected on your system, it automatically downloads and unpacks the latest release binary matching your OS and architecture (Linux, macOS, Windows x86_64 / arm64) into ~/.cache/moli.
• Existing installations of Moli v1.1.5+ are automatically detected on $PATH, ~/.local/bin/moli, or via the moliPath setting.

Manual installation (optional):

curl -sSL https://github.com/GreyRaphael/moli/releases/latest/download/moli-x86_64-unknown-linux-gnu.tar.gz | tar -xz -C ~/.local/bin
chmod +x ~/.local/bin/moli

Installation

# Option 1: Install directly via GitHub repository
dsh plugin --profile web add github:GreyRaphael/dsh-web-fetch-moli

# Option 2: Install via npm package name
dsh plugin --profile web add dsh-web-fetch-moli

After adding, restart dsh web to load the plugin.

Configuration

The settings card (设置 → 插件 → 插件配置 → Moli 网页爬取) configures web-fetch-moli in real time:

Field	Default	Description
backend	local	local (managed Moli CDP daemon) or cdp (remote CDP endpoint).
moliPath	(auto)	Path to moli binary. Blank = auto-discover on $PATH and standard locations.
cdpEndpoint	127.0.0.1:9222	Remote CDP endpoint (host:port or URL, used in cdp mode).
bypassCsp	true	Bypasses Content Security Policy via CDP Page.setBypassCSP, unlocking micro-frontend dynamic script loading.
autoScrollSentinel	true	Programmatically triggers IntersectionObserver load-more sentinels for structure-first geometry.
shareBrowserContext	true	Preserves cookies and localStorage across fetches in cdp mode. Unchecked (and always in local mode): fresh isolated context per fetch.
denoise	true	Runs LinkeDOM + Readability + mdream to strip ads, navbars, and footers before LLM-optimized Markdown conversion.
timeoutMs	90000	Total deadline budget (ms) for one fetch; heavy micro-frontend pages may need several layout + scroll rounds. The bundled layer ships 60000, the schema default is 90000.
maxConcurrency	(auto)	Max concurrent rendering slots (auto: local 20, remote CDP 50).
challengeWaitMs	5000	Bounded wait (ms) for Cloudflare verification to clear naturally (0 disables).
challengeRetries	0	Same-page retry attempts after challenge wait.

License

MIT