Vigil (Medusa) — Public-Discourse Intelligence

What it is

Vigil reads public discourse at scale and correlates it with structured reality. It ingests social media, news, and broadcast, runs an LLM intelligence layer over it (sentiment, entities, coordination), and overlays the result on electoral and socioeconomic data — to surface patterns and forecast outcomes for Ghana's electoral ecosystem. Governance and access control are designed in, not added later.

Ingestion — 28+ sources, continuous

• Social — X (GraphQL + API), Facebook & Threads, YouTube, TikTok, Telegram, Bluesky — with cookie/proxy rotation, rate limiting, and checkpoint resumability.
• News — 11 Ghanaian outlets on tiered hourly–4-hourly cadence, with an archive back to 2010, plus GDELT.
• Broadcast — live radio/TV stream capture (ffmpeg) + Whisper transcription across 12 stations, with heuristic commercial-break detection.
• A custom database-driven scheduler (Celery beat reading live-editable cron rows) means schedule changes propagate in seconds with no Airflow and no restart.

Intelligence layer

• Pluggable LLM backend — auto-selects vLLM (GPU) → Anthropic → Ollama (CPU), one abstraction, no lock-in.
• Sentiment, emotion (Plutchik), entity & topic extraction, political-leaning, and entity-directed sentiment attribution (who benefits, who amplified).
• Coordination detection — finds different authors posting near-identical content across platforms within a time window (normalized-text similarity).
• Entity resolution — LLM clustering + alias registry, behind an admin curation gate.

Correlation & forecast

• 100+ sub-national socioeconomic indicators (World Bank, FRED, IMF, census, health, food prices, nighttime lights) on an entity-attribute-value model, always at the finest available admin level.
• Geo-resolution over 15K localities → region/district/constituency.
• Election forecast — results 1992–2024, 40K+ geocoded polling stations, and EC "pink-sheet" tally PDFs parsed via Claude Vision OCR.

Architecture & governance

• Tight by design — single-responsibility Celery services over Redis, a shared medusa_core library, frozen-Pydantic data boundaries, PostgreSQL + Solr faceted search, a FastAPI layer (120+ routes), and a Next.js 16 dashboard.
• Self-hostable — Docker Compose, custom images, Nginx TLS, Flower monitoring; no external job queue or graph DB.
• Governance-first — role-based module permissions, tiered access, admin curation gates, ADR-governed conventions, and audit/lineage timestamps on every record. Documented for every audience (users, developers, deployment).

~1,100 source files · 359 commits.