RegAuth — Distributed Regulatory-Data Platform

What it is

RegAuth is a scrape → clean → NLP → index → serve pipeline over the U.S. federal legal corpus — CFR, USC, Public Laws, Executive Orders, Statutes, and the Federal Register. It is one of the few production systems that renders that entire corpus as structured, queryable data (obligation extraction, regulatory quantification). It productionizes and extends the RegData/QuantGov approach I helped pioneer at the Mercatus Center.

Architecture

• Distributed processing — PySpark / Spark SQL exports NLP features to partitioned Parquet with tuned writer configs (snappy compression, filter pushdown, partition sizing).
• Workers & state — Celery distributed workers with Redis-backed state; an S3-compatible storage abstraction.
• NLP / ML data products — obligation extraction, summarization, and content/language-feature pipelines that produce the structured datasets downstream algorithms and researchers consume.
• Search & serving — Postgres → Apache Solr indexing bridge; a React SPA + Express BFF; an operator CLI.
• Engineering rigor — a Python monorepo with a SemVer'd stable-core API (ADR-governed, 2-minor deprecation window), Azure DevOps CI/CD, four-audience documentation, and on-call runbooks.

Vision & team

I owned the vision and architecture, stayed hands-on, and hired and mentored the team of junior engineers who built it with me — original work that predates AI code generation.