07 · Regulatory data
RegAuth
Quantification is the point.
A scrape → parse → NLP → index → serve pipeline that turns the U.S. federal legal corpus into structured, searchable data.
- 01Seven corpora, one pipeline — CFR, U.S. Code, Public Laws, Executive Orders, Statutes, the Federal Register and Acts.
- 02Distributed processing — PySpark exports NLP features to partitioned Parquet with tuned compression, pushdown and partition sizes.
- 03Scheduled workers — Celery workers on two queues, one for browser-driven scrapes, with a nightly re-index.
- 04Search and serving — a Postgres → Solr indexer behind an Express BFF and a React SPA.
- 05Observability — opt-in Sentry error and performance reporting.
- 2 Celery queues
- Spark to Parquet
- Solr search
SurfacesWebSelf-hostedAPI
- PySpark
- Parquet
- Celery · Redis
- PostgreSQL
- Apache Solr
- BART
- React · Express