SYNCopated — Real-Time Collaborative Music Instruction

What it is

SYNCopated recreates the in-person piano lesson over the internet. A teacher and student work from the same score in real time — cursors, MIDI, metronome, and pen annotations stay in lock-step across devices while they see and hear each other. It's built around the actual teaching workflow: prepare materials, teach live, record practice takes, leave timestamped feedback, and track progress.

The hard parts

• Sub-100ms session sync. A stateless WebSocket relay broadcasts MIDI note-on/off, score-cursor position, metronome state, and exercise events between devices. Streaming A/V is the easy part (LiveKit handles it); keeping session state in lock-step at low latency is the moat.
• On-device audio DSP. AVAudioEngine with a sampled grand piano at ≤10ms latency; a custom, offline-tested YIN pitch-detection engine; OSMD MusicXML rendering with a measure-accurate cursor; a fingering engine.
• Live annotations. Teacher draws on the score with PencilKit; strokes appear on the student's device in real time and persist per score.

A full product, not a demo

• Rooms (1-on-1, group, sandbox), a score library (PDF + MusicXML, 2,800+ pieces), folders, and per-room material assignment.
• Assignment + feedback loop — student records a passage; teacher reviews in a scrubbable MIDI player and leaves notes anchored to a playback timestamp.
• AI lesson summaries — on-device (Foundation Models, iOS 26+) with a Claude Haiku fallback, plus bring-your-own-key stored in the iOS Keychain.
• Metronome (teacher-clocked), slow-down playback, transpose, multi-camera (overhead / hands-cam), practice timer, music-theory exercises, events/recitals, demo-student accounts, and a tiered entitlements model (free / premium / family).

Architecture & maturity

• iOS — Swift 6 / SwiftUI, 153 files across 18 modules, Sign in with Apple everywhere. Node/Express server (41 routes) on Azure Container Apps, PostgreSQL, Azure Blob, LiveKit, APNs, Application Insights.
• Operational discipline — Build 18 is a full release dedicated to observability and admin tooling (request-ID tracing, audit trails, synthetic probes, paging). Azure DevOps CI/CD to TestFlight; XCTest + Playwright.
• Cross-platform — syncopated-react (Expo + React-Native-Web) is an in-flight universal client for Android, web (PWA), and desktop.

14 builds in ~8 weeks, each with a runbook, QA matrix, and release notes — and a deliberate choice to cut a fragile feature rather than ship it. No half measures.