Upload a song, sing along to it, and see how closely you matched the original — broken down by pitch, tone, and lyrics, with a timeline showing which parts you nailed vs. struggled with.
frontend/— Next.js + TypeScript + Tailwind + recharts./— upload a song./song/[songId]— processing status, then a 3-2-1 countdown, sing-along recording./song/[songId]/attempt/[attemptId]— scoring status, then results with a timeline chart.
backend/— FastAPI + SQLAlchemy (SQLite). Background jobs (no Celery/Redis) do the heavy lifting: source separation, pitch extraction, lyric transcription, and attempt scoring.
Two terminals, no Docker required. Requires Python 3.11+ and Node 18+.
cd backend
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --reload --port 8000Backend runs at http://localhost:8000 (interactive docs at /docs). SQLite DB is created at
backend/storage/app.db on first run. Uploaded files and generated artifacts are saved under
backend/storage/songs/{song_id}/. (Everything lives under storage/ so a single volume mount
covers all persistent state — see Deployment below.)
Notes on first run:
demucs(source separation) andfaster-whisper(transcription) download their models the first time they're used — expect the first upload to take noticeably longer than subsequent ones.- Everything runs on CPU. A ~30s song takes roughly 30-90s to fully process on a modern laptop; scoring an attempt takes somewhat less.
- No system
ffmpegis required — audio decoding (including the browser's webm/opus recordings) goes throughav(PyAV), which bundles its own decoders.
cd frontend
npm install
npm run devFrontend runs at http://localhost:3000. It calls the backend at http://localhost:8000 by
default (override with NEXT_PUBLIC_API_BASE_URL if the backend runs elsewhere).
- Start both servers above.
- Open
http://localhost:3000, drag and drop (or click to browse) a song — MP3 or MP4, ≤50MB. - Wait for processing (separating → analyzing pitch → transcribing → ready).
- Click Start, allow microphone access, sing along through the 3-2-1 countdown.
- Recording stops automatically when the song ends and is submitted for scoring.
- Watch the results page for your overall score, pitch/tone/lyrics breakdown, and timeline chart.
When an attempt is submitted, the backend decodes the recording (via PyAV, so browser webm/opus
works directly) and extracts the same pitch contour (librosa pyin) and MFCC timbre features used
for the reference vocal, then DTW-aligns (dtw-python) the user's contour to the reference to
absorb timing drift before comparing them. Pitch accuracy is the average cents deviation on
frames voiced in both recordings, mapped to 0-100. Tone accuracy is average cosine similarity
between DTW-aligned MFCC frames. Lyrics accuracy is a Word Error Rate score from word-level
Levenshtein distance (python-Levenshtein) between the reference transcript and a fresh
faster-whisper transcription of the attempt. The overall score is a weighted average
(pitch 40% / tone 25% / lyrics 35%, see backend/app/scoring/config.py), computed both for the
whole recording and independently for each fixed 5-second segment to produce the timeline chart.
Recordings that are too short or near-silent are rejected with a clear error rather than scored.
POST /songs/upload— multipart form, fieldfile. Returns{ song_id, status }. Only.mp3/.mp4are accepted; rejects other extensions, empty files, and files over 50MB with a 400 + clear message. Accepted files are normalized to a plain WAV (original.wav) before processing, since Demucs/librosa can't read compressed containers directly.GET /songs/{song_id}/status—{ song_id, status, error_message, created_at }.status:pending→separating→analyzing_pitch→transcribing→ready|failed.GET /songs/{song_id}/assets— instrumental URL, duration, and which reference artifacts exist.GET /songs/{song_id}/instrumental— streams the separated instrumental audio.POST /songs/{song_id}/attempts— multipart form, fieldfile(the user's recording). Same validation as song upload. Returns{ attempt_id, status }.GET /songs/{song_id}/attempts/{attempt_id}— scoring status, and oncestatus: "scored", the full result:{ overall_score, pitch_score, tone_score, lyrics_score, segments: [...] }.status:scoring→scored|failed.
The frontend and backend are packaged as one Docker image (Dockerfile at the repo root):
Next.js (output: "standalone") and FastAPI run as separate processes in the same container,
with Caddy reverse-proxying /songs/* to the backend and everything else to the frontend. This
means one host, one URL, no CORS, and no separate NEXT_PUBLIC_API_BASE_URL wiring between two
services — but it's still a real, always-on server under the hood, so it needs to run somewhere
with genuine persistent compute, not Vercel or another serverless platform (see below).
Vercel runs Python as serverless functions: they freeze immediately after the HTTP response is
sent, so the BackgroundTasks jobs this app depends on (Demucs separation, transcription, scoring)
would get killed mid-run. The filesystem is also ephemeral, so SQLite and uploaded files wouldn't
persist. And torch + demucs + faster-whisper alone are hundreds of MB, past Vercel's function
size limit. None of this is a config problem — the backend needs a persistent server, full stop.
Budget 2GB+ RAM. Loading the Demucs model plus torch's runtime commonly uses 1GB+ during separation; smaller instances (Render/Fly/Railway's smallest free or hobby tiers, ~256-512MB) will likely OOM-crash on the first real song upload.
Oracle Cloud's Always Free tier is one of the few genuinely permanent free tiers with enough RAM for this app (up to 24GB on the ARM Ampere A1 shape).
-
Create the VM: oracle.com/cloud/free → sign up (a card is required for identity verification, but Always Free resources aren't charged) → Compute → Instances → Create Instance. Pick an Ampere A1 (ARM) shape, Always Free eligible (e.g. 2 OCPU / 12GB RAM), Ubuntu image, and add your SSH key (
ssh-keygenif you don't have one). Make sure it's assigned a public IP. -
Open port 80 — Oracle Cloud blocks it in two places by default, both need fixing:
- Cloud-level: the VM's subnet → Security Lists (or Network Security Groups) → add an
ingress rule: source
0.0.0.0/0, TCP, destination port80. - VM-level firewall: SSH in and run
sudo iptables -I INPUT -p tcp --dport 80 -j ACCEPT && sudo netfilter-persistent save(Ubuntu images ship with their own iptables rules that block it even after the cloud-level rule is open).
- Cloud-level: the VM's subnet → Security Lists (or Network Security Groups) → add an
ingress rule: source
-
Install Docker on the VM:
sudo apt update && sudo apt install -y docker.io docker-compose-v2 sudo systemctl enable --now docker sudo usermod -aG docker $USER && newgrp docker
-
Deploy:
git clone https://github.com/chinghar/SoundScorer.git cd SoundScorer docker compose up -d --build -
Visit
http://<VM_PUBLIC_IP>. First song upload will be slow (models download on first use, cached afterward in themodel_cachevolume so restarts don't re-download them).
To redeploy after pulling new code: git pull && docker compose up -d --build.
I haven't been able to test-build or run this image myself (no local Docker available in this
environment) — the Dockerfile/Caddyfile/compose setup follows standard, well-established patterns,
but treat the first docker compose up --build as the real first test.
- Single-process, in-memory background jobs (
BackgroundTasks) — fine for local/demo use, not for concurrent multi-user production load. A real deployment would want a proper task queue. - No auth, accounts, or history — every song/attempt is anonymous and world-readable by ID.
- CORS is locked to
http://localhost:3000; updatebackend/app/main.pyif you serve the frontend from elsewhere.