Whoop for your time. A multi-sensor AI time-analysis system: smart-glasses frames become a timestamped activity timeline, and an LLM coaching layer turns each day into a written report with objective scores — how much deep work you actually did, how often you got distracted, how well you took breaks.
No manual time tracking. No screenshots kept. Every frame is converted to text and deleted in the same breath.
flowchart LR
subgraph sensors["Sensors (event producers)"]
G["Meta glasses<br/>(Phase 1–2: MockDeviceKit)"]
L["Laptop agent<br/>(planned)"]
D["Desk cam<br/>(planned)"]
end
G -->|"frame"| V["Gemini vision<br/>gemini-flash-latest"]
L -.-> V
D -.-> V
V -->|"structured JSON event"| T[("SwiftData<br/>timeline")]
V -->|"frame deleted<br/>immediately"| X["🗑️"]
T -->|"one day of events"| C["CoachingProvider<br/>(Gemini default)"]
C -->|"summary + 4 scores"| R["Daily report"]
Two design rules hold everything together:
-
Frame-to-text-then-delete. A frame exists only in memory, makes exactly one trip to the vision API, and is destroyed the moment analysis returns. No image ever persists or leaves the device otherwise.
-
Sensors are just event producers. The timeline schema is sensor-agnostic, so new inputs (laptop screenshot agent, desk cam) plug in without touching the analysis or coaching layers:
{timestamp, source, activity, environment, screen_content, people_present, confidence}
The coaching layer sits behind a CoachingProvider protocol — implementations
receive value-type event snapshots (never persistence types) and return a
report. Gemini is the default; a Claude provider is a drop-in. Real-time
nudges (planned) are rules-based, never LLM calls, so they stay instant and
free.
Once a day (or on demand), the full event log goes to the coaching provider, which returns a written summary — what went well, where time leaked, one concrete suggestion — plus four scores:
| Score | What it measures |
|---|---|
| Deep work | Minutes in sustained, focused work blocks |
| Distraction rate | % of work-hours observations showing off-task activity |
| Context switches | Times you jumped between distinct tasks |
| Break hygiene | 0–100: regular off-screen breaks score high; screen marathons and phone-scrolling "breaks" score low |
Frame analysis sits behind a swappable backend, selectable in Debug ▸ Vision Engine:
| Engine | State |
|---|---|
| Cloud (Gemini) | Active. Frames are pool-routed to the Gemini API and deleted the moment analysis returns. |
| On-Device | Preview. The full seam is built and tested; no inference runtime ships yet, so frames continue to use Cloud. |
Selecting On-Device persists the choice and says plainly that frames still go
to Gemini — the picker never implies inference that isn't happening. The
fallback is a capability check, not a hardcoded branch: if no
VLMInferencing conformer is registered, routing resolves to Cloud and the
preview note appears. Register a conformer and both the fallback and the note
disappear with no other edit — a test proves exactly that.
Everything around the runtime is finished and covered by tests: the
VLMInferencing protocol, the settings gate, quota-router bypass (on-device
frames never touch the pool table), a thermal guard that skips a frame at
.serious rather than silently falling back to cloud, a tolerant parser for
small-VLM output, and a model downloader.
Why no runtime yet. The intended model was Qwen2-VL-2B-Instruct 4-bit (Apache 2.0, ~1.4 GB) — Apple's FastVLM is the better technical fit but its weights are Research-Only and expressly exclude "product development", so it can't ship here. Running it needs MLX, and MLX was rejected on two grounds:
- Dependency cost.
MLXVLM— the library containing the actual Qwen2-VL implementation — resolves to 16 transitive packages (swift-nio, swift-crypto, swift-asn1, swift-transformers, swift-jinja, gzipswift, yyjson, …). This project ships exactly one third-party dependency. - It can't run in the simulator. Per MLX's own docs, it requires Metal, which the iOS simulator doesn't provide. All 252 iOS tests run in the simulator with the app as the test host, so linking MLX would put the entire verification story at risk.
Core ML was the preferred route and is impractical for every candidate: no
full-pipeline .mlpackage exists (SmolVLM2's conversion is blocked by an
unsupported op, Moondream has no path, Qwen2-VL has no release). Even Apple's
own FastVLM iOS demo is a hybrid — Core ML vision encoder plus MLX decoder.
The plan: iOS 27's FoundationModels adds image input (announced at WWDC26;
iOS 26.x is text-only for third-party apps) with @Generable guided
generation — schema-constrained output rather than prompt-steered JSON,
which would delete the tolerant parser outright. Zero download, zero
dependencies, first-party, Neural Engine. It drops in behind VLMInferencing.
Runs at $0/month: Gemini free tier covers both the vision calls (~140/day at the default 60s capture interval) and the single daily coaching call. The only planned spend is Ray-Ban Gen 2 hardware for Phase 3.
- Xcode 26 (iOS 17+ deployment target).
- Gemini API key — copy the template and paste your key:
cp TimeLens/Secrets.example.plist TimeLens/Secrets.plist # then edit TimeLens/Secrets.plist and set GEMINI_API_KEYSecrets.plistis gitignored; never commit it. - Open
TimeLens.xcodeproj, let SPM resolve the Meta Wearables Device Access Toolkit, and run on an iOS simulator. No physical glasses needed — the capture pipeline runs against Meta's MockDeviceKit.
- Timeline tab — analyzed events, newest first.
- Report tab — today's coaching report and scores; generate or regenerate on demand.
- Debug tab — start/stop the capture pipeline (pairs a mock Ray-Ban Meta device), inject a test image, adjust the capture interval, and seed a synthetic test day: a scripted ~100-event workday (deep work, phone distraction spirals, a meeting, real breaks) so the coaching report can be exercised end-to-end without waiting for a real day.
For CLI verification there are launch flags: -TLAutoStart (start capture on
launch), -TLAutoReport (seed the synthetic day, generate a report, print
it), and -TLFakeGemini (stub both providers — no API key required).
SwiftUI · SwiftData · async/await · Meta Wearables DAT SDK (developer preview) via SPM. No other third-party dependencies.
- ✅ Mock capture pipeline — frames → Gemini → SwiftData → timeline UI, verified end-to-end in the simulator
- ✅ Coaching layer — daily report + four scores behind
CoachingProvider, Report tab, synthetic-day seeder - Real glasses — swap MockDeviceKit for DAT sessions on Ray-Ban Gen 2 (pairing via the Meta AI app, Developer Mode, firmware V20+)
- Laptop screenshot agent + desk cam as additional event producers
- Dashboard: scores over time, trends, rules-based nudges