Repository navigation
fix(analytics): measure open and click rates against tracked emails - #534
Open
joanproductivo wants to merge 4 commits into
Open
joanproductivo wants to merge 4 commits into
joanproductivo wants to merge 4 commits into
Conversation
With email tracking set to "Marketing only" or "Disabled", transactional emails (or all emails) go out without the open pixel or rewritten links, yet the dashboard and analytics still divided opens and clicks by every email sent, diluting both rates. Open and click rates are now measured against the emails sent with tracking under the project's current tracking mode. The streams a mode tracks come from shouldTrackEmail, plus the no-tracking configuration set toggle that SESService applies, so stats and the send path agree. Opens and clicks are filtered the same way, so neither rate can pass 100% after the mode changes. Everything stays within the columns of the stats index and needs no schema change. - analytics: a `tracked` count beside `sent` in every count, timeseries point and mailbox provider row; the campaign leaderboard tracks a campaign by its type, which sets the stream of all its emails - dashboard: rates measured like analytics (tracked, not simulated) - the tracking mode is part of both cache keys - the chart measures opens and clicks against `tracked` as well Projects tracking everything get the same analytics numbers as before. Refs useplunk#533 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard counted every email row created in the period as sent: emails still queued or that failed, inbound mail, and mail to the SES mailbox simulator. "Emails sent" now counts outbound emails that went out, the same scope as `sent` in analytics, so the dashboard and the analytics page agree. The onboarding banner keys off this number, so it now disappears once the first email actually goes out rather than when it is queued. Refs useplunk#533 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tracked With nothing sent with tracking in a period or stream, open and click rates read 0%, which says nobody opened rather than that nothing was measured. Each metric now knows the volume it is measured on (`volume`: emails sent, or emails sent with tracking for the figures built on opens and clicks), and a figure with nothing to measure shows "—" instead: - overview tiles show "—" with "Not tracked" under them, and the chart explains that none of the emails were sent with tracking - the by-stream table, leaderboards and mailbox providers show "—" - the dashboard's open rate card shows "—" and "Not tracked" - deltas compare against the previous period's tracked volume Refs useplunk#533 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Explain how the dashboard and analytics measure open and click rates under each tracking mode, that a period without tracked emails shows "Not tracked", and that the current mode decides which emails count. Refs useplunk#533 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Bug
With Settings → Email tracking set to Marketing only or Disabled, the open and click rates on the dashboard and in Analytics still count emails that were sent without tracking. Those emails carry no open pixel and no rewritten links, so they can never record an open or a click, and they dilute the rate: one untracked transactional email plus one opened campaign email reads 50% instead of 100%. The dashboard also counted every email row created in the period as sent (queued, failed, inbound and mailbox-simulator mail), so it disagreed with Analytics even with tracking enabled.
Root cause
Whether an email is tracked is decided at send time from the project's mode and the email's stream (
EmailService.shouldTrackEmail, plus the no-tracking configuration set toggle inSESService), but no stats query takes it into account:ActivityService.getStatsdivided opens by_count._all, andAnalyticsService.toMetricsdivided them bysent.Solution
EmailService.trackedSourceTypes(mode)derives the streams a mode tracks fromshouldTrackEmailand the same toggleSESServiceapplies, so stats and the send path follow one rule.EmailService.trackedStreams(projectId)resolves it for a project through the cachedProjectService.id, and falls back to "everything tracked" if the project cannot be read.trackedcount next tosentin every count, timeseries point and mailbox provider row. Open and click rates are measured againsttracked, andopened/clickedare filtered with the same predicate, so neither rate can pass 100% after the mode changes. The tracked streams go into the SQL as literals: a bound parameter cast to"EmailSourceType"is cast again on every row, which madecountsByStreamabout three times slower in my measurements.tracked: 0; tracked campaigns keep matching their campaign page.groupByon (sourceType,simulated) served fromemails_projectId_createdAt_stats_idx. In a separate commit, "Emails sent" counts the outbound emails that went out; the onboarding banner, which keys off that number, now hides once the first email actually goes out rather than when it is queued.sent, ortrackedfor open, click and click-to-open). With nothing tracked, the figure shows "—" instead of 0%: overview tiles add "Not tracked", the chart explains that none of the emails were sent with tracking, and the stream table, leaderboards and providers show "—".Projects with tracking Enabled get exactly the same Analytics numbers as before (there is a test for it). There is no schema change, migration or new index: every column the queries read is already in the
INCLUDEof the stats index.Worth knowing
trackedcolumn at send time, the waysimulatedis stamped, would make this exact and would also cover per-email overrides like feat: per-template, per-campaign and per-send tracking overrides #433. I kept this PR migration-free;trackedSourceTypeswould still be the fallback for rows sent before such a column. Happy to follow up with that if you prefer it.CampaignService.getStats) still measure againstsentCount; I left them out of scope.getStatsandemailScope(astatusfilter), so one of the two will need a small rebase.Type of Change
fix:Bug fix (PATCH version bump)Testing
AnalyticsService.test.ts(8 cases),ActivityService.getStats(5),EmailService.trackedSourceTypes/trackedStreams(7), andmetrics.test.tsfor the web (3). On the currentnextthe issue cases fail by assertion (25% or 50% instead of 100%), while the Enabled and mailbox-simulator cases already pass and keep passing.next. Each of the four commits builds and passes its own tests.yarn lintreports no errors andyarn build --filter=api --filter=websucceeds. Prettier and ESLint are clean on the lines this PR changes.background-index-builderdoes, andVACUUM ANALYZE. The analytics queries run through Prisma with real parameters (median of 15 runs):countsByStream14 → 18 ms, daily timeseries 32 → 36 ms. The dashboard, as the SQL Prisma generates for the oldaggregateand the newgroupByunderEXPLAIN ANALYZE: 14 → 16 ms. All of them are anIndex Only Scanonemails_projectId_createdAt_stats_idxwith 0 heap fetches.Checklist
Related Issues
Closes #533
Related: #531 (automatic pixel fetches counted as opens), #432 / #433 (per-email tracking override)
🤖 Generated with Claude Code