"Opened" Doesn't Mean Opened for Customer.io In-App Messages: How to Actually Measure Them
Email marketers have already lived through one version of this story. Apple's Mail Privacy Protection pre-fetches tracking pixels, so an email sent to an Apple Mail user registers as opened whether the person read it or never saw it. Customer.io's answer was to split the metric: since 20 March 2025 the platform reports a separate Human Opened metric that strips out MPP pre-fetches, bot user agents and known scanners. Its docs are blunt about why—all opens from Apple Mail are counted as non-human unless the person goes on to click. We covered what that did to behaviour-triggered journeys built on opens when the dust settled.
Email opens, though, were at least designed to measure a person's action before privacy tech broke them. In-app "opens" were never an engagement metric to begin with. Customer.io's Metrics Overview defines it in one line: "For in-app-messages, the opened metric means the message has been delivered and displayed to a person." Displayed. Not read, not tapped, not necessarily even noticed.
If you've been quoting an in-app open rate to leadership, you've been reporting reach and calling it engagement. This post resets the metric set: what each Customer.io in-app metric means, and which numbers belong on the slide instead. It ends with how to wire an in-app message to an outcome you can defend in a budget conversation.
Why the email mental model breaks for in-app
An email open is a proxy for human attention. An in-app open is a statement about rendering. The two share a name and nothing else.
For email, Customer.io counts an open when the recipient's client loads a tracking pixel or the recipient clicks a link. Flawed, gameable by pre-fetching, but aimed at a person doing something. In-app messages have no inbox to sit in. The in-app metrics documentation explains that messages display as soon as a person opens your app or visits your website: "a person doesn't have to do anything to 'open' the message, so in-app messages don't have a 'delivered' state."
So an 82% in-app open rate tells leadership exactly one thing: 82% of the people you targeted opened your app while the message was live. That's app activity wearing an engagement costume. A healthy product with daily active users will post in-app "open rates" that would be miraculous for email, and the number says nothing about whether anyone cared.
This isn't Customer.io mislabelling anything. The metric is named consistently with how the channel works, and the definition sits in plain sight in the docs. The misread happens on our side, when the email scorecard gets copied across channels without checking what each column means. It's the same vanity-metric trap we unpicked in the lifecycle marketing scorecard, playing out on a channel where the metric was never pretending in the first place. Even vendors who sell in-app tooling agree: Pushwoosh's guide to in-app measurement opens by noting that "in-app messages don't rely on a traditional open rate metric" because display requires no user action.
What each Customer.io in-app metric actually means
Customer.io tracks four states for an in-app message: Sent, Opened, Clicked and Failed. A message's status is the last metric reported for that individual delivery—the state of that message for that person.
Opened means displayed
The docs definition is short: "The message has been delivered and displayed to a person." The path there matters. When your workflow reaches an in-app message, Customer.io creates the delivery and marks it Sent. It stays Sent until the person opens your app or visits your website, at which point it displays and becomes Opened. If they never show up before the message expires, it expires still marked Sent.
That last detail has a reporting consequence: in-app metrics keep accruing until the message expires. The docs advise treating metrics as incomplete until then, because anyone who hasn't opened your app yet can still flip from Sent to Opened tomorrow.
Sent, Clicked, Failed and the per-person status
Sent means the delivery exists but hasn't reached a device. Clicked means the person tapped an action with a Tracked Name—a button, link or other clickable element you chose to track. Click an element without a Tracked Name, like an untracked close button, and the message stays Opened. Failed means the message never rendered, which most commonly traces back to unresolved Liquid—a variable with no value and no fallback.
The channel comparison table in the Metrics Overview confirms how small the in-app reporting surface is: Opened, Clicked and Converted. No Delivered, no bounces, no suppressions, no unsubscribes. Three numbers plus Failed. Which makes it all the stranger that most teams report the one number that requires no engagement at all.
The numbers to report instead
Drop the open rate from your in-app reporting and three useful numbers remain: click rate against displays, dismissals if you deliberately choose to track them, and conversion against a goal.
Click rate against displays
The click-to-open ratio—Clicked divided by Opened—is the one place where in-app's odd vocabulary works in your favour. Because "opened" means displayed, the denominator is honest: of everyone who saw this message, how many acted on it? The in-app metrics docs recommend the clicks-to-opens ratio for exactly this purpose, as the share of people who received a message and followed a call to action.
Make the clicks themselves legible while you're at it. Give every action a descriptive Tracked Name and Customer.io breaks clicks down by name in the Tracked Responses section of your campaign's Metrics tab—"start-trial" against "remind-me-later" rather than one undifferentiated Clicked count. Tracked responses export to CSV per day or hour, so the breakdown can land in your reporting warehouse, not just the UI.
Dismissals, on one condition
Customer.io has no native dismissed metric. The statuses are Sent, Opened, Clicked and Failed, and a tap on an untracked close button leaves the message sitting at Opened. The only way to count dismissals is to give the close action a Tracked Name—and the docs advise against it, because every dismissal then registers as a Clicked, polluting your engagement metric with its opposite.
If dismissal data matters to you, take the trade-off deliberately. Name the close action something unmistakable ("modal-dismissed"), then report engagement from your CTA action names in Tracked Responses and ignore the aggregate Clicked number, which now blends the two. If you won't commit to that discipline, leave close buttons untracked and accept that dismissals stay invisible. What you must not do is report the inflated Clicked figure with a dismissal action quietly inside it.
Conversion against a goal
The only in-app number that survives a budget conversation is conversion against a goal you set before launch. Customer.io's campaign metrics documentation is specific about the mechanics. A delivered message is marked converted when the person achieves your campaign's goal within the conversion criteria window. The message must also have been sent, opened or clicked within the specified period.
Two things follow. First, no conversion criteria means an empty Converted chart—the metric doesn't exist until you define the goal. Second, be careful which measurement layer you're in: Customer.io's workspace-level Goals are a separate object from per-campaign conversion criteria, and running both against the same outcome double-counts revenue. Pick the layer, document it, and report one number.
How to wire an in-app message to a measurable outcome
Work backwards from the outcome, not forwards from the message. The sequence that keeps an in-app message honest takes five steps, and four of them happen before you write a word of copy.
Pick the single behaviour the message exists to drive—an event you already track, like trial_started or push_opt_in. Set it as the campaign goal with a conversion window that matches how quickly the behaviour plausibly follows the message. Give every call to action a Tracked Name so Tracked Responses can tell you which path people took. Leave the close button untracked unless you've made the deliberate dismissal trade-off above. Then report Clicked and Converted, using Opened only as the denominator that turns them into rates.
Done this way, in-app messages measure well because they reach people at the moment of highest context—inside your product. Airship's 2025 mobile app benchmarks report that in-app messages drive a 14% average increase in push notification opt-in rates among opted-out users. The same report found apps combining push and in-app messaging with segmentation and automation saw an average 31% higher Engagement Score than their app category average. Both numbers describe in-app aimed at a specific, measurable behaviour... not a welcome modal shown to everyone. If you're choosing which messages to build first, we've made the case for three behaviour-triggered in-app messages that beat the welcome modal.
A reporting template that survives a CFO's questions
Replace the open-rate line on your in-app slide with four rows per message. Each one answers a question a finance person will eventually ask.
Displayed (the metric Customer.io calls Opened): how many people saw the message. Report it as reach and context—the denominator—never as a success measure. Click rate on CTA actions: clicks on named calls to action divided by displays, from Tracked Responses, with dismissal actions excluded. This is your engagement number. Converted against goal: the percentage who completed the campaign goal within the window, from one measurement layer only. This is your outcome number. Lift against holdout, if you're running one: conversion rate of people who saw the message minus the rate of those held back. Customer.io has made this a checkbox inside the A/B test flow, and it's the only row on the slide that proves causation rather than correlation.
A CFO reading that slide can trace every number to a behaviour. Nobody has to explain, three quarters in, that the 82% was people opening the app. And the same hierarchy—reach, engagement, outcome, incrementality—is the one we recommend for lifecycle reporting and attribution across every channel, not just this one.
The fix costs nothing. Rename the column from "open rate" to "displayed", add the conversion row, and watch the conversation change from defending a vanity number to discussing what the message earned.
Frequently asked questions
What does "opened" mean for a Customer.io in-app message?
It means the message was delivered and displayed to a person—nothing more. Customer.io's Metrics Overview states it directly: "For in-app-messages, the opened metric means the message has been delivered and displayed to a person." The person takes no action; the message renders automatically when they open your app or visit your website.
Why is my in-app open rate so high?
Because it measures app activity, not interest in the message. An in-app message counts as opened the moment it displays, so your open rate reflects how many targeted people opened your app or site while the message was live. A product with engaged daily users will post in-app open rates far above anything email can produce, and the number carries no information about engagement with the message itself.
Does Customer.io track a "delivered" status for in-app messages?
No. In-app messages display as soon as a person opens your app or visits your website, so there's no separate delivered state—messages move from Sent straight to Opened. The channel table in Customer.io's Metrics Overview shows in-app produces three audience metrics: Opened, Clicked and Converted.
How do I measure whether an in-app message actually worked?
Report two numbers: the click rate on named calls to action (Clicked divided by Opened, from Tracked Responses) and the conversion rate against the campaign goal you set before launch. Displays tell you reach; clicks tell you engagement; conversion tells you whether the message drove the behaviour it exists for. If you need proof of causation, add a holdout group and report lift.
Can I track clicks on in-app messages?
Yes, with one condition: each action needs a Tracked Name. Customer.io only counts a click when the person taps a button, link or element with a Tracked Name set. Clicks on untracked elements don't register, and the Tracked Responses section breaks your clicks down by action name so you can tell which call to action did the work.
Can I track dismissals of in-app messages?
Not natively—Customer.io's in-app statuses are Sent, Opened, Clicked and Failed, with no dismissed state. Tapping an untracked close button leaves the message at Opened. The workaround is to give the close action a Tracked Name, but every dismissal then counts as a Clicked, which is why the docs recommend leaving close buttons untracked. If you track dismissals anyway, report engagement from CTA action names rather than the aggregate Clicked metric.
How do I attribute a conversion to an in-app message?
Set conversion criteria on the campaign. Customer.io marks a message converted when the person achieves the campaign goal within the conversion criteria window, provided the message was sent, opened or clicked within the specified period. Without conversion criteria, the Converted metric stays empty. Keep workspace-level Goals and per-campaign conversion criteria measuring different outcomes, or you'll double-count.
What are Tracked Responses in Customer.io?
A per-action breakdown of in-app clicks. When actions in your message have Tracked Names and Track Clicks enabled, the Tracked Responses section of your campaign's Metrics tab shows how people responded, broken down by message and action name. You can export tracked responses to CSV per day or hour, which makes them usable in reporting outside the Customer.io UI.
Why does my in-app message still show as Sent?
The person hasn't opened your app or visited your website on a page where the message can display. A Sent message means Customer.io created the delivery, but it hasn't left the platform until it displays. If the person never shows up before the message expires, it expires still marked Sent.
What does Failed mean for an in-app message?
The message couldn't render and never left Customer.io. The most common cause is unresolved Liquid—a personalisation variable that didn't exist for that person and had no fallback, making the message unreadable. Check the Liquid in your message and add fallbacks for any variable that might be empty.
Do in-app messages have Human Opened and Machine Opened metrics like email?
No—those metrics apply to email only. Email needs the split because Apple's Mail Privacy Protection, Gmail prefetching and bots generate opens no human made. In-app doesn't have that problem: the opened metric is already a literal display count, so there's no machine noise to filter out.
When are my in-app metrics complete?
When the message expires. Customer.io continues gathering metrics until the in-app message stops being sent, because anyone still sitting at Sent can flip to Opened the next time they open your app. Judge performance on the post-expiry numbers, and treat anything earlier as provisional.
What should I report instead of open rate for in-app messages?
Three numbers: displays (Customer.io's Opened) as reach, click rate on named CTA actions as engagement, and conversion against the campaign goal as the outcome. Add lift against a holdout group when you need to prove the message caused the behaviour rather than coincided with it. Display count is the denominator for the other numbers, never a KPI on its own.
How should I judge my in-app conversion rate?
Against your own baseline and goal, not an industry benchmark. Published in-app benchmarks rarely share your audience, trigger logic or goal definition, so the comparison misleads more than it informs. Compare each message against your other in-app messages, against its own past performance, and—most rigorously—against a holdout group, which tells you the conversion lift the message earned rather than borrowed.
Sources
- Metrics Overview. Customer.io Docs, updated 8 June 2026.
- In-app metrics. Customer.io Docs, updated 8 June 2026.
- Campaign and Broadcast Metrics. Customer.io Docs, updated 9 June 2026.
- How to Measure the Effectiveness of In-App Messaging: 6 Key Metrics. Pushwoosh, 27 February 2024.
- 2025 Mobile App Push Notification Benchmarks. Airship, 2025.
