Validity
When churn models lie: leakage in mobile cohorts
Identity Flowcore journal · companion to Leakage & Validity
Leakage is not a theoretical hygiene issue. It is the usual reason a mobile churn model looks prophetic in a notebook and useless in CRM. The model has been allowed to peek at information that would not have been available at decision time.
The timestamp join that looks into next week
Event tables often carry ingest time and client time. Joining features on ingest time while labelling on client time is a classic time-travel. A payment that happened Monday night and landed in the warehouse Wednesday morning will “predict” Monday’s churn if you are sloppy. The clock contract in our practice notes exists to make this argument boring and repeatable.
Flags that only exist after the outcome
“Has completed KYC,” “received winback voucher,” “is_subscribed” — many of these columns are written after the behaviour you claim to predict. If the flag is null for everyone still in the funnel and true for everyone who converted, you have built a detector of your own backfill, not of future churn.
Campaign exposure as a secret label
Push and in-app campaigns are often logged only for users who opened them. Feeding that exposure into a model trained to predict inactivity is circular: people who disappeared never opened the message, so “no exposure” becomes a proxy for the label. Either log assignment at send time for everyone in the audience, or keep campaign features out of the score.
Holdouts that are not holdouts
Random splits on users who share a household device, or on days that include a festival, are not the same as a forward-in-time holdout. Finance is right to be suspicious. We ask students to show a time-based cut before they are allowed to quote AUC in the room.
Week three of the Cohort Signal Lab is entirely this work. It is not glamorous. It is the week people remember.