A churn model is only useful if it changes who you contact. This one scores 7,043 telecom customers, then turns each probability into an expected dollar value and only calls the customers worth calling. It is tracked in MLflow, served with FastAPI in Docker, and retrained by CI behind a quality gate.
DataIBM Telco customer churn: 7,043 customers, 21 columns, 26.5% churned. A single snapshot with no dates.
0.846Test ROC-AUC, tuned XGBoost
+$9,233Net value vs month-to-month rule
197Fewer calls for more value
15Tests gate every retrain
Data flow · how a record moves
From a CSV row to a yes or no
Each dot is a customer. Three models are compared under cross-validation; at the value step, customers whose expected value is not worth a $60 offer drop out and are not called. Click a stage to read what happens there.
01 · Models
Three models, almost a tie
On ranking quality the models are close. Logistic regression is as good as default XGBoost; tuning buys a little on PR-AUC and calibration. The tuned model wins on a shallow tree, depth 2, with 366 trees at a slow learning rate.
Cross-validationCV ROC-AUC was 0.847, 0.843 and 0.849. Test scores line up with CV, so the search did not overfit.
02 · The decision rule
Call only if it pays
A probability is not a decision. Each customer gets an expected value: P(churn) × 30% save rate × 12 × monthly bill. If that beats the $60 cost of the offer, send it. The same 1,409 test customers, four ways to pick who to call:
Assumed economicsThe 30% save rate and $60 offer cost are assumptions, not measured. Section 03 checks what happens when they change.
03 · Against the simple rule
Fewer calls, more value
The obvious rule without a model is "call every month-to-month customer". The model calls fewer people and hits more often.
Customers called0vs 773 for the rule
Hit rate0%vs 42.6% for the rule
Churners reached0%vs 88.0% for the rule
Net value$0vs $37,566 · +25%
Is the gain real?
Bootstrapping the test set 2,000 times, the model beats the rule in every one of 2,000 resamples. Across 9 sensitivity scenarios for save rate and offer cost it wins all 9, by $4.6k to $51k.
Trade-offThe rule reaches more churners (88% vs 75%) but spends on many customers who would have stayed. The model gives up some reach for a better hit rate.
04 · MLOps
Tracked, served, gated
01TrackEvery run logged to MLflow with its parameters, metrics and model artefact.
02ServeFastAPI: /predict, /predict/batch (up to 1,000 rows) and /health.
03ShipDocker image deployed on Render.
04GateCI retrains, runs 15 tests and fails the build if ROC-AUC drops below 0.82.
→SmokeThe built container is started and hit with real requests before it counts.
CaveatsThe economics are assumed. There are no dates, so no drift checks. And this is a churn model, not an uplift model: it finds who will leave, not who an offer will change.