DecideX

The Problem: Churn Scores Without Next Steps

Retention teams at subscription businesses usually find out a customer is leaving when the cancellation arrives. Churn models exist, but a probability on its own leaves the hard part to the person reading it: what to do, who should do it, how urgently, and whether it is worth a discount. The richest signal, the free-text notes from support calls and emails, rarely reaches the model at all.

I wanted a tool that a non-technical customer team could open and act on. It would read the notes, score the customer and turn both into a recommendation with reasons in plain English, a price tag on the risk and a draft of what to say.

The Constraints I Set

Four constraints drove most of the decisions. The audience is non-technical, so every screen had to explain itself in plain words and every recommendation had to carry its reasons. People stay in charge: uncertain calls go to a review queue, reviewers can override anything, and gender and senior-citizen status are dropped before training so they cannot influence a recommendation.

Idle cost had to be as close to zero as possible, so the backend runs with sleep mode enabled and must go quiet when nobody is using it. Finally, there are no accounts. Each browser generates a random workspace key and the server stores only an HMAC hash of it, which removes sign-up friction without storing anything that identifies a person.

Architecture: Four Stages and a Human

A decision runs as a four-stage pipeline in FastAPI. The note reader and the churn model run concurrently with asyncio.gather, their outputs feed a structured decision engine, and a review policy decides whether a person needs to check the result. Each stage records its status and timing, and the decision stores the exact inputs, so every recommendation carries its own audit trail.

On top of the pipeline sit the features that make it useful day to day: yearly revenue at risk (monthly bill times twelve times the probability), a best-offer search, LLM-drafted emails and call scripts, expiring share links for a one-page report, a fairness check and an activity log. None of them needs a user account; they all hang off the browser's workspace key.

• Next.js 16 frontend: the app, dashboards and printable reports, proxying /api to the backend.
• LLM note reader: gpt-oss-20b in JSON mode returns sentiment, complaint topics, competitor mentions, cancellation intent and evidence quotes; any quote not found verbatim in the notes is discarded.
• Churn model: a calibrated scikit-learn pipeline that returns a probability, a risk level and occlusion-based reasons, measured by swapping each feature for its typical training value.
• Decision engine: a structured-decision API answers four typed questions (action and owning team as choices, urgency as a score, escalation as yes or no), each with probabilities; if it is unreachable, deterministic backup rules answer and the decision is always flagged.
• Review policy: flags confidence below the workspace threshold, escalation probabilities between 40% and 60%, unreadable notes, and actions that contradict the risk level.
• Storage: async SQLAlchemy 2 on SQLite, with a startup step that adds missing columns so deploys never need a manual migration.

Why a Cost-Based Threshold Over the Default 0.5

I trained four candidates on a stratified 60/20/20 split of the IBM Telco dataset (7,043 customers, 26.5% churn): a class-prior baseline, logistic regression, a 300-tree random forest and LightGBM. I picked the winner by validation PR-AUC rather than accuracy, because when a quarter of customers leave, a model can look accurate while missing most of them. The random forest edged ahead at 0.643, against 0.640 for logistic regression and 0.633 for LightGBM, and I wrapped it in isotonic calibration with five-fold cross-validation so that a stated 30% means roughly 30%.

The bigger lever was the threshold: at the default 0.5, the model caught only 50.8% of leavers on the test set. I treated a missed churner as five times as costly as an unnecessary intervention and chose the threshold that minimises expected cost, which landed at 21%. Recall rose to 85.6% and missed churners fell from 184 to 54 out of 374, at the price of precision dropping from 64.6% to 48.3%. For a retention team a few extra check-in calls are cheap and a lost customer is not, so I accepted more false alarms and let the review queue absorb them.

Why SQLite on a Volume Over Postgres

The original production configuration used Postgres through asyncpg, with SQLite only for local development, and I ran the full test suite against a local Postgres 18 instance to prove it worked. But a managed database is a second service that runs around the clock, which cut against the zero-idle-cost goal, and open database connections count as activity that can stop the app sleeping. DecideX runs as a single instance with modest write volume, so Postgres was buying very little.

I switched to SQLite in WAL mode on a persistent volume mounted at /data and removed the Postgres driver. The trade-offs are real: volumes rule out multiple replicas, redeploys cause a few seconds of downtime, and backups become my responsibility. The biggest risk was silent data loss, because without a volume every redeploy would start from an empty file. So the app refuses to start in production unless the database file sits inside the mounted volume, and the log says why.

The Hardest Problem: Making the Backend Actually Sleep

The host sleeps a service after roughly five to ten minutes without outbound traffic. My first audit found two things keeping it awake around the clock: the dashboard refetched its data every 60 seconds, and the public landing page called the backend on every visit to fetch live model metrics. I removed the polling, moved the landing-page metrics and sample customers into static JSON snapshots generated by a small script, and made the guide, the new-decision form and bulk upload render without touching the API. The server now wakes only when someone picks a sample, starts typing, drops in a file or opens a page that needs saved data.

The subtler bug only showed up when I measured it. After a decision, lsof showed two HTTPS connections from the backend to the AI providers still established after twelve seconds of idle, because httpx keeps connections alive for reuse and does not close them proactively. When a provider eventually closes one of those sockets, the backend's reply is outbound traffic, which can reset the sleep timer. In serverless mode I set max_keepalive_connections to zero; the same check then showed no open connections two seconds after the last request, and a regression test now guards the setting.

Waking up had to feel acceptable too. I cut backend import time from 2.0 to 0.7 seconds by loading scikit-learn and LightGBM only when first needed, and the frontend retries unreachable responses with delays of 1, 2, 3.5 and 5 seconds while showing a short waking-up banner. Decision requests carry an idempotency key, so the retries are safe: in a test with the server stopped, the decision completed about nine seconds after the click once the server came back, and exactly one decision was created.

Results

On 1,409 held-out customers the calibrated model reaches ROC-AUC 0.844 and PR-AUC 0.653 with an expected calibration error of 0.018, and at the tuned threshold it catches 85.6% of customers who leave. In local tests with the live LLM and decision engine, a full decision takes about 1 to 1.5 seconds. The best-offer search re-scores every single and paired change in one batch in roughly 150 milliseconds, and for one sample customer it found that a two-year contract would take the chance of leaving from 62% to 19%, below the warning line.

The backend has 51 automated tests, and the suite also passed on Postgres before the switch to SQLite.

What I Would Do Differently

The best-offer finder re-scores a customer with one feature changed, but the model learned correlations from observational data, not causal effects. Customers on two-year contracts churn less partly because loyal customers choose them, so the predicted drop from offering one is optimistic. Before trusting those numbers with real discounts, I would validate offers with a holdout experiment or an uplift model.

I also relied on prompt instructions alone to keep the LLM drafts honest, and testing showed why that is not enough: one call script promised the customer could cancel any time, which was never in the facts it had been given. I tightened the prompt, but next time I would build a small evaluation set of drafts from day one and check automatically that every commitment in a draft traces back to the offer it was given.