Comet

The Problem With Permanent, Unchecked Links

Most URL shorteners assume a link should live forever and every click is worth tracking. The result is years of dead redirects that still resolve, visitor logs full of IP addresses, and a short domain that hides where it actually leads. That last part makes shorteners a favourite wrapper for phishing: the person clicking cannot see the destination, and the service rarely checks it either.

I wanted the opposite for the everyday case of sharing a meeting link, a paper or a file with a group. A link should check where it leads before anyone opens it, count visits without identifying anyone, and disappear when it is no longer needed.

The Constraints I Set

Four constraints shaped almost every decision. First, nothing lives longer than 7 days: LINK_MAX_TTL_HOURS is 168, and expiry is enforced on every read rather than left to a cleanup job. Second, no visitor IP addresses are stored, so analytics had to work from salted hashes.

Third, every destination is checked before its short code exists. That puts a network fetch and an AI call on the creation path, which made latency a design problem from the start. Fourth, it had to run on one small instance: the deployment is pinned to a single replica, mounts a single volume at /data and lets the app sleep when idle.

How Comet Is Put Together

Comet is a Next.js 16 frontend and a FastAPI backend, with SQLite in WAL mode as the only datastore. There are no user accounts: the browser generates a random UUID as a workspace key, and the server only ever stores an HMAC of it. A short link hits a Next.js route handler, which forwards the visitor's user agent and referrer to POST /api/v1/r/{code} and then redirects.

Expiry needed no timers. Expired rows are ignored on read, and a cleanup sweep piggybacks on ordinary requests every 15 minutes. That matters on a container that sleeps when idle, where a scheduled job would either never run or keep the machine awake.

• Next.js app: landing page, dashboard, safety check, live monitor, and the route handler that resolves and redirects short codes.
• Edge middleware: request IDs, an origin allow-list for writes, security headers, per-request trace spans and the X-Chaos fault header.
• Token-bucket rate limiter: separate budgets per action, such as 12 creates, 120 resolves and 6 AI calls per minute per client.
• Guardian: URL normalisation, a 12-signal rule-based score, a page fetch that refuses private network addresses, and the Jev decision, merged into one verdict.
• Lookup path: a Bloom filter sized for 100,000 codes at a 0.1% false-positive target, then an in-memory TTL cache of 5,000 entries, then SQLite.
• Storage: destinations encrypted with AES-256-GCM using the short code as associated data, and link passwords hashed with Argon2id.
• Analytics: counters by day, hour, device, browser, OS and referrer, plus a 4 KB HyperLogLog per link for unique visitors, written in a background task after the redirect is sent.

Why a Confidence-Gated Model Over a Raw AI Verdict

The obvious design was to send each URL to a model and do whatever it said. Instead I treated the model as one decider among several. Heuristics score the URL first, and the fetcher adds what it learnt from the page and its redirect chain. Jev (TypeSafe's decision API, model jev-latest) then answers three typed questions: whether the link is safe, suspicious or malicious; how likely it is to impersonate a brand; and what category of site it is.

The model's answer only changes the outcome when its confidence clears JEV_MIN_CONFIDENCE, which I set to 0.55. A malicious verdict at 0.8 or above blocks creation, an impersonation probability of 0.75 or above forces the danger level, and a heuristic score of 90 blocks on its own. Every decider's raw output is stored with the verdict, so the UI can show exactly why a link was flagged.

Running it against real destinations showed why the gate matters. The page at notion.so redirects to notion.com, and the model called it safe with only 0.38 confidence, alongside a 0.41 probability of suspicious. Acting on that would have put a warning on a legitimate site, so the gate ignored it. A phishing-shaped test URL on apple-id-verify-login.example-support.top scored 90 out of 100 on heuristics and malicious at 1.0 from the model, and was blocked.

Why SQLite and In-Process State Over Redis and Postgres

A conventional stack would be Postgres for links, Redis for the cache and rate limits, and a scheduled worker for expiry. For one instance that sleeps when idle, that means three services to run, and the always-on ones defeat the point of sleeping. I kept everything in one process: SQLite with a write connection and a separate read-only connection, an OrderedDict-based TTL cache, and a token-bucket limiter held in a dictionary.

The Bloom filter earns its place on the lookup path. Mistyped codes, and fresh candidate codes during creation, are answered "definitely absent" without touching the database. Codes are 7 characters from a 57-character alphabet with no look-alike characters, which gives about 1.95 trillion combinations. Because a Bloom filter cannot delete entries, the sweep rebuilds it from active codes, and the same sweep runs at start-up so a restart never serves stale membership.

The trade-off is explicit: this cannot scale horizontally as written. The cache, limiter and Bloom filter live in each process, so a second replica would double every rate limit and serve stale cache entries. I would move shared state into Redis before adding replicas.

The Hardest Problem: When the Model Is Slow or Down

The hardest problem was putting an external AI call where a user is waiting for their short link. If Jev is overloaded or slow, a naive version either hangs the request or fails link creation, and a safety feature becomes an outage.

I fixed it in layers. Each Jev call has a 4 second timeout and at most 2 attempts, with exponential backoff from 0.25 seconds plus jitter. It skips the retry when the provider asks for more than 2 seconds via Retry-After, and it retries only on status codes such as 429, 503 and 529. Two 4 second attempts plus backoff can exceed 8 seconds, so the Guardian also wraps all AI work in a 7 second budget with asyncio.wait and cancels anything still pending.

When the model fails, the verdict records the reason, such as http_529 or timeout, and the heuristics and page metadata still decide and summarise the link. To prove the fallback works, I added an X-Chaos request header that injects faults such as jev_down, jev_slow, fetch_timeout and slow_db into a single request. A test asserts that a jev_down request makes exactly two attempts with one backoff span in the trace. The same header drives a Chaos Lab in the UI, so anyone can watch the fallback happen.

Results

The shipped feature set covers AI-checked link creation; password, click-limit, time-lock, device-routing and A/B rules; bulk shortening streamed as NDJSON; QR codes; privacy-preserving analytics; and a live monitor. The backend has 70 passing tests. They cover the links API, link lifecycle, bulk creation, workspaces, timezone-correct analytics, the chaos paths and the domain data structures.

The built-in load tester ran 500 in-process lookups at a concurrency of 10 on my laptop: about 30,000 lookups per second, a median of 7 microseconds for cache hits, 0.88 milliseconds for SQLite reads and 5 microseconds for Bloom-filter rejections. These exclude network overhead, so they show the relative cost of each layer rather than end-to-end latency. One live safety check against a real page, including the fetch and the Jev decision, took 1.4 seconds locally.

What I Would Do Differently

I would take the deep scan off the synchronous create path. A better shape is to create the link immediately in a pending state, scan in the background, and hold the first visitor on the existing warning page only if the verdict is not back yet. The creator would get their link in milliseconds, and only the first visitor to a brand-new link could ever hit the 7 second worst case.

I would also build an evaluation set before tuning thresholds. The 0.55 confidence gate, the 0.8 block threshold and the heuristic weights, such as 35 points for a brand name in a subdomain, came from reasoning and spot checks rather than labelled data. A few thousand URLs from a public phishing feed and a top-sites list would let me report precision and recall, and choose thresholds from a curve instead of intuition.