Design a Feature Store

Every machine-learning model is a function of features: the driver's trips in the last hour, the rider's cancellation rate, the city's current demand. Computing those features is a data engineering problem, not a modeling one, and it hides a trap that has silently ruined more models than any algorithm choice. The feature a model sees while it trains, and the feature it sees while it serves a live request, have to be the same feature. When they quietly diverge, the model looks brilliant offline and falls apart in production.

Uber's Michelangelo platform put a name to the fix: a feature store, the shared layer where the offline and online worlds agree on what a feature means, when it was observed, and how fast it has to be. At Uber's scale that is on the order of 10,000 features feeding 1,000 models, serving surge pricing, ETA, and fraud detection. Maya, a data engineer on the ML platform team, has been handed the prompt: "build the thing that makes features consistent." This chapter is that build.

The difficulty is not throughput. It is that two consumers want opposite things from the same data, and the seams between them leak.

Step 1 - Understand the Problem and Establish Design Scope

"Build a feature store" hides at least three systems. The clarifying questions pin down which one, and which properties actually matter.

Candidate: Who consumes features, and how differently?

Interviewer: Two consumers. Online model serving needs a feature vector for a single entity in a few milliseconds, on the request path of a live prediction. Offline training needs the historical value of those same features across months of past events, to build a training set.

Candidate: How fresh do features need to be at serving time?

Interviewer: It varies by feature. "Driver's city" barely changes. "Driver's trips in the last five minutes" has to be seconds-fresh or it is useless. The store has to support both without forcing everything to be real-time.

Candidate: What is the latency budget on the serving path?

Interviewer: A prediction has a tight end-to-end budget, and the feature lookup is only one part of it. Assume single-digit milliseconds, p99, for reading a full feature vector for one entity.

Candidate: How correct does the training data have to be?

Interviewer: This is the whole point. The training set must reflect what the feature actually was at the moment each labeled event happened. If the model trains on a value it could not possibly have known yet, it learns to cheat, and it fails in production. No leakage from the future.

Candidate: Do the training-time and serving-time values of a feature have to match exactly?

Interviewer: Yes. If the two paths compute the same feature differently, the model is trained on one distribution and served another. That skew is the failure we are trying to design out.

Candidate: A few things I want to confirm are in scope: adding a brand-new feature and making it usable for training over past data, and recovering when a materialization job fails.

Interviewer: Both in scope. Backfilling a new feature correctly is exactly as hard as the point-in-time problem, and for the same reason.

With that, the requirements are clear.

Functional requirements

• Let engineers define a feature once (its source, transformation, entity, and freshness) and reuse that definition everywhere.
• Serve a feature vector for a given entity (a driver, a rider, a city) with a low-latency keyed lookup at inference time.
• Generate a training set: for a set of labeled events, produce each feature's value as it was at that event's timestamp.
• Support both batch-computed features (daily) and streaming-computed features (seconds), under one interface.
• Add a new feature and make it available for both serving and training, including over historical data.