1. Requirements Gathering

Functional Requirements

We need a system that:

• Recommends personalized content/products
• Learns from user behavior
• Supports millions of users and items
• Handles new users and new items
• Updates recommendations based on recent activity

Non-Functional Requirements

Requirement	Target
Relevance	High CTR/Conversion
Latency	<100ms
Availability	99.9%
Freshness	Minutes or seconds
Scalability	Millions of users

---

2. High-Level Approach

I would use a two-stage recommendation architecture:

Candidate Generation
        ↓
Candidate Ranking
        ↓
Recommendations

Reason:

If we have 100 million products/videos, we cannot rank all 100 million for every request.

Instead:

100M items
   ↓
1000 candidates
   ↓
Top 20 recommendations

This is the architecture used by companies like YouTube, Netflix, and Amazon.

---

3. Data Collection Layer

Before training any ML model, we need behavioral data.

User actions:

View
Click
Like
Watch Time
Purchase
Add To Cart
Share

Technology: Apache Kafka

Use:

• Apache Kafka

Why?

• High throughput
• Durable event storage
• Real-time streaming
• Industry standard for event ingestion

Architecture:

User
  ↓
Kafka Topics

Example:

{
  "userId": "123",
  "itemId": "456",
  "event": "click",
  "timestamp": "..."
}

---

4. Feature Engineering Layer

Raw events aren’t enough.

We need features like:

Last 7 day clicks
Favorite category
Average watch time
Purchase frequency

Technology: Apache Flink

Use:

• Apache Flink

Why?

Because requirements mention:

Freshness

We want recommendations to adapt within minutes.

Flink continuously computes:

User clicked 5 football videos

and updates features immediately.

Pipeline:

Kafka
   ↓
Flink
   ↓
Feature Store

---

5. Feature Store

Store ML features centrally.

Technology: Feast

Use:

• Feast

Why?

Without a feature store:

Training Features
!=
Serving Features

This causes training-serving skew.

Feast ensures:

Same feature definitions
Same feature values

for both training and inference.

Examples:

user_avg_watch_time
user_category_affinity
item_ctr
item_popularity

---

6. Candidate Generation

Now the ML starts.

Requirement:

100 million items

cannot be ranked directly.

Need:

100 million
      ↓
1000 candidates

---

Algorithm Choice: Two-Tower Neural Network

Why not collaborative filtering?

Problems:

• Doesn’t scale well
• Cold-start issues
• Hard to include rich features

Modern industry uses:

Two-Tower Retrieval

Architecture:

User Features
      ↓
User Tower
      ↓
User Embedding

Item Features
      ↓
Item Tower
      ↓
Item Embedding

Training objective:

Bring relevant user-item pairs closer in vector space.

---

Technology: PyTorch

Use:

• PyTorch

Why?

• Industry standard
• Flexible
• Excellent for deep learning

Alternative:

• TensorFlow

---

7. Vector Search

After training:

Each item becomes:

128-dimensional vector

Need nearest-neighbor search.

---

Technology: FAISS

Use:

• FAISS

Why?

Brute force:

100M comparisons

Too expensive.

FAISS provides:

Approximate Nearest Neighbor

search in milliseconds.

Flow:

User Embedding
      ↓
FAISS
      ↓
Top 1000 Candidates

Latency:

10-20 ms

---

8. Candidate Ranking

Now we have:

1000 candidates

Need:

Top 20

This is where ranking models shine.

---

Algorithm Choice: XGBoost

Why?

Recommendation data is mostly:

Tabular Features

Examples:

User Age
Item Popularity
Category Match
Historical CTR
Watch Time

Tree-based models perform extremely well.

Use:

• XGBoost

or

• LightGBM

---

Model Predicts

P(click)
P(purchase)
P(view)

Final score:

0.5 × CTR
+
0.3 × Conversion
+
0.2 × Freshness

---

9. Advanced Ranking (Optional)

For very large systems.

Use:

DLRM

Deep Learning Recommendation Model.

Created by:

• Meta

Why?

Captures complex interactions:

User × Item
User × Context
User × Device

better than trees.

Use:

• PyTorch

---

10. Sequential Behavior Modeling

Users have evolving interests.

Example:

Learn Python
Learn SQL
Learn Spark

Next recommendation:

Data Engineering

Need sequence understanding.

---

Algorithm: Transformers

Use:

• SASRec
• BERT4Rec

Why?

Transformers understand:

Order
Context
Long-term dependencies

better than traditional collaborative filtering.

Used heavily by:

• TikTok

---

11. Exploration vs Exploitation

Problem:

If user watches football videos:

Football
Football
Football
Football

Feed becomes repetitive.

Need exploration.

---

Algorithm: Contextual Bandits

Use:

Thompson Sampling

Why?

Balances:

Known Interests
+
Discovery

Example:

90% football
10% basketball

This prevents recommendation bubbles.

---

12. Real-Time Feedback Loop

Recommendations must improve continuously.

Pipeline:

Click
 ↓
Kafka
 ↓
Flink
 ↓
Feature Store
 ↓
Retraining

Implicit feedback:

Clicks
Watch Time
Purchases
Likes

is much more abundant than ratings.

---

13. Model Training Pipeline

Offline training:

Kafka
 ↓
Data Lake
 ↓
Feature Engineering
 ↓
Model Training
 ↓
Model Registry
 ↓
Deployment

Storage:

• Apache Iceberg
• Snowflake
• BigQuery

Training frequency:

Daily
or
Hourly

depending on freshness requirements.

---

14. Caching Layer

Requirement:

Latency <100ms

Use:

• Redis

Cache:

User Embeddings
Popular Items
Top Recommendations

This reduces repeated model inference.

---

15. Final Interview Summary

“I would design a two-stage recommendation system. User events are ingested through Kafka and processed by Flink to generate real-time features stored in Feast. For candidate generation, I’d train a Two-Tower retrieval model in PyTorch and store item embeddings in FAISS for low-latency ANN search. This retrieves around 1000 candidates from millions of items. For ranking, I’d use XGBoost or DLRM to predict click-through and conversion probability using user, item, and contextual features. To capture evolving interests I’d introduce transformer-based sequential models such as BERT4Rec. Recommendations would continuously improve through Kafka/Flink feedback loops, while contextual bandits like Thompson Sampling would balance exploration and exploitation. Redis would be used to meet sub-100ms latency requirements.”

This answer clearly connects requirement → component → ML algorithm → technology → justification, which is exactly what senior-system-design interviewers are looking for.

Production-Grade Recommendation System Architecture

                          ┌──────────────────────────────┐
                          │        Client Layer          │
                          │  Web / Mobile / API Clients  │
                          └──────────────┬───────────────┘
                                         │
                                         ▼
                          ┌──────────────────────────────┐
                          │     API Gateway Layer        │
                          │  Auth / Rate Limit / Router  │
                          └──────────────┬───────────────┘
                                         │
                                         ▼
                ┌────────────────────────────────────────────┐
                │        Recommendation Service              │
                │   (Low-latency Serving Layer <100ms)      │
                └──────────────┬───────────────┬────────────┘
                               │               │
        ┌──────────────────────┘               └──────────────────────┐
        ▼                                                              ▼
┌──────────────────────┐                                   ┌──────────────────────┐
│ Candidate Generation │                                   │  Real-time Cache     │
│ (Two-Tower Model)    │                                   │  Redis / Memcached   │
│ + ANN Retrieval      │                                   │                      │
└─────────┬────────────┘                                   └─────────┬────────────┘
          │                                                          │
          ▼                                                          │
┌────────────────────────────────────────────────────┐              │
│ Vector Search Layer                                │              │
│ FAISS / Milvus / ScaNN                            │              │
└──────────────┬─────────────────────────────────────┘              │
               │                                                    │
               ▼                                                    │
      ┌──────────────────────┐                                      │
      │ Candidate Set (1K)   │◄─────────────────────────────────────┘
      └─────────┬────────────┘
                │
                ▼
┌────────────────────────────────────────────────────┐
│ Ranking Layer (ML Model)                           │
│ - XGBoost / LightGBM                              │
│ - DLRM (Deep Learning Ranker)                     │
│ - Feature Cross Models                             │
└──────────────┬─────────────────────────────────────┘
               │
               ▼
┌────────────────────────────────────────────────────┐
│ Sequential / Contextual Re-ranking (Optional)      │
│ - Transformer (SASRec / BERT4Rec)                  │
│ - Contextual Bandits (Exploration)                │
└──────────────┬─────────────────────────────────────┘
               │
               ▼
        ┌──────────────────────┐
        │ Final Recommendations│
        └─────────┬────────────┘
                  │
                  ▼
         ┌─────────────────────┐
         │   User Response     │
         └─────────────────────┘

---

🔁 Offline + Real-Time ML Pipeline (Critical FAANG Part)

                    ┌──────────────────────────┐
                    │   Event Collection       │
                    │ (Clicks / Views / Buy)   │
                    └──────────┬───────────────┘
                               │
                               ▼
                    ┌──────────────────────────┐
                    │ Apache Kafka Stream      │
                    │ (Event Ingestion)        │
                    └──────────┬───────────────┘
                               │
            ┌──────────────────┴──────────────────┐
            ▼                                     ▼
┌──────────────────────┐              ┌──────────────────────┐
│ Real-time Processing  │              │ Offline Storage       │
│ Apache Flink          │              │ Data Lake (S3/HDFS)   │
│ Feature Aggregation    │              └─────────┬────────────┘
└──────────┬────────────┘                        │
           │                                      ▼
           ▼                         ┌──────────────────────────┐
┌──────────────────────┐           │ ML Training Pipeline      │
│ Feature Store        │           │ PyTorch / TensorFlow      │
│ (Feast)              │           │ XGBoost / DLRM           │
└──────────┬───────────┘           └──────────┬───────────────┘
           │                                  │
           ▼                                  ▼
┌──────────────────────┐        ┌──────────────────────────────┐
│ Online Serving       │        │ Model Registry               │
│ (Low latency APIs)   │        │ (Versioning / Deployment)    │
└──────────────────────┘        └──────────────────────────────┘

---

⚙️ What Makes This “FAANG-Grade”

1. Two-Stage ML System

• Candidate Generation → Two-Tower Model
• Ranking → XGBoost / DLRM

Why?

• Scales to billions of items
• Separates recall vs precision problems

---

2. Vector Search Layer

• FAISS / Milvus / ScaNN

Why?

• Replaces O(N) search with O(log N) approximate search
• Enables real-time retrieval

---

3. Real-Time Learning Loop

• Kafka + Flink

Why?

• User behavior changes quickly
• System adapts in minutes

---

4. Feature Store

• Feast

Why?

• Avoid training-serving skew
• Centralized feature management

---

5. Ranking ML Models

• XGBoost (fast, strong baseline)
• DLRM (deep learning ranking)

Why both?

• Trees = interpretability
• Deep models = complexity capture

---

6. Exploration Layer

• Contextual Bandits (Thompson Sampling)

Why?

• Avoid filter bubbles
• Introduce novelty

---

🎯 How to Present This in Interview (Key Line)

“I designed a two-stage ML recommendation system where a Two-Tower model generates candidates using FAISS-based vector search, and a ranking model like XGBoost or DLRM scores them using real-time features from a Kafka-Flink pipeline. A feature store like Feast ensures consistency, while Redis enables low-latency serving. Finally, contextual bandits introduce exploration to balance relevance and diversity.”

30-Second Whiteboard Diagram (MEMORISE THIS)

               ┌──────────────┐
               │   CLIENT     │
               │ Web / Mobile │
               └──────┬───────┘
                      │
                      ▼
             ┌─────────────────┐
             │ API GATEWAY     │
             └──────┬──────────┘
                    │
                    ▼
     ┌────────────────────────────────┐
     │ RECOMMENDATION SERVICE         │
     │ (Online Serving <100ms)        │
     └──────┬───────────────┬────────┘
            │               │
            ▼               ▼
 ┌────────────────┐   ┌──────────────┐
 │ TWO-TOWER NN   │   │ REDIS CACHE  │
 │ (Candidate Gen)│   └──────────────┘
 └──────┬─────────┘
        │
        ▼
 ┌────────────────┐
 │ FAISS / ANN    │
 │ Vector Search  │
 └──────┬─────────┘
        │
        ▼
 ┌────────────────┐
 │ 1000 CANDIDATES│
 └──────┬─────────┘
        │
        ▼
 ┌────────────────┐
 │ RANKING MODEL  │
 │ XGBoost / DLRM │
 └──────┬─────────┘
        │
        ▼
 ┌────────────────┐
 │ FINAL TOP K    │
 └──────┬─────────┘
        │
        ▼
     USER FEED

---

🔁 Side Pipeline (draw this on the LEFT or BOTTOM)

EVENTS (click, view, buy)
        │
        ▼
   KAFKA STREAM
        │
        ▼
   FLINK (real-time features)
        │
        ▼
 FEATURE STORE (Feast)
        │
        ├───────────────┐
        ▼               ▼
 OFFLINE DATA     MODEL TRAINING
 (S3 / Hive)      (PyTorch / XGBoost)
                        │
                        ▼
                 MODEL DEPLOYMENT

---

⚡ Exploration (add small note bubble)

Exploration Layer:
Contextual Bandits (Thompson Sampling)
→ inject 5–10% new content

---

🧠 30-Second Memory Trick (IMPORTANT)

Just memorise this chain:

MAIN FLOW:

CLIENT → API → TWO-TOWER → FAISS → RANKING → OUTPUT

DATA FLOW:

EVENTS → KAFKA → FLINK → FEATURE STORE → TRAINING

---

🎯 How to Draw in Interview (exact order)

1. Draw Client → API Gateway
2. Draw Recommendation Service box
3. Split into:
   • Two-Tower (left)
   • Redis (right)
4. Arrow → FAISS
5. Arrow → Ranking (XGBoost)
6. Arrow → Final Feed
7. Add bottom pipeline (Kafka → Flink → Feature Store)

---

💡 One-line recall mantra

“Two-Tower for retrieval, FAISS for search, XGBoost for ranking, Kafka + Flink for real-time features.”