Designing YouTube: Video Upload, Transcoding, and Streaming at Scale

Why Study YouTube?

YouTube ingests over 500 hours of video every minute from creators worldwide and serves over 2 billion logged-in users monthly. Unlike a licensed-content platform, YouTube must accept, process, and safely publish anything anyone uploads - from a 30-second phone clip to a multi-hour 4K livestream recording. That upload-side problem, not just playback, is what makes YouTube a distinct system design case study.

Requirements Analysis

Functional Requirements

1. Video Upload: Creators upload videos up to several hours long and tens of gigabytes in size
2. Transcoding: Every upload is converted into multiple resolutions and bitrates
3. Playback: Viewers stream video that adapts to their device and network conditions
4. View Counting: Track how many times a video has been watched
5. Metadata Management: Titles, descriptions, thumbnails, captions
6. Content Availability: Videos become watchable shortly after upload completes

Non-Functional Requirements

High-Level Architecture

Resumable, Chunked Upload

Why a Single HTTP Request Doesn't Work

A 4K video can easily be 20-50 GB. Uploading that as one HTTP PUT/POST has real problems at scale:

• Network interruptions are common. Mobile networks, hotel WiFi, and long-lived TCP connections fail. A single dropped connection at byte 19GB means restarting from zero.
• Server-side buffering. A naive server that buffers the whole request body before processing needs enormous memory/disk per concurrent upload.
• No progress visibility. Users have no way to resume or see partial progress with a monolithic request.
• Timeouts. Load balancers and proxies often cap request duration; a multi-hour upload on a slow connection can exceed those limits.

Chunked Upload Protocol

The client splits the file into fixed-size chunks (commonly 8-16 MB), uploads them independently, and each chunk is acknowledged and persisted before the next is sent. If the connection drops, the client asks the server which chunks it already has and resumes from there instead of restarting. This is the same principle behind the resumable upload protocol YouTube's own API documents and behind the TUS open protocol.

Chunked Upload Design Decisions

Async Transcoding Pipeline

Why Transcoding Must Be Asynchronous

A single uploaded video needs to become many output files: several resolutions (144p through 4K), several bitrates per resolution, multiple codecs (H.264 for compatibility, VP9/AV1 for efficiency), and audio tracks. Transcoding a long video can take longer than the upload itself. Doing this synchronously would mean holding an HTTP connection open for many minutes and tying up upload-tier capacity with CPU-bound work - two very different scaling profiles that shouldn't share infrastructure.

The Pipeline

Each resolution is transcoded by an independent worker pulling from a queue, so a slow 4K job never blocks a fast 144p job, and the system can prioritize low-resolution renditions first so the video becomes watchable (on smaller screens or slow connections) before every rendition finishes.

Transcoding Job Priorities

Adaptive Bitrate Streaming (HLS/DASH)

How the Player Chooses Quality

Adaptive bitrate (ABR) streaming works by splitting each rendition into short segments (typically 2-10 seconds) and publishing a manifest file that lists every available rendition and where its segments live. The player - not the server - decides which rendition to request for each segment, based on measured download throughput and buffer health.

HLS vs DASH

YouTube supports both, serving the appropriate manifest based on the requesting device and player. Both formats express the same idea: a hierarchy of manifests (master → per-rendition playlist) pointing at small, independently-fetchable segments.

CDN Delivery and Origin Shielding

Popular segments (viral videos) get requested by thousands of edge locations simultaneously. Without an origin shield, every edge cache miss goes straight to origin storage, multiplying load on the same hot object. An origin shield sits between edges and origin: it's a smaller set of regional caches that de-duplicate concurrent misses (many edges requesting the same segment collapse into a single origin fetch) and absorb the bulk of repeat traffic, so origin storage sees a small fraction of total request volume.

Database Schema

Videos Table

CREATE TABLE videos (
    id BIGINT PRIMARY KEY,
    uploader_id BIGINT NOT NULL,
    title VARCHAR(200) NOT NULL,
    description TEXT,
    status VARCHAR(20) NOT NULL DEFAULT 'processing',
    -- processing | ready | failed | removed
    duration_seconds INT,
    source_file_key VARCHAR(500) NOT NULL,
    thumbnail_url VARCHAR(500),
    visibility VARCHAR(20) NOT NULL DEFAULT 'private',
    -- private | unlisted | public
    created_at TIMESTAMP DEFAULT NOW(),
    published_at TIMESTAMP
);
 
CREATE INDEX idx_uploader_id ON videos(uploader_id, created_at DESC);
CREATE INDEX idx_status ON videos(status);

Renditions Table

CREATE TABLE renditions (
    id BIGINT PRIMARY KEY,
    video_id BIGINT NOT NULL REFERENCES videos(id),
    resolution VARCHAR(10) NOT NULL,   -- e.g. '1080p'
    codec VARCHAR(20) NOT NULL,        -- e.g. 'h264', 'vp9', 'av1'
    bitrate_kbps INT NOT NULL,
    manifest_path VARCHAR(500) NOT NULL,
    segment_prefix VARCHAR(500) NOT NULL,
    status VARCHAR(20) NOT NULL DEFAULT 'queued',
    -- queued | processing | ready | failed
    created_at TIMESTAMP DEFAULT NOW()
);
 
CREATE INDEX idx_video_renditions ON renditions(video_id, status);
CREATE UNIQUE INDEX idx_video_res_codec ON renditions(video_id, resolution, codec);

View Counts Table

CREATE TABLE view_counts (
    video_id BIGINT PRIMARY KEY REFERENCES videos(id),
    approximate_views BIGINT NOT NULL DEFAULT 0,
    last_batch_applied_at TIMESTAMP,
    updated_at TIMESTAMP DEFAULT NOW()
);
 
-- Raw view events land here first, batched and aggregated later
CREATE TABLE view_events_buffer (
    id BIGINT PRIMARY KEY,
    video_id BIGINT NOT NULL,
    viewer_session_hash VARCHAR(64) NOT NULL,
    watched_seconds INT NOT NULL,
    event_time TIMESTAMP NOT NULL
);
 
CREATE INDEX idx_events_video_time ON view_events_buffer(video_id, event_time);

The Approximate View Counter Problem

Why You Can't Just UPDATE ... SET views = views + 1

A synchronous increment per view sounds simple, but at YouTube's scale it breaks down fast:

• Write contention. A viral video can be watched thousands of times per second. Every view hitting the same row as a synchronous UPDATE creates a hot-row lock contention bottleneck.
• Fraud and replay. Naive counting is trivially gameable - refreshing a page, bot traffic, or a single viewer replaying a video shouldn't all count as new views. A raw increment can't apply "does this count as a real view" logic (e.g., minimum watch duration) without extra reads on the hot path.
• Playback latency coupling. If starting playback requires a synchronous write to a view counter, video start time now depends on database write latency - exactly the kind of coupling adaptive streaming and CDN delivery are designed to avoid.

Batched, Eventually-Consistent Counting

Instead of incrementing on every view, the player emits a lightweight, asynchronous event once a view is judged "real" (e.g., a minimum watch threshold is crossed). Events land in a stream/log rather than the database directly. A batch aggregation job periodically deduplicates and validates events, then applies a single bulk increment per video per batch window - collapsing potentially thousands of individual events into one write.

This is deliberately similar in spirit to Instagram's like/comment counters (also asynchronous and eventually consistent) but distinct in mechanism: Instagram's counters update relatively quickly per-action against a social graph, while view counters additionally need watch-time validation and heavier anti-fraud filtering before a "view" is even accepted into the aggregation batch - so the displayed number can lag the true count by design, and briefly "hovers" during viral spikes rather than climbing in real time.

Scaling Challenges & Solutions

Key Takeaways

1. Chunk uploads, don't stream them as one request: Independent, acknowledged chunks make multi-gigabyte uploads resumable and memory-safe on the server.
2. Decouple upload from transcoding: A queue between them lets CPU-bound transcoding scale independently of upload-tier capacity.
3. Publish incrementally: Don't wait for every rendition - make the video watchable as soon as the lowest usable rendition is ready.
4. Let the client choose quality: Adaptive bitrate streaming works because the player, closest to the actual network conditions, makes the switching decision - not the server.
5. Shield your origin: A caching tier between edges and origin storage is what keeps a viral spike from taking down the storage layer.
6. Approximate counters aren't a shortcut, they're a requirement: At sufficient write volume, eventual consistency for counters is the only design that scales - and it doubles as a natural point to filter fraud.

Follow-Up Questions to Consider

1. How would you support live streaming instead of pre-recorded video, where segments must be generated and published in near real-time?
2. How would you implement content moderation (copyright detection, policy violations) without delaying publish time for compliant creators?
3. How would you handle re-transcoding an already-published video (e.g., adding a new codec like AV1 to old content) without downtime?
4. How would you design closed captioning, including auto-generated captions for videos in many languages?
5. How would you rate-limit or throttle a single uploader who publishes an unusually large volume of content in a short window?