Advanced SQL: Joins, Window Functions, CTEs, and Pagination
A staff-engineer guide to joins, aggregation, window functions, CTEs, views, and production-safe pagination patterns.
Advanced SQL and Querying
Once your schema is sound, the day-to-day work of a backend engineer is writing queries that are both correct and fast. This guide goes past basic SELECTs into the query patterns that show up constantly in production: every join variant, aggregation with GROUP BY/HAVING, window functions, CTEs, views, and the pagination strategies that actually scale past page one.
1. Joins: The Complete Picture
A join combines rows from two or more tables based on a related column. Choosing the wrong join type is one of the most common sources of subtly wrong data in backend systems — usually silently dropped rows.
INNER JOIN — only matching rows
SELECT o.id AS order_id, c.name AS customer_name, o.total_cents
FROM orders o
INNER JOIN customers c ON c.id = o.customer_id;
-- Orders with no matching customer (shouldn't happen with a FK, but could
-- with orphaned data) are silently excluded.LEFT JOIN — keep every row from the left table
-- All customers, including those who have never placed an order
SELECT c.id, c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
GROUP BY c.id, c.name;
-- Customers with zero orders still appear, with order_count = 0A LEFT JOIN followed by a WHERE clause that filters on the right table's column silently turns it back into an INNER JOIN. WHERE o.status = 'CONFIRMED' on the query above would drop every customer with no orders at all, because NULL = 'CONFIRMED' evaluates to UNKNOWN, not TRUE. If you need to filter the right side while still keeping unmatched left rows, move the condition into the ON clause: LEFT JOIN orders o ON o.customer_id = c.id AND o.status = 'CONFIRMED'.
RIGHT JOIN and FULL OUTER JOIN
-- RIGHT JOIN: rarely used in practice — equivalent to swapping table order
-- and using LEFT JOIN. Most style guides ban it for readability.
SELECT c.name, o.id
FROM orders o
RIGHT JOIN customers c ON c.id = o.customer_id;
-- FULL OUTER JOIN: not supported natively in MySQL, but standard in PostgreSQL
-- PostgreSQL:
-- SELECT c.id, c.name, o.id AS order_id
-- FROM customers c
-- FULL OUTER JOIN orders o ON o.customer_id = c.id;
-- MySQL equivalent via UNION of LEFT and RIGHT JOIN:
SELECT c.id, c.name, o.id AS order_id
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
UNION
SELECT c.id, c.name, o.id AS order_id
FROM customers c
RIGHT JOIN orders o ON o.customer_id = c.id;Dialect gap: MySQL has no native FULL OUTER JOIN — you have to emulate it with a UNION of LEFT JOIN and RIGHT JOIN. PostgreSQL supports it directly. This is one of the more common dialect surprises when porting queries between the two.
SELF JOIN — a table joined to itself
-- Find every employee alongside their manager's name
SELECT e.name AS employee_name, m.name AS manager_name
FROM employees e
LEFT JOIN employees m ON e.manager_id = m.id;CROSS JOIN — cartesian product
-- Generate every (size, color) combination for a product configurator
SELECT s.size_label, c.color_name
FROM sizes s
CROSS JOIN colors c;
-- 5 sizes x 8 colors = 40 rows, deliberatelyAn accidental cross join is one of the most common query bugs — it happens when you list two tables in FROM (comma syntax) and forget the join condition, or when an ON clause typo makes every row match every row. On tables with millions of rows each, this doesn't just return wrong data — it can produce a result set so large it takes down the database connection or fills disk with a temp table. Always use explicit JOIN ... ON syntax and never the legacy comma-join style.
Join type comparison
| Join type | Left unmatched rows | Right unmatched rows | Use case |
|---|---|---|---|
INNER JOIN | Dropped | Dropped | Only rows that exist on both sides matter |
LEFT JOIN | Kept (NULLs for right cols) | Dropped | "All X, with optional Y" — e.g., all customers with their orders |
RIGHT JOIN | Dropped | Kept (NULLs for left cols) | Rare — prefer flipping table order and using LEFT JOIN |
FULL OUTER JOIN | Kept | Kept | Reconciliation / diffing two datasets |
CROSS JOIN | N/A | N/A | Deliberate combinatorial generation |
SELF JOIN | Depends on join type used | Depends on join type used | Hierarchies, comparing rows within the same table |
2. Aggregation: GROUP BY and HAVING
-- Total revenue and order count per customer
SELECT
customer_id,
COUNT(*) AS order_count,
SUM(total_cents) AS total_revenue_cents,
AVG(total_cents) AS avg_order_cents,
MAX(total_cents) AS largest_order_cents
FROM orders
WHERE status = 'CONFIRMED'
GROUP BY customer_id
HAVING COUNT(*) >= 5 -- filters GROUPS, evaluated after aggregation
ORDER BY total_revenue_cents DESC
LIMIT 20;Rule of thumb: WHERE filters rows before grouping (can't reference aggregates); HAVING filters groups after aggregation (can reference aggregates). Filtering on a non-aggregated condition in HAVING when it could go in WHERE is a common performance mistake — it forces the database to group rows it could have discarded earlier.
-- Multi-column grouping
SELECT
DATE(created_at) AS order_date,
status,
COUNT(*) AS count
FROM orders
GROUP BY DATE(created_at), status
ORDER BY order_date DESC, status;In MySQL's default (ONLY_FULL_GROUP_BY) SQL mode, every non-aggregated column in SELECT must appear in GROUP BY, or be functionally dependent on grouped columns. Older MySQL configs sometimes disable this mode, which silently permits nondeterministic column values in aggregated queries — a value picked essentially at random from one of the grouped rows. Never rely on that behavior; always be explicit about which row's value you want (typically via a window function or a subquery).
3. Window Functions
Window functions compute a value across a set of rows related to the current row without collapsing the result into one row per group, unlike GROUP BY. This is the single most powerful addition to modern SQL for analytics-style backend queries.
Ranking functions
-- Rank customers by revenue, per region
SELECT
region,
customer_id,
total_revenue_cents,
ROW_NUMBER() OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS row_num,
RANK() OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS rnk,
DENSE_RANK() OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS dense_rnk
FROM customer_revenue;| Function | Ties behavior | Gaps after ties |
|---|---|---|
ROW_NUMBER() | No ties — arbitrary but stable tiebreak | N/A, always sequential |
RANK() | Ties get the same rank | Next rank skips (1, 2, 2, 4) |
DENSE_RANK() | Ties get the same rank | No gap (1, 2, 2, 3) |
Top-N per group — the killer use case
-- Top 3 highest-value orders per customer (a query that's painful without window functions)
WITH ranked_orders AS (
SELECT
id, customer_id, total_cents,
ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total_cents DESC) AS rn
FROM orders
)
SELECT id, customer_id, total_cents
FROM ranked_orders
WHERE rn <= 3;Running totals and moving averages
-- Running total of daily revenue
SELECT
order_date,
daily_revenue_cents,
SUM(daily_revenue_cents) OVER (ORDER BY order_date) AS running_total_cents,
AVG(daily_revenue_cents) OVER (
ORDER BY order_date
ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
) AS trailing_7day_avg_cents
FROM daily_revenue;LAG and LEAD — comparing to neighboring rows
-- Month-over-month revenue change, per customer
SELECT
customer_id,
order_month,
revenue_cents,
LAG(revenue_cents) OVER (PARTITION BY customer_id ORDER BY order_month) AS prev_month_revenue,
revenue_cents - LAG(revenue_cents) OVER (PARTITION BY customer_id ORDER BY order_month) AS mom_change
FROM monthly_customer_revenue;Window functions execute after WHERE, GROUP BY, and HAVING, but before ORDER BY and LIMIT in the logical execution order — which is why you cannot reference a window function's alias in the same query's WHERE clause. If you need to filter on a window function's result, wrap the query in a CTE or subquery (as in the top-N-per-group example above) and filter in the outer query.
4. Common Table Expressions (CTEs)
A CTE (WITH clause) names a temporary result set for use within a single query. It exists purely for readability and structure in most databases — it does not automatically imply materialization.
WITH high_value_customers AS (
SELECT customer_id, SUM(total_cents) AS total_spent
FROM orders
WHERE status = 'CONFIRMED'
GROUP BY customer_id
HAVING SUM(total_cents) > 1000000
),
recent_activity AS (
SELECT customer_id, MAX(created_at) AS last_order_at
FROM orders
GROUP BY customer_id
)
SELECT
hvc.customer_id,
hvc.total_spent,
ra.last_order_at
FROM high_value_customers hvc
JOIN recent_activity ra ON ra.customer_id = hvc.customer_id
ORDER BY hvc.total_spent DESC;Recursive CTEs — traversing hierarchies
-- Walk an org chart: find every employee under a given manager, at any depth
WITH RECURSIVE org_chart AS (
-- Anchor member: the starting manager
SELECT id, name, manager_id, 0 AS depth
FROM employees
WHERE id = 1
UNION ALL
-- Recursive member: employees reporting to anyone already in the result
SELECT e.id, e.name, e.manager_id, oc.depth + 1
FROM employees e
JOIN org_chart oc ON e.manager_id = oc.id
)
SELECT * FROM org_chart ORDER BY depth, name;Recursive CTEs need a termination condition — the recursive member must eventually stop producing new rows (e.g., reaching leaf nodes with no reports). A cyclic graph (accidentally or via bad data — an employee who is their own indirect manager) can cause infinite recursion. MySQL and PostgreSQL both support a recursion depth limit (cte_max_recursion_depth in MySQL) as a safety net, but don't rely on it as your primary defense — validate the underlying data has no cycles.
CTEs are primarily a readability tool — think of them as naming intermediate steps of a query the way you'd extract a variable in application code. In PostgreSQL 12+ and MySQL 8+, the optimizer can inline non-recursive CTEs into the outer query (unlike older PostgreSQL versions, which always materialized them as an optimization fence). Don't assume a CTE forces materialization — check your engine's version and, if performance matters, verify with EXPLAIN.
5. Views
A view is a stored, named SELECT query — a virtual table that doesn't store data itself (unless it's a materialized view).
-- Standard view: re-executed on every query against it
CREATE VIEW active_customer_summary AS
SELECT
c.id,
c.name,
COUNT(o.id) AS order_count,
COALESCE(SUM(o.total_cents), 0) AS lifetime_value_cents
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id AND o.status = 'CONFIRMED'
GROUP BY c.id, c.name;
-- Query it like a table
SELECT * FROM active_customer_summary WHERE lifetime_value_cents > 500000;-- Materialized view (PostgreSQL): computed once, stored, refreshed on demand
-- CREATE MATERIALIZED VIEW active_customer_summary_mat AS
-- SELECT ... (same query as above);
--
-- REFRESH MATERIALIZED VIEW CONCURRENTLY active_customer_summary_mat;| Aspect | Standard view | Materialized view |
|---|---|---|
| Storage | None — query re-runs every time | Stores computed result on disk |
| Freshness | Always current | Stale until explicitly refreshed |
| Read performance | Same as the underlying query | Fast — pre-computed |
| Write overhead | None | Refresh cost, run on a schedule or trigger |
| MySQL support | Yes | No native support (emulate with a real table + scheduled refresh) |
| PostgreSQL support | Yes | Yes, native |
Views are useful for encapsulating complex joins behind a stable, simple interface — e.g., giving a reporting team a SELECT * FROM active_customer_summary instead of exposing raw schema internals they could query incorrectly. But a view is not a performance optimization by itself; a standard view's query still runs in full every time. If the underlying query is expensive and doesn't need to be real-time, that's when a materialized view (or a denormalized summary table, refreshed on a schedule) actually helps.
6. Pagination Patterns
Pagination looks trivial until your table has tens of millions of rows and OFFSET pagination starts timing out on page 500.
OFFSET/LIMIT — simple but doesn't scale
-- Page 3, 20 rows per page
SELECT id, name, created_at
FROM orders
ORDER BY created_at DESC
LIMIT 20 OFFSET 40;OFFSET pagination gets linearly slower as the offset grows — the database still has to scan and discard every row before the offset, even though it only returns 20 of them. OFFSET 1000000 LIMIT 20 on a large table can mean scanning a million rows to throw them away. It's also unstable under concurrent writes: if a row is inserted or deleted between page loads, rows can shift and get skipped or duplicated across pages.
Keyset (cursor) pagination — the production-safe pattern
-- First page
SELECT id, name, created_at
FROM orders
ORDER BY created_at DESC, id DESC
LIMIT 20;
-- Next page: use the last row's (created_at, id) as the cursor
SELECT id, name, created_at
FROM orders
WHERE (created_at, id) < ('2026-08-10 14:32:00', 918273)
ORDER BY created_at DESC, id DESC
LIMIT 20;| Aspect | OFFSET/LIMIT | Keyset (cursor) |
|---|---|---|
| Performance at large offsets | Degrades linearly | Constant — always an index seek |
| Stability under concurrent writes | Rows can shift, duplicate, or skip | Stable — cursor is a real value, not a position |
| Jump to arbitrary page N | Trivial | Not directly supported (sequential only) |
| Implementation complexity | Trivial | Requires a stable, indexed sort key (often a tuple) |
| Best for | Small tables, admin UIs needing page numbers | Infinite scroll, APIs, large tables |
Always use a tie-breaking column in the sort key for keyset pagination — created_at alone isn't unique enough (two rows can share a timestamp), so pair it with the primary key: ORDER BY created_at DESC, id DESC and cursor on (created_at, id). Without the tie-breaker, rows with duplicate values in the primary sort column can be skipped or repeated across pages.
7. Subqueries: Correlated vs Uncorrelated
-- Uncorrelated: inner query runs once, independent of the outer query
SELECT name FROM products
WHERE id IN (
SELECT product_id FROM order_items GROUP BY product_id HAVING COUNT(*) > 100
);
-- Correlated: inner query re-runs once per outer row, referencing the outer row
SELECT c.name,
(SELECT COUNT(*) FROM orders o WHERE o.customer_id = c.id) AS order_count
FROM customers c;A correlated subquery in the SELECT list runs once per outer row — on a table with 100,000 customers, that's 100,000 subquery executions. Most of the time this is better expressed as a LEFT JOIN with GROUP BY, or a window function, both of which the query planner can usually execute in a single pass. Reach for a correlated subquery only when the join/window alternative is genuinely more awkward, and always check the EXPLAIN plan before shipping one against a large table.
Key takeaways
LEFT JOIN+ aWHEREfilter on the right table silently degrades into anINNER JOIN— put such conditions in theONclause instead.- MySQL has no native
FULL OUTER JOIN; emulate it with aUNIONofLEFT JOINandRIGHT JOIN. HAVINGfilters groups after aggregation;WHEREfilters rows before — putting a filterable condition in the wrong one costs performance.- Window functions (
ROW_NUMBER,RANK,LAG/LEAD,SUM() OVER) solve "top-N per group" and running-total problems that are painful with plainGROUP BY. - Recursive CTEs are the standard tool for hierarchy/graph traversal in SQL — always confirm the underlying data has no cycles.
- Views encapsulate query complexity but aren't a performance optimization; materialized views (or scheduled summary tables) are, at the cost of freshness.
OFFSETpagination degrades linearly with page depth and is unstable under concurrent writes — use keyset (cursor) pagination for any large or high-write table.- Correlated subqueries execute once per outer row; prefer a
JOINor window function when the same result can be computed in one pass.
Interview Questions
- Explain the difference between
INNER JOIN,LEFT JOIN, andFULL OUTER JOINwith an example for each. - Why does adding a
WHEREcondition on the right-hand table of aLEFT JOINsometimes silently turn it into anINNER JOIN? How do you fix it? - How would you find the top 3 highest-paid employees per department using SQL?
- What is the difference between
RANK(),DENSE_RANK(), andROW_NUMBER()? - How do window functions differ from
GROUP BYin terms of the number of output rows? - What is a recursive CTE, and what safeguard do you need against infinite recursion?
- What is the difference between a standard view and a materialized view? Which one improves query performance?
- Why does
OFFSET-based pagination get slower on later pages, and how does keyset pagination avoid that? - Why is a single sort column often insufficient for keyset pagination, and what do you add to fix it?
- What is a correlated subquery, and why can it be a performance problem at scale?
- When would you use a
CROSS JOINdeliberately? How can aCROSS JOINhappen accidentally? - What does MySQL's
ONLY_FULL_GROUP_BYmode enforce, and what happens if it's disabled? - Write a query using
LAG()to compute the month-over-month change in a metric.