02-database-design-sql

Advanced SQL: Joins, Window Functions, CTEs, and Pagination

A staff-engineer guide to joins, aggregation, window functions, CTEs, views, and production-safe pagination patterns.

August 14, 2026
backend-engineerjoinswindow-functionscteaggregationviews

Advanced SQL and Querying

Once your schema is sound, the day-to-day work of a backend engineer is writing queries that are both correct and fast. This guide goes past basic SELECTs into the query patterns that show up constantly in production: every join variant, aggregation with GROUP BY/HAVING, window functions, CTEs, views, and the pagination strategies that actually scale past page one.


1. Joins: The Complete Picture

A join combines rows from two or more tables based on a related column. Choosing the wrong join type is one of the most common sources of subtly wrong data in backend systems — usually silently dropped rows.

INNER JOIN — only matching rows

sql
SELECT o.id AS order_id, c.name AS customer_name, o.total_cents
FROM orders o
INNER JOIN customers c ON c.id = o.customer_id;
-- Orders with no matching customer (shouldn't happen with a FK, but could
-- with orphaned data) are silently excluded.

LEFT JOIN — keep every row from the left table

sql
-- All customers, including those who have never placed an order
SELECT c.id, c.name, COUNT(o.id) AS order_count
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
GROUP BY c.id, c.name;
-- Customers with zero orders still appear, with order_count = 0
⚠️

A LEFT JOIN followed by a WHERE clause that filters on the right table's column silently turns it back into an INNER JOIN. WHERE o.status = 'CONFIRMED' on the query above would drop every customer with no orders at all, because NULL = 'CONFIRMED' evaluates to UNKNOWN, not TRUE. If you need to filter the right side while still keeping unmatched left rows, move the condition into the ON clause: LEFT JOIN orders o ON o.customer_id = c.id AND o.status = 'CONFIRMED'.

RIGHT JOIN and FULL OUTER JOIN

sql
-- RIGHT JOIN: rarely used in practice — equivalent to swapping table order
-- and using LEFT JOIN. Most style guides ban it for readability.
SELECT c.name, o.id
FROM orders o
RIGHT JOIN customers c ON c.id = o.customer_id;
 
-- FULL OUTER JOIN: not supported natively in MySQL, but standard in PostgreSQL
-- PostgreSQL:
-- SELECT c.id, c.name, o.id AS order_id
-- FROM customers c
-- FULL OUTER JOIN orders o ON o.customer_id = c.id;
 
-- MySQL equivalent via UNION of LEFT and RIGHT JOIN:
SELECT c.id, c.name, o.id AS order_id
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id
UNION
SELECT c.id, c.name, o.id AS order_id
FROM customers c
RIGHT JOIN orders o ON o.customer_id = c.id;
💡

Dialect gap: MySQL has no native FULL OUTER JOIN — you have to emulate it with a UNION of LEFT JOIN and RIGHT JOIN. PostgreSQL supports it directly. This is one of the more common dialect surprises when porting queries between the two.

SELF JOIN — a table joined to itself

sql
-- Find every employee alongside their manager's name
SELECT e.name AS employee_name, m.name AS manager_name
FROM employees e
LEFT JOIN employees m ON e.manager_id = m.id;

CROSS JOIN — cartesian product

sql
-- Generate every (size, color) combination for a product configurator
SELECT s.size_label, c.color_name
FROM sizes s
CROSS JOIN colors c;
-- 5 sizes x 8 colors = 40 rows, deliberately
🚨

An accidental cross join is one of the most common query bugs — it happens when you list two tables in FROM (comma syntax) and forget the join condition, or when an ON clause typo makes every row match every row. On tables with millions of rows each, this doesn't just return wrong data — it can produce a result set so large it takes down the database connection or fills disk with a temp table. Always use explicit JOIN ... ON syntax and never the legacy comma-join style.

Join type comparison

Join typeLeft unmatched rowsRight unmatched rowsUse case
INNER JOINDroppedDroppedOnly rows that exist on both sides matter
LEFT JOINKept (NULLs for right cols)Dropped"All X, with optional Y" — e.g., all customers with their orders
RIGHT JOINDroppedKept (NULLs for left cols)Rare — prefer flipping table order and using LEFT JOIN
FULL OUTER JOINKeptKeptReconciliation / diffing two datasets
CROSS JOINN/AN/ADeliberate combinatorial generation
SELF JOINDepends on join type usedDepends on join type usedHierarchies, comparing rows within the same table

2. Aggregation: GROUP BY and HAVING

sql
-- Total revenue and order count per customer
SELECT
    customer_id,
    COUNT(*)              AS order_count,
    SUM(total_cents)       AS total_revenue_cents,
    AVG(total_cents)       AS avg_order_cents,
    MAX(total_cents)       AS largest_order_cents
FROM orders
WHERE status = 'CONFIRMED'
GROUP BY customer_id
HAVING COUNT(*) >= 5           -- filters GROUPS, evaluated after aggregation
ORDER BY total_revenue_cents DESC
LIMIT 20;

Rule of thumb: WHERE filters rows before grouping (can't reference aggregates); HAVING filters groups after aggregation (can reference aggregates). Filtering on a non-aggregated condition in HAVING when it could go in WHERE is a common performance mistake — it forces the database to group rows it could have discarded earlier.

sql
-- Multi-column grouping
SELECT
    DATE(created_at) AS order_date,
    status,
    COUNT(*) AS count
FROM orders
GROUP BY DATE(created_at), status
ORDER BY order_date DESC, status;
⚠️

In MySQL's default (ONLY_FULL_GROUP_BY) SQL mode, every non-aggregated column in SELECT must appear in GROUP BY, or be functionally dependent on grouped columns. Older MySQL configs sometimes disable this mode, which silently permits nondeterministic column values in aggregated queries — a value picked essentially at random from one of the grouped rows. Never rely on that behavior; always be explicit about which row's value you want (typically via a window function or a subquery).


3. Window Functions

Window functions compute a value across a set of rows related to the current row without collapsing the result into one row per group, unlike GROUP BY. This is the single most powerful addition to modern SQL for analytics-style backend queries.

Ranking functions

sql
-- Rank customers by revenue, per region
SELECT
    region,
    customer_id,
    total_revenue_cents,
    ROW_NUMBER() OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS row_num,
    RANK()       OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS rnk,
    DENSE_RANK() OVER (PARTITION BY region ORDER BY total_revenue_cents DESC) AS dense_rnk
FROM customer_revenue;
FunctionTies behaviorGaps after ties
ROW_NUMBER()No ties — arbitrary but stable tiebreakN/A, always sequential
RANK()Ties get the same rankNext rank skips (1, 2, 2, 4)
DENSE_RANK()Ties get the same rankNo gap (1, 2, 2, 3)

Top-N per group — the killer use case

sql
-- Top 3 highest-value orders per customer (a query that's painful without window functions)
WITH ranked_orders AS (
    SELECT
        id, customer_id, total_cents,
        ROW_NUMBER() OVER (PARTITION BY customer_id ORDER BY total_cents DESC) AS rn
    FROM orders
)
SELECT id, customer_id, total_cents
FROM ranked_orders
WHERE rn <= 3;

Running totals and moving averages

sql
-- Running total of daily revenue
SELECT
    order_date,
    daily_revenue_cents,
    SUM(daily_revenue_cents) OVER (ORDER BY order_date) AS running_total_cents,
    AVG(daily_revenue_cents) OVER (
        ORDER BY order_date
        ROWS BETWEEN 6 PRECEDING AND CURRENT ROW
    ) AS trailing_7day_avg_cents
FROM daily_revenue;

LAG and LEAD — comparing to neighboring rows

sql
-- Month-over-month revenue change, per customer
SELECT
    customer_id,
    order_month,
    revenue_cents,
    LAG(revenue_cents) OVER (PARTITION BY customer_id ORDER BY order_month) AS prev_month_revenue,
    revenue_cents - LAG(revenue_cents) OVER (PARTITION BY customer_id ORDER BY order_month) AS mom_change
FROM monthly_customer_revenue;
💡

Window functions execute after WHERE, GROUP BY, and HAVING, but before ORDER BY and LIMIT in the logical execution order — which is why you cannot reference a window function's alias in the same query's WHERE clause. If you need to filter on a window function's result, wrap the query in a CTE or subquery (as in the top-N-per-group example above) and filter in the outer query.


4. Common Table Expressions (CTEs)

A CTE (WITH clause) names a temporary result set for use within a single query. It exists purely for readability and structure in most databases — it does not automatically imply materialization.

sql
WITH high_value_customers AS (
    SELECT customer_id, SUM(total_cents) AS total_spent
    FROM orders
    WHERE status = 'CONFIRMED'
    GROUP BY customer_id
    HAVING SUM(total_cents) > 1000000
),
recent_activity AS (
    SELECT customer_id, MAX(created_at) AS last_order_at
    FROM orders
    GROUP BY customer_id
)
SELECT
    hvc.customer_id,
    hvc.total_spent,
    ra.last_order_at
FROM high_value_customers hvc
JOIN recent_activity ra ON ra.customer_id = hvc.customer_id
ORDER BY hvc.total_spent DESC;

Recursive CTEs — traversing hierarchies

sql
-- Walk an org chart: find every employee under a given manager, at any depth
WITH RECURSIVE org_chart AS (
    -- Anchor member: the starting manager
    SELECT id, name, manager_id, 0 AS depth
    FROM employees
    WHERE id = 1
 
    UNION ALL
 
    -- Recursive member: employees reporting to anyone already in the result
    SELECT e.id, e.name, e.manager_id, oc.depth + 1
    FROM employees e
    JOIN org_chart oc ON e.manager_id = oc.id
)
SELECT * FROM org_chart ORDER BY depth, name;
⚠️

Recursive CTEs need a termination condition — the recursive member must eventually stop producing new rows (e.g., reaching leaf nodes with no reports). A cyclic graph (accidentally or via bad data — an employee who is their own indirect manager) can cause infinite recursion. MySQL and PostgreSQL both support a recursion depth limit (cte_max_recursion_depth in MySQL) as a safety net, but don't rely on it as your primary defense — validate the underlying data has no cycles.

CTEs are primarily a readability tool — think of them as naming intermediate steps of a query the way you'd extract a variable in application code. In PostgreSQL 12+ and MySQL 8+, the optimizer can inline non-recursive CTEs into the outer query (unlike older PostgreSQL versions, which always materialized them as an optimization fence). Don't assume a CTE forces materialization — check your engine's version and, if performance matters, verify with EXPLAIN.


5. Views

A view is a stored, named SELECT query — a virtual table that doesn't store data itself (unless it's a materialized view).

sql
-- Standard view: re-executed on every query against it
CREATE VIEW active_customer_summary AS
SELECT
    c.id,
    c.name,
    COUNT(o.id) AS order_count,
    COALESCE(SUM(o.total_cents), 0) AS lifetime_value_cents
FROM customers c
LEFT JOIN orders o ON o.customer_id = c.id AND o.status = 'CONFIRMED'
GROUP BY c.id, c.name;
 
-- Query it like a table
SELECT * FROM active_customer_summary WHERE lifetime_value_cents > 500000;
sql
-- Materialized view (PostgreSQL): computed once, stored, refreshed on demand
-- CREATE MATERIALIZED VIEW active_customer_summary_mat AS
-- SELECT ... (same query as above);
--
-- REFRESH MATERIALIZED VIEW CONCURRENTLY active_customer_summary_mat;
AspectStandard viewMaterialized view
StorageNone — query re-runs every timeStores computed result on disk
FreshnessAlways currentStale until explicitly refreshed
Read performanceSame as the underlying queryFast — pre-computed
Write overheadNoneRefresh cost, run on a schedule or trigger
MySQL supportYesNo native support (emulate with a real table + scheduled refresh)
PostgreSQL supportYesYes, native
💡

Views are useful for encapsulating complex joins behind a stable, simple interface — e.g., giving a reporting team a SELECT * FROM active_customer_summary instead of exposing raw schema internals they could query incorrectly. But a view is not a performance optimization by itself; a standard view's query still runs in full every time. If the underlying query is expensive and doesn't need to be real-time, that's when a materialized view (or a denormalized summary table, refreshed on a schedule) actually helps.


6. Pagination Patterns

Pagination looks trivial until your table has tens of millions of rows and OFFSET pagination starts timing out on page 500.

OFFSET/LIMIT — simple but doesn't scale

sql
-- Page 3, 20 rows per page
SELECT id, name, created_at
FROM orders
ORDER BY created_at DESC
LIMIT 20 OFFSET 40;
🚨

OFFSET pagination gets linearly slower as the offset grows — the database still has to scan and discard every row before the offset, even though it only returns 20 of them. OFFSET 1000000 LIMIT 20 on a large table can mean scanning a million rows to throw them away. It's also unstable under concurrent writes: if a row is inserted or deleted between page loads, rows can shift and get skipped or duplicated across pages.

Keyset (cursor) pagination — the production-safe pattern

sql
-- First page
SELECT id, name, created_at
FROM orders
ORDER BY created_at DESC, id DESC
LIMIT 20;
 
-- Next page: use the last row's (created_at, id) as the cursor
SELECT id, name, created_at
FROM orders
WHERE (created_at, id) < ('2026-08-10 14:32:00', 918273)
ORDER BY created_at DESC, id DESC
LIMIT 20;
AspectOFFSET/LIMITKeyset (cursor)
Performance at large offsetsDegrades linearlyConstant — always an index seek
Stability under concurrent writesRows can shift, duplicate, or skipStable — cursor is a real value, not a position
Jump to arbitrary page NTrivialNot directly supported (sequential only)
Implementation complexityTrivialRequires a stable, indexed sort key (often a tuple)
Best forSmall tables, admin UIs needing page numbersInfinite scroll, APIs, large tables

Always use a tie-breaking column in the sort key for keyset pagination — created_at alone isn't unique enough (two rows can share a timestamp), so pair it with the primary key: ORDER BY created_at DESC, id DESC and cursor on (created_at, id). Without the tie-breaker, rows with duplicate values in the primary sort column can be skipped or repeated across pages.


7. Subqueries: Correlated vs Uncorrelated

sql
-- Uncorrelated: inner query runs once, independent of the outer query
SELECT name FROM products
WHERE id IN (
    SELECT product_id FROM order_items GROUP BY product_id HAVING COUNT(*) > 100
);
 
-- Correlated: inner query re-runs once per outer row, referencing the outer row
SELECT c.name,
    (SELECT COUNT(*) FROM orders o WHERE o.customer_id = c.id) AS order_count
FROM customers c;
⚠️

A correlated subquery in the SELECT list runs once per outer row — on a table with 100,000 customers, that's 100,000 subquery executions. Most of the time this is better expressed as a LEFT JOIN with GROUP BY, or a window function, both of which the query planner can usually execute in a single pass. Reach for a correlated subquery only when the join/window alternative is genuinely more awkward, and always check the EXPLAIN plan before shipping one against a large table.


Key takeaways

  • LEFT JOIN + a WHERE filter on the right table silently degrades into an INNER JOIN — put such conditions in the ON clause instead.
  • MySQL has no native FULL OUTER JOIN; emulate it with a UNION of LEFT JOIN and RIGHT JOIN.
  • HAVING filters groups after aggregation; WHERE filters rows before — putting a filterable condition in the wrong one costs performance.
  • Window functions (ROW_NUMBER, RANK, LAG/LEAD, SUM() OVER) solve "top-N per group" and running-total problems that are painful with plain GROUP BY.
  • Recursive CTEs are the standard tool for hierarchy/graph traversal in SQL — always confirm the underlying data has no cycles.
  • Views encapsulate query complexity but aren't a performance optimization; materialized views (or scheduled summary tables) are, at the cost of freshness.
  • OFFSET pagination degrades linearly with page depth and is unstable under concurrent writes — use keyset (cursor) pagination for any large or high-write table.
  • Correlated subqueries execute once per outer row; prefer a JOIN or window function when the same result can be computed in one pass.

Interview Questions

  • Explain the difference between INNER JOIN, LEFT JOIN, and FULL OUTER JOIN with an example for each.
  • Why does adding a WHERE condition on the right-hand table of a LEFT JOIN sometimes silently turn it into an INNER JOIN? How do you fix it?
  • How would you find the top 3 highest-paid employees per department using SQL?
  • What is the difference between RANK(), DENSE_RANK(), and ROW_NUMBER()?
  • How do window functions differ from GROUP BY in terms of the number of output rows?
  • What is a recursive CTE, and what safeguard do you need against infinite recursion?
  • What is the difference between a standard view and a materialized view? Which one improves query performance?
  • Why does OFFSET-based pagination get slower on later pages, and how does keyset pagination avoid that?
  • Why is a single sort column often insufficient for keyset pagination, and what do you add to fix it?
  • What is a correlated subquery, and why can it be a performance problem at scale?
  • When would you use a CROSS JOIN deliberately? How can a CROSS JOIN happen accidentally?
  • What does MySQL's ONLY_FULL_GROUP_BY mode enforce, and what happens if it's disabled?
  • Write a query using LAG() to compute the month-over-month change in a metric.