Data Encryption and Secrets Management: At Rest, In Transit, and Everywhere Between

Encryption as a Design Constraint

Every system stores data somewhere and moves data between somewhere. If either of those states is unencrypted, a single stolen disk, a compromised network tap, or a misconfigured S3 bucket turns into a full data breach. Encryption doesn't prevent every attack — it prevents a successful intrusion from becoming a catastrophic one.

This article is the deeper, canonical reference on encryption and secrets. Authentication, authorization, and the OWASP Top 10 are covered in System Security: Authentication, Authorization, and Common Vulnerabilities — here we go deep on how data itself is protected, both while it sits on disk and while it travels.

Encryption at Rest

Encryption at rest protects data stored on disk — databases, backups, file systems, object storage — from anyone who gets physical or unauthorized access to the storage medium itself.

AES-256: The Default Choice

AES (Advanced Encryption Standard) with a 256-bit key is the industry-standard symmetric cipher for data at rest. It's fast (hardware-accelerated on nearly all modern CPUs via AES-NI), well-audited, and considered secure against brute force for the foreseeable future. Almost every managed database, disk encryption tool, and cloud storage service defaults to AES-256 under the hood.

Layers of Encryption at Rest

Encryption at rest isn't one decision — it happens at different layers, each with different tradeoffs:

Most production systems layer these: full-disk encryption on the underlying volume as a baseline, transparent data encryption (TDE) at the database layer for at-rest protection of data files and backups, and field-level encryption for the handful of columns that are regulated or especially sensitive.

Transparent Data Encryption (TDE)

TDE is a database feature (available in Postgres via extensions, MySQL, SQL Server, Oracle) that encrypts data files, logs, and backups on disk without changing application code. Queries against the running database see plaintext — encryption and decryption happen transparently at the storage engine level.

Application  --(plain SQL)-->  Database Engine  --(AES-256)-->  Disk
                                      ^
                              decrypts on read,
                              encrypts on write

Encryption in Transit

Data moving across a network — between browser and server, between microservices, between a service and its database — needs to be encrypted so that anyone intercepting the traffic (a compromised router, a malicious Wi-Fi hotspot, a man-in-the-middle) sees ciphertext, not payloads.

TLS 1.3 at a Conceptual Level

TLS (Transport Layer Security) 1.3 is the current standard, and it's meaningfully faster and simpler than TLS 1.2. The handshake establishes a shared symmetric key without ever sending that key over the wire:

Key improvements over TLS 1.2: the handshake completes in one round trip instead of two, weak ciphers (RC4, SHA-1, static RSA key exchange) are removed entirely, and every handshake supports forward secrecy — meaning even if a server's private key is stolen later, past recorded traffic can't be decrypted.

Certificate Management

TLS relies on certificates issued by a Certificate Authority (CA) to prove a server is who it claims to be. In production, certificate management means:

• Automated issuance and renewal. Let's Encrypt and ACME-based tooling (cert-manager on Kubernetes) issue short-lived certs (90 days) and renew them automatically. Manually-managed certs are a leading cause of outages — an expired cert silently breaks all TLS connections.
• Monitoring expiry. Alert at 30/14/7 days before expiry, independent of whether auto-renewal "should" have worked.
• Private key protection. The certificate's private key must never leave the server or the HSM/KMS that holds it. A leaked private key means anyone can impersonate your service until the cert is revoked.

mTLS for Service-to-Service Communication

Regular TLS authenticates the server to the client (the browser verifies it's really talking to your server). Mutual TLS (mTLS) goes further: both sides present certificates, so the server also verifies the client's identity.

This is the standard pattern inside a service mesh (Istio, Linkerd) for internal microservice traffic: every service gets a short-lived identity certificate issued by an internal CA, rotated automatically (often every 24 hours), and services reject connections from anything that doesn't present a valid cert. This means network-level access alone — being inside the VPC — is no longer sufficient to talk to a service, which matters a lot in a zero-trust architecture.

Symmetric vs Asymmetric Encryption

Both show up constantly in system design, and the tradeoff is speed vs. key distribution.

In practice, systems use both together: asymmetric cryptography establishes trust and exchanges a symmetric key (this is exactly what the TLS handshake above does), and then the fast symmetric cipher encrypts the actual data. You'll almost never see RSA used to encrypt a large payload directly — it's reserved for the parts of the system where identity and key exchange matter more than raw throughput.

Envelope Encryption and KMS

A naive design encrypts every piece of data directly with one master key. This has two serious problems: rotating the master key means re-encrypting everything it ever touched, and the master key is used so often that it's exposed to more attack surface than necessary.

Envelope encryption solves this with a layer of indirection:

The pattern: generate a random Data Encryption Key (DEK) for each object, encrypt the data with the DEK, then encrypt the DEK itself with a Key Encryption Key (KEK) — the master key, which never leaves the KMS or HSM. You store the encrypted DEK alongside the encrypted data. To decrypt, you send the encrypted DEK to the KMS, which decrypts it (without ever exposing the master key), and use the resulting plaintext DEK to decrypt the data.

Why this is better than encrypting directly with the master key:

• Rotating the master key is cheap. You re-encrypt only the small DEKs, not the underlying data — because the KMS re-wraps each DEK with the new KEK, the data itself never moves.
• The master key almost never touches plaintext data directly, which limits the blast radius if any single operation is compromised.
• Each object gets its own DEK, so compromising one DEK exposes only that object, not the entire dataset.

This is exactly how AWS KMS, GCP Cloud KMS, and Azure Key Vault implement "encrypt this data" APIs under the hood, and it's the pattern you should describe in an interview when asked how you'd design encryption for a multi-tenant system with millions of encrypted objects.

Secrets Management

Secrets — database passwords, API keys, TLS private keys, third-party credentials — are a different problem from encrypting data, but closely related: a leaked secret defeats encryption entirely, because the attacker doesn't need to break the cipher, just read the key.

Why Secrets Don't Belong in Env Vars, Config Files, or Git

• Git history is forever. A secret committed and then removed in a later commit is still recoverable from history. Rotation is the only real fix once this happens.
• Environment variables are visible. Anyone who can exec into a container, read /proc/<pid>/environ, or trigger a crash dump can often recover env vars. They also get logged accidentally more often than you'd expect (error trackers, CI logs).
• Config files get copied. Onto laptops, into backups, into Docker images that get pushed to a registry — each copy is a new place the secret can leak from.
• No audit trail. A secret sitting in a config file gives no record of who accessed it or when. A vault does.

Vault-Based Secret Injection

The alternative is a dedicated secrets manager — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager — that stores secrets encrypted, controls access with fine-grained policies, and logs every read.

The application doesn't hold a long-lived secret at all — it holds an identity (an IAM role, a Kubernetes service account token) that it uses to authenticate to the vault, which then hands back the actual secret at runtime. This means the secret can be rotated, revoked, or scoped without ever touching application code or deployment config.

Dynamic, Short-Lived Credentials vs Static Secrets

A static secret (a database password that's the same for months) is a standing liability — if it leaks, it's valid until someone notices and rotates it. Vault's dynamic secrets engine flips this: Vault generates a brand-new database user with a random password on demand, with a TTL of minutes to hours, and revokes it automatically on expiry.

# Example: request a dynamic database credential from Vault
$ vault read database/creds/readonly-role
 
Key                Value
---                -----
lease_id           database/creds/readonly-role/abcd1234
lease_duration     1h
username           v-readonly-a1b2c3d4
password           A1a-RandomGeneratedPassword

If this credential leaks, the blast radius is capped at one hour, and it's scoped to exactly the permissions the readonly-role policy grants — nothing more.

Key Rotation

Keys — encryption keys, API keys, database credentials, TLS certs — should be rotated regularly, both as a hygiene practice (limiting the value of a key even if it's never known to be compromised) and as an incident response tool (revoking a specific compromised key immediately).

Rotation Strategies

The overlap-window and versioned-key approaches are what production systems actually use. Envelope encryption (above) makes this tractable: rotating the KEK doesn't require touching the underlying data, only re-wrapping DEKs, so key rotation becomes a background operation rather than a blocking migration.

The Operational Challenge

Rotating without downtime means:

1. Generate the new key/credential, but don't retire the old one yet.
2. Roll out the new key to all consumers (services, config, KMS aliases).
3. Confirm nothing is still using the old key (via access logs / metrics).
4. Only then revoke the old key.

Skipping step 3 is the most common cause of rotation-induced outages — a forgotten batch job or a cached credential in a long-running process keeps using the old key, and it starts failing the moment the old key is revoked.

Hardware Security Modules (HSMs)

An HSM is a dedicated, tamper-resistant physical (or cloud-virtualized) device that generates and stores cryptographic keys, and performs cryptographic operations (signing, encryption) inside the device — the private key material never leaves it, even during use.

When You Need One

Most systems are well-served by a cloud KMS (AWS KMS, GCP Cloud KMS), which is backed by HSMs under the hood without you managing the hardware. Dedicated HSM access (AWS CloudHSM, Azure Dedicated HSM) becomes necessary when:

• Compliance mandates it explicitly. PCI-DSS for payment card processing, and certain government/financial regulations, require FIPS 140-2 Level 3 certified hardware key storage, not just "encryption at rest."
• You need single-tenant key isolation and can't accept keys being managed in a shared multi-tenant KMS, even one backed by HSMs.
• You're doing high-volume signing (e.g., a certificate authority, a payment processor) where the HSM's dedicated crypto hardware and stricter access model are a hard requirement, not a nice-to-have.

Field-Level Encryption

Sometimes encrypting the whole database is overkill, or the database itself isn't fully trusted (e.g., a third-party analytics platform with read access). Field-level encryption encrypts specific sensitive columns — SSNs, credit card numbers, health records — while leaving the rest of the row queryable in plaintext.

users table
+----+----------+----------------------+------------------------+
| id | name     | email                | ssn_encrypted           |
+----+----------+----------------------+------------------------+
| 1  | J. Doe   | j@example.com        | AES256(...)==base64     |
+----+----------+----------------------+------------------------+

The encryption/decryption happens in the application layer (or via a proxy) using a DEK obtained from the KMS — the database itself just stores ciphertext bytes and has no ability to decrypt them.

The Query-Ability Tradeoff

This is the fundamental cost of field-level encryption: you lose the ability to index, search, filter, or sort on the encrypted field directly.

• WHERE ssn = '123-45-6789' no longer works — AES with a random IV produces different ciphertext every time, even for the same input, so there's nothing to match against.
• Range queries (WHERE salary > 100000) are essentially impossible on an encrypted numeric field.
• The common workaround is a deterministic hash of the value stored in a separate indexed column purely for equality lookups (e.g., ssn_hash = SHA256(ssn + pepper)), while the actual encrypted value stays in ssn_encrypted and is only decrypted after the row is found.

Encryption and Secrets Checklist

• Data at rest encrypted with AES-256 at the appropriate layer (disk, TDE, or field-level)
• TLS 1.3 enforced for all external traffic; weak ciphers/TLS versions disabled
• Certificates auto-renew, and expiry is monitored independently of the renewal automation
• Service-to-service traffic uses mTLS, not just network-perimeter trust
• Bulk data encrypted symmetrically (AES); asymmetric crypto reserved for key exchange and identity
• Envelope encryption (DEK + KEK) used instead of direct master-key encryption for any multi-object system
• No secrets in env vars, config files, or source control — a vault or secrets manager is the source of truth
• Secrets are short-lived and dynamically generated where possible, not static and long-lived
• Key rotation uses an overlap window or key versioning, not a hard cutover
• HSM usage is tied to a named compliance requirement, not used by default
• Sensitive fields (SSN, card numbers, etc.) use field-level encryption, with query patterns decided upfront

What to Remember for Interviews

1. Encryption at rest has layers — disk, TDE, field-level — and each protects against a different threat, not all of them.
2. TLS 1.3 establishes a symmetric session key via asymmetric handshake — know that symmetric is fast/bulk, asymmetric is for exchange and identity.
3. Envelope encryption (DEK + KEK) is the standard answer for "how do you rotate keys at scale" — it avoids re-encrypting all the data.
4. Secrets belong in a vault, not in env vars or config — and the real upgrade is dynamic, short-lived credentials, not just centralized storage.
5. Field-level encryption trades away query-ability — know the deterministic-hash workaround for equality lookups.
6. HSMs are a compliance-driven choice, not a default — be ready to name the driver (PCI-DSS, FIPS 140-2) if you bring one up.