Encryption & Secrets Management

Part 1: The building blocks (in plain words)

Key point: encoding ≠ encryption ≠ hashing.

• Base64 is encoding. Encoding only changes how data is written. Anyone can decode it, so it is not security.
• Encryption can be reversed with the key. The scrambled result is called ciphertext.
• Hashing can't be reversed at all.

Use authenticated encryption (like AES-GCM). It protects both secrecy (nobody can read the data) and integrity (nobody can change it without being noticed).

Do not invent your own cryptography. Use well-reviewed libraries (like libsodium or Google Tink, or your platform's standard crypto library) and managed services. Most real failures come from misuse: reusing a nonce (a number that must be used only once), weak random numbers, keys written in the code, or home-made methods. Very few come from breaking the algorithms.

Part 2: Encryption in transit

Use TLS everywhere. TLS is the protocol that encrypts data on the network. It is what makes HTTPS secure. Use it not only between users and your load balancer (post 3), but also:

• between the load balancer and app servers (re-encryption),
• between services,
• between apps and databases, caches and message brokers,
• between data centres and clouds.

The old idea "it is inside our network, so it is safe" has been replaced by zero trust. Zero trust means you assume the internal network may already be hacked, so you check who is calling and encrypt every connection.

mTLS (mutual TLS). With normal TLS, only the server proves who it is. With mTLS, both sides show a certificate. A certificate is a digital ID card signed by a trusted authority. So service A knows it is really talking to service B, and B knows the caller is really A.

Managing certificates for hundreds of services by hand is painful, so:

• service meshes (Istio, Linkerd) give each service a certificate automatically, replace it often (rotate it), and force mTLS between sidecars. A sidecar is a small helper program that runs next to each service (post 13),
• workload identity systems (like SPIFFE/SPIRE) give services short-lived identities, so you do not manage keys by hand.

Certificate management:

• renew certificates automatically (ACME is the protocol for this, and Let's Encrypt is a free service that uses it; cloud certificate managers do it too),
• watch the expiry dates. An expired certificate is a classic cause of outages (post 40),
• use modern TLS versions (TLS 1.2 at least, TLS 1.3 preferred) and turn off old, weak ones.

Part 3: Encryption at rest

Layers of encryption at rest:

• Disk, volume and bucket encryption is usually on by default in the major clouds today. A volume is a virtual disk, and a bucket is a place to store files in the cloud. This protects against stolen or thrown-away hardware and some wrong settings.
• Database Transparent Data Encryption (TDE) encrypts database files and backups automatically. "Transparent" means your app does not notice it.
• Application-level (field-level) encryption means the app encrypts sensitive fields before saving them, so the database only ever sees ciphertext (the scrambled data).

Important: disk, volume and database encryption are transparent. Anyone who can query the database sees decrypted data. This includes an attacker with stolen app credentials, or an attacker who uses SQL injection (post 52). SQL injection means sending SQL commands through a form field. They mainly protect against physical theft and some storage leaks. For the most sensitive data, add field-level encryption, tokenisation, and strict access control.

Tokenisation replaces sensitive data (like a card number) with a meaningless token such as tok_9f3a…. The real value is kept in a separate, highly protected vault. Most of your systems handle only tokens, so if they are breached, nothing useful leaks. This is how payment providers keep merchants out of the scope of the card-data rules (PCI DSS, the security standard for card payments).

Backups must be encrypted too (post 45). Their keys must be available during a disaster, but not to an attacker.

Part 4: Key management and envelope encryption

Encryption changes the problem from "protect the data" to "protect the keys". Keys should:

• never be written directly in the source code,
• be stored in a Key Management Service (KMS), a service that keeps keys safe and does the encrypting for you, or in a Hardware Security Module (HSM), which is special hardware built so that keys cannot be copied out,
• have strict access control and audit logs of every use,
• be rotated regularly.

Envelope encryption means you encrypt your data with a data key, and then you encrypt that data key with a master key. It is how KMS systems (AWS KMS, Google Cloud KMS, Azure Key Vault, HashiCorp Vault) work at scale. Think of a letter (the data) in a locked box (the data key), and the box key is kept in a bank safe (the master key):

Why it's clever:

• Huge data never goes to KMS, only tiny keys, so it's fast and cheap.
• Rotating the master key means re-encrypting the small DEKs, not all your data.
• Access to KMS is controlled and logged, so you can see exactly which service decrypted what, and cut off access instantly.
• Crypto-shredding: if you delete the key, the data encrypted with it becomes unreadable. This is useful for "delete my data" requests and for retiring old backups.

Part 5: End-to-end encryption (E2EE)

With end-to-end encryption, data is encrypted on the sender's device and is decrypted only on the recipient's device. The server in the middle can't read it at all.

• ✅ The strongest privacy. Even the service provider, or anyone who breaches it, can't read messages.
• ❌ Server-side features become hard: search, spam filtering, content moderation (checking content for abuse), backups and multi-device sync all need special designs.
• ❌ Key management moves to the users' devices. If a user loses all their devices and backups, their messages may be lost for ever.

Part 6: Secrets management

What counts as a secret:

• database passwords,
• API keys (payment, SMS and email providers),
• OAuth client secrets (post 50),
• JWT signing keys (post 49),
• TLS private keys,
• encryption keys,
• webhook signing secrets,
• cloud access keys.

Anti-patterns (all very common):

• ❌ secrets in source code or committed config files (even in private repos),
• ❌ secrets baked into Docker images,
• ❌ secrets printed in logs, CI output or error messages,
• ❌ secrets shared over chat or email,
• ❌ one shared admin key used by every service and every engineer,
• ❌ long-lived secrets that are never rotated.

Better: a secrets manager. Store secrets in a dedicated, encrypted, access-controlled, audited system:

• HashiCorp Vault (or its open-source fork OpenBao),
• AWS Secrets Manager / Parameter Store, Google Secret Manager, Azure Key Vault,
• Kubernetes Secrets. ⚠️ By default, these are only base64-encoded in etcd (the database where Kubernetes keeps its data), not encrypted. Turn on encryption at rest for etcd and limit access with RBAC (role-based access control, post 49). You can also use the External Secrets Operator (it copies secrets from a cloud secrets manager), Sealed Secrets or SOPS (tools that let you store encrypted secrets in Git).

Apps get secrets at runtime, which means when they run. The platform puts them in as files or environment variables, or the app reads them through the secrets manager's API.

Best: avoid long-lived secrets entirely.

• Workload identity / IAM roles: your service runs with an identity that the platform gives it (like an AWS IAM role for a pod or VM). A pod is a running unit in Kubernetes, and a VM is a virtual machine. The service gets short-lived credentials automatically, so there are no keys in the configuration at all.
• Dynamic secrets: tools like Vault can create a new database username and password for each app instance. The login works for an hour, and then Vault deletes it.
• OIDC federation from CI/CD: CI/CD is the automatic pipeline that builds and releases your code. Instead of storing cloud keys in GitHub Actions or GitLab, the job shows a short-lived OIDC token (post 50), and the cloud swaps it for temporary credentials.

Rotation and revocation.

• Rotate secrets regularly, and immediately when someone leaves or a leak is suspected.
• Design apps so rotation causes no downtime. For example, accept both the old and the new key during a short change-over period, and reload secrets without restarting.
• Keep an inventory: which secrets exist, who owns them, what uses them, and when they were last rotated.

Detect leaks early.

• Pre-commit hooks and CI scanners (gitleaks, trufflehog, detect-secrets) check your code and block commits that contain secrets. A pre-commit hook is a script that runs before each commit.
• GitHub secret scanning and push protection detect known key formats and can block pushes. Many providers (like cloud and payment companies) automatically cancel their keys when they find them in public repositories.

If a secret leaks, respond in this order:

1. Revoke or rotate it immediately. Deleting the commit is not enough. The secret is still in the Git history and in the copies that bots already made.
2. Check logs for any use of the leaked secret.
3. Assess and contain any damage.
4. Clean up the repository history if needed.
5. Do a blameless postmortem (post 48) and add the controls that would have prevented it.