Container Security Best Practices for Production

Why Container Security Matters

Containers are everywhere. Over 92% of organizations use containers in production (Sysdig 2024 Container Security Report). But the speed and convenience of containers often comes at the cost of security.

The risk landscape:

Key insight: Most container breaches aren't sophisticated. They exploit basic misconfigurations — images running as root, unpatched base images, and overly permissive network policies.

---

The Container Security Lifecycle

Security must be addressed at every stage — build, ship, and run. A vulnerability missed during build becomes an incident at runtime.

Build Phase: Secure the Foundation

Choose Minimal Base Images

Every unnecessary package in your base image is a potential attack surface. Use the smallest image that meets your needs.

Image size comparison:

Best practices for base images:

• Use Alpine or distroless images whenever possible
• Pin base image versions — use node:20.11.0-alpine, not node:latest
• Use multi-stage builds to separate build dependencies from runtime
• Remove package managers from final images to prevent runtime package installation

Scan Images Before Publishing

Every image must be scanned before it reaches a registry:

• Trivy (Aqua Security) — Fast, comprehensive, open source
• Grype (Anchore) — Deep CVE scanning with SBOM integration
• Docker Scout — Native Docker scanning
• Snyk Container — Developer-friendly with fix suggestions

When to scan:

• In the Dockerfile build step (fail fast)
• In your CI/CD pipeline (gate deployments)
• On a schedule in your registry (catch newly published CVEs)
• At runtime (detect drift)

Don't Run as Root

Running containers as root is the single most common — and most dangerous — misconfiguration.

What to do:

• Create a dedicated non-root user in your Dockerfile
• Set USER appuser before the CMD/ENTRYPOINT instruction
• Use runAsNonRoot: true in Kubernetes security contexts
• Avoid chmod 777 — set specific file permissions

---

Ship Phase: Secure the Pipeline

Once images are built, the shipping process introduces its own risks.

Registry security:

• Use private registries — Never pull production images from Docker Hub without verification
• Enable image signing — Use Cosign (Sigstore) or Docker Content Trust to verify image authenticity
• Implement admission controllers — Reject unsigned or unscanned images in Kubernetes
• Automate image promotion — dev → staging → production with gates at each stage

Supply chain controls:

• Generate SBOMs for every image using Syft or Trivy
• Maintain an approved base image catalog — teams can only build from pre-approved bases
• Track image provenance — know exactly what source code and build environment produced each image

---

Run Phase: Runtime Protection

Build-time security catches known issues. Runtime security catches everything else.

Kubernetes Security Contexts

Every pod should have a restrictive security context:

• readOnlyRootFilesystem: true — Prevent filesystem modifications
• runAsNonRoot: true — Enforce non-root execution
• allowPrivilegeEscalation: false — Block processes from gaining additional privileges
• capabilities: drop: [ALL] — Remove all Linux capabilities, add back only what's needed

Network Policies

By default, Kubernetes allows all pod-to-pod communication. Lock it down:

• Default deny all ingress and egress — Start with zero access
• Allow only required communication — Frontend can talk to API, API can talk to database
• Block internet access — Most pods don't need outbound internet; block it unless explicitly required
• Use service mesh — Istio or Linkerd for mTLS between services

Resource Limits

Without resource limits, a single compromised container can consume all cluster resources:

• Set CPU and memory requests and limits for every container
• Configure LimitRanges to enforce defaults across namespaces
• Use ResourceQuotas to cap total namespace consumption
• Monitor for containers approaching their limits

Runtime Monitoring

Deploy runtime security tools to detect anomalous behavior:

• Falco (CNCF) — Detects unexpected system calls, file access, and network connections
• Sysdig Secure — Runtime threat detection + vulnerability management
• Aqua Security — Full lifecycle container security
• Tetragon (Cilium) — eBPF-based runtime enforcement

---

Container Security Checklist

Before deploying to production, verify each item:

Build:

• Non-root user configured
• Minimal base image (Alpine/distroless)
• Multi-stage build separating build and runtime
• Image scanned with zero high/critical CVEs
• SBOM generated and stored

Ship:

• Image signed and signature verified
• Stored in a private registry with access controls
• Admission controller blocking unsigned images
• Image promotion pipeline with gates

Run:

• Security context configured (non-root, read-only FS, no privilege escalation)
• Network policies in place (default deny)
• Resource limits set (CPU and memory)
• Runtime monitoring deployed
• Secrets managed via external secrets operator (not env vars or ConfigMaps)
• Logging configured and centralized

---

Further Reading

• CIS Kubernetes Benchmark
• NIST SP 800-190 — Container Security Guide
• NSA/CISA Kubernetes Hardening Guide
• Sigstore — Container Signing

Remember: Container security is a shared responsibility. Developers build secure images, platform teams configure secure clusters, and security teams monitor for threats. No single team can do it alone.