HighlyAvailableKubernetes

• Kubernetes zero downtime deployment: when theory meets the database - rolling update + backward/forward compatible database schema changes w/data duplication via database access abstraction
• Zero-downtime Deployment in Kubernetes with Jenkins - rolling update + blue/green
• Zero-Downtime Rolling Updates With Kubernetes - interesting use of lifecycle: preStop: sleep command to ensure no lost client connections from ingress
• Enable Rolling updates in Kubernetes with Zero downtime

Considerations

• CAP - consistency (eventual/data latency), availability, partition (network/quorum)

  • Consistency is acceptable drift by being eventual

• Backup/restore

  • RTO - Recovery Time Objective
  • RPO - Recovery Point Objective (PITR)

• Monitoring, alerting, escalation

  • Synthetic application monitoring (white, grey and blackbox)
    • Latency monitoring
  • Timeseries metrics
    • Latency monitoring

• Versioned Kubernetes Secrets and ConfigMaps

  1. So What’s Next?

• Upgrade/rollback Kubernetes application - relies on versioned product

• Fully ORM abstraction layer in all database communication to stage backward/forward compatible schema changes between application deltas - urgh potentially loads of effort (handled per application, or shared library or abstraction service)

• Geographical/datacentre fault tolerence: deploy all components to at least n+1 availability zones - mostly done

  • Autoscaling Kubernetes node capacity - easy

• Application fault tolerence: multiple replicas inside Kubernetes - done (?)

  • Pod resource requirements and quotas - easy
  • Deployment RollingUpdate strategy - easy
  • Deployment readinessProbe reflect genuinely readiness - easy
    • Consider use of livenessProbe probe but use it correctly - easy

• Versioned Kubernetes Secrets and ConfigMaps - investigate

• Load shedding: graceful service degredation at system limits - potentially some effort

• DoS protection - AWS shield?

Zero Downtime Infrastructure Upgrade

Kubernetes Control Plane & Node Groups

• Node groups - simply perform blue/green failover of node groups in Kubernetes control plane
• EKS - not found anything yet
• GKE - https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-upgrading-your-clusters-with-zero-downtime

Cloud Platform & Data Persistence

• “Depends”

Related Reading to End Goal

• AWS Well-Architected Framework - get pull out your finger and finally “read it all”

  • A Cloud Guru - Mastering the AWS Well-Architected Framework - or watch and listen to it all

• The Famous Twelve Factors Application

  • and Beyong the Twelve Factor App

• Google Cloud best practices for operating containers

  • Google Cloud best practices for building containers