Kubernetes Failure Stories & How to manage Cloud Native secrets with Vault


Jun 3, 2019, 2:30 – 6:00 PM

In-person event

About this event

Interested in Kubernetes? We have a great meetup lined up for you. We have the great pleasure of welcoming Henning Jacobs from Zalando who is visiting Aarhus. Henning is an experienced speaker, and recently created the Kubernetes Failure Stories repository at Github which features a comprehensive list of Kubernetes post-mortems. (https://github.com/hjacobs/kubernetes-failure-stories).

Further, Henrik Høegh will provide you with an introduction to how to handle secrets in a Kubernetes environment using HashiCorp Vault.

16:30: Doors Open
17:00: Welcome, by ORBIT Lab, Cloud Native and Tradeshift
17:15: "Manage your Cloud Native secrets with Vault" by Henrik Høegh, Praqma
18:00: Break and Food sponsored by Tradeshift
18:30: "Kubernetes Failure Stories and How to Crash Your Clusters" by Henning Jacobs, Zalando SE
19:15: Networking
20:00: Thank you and Good Night

TALK 1: "Manage your Cloud Native secrets with Vault", by Henrik Høegh, Praqma

In this talk we will look at the method used to integrate Vault with Kubernetes, and how to authenticate, write and read data from it. But running Vault de-coupled from Kubernetes is not "the Cloud Native way".

In a Cloud Native world, we want things to integrate with the orchestrator. We also have sensitive information in our cluster, which we want to keep secret. Kuberentes secrets are not really suitable for storing information like database password, usernames or alike, as they are simply stored with a base64 encoding.

Instead, we want to keep our "stuff" in a secure place, where we can rotate access tokens. This is where Vault comes into the picture. Vault can store and control access to sensitive information, and it integrates with Kubernetes allowing pods to authenticate with their service account.

BIO: Henrik Høegh, Praqma
As a DevOps consultant Henrik regularly works with build servers, Container technologies like Kubernetes, version control, and agile task management systems but also spend a great deal of time working with general organizational, architectural and process related challenges for customers. And yes, he drives a panda.

TALK 2: "Kubernetes Failure Stories and How to Crash Your Clusters" by Henning Jacobs, Zalando SE

Bootstrapping a Kubernetes cluster is easy, rolling it out to nearly 200 engineering teams and operating it at scale is a challenge. In this talk, we are presenting our approach to Kubernetes provisioning on AWS, operations and developer experience for our growing Zalando developer base. We will walk you through our horror stories of operating 100+ clusters and share the insights we gained from incidents, failures, user reports and general observations. Our failure stories will be sourced from recent and past incidents, so the talk will be up-to-date with our latest experiences.

Most of our learnings apply to other Kubernetes infrastructures (EKS, GKE, ..) as well. This talk strives to reduce the audience's unknown unknowns about running Kubernetes in production.

BIO: Henning Jacobs
Henning joined Zalando in the beginning of 2010 and accompanied the transformation of Zalando’s technology department through the eras of PHP/MySQL and Java/PostgreSQL to the new world of "Radical Agility". He helped to build the STUPS cloud infrastructure to make innovation scale across autonomous teams. His five teams help streamline the developer experience by providing a cloud-native application runtime to 200+ engineering teams.



Monday, June 3, 2019
2:30 PM – 6:00 PM UTC


  • Kasper Borg Nissen


    Lead Organizer

  • Rasmus Steiniche


    Lead Organizer

  • Ophelia Zhang Dalsgaard

    Danske Bank

    Squad & Chapter Lead

  • Henrik Høegh


    Digital Platform Owner

  • Henrik Løvborg

    Red Hat

    Tech Sales Lead Denmark

  • Stefan Daugaard


    Area Tech Lead

  • Ryan Gough

    Senior IT Operations Specialist / Platform at JYSK

  • Kasper Tygesen

    OK a.m.b.a.

    Chief Solutions Architect