5π Consulting

By Johannes Ziemke.
Cool, cool but need help? You can hire me!.

Case Studies

Koko: AI-Powered Community Moderation

From legacy infrastructure to over 100 million content moderation decisions per day and acquired by Airbnb. Building an agile cloud native infrastructure for a high velocity AI based content moderation startup.

…

Styla: Content Experience Engine

Advising a content commerce startup on Docker migration, AWS deployment and DevOps strategy. Debugging a ELB/ECS based pilot deployment. Implementation of and migration to lambda based machine learning pipeline.

…

Blog

Build securely with Github Actions and ECR using OpenID Connect

February 05, 2024

Intro Most companies and projects these days use some form of CI/CD to automatically build artifacts like binaries or container images when new code is pushed to a repository. Commonly this involves: Configuring where to push and how to name the artifacts in a config file in the repository Granting the build server permissions to your artifact…

…

Kubernetes at Home: Hardware

Diagram of Rack Disclaimer: While writing this article, I realize I could use Affiliate Links to the stuff I bought. So I gave it a try. Lemme know if that bothers you. I’ve been running Kubernetes on OpenWrt at home for two years now. Although I already believed that Kubernetes makes sense at small scale this was intended as experiment about how…

…

Kubernetes Native CI With Argo

Introduction As I discussed in Why Kubernetes at small scale, Kubernetes is more an infrastructure framework than just a container scheduler. Every resource in your infrastructure is a Kubernetes Object and new Resources can be defined using Custom Controllers and Resources. Argo One of the custom controllers I’m most excited about Argo. It extends…

…

Running Kubernetes on OpenWrt

Why.. Me, better with clean software than clean hardware As you might know, I’m living in Berlin, Germany. Maybe you also know that our Internet is slow. But I finally found a cable internet provider which is supposed to deliver up to 400 MBit to my home. Unfortunately their service seems to be really bad. So instead of canceling my old DSL aka…

…

Why Kubernetes at small scale?

December 18, 2018

Google Scale Wannabes I’ve frequently hear people say something like this: The number of people who adopt complicated stuff like Kubernetes for what is essentially a couple of web servers and a database is too high. They’re Google wannabies that thinks in Google’s scale but forget that it is utterly unnecessary in their case. reacharavindh, https…

…

Run Prometheus Blackbox Exporter as Lambda

December 10, 2018

Introduction When providing any service to a global customer base, you are probably interested in measuring the performance and availability of your site from multiple global locations. For this, I’ve created latency.at which unfortunately didn’t get the traction I hoped it would. (See this for more details). So today, we’re back where we were…

…

fluentd for data warehousing with athena

October 31, 2018

Or how I replaced AWS Firehose by fluentd instances and saved >80% EC2 traffic costs. Using AWS Kinesis Firehose koko is one of my current clients. They are running on the Kubernetes infrastructure I built earlier this year. For analytics purposes with AWS Athena, the backend services use AWS Kinesis Firehose to ship messages to S3. Basically it…

…

Found severe Azure Kubernetes Bug but ain't even got a lousy T-Shirt

February 17, 2018

About half a year ago at one of my last gigs, I built a Kubernetes Cluster on Azure. On Azure, the Kubernetes Volume Claims are commonly implemented by a “Virtual Hard Disk” (VHD) file stored on the Azure Blob Storage Service. The Kubernetes “cloud provider intergration” takes care of allocating and managing these disks. While I was working on some…

…

kubecfn: Cloudformation installer for reasonably secure multi-master Kubernetes Cluster

February 01, 2018

Introduction These days I’m helping koko to build their Kubernetes infrastructure on AWS. Initially, we used kops to deploy the cluster but weren’t very happy. I’ve researched various option and blogged about them earlier. tl;dr: There aren’t that many options that are secure, highly available and don’t require buying into a vendor ecosystem…

…

Production Grade Kubernetes

December 15, 2017

About a year ago I blogged about how to build a $15 Production Kubernetes Cluster on DigitalOcean and submitted it to Hacker News. HN being HN, soon after these comments trickled in: . Fair enough. If you only need $15 worth of resources, running a three node Kubernetes cluster might not be the best idea. As explained in that article, I was more…

…

Use Prometheus Vector Matching to get Kubernetes Utilization across any Pod Label

November 09, 2017

Prometheus was designed for dynamic environments like Kubernetes. Its powerful service discovery and query language allows you to answer all kind of questions that come up while operating a Kubernetes cluster. This flexibility comes with a somewhat steeper learning curve than some alternatives and hosted services, with Vector Matching being one of…

…

Frontend Testing Zoo - Or, Nightwatch without Selenium

What I always find hardest to figure it when it comes to unknown systems, it’s ‘how things fit together’. These project might have great documentation, but how the ecosystem plays together is usually left as an exercise for the reader. This happened to me again when diving into frontend testing. I figured out how to get nightwatch+selenium…

…

$15 Production Kubernetes Cluster on DigitalOcean

November 20, 2016

Four sweets or production Kubernetes for a month Introduction As you might already know, I’m into containers, static configuration and self-service infrastructures. Naturally, I love Kubernetes, which I consider the most promising cluster scheduler around. In fact, the biggest reason to use containers is that they make it possible for something…

…

The Future: Fabrication Cloud

October 05, 2015

The Cloud Given the ubiquity of “The Cloud” even in mainstream media, you might expect the readers of this blog know what it means. From my experience, that isn’t true. Too many consider it a meaningless buzzword, synonym for the Internet or the act of outsourcing handling of precious personal data. It has to do with outsourcing, the Internet and…

…

The Future

October 02, 2015

Since I was a kid, I was fascinated with sci-fi and today I’m thrilled to live in the unique age of unprecedenced velocity of invention. A few years ago, Elon Musk’s idea of building an electric car that can compete with regular cars was a moonshot. It was close to impossible, yet he succeeded. From todays perspective, this seems like a nobrainer…

…

Don't manage configuration unless you have to.

August 31, 2015

At my first job, there was no code driven infrastructure at all. A nightmare from today’s perspective. The next job, there were some perl scripts to massage systems in a imperative way. Then, at SoundCloud, I used modern configuration management for the first time. SoundCloud heavily invested into Chef which was, at some point, used to drive almost…

…

CloudFormation driven Consul in AutoScalingGroup

CloudFormation templates are a great way to manage AWS resources. All resources for a stack are configured in a json and CloudFormation takes care of creating or updating your stack. Consul is a distributed, consistent data store for Service Discovery and configuration which also features health checks and supports for distributed locks. Consul…

…

Scope and Ownership in Tech Companies

My first job was a typical corporate job at a financial service provider after which I worked at large hosting company before joining SoundCloud, a startup culture-wise closer to Silicon Valley startups before joining Docker end of 2013. Although I never was a manager, I was always interested in the company as a whole, how my work shapes it and how…

…

Building AWS AMIs from Scratch for Immutable Infrastructures

Why? I’m a big fan of what some might call “immutable infrastructures” which, to me, boils down to manging complexity by isolating change to specific points in the life cycle of your services. In practice this might mean you build your application image once and just recreate it if you need to update it. Or you install your complete server once and…

…

Find GHOSTs in your Docker images

January 27, 2015

A severe security vulnerability in glibc < 2.18, nicknamed GHOST was just reported. Here is a handy one-liner (Debian/Ubuntu only though) to walk through all your Docker images and see if they include a glibc older than 2.18:

…

Monitor Docker Containers with Prometheus

January 26, 2015

Monitoring Docker Running all your services in containers makes it possible to get in-depth resource and performance characteristics, since every container runs in their own cgroup and the Linux kernel provides us with all kind of useful metrics. Although there are a few other Docker monitoring tools out there, I’ll show you why I think that…

…

Containerize your Infrastructure

January 08, 2015

The Hype Everyone is talking about containers. Containers are a great way to bundle your application with all dependencies and isolate it from other applications running on the same host. That’s all great but we never cared that much about containerization even though it exists for quite some time: chroot() was born 1979 and IBM had tech similar to…

…

Using docker run --net=container:XX ... to debug network issues

November 13, 2014

Sharing the network namespace with a existing container is a less known feature of Docker. If you run: Your container runs in the same network namespace, sharing the same network configuration as . This is very useful for debugging purposes: I just wanted to verify the current outgoing connections from a container in our infrastructure and realized…

…

Running a highly available load balancer on Docker

November 10, 2014

For quite some time I felt like I should do a blog - again. Instead of spending time writting rants on facebook or commenting other peoples posting, I should write some blog articles! Thing is, it’s not that easy. Where to start? What should I talk about? What are you interested in reading? Instead of diving into abstract thoughts about the…

…