If you develop kubernetes applications, you must have experienced a very slow and painful debugging process at some point. With every change you make you must rebuild the image and deploy it to the cluster (whether local or remote). Every change.
The ultimate dev experience#
I have a strong opinion on what modern development should look like. There must be nearly zero delay between adding a print debug statement and seeing the result. Even better is to use the debugger - step through code one line at a time while inspecting the call stack and state of variables. With the magic of Python this gets even more powerful. Thanks to its interpreted dynamic nature, it allows you to run expressions in a REPL console while the program is in a paused state. You can run code that was not even imported when you hit the breakpoint!
This for me has by far been the fastest way to debug and develop new features and it’s the way my brain works. Instead of trying to build an entire feature and then running it and hoping it would work first try, I often write a chunk, inspect the state and run experiments (even call APIs and database queries!). All while my program is paused exactly on the line I am focusing on.
Embrace kubernetes#
When working with kubernetes applications, what you would typically do to achieve this level of dev comfort is to duplicate the entire stack locally using docker-compose. This might be good enough for a lot of projects, but what if you need to access kubernetes APIs to spin up jobs or if your app receives requests from other services in the cluster? What if you have a complex helm chart and for some reason all bugs you are getting can be replicated only if you spin up the whole tractor, the classical “it works on my machine” problem?
With Agent Stack we were hitting all of these issues and with my obsession for local debugger and mimicking the production infrastructure as closely as possible, I was determined to find a solution.
Devspace#
While this sounds great, a few years ago I was setting up devspace for a project and I remember that the reality was nowhere near the
promise. First of all you have to write a devspace.yaml, yet another yaml config to learn and believe me it’ll take
you at least one afternoon before you put together a decent workflow and then a lot of trial and error to make it actually work.
And I haven’t even attempted the remote debugger, I have too many bad memories from trying to set up a remote docker interpreter
in PyCharm. These workflows are so rarely used that they are often full of bugs and compromises, maybe VS Code does it better?
Once you configure it though, it should just work for other people on my team right? Use devspace dev and life is beautiful.
Truth is with these tools that no matter how simple the commands look at first sight, you still need to understand deeply what
is going on, or you’ll get blocked on the first hurdle that occurs. Developing applications is not just writing code and debugging.
You need to add dependencies, rebuild images, write and test migrations, access the database, other services, etc.
All in all, this was about a week of lost productivity and the adoption in my team was not what I was hoping for, due to the learning curve and complexity.
Haunted by this experience, I was looking for something better and I found a project that blew my mind 🤯.
Telepresence#
As the name suggests, Telepresence will put your local machine into the cluster:
- you can run and debug your application as a local process as you would normally
- kubernetes hostnames will resolve on your desktop, you can call them from your app or open them in the browser ->
postgres,myapp-svc,myapp-svc.default.svc.cluster.local - traffic that other services send to your kubernetes app will be received by your local process
As far as network is concerned, your local machine is now a pod in the cluster.
To set this up you need just a few things:
- install telepresence CLI
- run a kubernetes cluster, for example using
rancher-desktop
and install the traffic manager using
telepresence helm install - deploy a stable version of your app (
helm install your-app) - connect to the cluster using
telepresence connect - replace the pod using
telepresence replace <app-pod-name>and run your app locally on the same port as in kubernetes
telepresence replace command will
physically replace the pod with a thin traffic agent that receives and forwards incoming traffic
out to your local process. If you want to learn more about the internals, check the official
documentation on architecture.If you install sshfs, telepresence will even forward volumes mounted to the pod to your local filesystem.
Conclusion#
Telepresence is a game changer. You no longer have to think about port-forwarding,
rebuilding images, duplicating infrastructure stack or even having a separate app configuration.
Does your app connect to the database using postgresql-svc hostname by default? Just teleport
and it will work locally too! And all of this with just a few commands, zero configuration!
As with any software, it’s not without its faults. I’ve been using this tool for a couple of months now and there are a few annoyances here and there. I need to reconnect when I close my laptop, it doesn’t work if I am connected to a VPN tunnel, it breaks if I start uploading gigabytes of data in parallel and sometimes it will eat up my CPU and memory, probably due to some memory leak.
With all that said though, telepresence has become an indispensable tool for me when working on Agent Stack, combined with the local lima VM stack covered by the previous article. We now code and debug in exactly the same environment that gets shipped to users, which makes replicating bugs and infrastructure a lot easier. Every project is different, but if you are working on a cloud native application, I would recommend giving it a try. Maybe you will be as amazed by the simplicity and ease of use as I was.
