Investigating and Debugging Microservices on Kubernetes: How to Avoid a Murder Mystery
Investigating issues with containerized applications running in Kubernetes requires new tools and techniques
As any developer will tell you, we all accidentally add bugs to our software. Although unit testing can catch a great deal of potential issues it is inevitable that a few bugs end up in production. Once these have been identified it is vitally important to be able to quickly investigate, triage, and diagnose the issue. While murder mysteries can be fun, no one likes to be playing a game of murder mystery or “whodunnit” when facing a bug; especially not a 3am after you’ve been woken by PagerDuty!
Cloud Native: More Services, More Layers, More Challenges
Before the era of cloud native, diagnosing and debugging systems was relatively straightforward. There was typically a monolithic application (or small number of applications) and so the search space for the problem was constrained primarily by the size of a single code base. Applications were also typically executed as a standalone process on a virtual machine. Viewing logs or attaching a remote debugger was often as simple as SSHing into the machine.
All of this changed with the adoption of microservices, containers, and Kubernetes. These new architectures and technologies enable rapid evolution of systems, but the cost is often related to increased complexity and reduced understandability. This can lead to the “whodunnit” style of bug hunting!
Locating an issue in a system composed of ten microservices now means that there are ten code bases where a potential bug could be lurking. And this isn’t counting the integrations and gaps between services where bugs can also hide. Containerizing applications and running them via Kubernetes also adds additional layers of complexity for viewing logs and debugging, and the applications can be rescheduled at a moment's notice.
Traditional | Cloud Native | |
|---|---|---|
| Number of Services | 1 (or a small number) | Many |
| Infrastructure Layers | VM | VM, Kubernetes, containers |
| Log Access | Tail single process | Tail multiple processes, or logs shipped to centralized location |
| Correlation of User Requests | Single/multiple threads in single process | Multiple threads in multiple processes separate by network boundaries |
| Debugging | Debug locally running instances, or open ports in VM to enable remote debugging | Open ports in firewall, VPC, K8s and containers to enable remote debugging |
Become a Cloud Native Sherlock Holmes
Being able to effectively debug services in Kubernetes is not dependent on a single tool or technique. A combination of approaches is required (and made easier by free, community tools):
- Service Insight
- System Visualization
- Correlating User Requests
- Centralized Log Management
- Kubernetes-Native Debugging
Ready to investigate and debug your own Kubernetes woes?
This learning journey walks you through the primary concepts and hands-on activities required to debug issues across your cluster and multi-service applications.
- Annotating services to quickly identify key debugging information
- How distributed tracing helps follow requests across multiple services
- Debugging your cluster when things go wrong
- Using Telepresence to debug services locally
Related Resources
Easy Debugging of Spring Boot Microservices Running on Kubernetes
[Podcast] Livin' on the Edge Podcast #3: Matt Klein on Testing Microservices and Building Cloud Platforms
![[Podcast] Livin' on the Edge Podcast #3](/static/podcast-3-aaacde743799fa08721d7ec4031c6903.png)
[Podcast] Livin’ on the Edge #10: Sam Newman on Microservice Ownership, Local Development, and Release Trains
![[Podcast] Livin’ on the Edge #10](/static/podcast-10-4c29aaa1c4fe418ab1f4ab04cf5999b5.png)
[Podcast] Livin’ on the Edge #11: Charity Majors on Instrumenting Systems, Observability-Driven Development, and Honeycomb
![[Podcast] Livin’ on the Edge #11](/static/podcast-11-01c7a4b5899078628387969c6e4872e1.png)