Use Zipkin distributed tracing system to troubleshoot latency problems

Zipkin is a distributed tracing system. It helps gather /manage timing data needed to troubleshoot latency problems in microservice architectures. Twitter developed the technology using a Google paper that described Google’s internally-built distributed app debugger, Dapper.

In the Microservices architecture, application is usually structured as a set of loosely coupled, collaborating services. Each service implements a set of related functions. For example, an application might consist of services such as the order management service, the customer management service, etc. Services communicate using either synchronous protocols such as HTTP/REST or asynchronous protocols such as AMQP. Services can be developed and deployed independently of one another. Each service has its own database in order to be decoupled from other services. For more intro on Microservice, check out here.

Services requests often span multiple services. Each service handles a request by performing one or more operations, e.g. database queries, publishes messages, etc. To understand the behavior of an application and troubleshoot problems there is a need to monitor each service but if you use any of the external monitoring tools, it only tells you the overall response time and number of invocations but there would be no insight into the individual operations. To understand how each service is performing we would need to instrument services with code that will do the following

1. Assign each external request a unique external request id
2. Pass the external request id to all services that are involved in handling the request
3. Include the external request id in all log messages
4. Record information (e.g. start time, end time) about the requests and operations performed when handling an external request in a centralized service.

Applications are instrumented to report timing data to Zipkin server. The Zipkin UI presents a Dependency diagram showing how many traced requests went through each application. If you are troubleshooting latency problems or errors, you can filter or sort all traces based on the application, length of trace, annotation, or timestamp. Once you select a trace, you can see the % of the total trace time each span takes.

Whenever a request comes in, Zipkin application traces it as it goes through the system. Each request gets a unique identifier, which is passed along with the request to each microservice. For Zipkin to work, each microservice is instrumented with Zipkin library that the service then uses to identify the request’s entry and exit ports. Libraries are available for C#, Java, JavaScript, Python, Go, Scala and Ruby.

Request data is transmitted back to a Zipkin server, which is captured by Node.js and stored in Cassandra. It is left to the user to establish the communication protocol between the emitter and the collector; for his class, Gehard uses RabbitMQ. Scribe, HTTP, and Kafka are also recommended as transport mechanisms.

Zipkin comes with a Web interface that shows the amount of traffic each microservice instance is getting. The log data can be filtered by application, length of trace, annotation, or timestamp.

Below is Intro video :

Checkout Quickstart to get started.

Like this post? Don’t forget to share it!

Useful Resources :

• Kubernetes Tutorial : Distributed tracing with Jaeger
• Monitoring Docker containers using Prometheus + cAdvisor + Grafana
• 10 BEST Kubernetes monitoring tools
• Google Cloud Courses Collection
• How to aggregate Docker Container logs and analyse with ELK stack ?
• TOP 20 Microservices Design Patterns