Instrumenting Traces with Zipkin

Last time we ran a small, three service pre-instrumented system that reported spans into Zipkin. In this article we’re going to take a look at how we instrumented that system using Zipkin’s free and open source libraries.

This article assumes you have a version of NodeJS on your computer, a Zipkin instance running on port 9411, and have a copy of our sample services checked-out/cloned on your computer. If you’re not sure what we’re talking about, checkout part one of this series.

What is Zipkin?

Zipkin is a system designed to create, store, and display distributed traces. Each span in Zipkin represents a single request to a service. This service could be HTTP based, but it doesn’t need to be.

In order to instrument a service, Zipkin will need to

1. Do some work whenever code makes a request to a service
2. Do some work when a service starts processing a request

Whenever we make a service request, we need to either create a new trace ID or determine what the current trace ID is. We also need to pass that trace ID along with the service request. This trace ID allows Zipkin to connect all the traces together.

We need to pass that trace ID along so that when a service starts processing a request it knows what trace the request is a part of.

Our Service

Let’s take a look at one of our un-instrumented sample services from our previous article.

// File: uninstrumented/src/service-main.js

const express = require('express')
const fetch = require('node-fetch')
const {getUrlContents} = require('./util')

const setupMainProgram = function() {
  const app = express()
  const port = 3000

  app.get('/main', async function (req, res) {
    // fetch data from second service running on port 3001
    const resultString = await getUrlContents(
      'http://localhost:3001/hello', fetch)
    const resultJson = JSON.parse(resultString)
    res.type('json')
    res.send(JSON.stringify({main:resultJson.message}))
  })

  app.listen(
    port,
    function() {
      console.log(`Example app listening at http://localhost:${port}`)
    }
  )
}

setupMainProgram()

As programs go it’s not too complex. If there’s concepts in here that you haven’t encountered before, the Mozilla Developers Network express tutorials are a great place to get up to speed on express services.

The two parts of this program we’re interested in are the code that sets up our /main route/URL, and the code that “makes a service call” — i.e. the code that fetches the contents of http://localhost:3001/hello. You can see the route setup here.

// File: uninstrumented/src/service-main.js

const fetch = require('node-fetch')
/* ... */
app.get('/main', async function (req, res) {
  // fetch data from second service running on port 3001
  const resultString = await getUrlContents(
    'http://localhost:3001/hello', fetch)
  const resultJson = JSON.parse(resultString)
  res.type('json')
  res.send(JSON.stringify({main:resultJson.message}))
})

The function passed as the second argument to app.get is the function that express will call when handling a request to http://localhost:3001/main

The getUrlContents function is what fetches a URL using the node-fetch library. We can see the source of getUrlContents here

//File: uninstrumented/src/util.js

const getUrlContents = function(url, fetch) {
  return new Promise((resolve, reject)=>{
    fetch(url)
    .then(res => res.text())
    .then(body => resolve(body));
  })
}

module.exports = {
  getUrlContents
}

This function uses the passed in node-fetch library (the fetch variable above) to retrieve the URL’s contents, and then the route handler uses those results as part of its own response.

Instrumenting with Zipkin

Earlier, we said that in order to instrument a service with Zipkin, we need to

either create a new trace ID or determine what the current trace ID is. We also need to pass that trace ID along with the service request.

For this particular service, that means we’ll need to

1. Add a Zipkin middleware to our express application (this creates-or-determines the trace ID)
2. Wrap the node-fetch library with a Zipkin wrapper (this passes trace ID along to the next service)

We can see the instrumented version of this service here.

Instrumenting Express

Adding the express middleware is the more straight forward of the two tasks — we can see the code required in this partial fragment

// File: instrumented/src/util.js

const createTracer = (localServiceName) => {
  const tracer = new Tracer({
    ctxImpl: new CLSContext('zipkin', true),
    recorder: new BatchRecorder({
      logger: new HttpLogger({
        endpoint: 'http://localhost:9411/api/v2/spans',
        jsonEncoder: JSON_V2
      })
    }),
    localServiceName: localServiceName // name of this application
  });
  return tracer;
}

// File: instrumented/src/service-main.js
const tracer = createTracer('service-main')

/* ... */

app.use(createZipkinMiddleware({tracer}));

app.get('/main', async function (req, res) {
  // fetch data from second service running on port 3001
  const zipkinFetch = createFetcher('service-hello', tracer)
  const resultString = await getUrlContents(
    'http://localhost:3001/hello', zipkinFetch)
  const resultJson = JSON.parse(resultString)
  res.type('json')
  res.send(JSON.stringify({main:resultJson.message}))
})

This code creates a tracer object, which includes the name we want to use for the service (service-main)

// File: instrumented/src/service-main.js

const tracer = createTracer('service-main')

Then we load the Zipkin library for instrumenting express

// File: instrumented/src/service-main.js

const createZipkinMiddleware = require('zipkin-instrumentation-express').expressMiddleware;

and then we use the returned createZipkinMiddleware function to create a middleware. Finally the code adds that express middleware to the current application.

// File: instrumented/src/service-main.js

app.use(createZipkinMiddleware({tracer}));

Important: Be sure to add this app.use line before your call to app.get. Route-less Express middleware will not fire if they’re added after another middleware that calls res.end().

Instrumenting node-fetch

Next — we’ll take a look at the important bits for instrumenting node-fetch

// File: instrumented/src/util.js

const createFetcher = (remoteServiceName, tracer) => {
  const wrapFetch = require('zipkin-instrumentation-fetch');
  return wrapFetch(fetch,
    {
      tracer:tracer,
      remoteServiceName:remoteServiceName
    }
  );
}

// File: instrumented/src/service-main.js
const fetch = require('node-fetch')
/* ... */
const zipkinFetch = createFetcher('service-hello', tracer)
const resultString = await getUrlContents(
  'http://localhost:3001/hello', zipkinFetch)

Unlike express, node-fetch doesn’t have the concept a middleware or plugin system. In order to instrument the node-fetch module, Zipkin needs to wrap the node-fetch module. That is, the wrapFetch function provided by the zipkin-instrumentation-fetch module will take an object that implements the fetch api, and returns a version with its methods changed such that they perform the necessary steps to add the trace ID to the outgoing request.

// File: instrumented/src/util.js

const zipkinFetch = createFetcher('service-hello', tracer)

// File: instrumented/src/util.js

const createFetcher = (remoteServiceName, tracer) => {
  const wrapFetch = require('zipkin-instrumentation-fetch');
  return wrapFetch(fetch,
    {
      tracer:tracer,
      remoteServiceName:remoteServiceName
    }
  );
}

The wrapFetch function requires us to provide our previously instantiated tracer, as well as provide the name of the remote service.

With this new version of fetch in hand (stored in the zipkinFetch variable below)

// File: TO DO find file, use up to date code

const fetch = require('node-fetch')
/* ... */
const zipkinFetch = createFetcher('service-hello', tracer)

we can then use it in our getUrlContents function call

// File: TO DO find file, use up to date code

const resultString = await getUrlContents(
  'http://localhost:3001/hello', zipkinFetch)

What is the Tracer

You probably noticed that both the express and fetch instrumentation required an instantiated Tracer object, created via our createTrace function.

// File: instrumented/src/util.js

const {
  Tracer,
  BatchRecorder,
  jsonEncoder: {JSON_V2}
} = require('zipkin');
const CLSContext = require('zipkin-context-cls');
const {HttpLogger} = require('zipkin-transport-http');

const createTracer = (localServiceName) => {
  const tracer = new Tracer({
    ctxImpl: new CLSContext('zipkin', true),
    recorder: new BatchRecorder({
      logger: new HttpLogger({
        endpoint: 'http://localhost:9411/api/v2/spans',
        jsonEncoder: JSON_V2
      })
    }),
    localServiceName: localServiceName // name of this application
  });
  return tracer;
}

The tracer is a single instance object that contains the basic configuration that other instrumentation modules will need to trace the application.

The localServiceName parameter is our Zipkin name for the service we’re instrumenting.

The recorder parameter is an object we want to use to send data to our actual Zipkin system. You’ll see we’ve used a BatchRecorder object configured with a HttpLogger object that’s pointed at our http://localhost:9411 Zipkin URLs. We won’t get too into how these object work, but it’s worth investigating if you’re planning on using Zipkin to trace your systems.

The ctxImpl parameter is an object that implements Zipkin’s (implicit) context interface. It’s beyond the scope of this article to explain tracing context in greater detail, but in Zipkin it’s the thing that makes sure that a trace ID stays consistent between the asynchronous callbacks in Node’s network handling code.

Zipkin Challenges

So that’s how you instrument an express service using Zipkin.

If you weren’t using express (or an express compatible framework) you’d need to find a Zipkin library for that framework. For example, there’s also a library for instrumenting applications built using the Koa framework.

Similarly — if your application uses a library other than node-fetch for making HTTP requests, you’d need to find another Zipkin library. For example, if you were using the got framework to make HTTP requests, there’s a zipkin-instrumentation-got library.

If you can’t find a library for the package you’re using, you’re of out of luck. While it’s certainly possible to implement your own instrumentation library, there’s not a well documented path forward for doing so. Even if you’re willing to figure out how these Zipkin libraries work, the Zipkin project itself doesn’t (appear to?) offer a formal, supported API for writing instrumentations.

And then — even if you do invest the effort into writing custom instrumentation — those instrumentations are Zipkin specific. You have no easy way to switch to a different tracing library. Similar to what we saw with Prometheus, even though Zipkin is an open system based on open source code, its complexity has created a situation where you can still end up locked into a specific open source ecosystem.

What to do? In out next article, we’ll take a look at the OpenTelemetry project’s tracing code, and how it can offer you a path out of this open-source-vendor-lock-in.