Categories


Archives


Recent Posts


Categories


What are Open Telemetry Metrics and Exporters

astorm

Frustrated by Magento? Then you’ll love Commerce Bug, the must have debugging extension for anyone using Magento. Whether you’re just starting out or you’re a seasoned pro, Commerce Bug will save you and your team hours everyday. Grab a copy and start working with Magento instead of against it.

Updated for Magento 2! No Frills Magento Layout is the only Magento front end book you'll ever need. Get your copy today!

This entry is part 3 of 3 in the series An Introduction to Prometheus and Metrics. Earlier posts include What is Prometheus?, and What are Prometheus Exporters?. This is the most recent post in the series.

Last time we ended by musing that Prometheus, while an open source project, still creates a case of vendor lock-in for its users. Getting metrics IN to Prometheus is easy. Getting metrics out is less easy. Changing-to or trying-out a new metric system is even less easy.

Today we’re going to take a look at the Open Telemetry project. Open Telemetry is an open source project whose aim is to create a common, Open Telemetry platform that’s capable of sending your telemetry data to any monitoring platform. Instrument things once with Open Telemetry, and you can send your data to any system that’s capable of importing it.

Today we’ll take a look at creating a counting metric using the Open Telemetry metrics library. Then we’ll show you how you can export this metric data to any platform — including Prometheus!

To get the most out of this tutorial you’ll want to be familiar with some starting Prometheus concepts we covered in the first two articles in this series.

Instrumenting a Service

Let’s start with the same small NodeJS program from our first article.

// File: index.js
const express = require('express')

const app = express()
const port = 3000

app.get('/stuff', function (req, res) {
res.type('json')
  res.send(JSON.stringify({hello:"world"}))
})

app.listen(
  port,
  function() {
    console.log(`Example app listening at http://localhost:${port}`)
  }
)

We’re going to do the same thing we did in part one: Create a metric that counts how many times the /stuff endpoint gets called. However, this time we’ll use the Open Telemetry metrics system instead of Prometheus.

To start, we’ll want to install the @opentelemetry/metrics package into our project.

$ npm install @opentelemetry/metrics

Then, we’ll use this library to instrument our program.

const express = require('express')
// NEW: The `MeterProvider` class creates factory-ish objets that
//      allows us to create `Meter` objects.  `Meter` objects are
//      a sort of global-ish singleton-ish object that we'll use to create
//      all our metrics.
const { MeterProvider } = require('@opentelemetry/metrics');

// NEW: Here's where we instantiate the `MeterProvider`.  The `my-meter`
//      string that uniquely identifies this meter.  The `getMeter` method
//      will fetch a meter if it already exists, or create a new one
//      if it does not.
const meter = new MeterProvider().getMeter('my-meter');

// NEW: Here we use our `Meter` object to create a counter metric.
//      This is similar to `require('prom-client').Counter(...)`
//      from the prom-client library
const counter = meter.createCounter('my-open-telemetry-counter', {
  description: 'A simple counter',
});

const app = express()
const port = 3000

app.get('/stuff', function (req, res) {
  // NEW: Here we increment our counter by 1.  This is similiar to
  //      the counter.inc() in prom-client, except that we're passing
  //      in the amount to increment a counter by.

  // NOTE: Ideally we should be able to invoke this via
  //       `counter.add(1)` once this Issue lands
  //
  //       https://github.com/open-telemetry/opentelemetry-js/issues/1031
  //
  //       If this issue hasn't landed, you'll need to pass an empty
  //       object in as the second paramter
  counter.add(1, {})
  res.type('json')
  res.send(JSON.stringify({hello:"world"}))
})

app.listen(
  port,
  function() {
    console.log(`Example app listening at http://localhost:${port}`)
  }
)

Here we see patterns that are similar patterns to the Prometheus instrumentation library. We create a counter metric

const counter = meter.createCounter('my-open-telemetry-counter', {
  description: 'A simple counter',
});

and then we increment that metric.

// counter.add(1, {})
counter.add(1, {})

What’s a little different here is the MeterProvier and Meter objects

const { MeterProvider } = require('@opentelemetry/metrics');

const meter = new MeterProvider().getMeter('my-meter');

A Meter object (created with a MeterProvider object) is what we use to create our metrics. This is just a case of the Open Telemetry project being a little more abstracted than the prom-client library we used with Prometheus. This will be important later, but for now let’s keep our focus on the metrics themselves.

An Exporter by any Other Name

So far this all seems similar to what we saw with Prometheus. Open Telemetry’s big differentiation is how we, or other systems, are able to view these metrics. There is not, at the time of this writing, a client for scraping Open Telemetry metrics or a UI for viewing them. Instead, we need to rely on Open Telemetry Exporters to move our data into other systems.

These Open Telemetry exporters are not Prometheus exporters. In Prometheus, an exporter allows you to “export” metrics from a system that isn’t instrumented by Prometheus.

An Open Telemetry Exporter has a different job. In Open Telemetry, an exporter allows you to export metrics from an instrumented application to another third party system.

There is an Open Telemetry exporter capable of exporting Open Telemetry metrics in a way where the Prometheus client can consume them. Put another, even more confusing way, there’s an an Open Telemetry Prometheus Exporter that is not a Prometheus Exporter, but is an Open Telemetry Exporter.

Yes, naming things remains hard. Hopefully some code will clear things up.

Remember that Meter object we created? Let’s change its instantiation so it looks like this

// NEW: pulls in the `PrometheusExporter` class from the
//      '@opentelemetry/exporter-prometheus library
const { PrometheusExporter } = require('@opentelemetry/exporter-prometheus');

// NEW: Instantiates a `PrometheusExporter` object.
const exporter = new PrometheusExporter(
  {startServer: true,},
  () => {
    console.log('prometheus scrape endpoint: http://localhost:9464/metrics');
  },
);

// NEW: When creating our meter object
const meter = new MeterProvider({
  exporter:exporter,
  interval: 1000,
}).getMeter('my-meter');

This new code uses a class from the @opentelemetry/exporter-prometheus package, so we’ll want to pull that into our project

$ npm install @opentelemetry/exporter-prometheus

Run your program,

$ node index.js

send some traffic to /stuff,

$ curl http://localhost:3000/stuff

and then hit the /metrics URL exposed by the Open Telemetry Prometheus exporter

$ curl http://localhost:9464/metrics
# HELP my_open_telemetry_counter A simple counter
# TYPE my_open_telemetry_counter counter
my_open_telemetry_counter 3 1589046070557

Congratulations! You just exported your Open Telemetry metrics. From here all you’d need to do is point your Prometheus client at this URL (as we did in part one of this series), and your Open Telemetry metrics would be imported into Prometheus

What just Happened?

A Meter object is aware of every metric that’s created and, in turn, aware of the current value of every metric. When you instantiate a Meter, you can provide it with an exporter object.

const meter = new MeterProvider({
  exporter:exporter,
  interval: 1000,
}).getMeter('my-meter');

When you configure an exporter, the Meter object will send that exporter a list of every metric object on a timed interval. That’s what the interval configuration field is for. The Meter will send the exporter every metric in the system once every 1000 milliseconds (i.e. once a second). It’s the job of an exporter to examine these records, and then take steps to make that data available to other systems. What these steps are will depend entirely on the system you’re trying to export your metrics to.

In the case of the Prometheus exporter, those steps are exposing an HTTP endpoint that a Prometheus client can read, and printing out the metric data in a form that Prometheus understands. If you wanted to export your metrics to another system, you’d probably take different steps (POSTing those metric values to an HTTP endpoint, streaming them over GRPC, writing them out to disk for consumption by a cron job, etc.)

For example, here’s “The World’s Simplest Exporter™”: It sends metric data to the console of the running application.

const simpleExporter = {
  export: function(metrics, resultCallback) {
    for(const [,metric] of metrics.entries()) {
      console.log(metric)
    }

    // you'll need to npm install @opentelemetry/base to get
    // access to the status code expected by
    resultCallback(
      require("@opentelemetry/base").ExportResult.SUCCESS
    )
  },
  shutdown: function(shutdownCallback) {
    console.log('do any work needed to shutdown your exporter')
    // call shutdownCallback to indicate exporter is shutdown
    shutdownCallback()
  }
}

All we need to do is add this simpleExporter object to the configuration when you’re instantiating your Meter object

const meter = new MeterProvider({
  exporter: simpleExporter,
  interval: 1000,
}).getMeter('my-meter');

Run your program with this exporter in place, send some traffic to /stuff, and you’ll see your metric printed out to the console about once a second.

$ node index.js
Example app listening at http://localhost:3000
{
  descriptor: {
    name: 'my-open-telemetry-counter',
    description: 'A simple counter',
    unit: '1',
    metricKind: 0,
    valueType: 1,
    labelKeys: [],
    monotonic: true
  },
  labels: {},
  aggregator: CounterSumAggregator {
    _current: 1,
    _lastUpdateTime: [ 1589046826, 890210944 ]
  }
}

Open Telemetry is built around the idea that there should be a central way to create instrumentation data, but that anyone should be able to export and use that data. While it would be easy to dismiss Open Telemetry as just one more standard, the Open Telemetry team is doing everything they can to make this system compatible with your existing solution.

Wrap Up

There’s a lot more to Open Telemetry than what we discussed in these three articles. We only scratched the surface on Metrics, and we didn’t even get into Traces/Spans. If you want to learn more the specification overview document, while dense, is as good a place as any to start. From there exploring the code and engaging with the community via their mailing lists and Gitter Chat are the best ways to get started.

I suspect I’ll have more to say on the subject as time goes on.

Series Navigation<< What are Prometheus Exporters?

Copyright © Alan Storm 1975 – 2020 All Rights Reserved

Originally Posted: 13th May 2020