Exporting to Prometheus Grafana

The Opsian Ocaml agent can export data to both the https://opsian.com/ backend and also to prometheus with the profiling data visualised in Grafana. This is a set of instructions for how to configure Opsian to output to Prometheus / Grafana.

Requirements

This guide assumes that you have a working Prometheus and Grafana instances, with Prometheus configured as a data source in Grafana. See https://prometheus.io/docs/visualization/grafana/ for details.

Configuring Opsian

In order to use the Opsian agent, you need to add opsian as a dependency in your Ocaml project, our repo contains an example. In order to configure Opsian you need to set the OPSIAN_OPTS environment variable in the shell where the system is run. The normal Opsian agent options apiKey, host, applicationVersion and agentId are not used in prometheus mode. Here are the prometheus options:

• prometheusPort - the local port to bind to. Required in order to enable prometheus. Multiple ports can be provided, separated by | character, they will be attempted in the order provided and the first that succeeds at binding will be used.
• prometheusHost - the local IP address to bind on. This is useful if you have a public IP address and private IP address on the same host and for security reasons you only want to bind to the private network. Optional, if not set the agent binds to 0.0.0.0
• prometheusSegment - sets the name to be used for a segment tag on the prometheus metrics. This can be used to separate out different hosts, apps or machines and give them separate profiles. Optional.
• prometheusProcessSampleRate - sets the sample rate for process time profiling, often known as CPU time profiling.
• prometheusElapsedSampleRate - sets the sample rate for elapsed time profiling, often known as Wallclock time profiling.

Here's an example of these options being used in conjunction with each other:

export OPSIAN_OPTS="prometheusPort=9100,prometheusHost=192.168.2.101,prometheusSegment=test-example,prometheusProcessSampleRate=50,prometheusElapsedSampleRate=50,debugLogPath=debug.log"

You should be able to test that your configuration has worked by performing an http get on the /metrics endpoint of the specified port and IP. For example, with the previous configuration option:

curl "http://192.168.2.101:9100/metrics"

Prometheus

In order for Prometheus to import the data from the Opsian agent, you need to add it as a Scrape Target. In order to test that this has worked, query prometheus for “promfiler_cpu_profile” and check that the data is there.

Grafana Flamegraph Plugin

Install the Plugin

Copy dist directory of the Grafana Flamegraph Panel to your plugin directory:

git clone https://github.com/Opsian/grafana-flamegraph-panel.git 
cp -r dist/ /path/to/grafana-8.4.3/data/plugins/grafana-flamegraph-panel

You need to use our fork of the flamegraph panel since we’ve forked off a version that works with Grafana 6+. The original Grafana Flamegraph Plugin is unmaintained.

Enable unsigned plugins:

In your conf/defaults.ini file, under the [plugins] section set the following line:

allow_loading_unsigned_plugins = grafana-flamegraph-panel

You should now restart Grafana and confirm that the plugin has been loaded.

Add a dashboard with a flamegraph panel inside

Follow normal Grafana process for adding the panel and when adding the panel (https://grafana.com/docs/grafana/next/panels/working-with-panels/add-panel/). Then click in the top right corner “click to change visualisation” and pick “Flamegraph”. Then setup the query parameters, these are located in the bottom of the panel.

• Metrics Browser = promfiler_cpu_profile{type="cpu"} or promfiler_cpu_profile{type="wallclock"} depending upon which type of profiling data you want. You can also add in additional prometheus query parameters here in order to separate out different instances, for example using the job or segment query types, eg: promfiler_cpu_profile{type="cpu",job="tezos-profiling"}
• Min Step = 1
• Format = Table
• Instant = On

Save the dashboard and view it.