For an overview of other options for gathering statistics on Emissary-ingress, see the Statistics and Monitoring overview.
At the core of Emissary-ingress is Envoy Proxy, which has built-in support for exporting a multitude of statistics about its own operations to StatsD (or to the modified DogStatsD used by Datadog).
If enabled, then Emissary-ingress has Envoy expose this information via the
ubiquitous and well-tested StatsD
protocol. To enable this, you will simply need to set the environment
STATSD_ENABLED=true in Emissary-ingress's deployment YAML:
When this variable is set, Emissary-ingress by default sends statistics to a
Kubernetes service named
statsd-sink on UDP port 8125 (the usual
port of the StatsD protocol). You may instead tell Emissary-ingress to send
the statistics to a different StatsD server by setting the
STATSD_HOST environment variable. This can be useful if you have an
existing StatsD sink available in your cluster.
We have included a few example configurations in the
statsd-sink/ directory to help you get started. Clone the
repository to get local, editable copies.
Graphite is a web-based real-time graphing system. Spin up an example Graphite setup:
This sets up the
statsd-sink service and a deployment that contains
Graphite and its related infrastructure. Graphite's web interface is
http://statsd-sink/ from within the cluster. Use port
forwarding to access the interface from your local machine:
This sets up Graphite access at
Prometheus is an open-source monitoring and alerting system.
Prometheus does not natively understand the StatsD protocol, but you
can deploy the Prometheus StatsD Exporter to act as the StatsD
sink, and it will translate from StatsD to the exposition format
that Prometheus requires. An example of how deploying Prometheus
StatsD Exporter is available in
To finally get the statistics to Prometheus, you then configure a
Prometheus target to read from
statsd-sink on port 9102.
You could instead also add the
statsd-sink service and Prometheus StatsD
Exporter as a sidecar on the Emissary-ingress pod. If you do this, make
sure to set
STATSD_HOST=localhost so that UDP packets are routed to
It may be desirable to change how metrics produced by the
statsd-sink are named, labeled and grouped when they finally make it
For example, by default, each service that Emissary-ingress serves will
create a new metric using its name. For the service called
you will see this metric
envoy.cluster.usersvc_cluster.upstream_rq_total. This may lead to
problems if you are trying to create a single aggregate that is the
sum of all similar metrics from different services. In this case, it
is common to differentiate the metrics for an individual service with
label. This can be done by configuring a Prometheus StatsD
Exporter "mapping" (not to be confused with an Emissary-ingress
Mapping). See Metric Mapping and Configuration in
the Prometheus StatsD Exporter documentation to learn how to modify
If you deploy Prometheus using Helm the value that you should change
in order to add a mapping is
it to something like this:
If you are using the
prom-statsd-sink.yaml example, edit its
ConfigMap with your Prometheus mappings, for example:
If you don't already have a Prometheus setup, the Prometheus Operator is a powerful way to create and deploy Prometheus instances. Use the following YAML to quickly configure the Prometheus Operator with Emissary-ingress:
statsd-sink.yamlCreates the Prometheus Stats Exporter deployment and
statsd-sinkservice that receives the statistics from Emissary-ingress and translates them to Prometheus metrics. It also creates a
ServiceMonitorresource that tells the Prometheus Operator to configure Prometheus to fetch those metrics from the StatsD Exporter.
prometheus.yamlDeploys the Prometheus Operator and creates
Prometheusresource that tells the Prometheus Operator to create the actual Prometheus deployment.
Make sure that the
ServiceMonitor is in the same namespace as
Emissary-ingress. A walk-through of the basics of configuring the
Prometheus Operator with Emissary-ingress is available
STATSD_ENABLED is set to
"true" and apply the YAML with
Wait for a minute after the pods spin up and then access the
Prometheus dashboard by port-forwarding the Prometheus pod and going
http://localhost:9090/ on a web-browser.
If you're using Grafana, Alex Gervais has written a template
Emissary-ingress dashboard for Grafana that works with either the
metrics exposed by the Prometheus StatsD Exporter, or by [the
If you are a user of the Datadog monitoring system, pulling in the Envoy statistics from Emissary-ingress is very easy.
Because the DogStatsD protocol is slightly different than the normal
StatsD protocol, in addition to setting Emissary-ingress's
STATSD_ENABLED=true environment variable, you also need to set the
DOGSTATSD=true environment variable:
Then, you will need to deploy the DogStatsD agent in to your cluster to act as the StatsD sink. To do this, replace the sample API key in our sample YAML file with your own, then apply that YAML:
This sets up the
statsd-sink service and a deployment of the
DogStatsD agent that forwards the Emissary-ingress statistics to your
Additionally, Emissary-ingress supports setting the
statitics tag using the
DD_ENTITY_ID environment variable. If this value
is set, statistics will be tagged with the value of the environment variable.
Otherwise, this statistics tag will be omitted (the default).