Monitoring with Prometheus and Grafana

Observability

Monitoring and Observability is one of the key pillars of creating a robust distributed system. Envoy outputs a wide variety of statistics that provide insights to how well your application is performing.

With Ambassador as your gateway, Prometheus as your stats sink, and Grafana providing dynamic plotting of these statistics, we can quickly create a system to easily monitor ingress into your cluster.

Using the provided reference architecture, the following instructions will set up a minimal monitoring environment that will display:

  • Requests per second
  • 95th percentile peak RPS
  • Requests per minute that respond with HTTP response codes 2xx, 3xx, 4xx, and 5xx
  • Total open connections
  • Connection lengths
  • Number of registered routes
  1. Install Ambassador Pro

    Ambassador Pro is a commercial version of Ambassador that includes integrated Single Sign-On, powerful rate limiting, custom filters, and more. Ambassador Pro also uses a certified version of Ambassador OSS that undergoes additional testing and validation.

    1. Clone the Ambassador Pro configuration repository

      Ambassador Pro consists of a series of modules that communicate with Ambassador. The core Pro module is typically deployed as a sidecar to Ambassador. This means it is an additional process that runs on the same pod as Ambassador. Ambassador communicates with the Pro sidecar locally. Pro thus scales in parallel with Ambassador. Ambassador Pro also relies on a Redis instance for its rate limit service and several Custom Resource Definitions (CRDs) for configuration.

      For this installation, we'll start with a standard set of Ambassador Pro configuration files.

      git clone https://github.com/datawire/pro-ref-arch
    2. License Key

      Copy env.sh.example to env.sh, and add your specific license key to the env.sh file. If you don’t have a license key, you can request a free 14-day trial key now.

      Note: Ambassador Pro will not start without a valid license key.

    3. Deploy Ambassador Pro

      If you're on GKE, first, create the following ClusterRoleBinding:

      kubectl create clusterrolebinding my-cluster-admin-binding \
      --clusterrole=cluster-admin \
      --user=$(gcloud info --format="value(config.account)")
      

      Then, deploy Ambassador Pro:

      make apply-ambassador

      This make command will use kubectl to deploy Ambassador Pro and a basic test configuration to the cluster.

      Verify that Ambassador Pro is running:

      kubectl get pods | grep ambassador
      ambassador-79494c799f-vj2dv            2/2       Running            0         1h
      ambassador-pro-redis-dff565f78-88bl2   1/1       Running            0         1h
      

      Note: If you are not deploying in a cloud environment that supports the LoadBalancertype, you will need to change the ambassador/ambassador-service.yaml to a different service type (e.g., NodePort).

      By default, Ambassador Pro uses ports 8081 and 8082 for rate-limiting and filtering, respectively. If for whatever reason those assignments are problematic (perhaps you set service_port to one of those), you can set adjust these by setting environment variables:

      • GRPC_PORT: Which port to serve the RateLimitService on; 8081 by default.
      • APRO_AUTH_PORT: Which port to serve the filtering AuthService on; 8082 by default.

      If you have deployed Ambassador with AMBASSADOR_NAMESPACE, AMBASSADOR_SINGLE_NAMESPACE, or AMBASSADOR_ID set, you will also need to set them in the Pro container.

  2. Configure Grafana behind Ambassador

    1. Get the IP address or hostname of your ambassador service

      kubectl get svc ambassador
    2. In your env.sh file, replace {http(s)://{AMBASSADOR_IP/HOSTNAME}:{AMBASSADOR_INGRESS_PORT}} with the protocol, IP or hostname, and port.

      Note: If you are doing a standard installation on a cloud provider this will be

      AMBASSADOR_URL=https://$AMBASSADOR_IP

    Grafana will now know to use AMBASSADOR_URL/grafana as the root URL

  3. Deploy Prometheus and Grafana

    make apply-monitoring

    After that command completes, Prometheus and Grafana will be deployed. Send a couple of requests through Ambassador to generate some statistics.

    Note: Prometheus takes a couple of minutes to pull the metrics from statsd

  4. Access the Grafana dashboard

    1. Go to https://${AMBASSADOR_IP}/grafana/ from a web browser

    2. Log in with the credentials

      username: admin
      password: admin
    3. On the left panel, hover over the Dashboards icon (looks like a 2x2 matrix) and click Manage

    4. Click on Ambassador to pull up the pre-configured Grafana dashboard.

    You will now see the different panels in the dashboard showing the data stated above.

    Note: The "Percentile" statistics are collected and aggregated over over an hour time frame. To get an accurate estimate of this data, send a sample of production load through this Ambassador for over a day and check back then.