12 min • read

Istio integration

Emissary-ingress and Istio: Edge Proxy and Service Mesh together in one. Emissary-ingress is deployed at the edge of your network and routes incoming traffic to your internal services (aka "north-south" traffic). Istio is a service mesh for microservices, and is designed to add application-level Layer (L7) observability, routing, and resilience to service-to-service traffic (aka "east-west" traffic). Both Istio and Emissary-ingress are built using Envoy.

Emissary-ingress and Istio can be deployed together on Kubernetes. In this configuration, Emissary-ingress manages traditional edge functions such as authentication, TLS termination, and edge routing. Istio mediates communication from Emissary-ingress to services, and communication between services.

This allows the operator to have the best of both worlds: a high performance, modern edge service (Emissary-ingress) combined with a state-of-the-art service mesh (Istio). While Istio has introduced a Gateway abstraction, Emissary-ingress still has a much broader feature set for edge routing than Istio. For more on this topic, see our blog post on API Gateway vs Service Mesh.

This guide explains how to take advantage of both Emissary-ingress and Istio to have complete control and observability over how requests are made in your cluster:

If desired, you may also

To follow this guide, you need:

  • A Kubernetes cluster version 1.15 and above
  • kubectl
  • Istio version 1.10 or higher

Install Istio

Start by installing Istio. Any supported installation method for Istio will work for use with Emissary-ingress.

Configure Istio Auto-Injection

Istio functions by supplying a sidecar container running Envoy with every service in the mesh (including Emissary-ingress). The sidecar is what enforces Istio policies for traffic to and from the service, notably including mTLS encryption and certificate handling. As such, it is very important that the sidecar be correctly supplied for every service in the mesh!

While it is possible to manage sidecars by hand, it is far easier to allow Istio to automatically inject the sidecar as necessary. To do this, set the istio-injection label on each Kubernetes Namespace for which you want auto-injection:

yaml
kubectl label namespace $namespace istio-injection=enabled --overwrite

Install Emissary-ingress with Istio Integration

Properly integrating Emissary-ingress with Istio provides support for:

  • Mutual TLS (mTLS), with certificates managed by Istio, to allow end-to-end encryption for east-west traffic;
  • Automatic generation of Prometheus metrics for services; and
  • Istio distributed tracing for end-to-end observability.

The simplest way to enable everything is to install Emissary-ingress using Helm, though you can use manual installation with YAML if you wish.

To install with Helm, write the following YAML to a file called istio-integration.yaml:

yaml
# Listeners are required in Emissary-ingress 2.0.
# This will create the two default Listeners for HTTP on port 8080 and HTTPS on port 8443.
createDefaultListeners: true
# These are annotations that will be added to the Emissary-ingress pods.
podAnnotations:
# These first two annotations tell Istio not to try to do port management for the
# Emissary-ingress pod itself. Though these annotations are placed on the Emissary-ingress
# pods, they are interpreted by Istio.
traffic.sidecar.istio.io/includeInboundPorts: "" # do not intercept any inbound ports
traffic.sidecar.istio.io/includeOutboundIPRanges: "" # do not intercept any outbound traffic
# We use proxy.istio.io/config to tell the Istio proxy to write newly-generated mTLS certificates
# into /etc/istio-certs, which will be mounted below. Though this annotation is placed on the
# Emissary-ingress pods, it is interpreted by Istio.
proxy.istio.io/config: |
proxyMetadata:
OUTPUT_CERTS: /etc/istio-certs
# We use sidecar.istio.io/userVolumeMount to tell the Istio sidecars to mount the istio-certs
# volume at /etc/istio-certs, allowing the sidecars to see the generated certificates. Though
# this annotation is placed on the Emissary-ingress pods, it is interpreted by Istio.
sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-certs"}]'
# We define a single storage volume called "istio-certs". It starts out empty, and Istio
# uses it to communicate mTLS certs between the Istio proxy and the Istio sidecars (see the
# annotations above).
volumes:
- emptyDir:
medium: Memory
name: istio-certs
# We also tell Emissary-ingress to mount the "istio-certs" volume at /etc/istio-certs in the
# Emissary-ingress pod. This gives Emissary-ingress access to the mTLS certificates, too.
volumeMounts:
- name: istio-certs
mountPath: /etc/istio-certs/
readOnly: true
# Finally, we need to set some environment variables for Emissary-ingress.
env:
# AMBASSADOR_ISTIO_SECRET_DIR tells Emissary-ingress to look for Istio mTLS certs, and to
# make them available as a secret named "istio-certs".
AMBASSADOR_ISTIO_SECRET_DIR: "/etc/istio-certs"
# AMBASSADOR_ENVOY_BASE_ID is set to prevent collisions with the Istio sidecar's Envoy,
# which runs with base-id 0.
AMBASSADOR_ENVOY_BASE_ID: "1"

To install Emissary-ingress with Helm, use these values to configure Istio integration:

  1. Create the emissary Namespace:

    yaml
    kubectl create namespace emissary
  2. Enable Istio auto-injection for it:

    yaml
    kubectl label namespace emissary istio-injection=enabled --overwrite
  3. Make sure the Helm repo is configured:

    bash
    helm repo add datawire https://app.getambassador.io
    helm repo update
  4. Use Helm to install Emissary-ingress in emissary:

    bash
    helm install emissary-ingress --namespace emissary -f istio-integration.yaml datawire/emissary-ingress && \
    kubectl -n emissary wait --for condition=available --timeout=90s deploy -lapp.kubernetes.io/instance=emissary-ingress

Installation Using YAML

To install using YAML files, you need to manually incorporate the contents of the istio-integration.yaml file shown above into your deployment YAML:

  • pod-annotations should be configured as Kubernetes annotations on the Emissary-ingress Pods;
  • volumes, volumeMounts, and env contents should be included in the emissary-ingress Deployment; and
  • you must also label the emissary Namespace for auto-injection as described above.

Configuring an Existing Installation

If you have already installed Emissary-ingress and want to enable Istio:

  1. Install Istio.
  2. Label the emissary namespace for Istio auto-injection, as above.
  3. Edit the Emissary-ingress Deployments to contain the annotations, volumes, volumeMounts, and env elements shown above.
    • If you installed with Helm, you can use helm upgrade with -f istio-integration.yaml to modify the installation for you.
  4. Restart the Emissary-ingress pods.

Configure an mTLS TLSContext

After configuring Emissary-ingress for Istio integration, the Istio mTLS certificates are available within Emissary-ingress:

  • Both the istio-proxy sidecar and Emissary-ingress mount the istio-certs volume at /etc/istio-certs.

  • The istio-proxy sidecar saves the mTLS certificates into /etc/istio-certs (per the OUTPUT_CERTS environment variable).

  • Emissary-ingress reads the mTLS certificates from /etc/istio-certs (per the AMBASSADOR_ISTIO_SECRET_DIR environment variable) and creates a Secret named istio-certs.

To make use of the istio-certs Secret, create a TLSContext referencing it:

shell
$ kubectl apply -f - <<EOF
---
apiVersion: getambassador.io/v3alpha1
kind: TLSContext
metadata:
name: istio-upstream
namespace: emissary
spec:
secret: istio-certs # This Secret name cannot currently be changed.
alpn_protocols: istio # This is required for Istio.
EOF

Once the TLSContext is created, a Mapping can use it for TLS origination. An example might be:

shell
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: mtls-mapping
spec:
hostname: my-secret-host.example.com # Make sure this matches a configured Host!
prefix: /my-secret-mapping/
tls: istio-upstream

This Mapping will use mTLS when communicating with its upstream service.

Route to Services Using mTLS

After integrating Emissary-ingress with Istio, Emissary-ingress's feature-rich routing capabilities and Istio's mTLS and observability are all available for all incoming traffic. To take full advantage of both, you need to:

  • configure upstream services with the Istio sidecar;
  • configure Mappings to use mTLS; and
  • verify your service port configuration.

Configure Upstream Services with the Istio Sidecar

Upstream services must have the Istio sidecar configured. The easiest way to arrange for this is to use Istio automatic sidecar injection as discussed above.

shell
kubectl label namespace default istio-injection=enabled

This will tell Istio to automatically inject the istio-proxy sidecar container into pods in this namespace.

Configure Mappings to Use mTLS

Traffic routing in Emissary-ingress is configured with the Mapping resource. This is a powerful configuration object that lets you configure different routing rules for different services.

To configure a Mapping to use mTLS, you need to use the tls element of the Mapping to tell it to originate TLS using the istio-upstream TLSContext above:

yaml
tls: istio-upstream

For example, if you have installed the Quote service as described on the Getting Started page, you will have a Mapping as follows:

yaml
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: quote-backend
spec:
hostname: "*"
prefix: /backend/
service: quote

To take advantage of Istio mTLS, update the above Mapping to originate TLS using the Istio mTLS certificates and to force access on port 80:

yaml
$ kubectl apply -f - <<EOF
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: quote-backend
spec:
hostname: "*"
prefix: /backend/
service: quote:80 # Be explicit about port 80. THIS IS IMPORTANT: see below
tls: istio-upstream # Originate TLS with the mTLS certificate
EOF

The behavior of your service will not seem to change, even though mTLS is active:

shell
$ curl -k https://{{AMBASSADOR_HOST}}/backend/
{
"server": "bewitched-acai-5jq7q81r",
"quote": "A late night does not make any sense.",
"time": "2020-06-02T10:48:45.211178139Z"
}

This request first went to Emissary-ingress, which routed it over an mTLS connection to the quote service in the default namespace. That connection was intercepted by the istio-proxy which authenticated the request as being from Emissary-ingress, exported various metrics, and finally forwarded it on to the actual quote service.

Configure Service Ports

When mTLS is active, Istio makes TLS connections to your services. Since Istio handles the TLS protocol for you, you don't need to modify your services however, the TLS connection will still use port 443 if you don't configure your Mappings to explicitly use port 80.

If your upstream service was not written to use TLS, its Service resource may only map port 80. If Istio attempts a TLS connection on port 443 when port 443 is not defined by the Service resource, the connection will hang even though the Istio sidecar is active, because Kubernetes itself doesn't know how to handle the connection to port 443.

As shown above, one simple way to deal with this situation is to explicitly specify port 80 in the Mapping's service:

yaml
service: quote:80 # Be explicit about port 80.

Another way is to set up your Kubernetes Service to map both port 80 and port 443. For example, the Quote (which listens on port 8080 in its pod) might use a Service like this:

yaml
apiVersion: v1
kind: Service
metadata:
name: quote
spec:
type: ClusterIP
selector:
app: quote
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 8080

Note that ports 80 and 443 are both mapped to targetPort 8080, where the service is actually listening. This permits Istio routing to work whether mTLS is active or not.

Enable Strict mTLS

Istio defaults to permissive mTLS, where mTLS is allowed between services, but not required. Configuring strict mTLS requires all connections within the cluster be encrypted. To switch Istio to use strict mTLS, apply a PeerAuthentication resource in each namespace that should operate in strict mode:

yaml
$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: global
namespace: istio-system
spec:
mtls:
mode: STRICT
EOF

To test strict mTLS, remove the tls configuration from the quote-backend Mapping and send a request:

shell
$ kubectl apply -f - <<EOF
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
name: quote-backend
spec:
hostname: "*"
prefix: /backend/
service: quote
EOF
shell
$ curl -k https://{{AMBASSADOR_HOST}}/backend/
upstream connect error or disconnect/reset before headers. reset reason: connection termination

Make sure to restore the tls configuration when testing is complete!

Configure Prometheus Metrics Collection

By default, the Istio sidecar provides Prometheus metrics using prometheus.io annotations. To take advantage of these metrics, you must install Prometheus.

Configure Istio Distributed Tracing

The Istio sidecar also supports distributed tracing by default. To take advantage of this support, you need to:

  1. Install a tracing provider, for example Zipkin into your cluster.

  2. Add a TracingService to tell Emissary-ingress to send tracing to your tracing provider, for example:

    yaml
    ---
    apiVersion: getambassador.io/v3alpha1
    kind: TracingService
    metadata:
    name: tracing
    namespace: emissary
    spec:
    service: "zipkin:9411"
    driver: zipkin
    config: {}
    tag_headers:
    - ":authority"
    - ":path"

After adding a TracingService, restart Emissary-ingress for the configuration to take effect. Istio propagates the tracing headers automatically, allowing for end-to-end observability within the cluster.

FAQ

How to test Istio certificate rotation

By default, Istio mTLS certificates are valid for 90 days, but get rotated every day.

Emissary-ingress updates the mTLS certificates as they are rotated, so you don't need to worry about certificate expiration.

To test that Emissary-ingress is properly rotating certificates, shorten the TTL of the Istio certificates by setting the following environment variables in the istiod container in the istio-system Namespace:

yaml
env:
- name: DEFAULT_WORKLOAD_CERT_TTL
value: 30m
- name: MAX_WORKLOAD_CERT_TTL
value: 1h

This makes the certificates Istio issues expire in one hour so testing certificate rotation is much easier.