Docsright arrowEmissary-ingressright arrowRunning and deployment

16 min • read

Running and deployment

This section is intended for operators running Emissary-ingress, and covers various aspects of deploying and configuring the Emissary-ingress in production.

Emissary-ingress and Kubernetes

Emissary-ingress relies on Kubernetes for reliability, availability, and scalability. This means that features such as Kubernetes readiness and liveness probes, rolling updates, and the Horizontal Pod Autoscaling should be utilized to manage Emissary-ingress.

Default configuration

The default configuration of Emissary-ingress includes default resource limits, as well as readiness and liveness probes. These values should be adjusted for your specific environment.

Accessing the Admin Endpoint

Emissary-ingress exposes access to Envoy's admin endpoint which can be used to set runtime flags, dump the envoy config, and much more.

  1. Port-forward to a Emissary-ingress pod: kubectl port-forward -n ambassador <POD_NAME> 8001:8001

  2. In a web browser, visit

Dump the Envoy Config

  1. Visit

    • If you are in Firefox or a similar browser, click on the Raw Data button at the top
  2. Right click the page and save the contents as something like config_dump.json so you can explore it using an editor of your choice.

  3. There are plenty of other functions in Envoy's Admin interface you can take advantage of while the port-forward is active if you are comfortable exploring and debugging Envoy Proxy. Visit in a web browser to check out some of the other functions.

Running as non-root

Starting with Emissary-ingress 0.35, we support running Emissary-ingress as non-root. This is the recommended configuration, and will be the default configuration in future releases. We recommend you configure Emissary-ingress to run as non-root as follows:

  • Have Kubernetes run Emissary-ingress as non-root. This may happen by default (e.g., OpenShift) or you can set a securityContext in your Deployment as shown below in this abbreviated example:
  • Set the service_port element in the ambassador Module to 8080 (cleartext) or 8443 (TLS). This is the port that Emissary-ingress will use to listen to incoming traffic. Note that any port number above 1024 will work; Emissary-ingress will use 8080/8443 as its defaults in the future.

  • Make sure that incoming traffic to Emissary-ingress is configured to route to the service_port. If you're using the default Emissary-ingress configuration, this means configuring the targetPort to point to the service_port above.

  • If you are using redirect_cleartext_from, change the value of this field to point to your cleartext port (e.g., 8080) and set service_port to be your TLS port (e.g., 8443).

Changing the configuration directory

While running, Emissary-ingress needs to use a directory within its container for generated configuration data. Normally this is /ambassador, but in some situations - especially if running as non-root - it may be necessary to change to a different directory. To do so, set the environment variable AMBASSADOR_CONFIG_BASE_DIR to the full pathname of the directory to use, as shown below in this abbreviated example:

With AMBASSADOR_CONFIG_BASE_DIR set as above, Emissary-ingress will create and use the directory /tmp/ambassador-config for its generated data. (Note that, while the directory will be created if it does not exist, attempts to turn an existing file into a directory will fail.)

Running as DaemonSet

Emissary-ingress can be deployed as a DaemonSet to have one pod per node in a Kubernetes cluster. This setup is especially helpful when you have a Kubernetes cluster running on a private cloud.

  • In an ideal example scenario, you are running containers on Kubernetes alongside with your non-containerized applications running exposed via VIP using BIG-IP or similar products. In such cases, east-west traffic is routed based on iRules to certain a set of application pools consisting of application or web servers. In this setup, alongside traditional application servers, two or more Emissary-ingress pods can also be part of the application pools. In case of failure there is at least one Emissary-ingress pod available to BIG-IP that can take care of routing traffic to the Kubernetes cluster.

  • In manifest files kind: Deployment needs to be updated to kind: DaemonSet and replicas should be removed in spec section.


Emissary-ingress supports multiple namespaces within Kubernetes. To make this work correctly, you need to set the AMBASSADOR_NAMESPACE environment variable in Emissary-ingress's container. By far the easiest way to do this is using Kubernetes' downward API (this is included in the YAML files from

Given that AMBASSADOR_NAMESPACE is set, Emissary-ingress Mappings can operate within the same namespace, or across namespaces. Note well that Mappings will have to explicitly include the namespace with the service to cross namespaces; see the Mapping documentation for more information.

If you want Emissary-ingress to only work within a single namespace, set AMBASSADOR_SINGLE_NAMESPACE as an environment variable.

With Emissary-ingress, if you set AMBASSADOR_NAMESPACE or AMBASSADOR_SINGLE_NAMESPACE, set it in the deployment container.

If you want to set a certificate for your TLScontext from another namespace, you can use the following:


Emissary-ingress supports running multiple Emissary-ingresses in the same cluster, without restricting a given Emissary-ingress to a single namespace. This is done with the AMBASSADOR_ID setting. In the ambassador Module, set the ambassador_id, e.g.,

Then, assign each Emissary-ingress pod a unique AMBASSADOR_ID with the environment variable as part of your deployment:

With Emissary-ingress, if you set AMBASSADOR_ID, you will need to set it in the deployment container.

Emissary-ingress will then only use YAML objects that include an appropriate ambassador_id attribute. For example, if Emissary-ingress is given the ID ambassador-1 as above, only the first two YAML objects below will be used:

ambassador_id is always a list, and may (as shown in mapping-used-2 above) include multiple IDs to allow a given object in the configuration for multiple Emissary-ingress instances. In this case, mapping-used-2 will be included in the configuration for ambassador-1 and also for ambassador-2.

Note well that any Emissary-ingress configuration resource can have an ambassador_id included so, for example, it is fully supported to use ambassador_id to qualify the ambassador Module, TLSContext, and AuthService objects. You will need to set ambassador_id in all resources you want to use for Emissary-ingress.

If no AMBASSADOR_ID is assigned to an Emissary-ingress, it will use the ID default. If no ambassador_id is present in a YAML object, it will also use the ID default.


Emissary-ingress supports running side-by-side with other envoy-based projects in a single pod. An example of this is running with an istio sidecar. This is done with the AMBASSADOR_ENVOY_BASE_ID environment variable as part of your deployment:

If no AMBASSADOR_ENVOY_BASE_ID is provided then it will use the ID 0. For more information on the Envoy base-id option please see the Envoy command line documentation.


By default, Emissary-ingress will verify the TLS certificates provided by the Kubernetes API. In some situations, the cluster may be deployed with self-signed certificates. In this case, set AMBASSADOR_VERIFY_SSL_FALSE to true to disable verifying the TLS certificates.


If AMBASSADOR_UPDATE_MAPPING_STATUS is set to the string true, Emissary-ingress will update the status of every Mapping CRD that it accepts for its configuration. This has no effect on the proper functioning of Emissary-ingress itself, and can be a performance burden on installations with many Mappings. It has no effect for Mappings stored as annotations.

The default is false. We recommend leaving AMBASSADOR_UPDATE_MAPPING_STATUS turned off unless required for external systems.


Setting AMBASSADOR_LEGACY_MODE to true will result in Emissary-ingress disabling certain features and reverting to older codepaths which may be better preserve certain older behaviors. Legacy mode currently has the following effects:

  • Emissary-ingress will switch back to the Emissary-ingress 1.6 input-resource validator (which can significantly increase configuration latency for Emissary-ingress installations with many resources).
  • Emissary-ingress will use the shell boot sequence that was the default up through 1.9.1, rather than the Golang boot sequence that became the default in 1.10.0.
  • AMBASSADOR_FAST_RECONFIGURE (see below) is not supported in legacy mode.


Setting AMBASSADOR_FAST_RECONFIGURE to "true" enables incremental reconfiguration. When enabled, Emissary-ingress will track deltas from one configuration to the next and recalculate only what is necessary to follow the change. When disabled (the default), Emissary-ingress will recompute the entire configuration at every change.


Configuration from the filesystem

If desired, Emissary-ingress can be configured from YAML files in the directory $AMBASSADOR_CONFIG_BASE_DIR/ambassador-config (by default, /ambassador/ambassador-config, which is empty in the images built by Datawire). You could volume mount an external configuration directory here, for example, or use a custom Dockerfile to build configuration directly into a Docker image.

Note well that while Emissary-ingress will read its initial configuration from this directory, configuration loaded from Kubernetes annotations will replace this initial configuration. If this is not what you want, you will need to set the environment variable AMBASSADOR_NO_KUBEWATCH so that Emissary-ingress will not try to update its configuration from Kubernetes resources.

Also note that the YAML files in the configuration directory must contain the Emissary-ingress resources, not Kubernetes resources with annotations.

Log levels and debugging

The Emissary-ingress and Ambassador Edge Stack support more verbose debugging levels. If using Emissary-ingress, the diagnostics service has a button to enable debug logging. Be aware that if you're running Emissary-ingress on multiple pods, the debug log levels are not enabled for all pods -- they are configured on a per-pod basis.

If using Ambassador Edge Stack, you can adjust the log level by setting the AES_LOG_LEVEL environment variable; from least verbose to most verbose, the valid values are error, warn/warning, info, debug, and trace; the default is info.

Log format

By default, Emissary-ingress writes its own logs in a plain text format. To produce logs as JSON instead, set the AMBASSADOR_JSON_LOGGING environment variable:

Port assignments

Emissary-ingress uses some TCP ports in the range 8000-8499 internally, as well as port 8877. Third-party software integrating with Emissary-ingress should not use ports in this range on the Emissary-ingress pod.

Emissary-ingress usage telemetry (Scout)

Emissary-ingress integrates Scout, a service that periodically checks with Ambassador Labs servers to send anonymized usage data and the Emissary-ingress version. This information is important to us as we prioritize test coverage, bug fixes, and feature development. Note that Emissary-ingress will run regardless of the status of Scout (i.e., our uptime has zero impact on your uptime.)

We do not recommend you disable Scout. This check can be disabled by setting the environment variable SCOUT_DISABLE to 1 in your Emissary-ingress deployment.

Each Emissary-ingress installation generates a unique cluster ID based on the UID of its Kubernetes namespace and its Emissary-ingress ID: the resulting cluster ID is a UUID which cannot be used to reveal the namespace name nor Emissary-ingress ID itself. Emissary-ingress needs RBAC permission to get namespaces for this purpose, as shown in the default YAML files provided by Datawire; if not granted this permission it will generate a UUID based only on the Emissary-ingress ID. To disable cluster ID generation entirely, set the environment variable AMBASSADOR_CLUSTER_ID to a UUID that will be used for the cluster ID.

Unless disabled, Emissary-ingress will also report the following anonymized information back to Datawire:

cluster_countinttotal count of clusters in use
cluster_grpc_countintcount of clusters using GRPC upstream
cluster_http_countintcount of clusters using HTTP or HTTPS upstream
cluster_routing_envoy_rh_countintcount of clusters routing using Envoy ring_hash
cluster_routing_envoy_maglev_countintcount of clusters routing using Envoy maglev
cluster_routing_envoy_lr_countintcount of clusters routing using Envoy least_request
cluster_routing_envoy_rr_countintcount of clusters routing using Envoy round_robin
cluster_routing_kube_countintcount of clusters routing using Kubernetes
cluster_tls_countintcount of clusters originating TLS
custom_ambassador_idboolhas the ambassador_id been changed from 'default'?
custom_listener_portboolhas the listener port been changed from 80/443?
diagnosticsboolis the diagnostics service enabled?
endpoint_grpc_countintcount of endpoints to which Emissary-ingress will originate GRPC
endpoint_http_countintcount of endpoints to which Emissary-ingress will originate HTTP or HTTPS
endpoint_routingboolis endpoint routing enabled?
endpoint_routing_envoy_rh_countintcount of endpoints being routed using Envoy ring_hash
endpoint_routing_envoy_maglev_countintcount of endpoints being routed using Envoy maglev
endpoint_routing_envoy_lr_countintcount of endpoints being routed using Envoy least_request
endpoint_routing_envoy_rr_countintcount of endpoints being routed using Envoy round_robin
endpoint_routing_kube_countintcount of endpoints being routed using Kubernetes
endpoint_tls_countintcount of endpoints to which Emissary-ingress will originate TLS
extauthboolis extauth enabled?
extauth_allow_bodyboolwill Emissary-ingress send the body to extauth?
extauth_host_countintcount of extauth hosts in use
extauth_protostrextauth protocol in use ('http', 'grpc', or null if not active)
group_canary_countintcount of Mapping groups that include more than one Mapping
group_countinttotal count of Mapping groups in use (length of the route table)
group_header_match_countintcount of groups using header matching (including host and method)
group_host_redirect_countintcount of groups using host_redirect
group_host_rewrite_countintcount of groups using host_rewrite
group_http_countintcount of HTTP Mapping groups
group_precedence_countintcount of groups that explicitly set the precedence of the group
group_regex_header_countintcount of groups using regex header matching
group_regex_prefix_countintcount of groups using regex prefix matching
group_resolver_consulintcount of groups using the Consul resolver
group_resolver_kube_endpointintcount of groups using the Kubernetes endpoint resolver
group_resolver_kube_serviceintcount of groups using the Kubernetes service resolver
group_shadow_countintcount of groups using shadows
group_shadow_weighted_countintcount of groups using shadows but not shadowing all traffic
group_tcp_countintcount of TCP Mapping groups
host_countintcount of Host resources in use
k8s_ingress_class_countintcount of IngressClass resources in use
k8s_ingress_countintcount of Ingress resources in use
listener_countintcount of active listeners (1 unless redirect_cleartext_from or TCP Mappings are in use)
liveness_probeboolare liveness probes enabled?
managed_bystringtool that manages the Emissary-ingress deployment, if any (e.g. helm, edgectl, etc.)
mapping_countintcount of Mapping resources in use
ratelimitboolis rate limiting in use?
ratelimit_custom_domainboolhas the rate limiting domain been changed from 'ambassador'?
ratelimit_data_plane_protoboolis rate limiting using the data plane proto?
readiness_probeboolare readiness probes enabled?
request_4xx_countintlower bound for how many requests have gotten a 4xx response
request_5xx_countintlower bound for how many requests have gotten a 5xx response
request_bad_countintlower bound for how many requests have failed (either 4xx or 5xx)
request_elapsedfloatseconds over which the request_ counts are valid
request_hr_elapsedstringhuman-readable version of request_elapsed (e.g. "3 hours 35 minutes 20 seconds"
request_ok_countintlower bound for how many requests have succeeded (not a 4xx or 5xx)
request_total_countintlower bound for how many requests were handled in total
statsdboolis StatsD enabled?
server_nameboolis the server_name response header overridden?
service_resource_totalinttotal count of service resources loaded from all discovery sources
tls_origination_countintcount of TLS origination contexts
tls_termination_countintcount of TLS termination contexts
tls_using_contextsboolare new TLSContext resources in use? ?
tls_using_moduleboolis the old TLS module in use
tracingboolis tracing in use?
tracing_driverstrtracing driver in use ('zipkin', 'lightstep', 'datadog', or null if not active)
use_proxy_protoboolis the PROXY protocol in use?
use_remote_addressboolis Emissary-ingress honoring remote addresses?
x_forwarded_proto_redirectboolis Emissary-ingress redirecting based on X-Forwarded-Proto?
xff_num_trusted_hopsintwhat is the count of trusted hops for X-Forwarded-For?

The request_* counts are always incremental: they contain only information about the last request_elapsed seconds. Additionally, they only provide a lower bound -- notably, if an Emissary-ingress pod crashes or exits, no effort is made to ship out a final update, so it's very easy for counts to never be reported.

To completely disable feature reporting, set the environment variable AMBASSADOR_DISABLE_FEATURES to any non-empty value.