Understanding Envoy Proxy HTTP Access Logs
March 22, 2021 | 4 min read
Table of contents
Edge Stack API Gateway uses Envoy Proxy as its core L7 routing engine. Envoy Proxy provides a configurable access logging mechanism. Edge Stack uses the default format string for Envoy’s access logs. These access logs provide an extensive amount of information that can be used to troubleshoot issues.
Reading Edge Stack Access Logs
You can read the log file using
(⎈ | gke:default)$ kubectl logs ambassador-796cb689d9-jsptz ambassadorACCESS [2019-08-22T09:14:59.499Z] "GET /.ambassador-internal/openapi-docs HTTP/1.1" 200 - 0 989 5 1 "10.52.2.21" "Go-http-client/1.1" "bc391742-0ad4-4f0d-9e00-6e81266a1480" "ambassador" "10.55.253.138:5000"ACCESS [2019-08-22T09:14:59.506Z] "GET /callback/.ambassador-internal/openapi-docs HTTP/1.1" 503 UH 0 19 3 - "10.52.2.21" "Go-http-client/1.1" "c8d23a4b-c203-468a-abfb-ef47aca58e23" "ambassador" "-"...
Edge Stack API Gateway access log format
Let’s dissect each entry.
The start time of the request.
The HTTP method used for the request.
The original HTTP path requested by the client.
Either HTTP/1.1 or HTTP/2. If the protocol is TCP, the value will be
The HTTP response code. If the request is a TCP request, the value will be
These provide additional details about the response or connection if any above and beyond the standard response code. Possible values for HTTP and TCP requests include
(no healthy upstream hosts);
(upstream connection failure);
(no route configured);
(rejected because of upstream retry limit or maximum connection attempts reached). For HTTP requests, an additional set of values are possible, including:
- downstream connection termination
- Local service failed health check request
- Upstream request timeout
- Connection local reset
- Upstream remote reset
- Upstream connection termination
- The request processing was delayed for a period specified via fault injection. Note that Edge Stack does not currently support fault injection.
- The request was aborted with a response code specified via fault injection. Note that Edge Stack does not currently support fault injection.
- The request was ratelimited locally by the rate limiting filter.
- The request was denied by the external authorization service.
- The request was rejected because there was an error in rate limit service.
- The request was rejected because it set an invalid value for a strictly-checked header in addition to 400 response code.
- Stream idle timeout in addition to 408 response code.
- The downstream request had an HTTP protocol error.
- The upstream response had an HTTP protocol error.
- The upstream request reached max stream duration.
Bytes Received / Bytes Sent
The body bytes received or sent. For WebSocket connections, the Bytes Sent will include response header bytes.
The total duration, in milliseconds, of the request from the start time to the first byte read from the upstream host.
Upstream Service Time
Upstream Service Time
The time, in milliseconds, spent by the upstream host processing the request. This is useful if you want to compare the service time compared to network latency.
The XFF HTTP header field identifies the originating IP address of the client. Edge Stack enables XFF by default.
The user agent string, which allows the server to identify the specific type of software request agent.
header is used by Envoy to uniquely identify each request. This is especially important for distributed tracing and stable access logging across multiple microservices.
Host (or Authority)
The value of the
The upstream host URL, i.e., the target destination for the request.
For more details about the access log configuration, see the Envoy Proxy access log documentation. Thanks to Megan O’Keefe for her original tweet about Envoy access logs in Istio.
You’ll see some strong similarities between Istio and Edge Stack access logs (after all, both are based on Envoy Proxy). That said, there are some subtle differences as Edge Stack is solely an edge gateway, while Istio is a broader mesh (what’s the difference?).