What are the causes of various Envoy RESPONSE_FLAGS

Question

I'm trying to debug a mesh of services, with Envoy sat in the middle. The access logs are showing a lot of 503s, which vary in their %RESPONSE_FLAGS%. Other than the access logging docs, I can't find any explaination of what the flags actually mean.

Eg, pithily, NR means "I sent a 404 downstream, but not because upstream sent me one, but because I have no matching route, so I, envoy generated the 404". I'd love a one-liner like that for some of the other. Specially I'm struggling with (these all appear in our logs...)

• UR vs UC - I think these are "I sent a 503 downstream because I connected upstream but then there was a subsequent problem". What problem? Specifically, what do remote reset vs connection termination mean?
• LR connection local reset - what does this mean? Envoy sent a 503 downstream becuase it decided to stop talking to upstream, mid-way through? Why would it do that?
• <no flag> - am I right in assuming that if there's an error code like 404 or 503 and no RESPONSE_FLAGS, that that code was sent by upstream and is just being forwarded?

Answer 1

From the source code below,

• UR: UPSTREAM_REMOTE_RESET If a remote codec level reset was received on the stream.
• UC: UPSTREAM_CONNECTION_TERMINATION If the stream was locally reset due to connection termination.
• LR: If a local codec level reset was sent on the stream

/**
 * Stream reset reasons.
 */
enum class StreamResetReason {                                                                                                                                                                                                                                // If a local codec level reset was sent on the stream.
  LocalReset,
  // If a local codec level refused stream reset was sent on the stream (allowing for retry).
  LocalRefusedStreamReset,
  // If a remote codec level reset was received on the stream.
  RemoteReset,
  // If a remote codec level refused stream reset was received on the stream (allowing for retry).
  RemoteRefusedStreamReset,
  // If the stream was locally reset by a connection pool due to an initial connection failure.
  ConnectionFailure,
  // If the stream was locally reset due to connection termination.
  ConnectionTermination,
  // The stream was reset because of a resource overflow.
  Overflow,
  // Either there was an early TCP error for a CONNECT request or the peer reset with CONNECT_ERROR
  ConnectError,
  // Received payload did not conform to HTTP protocol.
  ProtocolError
};

StreamInfo::ResponseFlag
Filter::streamResetReasonToResponseFlag(Http::StreamResetReason reset_reason) {                                                                                                                                                                               switch (reset_reason) {
  case Http::StreamResetReason::ConnectionFailure:                                                                                                                                                                                                              return StreamInfo::ResponseFlag::UpstreamConnectionFailure;
  case Http::StreamResetReason::ConnectionTermination:
    return StreamInfo::ResponseFlag::UpstreamConnectionTermination;
  case Http::StreamResetReason::LocalReset:
  case Http::StreamResetReason::LocalRefusedStreamReset:
    return StreamInfo::ResponseFlag::LocalReset;
  case Http::StreamResetReason::Overflow:
    return StreamInfo::ResponseFlag::UpstreamOverflow;
  case Http::StreamResetReason::RemoteReset:
  case Http::StreamResetReason::RemoteRefusedStreamReset:
  case Http::StreamResetReason::ConnectError:
    return StreamInfo::ResponseFlag::UpstreamRemoteReset;
  case Http::StreamResetReason::ProtocolError:
    return StreamInfo::ResponseFlag::UpstreamProtocolError;
  }

  NOT_REACHED_GCOVR_EXCL_LINE;
}

constexpr static absl::string_view NONE = "-";                                                                                                                                                                                                              constexpr static absl::string_view DOWNSTREAM_CONNECTION_TERMINATION = "DC";                                                                                                                                                                                constexpr static absl::string_view FAILED_LOCAL_HEALTH_CHECK = "LH";                                                                                                                                                                                        constexpr static absl::string_view NO_HEALTHY_UPSTREAM = "UH";                                                                                                                                                                                              constexpr static absl::string_view UPSTREAM_REQUEST_TIMEOUT = "UT";                                                                                                                                                                                         constexpr static absl::string_view LOCAL_RESET = "LR";                                                                                                                                                                                                      constexpr static absl::string_view UPSTREAM_REMOTE_RESET = "UR";                                                                                                                                                                                            constexpr static absl::string_view UPSTREAM_CONNECTION_FAILURE = "UF";                                                                                                                                                                                      constexpr static absl::string_view UPSTREAM_CONNECTION_TERMINATION = "UC";                                                                                                                                                                                  constexpr static absl::string_view UPSTREAM_OVERFLOW = "UO";                                                                                                                                                                                                constexpr static absl::string_view UPSTREAM_RETRY_LIMIT_EXCEEDED = "URX";                                                                                                                                                                                   constexpr static absl::string_view NO_ROUTE_FOUND = "NR";                                                                                                                                                                                                   constexpr static absl::string_view DELAY_INJECTED = "DI";                                                                                                                                                                                                   constexpr static absl::string_view FAULT_INJECTED = "FI";                                                                                                                                                                                                   constexpr static absl::string_view RATE_LIMITED = "RL";                                                                                                                                                                                                     constexpr static absl::string_view UNAUTHORIZED_EXTERNAL_SERVICE = "UAEX";                                                                                                                                                                                  constexpr static absl::string_view RATELIMIT_SERVICE_ERROR = "RLSE";                                                                                                                                                                                        constexpr static absl::string_view STREAM_IDLE_TIMEOUT = "SI";                                                                                                                                                                                              constexpr static absl::string_view INVALID_ENVOY_REQUEST_HEADERS = "IH";                                                                                                                                                                                    constexpr static absl::string_view DOWNSTREAM_PROTOCOL_ERROR = "DPE";                                                                                                                                                                                       constexpr static absl::string_view UPSTREAM_MAX_STREAM_DURATION_REACHED = "UMSDR";                                                                                                                                                                          constexpr static absl::string_view RESPONSE_FROM_CACHE_FILTER = "RFCF";                                                                                                                                                                                     constexpr static absl::string_view NO_FILTER_CONFIG_FOUND = "NFCF";                                                                                                                                                                                         constexpr static absl::string_view DURATION_TIMEOUT = "DT";                                                                                                                                                                                                 constexpr static absl::string_view UPSTREAM_PROTOCOL_ERROR = "UPE";                                                                                                                                                                                         constexpr static absl::string_view NO_CLUSTER_FOUND = "NC";                                                                                                                                                                                                                                                                                                                                                                                                                                                             static constexpr std::array ALL_RESPONSE_STRING_FLAGS{                                                                                                                                                                                                          FlagStringAndEnum{FAILED_LOCAL_HEALTH_CHECK, ResponseFlag::FailedLocalHealthCheck},                                                                                                                                                                         FlagStringAndEnum{NO_HEALTHY_UPSTREAM, ResponseFlag::NoHealthyUpstream},                                                                                                                                                                                    FlagStringAndEnum{UPSTREAM_REQUEST_TIMEOUT, ResponseFlag::UpstreamRequestTimeout},                                                                                                                                                                          FlagStringAndEnum{LOCAL_RESET, ResponseFlag::LocalReset},                                                                                                                                                                                                   FlagStringAndEnum{UPSTREAM_REMOTE_RESET, ResponseFlag::UpstreamRemoteReset},                                                                                                                                                                                FlagStringAndEnum{UPSTREAM_CONNECTION_FAILURE, ResponseFlag::UpstreamConnectionFailure},                                                                                                                                                                    FlagStringAndEnum{UPSTREAM_CONNECTION_TERMINATION,                                                                                                                                                                                                                            ResponseFlag::UpstreamConnectionTermination},                                                                                                                                                                                             FlagStringAndEnum{UPSTREAM_OVERFLOW, ResponseFlag::UpstreamOverflow},                                                                                                                                                                                       FlagStringAndEnum{NO_ROUTE_FOUND, ResponseFlag::NoRouteFound},                                                                                                                                                                                              FlagStringAndEnum{DELAY_INJECTED, ResponseFlag::DelayInjected},                                                                                                                                                                                             FlagStringAndEnum{FAULT_INJECTED, ResponseFlag::FaultInjected},                                                                                                                                                                                             FlagStringAndEnum{RATE_LIMITED, ResponseFlag::RateLimited},                                                                                                                                                                                                 FlagStringAndEnum{UNAUTHORIZED_EXTERNAL_SERVICE, ResponseFlag::UnauthorizedExternalService},                                                                                                                                                                FlagStringAndEnum{RATELIMIT_SERVICE_ERROR, ResponseFlag::RateLimitServiceError},                                                                                                                                                                            FlagStringAndEnum{DOWNSTREAM_CONNECTION_TERMINATION,                                                                                                                                                                                                                          ResponseFlag::DownstreamConnectionTermination},                                                                                                                                                                                           FlagStringAndEnum{UPSTREAM_RETRY_LIMIT_EXCEEDED, ResponseFlag::UpstreamRetryLimitExceeded},                                                                                                                                                                 FlagStringAndEnum{STREAM_IDLE_TIMEOUT, ResponseFlag::StreamIdleTimeout},                                                                                                                                                                                    FlagStringAndEnum{INVALID_ENVOY_REQUEST_HEADERS, ResponseFlag::InvalidEnvoyRequestHeaders},                                                                                                                                                                 FlagStringAndEnum{DOWNSTREAM_PROTOCOL_ERROR, ResponseFlag::DownstreamProtocolError},                                                                                                                                                                        FlagStringAndEnum{UPSTREAM_MAX_STREAM_DURATION_REACHED,                                                                                                                                                                                                                       ResponseFlag::UpstreamMaxStreamDurationReached},                                                                                                                                                                                          FlagStringAndEnum{RESPONSE_FROM_CACHE_FILTER, ResponseFlag::ResponseFromCacheFilter},                                                                                                                                                                       FlagStringAndEnum{NO_FILTER_CONFIG_FOUND, ResponseFlag::NoFilterConfigFound},                                                                                                                                                                               FlagStringAndEnum{DURATION_TIMEOUT, ResponseFlag::DurationTimeout},                                                                                                                                                                                         FlagStringAndEnum{UPSTREAM_PROTOCOL_ERROR, ResponseFlag::UpstreamProtocolError},                                                                                                                                                                            FlagStringAndEnum{NO_CLUSTER_FOUND, ResponseFlag::NoClusterFound},                                                                                                                                                                                      };