Unable to export to Monitering service because: GaxError RPC failed, caused by 3

Question

I have a Java Application in App Engine, and recently I started getting following error:

Unable to export to Monitering service because: GaxError RPC failed, caused by 3:One or more TimeSeries could not be written: Metrics cannot be written to gae_app. See https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource for a list of writable resource types.: timeSeries[0]

and this happens every time after health check log:

Health checks: instance=instanceName start=2020-01-14T14:28:07+00:00 end=2020-01-14T14:28:53+00:00 total=18 unhealthy=0 healthy=18

and after some time my instances would be restarted and the same thing starts to happen again.

app.yaml:

 #https://cloud.google.com/appengine/docs/flexible/java/reference/app-yaml

#General settings
runtime: java
api_version: '1.0'
env: flex
runtime_config:
  jdk: openjdk8
#service: service_name #Required if creating a service. Optional for the default service.

#https://cloud.google.com/compute/docs/machine-types
#Resource settings
resources:
  cpu: 2
  memory_gb: 6 #memory_gb = cpu * [0.9 - 6.5] - 0.4
#  disk_size_gb: 10 #default

##Liveness checks - Liveness checks confirm that the VM and the Docker container are running. Instances that are deemed unhealthy are restarted.
liveness_check:
  path: "/liveness_check"
  timeout_sec: 20         #1-300   Timeout interval for each request, in seconds.
  check_interval_sec: 30 #1-300   1-300Time interval between checks, in seconds.
  failure_threshold: 6   #1-10    An instance is unhealthy after failing this number of consecutive checks.
  success_threshold: 2   #1-10    An unhealthy instance becomes healthy again after successfully responding to this number of consecutive checks.
  initial_delay_sec: 300 #0-3600  The delay, in seconds, after the instance starts during which health check responses are ignored. This setting can allow an instance more time at deployment to get up and running.

##Readiness checks - Readiness checks confirm that an instance can accept incoming requests. Instances that don't pass the readiness check are not added to the pool of available instances.
readiness_check:
  path: "/readiness_check"
  timeout_sec: 10             #1-300      Timeout interval for each request, in seconds.
  check_interval_sec: 15      #1-300      Time interval between checks, in seconds.
  failure_threshold: 4       #1-10    An instance is unhealthy after failing this number of consecutive checks.
  success_threshold: 2       #1-10    An unhealthy instance becomes healthy after successfully responding to this number of consecutive checks.
  app_start_timeout_sec: 300 #1-3600  The maximum time, in seconds, an instance has to become ready after the VM and other infrastructure are provisioned. After this period, the deployment fails and is rolled back. You might want to increase this setting if your application requires significant initialization tasks, such as downloading a large file, before it is ready to serve.

#Service scaling settings
automatic_scaling:
  min_num_instances: 2
  max_num_instances: 3
  cpu_utilization:
    target_utilization: 0.7

Answer 1

The error is caused by an upgrade of the stackdriver logging sidecar to 1.6.25 version, which starts to push FluentD metrics to Stackdriver monitoring via OpenCensus. However the integration with App Engine Flex doesn't work yet.

These errors should be logs only. It is not relative to the health check logs. It should not impact VM restart. If your VM instances are restarted frequently, there may be caused by some other reason. In Stackdriver logging UI, you can search Free disk space under vm.syslog stream and unhealthy sidecars under vm.events stream. If some logs show up, your instances restart may be caused by low free disk size or any unhealthy sidecar containers.