Known issue: AWS instance can go into an unresponsive state after running for a few days

[peiwenhu]

We discovered that if the server runs on AWS for a few days it may get killed due to failing autoscaling health checks. When we disabled the health checks we noticed that the instance would be unresponsive, failing EC2 status checks and sshing to it times out.

Work is in progress to fix this.

[MarcoLugo]

Any updates on the progress/timeline for the fix? The context offered here seems rather concerning to us so any additional information would be helpful. Thank you.

[peiwenhu]

We think it is an out of memory issue due to the continuously ingested test data in that environment. memory consumption in TEE is a bit tricky so we're still in the process of getting concrete evidence.

[peiwenhu]

It turns out that we set Envoy logging verbosity level to DEBUG which generated too much logs and it used up our test machine disk over time. It is unrelated to the server code itself. The verbosity level is fixed.