Inaccessible APM nodes within GCP and Azure.
Incident Report for ESS (Public)
Resolved
We confirm that issue was resolved by actions mentioned in previous updates; thank you for your patience.
Posted Nov 22, 2019 - 10:59 UTC
Monitoring
The code release has been reverted, and the known symptoms of the issue are remediated. We are currently monitoring systems closely to confirm that services are stable and no further action is required. Please expect the next update within 1h.
Posted Nov 22, 2019 - 09:34 UTC
Update
We've noticed an increased error rate when connecting to APM hosts. Customers with deployments within GCP and Azure may be noticing failures when connecting to APM nodes.

We're currently working on reverting the recent release and expect connectivity to be restored once this has been reverted. We expect this to take several hours to be completely reverted and will keep this page updated as we progress.

We'll update this page in the next hour, or as the situation changes.
Posted Nov 22, 2019 - 06:52 UTC
Identified
An ongoing release is causing APM nodes to be reported as unhealthy within the user console. Customers may notice cluster nodes being reported as unhealthy in the user console, and may experience errors within the console when viewing the details of APM clusters.

Monitoring indicates that all cluster nodes are healthy and accepting requests, with no data being lost.

We're working on mitigating the impacted deployments and will provide an update in the next 30 minutes.
Posted Nov 22, 2019 - 06:26 UTC
This incident affected: Global services (Cloud console), GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3, APM connectivity: GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1, APM connectivity: GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2, APM connectivity: GCP europe-west2), GCP N. Virginia (us-east4) (APM connectivity: GCP us-east4, Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east4), GCP Tokyo (asia-northeast1) (APM connectivity: GCP asia-northeast1, Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast1), GCP Oregon (us-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1, APM connectivity: GCP us-west1), GCP Sydney (australia-southeast1) (APM connectivity: GCP australia-southeast1, Deployment orchestration (Create/Edit/Restart/Delete): GCP australia-southeast1), and GCP Iowa (us-central1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1, APM connectivity: GCP us-central1).