All Systems Operational
Cluster Management Operational
Cluster Management Console Service Operational
Cluster Management API Operational
Cluster Orchestration ? Operational
Cluster Metrics Operational
Cluster Snapshots ? Operational
AWS Marketplace Operational
GCP us-central1 Operational
Cluster Connectivity: GCP us-central1 Operational
Kibana Connectivity: GCP us-central1 Operational
APM Connectivity: GCP us-central1 Operational
GCP us-west1 Operational
Cluster Connectivity: GCP us-west1 Operational
Kibana Connectivity: GCP us-west1 Operational
APM Connectivity: GCP us-west1 Operational
GCP europe-west1 Operational
Cluster Connectivity: GCP europe-west1 Operational
Kibana Connectivity: GCP europe-west1 Operational
APM Connectivity: GCP europe-west1 Operational
GCP europe-west3 Operational
Cluster Connectivity: GCP europe-west3 Operational
Kibana Connectivity: GCP europe-west3 Operational
APM Connectivity: GCP europe-west3 Operational
Google Cloud Platform ? Operational
Google Compute Engine ? Operational
Google Cloud Storage ? Operational
AWS N. Virginia (us-east-1) Operational
Cluster Connectivity: AWS us-east-1 Operational
AWS EC2 Health: us-east-1 ? Operational
Snapshot Storage Infrastructure (S3): us-east-1 ? Operational
Kibana Connectivity: AWS us-east-1 ? Operational
APM Connectivity: AWS us-east-1 Operational
AWS N. California (us-west-1) Operational
Cluster Connectivity: AWS us-west-1 Operational
AWS EC2 Health: us-west-1 ? Operational
Snapshot Storage Infrastructure (S3): us-west-1 ? Operational
Kibana Connectivity: AWS us-west-1 Operational
APM Connectivity: AWS us-west-1 Operational
AWS Ireland (eu-west-1) Operational
Cluster Connectivity: AWS eu-west-1 Operational
AWS EC2 Health: eu-west-1 ? Operational
Snapshot Storage Infrastructure (S3): eu-west-1 ? Operational
Kibana Connectivity: AWS eu-west-1 Operational
APM Connectivity: AWS eu-west-1 Operational
AWS Frankfurt (eu-central-1) Operational
Cluster Connectivity: AWS eu-central-1 Operational
AWS EC2 Health: eu-central-1 Operational
Snapshot Storage Infrastructure (S3): eu-central-1 Operational
Kibana Connectivity: AWS eu-central-1 Operational
APM Connectivity: AWS eu-central-1 Operational
AWS Oregon (us-west-2) Operational
Cluster Connectivity: AWS us-west-2 Operational
AWS EC2 Health: us-west-2 ? Operational
Snapshot Storage Infrastructure (S3): us-west-2 ? Operational
Kibana Connectivity: AWS us-west-2 Operational
APM Connectivity: AWS us-west-2 Operational
AWS São Paulo (sa-east-1) Operational
Cluster Connectivity: AWS sa-east-1 Operational
AWS EC2 Health: sa-east-1 ? Operational
Snapshot Storage Infrastructure (S3): sa-east-1 ? Operational
Kibana Connectivity: AWS sa-east-1 Operational
APM Connectivity: AWS sa-east-1 Operational
AWS Singapore (ap-southeast-1) Operational
Cluster Connectivity: AWS ap-southeast-1 Operational
AWS EC2 Health: ap-southeast-1 ? Operational
Snapshot Storage Infrastructure (S3): ap-southeast-1 ? Operational
Kibana Connectivity: AWS ap-southeast-1 Operational
APM Connectivity: AWS ap-southeast-1 Operational
AWS Sydney (ap-southeast-2) Operational
Cluster Connectivity: AWS ap-southeast-2 Operational
AWS EC2 Health: ap-southeast-2 ? Operational
Snapshot Storage Infrastructure (S3): ap-southeast-2 ? Operational
Kibana Connectivity: AWS ap-southeast-2 Operational
APM Connectivity: AWS ap-southeast-2 Operational
AWS Tokyo (ap-northeast-1) Operational
Cluster Connectivity: AWS ap-northeast-1 Operational
AWS EC2 Health: ap-northeast-1 ? Operational
Snapshot Storage Infrastructure (S3): ap-northeast-1 ? Operational
Kibana Connectivity: AWS ap-northeast-1 Operational
APM Connectivity: AWS ap-northeast-1 Operational
Heroku ? Operational
Elastic Maps Service ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jun 16, 2019

No incidents reported today.

Jun 15, 2019

No incidents reported.

Jun 14, 2019

No incidents reported.

Jun 13, 2019

No incidents reported.

Jun 12, 2019

No incidents reported.

Jun 11, 2019

No incidents reported.

Jun 10, 2019
Resolved - We have tested extensively over the weekend and monitored our failure rates. The rate of failures has dropped to our usual level and we are now resolving this incident. Thank you for your patience!
Jun 10, 15:02 UTC
Monitoring - We have completed the rollout to all production versions and confirmed that hot-warm deployment changes are now succeeding. We will monitor for the next day.
Jun 8, 00:28 UTC
Update - We have tested the fix in our staging environment and are working on getting it rolled out to production. We have confirmed that the rate of errors on deployment changes for hot-warm has decreased to normal levels. The production rollout is ongoing, and we will update you when it is completed.
Jun 7, 22:26 UTC
Identified - Identified:

This incident is impacting all hot-warm cluster deployment changes in AWS regions. Connectivity to clusters is not impacted.

The preliminary mitigation that we rolled out last evening was a timeout increase, which allowed more hot-warm plans to succeed. This gave us a little bit of breathing room to dig in deep into the code base and spend some time profiling thread contention, and we've had a breakthrough. We discovered that a networking method was writing a state file to disk with every run, rather than when it changed, causing an increased load on disk and additional thread contention. We have merged a fix and have begun the testing and rollout process.

We will update you in six hours with our progress.
Jun 7, 16:48 UTC
Update - Based on preliminary findings the team was able to conclude recent configuration change has improved the success rate of new deployment creations. We are still in process of determining the root cause of increased failure rate in plan configuration changes of hot-warm deployments.

We will provide an update within the next 6 hours.
Jun 7, 09:50 UTC
Update - We've deployed our interim mitigations which increase the thresholds causing hot-warm plan changes to fail. We're awaiting data on the effectiveness of these changes, but expect these to reduce the failure rates of hot-warm deployment changes.

We'll update this issue within the next 4 hours as data comes to hand.
Jun 7, 06:24 UTC
Investigating - We are observing an elevated error rate in Deployment changes for Elasticsearch clusters in a hot-warm setup.

Customers making changes to or creating new hot-warm deployments may be experiencing errors, shown in the Deployment Activity tab.

We have identified that containers holding the Elasticsearch node processes are taking longer than expected to start, exceeding a threshold that leads to marking deployment changes as failed.

A team of cloud engineers are working on an interim mitigation that addresses the effects of this issue, and we expect a decrease in failure rates soon.

A new update will be posted within 4 hours.
Jun 6, 19:23 UTC
Jun 9, 2019

No incidents reported.

Jun 7, 2019
Resolved - Based on the data gathered over the past hour we can conclude the connectivity between Kibana/APM and Elasticsearch clusters has been successfully restored. Customers should not experience any issues with connectivity to Kibana instances.

What happened: The Elastic Cloud Proxy tier is currently at the end of an implementation phase which has introduced a new Cloud Proxy for handling client requests. Part of the implementation phase involved using haproxy to conditionally route traffic between the old and new proxies so that we could introduce the change gradually and with least impact.
As part of the work to remove haproxy from this stack a configuration change was made to the AWS eu-central-1 region which resulted in inter-deployment (Kibana -> Elasticsearch; APM -> Elasticsearch) communication issues.

This change has now been reverted.
Jun 7, 13:03 UTC
Monitoring - Problematic proxy release has been successfully rolled back. Connectivity to Kibana instances should now be restored. We are currently monitoring the situation and will provide more details about the incident within the next 60 minutes.
Jun 7, 11:12 UTC
Update - Latest proxy release in AWS eu-central-1 region has caused issues with network connectivity from Kibana/APM to Elasticsearch. The release is being rolled back now. Customers might experience issues connecting to their Kibana instances. Functionality of APM might be affected.

Direct access to Elasticsearch clusters is not impacted.

We will provide status update within the next 60 minutes.
Jun 7, 11:02 UTC
Investigating - We are currently investigating this issue.
Jun 7, 10:47 UTC
Jun 6, 2019
Resolved - All affected components have been re-deployed with the correct versions. This incident has been resolved.
Jun 6, 06:44 UTC
Monitoring - The deployment of one of our API services has completed and users should no longer be experiencing any issues when interacting with deployments via the console. Executing requests via the API Console, and accessing deployment snapshots should be succeed.

We're still in the process of re-deploying the second API service, however there should be no further user impact whilst this completes.
Jun 6, 06:15 UTC
Update - We've reviewed the versions of other related components and have found another service which requires re-deployment. We're working on re-deploying this second component with the correct version, however this will delay the resolution of this incident.

We expect both components to be fully deployed within the next 45 minutes, and will update this incident at that time or as any changes come to hand.
Jun 6, 05:58 UTC
Identified - We've identified a version mismatch in some recently deployed components. We believe this to be causing the failures in executing requests via the API Console.

We're currently working on redeploying this component with the correct version, and anticipate this will be complete within the next 15 minutes.

We'll update this issue when this deployment is complete.
Jun 6, 05:38 UTC
Investigating - We're encountering issues executing requests against a deployment via the customer console. Users may experience errors when executing requests via the 'API Console' or when viewing, or restoring snapshots for a deployment.

There are no general cluster connectivity issues.

We're currently investigating this issue and will update this status page within the next hour, or as more information comes to hand.
Jun 6, 05:26 UTC
Jun 5, 2019

No incidents reported.

Jun 4, 2019

No incidents reported.

Jun 3, 2019

No incidents reported.

Jun 2, 2019

No incidents reported.