The incident has been resolved. All services are fully operational and stable on the new hardware. Monitoring data collection has resumed normally.
We apologize for the disruption and the false positive alerts during recovery. All the wrongly created incidents have been deleted.
Thank you for your patience. A post-incident review is underway to improve our infrastructure resilience.
Resolved
The incident has been resolved. All services are fully operational and stable on the new hardware. Monitoring data collection has resumed normally.
We apologize for the disruption and the false positive alerts during recovery. All the wrongly created incidents have been deleted.
Thank you for your patience. A post-incident review is underway to improve our infrastructure resilience.
Monitoring
All services have been fully restored. The primary server is back online and monitoring, alerting, dashboard, and API are all operational.
You may have received false positive alerts during the recovery, we sincerely apologize for the noise. Please disregard any alerts triggered between 11:00 CET today and 13:20 CET. There will be a gap in monitoring data for this period.
We are actively monitoring the situation to ensure stability on the new hardware. We will be conducting a post-incident review to improve our infrastructure resilience and prevent similar outages in the future.
Thank you for your patience and we apologize for the disruption.
Identified
Hetzner has identified a hardware failure and replaced the server. Our drives have been transferred to the new hardware. However, the server is currently unreachable due to a network configuration issue on the new machine. We are working with Hetzner to restore network connectivity.
All monitoring agents will automatically reconnect once the service is back online, no action is needed on your end. We will post another update as soon as we have network access and can bring services back up.
Investigating
We are currently experiencing a complete service outage affecting all fivenines.io services, including server monitoring, alerting, the dashboard, and the API.
The outage is caused by a hardware failure on our primary server hosted at Hetzner. The server became unresponsive on February 26 at approximately 11:00 CET and is currently unable to boot. We have opened an urgent support ticket with Hetzner for hardware diagnosis and repair.
What this means for you:
Monitoring data is not being collected during this outage
Alerts will not fire until service is restored
The dashboard and API are unreachable
Your monitoring agents will automatically reconnect and resume reporting once the service is back online, no action is needed on your end
Please ignore all emails you're receiving about your monitored instances, the web/API server does not receive metrics anymore making daemon notifiers sending false positives.
We are actively working with Hetzner to resolve this as quickly as possible and will post updates here as we have them. We sincerely apologize for the disruption.