Service instability due to network incident
Resolved - Due to an undocumented change in an operating system update shipped by one of our network equipment vendors, network devices in our Frankfurt datacenter experienced an unexpected partial failure.
This incident impacted primarily Proton Mail, with approximately 50% of users who were routed to the impacted datacenter experiencing intermittent downtime for approximately 1 hour. Due to redundant systems, no data or emails were lost, but some email delivery may have been delayed.
Incident report:
Because the failure was partial, it was not sufficient to trigger a failover. Due to the unique circumstances surrounding this failure, a significant amount of confusion led to a longer than usual delay before the infrastructure engineers on shift made the call to failover to an alternative site.
That restored services, with approximately 30 minutes of lingering low-level instability while load was rebalanced. Investigation that took place in parallel uncovered the undocumented operating system change in the network device update that was rolled out earlier this month. Impacted network devices were updated, and the Frankfurt datacenter brought back into production with no user impact.
Proton routinely conducts testing before rolling out software patches to our network equipment and rolls them out gradually.
Unfortunately, this problematic undocumented change was not discovered because it only created issues under specific load conditions (indeed, the new software had been running for weeks without issues).
We apologize for the longer than usual incident response time. In the coming days, we will be analyzing our response to this incident to reduce future reaction times.
Yeah, incredibly frustrating.
The only acknowledgement is from a volunteer mod on reddit that said an hour ago that “the team is aware and the status page will be updated shortly”.
The fact I had to dig around to find that is really not a pleasing experience.
It’s also a business decision. Many times companies will massage their verbiage and have a plan in place before they even change the status to “investigating” simply to appease when they have SLAs. It’s stupid, but that’s often the reason.
Maybe somehow the problem was triggered in a way that the status page didn’t automatically detect it (for example, mine still works)? I’m really grasping at straws with that one. If it isn’t automatic, it categorically needs to be; if it is automatic but missed what’s apparently a major outage, then it needs to be fixed.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !protonprivacy@lemmy.world
Empowering you to choose a better internet where privacy is the default. Protect yourself online with
Proton Mail, Proton VPN, Proton Calendar, Proton Drive. Proton Pass and SimpleLogin.
Proton Mail is the world’s largest secure email provider. Swiss, end-to-end encrypted, private, and free.
Proton VPN is the world’s only open-source, publicly audited, unlimited and free VPN. Swiss-based, no-ads, and no-logs.
Proton Calendar is the world’s first end-to-end encrypted calendar that allows you to keep your life private.
Proton Drive is a free end-to-end encrypted cloud storage that allows you to securely backup and share your files. It’s open source, publicly audited, and Swiss-based.
Proton Pass Proton Pass is a free and open-source password manager which brings a higher level of security with rigorous end-to-end encryption of all data (including usernames, URLs, notes, and more) and email alias support.
SimpleLogin lets you send and receive emails anonymously via easily-generated unique email aliases.
Me when I test something in production
It had signed me out of the Proton Mail App on Android, first time that’s ever occurred, not sure it’s related though?
Their status page has an update on what happened.
No problems with mine all day.
It must be regional. It’s been fine for me all day.
The iPhone app kept working for me, but the proton mail website was inaccessible for about 2 hours.
FYI my proton mail mail works.
Mine too
Same. Which is whatever, I’m more annoyed they haven’t updated their status page.
Yeah, I’m used to company status pages being the last to know.
Yeah, incredibly frustrating.
The only acknowledgement is from a volunteer mod on reddit that said an hour ago that “the team is aware and the status page will be updated shortly”.
The fact I had to dig around to find that is really not a pleasing experience.
Why does a status page need to be updated manually?
Servers could still be up and responding to pings, yet backend databases could be down.
Or it could be a caching problem with the status service.
It’s bad ways of handling your status page but it happens.
There’s also a insurmountable amount of potential issues to cover, not worth the automation
It depends on the services, but in the end it’s pretty easy to spoof handshake packets to see if a service on a server is still running.
nmap is a great example.
I meant on the logic side of things
It’s also a business decision. Many times companies will massage their verbiage and have a plan in place before they even change the status to “investigating” simply to appease when they have SLAs. It’s stupid, but that’s often the reason.
Maybe somehow the problem was triggered in a way that the status page didn’t automatically detect it (for example, mine still works)? I’m really grasping at straws with that one. If it isn’t automatic, it categorically needs to be; if it is automatic but missed what’s apparently a major outage, then it needs to be fixed.