As you probably know, we were able to restore the basic functionality of the waze application (client) for mobile after the Amazon Web Services outage which occurred on Sunday. We are still having issues with some of our server which will require further work before they are fully operational again.
Password reset - though the link on world.waze.com/forgot_password seems to be working properly, the password will not change. This should be resolved by tomorrow (Wednesday)
The daily procedures - such as points, ranks, archived routes, etc
Suggested routes (‘are you going to “X”?’)
We will keep on updating as issues are resolved.
thank you for your patience.
I suspect there is no timescale because Waze does not want to over commit. Also, they may not be sure how long this is going to take if they have not gone through this type of issue previously. I am sure they are working on resolving everything ASAP.
This is only speculation, but likely their servers shut down abnormally in the AWS data center when the power went out. The are probably in the process of restoring servers/services from some type of snap shots or backups. It is also possible that they have some database issue that require the rolling in or out of transaction logs. After restoring functionality they need to test. And I imagine there is data and communications passing between servers that require certain things to be in sync. I am sure Waze is having lots of fun with this…
Amazon stated it could up to 2 days to get all servers up and running again. I don’t know if the Waze guys are affected by this, or that they have other issues due to the downtime.
Likely they are. This could be another reason not to commit on a time that it will be fixed. If Amazon is doing some of the service restoration work, Waze has to wait until that’s done first.
If the issue resides mostly with Amazon, then Waze definitely has gone through it when AWS EC2 had a major failure in the North Virginia data center a few months ago which knocked services out for several days. Interestingly, the N Virginia EC2 had problems again just yesterday.
Looks like those were connectivity (network) issues that lasted about < 20 minutes around 8pm last night. I was editing at the time and didn’t notice any issues getting to Papyrus.
Started editing. Two minutes after I get a prompt to enter my credentials again. Now all I get is Internal Server Error U1 when trying to sign in. Suggestions?
“INTERNATIONAL WAZERS CAN NOW RESET PASSWORDS AGAIN
The password reset mechanism is now working again. So if you forgot your password and were logged out during the service issues we had in the last couple of days, you can now reset your password and use waze again.
If you had problems in the last couple of days with signing in to your account, please go to waze.com/forgot_password and reset the password for your account.
If you’re still having problems, please email us at alpha@waze.com”
"INTERNATIONAL USERS CAN NOW EDIT AGAIN
In both cartouche and the new map editor.
We’re not 100% back to normal yet but we’re getting there.
thanks for your patience!"
As a side note, they changed the page design of status.waze.com
I thought that was a good article, thanks for highlighting it. FWIW I do some work in business continuity and resilience. As the article implies, it’s possible to protect against these kinds of issues, but at a price. At the top level (Six Sigma, or 99.9999% availability) the protection comes at a price that’s usually only paid by government institutions and financial services companies who can cost justify it.
Looking how long it takes to revive Waze I have come to opinion that they don’t have a backups. Most of affected Amazon clients were already running in 24 hours.
This depends on a lot of factors. Waze is much more than just a static website. I bet the back-end is very complex. Waze is also still a fairly small company, maybe not quite a start up anymore. It’s my understanding that the infrastructure for Waze was created and Israel and replicated to build the US/Canada and World infrastructures. Who knows how well it was originally designed and how many of these early design issues are causing trouble today when they now > 5 million user accounts.
From the multitude of issues they have been having recently, seems likely that it’s not scaling as well as they need and not easy to support.
the blackout at Amazon WS ist almost one week ago - nothing much has changed since then. I only have started wazing 10 days ago and I’m still very excited about the idea but all the delays and errors just take the fun… I know, I don’t pay anything for this service and Waze is still a small company but though I expect at least a running service and proper information which is not given to us as European users so far. Rather tell us “world-Waze won’t be available for the next week” than keep us hoping whole day and checking and checking and checking again if there’s any progress in restoring the service.
Looking at this I wonder what will happen to the 25M US $ Microsoft has put into Waze in July. Probably they have to pay new programmers to suite waze’s code to the 5M users. It’s probably not made for handle such a lot of users…
Anyway,… We should give another chance to Waze and wait for 2 more weeks.
