Open letter to Waze on infrastructure

Moderators: krankyd, Unholy

Open letter to Waze on infrastructure

Postby AlanOfTheBerg » Fri Oct 21, 2011 3:05 pm

I just sent this off to alpha this morning. This is no big deal. I'm sure others have sent similar thoughts. Just figured I'd post this and perhaps Waze can respond here.
Friends,

I am a Waze fan and supporter. I and many other spend an inordinate amount of time improving the maps and requesting changes meant to improve Waze. You probably hear from me more often that you like. :)

One thing we notice, and I know you know about it too, is that the routing and/or map server infrastructure software is unstable. Whenever the Israeli weekend or a holiday rolls around, it is a running joke that routing problems and map editing errors and slowness increase exponentially during those times. It's a joke, but it's not really funny. We should not be able to predict the days that routing errors and timeouts will increase.

I work in IT, so I know about these kinds of things. Whenever a problem rears its head (a problem is a series of incidents), a cross-functional team is put together and they spend their entire days and sometimes weeks to thoroughly research the problem and make and execute plans to bring that problem or problems to an end. Sometimes it is hardware, sometimes OS or patching, sometimes database settings tweaks or patching. Whatever it is, we dedicate resources to eradicating that problem. It takes time, it takes money, it takes discipline and it takes patience.

I humbly request that Waze do the same to get the back-end stable. I know you keep advertising how many users you have now, but I know from my own experience with friends, that number of active Wazers is likely a small percentage of the accounts created by users. They like it at first, but then run into issues with ridiculous routing and/or routing errors or routing timeouts and they just put it away, delete it and don't look back. I don't want that to happen, but it does and I understand why it does.

I understand you need to get Android 3.0 out the door. Got it. After that, before enacting more features in Papyrus or whatever else is next on the list, I suggest it be put on hold or delayed. Put together that team to research why routing or map editing is really only stable during the week, and fix it once and for all. Waze needs to be stable and fast. Routing timeouts should never happen due to a server issue, only network. Alternate routes should return to the user within 5 seconds, just like other routing apps.

If you aren't getting app updates out during this time, or getting the latest Papyrus upgrades into production and available to map editors, I think we can live with that. And I think you can too. The positive effects will be to retain more Waze users, which increases the attractiveness of your traffic data to potential clients and also to investors. I would much rather wait an extra six weeks for Papyrus to get a new feature if I can get a rock solid routing and mapping infrastructure out of that time.

Thanks for taking the time to read and consider this request.
-Alan
Wiki Resources: Map Editing Manual | Oregon Project/To-Do List
Oregon-based US Country Manager | iPhone5 - VZ - iOS 7.0.6 | Waze v3.7.9.992
AlanOfTheBerg
Waze Global Champs
Waze Global Champs
 
Posts: 19485
Joined: Sat Aug 28, 2010 8:48 pm
Location: US Country Manager - Oregon, USA
Has thanked: 653 times
Been thanked: 2253 times

Re: Open letter to Waze on infrastructure

Postby AlanOfTheBerg » Fri Oct 21, 2011 10:53 pm

No. This "surge" shouldn't, if Waze has their infrastructure properly diversified and scaled, be a reason to turn stuff off. This "surge" shouldn't go away. They want and need more users. This should be the new normal with thousands of new users daily and not going away due to poor service.

This is more of a band-aid, and though I know Fej didn't mean it to sound this way, it reads as if they hope the new users don't stick around so they can turn stuff back on again.
Wiki Resources: Map Editing Manual | Oregon Project/To-Do List
Oregon-based US Country Manager | iPhone5 - VZ - iOS 7.0.6 | Waze v3.7.9.992
AlanOfTheBerg
Waze Global Champs
Waze Global Champs
 
Posts: 19485
Joined: Sat Aug 28, 2010 8:48 pm
Location: US Country Manager - Oregon, USA
Has thanked: 653 times
Been thanked: 2253 times

Re: Open letter to Waze on infrastructure

Postby AlanOfTheBerg » Fri Oct 21, 2011 11:00 pm

Yeah, I read the whole thing. I want them to succeed.
Wiki Resources: Map Editing Manual | Oregon Project/To-Do List
Oregon-based US Country Manager | iPhone5 - VZ - iOS 7.0.6 | Waze v3.7.9.992
AlanOfTheBerg
Waze Global Champs
Waze Global Champs
 
Posts: 19485
Joined: Sat Aug 28, 2010 8:48 pm
Location: US Country Manager - Oregon, USA
Has thanked: 653 times
Been thanked: 2253 times

Re: Open letter to Waze on infrastructure

Postby dmcconachie » Fri Oct 21, 2011 3:13 pm

Good letter, Alan. I could add several facets to it but I think that covers the broad problem. The sad thing is I don't think it will come as a surprise to them. I just hope that having secured the recent funding they can focus less on proving they can get numbers in and a little more on giving us a rock-steady infrastructure!
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Open letter to Waze on infrastructure

Postby dmcconachie » Fri Oct 21, 2011 3:35 pm

unwallflower wrote:The instability and unpredictability of Waze is by far my biggest frustration.

I caught a Facebook post from them this morning, saying that there may be a large influx of users due to their current high ranking on the app store. My first thought was: GREAT, we're going to have even more processing delays on top of the ones we're already having. They still seem concerned mainly about recruiting new users rather than retaining their most loyal.

Yeah that was my first reaction too. I've been harping on at them about quality v quantity for most of this year! Frustrating!
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Re: Open letter to Waze on infrastructure

Postby dmcconachie » Fri Oct 28, 2011 5:19 pm

leo_lau wrote:One thing I can't understand is how difficult to make waze stable. If we take internet banking system or insurance system as example, their systems have to take more data but without any error such as...'your money cannot be transfer, please try it later...' :lol:
I've been using waze more than a year and hope waze can use the US$30M to rebuild its infrastructure.

I would say waze has *significantly* more data throughput!
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Re: Re: Open letter to Waze on infrastructure

Postby dmcconachie » Fri Oct 28, 2011 8:16 pm

Timbones wrote:
dmcconachie wrote:I would say waze has *significantly* more data throughput!

I would say banks and insurance companies have *significantly* more money!

Very true but working for one I can assure very little of it goes towards customer IT!
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Open letter to Waze on infrastructure

Postby dmcconachie » Fri Oct 28, 2011 11:35 pm

Completely off topic but most banks' IT is outsourced now and the development of online banking would definitely be outsourced!
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Open letter to Waze on infrastructure

Postby dmcconachie » Sat Oct 29, 2011 12:22 am

You know it's already available outside the US right? Europe or even India. China won't be adding 4x the current load by any means.

http://www.wolframalpha.com/input/?i=po ... a+v+europe
dmcconachie • UK Country Admin • Forum Moderator • Former Global Champ
Nexus 4 • Android 4.3
UK WikiFB PageForum FAQCurrent status
dmcconachie
 
Posts: 6021
Joined: Sat May 22, 2010 11:44 pm
Location: Edinburgh, UK
Has thanked: 80 times
Been thanked: 260 times

Re: Open letter to Waze on infrastructure

Postby fvwazing » Fri Oct 28, 2011 11:23 pm

Also, financial systems are designed bottom up with integrity as well as scalability in mind, by experts for high-volume transactional systems.

That is probably not how Waze started. I would not be surprised if Waze started as a single CPU, that got replaced with a handfull of CPUs as the load got too high, then ever growing organically, somehow just coping with the need.

This can work only as long as growth in users and in CPUs needed is somehow lineairly correlated which means that 3 times the number of users need 3 times the number of servers. Probably this is not the case with Waze; there are many effects in a complex system like this that make that correlation NONlinear. For instance: the closer Wazers are on the map, the bigger the chance that an event reported by one Wazer forces rerouting by whole hordes of Wazers nearby. Routing that takes place in a map that gets ever more detailed over time (anyone *reduced* the number of streets in his area, lately?), that needs ever more time and power to maintain indexed, to create ever more square km of tiles. So - if the number of Wazers doubles, the number of servers may have to triple, or quadruple. You can tweak a system like that forever, adding an index here, an extra table there, but to make a significant gain in performance that way you will make the system more complex and prone to failure.

From what I gather, Waze has made architectural changes in the past. Currently the architecture is allowing it to maintain serverfarms in Amazone, thus keeping pace with a lineair growth. Technically, they may order a bunch of servers today and have them operational tomorrow - as long as there is a budget for that. You could create clones of machines, basically reducing the chance that one of them breaking can stop the whole system. Perfect. But there is a tradeoff; every machine you add into this can either create extra capacity, or extra redundancy - but not both. This is where the strategic decisions work out - going for more users, or for a reliable service. You will end up with more servers than google yet loosing out in reliability, as the more components a system has the more vulnerable it gets. So, you really should have double everything to be safe. Imagine the budget it takes!

Waze may go for capacity, but when lightning strikes we are left in the dark. 1 flashstorm at the wrong place and you are game.

When growth of users gets *exponential*, doubling every few month or so, there is no way the need for "Wazepower" can be satisfied with an exponential growth in serverhardware. It is very clear that something has to be done.

Personally, I think that Waze should take the crowdsourcing-concept to the next level. Imagine that Waze could go the SETI way and have its most calculation-intensive functions running on *YOUR* hardware. No, not in your iPhone, but on your PC. A screensaver that connects itself with Waze HQ for instructions on what to do, in the background, every time you leave your PC unattended for a few minutes. Waze could reduce its infrastructure to a central database for the map of the world (at several physical locations please) and the distribution of maptiles from the PCs to the clients, and hundreds? thousands? of PCs could do the hard work. If bandwidth is a problem, a peer2peer-style (think torrent) layer could take that worry away. Hey, you could earn Wazepoints with running that screensaver (or a daemon/service/whatever) based on the thruput of your machine. It is enough if a small percentage of Wazers participated in this scheme as 1 could calculate hundreds of rides over a day. There is your scalability!

That would really look like Waze 3.0. I would not mind waiting half a year or longer for a solution like that.
Last edited by fvwazing on Fri Oct 28, 2011 11:31 pm, edited 2 times in total.
Waze jezelf!
600K+ edits
AM for some 25.000 km² around Maastricht, the Netherlands
iPhone 3Gs / iOS 6.0.1
fvwazing
 
Posts: 3585
Joined: Sat Nov 14, 2009 2:48 pm
Has thanked: 405 times
Been thanked: 373 times

Next

Return to Website, Community, General

Who is online

Users browsing this forum: No registered users