Comments: "Infrastructure, infrastructure, infrastructure!"

Moderators: krankyd, Unholy

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Fri Nov 04, 2011 6:45 pm

noam wrote:Just to be clear - the routing server performance IS being worked on. We are working on better distribution and scaling of the server. We don't have an ETA on it but it obviously keeps us up at night (literally)

Noam,

it's not only the routing server performance. For months now I get no learning from the routing. Every time during the day Waze is trying to calculate the silly route to drive to KLCC in Kuala Lumpur, which isn't my daily and I only drive there once in a while. Yet, I drive almost daily to work and back home using probably 3-4 different routes depending on traffic. Guess what, Waze routing server is only giving me one route if I enter either work or home as target and it's in many cases not the fastest one.

I've checked the routes, they're all connected but it seems the learning has stopped. Probably due to the infrastructure as well.

But what is a routing software worth if it doesn't route? Still your no 1 problem is that you guys have retention rate similar to Zynga, hardly users stay longer than a couple of days, you're lucky if the stay more than 1 month.

That's what needs to be addressed as otherwise your potential prospects are all burned by a bad user experience in the first place. And I know quite a bit about that as I'm responsible for our 12+ million paying users here at the no 2 Telco in Malaysia, in my role as SVP of Customer Value Management. If we wouldn't had developed a proper welcome program, followed by a proper cross/upsell our clients would be fleeing in the same way as it happens with new Wazers. Where is your customer lifecycle program? How are you guiding the user from lifestage to lifestage? Where's your prediction of attrition from the user, do you react to changes and decline in daily usage? No, nothing (probably as you don't have anybody who a) has a clue about that and b) you didn't even know that it's important.

A large percentage of them is leaving as they don't find anything on the map, not their destination nor their current position. That's due to the map not well deployed. That's due because your company doesn't place enough emphasis on supporting new users by creating a welcome program. How do you react to the fact that a new user is starting at a location where there is no map yet? Where's your tutorial to react to this event and turn maybe the bulldozer on for him? How do you communicate at all to new users, to react to different events? Just giving them points for their first update request send isn't good enough. You have to react automatically to it with the right explanation/tutorial. Go and play any Zynga game on Facebook and you will understand how perfectly they take care of their new joiners and keep them connected to their game. What do you do?

Then your gamification approach isn't working for two reasons. One is the most obvious one, no points or not getting what is promised (+100 points) as the user fulfills his part (like first week you drive 2 days) is a complete turn-off point and as mentioned by dohartman is another main reason for leaving after just coming on board. The second thing is that incentifying with extrinsic motivators only works for a certain period of time (see Foursquare as the perfect example how quickly Badges and Leaderboard/Points wear off) and even leading to less usage/fun which is called overjustification. So you need to find a way that the intrinsic motivation (like for me building something that is potentially saving gasoline by being able to avoid jams and taking the fastest route, therefore saving our planet and some lives along the way as no jams means also less accidents) is replacing the extrinsic motivation to climb up your leaderboard. The intrinsic motivation is the pleasure of beating the obstacles on our way to home/work by using Waze to avoid the jams, being faster, being smarter than those not using it. Yet, you're not communicating to the user on that intrinsic motivation to connect him further to Waze (like "Congratulations, by using this route suggested by Waze rather than your standard route you arrived 5 minutes earlier!").

Lastly, focusing on crowdsourcing is a great idea but it's worth nothing if your users don't get it (hardly any of the new users understand that they are responsible for building the map) and if you're not focusing as well on helping your superusers (like us in the Top 15 worldwide with all more than 150000 edits) to build up that map. The same rule applies like in many other internet based apps (like Twitter eg), 90% is passive, 10% is active and less than 1% is doing 90% of the work!

But yet we're bothered on each of those 150000 edits with the software being buggy and not supporting us in how we spend our free, unpaid time! We still have to switch between the old Cartouche (for seeing update requests) and the new Papyrus as we eg clear stray roads left by bad GPS reception (as deleting them is much faster on Papyrus vs the one after the other deleting on Cartouche). Same for working on update requests, finding errors in the road segments.

Not even speaking about not giving us any proper communication tool to interact with the endusers, on their update request. Hence 90% of them is useless as there is no comment and we have no time to spent 5-10 minutes figuring out why something was reported (and often wrongly classified as eg missing bridge). I setup a Facebook page on "Waze in Malaysia" which has 177 engaged users but getting them there is a nightmare as your tool is missing the communication part between users (and don't tell me to use the PM system here, it's a nightmare as well).


You see Noam, there's a lot more that can be done, a lot more that can be done better but after now more than 2 years it's time to get your act together. Upgrading the infrastructure is actually hygiene, not even worth to talk about and it should work (but we accept some glitches as it's a free service). You have more than 4 million email addresses from former users (assuming all of them registered), that's the perfect basis to start your Customer Lifecycle Program!


Hope this lengthy post is helping making Waze better, I haven't fully given up on you guys after my rant a couple weeks ago but I almost stopped all work. I've overcome some of my frustration and I'm willing to help again in my sparse free time, as it somehow relaxes me from the stress at work to work on map edits (another intrinsic motivation BTW). I just don't know how long my patience lasts.

Cheers,

Andreas
Country Manager for Malaysia and Singapore (and Brunei it seems)
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sat Nov 05, 2011 5:47 am

andrewfatcat wrote:I have never been this satisfied with the current V3.0 client and Cartouche. V3.0 has never crashed and the new Cartouche really makes editing much easier.

There is only one thing I think that needs to be done as soon as possible: I would like to see Toll-road and Carpool lanes as new road types. It shouldn't be difficult to add just these two road types for map editing and we have been asking for this but it has not been done. The routing function to avoid them can be added later, but you got to have these road types first.

I disagree, what is the purpose of adding the new road types if you don't use them immediately in your routing?

It's a good idea to implement, but implement it in the right way and not half finished.

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sat Nov 05, 2011 5:52 am

gettingthere wrote:Andreas - excellent post!

You are absolutely correct that if Waze want's to be successful, they need to focus on retaining users. Of course the infrastructure working correctly is critical to keeping users engaged and using the product.

Also, great points regarding Waze needing to reach out to the 1% of the users doing 90% of the map editing work. The new Cartouche is great, but really needs to be finished up. The periodic slow downs in the editor as well as the very frustrating road layer not updating that occurs from time to time needs to be resolved. And as you say, the improved Update Request mechanism needs to be deployed - promised many times, but not yet delivered.

Patience is needed though. Waze needs to clear the nagging infrastructure issues that they are currently focusing on.

Thanks for the positive feedback and reinforcement. Yet I believe the infrastructure improvements are hygiene, anybody is expecting the server to be up and give him a route, not an error message it can't calculate right now.

A service that isn't working is worth nothing, we're experiencing this here in Malaysia with our massive broadband demand. You can't use it, it's worth nothing and will be worth nothing even if you give it for free, that's feedback from our customer but I think Waze users think the same.

Problem is that people working on the Internet think that acquisition is basically 7 billion prospects, the total world population. It's not and I do think that after two years or more in operation Waze could have been way more successful if they had intended to work on retaining customer from the beginning on, not just focusing on acquiring them and then do nothing when they lose them after 5 minutes.

But Noam is the CEO and making the calls here

Just my two cents worth,

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sat Nov 05, 2011 5:55 am

shmupi2 wrote:Well done Andreas !

I believe that you expressed our feelings as well!!!

I've posted a link in the Israeli forum here :

http://www.waze.co.il/forum/viewtopic.php?f=31&t=12679

Thanks,
Shmupi

Shalom Shmupi,


many thanks for your encouraging positive feedback. Would love to have such a Waze loving community like you guys in Israel, even though I wonder how Waze can improve Israelian traffic (memories back when I was working Tel Aviv, crazy sometimes late into the night).

BTW, as my hebrew is bad (I remember only a couple of words), what did you post there?

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sat Nov 05, 2011 12:09 pm

asterix06 wrote:@a4xrbj1
If you believe Noam did a big mistake on organizing a proper welcome service
I would like to invite you (Noam pls do not hate my worlds) to download OSM navigator Iphone or Android version and use it in a massive way….. making a comparison with waze ………and you will go back to waze as fast as you can

Waze , on top, is providing written info to users how to use the navigator and how to edit : if people does not read it is not a Noam fault!
Asterix06

Ciao Asterix06,

I think you got my comment about proper welcome service wrong, I'm not comparing Waze to other GPS services and I agree with you that with it's realtime traffic updates on every road it is no 1 in being the most visionary!

You said: "Providing written info to users how to use the navigator and how to edit"? Do you think this is a welcome program? That's were the two of us disagree. Obviously as someone who's working the last 20 years in business on retaining customer, making sure they buy more products and increase their usage I do have another view on what a proper welcome program is ;-)

I'm 100% sure that Noam and his team will precisely understand what I mean and I do hope they pick up some of the points I mentioned and incorporate them into their long-term strategy.

asterix06 wrote:I am just one of the 1 million points Wazers , I am fully aware of the problem we have at the moment, but I can only tell you that Waze is a couple of years ahead all other satnav , is the fastest in innovation and the others can only try to copy.
Waze is now a child on his way to become an adult: let’s us help him to do it in the best possible way.
It will pay back us in a way you cannot even imagine.
I feel proud to participate to waze project!
And you ?
Asterix06

Like I said above, well aware your status as no 1 and having almost 3 times more points than me, the best I managed was getting to no 10 spot in the world but still light years away from what you have done for Waze.

But are you indicating that people that come up with feedback to improve Waze like I did cannot be proud to participate in this project?

Saluti da Kuala Lumpur,

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sun Nov 06, 2011 6:49 am

cymike wrote:
andrewfatcat wrote:the new Cartouche really makes editing much easier.

Could you elaborate?
I for one find the old Cartouche much more usable then the new one (Papyrus). Cartouche feels much more smooth/responsive when editing road geometry for example. Not to mention that Papyrus is feature-incomplete and buggy. Just now (after reading your praise to it) I've tried to give it another shot only to get "an error occurred, please try again later" message while trying to save my edits :(

Same here, gave a long try yesterday and encountered two other points on the map where I get the mysterious "an error occured, please try again later" message. Just found another spot right now whilst editing (yeah, I broke my non-editing phase).

It seems very closely related to map server problems. The last couple of days the Papyrus server takes about 10 seconds to save 1-10 edit steps, very slow (seems Cartouche is now the faster one, let's all go back there, haha!).

A lot of these problems are what seems roads build under other roads, or maybe it's the server again when I move the road to the right position via GPS data, the "original" road doesn't move at all.

Want to see where? Try this segment of road: https://world.waze.com/cartouche/?zoom=7&lat=2.3221&lon=102.21435&layers=BTFFTTFTTFTTTFTTTTT&seg-1=143552314&segments=143552314

I've changed it to Primary road and move the road segment. Still on the Papyrus editor it's out-of-place and sometimes reveals an underlying road (move further south on the above link to see what I mean).

I'm switching between both editors all the time, copying the permalinks (tedious) from one to the other.


As the Waze development team has a lot on their plate (according to this original announcement), could you share the priority with us and ask for our feedback? I know companies normally don't care about feedback from their users (or super-users) but that would surely make a difference and I know you guys care!

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Sun Nov 06, 2011 7:00 am

shmupi2 wrote:Anyway..this is not the right way to communicate with your users and with your area managers that are spending their precious time on mapping and managing the maps.

The problem is not the Blackberry version...the problem is the way that the Waze management team is making decisions and the way that they are communicating with all the area managers around the world , among other things and issues.

Regards,
Shmupi

I second that from Shmupi, I'm not expecting the Waze people to respond to the thousands of threads and every topic. It should be solved by us, the superusers here in the forum like GiffGaff is doing it in the UK (it's a MVNO Telco, they have no support, just a forum kept by their users alone).

But in order for that to work they need to figure out a structured way how the Area Managers and Country Managers in the world can be updated and a two-way communication can be setup. Maybe a challenge for your corp comm department?

Remind you about what I posted, 1% of the people on Waze (probably more 0,01%) are doing 90% of the edits and keep this going!

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby a4xrbj1 » Thu Jan 05, 2012 4:56 pm

dmcconachie wrote:You're going to disenfranchise an awful lot of early adopters (of which I consider myself one) if this goes ahead. I can see an argument to create a yearly and/or monthly leaderboard certainly but removing the All-time board is a no-go for me!

Totally agree, even though I came in about 1 year too late to be considered early adopter (call me laggard).

But you can't take away whatever was achieved so far, unless you want that all those guys in the top 50 immediately turn their back to Waze and do somethings else in their hard time. Respect what these people have done for Waze community, if they are still using it or not.


Just my two cents though,

Andreas
Country Manager of Malaysia, Singapore & Brunei
My Waze videos on editing: http://bit.ly/YTr3z0
Join our FB group: http://on.fb.me/12t1CVU
or Google+ group: http://bit.ly/VDNSjO
Image
a4xrbj1
Waze Global Champs
Waze Global Champs
 
Posts: 197
Joined: Tue Aug 31, 2010 3:35 pm
Location: Kuala Lumpur, Malaysia
Has thanked: 19 times
Been thanked: 3 times

Comments: "Infrastructure, infrastructure, infrastructure!"

Postby AlanOfTheBerg » Fri Nov 04, 2011 3:00 pm

Please use this thread for comments and discussion of Ehud's Infrastructure, infrastructure, infrastructure! post announcement.

eshabtai wrote:We've recently released our new 3.0 client for the iPhone (Android is coming soon!) and got really incredible feedback on it. However, some of you shared your feelings with us, telling us that our priorities are wrong. Instead of a redesigned client or new features you think that we'd better focus on our infrastructures first as building new stuff on top of a shaky base does not make much sense.

While infrastructure is a general term and includes different services, we are fully aware that a lot of the things we promised from day one, are not working as they should. This is mostly true regarding our "daily process" which includes points, client tiles update and map editing. In fact, from your perspective they have not improved over the last year or so, despite us telling you that "we are working on it".

The fact is that the majority of our dev team is actually working on our infrastructure and we have not stopped that work in order to add new features. We have always been constantly redesigning and deploying new improvements to our backend servers. So why aren't you seeing any real world improvements? There are several reasons for that:

  • We are growing fast. While a lot of things have not improved they have also did not deteriorated which means we did improve on our capacity. But not fast enough. What we saw is that every time we deployed a new redesigned backend service, in just a few days new users would come and fill in the available capacity.
  • Some improvements are transparent. Twice in the last year we had a big down time of several hours. We rely on AWS to run our services and when they had issues our service went down. So we have been working on the last few months (and still are) on redeploying all of our core services in a redundant configuration where each service runs on at least two AWS availability zones. That way if one zone goes down our service will keep running on another zone.
  • Serialized processes and dependencies. Our "daily process", historically was a serialized process which involved analyzing users tracks, updating points (partially based on the tracks analysis), updating tiles, switching over to the new tiles, etc'. Since every such task depends on another one, every issue in every task causes a delay in the whole process. So while we have been improving and optimizing the different parts of the chain, there was always a weak link causing delays.
  • Deployment and testing. Deploying infrastructure changes is a slow process as the effect of a bad deployment is pretty much disastrous.
So, while we have been working on improving our infrastructure, at the end of the day, it was not fast enough to address your demand and we have been actually limiting your willingness to contribute time and help us improve the service. We have also lacked in communicating the issues to you, the things we are actually doing to solve them and how much time it is going to take. Once again I'm now going to tell you that things are going to change soon for the better. Instead of just promises I'll try and describe where we stand and when is it coming.

  • Cartouche. We have recently relaunched the new cartouche with a new UI which was a big improvement over the old one. But this was not just a UI project. We have also rewritten the backend on which the old cartouche was running. The new backend allows us to simply add more and more servers as the demand grows and I'm sure you noticed that it runs much faster. We are still limited by the amount of saves we can process per second but for now we have a lot of room to grow there.
    We launched the new backend sooner than we planned as the old backend just couldn't handle the load and we had no way to scale it. Unfortunately we still have some issues we are looking into and while we solved a lot of them our cache currently has some bad data which we need to rebuild. We plan to start working on that in a few days and hopefully this will solve a lot of the inconsistencies you encounter.
    Until we rebuild everything, in a few days we will launch a new update which will bring back the detailed error messages. These messages will help you figure our what's wrong with your edits and in many cases fix the issue and resave.
  • Points. We have been working on breaking most of the dependencies which were required for our points calculation. The calculations rely on a huge amount of events which our different services produce. As part of scaling out our backend, we are now using Hadoop to process it. That way we can easily add more instances as the amount of data grows. Our first goal is to be able to predictably update the points every day without relying on other processes. Most of the development was done and we plan to deploy it in a week or two. Our next step, which have already started to work on, is to be able to update the points in near real time. More updates on that as soon as we finish our next phase design.
  • Merging you tracks. We have previously rewritten the process which merges tracks and it is already able to scale out by just adding more processes. However, only now we have finally developed a supporting system which can actually accommodate and distribute the data for merging. This now enables us to do some parts of the merging process in near real time. Our goal is that within a hour from a drive you will see your track on cartouche along with any new roads and cameras you added. You will also get permissions to edit along your drive immediately.
    Some parts of the merging process such as detecting turn restrictions and road directions is still done in offline once a day. But we are working on improving that part too and make it near real time as well.
    The new system is under final testing and is planned to be deployed within a month.
  • US/World deployment differences. Map problems are still shown only on the US system. We are planning to deploy it on the world servers as well but I don't have an ETA yet. Our policy is not to have differences as it makes our life harder to manage different configurations, however in this case a gap was created and we are trying to close it as soon as possible.
Improving our "daily process" is currently our first priority on the infrastructure tasks. We are working hard on breaking all the dependencies and redesigning everything to allow scaling out easily. Our first goal is to reach predictable iterations as currently I'm sure it's pretty frustrating working on the map and having no idea when will it be available on the client. As we progress, we'll keep updating you!

Ehud
Last edited by AlanOfTheBerg on Sat Dec 03, 2011 3:41 pm, edited 1 time in total.
Reason: Unsticky due to time
AlanOfTheBerg
Waze Global Champs
Waze Global Champs
 
Posts: 19467
Joined: Sat Aug 28, 2010 8:48 pm
Location: US Country Manager - Oregon, USA
Has thanked: 647 times
Been thanked: 2247 times

Re: Comments: "Infrastructure, infrastructure, infrastructur

Postby AlanOfTheBerg » Fri Nov 04, 2011 3:04 pm

I am very appreciative of the announcement and the information provided. One glaring omission, unless I am misunderstanding, is the performance and upgrades/updates to the routing servers. Those are nearly the first thing a new user interacts with as they attempt to get a route to a destination, and today, these routes are still failing to have full integrity on the client. Every day I still get a route which has gaps in it. Maybe that is actually a problem with the map itself and not an issue with routing?
Wiki Resources: Map Editing Manual | Oregon Project/To-Do List
Oregon-based US Country Manager | iPhone5 - VZ - iOS 7.0.6 | Waze v3.7.9.992
AlanOfTheBerg
Waze Global Champs
Waze Global Champs
 
Posts: 19467
Joined: Sat Aug 28, 2010 8:48 pm
Location: US Country Manager - Oregon, USA
Has thanked: 647 times
Been thanked: 2247 times

Next

Return to Website, Community, General

Who is online

Users browsing this forum: No registered users