Get a sneak peek at whats next for Permanent Hazards on our April 7th Office Hours!
Post by Twister-UK
First bit of data from the new routing tester...

http://i1056.photobucket.com/albums/t37 ... f8afce.png

Using our trusty old test route once again, this plot shows the server response time (in milliseconds) to each routing request (grey line) with the usual yellow and red background areas denoting requests where the returned route was incomplete or non-existent. The black line is a 5-point averaging of the raw response times. The plot starts at 00:00 today and runs through to around 10:15.

What's interesting here is that throughout the period where no routes were returned, there's a corresponding drop in average response times, which suggests the server is simply rejecting the requests as they arrive rather than attempting to calculate any routes and then timing-out during the process (the request timeout parameter was set to the default 60000 for this test). Bear in mind that the response times recorded by the test client include network transit times for both the request and response packets, in addition to the processing time of the routing server itself, so the odd spike in raw response times may be down to network congestion rather than server slowdown.

Raw data captured during this test: http://chizzum.com/lmr/test1.zip
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Last edited by Twister-UK on Mon Sep 02, 2013 9:06 pm, edited 1 time in total.
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Test has now been restarted with the client requesting just a single route each time...
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Requesting just a single route doesn't seem to make the slightest difference...

http://i1056.photobucket.com/albums/t376/chriscoulson2012/th_lmr_responsetimes_singleroute_zpsfcd3cef4.png

This plot starts at 14:00 on the 12th, and runs through until 20:25 today. The x axis ticks are every hour, and the y axis is response time in milliseconds as before. Clearly, the poor routing server is so badly overworked and/or underpowered that, even asking for just a single route, it follows the now quite familiar pattern of cycling between periods of returning incomplete routes to periods where no routes are returned at all, only resuming normal operation during the late evening-overnight-early morning quiet time (for the UK and western Europe timezones at least - does this mean there are different routing servers allocated to world.waze.com users depending on whereabouts in the world they are?).

Endpoint plot is also familiar...

http://i1056.photobucket.com/albums/t376/chriscoulson2012/th_lmr_endpoints_singleroute_zps530d0b9c.png

Raw data captured during this test: http://chizzum.com/lmr/test2.zip
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Unless you'd prefer a different route, then for consistency and comparison I'll start logging again tonight using the same start/end points as the previous tests.

Will 24 hours of data be sufficient or would you like it to run for longer? And do you need the raw data or just the csv summary?
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Logging was started at around 10pm last night. Monitoring the live output whilst having breakfast this morning (around 7am) I noticed a few requests that were returning only a single route or none at all, so in that respect the routing server appears to be behaving as per the previous tests, and this test should be able to supply the data that Shira and her team requires.

Slightly worrying though is that for all of the requests I observed (even during setting up the test last night) where both routes were returned, the quickest route never quite made it to the requested endpoint. Whilst this is expected behaviour in busy periods, 10pm on a Sunday evening isn't what previous testing would indicate to be a busy period - when I get a chance to examine the full logs in detail tonight it'll be interesting to see if any requests returned the correct endpoints for both routes...
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
With almost 2 full days of data captured, logging has now stopped and a summary of the results is as follows:

Day 1 (9th September):

At 01:44 and 01:46 the returned routes flipped around, with the shortest route returned as the first result rather than the second.

The shortest route failed to reach the expected endpoint between 06:17-06:49.

Between 06:50-07:00 only a single route was returned.

Routing failed completely between 07:01-07:24, and again between 17:04-17:31


The quickest route never reached the expected endpoint, although in most of the results it was very close (51.4595492,-0.067670527 vs the expected 51.45954977,-0.06766258).


Day 2 (10th September):

The shortest route failed to reach the expected endpoint between 06:03-06:42.

Routing failed completely between 07:47-09:05, and again between 16:33-18:35.

The quickest route reached the expected endpoint between 02:57-06:19 and from 19:24 to the end of logging at 22:08.


Raw data for this test: http://chizzum.com/lmr/test3.zip


Notes on the CSV file layout...

The first 4 columns contain data common to all routes returned in the request:
Columns 1&2 - Date/time of request
Column 3 - Elapsed time (ms) between submitting request and receiving response
Column 4 - 0 if any routes were returned, 1 if no routes were returned

Each returned route then adds 5 columns as follows:
Column 5 - 0 if route reached the exact expected endpoint, 1 if it did not (no matter how close it got)
Columns 6&7 - actual endpoint coordinates
Column 8 - route distance
Column 9 - route time
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
I'll have to check and see if the route tester still works after the Livemap changes, but yeah, should be able to do another run next week.
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Slight change in the request arguments, plus enforced use of a https request which meant having to set up SSL support in Indy... With that now working OK, a quick check of the revised code shows that it's able once again to retrieve the test route used previously - I'll leave it running overnight to make sure it's stable on an extended run, and then fingers crossed we should be good to go next week.
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Using the same routing parameters as before, another 2-day test was run earlier this week. Comparison with the results from the previous test suggests a whole load of absolutely nothing has been done to improve routing server behaviour...

Day 1 (21st October):

Between 06:04 and 06:39 the returned routes flipped around, with the shortest route returned as the first result rather than the second.

The shortest route failed to reach the expected endpoint between 06:04-06:51.

Between 06:40-06:51 only a single route was returned.

Routing failed completely between 06:52-07:49, and again between 17:17-17:23 and 17:25-17:33 (the server managed to return both routes at 17:24)

The quickest route never reached the expected endpoint, although in most of the results it was very close (51.4595492,-0.067670527 vs the expected 51.45954977,-0.06766258).


Day 2 (22nd October):

IIntermittently between 05:53 and 06:44 the returned routes flipped around, with the shortest route returned as the first result rather than the second.

The shortest route failed to reach the expected endpoint between 06:09-06:10, 06:12-06:18, 06:20-06:21, 06:24-06:37, and 06:41-06:44.

Routing failed completely between 07:33-08:50, and again between 16:36-18:37.

Between 01:57-02:02, no responses were received from the server.

The quickest route consistently reached the expected endpoint between 03:08-06:23 (with intermittent complete routes returned between 02:29-03:05 and 06:34-06:36), at 11:54, 12:59-13:00 and 13:06, and then from 19:24 until 23:59 where logging ended.


Raw data for this test: http://chizzum.com/lmr/test4.zip


Notes on the CSV file layout...

The first 4 columns contain data common to all routes returned in the request:
Columns 1&2 - Date/time of request
Column 3 - Elapsed time (ms) between submitting request and receiving response
Column 4 - 0 if any routes were returned, 1 if no routes were returned

Each returned route then adds 5 columns as follows:
Column 5 - 0 if route reached the exact expected endpoint, 1 if it did not (no matter how close it got)
Columns 6&7 - actual endpoint coordinates
Column 8 - route distance
Column 9 - route time
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png

Post by Twister-UK
Just fired up the automated test code used to generate the previous sets of results, and using the same routing request as before (see Shira's earlier post for details) I'm not getting any routes returned at the moment (6 requests made so far). I'll leave the code running for a few hours to see when routes start to appear...
Twister-UK
Waze Local Champs
Waze Local Champs
Posts: 4670
Answers: 2
Has thanked: 736 times
Been thanked: 4677 times
Send a message
Chris (not to be confused with Chris or Chris, or even Tim, Stu, or any of the other champs team...)
AM SE England & Shetland Islands, UK Local Champ, WME Beta Tester & ScriptMangler
WME/Livemap enhancement scripts @ GreasyFork


https://chizzum.com/greasemonkey/images/beta.pnghttps://chizzum.com/greasemonkey/images/s0400.pnghttps://chizzum.com/greasemonkey/images/c5s.png