Discussion for the unofficial, community-developed addons, extensions and scripts built for the Waze Map Editor.

The official index of these tools is the Community Plugins, Extensions and Tools wiki page.
Sort by
Author
Post time
Subject
Display
All posts
1 day
2 weeks
1 month
3 months
6 months
1 year
Post Reply
Forum rules
Discussion for the unofficial, community-developed addons, extensions and scripts built for the Waze Map Editor.

DO NOT START a new thread unless it is about a new idea. Keep discussion of existing tools within the main thread for that tool.

The official index of these tools is the Community Plugins, Extensions and Tools wiki page.

WazeDataArchive - Export your drives to GPX file (EU only?)

Post by kumiankka
Dear fellow Wazers!

Presented here is the WazeDataArchive module for (Windows) PowerShell.

As required by EU's General Data Protection Regulation, the GDPR, Waze has made possible to download your personal data archive. This data might be available for Wazers all over the world I haven't been able to verify it, so please let me know if you are outside EU and can still download your data!

The archive contains, among other things, all your drives from past three months including all the GPS coordinates. Unfortunately, in its default form, the location history section in the archive (the drives) cannot be used pretty much for anything except visual inspection, so I created a PowerShell module that can export the data in couple of formats, including JSON, for further processing, and convert the exported data into GPX format, for importing into popular mapping programs like Garmin BaseCamp or Google Earth.

Using this module requires some basic PowerShell skills but it's not overly complicated and comes with complete help including examples.

Install instructions
Download the attached WazeDataArchive.zip and extract the contents into this folder

Code: Select all

C:\Users\<username>\Documents\WindowsPowerShell\Modules\WazeDataArchive
Adjust the path to match your "Documents" or "My Documents" folder location but do not alter the part starting with WindowsPowerShell unless you are familiar with $env:PSModulePath, importing modules from custom locations etc.

Run the command Import-Module WazeDataArchive or open a new PowerShell window, after which you should have access to these commands:
  • ConvertFrom-WDADriveSummary
  • ConvertFrom-WDAFavorites
  • ConvertFrom-WDALocationDetails
  • Convert-WDADriveSummaryToGpx
  • Convert-WDAFavoritesToGpx
  • Convert-WDALocationDetailsToGpx
  • Import-WDADriveSummary
  • Import-WDAFavorites
  • Import-WDALocationDetails
  • Select-WDASection
Use the command help <name of the command> -Full to read the included help.

This module has been developed and tested on a Windows 10 machine running PowerShell v5 and using only one sample of archived data. Code doesn't yet run on PowerShell Core on other operating systems due to missing types but I'm planning to make it work. Please let me know if (=when) you find bugs, or features that doesn't work on your system or with your data. Thank you!

Happy hacking!

Download the module
WazeDataArchive-2.5.4.zip
(8.93 KiB) Downloaded 116 times
kumiankka
Map Editor - Level 5
Map Editor - Level 5
Posts: 955
Answers: 1
Answers: 1
Has thanked: 197 times
Been thanked: 428 times

POSTER_ID:17473307

1

Send a message
Last edited by kumiankka on Sun Nov 06, 2022 10:27 am, edited 8 times in total.

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by BigDaddyTurbo
Hi,

I've downloaded the Waze archive data and can see there are distinct sections in the CSV (the CSV file I have that contains all the data is called account_activity_3.csv); Drive summary, Favorites, Location details, etc.

I've tried:

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Select-Object -Skip 3 -First 1 | Convert-WDADriveSummaryToGpx -AsString

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<gpx xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.topografix.com/GPX/1/1 http://www.topografix.com/gpx/1/1/gpx.xsd" version="1.1" creator="WazeDataArchive module for Windows PowerShell" xmlns="http://www.topografix.com/GPX/1/1">
  <metadata>
    <time>2018-12-14T10:58:24.2409484Z</time>
  </metadata>
</gpx>
And then:

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Convert-WDADriveSummaryToGpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route from my mobile device' -AsString | Out-File -FilePath .\drives.gpx -Encoding utf8

VERBOSE: Begin converting routes to GPX.
VERBOSE: Finished. Total of 0 routes converted to GPX.
So I tried to use the Convert-WDALocationDetailsToGpx but seems I've carried over the route comment incorrectly:

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Convert-WDALocationDetailsToGpx -Verbose -FileComment 'My drives from Waze' -
RouteComment 'Route from my mobile device' -AsString | Out-File -FilePath .\drives.gpx -Encoding utf8
Convert-WDALocationDetailsToGpx : A parameter cannot be found that matches parameter name 'RouteComment'.
At line:1 char:130
+ ... Gpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route  ...
+                                                     ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Convert-WDALocationDetailsToGpx], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Convert-WDALocationDetailsToGpx
Took that offending line out but get zero results then:

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Convert-WDALocationDetailsToGpx -Verbose -FileComment 'My drives from Waze'
| Out-File -FilePath .\drives.gpx -Encoding utf8
VERBOSE: Begin converting data to GPX.
VERBOSE: Finished. Total of 0 tracks converted to GPX.
I'd like to convert the archive into GPX with the specific aim of figuring out journey lengths (distance).

Apologies my Powershell is virtually non-existent even though I've been in IT for 18 years.

Where am I going wrong with this?
BigDaddyTurbo
Posts: 7
Has thanked: 2 times
Been thanked: 5 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by BigDaddyTurbo
Looks like it's working!

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Convert-WDADriveSummaryToGpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route from my mobile device' -AsString | Out-File -FilePath .\drives.gpx -Encodi
ng utf8
VERBOSE: Begin converting routes to GPX.
VERBOSE: Route 2018-09-19 17:36:58 added to GPX (#1, From: unknown location, To: unknown location).
### output cut ###
VERBOSE: Route 2018-12-10 19:14:42 added to GPX (#245, From: unknown location, To: unknown location).
VERBOSE: Finished. Total of 245 routes converted to GPX.
I couldn't find a decent tool to convert the GPX into a format that would give me what I want. Ended up turning it into a CSV and manipulating it in Excel. I was looking for a tool which would show me the length of each journey. Seems that's tricky when the GPX file contains multiple journeys.

Again, thanks for contributing this to the community and for taking the time to correct the script. It's very much appreciated.
BigDaddyTurbo
Posts: 7
Has thanked: 2 times
Been thanked: 5 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by BigDaddyTurbo
Like this?

Code: Select all

Import-WDALocationDetails .\account_activity3.csv | Convert-WDALoc
ationDetailsToGpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route from my mobile device' -AsString | Ou
t-File -FilePath .\drives.gpx -Encoding utf8
Convert-WDALocationDetailsToGpx : A parameter cannot be found that matches parameter name 'RouteComment'.
At line:1 char:129
+ ... Gpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route  ...
+                                                     ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Convert-WDALocationDetailsToGpx], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Convert-WDALocationDetailsToGpx
I seem to be doing something wrong here.
BigDaddyTurbo
Posts: 7
Has thanked: 2 times
Been thanked: 5 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by BigDaddyTurbo
It worked!

https://i.ibb.co/ZYXGb3g/wazegpx.jpg

Thank you very much for your help. Now I have to try and analyse the data to see what percentage of trips are less than 50km.

In case you're interested, this is all about deciding whether to buy a diesel or a petrol car. I need to figure out the exact type of driving I'm doing.
BigDaddyTurbo
Posts: 7
Has thanked: 2 times
Been thanked: 5 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by KhR0N1K
this guy is a god and very helpful! i tried this outside of EU (in Canada) and just pasted the "download archive" URL which is "https://www.waze.com/user/download_data" and it worked perfectly fine from there. i had a couple issues with the script, powershell and .gpx file since i had no experience with it but i just msged kumiankka and he was very polite and quick to help me out. thanks again brother.
KhR0N1K
Posts: 1
Has thanked: 9 times
Been thanked: 1 time
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by kumiankka
Hey! Thanks for the testing and feedback. Much appreciated! I found couple of bugs from the code and some misinformation from the help texts. The major bug being that the script didn't do anything. Sorry for that! :oops:

Bugs are now fixed, and I updated the attachment in the opening post. Please download the latest version and try again. I don't guarantee that the code is 100 % bug free but at least it should work for the most part. Also, examples in the help of each command should now work as-is.

About the commands. They are PowerShell functions that are named according to PowerShell standard naming rules, Verb-Noun. The functions do pretty much what the name says but here is quick summary.

Import-functions import data into PowerShell session, where it can be either stored to a variable or piped to other commands for further processing. In this module, Import-functions are simple wrappers that just call other commands to simplify importing. They can also convert dates and coordinates to other forms. This is useful if the data needs to be converted to JSON or some other text-based syntax.

Convert-functions convert the imported data to GPX. You can pipe data from an Import-function to a Convert-function if the noun parts of their names match. Convert-function needs date and time data in PowerShell's native DateTime format, so -AsISO8601 switch of an Import-function is not compatible with Convert-function. Convert-functions output PowerShell's native XmlDocument object by default but the output can be switched to plain text using -AsString switch. XmlDocument object can be stored to a variable for further processing or saved to a file using methods of the object.

Select-WDASection function extract raw row data from input stream, thus the data must be piped into the function or supplied as a parameter. Select-WDASection function supports more sections than there are converters for, so not all data can be converted. Import-functions call this function internally, but this can also be used manually to extract data for further processing in other PowerShell commands.

ConvertFrom-functions convert data that has been extracted by the Select-WDASection function to format that is compatible with many other PowerShell commands. Import-functions call this internally but this function can also be used to manually process data that has been stored in a variable or otherwise filtered or pre-processed. You can pipe data from the Select-WDASection function to a ConvertFrom-function if the -SectionName parameter match the noun part of the ConvertFrom function. Additionally, when piping drive summary data or favorites, section header needs to be removed with -NoHeader switch.
kumiankka
Map Editor - Level 5
Map Editor - Level 5
Posts: 955
Answers: 1
Has thanked: 197 times
Been thanked: 428 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by kumiankka
BigDaddyTurbo wrote:Looks like it's working!

Code: Select all

Import-WDALocationDetails .\account_activity_3.csv | Convert-WDADriveSummaryToGpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route from my mobile device' -AsString | Out-File -FilePath .\drives.gpx -Encodi
ng utf8
VERBOSE: Begin converting routes to GPX.
VERBOSE: Route 2018-09-19 17:36:58 added to GPX (#1, From: unknown location, To: unknown location).
### output cut ###
VERBOSE: Route 2018-12-10 19:14:42 added to GPX (#245, From: unknown location, To: unknown location).
VERBOSE: Finished. Total of 245 routes converted to GPX.
I couldn't find a decent tool to convert the GPX into a format that would give me what I want. Ended up turning it into a CSV and manipulating it in Excel. I was looking for a tool which would show me the length of each journey. Seems that's tricky when the GPX file contains multiple journeys.

Again, thanks for contributing this to the community and for taking the time to correct the script. It's very much appreciated.
The data you are using in the code block is DriveSummary. It doesn't contain distances or gps data and therefore cannot be used to calculate total distances. You probably want to use LocationDetails section of the file. It's the most valuable part of the data because it contains all the gps goodies. Put that into a .gpx and use tool like Garmin BaseCamp to read it in. BaseCamp is the software I use and had in mind when I started writing the script. BaseCamp can be downloaded for free from Garmin web site:
https://www.garmin.com/en-US/shop/downloads/basecamp

BasCamp can calculate distances from gps data points:
basecamp1.png
(12.76 KiB) Downloaded 1881 times
It can also calculate total distance of multiple tracks:
basecamp2.png
(10.87 KiB) Downloaded 1833 times
BaseCamp can overlay tracks on a map and it comes with very rough world base map. Usually it's enough to give you general overview where you have been but more detailed maps are also available.
kumiankka
Map Editor - Level 5
Map Editor - Level 5
Posts: 955
Answers: 1
Has thanked: 197 times
Been thanked: 428 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by kumiankka
BigDaddyTurbo wrote:Like this?

Code: Select all

Import-WDALocationDetails .\account_activity3.csv | Convert-WDALoc
ationDetailsToGpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route from my mobile device' -AsString | Ou
t-File -FilePath .\drives.gpx -Encoding utf8
Convert-WDALocationDetailsToGpx : A parameter cannot be found that matches parameter name 'RouteComment'.
At line:1 char:129
+ ... Gpx -Verbose -FileComment 'My drives from Waze' -RouteComment 'Route  ...
+                                                     ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (:) [Convert-WDALocationDetailsToGpx], ParameterBindingException
    + FullyQualifiedErrorId : NamedParameterNotFound,Convert-WDALocationDetailsToGpx
I seem to be doing something wrong here.
Hi! Yes, your're almost there! Just change the parameter name -RouteComment to -TrackComment and the command should work.

Parameter name differ from command to command and this is by design but, I'm still not sure if that was a good idea. Let's wait and see. I might change it in the future to be something more generic (like -ItemComment or something). You can also use PowerShell's tab completion to cycle thru available parameters when typing commands:
https://ss64.com/ps/syntax-tab-completion.html
kumiankka
Map Editor - Level 5
Map Editor - Level 5
Posts: 955
Answers: 1
Has thanked: 197 times
Been thanked: 428 times
Send a message

Re: WazeDataArchive - Export your drives to GPX file (EU onl

Post by kumiankka
Olestas wrote:467km/h? really? :]
Waze stores so called cooked data, which means all coordinates are clamped to nearest road and individual GPS coordinates have no speed data attached to them. This means that Garmin BaseCamp software have to calculate speed from coordinates that are not in their original (raw) location and sometimes the results are bit extreme. Average speed over multiple coordinates is usually accurate enough for statistical purposes but it cannot be used, for example, to fight a speed ticket.
kumiankka
Map Editor - Level 5
Map Editor - Level 5
Posts: 955
Answers: 1
Has thanked: 197 times
Been thanked: 428 times
Send a message
Post Reply