Generating townland geometries from open data for WME

[ img ]

Moderators: Unholy, davidg666, arthurohara

Generating townland geometries from open data for WME

Postby cw1998 » Tue Apr 24, 2018 9:37 pm

Previous text
[moderator edit by davidg666: split topic to move this post to a new thread]

I should probably make a new thread for this... but I have completed one part of my script - county extraction of townland geometries. See the screenshot here. Performance is good with just one county's worth of geometry loaded. The file is only about 2MB in size for Monaghan, so I image it will be similar for the majority of the rest. Maybe Mayo, Cork and Galway will be larger; testing needs to be done for this.


In order to facilitate the addition of townlands, I have been looking into a way to show townland boundaries in WME. This is possible because data.gov.ie provides open data sets with townland geometry in them. However, these files are very large and often impossible to work with in WME.

To get around this, I have written a script to extract a county's set of townlands that can be loaded into WME on its own (without the rest of the country) in order to improve performance.
Performance is still an issue for larger counties, so I am looking to split these files up further, possibly into half counties.

The script is written in Python (this may be ported to another language in the future to improve its performance), and it accepts both generalised and ungeneralised geoJSON files provided by data.gov.ie.

It is worth noting that counties in Northern Ireland aren't included in the datasets, so they are therefore not included as counties in the script.

The script can be found on GitHub here.
Usage: Just run the script in a terminal window. It will guide you through the rest.
I haven't got a chance yet properly test it yet.
Dependencies: ijson and simplejson (these are more performant than the built-in python equivalents)
Both can be installed with
Code: Select all
pip3 install <package>
in your virtual environment.
Last edited by cw1998 on Sat Apr 20, 2019 8:53 pm, edited 4 times in total.
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Re: displaying townlands in WME

Postby davidg666 » Wed Apr 25, 2018 7:41 am

cw1998 wrote:I should probably make a new thread for this...

I took the liberty of moving your post into a new thread...
cw1998 wrote:I have completed one part of my script - county extraction of townland geometries. See the screenshot here.

Looks great! If this works well (and it's easy to set up) I think I'm going to love it.
cw1998 wrote:Performance is good with just one county's worth of geometry loaded. The file is only about 2MB in size for Monaghan, so I image it will be similar for the majority of the rest. Maybe Mayo, Cork and Galway will be larger; testing needs done for this.

The next step is testing this with the ungeneralised data set, which is many times larger than the generalised file...

Edit: Yeah this may be a challenge. I just get memory errors trying to load in the ungeneralised file. Will keep working on this tomorrow.

Which browser and OS are you using? I'm using Chrome on a Mac (it was decent in 2015, but it sometimes struggles a bit with WME now).

++David
Country manager and local champ for Ireland
AM for Dublin, Wicklow, Kilkenny, Wexford, Graiguenamanagh and Enniscorthy

[ img ]
davidg666
Waze Local Champs
Waze Local Champs
 
Posts: 911
Joined: Thu Jan 12, 2012 6:58 pm
Location: Wicklow, Ireland
Has thanked: 75 times
Been thanked: 192 times

Re: displaying townlands in WME

Postby cw1998 » Wed Apr 25, 2018 10:47 am

davidg666 wrote:Which browser and OS are you using? I'm using Chrome on a Mac (it was decent in 2015, but it sometimes struggles a bit with WME now).


I'm using Firefox on Windows 10 if I'm at home, or Linux if I'm using my laptop. My PC at home is almost 7 years old now, and WME is still manageable with the generalised townlands loaded in. (8GB RAM and an old, I think 3rd gen, intel core i5)

I've managed to get around the memory issues by loading the ungeneralised file in line by line (which in itself was a challenge because the original file was all on one line!), So the only drawback now is speed as the new file is a couple of million lines long. It takes a good 15 minutes to extract a county.
Python isn't the best language to be using for speed (I didn't think it would be this bad!) So I'll probably port it over to something like C++ if optimisation efforts in python don't improve anything.

In the meantime, I can throw this script onto a server and have it work away at extracting each county. It might take a while! I'll then make these smaller files publicly available (I'll maybe set up a site at waze.cw1998.uk)

If these smaller county sized ungeneralised files are still too big to be managed efficiently in WME, I'll split them in half again. The benefit here being the smaller files should take much less time to process! At this stage it's just a maths problem - working out which townlands need included by coordinates.
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Re: Displaying Townlands in WME

Postby cw1998 » Wed Apr 25, 2018 12:22 pm

I have updated the first post with some more information about this. I also have a server going through the dataset and making a separate file for each county. I'll update you here when this is finished!
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Re: Displaying Townlands in WME

Postby davidg666 » Wed Apr 25, 2018 12:36 pm

cw1998 wrote:I have updated the first post with some more information about this. I also have a server going through the dataset and making a separate file for each county. I'll update you here when this is finished!

Lovely - thank you!

When you get a chance, would you mind writing a little cheat sheet telling people what to do to get Tim's script and your datafiles working with WME? Without having tried it yet, I imagine it's straightforward once you've done it once :)

++David
Country manager and local champ for Ireland
AM for Dublin, Wicklow, Kilkenny, Wexford, Graiguenamanagh and Enniscorthy

[ img ]
davidg666
Waze Local Champs
Waze Local Champs
 
Posts: 911
Joined: Thu Jan 12, 2012 6:58 pm
Location: Wicklow, Ireland
Has thanked: 75 times
Been thanked: 192 times

Re: Displaying Townlands in WME

Postby cw1998 » Wed Apr 25, 2018 12:46 pm

davidg666 wrote:When you get a chance, would you mind writing a little cheat sheet telling people what to do to get Tim's script and your datafiles working with WME? Without having tried it yet, I imagine it's straightforward once you've done it once :)


Yep, it's on the todo list. Just need to wait for a few more county files to get generated (it's at 6% overall at the time of writing). I've also just added a new DNS record to my domain, so waze.cw1998.uk should be live soon (if it isn't already). All the county files will live there.
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Re: Displaying Townlands in WME

Postby davidg666 » Wed Apr 25, 2018 12:49 pm

cw1998 wrote:
davidg666 wrote:I've also just added a new DNS record to my domain, so waze.cw1998.uk should be live soon (if it isn't already). All the county files will live there.

Ah, that's why it wouldn't load for me (in Dublin, just checked at 12:48 UTC).

++David
Country manager and local champ for Ireland
AM for Dublin, Wicklow, Kilkenny, Wexford, Graiguenamanagh and Enniscorthy

[ img ]
davidg666
Waze Local Champs
Waze Local Champs
 
Posts: 911
Joined: Thu Jan 12, 2012 6:58 pm
Location: Wicklow, Ireland
Has thanked: 75 times
Been thanked: 192 times

Re: Generating townland geometries from open data for WME

Postby cw1998 » Fri Apr 27, 2018 2:33 pm

Just to update this: I made the script half the precision of the coordinates for the points on the townland polygons and remove unneeded polygon properties.
It looks a little bit messier (see this screenshot), but the file size is essentially half of the original, and it certainly is still workable.
I have only run this with Co. Clare so far, and the file size went from 39.3MB to 20.1MB.
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Re: Generating townland geometries from open data for WME

Postby davidg666 » Fri Apr 27, 2018 2:51 pm

cw1998 wrote:Just to update this: I made the script half the precision of the coordinates for the points on the townland polygons and remove unneeded polygon properties.
It looks a little bit messier (see this screenshot), but the file size is essentially half of the original, and it certainly is still workable.
I have only run this with Co. Clare so far, and the file size went from 39.3MB to 20.1MB.

So the lower-precision one is the purple line on your screenshot?

Does it make any noticeable speed difference in WME?

++David
Country manager and local champ for Ireland
AM for Dublin, Wicklow, Kilkenny, Wexford, Graiguenamanagh and Enniscorthy

[ img ]
davidg666
Waze Local Champs
Waze Local Champs
 
Posts: 911
Joined: Thu Jan 12, 2012 6:58 pm
Location: Wicklow, Ireland
Has thanked: 75 times
Been thanked: 192 times

Re: Generating townland geometries from open data for WME

Postby cw1998 » Fri Apr 27, 2018 3:22 pm

davidg666 wrote:So the lower-precision one is the purple line on your screenshot?

Yes.

davidg666 wrote:Does it make any noticeable speed difference in WME?

Yes, zooming in and out is definitely smoother. I don't really have much of a problem with panning around with the larger files loaded, but on a lower end device, I'm guessing it will be smoother.
cw1998
Area Manager
Area Manager
 
Posts: 149
Joined: Sat Jul 23, 2011 10:02 am
Location: Waringstown, Northern Ireland
Has thanked: 90 times
Been thanked: 32 times

Next

Return to Ireland

Who is online

Users browsing this forum: No registered users