Generate OSM raster tiles for offline browsing🔗

This job has been achieved with Debian Jessie (aka testing for the moment). The goal is to build raster OSM rendered tiles to read them offline with a GIS tool like QGis or even Javascript OpenLayers.

No explanations here, just the code or commands for those who don't have time…

Here are the steps to render OSM tiles:

• We need to grab the data from OSM and inject them into a PostGIS database.
• Then, we need to install rendering tools. The main one is mapnik which just do the rendering. But MapNik is not a tool, it is a library. So we need to install it and grab a few command-line tools to do the effective rendering.
• MapNik uses style files (XML) to style the OSM data before rendering. So we just need the official OSM Mapnik style files…
• … but OSM official Mapnik files doesn't exists. They are built using cartocss stylesheets. So we need a tool to convert from cartocss to mapnik style files.
• This tool is called carto-css (node-carto on debian).

Here we go !

I am building this from scratch. So I consider that there is no postgresql installation…

// install postgresql and postgis
# aptitude install postgresql-9.3-postgis-2.1
// create a new database as postgres user, we name it geobase
# su postgres
$ createdb geobase
$ psql -d geobase -c "CREATE EXTENSION postgis;"
$ psql -d geobase -c "CREATE EXTENSION hstore;"

You will have to modify pg_hba.conf in order to allow trust connexion for the user postgres on Unix socket. This is completely INSECURE but I need to go fast ! I assume you are working in a non connected environnement:

local   all         postgres                                trust
host    all	    postgres	     127.0.0.1/32	    trust
host    all	    postgres	     ::1/128		    trust

// prepare a repository
$ mkdir -p osm_tiles/tools
$ cd osm_tiles
// get the carto css stylesheets and the Nik4 tool
$ git clone https://github.com/gravitystorm/openstreetmap-carto

$ cd openstreetmap-carto
$ wget http://svn.openstreetmap.org/applications/rendering/mapnik/generate_tiles_multiprocess.py
$ chmod 755 generate_tiles_multiprocess.py
// get all of the basic layers
$ ./get-shapefiles.sh
// grab OSM data: this time it will be Ireland
$ wget http://download.geofabrik.de/europe/ireland-and-northern-ireland-latest.osm.pbf

$ osm2pgsql -k -c -S openstreetmap-carto.style -d geobase -U postgres ireland-and-northern-ireland-latest.osm.pbf

We need to edit the project file as the default postgis access are changed on our installation.

$ sed -i '/"dbname": "gis"/c \\t"dbname": "geobase"' project.mml
$ sed -i '/"dbname": "geobase"/ i \\t"user": "postgres", \n\t"host": "localhost",' project.mml

Then we can generate our MapNik XML file:

$ carto project.mml > mapnik.xml

Use OpenstreetMap Export utility and grab the overpass URL:

http://overpass-api.de/api/map?bbox=-10.673,51.406,-5.938,54.304

For zoom levels, use OSM to determine the max zoom level by zooming at the maximum you want on your map:

http://www.openstreetmap.org/#map=18/52.44848/-9.05243

The max zoom level is 18.

generate_tiles_multiprocesspy is a tool to build tiles from a mapnik xml file. It is written in Python and use mapnik to do the rendering. Before launching it, you have to modify the code to indicate the extent and the zoom level. You'll have to modify lines at the end of the script. Here is what I've done to it (very dirty things but quite fast modifications):

...
if __name__ == "__main__":
    home = os.environ['HOME']
    try:
	mapfile = os.environ['MAPNIK_MAP_FILE']
    except KeyError:
	   mapfile = '/home/medspx/osm_tiles/openstreetmap-carto/mapnik.xml'
    try:
	tile_dir = os.environ['MAPNIK_TILE_DIR']
    except KeyError:
	tile_dir = '/home/medspx/osm_tiles/wtms/'

    if not tile_dir.endswith('/'):
	tile_dir = tile_dir + '/'

    #-------------------------------------------------------------------------
    # Change the following for different bounding boxes and zoom levels
    #
    # Start with an overview
    # Ireland
    bbox = (-10.673, 51.406, -5.938, 54.304)
    render_tiles(bbox, mapfile, tile_dir, 0, 18, "World")

Then you'll have to modify your Mapnik xml style file. The problem comes from the Fonts required to generate the tiles. Mapnik made them mandatory. On general GNU/Linux distributions, DejaVu fonts are installed by default. So remove all of the Font declarations except this one.

After modifications, just launch the script:

$ ./generate_tiles_multiprocess.py

All of the files are located under the osm_tiles/wtms directory. The process can be very long: for the whole Ireland, my computer had to work for about more than 48 hours !

You just have to create an XML file to emulate a TMS server with some files:

<GDAL_WMS>
<Service name="TMS">
	<ServerUrl>file:///home/medspx/osm_tiles/wtms/${z}/${x}/${y}.png</ServerUrl>
</Service>
<DataWindow>
	<UpperLeftX>-20037508.34</UpperLeftX>
	<UpperLeftY>20037508.34</UpperLeftY>
	<LowerRightX>20037508.34</LowerRightX>
	<LowerRightY>-20037508.34</LowerRightY>
	<TileLevel>18</TileLevel>
	<TileCountX>1</TileCountX>
	<TileCountY>1</TileCountY>
	<YOrigin>top</YOrigin>
</DataWindow>
<Projection>EPSG:3857</Projection>
<BlockSizeX>256</BlockSizeX>
<BlockSizeY>256</BlockSizeY>
<BandsCount>3</BandsCount>
<Cache />
</GDAL_WMS>

Send this into ireland.xml and open it as a raster under QGis !

Here is the result:

You have now rendered OSM tiles under QGis. But you'd like to change things. For example, you'd like to show campings at a higher zoom level on the tiles ! Just hack the Carto-CSS styles !

Go edit amenity-points.mss, find eveything with camp_site and change the zoom level in it:

[tourism = 'camp_site'][zoom >= 14]::tourism {
  point-file: url('symbols/camping.n.16.png');
  point-placement: interior;
}
...
[tourism = 'camp_site'][zoom >= 14]::tourism {
  text-name: "[name]";
  text-size: 10;
  text-fill: #0066ff;
  text-dy: 15;
  text-face-name: @book-fonts;
  text-halo-radius: 1;
  text-wrap-width: 70;
}

Just use Nik4:

$ cd ~/osm_tiles
$ git clone https://github.com/Zverik/Nik4
$ cd ~/osm_tiles/openstreetmap-carto
$ ../Nik4/nik4.py -b -10.673 51.406 -5.938 54.304 -z 13 mapnik.xml ireland.tiff --wld ireland.wld

Use my dirty hack (get time to study code): nik4.py !

$ ../Nik4/nik4.py -b -10.673 51.406 -5.938 54.304 -z 14 mapnik.xml ireland.tiff --wld ireland.wld --tiles 4

Then use gdalbuildvrt and gdal_translate and gdaladdo

$ gdalbuildvrt -a_srs EPSG:3857 Ireland_13.vrt *.tiff
$ gdal_translate -of GTiff -a_srs EPSG:3857 -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE -co ZLEVEL=9 -co PREDICTOR=2 Ireland_13.vrt Ireland_13.tiff
$ gdaladdo -r average Ireland_13.tiff 2 4 8 16 32