Howto show ArcGIS 9.3 Cache maps under QGis🔗

QGis can open a wide bunch of raster formats, even proprietary ones. If you have ArcGis 9.3 (well, if you are forced to use it!), you have perhaps generated map services. Those services can be displayed on cartographic web applications by using some kind of tile map service (TMS). This service from Esri ArcGIS doesn't respect the TMS standard. The TMS standard uses a special projection (EPSG:3857) and a well-known algorithm to determine which tile to load. Google Maps and OpenStreet map tiling services use the TMS standard and you can use them in QGis with just an XML description file (see this).

But it is not the case with ArcGIS 9.3 which uses a different algorithm to calculate which tile should be requested. This "no-standard" seems to have been discarded in the next version of ArcGIS (10.x). But after a bit of search, I finally manage to display an ArcGIS 9.3 map service on QGis 2.4 and that's what will be discussed in this article.

I've searched what tool could open ArcGIS 9.3 cache maps and found that nearly nothing can do it, with the exception of ESRI ArcGIS of course ! Actually, in the opensource world, nothing else than OpenLayers 2 (starting to version 2.11) can open it ! I think that it is so exotic that it is not supported by OpenLayers 3. On the other side, ESRI seems to have changed its way of dealing with TMS in the version 10.x of ArcGIS. I haven't tested it myself but there are plenty of examples on the Internet that proves that QGis can open ArcGIS 10.x Map Services with only an XML description file (for those who want to go further, they can try the experimental "ArcGIS REST API connector" plugin).

Then I realised that there is a QGis plugin called "OpenLayers". It is used to show GoogleMaps/Bing/OSM tiles by using a workaround at the times where QGis (actually GDAL) was not able to handle those rasters. It is still installed in the version 2.4 of QGis even if the plugin has been refactored.

Here what I've found about "OpenLayers" QGis Plugin underwork:

• There is a core plugin (in Python) code.
• This code uses HTML description files which are a simple page which calls Javascript to print a Map. You can open those files in your Web browser and content will be shown.
• Core code executes Javascript and grabs the images.
• When you zoom or move onto the map, core code execute an Openlayers function and grabs the images.

Quite simple. It is true that this seems to be an ugly hack. Using a browser engine (thanks to Qt) to grab some raster files from the web is not what I call a web service. But it works !

In order to open your ArcGIS 9.3 Map Service, you have to write an HTML description file which uses OpenLayers 2 special functions for ArcGIS 9.3 and let do the magic of the plugin operates ! Actually it is a bit more complicated…

First of all, you need the good version of OpenLayers 2. Grab the latest from the Internet and save it in the plugin repository: .qgis2/python/plugins/openlayers_plugin/weblayers/html. Name it OpenLayers2.js for disambiguation with other version of the library that are already installed in the directory (OpenLayers.js and OpenLayers-2.8.js). You need the full version of the Javascript code, don't use mobile or light version because they do not contain the ArcGIS code.

Before diving into the code, be sure to have a global vision of what to change or to add (the root of the plugin is .qgis2/python/plugins/openlayers_plugin):

• Modify 'weblayers/weblayer.py': Add a class to handle ArcGIS Cache services not projected in ESPG:3857.
• Create 'weblayers/html/layer_x.html': New HTML(+JS) file(s) to add access to one service from ArcGIS 9.3 Cache Map. Add a new file for every new service.
• Create 'weblayers/arcgis.py': New file to create a group of layers which will be named "ArcGIS Layers" in the plugin menu.
• Modify 'openlayers_plugin.py': Import the new group of layers in the plugin and add them to the menu.

Once everything has been made, your layers will be available under the menu Internet → OpenLayers plugin.

Once you've got the good version of OpenLayers, you have to modify the core code of the plugin (but just a very little bit). The hack is just a workaround to deal with projection. If you inspect .qgis2/python/plugins/openlayers_plugin/weblayers/weblayer.py, you'll find a definition for the WebLayer class from which the code deals with the declaration of the layers. Particulary, there is code to handle the projection. Then you find a children class called WebLayer3857. This class is used in every python group declaration file (a file that just declares what are the name of your layers and which HTML file is linked). It just declares a layer with the projection EPSG:3857 which is the projection of all TMS services (read the introduction of this article for further details).

The problem with this static projection declaration is that the plugin cannot handle TMS that are not projected into this projection. Actually, the ArcGIS 9.3 services I used for this test QGis were using EPSG:27592 and even with the correct HTML definition map file, I was not able to show them in QGis if I use a WebLayer3857 declaration. The requested tiles were always too far away from what the service was able to provide.

As a workaround, we just have to declare a new class which is derived from WebLayer and which just says that the projection of the layers declared with this class is always 27592. Just add the following lines at the end of weblayer.py:

class ArcGISLayer27592(WebLayer):
    epsgList = [27592]

The code of WebLayer class is just self-sufficient to handle the different projection. Then, you'll have to use the new ArcGISLayer27592 class to declare your layers in the group layer definition file.

Note to myself: the official plugin code for WebLayer could be updated to handle different kinds of projections…

You just have to write down the definition of the map in an HTML file. Actually, it is much more Javascript than HTML. In the Javascript code, you'll have to use the OpenLayers functions that are able to open ArcGIS 9.3 Cache map. Be sure to load the good version of the OpenLayers2 library (OpenLayers2.js). Here is such a file:

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>ArcGIS Cache Map</title>
    <link rel="stylesheet" href="qgis.css" type="text/css">
    <script src="OpenLayers2.js"></script>
    <script type="text/javascript">
        var map;
        var loadEnd;
    	var layerURL = "http://arcgis.example.com/arcgis/rest/services/folder_name/layer_name/MapServer";
        var layerData = {
	"serviceDescription": "",
	"mapName": "Couches",
	"description": "",
	"copyrightText": "",
	"layers": [
		{
			"id": 0,
			"name": "The layer",
			"parentLayerId": -1,
			"defaultVisibility": true,
			"subLayerIds": null
		}
	],
	"spatialReference": {
		"wkid": 27592
	},
	"singleFusedMapCache": true,
	"tileInfo": {
		"rows": 512,
		"cols": 512,
		"dpi": 96,
		"format": "PNG32",
		"compressionQuality": 0,
		"origin": {
			"x": 225000,
			"y": 401000
		},
		"spatialReference": {
			"wkid": 27592
		},
		"lods": [
			{
				"level": 0,
				"resolution": 0.6614596562526459,
				"scale": 2500
			},
			{
				"level": 1,
				"resolution": 0.26458386250105836,
				"scale": 1000
			},
			{
				"level": 2,
				"resolution": 0.13229193125052918,
				"scale": 500
			},
			{
				"level": 3,
				"resolution": 0.06614596562526459,
				"scale": 250
			}
		]
	},
	"initialExtent": {
		"xmin": 301177.13180509704,
		"ymin": 258249.61360172715,
		"xmax": 301317.7581280163,
		"ymax": 258352.66901617133,
		"spatialReference": {
			"wkid": 27592
		}
	},
	"fullExtent": {
		"xmin": 270184,
		"ymin": 235952,
		"xmax": 332136,
		"ymax": 281008,
		"spatialReference": {
			"wkid": 27592
		}
	},
	"units": "esriMeters",
	"supportedImageFormatTypes": "PNG32,PNG24,PNG,JPG,DIB,TIFF,EMF,PS,PDF,GIF,SVG,SVGZ",
	"documentInfo": {
		"Title": "Layer",
		"Author": "author",
		"Comments": "",
		"Subject": "",
		"Category": "",
		"Keywords": "",
		"AntialiasingMode": "Normal",
		"TextAntialiasingMode": "Force"
	}
    };
        function init() {

            loadEnd = false;
            function layerLoadStart(event)
            {
              loadEnd = false;
            }
            function layerLoadEnd(event)
            {
              loadEnd = true;
            }
	    var baseLayer = new OpenLayers.Layer.ArcGISCache(
                "AGSCache",
                layerURL,
                {
                  layerInfo: layerData,
                  eventListeners: {
                    "loadstart": layerLoadStart,
                    "loadend": layerLoadEnd
                  }
                }
            );

            map = new OpenLayers.Map('map', {
	        maxExtent: baseLayer.maxExtent,
                units: baseLayer.units,
                resolutions: baseLayer.resolutions,
                numZoomLevels: baseLayer.numZoomLevels,
                tileSize: baseLayer.tileSize,
                displayProjection: baseLayer.displayProjection
            });
            map.addLayer(baseLayer);
        }
    </script>
  </head>
  <body onload="init()">
    <div id="map"></div>
  </body>
</html>

You can see that there is a big variable called layerData. It describes the whole content definition of the ArcGIS 9.3 service. Without this variable, you are not able to open the layer in an OpenLayers map… you can grab the variable value by using this example from OpenLayers2 which uses the autoconfig mechanism of ArcGISCache. Just open the Firefox Developper Tools and get the variable value (or use console.info()).

Creating HTML definition map file is something that can be very fastidious. Actually, you have to grab the JSON value for the layerData variable which describes the content of the service. You can do it by using Firefox Developper Tools but this is not an effective method. To ease the pain, just load this HTML (+JS) file in your web browser and put the URL of an ArcGIS 9.3 Map service (for example: http://arcgis.example.com/arcgis/rest/services/name_of_folder/name_of_layer/MapServer) before clicking on the load button. You'll have to save the resulting HTML file in the plugin directory: .qgis2/python/plugins/openlayers_plugin/weblayers/html.

You need to have Internet access to use this code because I just grab OpenLayers2 from cloudflare CDN. But you can replace it with any other javascript file that contains OpenLayers.

    <!DOCTYPE html>
    <html>
  <head>
    <title>QGis OpenLayers ArcGIS 9.3 Cache config helper</title>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <style>
    .smallmap {
      width: 712px;
      height: 556px;
      border: 1px solid #ccc;
    }
    </style>

    <script src="http://cdnjs.cloudflare.com/ajax/libs/openlayers/2.11/OpenLayers.js"></script>

    <script type="text/javascript">
        var map;
	var layerURL;

        function init() {
	    var url = document.getElementById("url").value;
	    if (url) {
	      layerURL = url;
              var jsonp = new OpenLayers.Protocol.Script();
              jsonp.createRequest(layerURL, {
                  f: 'json', 
                  pretty: 'true'
              }, initMap);
            }
        }

        function initMap(layerInfo){

	    var layerData = JSON.stringify(layerInfo, null,'\t');

            var html_doc = "<html xmlns=\"http://www.w3.org/1999/xhtml\">\n\
  <head>\n\
    <title>ArcGIS Cache Map<\/title>\n\
    <link rel=\"stylesheet\" href=\"qgis.css\" type=\"text\/css\">\n\
    <script src=\"OpenLayers2.js\"><\/script>\n\
    <script type=\"text/javascript\">\n\
        var map;\n\
        var loadEnd;\n\
	var layerURL = \"" + layerURL + "\";\n\
        var layerData = " + layerData +";";

           html_doc = html_doc + "\n\
        function init() {\n\n\
            loadEnd = false;\n\
            function layerLoadStart(event)\n\
            {\n\
              loadEnd = false;\n\
            }\n\
            function layerLoadEnd(event)\n\
            {\n\
              loadEnd = true;\n\
            }\n\
	    var baseLayer = new OpenLayers.Layer.ArcGISCache(\n\
                \"AGSCache\",\n\
                layerURL,\n\
                {\n\
                  layerInfo: layerData,\n\
                  eventListeners: {\n\
                    \"loadstart\": layerLoadStart,\n\
                    \"loadend\": layerLoadEnd\n\
                  }\n\
                }\n\
            );\n\n\
            map = new OpenLayers.Map('map', {\n\
	        maxExtent: baseLayer.maxExtent,\n\
                units: baseLayer.units,\n\
                resolutions: baseLayer.resolutions,\n\
                numZoomLevels: baseLayer.numZoomLevels,\n\
                tileSize: baseLayer.tileSize,\n\
                displayProjection: baseLayer.displayProjection\n\
            });\n\
            map.addLayer(baseLayer);\n\
        }\n\
    <\/script>\n\
  <\/head>\n\
  <body onload=\"init()\">\n\
    <div id=\"map\"><\/div>\n\
  <\/body>\n\
    <\/html>";

   	    var html_textarea = document.getElementById("qgis_html");
	    html_textarea.value = html_doc;

            // Send the HTML file to save in the local filesystem:
            var a = document.createElement('a');
            a.href = 'data:attachment/html,' + encodeURIComponent(html_doc);
            a.target = '_blank';
            a.download = 'arcgis_layer.html';

            document.body.appendChild(a);
            // send it to the browser
            a.click();

            var baseLayer = new OpenLayers.Layer.ArcGISCache("AGSCache", layerURL, {
                layerInfo: layerInfo
            });

            /*
             * Make sure our baselayer and our map are synced up
             */

            map = new OpenLayers.Map('map', { 
                maxExtent: baseLayer.maxExtent,
                units: baseLayer.units,
                resolutions: baseLayer.resolutions,
                numZoomLevels: baseLayer.numZoomLevels,
                tileSize: baseLayer.tileSize,
                displayProjection: baseLayer.displayProjection  
            });
            map.addLayer(baseLayer);

            map.addControl(new OpenLayers.Control.LayerSwitcher());
            map.addControl(new OpenLayers.Control.MousePosition() );            
     	    var bbox = layerInfo.initialExtent;
	    map.zoomToExtent(new OpenLayers.Bounds(bbox.xmin, bbox.ymin, bbox.xmax, bbox.ymax));

        }
    </script>
  </head>
  <body>
    <h1 id="title">OpenLayers ArcGIS 9.3 Cache config helper</h1>

    <p id="shortdesc">
    <form id="ServiceURL">
            <p>
                <label for="url">Service URL:</label>
                <input id="url" type="url" style="width: 30%;" placeholder="http://arcgis.example.com/arcgis/rest/services/folder_name/layer_name/MapServer"/>
                <input type="button" value="Load" onclick="init();"/>
            </p>
        </form>
    </p>

    <form id="serviceFolders">

    </form>

    <div id="map" class="smallmap"></div>

    <div id="docs">
      <p>HTML to save in a file</p>
      <textarea id="qgis_html" style="width: 40%; height: 300px;"></textarea>
    </div>
  </body>
</html>

You have to build a group definition file. This is a purely declarative python file which just describes what are your layers. It is used to build the menu shown by the plugin and to make the link with the HTML files. The first class is the Group one. Then all of the layers that are described in HTML file have to be children class of the group one. Be sure to use strings (and not unicode things) in the name of the layers.

# -*- coding: utf-8 -*-
from weblayer import ArcGISLayer27592

class OlArcgisLayer(ArcGISLayer27592):

emitsLoadEnd = True

def __init__(self, name, html):
	ArcGISLayer27592.__init__(self, groupName="ArcGIS 9.3", groupIcon="arcgis_icon.png",
						  name=name, html=html)

class OlArcgisFondDePlanLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name="Fond de plan", html="arcgis_fond_de_plan.html")

class OlArcgisEquipementPublicLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Equipement public', html='arcgis_equipement_public.html')

class OlArcgisAdresseLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Adresse', html='arcgis_adresse.html')

class OlArcgisArreteCirculationLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Arrete de circulation', html='arcgis_arrete_circulation.html')

class OlArcgisAssainissementLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Assainissement', html='arcgis_assainissement.html')

class OlArcgisCadastreNapoleonLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Cadastre Napoleonien', html='arcgis_cadastre_napoleon.html')

class OlArcgisParcelleLayer(OlArcgisLayer):

def __init__(self):
	OlArcgisLayer.__init__(self, name='Parcelle', html='arcgis_parcelle.html')

You have to modify openlayers_plugin.py to add the entry menus from your group definition file. The entries are built upon the python class, so you have to import them with a simple line like:

…
from weblayers.arcgis import OlArcgisFondDePlan, OlArcgisEquipementPublic, …
…

Once the modules (classes) are declared, you need to manually add them into the function initGui:

…
self._olLayerTypeRegistry.register(OlArcgisFondDePlanLayer())
self._olLayerTypeRegistry.register(OlArcgisEquipementPublicLayer())
…

Once this is done, you should be able to find your menu entries in the menu with a group of your own layers. Be careful about the names which can be a little type error prone.

The OpenLayers QGis plugin is far from being perfect ! Actually it seems to be an ugly hack that works with very few efforts, so on… I've experienced some troubles by using this modified plugin with ArcGIS 9.3:

• You should open another layer which is in the extent of your raster layer before opening the raster layer. Otherwise, the HTML script may crash because you are asking for a too much wide extent. It can block QGis, so be careful !
• Sometimes when you zoom, not of all the tiles are shown on the map dialog. You have to unzoom and rezoom to have the whole map updated and shown.
• Same problem with panning or moving. There you have to unzoom and rezoom to grab all of the tiles.
• It seems to have a sort of cache: once you grabbed all the tiles at different zoom levels, map loading is more fast.
• You can't have more than one OpenLayers layer on the map. Only the latest opened layer is shown, whatever the layer order. It is a limitation of the whole plugin, not restricted to ArcGIS services.
• You can't save an OpenLayer layer on your project ! You have to reopen the layer each time you open your project. It is also a limitation of the plugin, not restricted to ArcGIS services.

I have taken time to test print composer with those services and it seems to be unaffected by the bugs I've seen in the map window. You will be able to generate PDF or print maps with your ArcGIS Map service layers. So far, so good…

QGis is good for you. It can even be interfaced with its greatest opponent in GIS Desktop: ESRI ArcGIS. If you haven't migrated to ArcGIS 10.x, you are still able to use your old ArcGIS 9.3 Cache map services for a while under QGis. There are great chances that no other GIS Desktop software is able to deal with these services nowadays. But QGis is one of them…

To go further, I can only advice you to migrate to a true free software tiling solution like MapNik and carto-css. By using the TMS standard, you are 100% sure that QGis will be able to do it. Please read my wiki article to build your opinion on the ease to generate tiles with those tools.