In this installment, we’ll use the result as a Mapnik layer, and check out some different options for using raster data in your Mapnik maps.

Before we dive in, a note about versions. In this tutorial I’m using Mapnik2, which is now in beta release. I’m doing this both because Mapnik2 adds some exciting new capabilities, and so this tutorial is worth anything for more than a few weeks. Even though Mapnik2 is still beta, I’ve been running it for several weeks now and so far haven’t run into any serious difficulties, so especially if you’re on Mac OS X, I’d recommend giving version 2 a shot.

Even if you do run into problems, the Mapnik listserv is very active, and the developers themselves are very quick to respond with help and to fix bugs. If only every open-source project was so well-maintained!

If you’re not running version 2, a lot of this will still work, though you might need to tweak syntax a little bit. I’ll note anything you just can’t do in Mapnik before version 2.

Data mis en place

Here’s the data we’ll use for this part of the tutorial:

• A California border shapefile in Mercator projection (SRID 3395)
• A California county/shorelines shapefile (SRID 4326)
• A California water shapefile (SRID 4269)
• The final GeoTIFF cutout from part 1 (This is an 88 MB file, so don't download if you don't have to!)

Now on to the fun!

First, let’s do a basic Mapnik map with our GeoTIFF layer. For many Mapnik maps, Cascadenik makes styling much simpler for anyone familiar with CSS syntax, but currently (February 2011) Cascadenik doesn’t support raster layers. So we’ll start with this XML:

<Map srs="+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs">
  <Style name="hillshade style">
    <Rule name="rule 1">
      <RasterSymbolizer opacity="1" scaling="bilinear" mode="normal" />
    </Rule>
  </Style>
  <Layer name="hillshade">
    <StyleName>hillshade style</StyleName>
    <Datasource>
      <Parameter name="type">gdal</Parameter>
      <Parameter name="file">/Users/yourname/Documents/gdaltutorial/ca-dem-combined-merc-cutout.tif</Parameter>
    </Datasource>
  </Layer>
</Map>

Let’s break this down in stages. First, the overall <Map> wrapper:

<Map srs="+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs">
</Map>

This is where we set the projection of the overall map. Don’t worry, you don’t need to understand what all of it means (although it’s fun to find out!). As you no doubt noticed, this is the same srs data we used to reproject our GeoTIFF in part 1.

One of the great features of Mapnik is that even if the data for separate layers are in different projections, in most cases Mapnik can simply reproject that data on the fly. You can’t reproject raster data within Mapnik, however, so you’ll always want to be sure that you get your GeoTIFF into the projection you want your final map to use before you get to Mapnik (as we did in part 1).

Now let’s drop down a bit to the layer data:

  <Layer name="hillshade">
    <StyleName>hillshade style</StyleName>
    <Datasource>
      <Parameter name="type">gdal</Parameter>
      <Parameter name="file">/Users/yourname/Documents/gdaltutorial/ca-dem-combined-merc-cutout.tif</Parameter>
    </Datasource>
  </Layer>

This is where we tell Mapnik what type of data we’re using for this layer. Inside the <Datasource></Datasource> tags, we specify the data type (gdal), and point to the location of our GeoTIFF file. It’s good practice to use the absolute path to your files so if you move your Mapnik files you won’t break the script.

The remaining line inside our <Layer></Layer> tags is the <StyleName></StyleName>. This tells Mapnik which style definition to use for this layer. So let’s take a look at that style definition.

  <Style name="hillshade style">
    <Rule name="rule 1">
      <RasterSymbolizer opacity="1" scaling="bilinear" mode="normal" />
    </Rule>
  </Style>

There’s a lot you can do with raster data, especially in Mapnik2. This style definiton is pretty basic, though. We’re saying that we want the GeoTIFF to be rendered at 100% opacity. Scaling controls how Mapnik renders the image when scaling it down from its original size, and the mode controls how the raster layer is blended with layers below it. Since there’s nothing currently behind the layer, mode is somewhat moot at the moment. <a href=”http://trac.mapnik.org/wiki/RasterSymbolizer#Usage”</a>Feel free to play with different values for both scaling and mode.</a>

That’s about it. Just make sure your style name matches the <StyleName></StyleName> tag in your layer.

That’s it for our first, pretty basic GeoTIFF Mapnik map. Save this file as ca-map.xml.

Now we’ll use Python to compile this XML into an image. I highly recommend that once you get the basics down, you simplify your life and download Cascadenik and nik2img to do a little of this coding for you, but at this point you’ve installed enough for one tutorial. So we’ll stick to a basic Python script. Even if you’ve never used Python before, now’s the time to start. Trust me, if you have any programming experience this will make sense.

Open a new text file, and save the following code as ca-stuff-compile.py into the same directory as your XML document:

#!/usr/bin/env python
import mapnik2
mapfile = 'ca-stuff.xml'
map_output = 'ca-map.png'
m = mapnik2.Map(1200, 800)
mapnik2.load_map(m, mapfile)
bbox = mapnik2.Box2d(mapnik2.Coord(-14000000, 3700000), mapnik2.Coord(-12500000, 5200000))
m.zoom_to_box(bbox)
mapnik2.render_to_file(m, map_output)

Right off the bat, if you’re not using Mapnik2 for this, you’ll need to change everthing that says mapnik2 to simply mapnik.

Next we assign some variables. We specify our XML file for Mapnik to chew on, and tell Mapnik what filename to use for the final image.

mapfile = 'ca-stuff.xml'
map_output = 'ca-map.png'

Next we instantiate a new mapnik map, and pass our XML to Mapnik’s load_map() function. We also specify the pixel dimensions we want to give our final map, 800 pixels wide by 600 pixels deep.

m = mapnik2.Map(800, 600)
mapnik2.load_map(m, mapfile)

Now we tell Mapnik what bounding box we want the map to use. We’ll use this to control how much space will show up around our map content, but you could also specify only a section of the map or any other arbitrary bounding box. But how do we know what to use?

Remember back in part 1 when we used ogrinfo to get the bounding box of our California shapefile that we used to cut out the GeoTIFF? Here’s what that returned:

Extent: (-13849389.898804, 3810165.061203) - (-12704836.275367, 5133847.169980)

So if you want to clip your map at California’s exact boundaries, you can use that extent exactly. But generally we want our maps to have a little padding, so we’ll round up and down a bit:

bbox = mapnik2.Box2d(mapnik2.Coord(-14000000, 3700000), mapnik2.Coord(-12500000, 5200000))
m.zoom_to_box(bbox)

Note that, mercifully, Mapnik uses the same format for its bounding box as ogrinfo. The points represent the lower left and upper right corners of our image.

Finally, the script renders the map to our file:

mapnik2.render_to_file(m, map_output)

If you haven’t already save this file into the same directory as your XML document as ca-stuff-compile.py.

Now, in a terminal window, change into this directory, and let’s build that map:

$ python ca-stuff-compile.py

If you don’t get any error messages, check your directory. You should see a new file called ca-map.png, and hopefully your hillshade map is in the middle.

Neato! Well, sort of - it’s pretty boring, right? Not much different from our original GeoTIFF. Let’s change that now by adding more styles and more map layers to our XML. We’ll add a lot at once here, but not so much conceptually:

<Map background-color="rgb(51,122,207)" srs="+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs">
  <Style name="hillshade style">
    <Rule name="rule 1">
      <RasterSymbolizer opacity="0.8" scaling="bilinear" mode="multiply" />
    </Rule>
  </Style>
  <Style name="polygon style">
    <Rule name="rule 1">
      <PolygonSymbolizer fill="rgb(255,204,102)" fill-opacity="1" />
    </Rule>
  </Style>
  <Style name="county style">
    <Rule name="rule 1">
      <LineSymbolizer stroke="#CCCCCC" stroke-width="0.4" />
    </Rule>
  </Style>
  <Style name="water style">
    <Rule name="rule 1">
      <PolygonSymbolizer fill="rgb(51,122,207)" fill-opacity="1" />
    </Rule>
  </Style>
  <Layer name="stateborder">
    <StyleName>polygon style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/ca-mercator/ca-mercator.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>
  <Layer name="hillshade">
    <StyleName>hillshade style</StyleName>
    <Datasource>
      <Parameter name="type">gdal</Parameter>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/ca-dem-combined-merc-cutout.tif</Parameter>
    </Datasource>
  </Layer>
  <Layer name="water" srs="+proj=longlat +ellps=WGS84 +no_defs">
    <StyleName>water style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/california_water/california_water.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>
  <Layer name="counties" srs="+proj=longlat +ellps=GRS80 +datum=NAD83 +no_defs">
    <StyleName>county style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/california_county_shoreline/california_county_shoreline.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>
</Map>

We’ve added 3 shapefile layers: one in Mercator projection that you may recognize as the state border shapefile we used to cutout the GeoTIFF in part 1:

  <Layer name="stateborder">
    <StyleName>polygon style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/ca-mercator/ca-mercator.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>

You’ll also notice that we’ve put this layer before the GeoTIFF in our XML. This is because we want the state border to be behind the GeoTIFF in the rendered image.

We’ve specified that this layer uses <StyleName>polygon style</StyleName>, so let’s take a look at the polygon style style, defined above the layers:

  <Style name="polygon style">
    <Rule name="rule 1">
      <PolygonSymbolizer fill="rgb(255,204,102)" fill-opacity="1" />
    </Rule>
  </Style>

This one’s pretty simple. We set a fill color for features in this layer using standard Web RGB syntax and Mapnik’s PolygonSymbolizer, and specify that we want 100% opacity for this layer.

Next let’s turn to the style for the GeoTIFF, which we’ve now changed so it will blend with our state border background layer:

  <Style name="hillshade style">
    <Rule name="rule 1">
      <RasterSymbolizer opacity="0.8" scaling="bilinear" mode="multiply" />
    </Rule>
  </Style>

We’ve changed the opacity of the GeoTIFF layer to 80%, to keep the level of gray down, and we’ve changed the blend mode to multiply. This tells Mapnik how to blend the GeoTIFF with the underlying layers. There are several other modes you can experiment with, but in most cases you’ll probably want multiply.

Now we’ll add a water layer so we can see some of California’s major lakes, and then county borders. Here’s another great thing about Mapnik: You can use many types of data as layers: shapefiles, PostGIS tables and many other ogr types.

Here’s the top 2 layers:

  <Layer name="water" srs="+proj=longlat +ellps=WGS84 +no_defs">
    <StyleName>water style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/california_water/california_water.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>
  <Layer name="counties" srs="+proj=longlat +ellps=GRS80 +datum=NAD83 +no_defs">
    <StyleName>county style</StyleName>
    <Datasource>
      <Parameter name="file">/Users/youruser/Documents/gdaltutorial/california_county_shoreline/california_county_shoreline.shp</Parameter>
      <Parameter name="type">shape</Parameter>
    </Datasource>
  </Layer>

The main thing to notice about these layers is that we’ve specified an additional parameter in the <Layer></Layer> tag: srs. We’re doing this because each of these layers is in a different spatial reference system than our main map. The water layer is in SRID 4326, or standard latitude/longitude pairs. The county layer is in SRID 4269, otherwise known as NAD83. As long as you specify the correct srs, Mapnik will do any reprojection needed on the fly (though not on raster layers, remember), so there’s no need to reproject everything into the same projection on your own.

All that’s left is the style definitions for those two layers:

  <Style name="county style">
    <Rule name="rule 1">
      <LineSymbolizer stroke="#CCCCCC" stroke-width="0.4" />
    </Rule>
  </Style>
  <Style name="water style">
    <Rule name="rule 1">
      <PolygonSymbolizer fill="rgb(51,122,207)" fill-opacity="1" />
    </Rule>
  </Style>

Nothing too fancy here. The county style is using a LineSymbolizer instead of the PolygonSymbolizer we saw earlier. Look - we can use hexadecimal color codes too! And we set the stroke width to 0.4 pixels. The water style colors the water blue – not much more to say than that.

OK, time to render this map. In your Terminal:

$ python ca-stuff-compile.py

If you did everything right, you should have a simple but pretty nice multi-layer Mapnik map.

Since we used a fairly high-resolution GeoTIFF, you should be able to change the image dimensions in ca-stuff-compile.py to make the image quite large for print, HD video, or a poster for your bedroom. But even I don’t do that with my maps!

Unless you want to send me this one.

You’ve just read "Tutorial part 2: Create beautiful hillshade maps from digital elevation models with GDAL and Mapnik" on "Michael Corey, data journalist".

Though for your first project I might recommend a somewhat lower-priority project, I like to think I’m the only person to ever use Mapnik to make wedding invitations.

(If I’m wrong, I’d love to see more examples!)

One of Mapnik’s greatest strengths is its ability to combine multiple types of data into a single map: shapefiles, PostGIS data and raster data are all easy to combine.

When I began working with GIS data I started with exclusively vector data, because vector data are so easily scaled to any size and don’t take up much storage space. Vector layers are also easy to color and style, and are great at keeping your thematic maps clean and easy to read.

But sooner or later you’ll want to add some texture to your maps, and it’s time to add elevation models with raster data.

Luckily, there’s lots of raster data available to get started. The USGS publishes digital elevation models (.DEM) of most of the United States (everywhere except Alaska, I believe). This is definitely high-enough resolution for adding texture to a state-level visualization.

Less luckily, the digital elevation models are sliced up into small pieces. This can be nice because it cuts down on file sizes, but if you want a statewide elevation map, you’ll have to put many pieces together.

A statewide elevation model of California, for example, requires merging 70 .dem files at the 2 arc-second resolution.

However, never fear! GDAL is here to help.

GDAL is a powerful, open-source library of translation scripts for raster geographic data.

So with GDAL, a shapefile of California’s border and a set of digital elevation models and Mapnik, it’s straightforward to make a seamless high-resolution .PNG map of California with nice elevation hillshades.

This tutorial isn’t for people without any GIS or programming experience, but if you have a little of both you’ll be fine. This tutorial is for Mac OS X Snow Leopard users, but will probably work for Linux users with a few OS-related tweaks.

Using GDAL requires knowledge of basic GIS concepts and basic command-line programming. I’ve had good luck installing the GDAL Complete package from kyngchaos.com. Scroll down to the “GDAL 1.8 Complete” .dmg installer.

To get started with raster data, you mostly just need to know what a TIF image is.

To do any Mapnik work, you should feel comfortable with:

• GIS concepts like map projections
• Shapefiles and/or PostGIS
• XML
• Basic command-line programming

The Mapnik website has detailed installation guides and tutorials on how to use it. There’s an excellent OS X-specific guide to installing Mapnik and dependencies (including GDAL and many other useful spacial libraries) here. There’s many ways to do it and I won’t get into that here. Just make sure you have Mapnik running with GDAL support.

The tutorial is divided into two parts. Part 1, this post, will focus on merging, converting, projecting and clipping the digital elevation models so they can be used with Mapnik. We’ll handle the Mapnik process in part 2.

Data mis en place

Here’s the data we’ll use for this tutorial:

• A California border shapefile in Mercator projection (SRID 3395)
• A g-zipped archive of all the digital elevation models you'll need for California

Fun with GDAL

Download and upzip the shapefile and the .dem folder. Then open a terminal window and navigate to wherever you downloaded the shapefile and dem folders.

Then, in the terminal:

$ cd raw-ca-dems

First we’ll merge the digital elevation model tiles into a single .dem file using gdal_merge. We need to specify that we want any areas without data to be white by specifying -init “255”. Then we specify our desired output filename with -o (filename) and then list the files we want to merge together. Like so:

$ gdal_merge.py -init "255" -o ca-dem-combined.dem alturas-e.dem alturas-w.dem bakersfield-e.dem bakersfield-w.dem chico-e.dem chico-w.dem crescent_city-e.dem death_valley-e.dem death_valley-w.dem el_centro-e.dem el_centro-w.dem eureka-e.dem fresno-e.dem fresno-w.dem goldfield-w.dem kingman-e.dem kingman-w.dem klamath_falls-w.dem las_vegas-w.dem long_beach-e.dem long_beach-w.dem los_angeles-e.dem los_angeles-w.dem lovelock-w.dem mariposa-e.dem mariposa-w.dem medford-e.dem medford-w.dem monterey-e.dem monterey-w.dem needles-e.dem needles-w.dem noyo_canyon-e.dem redding-e.dem redding-w.dem reno-e.dem reno-w.dem sacramento-e.dem sacramento-w.dem salton_sea-e.dem salton_sea-w.dem san_bernardino-e.dem san_bernardino-w.dem san_clemente_island-e.dem san_diego-e.dem san_diego-w.dem san_francisco-e.dem san_francisco-w.dem san_jose-e.dem san_jose-w.dem san_luis_obispo-e.dem san_luis_obispo-w.dem santa_ana-e.dem santa_ana-w.dem santa_cruz-e.dem santa_maria-e.dem santa_rosa_island-e.dem santa_rosa-e.dem santa_rosa-w.dem susanville-e.dem susanville-w.dem trona-e.dem trona-w.dem ukiah-e.dem ukiah-w.dem vya-w.dem walker_lake-e.dem walker_lake-w.dem weed-e.dem weed-w.dem

As long as you don’t get any errors, you should now have a new file in your folder called ca-dem-combined.dem. If you have GIS software like Quantum GIS you should be able to add this as a raster layer. If you do open it up, you shouldn’t see any seams between the former tiles – that’s what we want for a good-looking Mapnik layer.

But for this to work in Mapnik, we need a GeoTIFF. To convert the merged .dem to a GeoTIFF, we’ll use gdaldem. The gdaldem command has several modes; we’ll use hillshade to create a shaded relief map, the most common way to visualize texture. We also need to specify the ratio of vertical units to horizontal. Since right now the .dem is in WGS84 projection (standard latitude/longitude), we’ll use degrees, so -s 111120. Then we specify the input file and the output file. Here’s the full command:

$ gdaldem hillshade -s 111120 ca-dem-combined.dem ca-dem-combined.tif

Even if you don’t have a GIS viewer, you should be able to open the .tif file in any image browser like Preview or Photoshop. (Be careful you don’t resave the file, though – most image editors will remove necessary geometric data from the file if you re-save it.) You’ll see a few thin black lines around the outside edge of the old tile data, but don’t worry – we’ll cut this out later.

So now we have a combined .tif, but most of the time we want to display our finished project in Mercator projection – most non-GIS types will think non-Mercator maps look a bit strange. We need to reproject our .tif to Mercator with gdalwarp. We’ll use the -t_srs option to set the target projection, and use the projection specification for the version of Mercator that’s also used by Google Maps and OpenLayers.

$ gdalwarp -t_srs '+proj=merc +lon_0=0 +k=1 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs' ca-dem-combined.tif ca-dem-combined-merc.tif

If you open up the resulting file, you should see that the projection has changed.

Now we have our final hillshade data, but we want to only show California – not all the extra stuff outside the border. We’ll use our California border shapefile as a mask to cut out just the hillshade data inside the boundary. This is a three-step process.

First, we determine the bounding box, or extent, of our clipping mask – the California border shapefile in this case. We use ogrinfo for this, which comes along with GDAL Complete.

$ ogrinfo -al ../ca-mercator/ca-mercator.shp

This will return a lot of details about the shapefile. But if you scroll back to the top of the output you’ll see:

Extent: (-13849389.898804, 3810165.061203) - (-12704836.275367, 5133847.169980)

These represent the longitude and latitude points (in Mercator projection) of the southwest and northeast corners of our shapefile data.

Next, we trim our merged Mercator .tif to those exact boundaries with gdal_translate.

$ gdal_translate -projwin -13849389.898804 5133847.169980 -12704836.275367 3810165.061203 ca-dem-combined-merc.tif ca-dem-combined-merc-box.tif

Warning: Notice that in the above command we reversed the order of the two y-axis values (5133847.169980 and 3810165.061203) from how they appeared in our ogrinfo query. This is because gdal_translate’s -projwin option is looking for the upper left and lower right coordinates – exactly the opposite of the extent we got from ogrinfo. Annoying, but that’s just the way it is.

Finally, we’ll use the shapefile’s polygon boundaries as a clipping mask on the merged, projected and trimmed .tif, which is now trimmed to show the same area as the shapefile. We’ll use gdalwarp for this, using some different options than before. The -co COMPRESS=DEFLATE option is a generic “creation option” that can have many values. We specify -dstalpha to create a “nodata” band in our resulting .tif, so we are left with a transparent background. And finally we use -cutline to specify a file to use as the clipping mask. As before we then specify our input file and desired output filename.

$ gdalwarp -co COMPRESS=DEFLATE -dstalpha -cutline ../ca-mercator/ca-mercator.shp ca-dem-combined-merc-box.tif ca-dem-combined-merc-cutout.tif

And there’s our textured California border. If you open ca-dem-combined-merc-cutout.tif in an image viewer, you should see a beautifully clipped grayscale image.

In the second half of our tutorial, we’ll use this cutout .tif as a layer in Mapnik. We’re more than halfway there!

You’ve just read "Tutorial part 1: Create beautiful hillshade maps from digital elevation models with GDAL and Mapnik" on "Michael Corey, data journalist".