We’ve mentioned in previous posts how Rendering Transformations in GeoServer can produce rich visualization effects such as contour maps and interpolated surfaces, all generated dynamically.

Rendering transformations are implemented as standard GeoServer WPS processes. As processes can be chained together—that is, the output of one process can be the input of another—rendering transformations can be made even more complex and powerful.

One example of a chained rendering transformation is dynamic visualization of temperature contours (also known as isotherms), given a data set consisting of scattered temperature observations from around the globe.

Temperature Observations

A refresher on rendering transformations

Recall that rendering transformations are defined in SLD styles by using the <Transformation> element within a <FeatureTypeStyle>. The process is indicated by a <Function> whose name attribute is the name of the underlying WPS process. The parameters to the process are provided by the arguments to this function, and are themselves wrapped in calls to the special parameter function. This function takes as arguments a <Literal> containing the name of the process parameter being specified, followed by other (optional) <Literal>s containing the argument values.

<Transformation>
  <ogc:Function name="gs:someProcess">
    <ogc:Function name="data">
      <ogc:Literal>someParameter</ogc:Literal>
      <ogc:Literal>someValue</ogc:Literal>    </ogc:Function>
    <ogc:Function name="parameter">
      ...   
    </ogc:Function>
  ...

To be able to be used as a rendering transformation, a process must include one parameter that accepts the data to be transformed (which in this case is the layer being styled). As with other process parameters, this is expressed using the parameter function. The name of the process parameter is determined by the particular process being used. In the simple non-chained case, no arguments are supplied with this parameter; the data is provided implicitly by the SLD rendering engine.

In order to chain transformations, an upstream (inner) process is called as the argument to a downstream (outer) process’s data parameter. The upstream process must also include its own data parameter, which will not have an explicit value (if this is the first/innermost process in the chain). During rendering, the upstream process is executed on the original layer data, and the downstream process is executed on the result of the upstream process. The symbology defined in the SLD must be appropriate for the output format of the final process in the chain, regardless of the original or intermediate formats.

Making isotherms

Getting back to our example, the rendering transformation for isotherms consists of the Barnes Surface process chained to the Contour process. When applied to the temperature data this generates a contour map of estimated average temperatures, as shown in the image below.

Barnes Surface and Contours

The image above is actually displaying three different layers:

1. base map showing world countries
2. layer showing the estimated temperature surface, computed using the Barnes Surface rendering transformation
3. layer showing the isotherm lines generated by contouring the Barnes Surface

Here are the parameters used for this chain:

Barnes Surface:

Contour:

Download the SLD for the Barnes Surface/Contour style

The relevant snippet for the <Transformation> element expressing the chained rendering transformation is shown below. Note how the downstream gs:Contour process has the upstream gs:BarnesSurface process nested inside the data parameter. It’s worth pointing out that the data gets converted from vector to raster and back to vector during this chained process. The symbology in the SLD uses symbolizers appropriate for vector data, as that is the final data format.

One final note: The contour levels are defined explicitly with the Contour process levels parameter. These need to match the range of data values which are computed for the surface.

<Transformation>
  <ogc:Function name="gs:Contour">
    <ogc:Function name="parameter">
      <ogc:Literal>data</ogc:Literal>
      <ogc:Function name="gs:BarnesSurface">
        <ogc:Function name="parameter">
          <ogc:Literal>data</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>valueAttr</ogc:Literal>
          <ogc:Literal>MxTmp</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>dataLimit</ogc:Literal>
          <ogc:Literal>500</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>scale</ogc:Literal>
          <ogc:Literal>15.0</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>convergence</ogc:Literal>
          <ogc:Literal>0.2</ogc:Literal>
        </ogc:Function><ogc:Function name="parameter">
          <ogc:Literal>passes</ogc:Literal>
          <ogc:Literal>3</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>minObservations</ogc:Literal>
          <ogc:Literal>2</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>maxObservationDistance</ogc:Literal>
          <ogc:Literal>15</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>pixelsPerCell</ogc:Literal>
          <ogc:Literal>8</ogc:Literal>
        </ogc:Function>
        <ogc:Function name="parameter">
          <ogc:Literal>outputBBOX</ogc:Literal>
          <ogc:Function name="env">
        <ogc:Literal>wms_bbox</ogc:Literal>
        </ogc:Function>
      </ogc:Function>
      <ogc:Function name="parameter">
        <ogc:Literal>outputWidth</ogc:Literal>
        <ogc:Function name="env">
          <ogc:Literal>wms_width</ogc:Literal>
        </ogc:Function>
      </ogc:Function>
      <ogc:Function name="parameter">
        <ogc:Literal>outputHeight</ogc:Literal>
          <ogc:Function name="env">
            <ogc:Literal>wms_height</ogc:Literal>
          </ogc:Function>
      </ogc:Function>
      <ogc:Function name="parameter">
        <ogc:Literal>queryBuffer</ogc:Literal>
        <ogc:Literal>40</ogc:Literal>
      </ogc:Function>
    </ogc:Function>
  </ogc:Function>
  <ogc:Function name="parameter">
    <ogc:Literal>levels</ogc:Literal>
    <ogc:Literal>-10</ogc:Literal>
    <ogc:Literal>-5</ogc:Literal>
    <ogc:Literal>0</ogc:Literal>
    <ogc:Literal>5</ogc:Literal>
    <ogc:Literal>10</ogc:Literal>
    <ogc:Literal>15</ogc:Literal>
    <ogc:Literal>20</ogc:Literal>
    <ogc:Literal>25</ogc:Literal>
    <ogc:Literal>30</ogc:Literal>
    <ogc:Literal>35</ogc:Literal>
    <ogc:Literal>40</ogc:Literal>
  </ogc:Function>
  <ogc:Function name="parameter">
    <ogc:Literal>simplify</ogc:Literal>
    <ogc:Literal>true</ogc:Literal>
  </ogc:Function>
  </ogc:Function>
</Transformation>

Ideas for new rendering transformations welcome

At the moment there are only a few processes in GeoServer which are suitable for implementing chained rendering transformations. We want to continue expanding our offerings but not in a vacuum. What do you wish you could do with rendering transformations? Leave a comment below to let us know.

My spatial story began when I was a senior in college, as I was completing an Electrical Engineering degree. Throughout my academic career math and physics always came easiest to me and I chose my college major accordingly. Despite the ease I felt with engineering, and the solid job prospects it offered, it left me longing to learn more about society, culture and history. The gap between what I excelled at and what interested me was large, and seemingly impossible to close.

To fulfill graduation requirements I enrolled in the course Historical Geography of the Silk Road and much to my surprise it changed my view of the world. With a pile of historical maps my professor walked us through two thousand years of history, recounting the evolution of the Silk Road, the rise and fall of great cities, how the environment facilitated or challenged the route, and conversely how the road shaped the environment itself. After learning how often the route changed and how strong of a role geography played it it’s history, a flip switched for me. I enrolled in additional geography courses, from urban geography and geomorphology, to more technical-oriented spatial analysis and subsequently a whole series of GIS courses. I began to understand the power of space in explaining social phenomena, and the usefulness of spatial data and technology in helping make critical decisions. Ultimately this led to my PhD study where I explored a range of spatial topics, from urban development to cyberspace security and even advanced data visualization techniques.

Routes of the Silk Road by Han (cyan), Tang (yellow), and Song (green) dynasties. Environmental changes caused some routes obsolete and cities deserted, for example, the city of Loulan.

During the course of my PhD study, two significant technology events occurred, both strongly related to GIS:

• The rise of smart phones, which provide unlimited possibilities to collect spatial data.
• The emergence and acceptance of cloud computing, which empowered spatial analysis and data sharing to an unprecedented level.

These events prompted me to learn more about the geospatial industry, introducing me to OpenGeo. When Jeff Johnson of OpenGeo offered me an opportunity to work as an intern on their development team I jumped at the opportunity. My strong belief that spatial technology can be used to benefit society meshed well with OpenGeo’s commitment to open source spatial technology and I was very excited to join them for the summer of 2012.

I had the pleasure of working with Jeff out of Carlsbad CA, he was a great mentor and had me hit the ground running, coaching me on how to use Git on my very first day. Soon after I was contributing by writing unit and integration tests for GeoNode, increasing testing coverage over 20%. We followed the testing by working on internationalization (known as i18n to the software development industry). By teasing out all necessary terms in GeoNode that needed translating and setting up a user-friendly interface for crowd sourcing translations we were able to draw contributors from many language backgrounds to help with the translation effort.

Working with Jeff Johnson in his favorite café.

While working with Jeff this summer I found myself wondering what benefits OpenGeo could bring to academia. In the university sitting people seek openness of knowledge and access to information and tools. OpenGeo builds strong open and freely available spatial technologies. To me the connection is self-evident; I believe there are at least three things worth considering.

My academic colleagues have a need to share and showcase their spatial analysis with a larger audience. During my PhD work alone I amassed over 70 years of urban data from San Diego and Tijuana. That data took me six months to collect and if it cannot easily be shared it would take every subsequent researcher a similar amount of time. GeoNode, with its tuned upload/download/metadata functionalities, provides a solution to share this information. Moreover, OpenGeo supports some of the best available web mapping technology where advanced data visualization can be implemented (figure below). Additionally, demands for through the web spatial analysis and processing tools are only growing, and the 3.0 release of OpenGeo suite has added a comprehensive set of tools to address these needs

Soon we may find an online database of the Silk Road, with extensive historical and current geographical data, and even exploratory processing tools which will be available and open to everyone. When students and researchers have access to such tools some may be inspired to set up their own expedition and discover a long forgotten city.

A music genres map based on songs played on Last.fm, an online radio station, and users’ tags. Cartographic language combined with online mapping services could provide a more effective way to explore complex multidimensional dataset. For more information please refer to http://cns.iu.edu/research/10-Last.fm.pdf.

The 3.0 release of the OpenGeo Suite added a variety of software features and improvements, including spatial processing capabilities. With scripting support, languages like JavaScript and Python could be leveraged to fit your needs. These scripting languages expose the GeoScript library, allowing users to leverage all GeoScript functions in processing scripts. While OpenGeo Suite 3.0 ships with many useful processes, the additional server-side scripting allows you to deploy many more to fit your enterprise’s specific wants and needs.

You may be surprised to hear that the GeoScript project is nearly 3 years old. In this time it’s accumulated a good deal of functionality. I find myself using the library daily to perform a variety of tasks and I thought sharing some of the most common things I do with GeoScript might be helpful to you.

Data Conversion

Recently I worked on a project that involved analyzing and improving GeoServer rendering performance with Oracle. The first step was to gather a baseline measurement of “acceptable performance”.  In order to measure respective rendering time between the two databases with the same data our plan was to port all the data to PostGIS. As requested the client sent a dump of their Oracle database and we got to work on converting it:

from geoscript.workspace import Oracle, PostGIS

# connect to oracle server
ora = Oracle('oradb', host='10.0.1.3', user='jdeolive', 'passwd=...')

# connect to local postgis
pg = PostGIS('postgis')

# grab all the layers from oracle and dump them into postgis
for layer_name in ora.layers():
layer = ora[layer_name]
pg.add(layer)

ora.close()
pg.close()

Voila! After some GeoScript magic we had a PostGIS database with all the client data.

Styling

One of the most common uses I find for GeoScript is to quickly prototype and create styles. Users of GeoServer know the pain of authoring styles directly in SLD; even with visual tools common tasks like creating “road casings” are often tedious. Here’s an example of some OSM styling that GeoScript made easy.

First I grabbed some OSM Shapefile data:

from geoscript.layer import Shapefile
from geoscript.geom import Bounds
from geoscript.render import draw

# load the data
shp = Shapefile('alberta_highway.shp')

# the area we care about
area = Bounds(-12842844.95, 6636065.22, -12841546.23, 6636833.98, 'epsg:900913')

# draw it
draw(shp, bounds=area, bgcolor='#efebe3')

Then I created a quick casing style with a label:

from geoscript.style import *

# create the casing 
style = Stroke('black', 12, cap='round')
style += Stroke('white', 11, cap='round').zindex(1)

# add a label, following the lines
label = Label('NAME', 'DejaVuSans').line(group=True, follow=True)
style += label

draw(shp, style, bounds=area, bgcolor='#efebe3')

Still a bit cluttered. Time to filter it down to just what I want to see.

# filter to roads we care about
style = style.where("TYPE IN ('residential', 'tertiary')")

draw(shp, style, bounds=area, bgcolor='#efebe3')

Better. Time to export the underlying SLD to dump into GeoServer:

writeSLD(style, 'road_casing.sld')

The end result: road_casing.sld

Fun with Bounding Boxes

I’ve been load-testing GeoServer quite a bit lately and do so I have to generate test plans that execute a number of WMS requests. In this case I wanted the tests to simulate a typical scenario of concentrated access centering on major cities.

The first step was to grab a dataset with city limits:

from geoscript.layer import Shapefile

# builtup area from 
shp = Shapefile('world_boundaries/builtup_area.shp')

# city names to include
cities = ['CALGARY', 'VANCOUVER', 'DENVER', 'CHICAGO']

# world bounds in web mercator
box = Bounds(-20037508,-19929239,20037508,19929239, 'epsg:900913')

Next we’ll loop through all the cities I’m interested in and get the city bounds in Web Mercator.

for city_name in cities:
  # union all the polygons into one geometry
  city_area = reduce(lambda x,y: x.union(y),
    [f.geom for f in shp.features("nam = '%s'" % city_name)])

  # reproject to web mercator
  city_bnds = Bounds(env=city_area.bounds(), prj='epsg:3395').reproject('epsg:900913')

Last we generate bounding box tiles that intersect the city limits for the zoom levels I want available:

  # go from min to max zooms
  for z in range(8, 15):
    # get resolution of this zoom level
    res = 1.0/pow(2,z)

    # tile our main bbox at this resolution within the city bounds
    for t in box.tiles(r, city_bnds):
      print '%f,%f,%f,%f\n' % (t.west, t.south, t.east, t.north)

What’s Next?

The next big thing on the GeoScript roadmap is raster data support. It’s something I’m particularly excited about  and happy to say that some of us at OpenGeo have been busy working on. Here is a quick sample of what’s to come:

Grab some raster data:

from geoscript.layer import GeoTIFF
dem = GeoTIFF('sfdem.tif')

Do some analysis on it:

dem.bands()
# [GRAY_INDEX]

dem.extrema()
# ([-9.999999933815811e+36], [1840.0])

histo = dem.histogram(low=0, high=1840)

Render the histogram:

from geoscript.plot import bar

data = zip(map(lambda (x,y): y, h.bins()), h.counts())
bar.xy(data).show()

Build a color map and render the dem:

from geoscript.style import *

# create the intervals
intervals = range(0,2000,10)

# interpolate some colors
colors = Color('white').interpolate(Color('black'), n=len(intervals))

# build the color map and render
draw(dem, ColorMap(zip(intervals, colors)))

And there you have it, raster analysis made easy with GeoScript. Remember that these are just a few examples of what’s possible with server-side scripting. Download the OpenGeo Suite and try it out for yourselves. We love feedback so if you have any questions or comments about GeoScript, or wrote something that you want to share please let us know. Leave a comment on this post or send us an email. Thanks for reading and happy scripting.

Last week registration opened for FedGeo Day, a one-day conference being held on February 28 in Washington, DC. The event offers a platform for those who work in or with federal agencies to share their experiences with modern open source geospatial technologies.

We’re looking forward to seeing case studies of how these tools are being used in government, what benefits they bring, and why so many agencies are shifting to open source technology. There will also be interactive demos that explore specific open source tools and applications. The agenda will be particularly attractive for those for those who make technology decisions within the federal government. OpenGeo will be presenting how the advances to server-side processing (added to the OpenGeo Suite in the 3.0 release) are altering the landscape for ‘traditional’ GIS.

OpenGeo and MapBox are acting the primary organizers and we thank RadiantBlue, MapStory, GeoEye, and Arc2Earth for joining us in sponsoring the event. A full schedule will be available soon at fedgeoday.com. Sign up now to reserve your spot, as space is limited.

Following the success of the OpenGeo Suite 3.0 release, we’re proud to release 3.0.1. This release is primarily a maintenance and quality assurance release that contains several bug fixes, improved documentation, and component updates like GeoServer 2.2.1. OpenGeo Suite 3.0.1 is available for download free of charge with a 30-day trial of OpenGeo’s commercial support.

We’ve also introduced new offerings—Community One-Time and Enterprise Plus—to meet growing demand from smaller organizations seeking support for open source geospatial software. Community One-Time offers enterprise support to Community Edition users for one incident for up to 15 business days. Enterprise Plus provides a year of enterprise support for smaller production environments with straightforward support requirements, such as those publishing data from basic geospatial formats and serving a limited number of users.

More information is available in the release notes and our pricing page.

OpenGeo is excited to share the GeoServer CSS module, an alternative to SLD for thematic map styling in the OpenGeo Suite. Compared to SLD, GeoServer CSS brings many advantages for web cartographers.

GeoServer CSS is customized for use in GeoServer and provides access to spatial utilities such as powerful filter expressions that are useful not only for choosing which features to render but also for transforming geometries and other attributes on-the-fly during the rendering process. Thanks to its custom syntax (which builds on OGC CQL the same way that SLD builds on OGC Filter Encoding XML) it is much more compact, allowing authors to see more of the style without scrolling and to make edits without worrying about details such as closing nested XML tags in the proper order. GeoServer CSS is also simpler because it eschews some non-styling aspects of SLD such as embedded geometries.

As an example, let’s say we are styling polygons representing the habitats of several bird species. We would like to distinguish species by color (orioles have an orange area, bluebirds have a blue one, etc.), but all polygons should have a consistent opacity to help us identify places where habitats overlap. A style in CSS notation might look like this:

[species="bluebird"] {
    fill: blue;
    fill-opacity: 70%;
}

[species="oriole"] {
    fill: orange;
    fill-opacity: 70%;
}

[species="robin"] {
    fill: brown;
    fill-opacity: 70%;
}

Read the rest of this entry »

And, late on a Friday afternoon, the plaintive cry was heard!

And indeed, into the sea they do go!

And ‘lo, the SQL faeries were curious, and gave it a shot!

##### Commandline OSX/Linux #####

# Get the Shape files
# http://www.elections.bc.ca/index.php/voting/electoral-maps-profiles/
wget http://www.elections.bc.ca/docs/map/redis08/GIS/ED_Province.exe

# Exe? No prob, it's actually a self-extracting ZIP
unzip ED_Province

# Get a PostGIS database ready for the data
createdb ed_clip
psql -c "create extension postgis" -d ed_clip

# Load into PostGIS
# The .prj says it is "Canada Albers Equal Area", but they
# lie! It's actually BC Albers, EPSG:3005
shp2pgsql -s 3005 -i -I ED_Province ed | psql -d ed_clip

# We need some ocean! Use Natural Earth...
# http://www.naturalearthdata.com/downloads/
wget http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/10m/physical/ne_10m_ocean.zip
unzip ne_10m_ocean.zip

# Load the Ocean into PostGIS!
shp2pgsql -s 4326 -i -I ne_10m_ocean ocean | psql -d ed_clip

# OK, now we connect to PostGIS and start working in SQL
psql -e ed_clip

##### SQL ######

# How big is the Ocean table?
SELECT Count(*) FROM ocean;

# Oh, only 1 polygon. Well, that makes it easy... 
# For each electoral district, we want to difference away the ocean.
# The ocean is a one big polygon, this will take a while (if we
# were being more subtle, we'd first clip the ocean down to 
# a reasonable area around BC.)
CREATE TABLE ed_clipped AS
SELECT 
  CASE 
  WHEN ST_Intersects(o.geom, ST_Transform(e.geom,4326))
  THEN ST_Difference(ST_Transform(e.geom,4326), o.geom)
  ELSE ST_Transform(e.geom,4326)
  END AS geom,
  e.edabbr,
  e.edname
FROM ed e, ocean o;

# Check our geometry types...
SELECT DISTINCT ST_GeometryType(geom) FROM ed_clipped;

# Oh, they are heterogeneous. Let's force them all multi
UPDATE ed_clipped SET geom = ST_Multi(geom);

##### Commandline OSX/Linux #####

# Dump the result out of the database back into shapes
pgsql2shp -f ed2009_ocean ed_clip ed_clipped
zip ed2009_ocean.zip ed2009_ocean.*
mv ed2009_ocean.zip ~/Dropbox/Public/

No more districts in oceans!

And the faeries were happy, and uploaded their polygons!

Update: And the lamentations ended, and the faeries also rejoiced.

This year GIS Day fell on Wednesday, November 14, which means that today is PostGIS Day! How are you celebrating? At OpenGeo we’re spending time with one of our newest clients, Washington University. Today in St. Louis, superstar trainer Sam Smith will be kicking off our relationship with Washington University by leading various sessions on open source geospatial technology and OpenGeo Suite training for students and staff.  If you’re not lucky enough to work or attend Washington University we’d like to offer all educators and students a special PostGIS day rate for our online training courses. If you’re interested contact us with a valid .edu email address and we’ll set you up with a discount off the listed prices.

We know that many higher education institutions have recognized that open source geospatial software not only outperforms proprietary alternatives by many measures, but learning it has become a critical job skill students are expected to have acquired. We saw this first hand at the 2012 AAG Annual Meeting where we met a handful of forward thinking educators who told us that they were actively adding web mapping and open source technology courses to their curriculums. It was a great opportunity to speak with students and educators committed to working with open source geospatial tools. We think that this enthusiasm will only grow and are eager to see GIS courses focus an increasing amount of lecture time on open source geospatial software.

If you work at a college or university and are interested in finding out how an OpenGeo academic support contract can assist you with your teaching and research goals send us a note to start a discussion.

Happy PostGIS day!

Below you’ll find a fresh round of spatial IT job postings. We hope these posts help connect organizations seeking to develop and support geospatial technologies with the talent that knows the best ways to do so. Our special thanks to Josh Campbell for letting us know about the State Department job openings.

Job Listings: 

• Lead Software Engineer, Forward Slope
• GIS Developer, Computech
• GIS Business Analyst, US Department of State
• GIS Developer, US Department of State
• Software Engineer, LMN Solutions
• Staff GIS Analyst, Telenav
• Java/GIS Developer, General Dynamics
• GIS Web Application Developer, Farallon Geographics
• Technical Writer, SAIC
• GIS Applications & System Administrator, Pacific Northwest National Laboratory
• Project Scientist, NGA
• Senior Software Engineer, Columbia University

As always, please let us know if we’ve missed any job postings, either leave a comment or email us and we’ll be sure to include them next month.

OpenLayers recently launched a crowd-funding campaign for their 3.0 release. I want to share a bit about why OpenLayers is important to me, and why OpenGeo has supported OpenLayers and will continue to invest in it. If you’re interested in helping, visit the OpenLayers 3.0 campaign for more information and ways to support.

When OpenLayers arrived six years ago it was a revelation; it took the awesome tile-based AJAX-ified magic of Google Maps and made it available to the entire geospatial world. The community was active and growing fast; they were rolling out new formats and innovations daily. Back then, OpenLayers truly pushed the limits of what was possible in the browser, establishing a new standard for the geospatial world.

In the past six years OpenLayers has matured into the most widely used geospatial JavaScript library, but the toolkit has begun to show its age. The wider JavaScript world has pioneered new techniques expanding the limits of browsers and younger front-end mapping libraries have been taking advantage of these innovations. On the other hand, OpenLayers has accumulated some quirks which stem from the desire to keep everything backwards compatible, never breaking old applications. Keeping the code of a maturing library backwards-compatible is a great asset for its existing users, but it makes things difficult for newer ones. It’s time to give OpenLayers the overhaul it needs to take take advantage of new JavaScript technology

Read the rest of this entry »