Archive for the ‘Technology’ Category

GeoNode: Call for Testers

For the past year we’ve been excited to develop a new software project in partnership with the World Bank: GeoNode.

GeoNode aims to be the next generation of spatial data infrastructure. Some of it—such as GeoExplorer with integrated styler or data publishing through open standards—will be familiar to users of the OpenGeo Suite. What GeoNode provides on top is search enabled by an ISO metadata catalog and an integrated front-end for user management, security, data browsing, cartography, and more.

The best way to learn about it is to try it yourself on the live demo!

We are almost ready for our 1.0 release, and we need your help testing GeoNode.  So please, make an account and give it a shot.  If you find any issues, please don’t hesitate to report them on our project Trac.  We’ll be squashing bugs and redeploying regularly in the upcoming weeks.

Our hope is to have a robust release candidate available for a grand unveiling at FOSS4G this year.  See you there!

WMS Benchmarking

As in past years, OpenGeo is participating in the annual WMS benchmarking exercise that OSGeo sponsors. The exercise is a great opportunity for development teams to learn about where their project needs to concentrate development effect to make things faster and more compliant in the next year. Our coordinator this year (as in the past couple years) is Andrea Aime, who is working hard on both making sure GeoServer is properly configured for the tests, and is providing JMeter expertise to the entire exercise. The results of the benchmarking will be presented at FOSS4G 2010 next month.

In previous years, the exercise has been just MapServer versus GeoServer, and both projects have learned a lot in the process. Last year ESRI joined the process, but unfortunately ran out of staff time before the results were complete. This year the participation has exploded! The final number of teams that complete the testing might be smaller, but as it stands now the participants include: MapServer, GeoServer, CadCorp GeognoSIS, Constellation SDI, ERDAS Apollo, Mapnik, Oracle MapViewer, and QGIS mapserver.

As the participants get their servers online and configured to use identical data and styles, it becomes possible to compare their results visually. To make the comparison a little more dynamic, I wrote a small multi-map page that uses OpenLayers to view all the servers at once, looking at the same area.

WMS Browser

It’s really fun to see so many different implementations chew up the same data and styles and spit out the same map. Standards work? Standards work!

Shape of a Polygon

What shape is this polygon?

POLYGON((-40.0 52.0, 102.0 -6.0, -67.0 -29.0, -40.0 52.0))

Of course, that depends on what projection it is in, but let’s skip cartesian systems and look at it as a geographic shape on the sphere. Google Earth thinks it is shaped like this:

GEarth

But I don’t really think so. It all comes down to how you interpret the edges. I’m not sure what rule Google Earth is using to interpret the edges, but they aren’t Great Circles, because this is what a Great Circle map of the polygon looks like:

See how far south and north they get? Way different from the Google Earth interpretation. If anyone knows what rule Google Earth uses, I’d love to hear it.

Update: Here is the KML if you want it.

<?xml version="1.0" encoding="utf-8" ?>
<kml>
  <Placemark>
    <Polygon>
    <outerBoundaryIs><LinearRing><coordinates>
    -40,52 102,-6 -67,-29 -40,52
    </coordinates></LinearRing></outerBoundaryIs>
    </Polygon>
  </Placemark>
</kml>

Update 2: In the comments, GeoXP posits and Mano Marks confirms that the polygon edges are rhumb lines. Mano notes that curiously, if you draw the same lines as, er, lines, instead of polygons, they will follow great circles instead.

Update 3: Once I take the bit between my teeth, you can’t stop me. In the KML 2.2 specification, the OGC has also ratified this fun interpolation scheme! “The boundary control points of each descendent kml:LinearRing are first projected to the plate carrée plane (where altitude is dropped), then straight line segment interpolation in the plate carrée (long,lat) plane is used between consecutive control points. The interior points are then filled in linearly in the plate carrée plane. Finally, the (long,lat) points of the polygon in the plate carrée plane are mapped back to (long.lat,alt) points on the earth’s terrain surface model.”

Update 4: In the comments, Mike Sumner suggested that I try one over the dateline because it’s “much dumber”. Who can resist an offer like that? So I just translated the whole polygon 150 degrees to the west, giving us this KML:

<?xml version="1.0" encoding="utf-8" ?>
<kml>
  <Placemark>
    <Polygon>
    <outerBoundaryIs><LinearRing><coordinates>
    170,52 -50,-6 143,-29 170,52
    </coordinates></LinearRing></outerBoundaryIs>
    </Polygon>
  </Placemark>
</kml>

And here is what it looks like in the venerable Google Earth.

Dateline Crossing

Basically, it refuses to cross the dateline, it takes the long way around. Which, if you read the OGC description, is entirely unsurprising, since all the interpretation of the edges is happening in cartesian space, in plate carrée, where there is no “dateline” and Siberia and Alaska are at opposite sides of the plane, farther apart than New York and London.

Update 5: Last one. Because the effect is hard to see in the last example, see how this small triangle over the dateline is rendered (in green, below):

POLYGON((170 72, -170 70, -165 54, 170 72))

A polygon enclosing the pole looks even cooler (in yellow, below).

POLYGON((45 85, 135 80, -135 85, -45 80, 45 85))

That dateline is sacrosanct!

Google Earth Dateline 2

Network Walking in PostGIS

One of the new features in PostgreSQL 8.4 was the “WITH RECURSIVE” clause available for queries. It allows you to define a subquery based on a recursive term — fancy language for a function that calls itself. One of the favorite uses of recursion is walking a network. Geospatial applications use networks all the time: electrical grids, stream systems, and storm sewers are all directed networks (they have unidirectional flow).

Here’s an example of network walking using a simple collection of segments. As is common in many GIS applications, the segment are implicitly connected — their end points are coincident with the start points of other segments.

CREATE TABLE network ( segment geometry, id integer primary key );

INSERT INTO network VALUES ('LINESTRING(1 1, 0 0)', 1);
INSERT INTO network VALUES ('LINESTRING(2 1, 1 1)', 2);
INSERT INTO network VALUES ('LINESTRING(1 2, 1 1)', 3);
INSERT INTO network VALUES ('LINESTRING(3 1, 2 1)', 4);
INSERT INTO network VALUES ('LINESTRING(3 2, 2 1)', 5);
INSERT INTO network VALUES ('LINESTRING(2 3, 1 2)', 6);
INSERT INTO network VALUES ('LINESTRING(1 3, 1 2)', 7);
INSERT INTO network VALUES ('LINESTRING(4 2, 3 2)', 8);
INSERT INTO network VALUES ('LINESTRING(3 4, 2 3)', 9);
INSERT INTO network VALUES ('LINESTRING(2 4, 2 3)', 10);
INSERT INTO network VALUES ('LINESTRING(1 4, 1 3)', 11);
INSERT INTO network VALUES ('LINESTRING(4 3, 4 2)', 12);
INSERT INTO network VALUES ('LINESTRING(4 4, 3 4)', 13);

CREATE INDEX network_gix ON network USING GIST (segment);

Visually, the network looks like this:

Network

To walk our network, use a WITH clause that starts with one segment, then repeatedly adds the next downstream segment to the collection. In our case, the “next downstream segment” is defined as a segment whose start point is close to the end point of the current segment. We’ll walk down from segment 6.

WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = 6
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT id
FROM walk_network

Which returns:

 id
----
  6
  3
  1
(3 rows)

From 6 to 3 to 1, correct! Once we have our walker producing the results we want, we can wrap more PostGIS and PostgreSQL functions around the walker to produce a finished product. Here’s a function that takes in an edge identifier and outputs a linestring based on the downstream path.

CREATE OR REPLACE FUNCTION downstream(integer)
RETURNS geometry
LANGUAGE sql
AS '
WITH RECURSIVE walk_network(id, segment) AS (
    SELECT id, segment FROM network WHERE id = $1
  UNION ALL
    SELECT n.id, n.segment
    FROM network n, walk_network w
    WHERE ST_DWithin(ST_EndPoint(w.segment),ST_StartPoint(n.segment),0.01)
  )
SELECT ST_MakeLine(ST_EndPoint(segment))
FROM walk_network
' IMMUTABLE;

And here’s the function in action, generating the downstream path from segment 12.

=# SELECT ST_AsText(Downstream(12));

            st_astext
---------------------------------
 LINESTRING(4 2,3 2,2 1,1 1,0 0)
(1 row)

Check the generated path against our network picture — looks good!

Path 12

Social goals are (more) motivating

I work for a purpose-driven organization, the social purpose of the organization aligns with my own social purposes, and the work is challenging and sophisticated.

Open source fits the profile of an intrinsically motivating pursuit (for folks of the right technical bent) exceedingly well. A great talk and visual presentation.

OpenGeo Suite 1.9 – Community Edition

We have been hard at work on improving the OpenGeo Suite. Part of our development process is to make interim releases available as Community Edition downloads.  These downloads are functional and ready to try out, but they could still use some more testing. Please download and try out the latest Community Edition and give us feedback on our new community forum.

And you definitely want to try out this new release. Why? Check out some of the new features:

  • PostGIS is now integrated into the OpenGeo Suite install experience! You now get a complete spatial mapping stack: database, map server, tile cache, web interface. Or, in project terms: PostGIS, GeoServer, GeoWebCache, OpenLayers, GeoExt!
  • pgShapeLoader, a graphical interface for loading shapefiles into your PostGIS database
  • pgAdmin a graphical interface for database administration.
  • GeoEditor, a web application. GeoEditor allows for web based editing of data, regardless of the underlying data format (shapefiles, PostGIS, ArcSDE, Oracle Spatial, DB2, etc.). This offers a great opportunity to integrate web based applications with traditional GIS workflows.

We currently have one-click installers available for Windows and Mac OS X. We will be adding an installer for Linux very soon.

It’s worth noting what you’re getting with the OpenGeo Suite. With a simple and quick installation, you have the functional equivalent of ArcGIS Server and Oracle Spatial.

Download the OpenGeo Suite Community Edition

PostgreSQL Big Ideas

Robert Haas has a nice post looking forward to the release-after-this-one of PostgreSQL (the next release will be 9.0, but Robert is looking towards 9.1). He contextualizes his ideas like this:

But I think it might be good to step back and look at the problem a bit more broadly. Ignoring for a minute what people actually are working on, what should they be working on? Where is PostgreSQL strong as a product and where does it need improvement?

I strongly recommend you go read Robert’s ideas on new features, but here are my favorites, from the point of view of making PostGIS even better:

  • KNNGiST, almost made it into 9.0, an extremely important feature to allow PostGIS to do index-assisted nearest neighbor searches (”find the N closest items to this item”).
  • Formal packaging system for extensions. Right now there is no way to distinguish functions installed by PostGIS with other functions installed by the user. So an “uninstall PostGIS” routine, or a “backup everything but not the PostGIS functions” has to be needlessly complex.
  • Framework for multi-threading. PostGIS is on the front-lines of database functions needing a standardized system for doing parallel processing. Calculations like point-in-polygon are easily parallelizeable, but it would be nice to do so in a way that is standard for PostgreSQL. Also, aggregate calculations are also often susceptible to parallel processing.

What features are you dreaming of for PostgreSQL and PostGIS? Maybe it is something on the core development menu for PostGIS already.

The Real World

I spent three days at the Washington GIS conference last week, and it was a useful reminder about the real world. That is, the world where technology is a means and not an end.

The comparison with an event like Where 2.0 is pretty stark: the metric of evaluation is decidedly not “is it cool”, but rather “is it useful”.

It was gratifying therefore to see “open source” moving into the “useful” column in such practical organizations. A surprising number of presentations were about the use of open source in deployed projects.

Even so, most of the organizations there were still at the “intrigued but interested in learning more stage”. And being practical people, they asked practical questions like “what is the first thing I should do to start experimenting with open source?

That question put me back on my heels. There’s lots of things one can do to start with open source, but what is the logical very first thing?

The self-serving answer would have been “download the OpenGeo Suite and get to it!”. But the question came from a GIS Manager, so it was not a matter of personal enlightenment, but organizational direction. What can he ask his subordinates to do that will make this new technology option available to his organization?

Fortunately, there is a success story right in the area, Pierce County. A big chunk of the talks about use of open source were from Pierce County employees. Back in 2007, the GIS Manager sent a group of her best technical staff up to Victoria, BC, for the FOSS4G conference to “look for some options” in solving some nagging technology issues they had.

And they came back with lots of options.

So then she empowered them to pick some and try them on pilot projects, which they did. And the ideas that worked found their way into core standards of the county. And now GeoServer is used for some of the map rendering, and OpenLayers is the new standard web map component. And they haven’t thrown out their old infrastructure, much of it is still there: they’ve just improved on it and provided a more diverse road map for the future.

So, here is a draft program for curious managers:

  • Decide whether you have a problem. Change will always cost a bit, in terms of convenience and time and learning new ways of doing thing at least. So if you are happy the way you are, maybe change isn’t for you. It might be a good idea to talk to your technical team and see if they are happy too.
  • Select a technical core team to learn about the options and empower them to do so. That might mean bringing in training, or just providing “free time” to learn about and report on options, or if you’re lucky sending them to a conference like FOSS4G.
  • Encourage the team to learn not just the technology but also the culture of open source. Sign on to mailing lists, attend local user group meetings, get to know how open source is actually developed and by whom. That soft knowledge will be incredible valuable if you start to depend more on open source. (Why? Questions like “will that feature be in the next release?”, “how can I get that bug fixed by Wednesday?”, “does it work (well) with component X?” are all easier to answer for people with the soft knowledge of community interaction and resources.)
  • Give that team some problems to solve. Pilot projects, one-time projects, etc. At the same time, ask them to integrate their solutions into the existing infrastructure as much as they can. You want your team to develop knowledge in integration because you’ll never change your whole infrastructure in one go. Having your team learn to build silo’ed all-open-source stacks won’t help in the long run.

The organizational reality of most counties and cities in Washington, and presumably in most of North America, is that their GIS technology is provided overwhelmingly by ESRI. That means that new solutions have to integrate with that stack. Fortunately, there are a number of integration points: at the database level, below the database level, at the user interface level.

Have a look at the OpenGeo architecture for a generic discussion of heterogeneous technology integration. We’ll be publishing a white paper about the specifics of integration with ESRI this spring, based on the many useful things I learned from talking with the good folks at Pierce County!

Gooey

The other thing I really enjoyed while re-tooling my geo-stack workshop was re-writing the PostGIS section to use the new GUI loader instead of the command-line tools.

shp2pgsql-gui.exe

The GUI was actually one of the first projects I worked on after coming to OpenGeo, and was a part of the 1.4 release. But it didn’t build in Windows, so nobody used it! For 1.5 I got myself a Windows build environment and made it work, so here it is! Another couple pages of boilerplate removed from the workshop.

Update: Soon to be available as “gooey apple” in the next release of the Suite.

OSX shp2pgsql-gui

Sweet Suite

As egregious as this self-horn-tooting is, I have to post about how enjoyable I have found using the OpenGeo Suite. It might sound odd, coming from an OpenGeo person, to “discover” the Suite at this late date but my personal reality is that I spend much of my time with PostGIS, and until recently only did short testing passes over the Suite.

But this last week, I have been revising a workshop that I will be giving with Steven Citron-Pousty at Where 2.0 this month, possibly at GeoWeb in the summer (and on my own at the Minnesota GIS/LIS conference in the fall). And included in that revision is using the OpenGeo Suite instead of vanilla GeoServer.

Among the things I’ve really enjoyed are:

  • How easy the install process is. No messing around installing JDKs or JAI extension libraries.
  • How clean the new UI is. Much more sensible names for things and less clicky workflow.
  • The instant gratification of the Suite tools. Installer creates datastore and layers and associated styles in one go. Styler writes new SLD with a click click click.
  • The little On/Off button in the Dashboard. For some reason I really like that.

The net effect of all the cleanups is I’ve removed about 6 screens of configuration boilerplate from the workshop. More time to talk concepts and scare the students with JavaScript examples!