Blog

Blog

Look Out DC!

We’re super excited about  this year’s FOSS4G North America Conference. OpenGeo is sending pretty big group to DC. Not only are we Gold Sponsors, we also have nearly a dozen OpenGeo presentations on the program, our own Sponsor Day event, and Paul Ramsey volunteered as conference chair! We’re looking forward to an exciting (and busy!) week.

Remember to register for the conference. We hear there are still spots but they are filling up quickly. If you’re in attendance make sure to come by our exhibition table, you never know who you’ll run into.

Interested in hearing us speak? Scroll down to see  list of all of the OpenGeo talks on the program:

Tuesday April 10
Time Session Title Location
09:00am Plenary
Paul Ramsey
West Salon
10:30am Introduction to PostGIS
Paul Ramsey
Room 103ab
01:00pm The State of Geoserver
Justin Deoliveira
Room 102b
01:30pm JEQL – A language for Spatial Processing
Martin Davis
Room 102b
02:00pm Scripting GeoServer with GeoScript
Tim Schaub
Room 102b
04:00pm OpenLayers: The Rebirth of Cool
Tim Schaub
Room 103ab
Wednesday April 11
Time Session Title Location
10:30am What’s New in PostGIS 2.0
Paul Ramsey
Room 103ab
01:00pm What’s new in GeoServer 2.2
Justin Deoliveira
Room 103ab
01:30pm What’s New in the JTS Topology Suite
Martin Davis
Room 103ab
02:00pm GeoServer in Production
Juan Marin
Room 103ab
03:00pm The State of the GeoNode Project
Jeff Johnson
Room 102b
03:30pm Taking Control: How GeoNode automates GeoServer configuration
David Winslow
Room 103ab

OpenGeo Suite 2.4.5 released

We are excited to release a new version of the OpenGeo Suite! In order to capture the many improvements and bug fixes happening in the open source community, we are moving toward a more rapid release cycle. For example, GeoServer now has JDBC datastore session startup/teardown SQL comments, as well as support for paletted PNG images with alpha transparency.

In GeoExplorer (which really is pretty amazing, if you haven’t seen it recently) there is now smoother tile display, including fade-in. Also, the map legend has now been integrated directly into the layer tree. Finally, we have changed the default base layer to be MapQest OSM, moving away from Google (though Google base layers are still available).

All of these new features are available in the Community Edition, Enterprise Edition (which includes a free 30 day trial of our support), and all Cloud Editions! Try any version you’d like and contact us to purchase the support you need to put your project into production!

PostGIS 2.0 New Features: 3D/4D Indexing

Another feature that has received relatively scant attention in the upcoming release is the ability to index in higher dimensions than the usual two.

Initially I had hoped to made n-dimensional indexing the default, so that multi-dimensional indexes would just automagically be available. Unfortunately it turned out that the overhead involved in having a n-dimensional index key caused the index to be slower.

Rather than slow down the standard use case (2D searching), I re-instated fixed two-dimensional indexing as the default for geometry, and added a new N-D index type for those who want the thrill of extra dimensions.

To build an N-D index, use the following CREATE INDEX statement:

CREATE INDEX my_nd_index
ON my_table USING GIST (geom gist_geometry_ops_nd);

To query your N-D index, use the &&& operator (note! three ampersands!), like so:

SELECT * FROM my_table
WHERE geom &&& 'LINESTRING(0 0 0, 2 2 2)';

(“Hey, what’s with the LINESTRING?” Because the index-only query is bounding-volume based, a linestring from the lower-left to the upper-right of my desired bounding volume will have exactly the bounding volume I want.)

Some of the fancy new 3D functions like ST_3DDistance can make use of the 3D index for fast searches in high dimensional spaces. I imagine folks working with XYT data may also find the new indexes useful.

OpenGeo Connections: Meet David Winslow

Welcome to another edition of OpenGeo Connections. Today we’re sitting down with David Winslow, technical lead on GeoNode. David works out of OpenGeo’s NYC headquarters, is a member of the GeoNode Project Steering Committee, and a core commiter to GeoServer. David has been with OpenGeo for over four years and is an integral part of our GeoNode team.

OpenGeo’s David Dubovsky: Hi David, thanks for sitting down to chat.
David Winslow: It’s my pleasure.

(DD): Let’s jump right in –  where are you from, where are you now, and how long have you been with OpenGeo?
(DW): I’m a Duke University comp-sci alum, I grew up in Virginia and now I live in Brooklyn. I’ve been with OpenGeo for about four and a half years now.

(DD): Four and a half years? Does that make you an elder statesman at OpenGeo?
(DW): We’ve grown a lot over that time but I wouldn’t say I’m an elder anything.

(DD): So, what brought you to join OpenGeo?
(DW): Before I came here I was studying computer science at Duke. I was attending a carrer fair when  I met Vanessa Hamer, now our Vice President of Operations. The rest is history, I found her pitch intriguing as I wanted to work at a place that was devoted to open source and open data.

(DD): So you studied computer science? What about web mapping and open source?
(DW): At the time mapping was new to me. When I joined OpenPlans, which OpenGeo is a division of, we were called The Open Planning Project (TOPP). I was hired to work on a OpenCore, a tool that provided basic project infrastructure for those rallying around a cause – it included things like a wiki, calendar, blog etc.

(DD): And how did web mapping get onto your resume?
(DW): I really consider myself more of a nerd than a geo-nerd. Chris Holmes had helped develop GeoServer while at TOPP. OpenCore was one of his first implementations of GeoServer. The idea was for users would to utilize maps to share and disperse information. This would enable volunteers to identify points of interest, concern, in need of attention etc.

(DD): And Chris pulled you in to help with that?
(DW): Yes, I helped Chris integrate GeoServer, and that was my first experience with open source geospatial.

(DD): That’s great. So what are you primarily working on these days?
(DW): For OpenGeo I work mostly on GeoNode, both supporting the community work – making sure releases come out, helping with new development etc, and for OpenGeo I perform custom deployments and bringing new features into the project. Recent deployments required customization that would be beneficial to the broader community; right now I’m working on brigning those into main GeoNode project

In my spare time I’ve been working on GeoServer styling with CSS. Right now you can only edit styles for GeoServer in an XML format called SLD. CSS is a lot less complex to edit and would would really be a benefit to the community.

(DD): Interesting, the word is that your GeoNode work has had you traveling a lot, where have you been?
(DW): You can say that, last year I traveled to Fiji, Portland, Zurich, Jakarta, Denver for FOSS4G, the list goes on.

(DD): Wow, what were you doing in the South Pacific?
(DW): I was in Fiji for a GeoNode deployment at the Secretariat of the Pacific Community (SOPAC). The SPC acts as an information broker for all the South Pacific nations. They’re using GeoNode to serve up data they’ve collected and the results of a large study on disater management

In Jakarta I worked with The Australia-Indonesia Facility for Disaster Reduction (AIFDR) on a project called Risk in a box. They’re helping inform people who don’t have the ability to make disaster models what would happen if a disaster did strike their area. For example, what would happen to their community in the event of a four meter storm surge. The online calculator is based on GeoNode and all of the calculations are published as GeoNode data sets and layers.

(DD): Very interesting, has the travel slowed down at all
(DW): Well OpenGeo will be in Washington D.C. for FOSS4G North America and then in Portland for an organization wide get-together - so no, but the flights are shorter.

(DD): So what’s your claim to fame in OpenGeo?
(DW): I’ve touched at least one line of code in every piece of the OpenGeo stack – PostGIS, GeoExt, GeoServer, REST.API, KML etc. I don’t think many others in our organization can say that.

(DD): Well, I’m sure you’ll get some competition for that now. Thanks so much for letting us pick your brain.
(DW): I’m happy to talk any time.

If you want to get in touch with David look out for dwins in #geoserver or #geonode. He’s also active on the the GeoServer and GeoNode mailing lists.

PostGIS 2.0 New Features: Typmod

The new raster type gets most of the press in the PostGIS 2.0 features round-ups, but one of my personal favorites is the support for “typmod” on geometry objects.

A “typmod” is a “type modifier”. If you’ve used SQL databases you’re probably familiar with type modifiers on variable length character columns, like this one:

CREATE TABLE test (
  description VARCHAR(255)
)

The “255″ after the “VARCHAR” is a modifier, specifying more information about the column. For geometry columns in PostGIS 2.0, it’s now possible to define a column like this:

CREATE TABLE geotbl (
  geom GEOMETRY(PointZ, 4326)
)

Note the modifiers on the geometry, specifying the specific type “Point”, the dimensionality “Z” and the spatial reference identifier “4326″. And the table description reflects the typmod information:

           Table "public.geotbl"
 Column |         Type          | Modifiers
--------+-----------------------+-----------
 geom   | geometry(PointZ,4326) | 

So what, you say? So, now the “geometry_columns” table can automatically be kept up to date as a view on the system tables. Once a spatial table is created, it now automatically appears in “geometry_columns”.

 f_table_catalog | f_table_schema | f_table_name | f_geometry_column | coord_dimension | srid  |      type
-----------------+----------------+--------------+-------------------+-----------------+-------+-----------------
 cded            | public         | geotbl       | geom              |               3 |  4326 | POINT

Still not impressed? The classic problem, “how do I reproject the data in my table” used to require multiple steps: drop spatial reference constraint, update the column, re-add the constraint, refresh the geometry_columns table. Now, it’s a one-liner, converting the contraints, global metadata, and table data in one step.

ALTER TABLE geotbl
  ALTER COLUMN geom
  SET DATA TYPE geometry(Point,26910)
  USING ST_Transform(ST_Force_2D(geom),26910)

And as a bonus, we also converted the type from 3D to 2D. And all the metadata is up-to-date automagically.

           Table "public.geotbl"
 Column |         Type          | Modifiers
--------+-----------------------+-----------
 geom   | geometry(Point,26910) | 

The manual management of the “geometry_columns” table has bothered me ever since it was introduced. After only 10 years, I’m pleased to see it finally fixed! This update required the new extended typmod support brought in to PostgreSQL 8.4 and was prototyped on the “geography” type in PostGIS 1.5.

Flattening the Peel

A PostGIS user asked me an interesting question this week, looking for the intersection of these two polygons:

POLYGON((170 50,170 72,-130 72,-130 50,170 50))
POLYGON((-170 68,-170 90,-141 90,-141 68,-170 68))

Now, from the coordinate sizes, you can see these shapes are described in longitude/latitude. So naturally you should consider using the PostGIS “geography” type, particularly since the shapes are quite big.

However, when you plug the shape into SQL and try out the calculation:

SELECT ST_AsText(
  ST_Intersection(
    ST_GeogFromText('POLYGON((170 50,170 72,-130 72,-130 50,170 50))'),
    ST_GeogFromText('POLYGON((-170 68,-170 90,-141 90,-141 68,-170 68))')
  ));

You get a bad result:

ERROR: transform: couldn't project point (-170 90 0): tolerance condition error (-20)

Hmm. Transform? This sounds like reprojection. We’re working in geography though, what’s going on? Check the definition of the ST_Intersects(geography, geography) function.

CREATE OR REPLACE FUNCTION ST_Intersection(geography, geography)
  RETURNS geography
  AS 'SELECT
    geography(
      ST_Transform(
        ST_Intersection(
          ST_Transform(
            geometry($1),
            _ST_BestSRID($1, $2)),
          ST_Transform(
            geometry($2),
            _ST_BestSRID($1, $2))
          ), 4326))'
  LANGUAGE 'SQL' IMMUTABLE STRICT;

Crazy stuff. Geographies are being cast to geometry, transformed into a planar projection (magically selected by _ST_BestSRID) intersected, then transformed back into geographics and cast back to geography.

The problem is that _ST_BestSRID is failing to pick an appropriate projection to do the calculation in. Now, there are some shapes that are just too big to pick any projection for, but in this case the shapes are a good deal smaller than a hemisphere. However, they are inconveniently located:

One of them goes all the way up to the pole, but the other extends moderately far south. The “best SRID” routine evaluates first whether the shapes can fit into a single UTM zone (they can’t), if they are sufficiently far north to use a polar stereographic (only one is), and finally falls back to a world mercator projection. But, one of the shapes goes up to the pole (which mercator places at infinity), so in the end even the mercator projection fails.

What to do? In order to accurately derive a new shape in planar space that reflects the result on the sphere, we need a projection that represents great circles (used to connect points on the sphere) as straight lines (used to connect points on the plane). We need a Gnomic projection.

Fortunately the projection library underneath PostGIS — Proj4supports the gnomic projection. Now, we need a gnomic projection that is parameterized to optimally fit our geography shapes. The approximate center of the two shapes falls at (-160, 70) so we’ll use that as the center of the projection.

Now we insert a record into the PostGIS “spatial_ref_sys” table, giving our new projection an identifier number of 55000:

INSERT INTO spatial_ref_sys (srid, proj4text)
VALUES (55000, '+proj=gnom +lat_0=70 +lon_0=-160');

Now we can run the intersection again, this time in our specially chosen gnomic:

WITH shapes AS
(
SELECT
  ST_GeogFromText('POLYGON((170 50,170 72,-130 72,-130 50,170 50))') AS p1,
  ST_GeogFromText('POLYGON((-170 68,-170 90,-141 90,-141 68,-170 68))') AS p2
)
SELECT ST_AsText(
  geography(ST_Transform(
    ST_Intersection(
      ST_Transform(geometry(p1), 55000),
      ST_Transform(geometry(p2), 55000)
  ),4326)))
FROM shapes;

And this time we get a result:

POLYGON((-169.908 74.035,-141.155 73.424,-141 68,-170 68,-169.908 74.035))

As you can see eyeballing the globe above the resultant shape should have four corners (check), the southern two corners should be the southern two corners of the second input shape (check), and the northern two corners should be derived from the intersection of the northern edge of the first shape and the eastern/western edges of the second (check).

Astute readers of the geometry text will note that the northern edge of the first shape runs from (170 72) to (-130 72): why doesn’t the north edge of the derived shape fall on the 72d parallel? Because the shortest path between those points doesn’t run along the parallel, it’s a great circle arc that stretches to the north along its path.

This example has made it clear to me that the “best SRID” code needs a little upgrade, so that the logic first checks if UTM is acceptable, then switches into finding a best gnomic, and only breaks down and tries mercator if the shape is drastically too large to fit in a hemisphere.

“Best SRID” finding  will always remain a hack, and direct math on the sphere is preferable, but for now we only have a few operations implemented on the sphere (area, length, distance) so we need some hacks to get our work done.

OpenGeo Workshops at FOSS4G North America

Interested in a full day, hands-on workshop on Monday April 9, just before the FOSS4G North America conference? Please fill out this form to let us know what you’d like to see.
If you can’t make it, don’t worry! This is not a commitment to attend the course, we're using this opportunity to gauge interest in our training offerings. Please note that full day workshops cost $450 and half day workshops cost $250.

More information about our workshops can be found here: http://workshops.opengeo.org/

Not at all Interested Extremely Interested

Not at all Interested Extremely Interested

Not at all Interested Extremely Interested


Not at all Interested Extremely Interested


Not at all Interested Extremely Interested


Not at all Interested Extremely Interested


Not at all Interested Extremely Interested


Not at all Interested Extremely Interested


Not at all Interested Extremely Interested





Down with Podal!

Are you anti-podal? So are many geographic edges!

Actually, not many, but it feels like a common first test case people try when they start playing with the geography type. “How far is it between 0,0 and 180,0?”

There’s a big problem with anti-podal edges though: they don’t have a determinate path. That is, to get from (0,0) to (180,0) it doesn’t matter what direction you travel, just start moving. Any other pairing of points generates a single great circle describing the shortest path joining them. So anti-podal points make very bad components of geometry: they don’t define a path, and they can’t bound an area, because only the end points have a determinate location.

Which brings me to my problem. How do I handle geography objects in PostGIS that include anti-podal segments? On the one hand, since they are impossible to do calculations against, I should just disallow them in all cases, and throw an error. On the other hand, people think they have meaning and stick them into functions all the time. There are also a few functions (like ST_Length) where it’s actually possible to calculate a valid answer given an antipodal input (because we know that antipodal points are exactly one half-circumference apart, even if we don’t know what direction someone might travel between them).

What do you think? Is there a best answer? Comments most welcome!

Building on the Chain, Gang.

Programming languages haven’t changed much over the years: the latest languages to take over large swathes of the industry (Java and C#) are unashamed about being cleaned-up versions of C++, which itself was a melding of C with object concepts. What has changed immensely recently is the state of the art in how large programs are built and tested, the “build chain”.

We don’t talk about this much because it isn’t very client-facing, but thanks to the efforts of Justin Deoliveira and Tim Schaub, OpenGeo has a quite robust build environment. Early on in the development on the OpenGeo Suite we found that the number of steps necessary to move from a particular version of the code to an installable and testable artifact was very high—so high that cycles of test/fix/re-test were just too long.

So we automated this chain, and not just the build. Our software is now automatically built out from source code all the way to installers (for Mac OS X and Windows) and packages (for Ubuntu Linux and CentOS Linux) and machine images (for Amazon AWS) every hour. The industry term for what we’re doing is called “continuous integration“.

The complexity of a system that builds multiple components (GeoServer, PostGIS, PostgreSQL, GeoExt, etc.) in multiple languages (C, Java, JavaScript) on multiple operating systems (Linux, OS X, Windows) is quite substantial, but it is all worth it to be able to incorporate a bug fix into a new installer in short order without human intervention. As we have many clients who are depending on our software for their deployments, this reliable turnaround is critical.

Our system has grown so large that we are now devoting a full-time engineer (welcome, Michael Weisman) just to maintaining and improving it. In time, we plan to add even more components and functions into the mix, such as continuous builds of GDAL and continuous unit testing of all components against multiple databases. The benefits in flexibility, quality, and development speed is well worth the investment.

So if you’re looking for us, you’ll find us building on the chain.

It goes up to 2.0

Maybe someday PostGIS will go to 11, but for now, we’re still shooting for 2, point oh. And happily we are getting closer and closer. We have moved to a weekly schedule of alpha releases (this week was alpha3) and have started cleaning down the list of tickets against the 2.0 milestone.

Last month, much of the time spent by me and Sandro Santilli on PostGIS 2.0 preparation was funded by the Humanitarian Information Unit of the US Department of State.  So, from the PostGIS development team, and the PostGIS community in general: thanks, HIU!  Why is HIU funding PostGIS? Because the kinds of tools that HIU and its partners use for humanitarian response are backed by PostGIS, and they want to see those tools get better. Funding PostGIS development is an economical way to simultaneously raise the capabilities of a whole ecosystem of tools in HIU’s space.