Let’s start with the videos we recorded during the sponsor day presentations.

Eddie Pickle kicked things off with the “Annual OpenGeo Update“, highlighting our recent spin-out, and how OpenGeo and FOSS4G have grown together:

Next up we saw Alyssa Wright and Tim Schaub formally launch, and then demo, Mapmeter beta. If you didn’t catch the announcement , Mapmeter is our new tool for analyzing geospatial deployments. Head over to http://mapmeter.com to learn more, connect with the development team and join us for this beta program.

David Winslow followed, giving a presentation on GeoGit, a project exploring the use of distributed management of spatial data. David has made his slides available for download.

Tim Schaub and Andreas Hocever closed out the day with an overview and demonstration of OpenLayers 3.

We’re sorry to say we didn’t catch everything on video, but we are happy to share the slides from all of our other presentations:

• Tim Schaub and Andreas Hocever: Say Hello to OpenLayers 3
• Paul Ramsey: LIDAR in PostgreSQL with PointCloud
• Alyssa Wright: OpenThreads, the community of open source mailing lists
• Juan Marin: GeoServer In Production
• David Winsow: GeoServer CSS Module
• Tim Schaub and Justin Deoliveira: Scripting GeoServer with GeoScript
• Paul Ramsey: PostGIS Feature Frenzy
• Justin Deoliveira: The State of GeoServer 2013

We had a great week in Minneapolis and are happy to see the North America conference continue to grow. Without fail, the FOSS4G community puts together the most interesting and informative conference of the year. FOSS4G-NA 2013 certainly lived up to our expecations. If you had a conversation with someone at our booth, or at a presentation, and would like to follow up please feel free to contact us. See you in Nottingham!

Mapmeter enables organizations to monitor the health of production deployments, optimize applications during development and diagnose critical issues. With these details, administrators and managers can better — and more cost effectively — make decisions about their geospatial deployments. With Mapmeter, spatial monitoring and reporting become a primary component in your spatial IT workflow.

You may have heard us talk about this product in recent months. We started with an announcement at FedGeo, then Alyssa Wright showed a live demo with James Fee, and some of you were able to get in on a private beta. With the help and feedback of our early testers, we are now able to open up the software to all.

If you want to learn more or see a demo, please join us for the Mapmeter launch at Sponsor Day at FOSS4G-North America at 9 a.m. Friday, May 24 (look for the OpenGeo room). Sponsor Day is free, but you need to register. We’ll also be holding office hours at our FOSS4G North America booth today (5/22/2013) from 2:00pm to 3:00pm.

If you won’t be able to join us in Minneapolis, head over to http://mapmeter.com to learn more about these exciting new features, connect with the development team and join us for this beta program.

We hope you’re as excited as we are about Mapmeter! If you have questions, please contact us. You can also contact the Mapmeter team directly for personalized help with the public beta.

Here’s the full list, please apply and/or spread the word!

Vanedge Capital, a Vancouver-based venture capital firm, led the Series A round of investment that made this possible. We are truly excited to begin a partnership with Vanedge, an innovative fund led by partners who know how to grow and manage software technology companies.

This investment provides the capital we need to meet our objectives and continue to develop innovative technologies. If you’re a regular reader of this blog or have seen us lately at conferences or events, you know about the ambitious projects we’ve been working on: through-the-web-processing, breaking out of the GIS work-flow with Spatial IT, geospatial web-analytics and distributed versioning for geospatial – to name a few. This type of development requires not just the strong technical skills and forward-looking leadership that our team has, but it also requires resources, which Vanedge’s investment provides.

This investment also allows us to achieve our long-planned separation from OpenPlans, which founded and incubated us. We are grateful for the support and vision of OpenPlans over the years. And, since OpenPlans remains an investor in our new company, we’re looking forward to our continued partnership with them.

Our mission remains the same: to build the highest quality software for location and mapping, available to all. This investment gives us a stronger base of resources to support the open source communities we work with. We remain committed to the open source principles of collaboration, transparency, and freedom. We’ll be doing even more to develop the best geospatial tools while supporting the open source communities and our customers alike.

Look for more from us about the future of Spatial IT and how we can help you get there.

Find out more about this important step forward and please contact us if you have any questions.

Clustering Scenario

In this following scenario, we will work through the installation and configuration of two GeoServers each inside their own servlet container instances on the same machine. Each servlet container will use the same JRE and the same container binaries (Apache Tomcat 7), but they will have independent configurations that allow them to run on different ports. These two GeoServer/Tomcat instances will be fronted by a local software proxy called HAProxy which acts as a HTTP/TCP load balancer. Load balancer configurations provide very basic “round robin” balancing of GeoServers. More sophisticated load-balancing configurations are possible, but are beyond the scope of this example. All GeoServers will be deployed as WAR files placed into each of the Tomcat webapps directories. It is possible to have multiple instances of Tomcat share a single web-application through the use of contexts. This is useful if you anticipate your web-application (GeoServer) will be changed/updated frequently, but isn’t necessary.

Implementation

The following steps will walk through the installation and configuration of a basic cluster containing two GeoServers in separate Tomcat servlet containers, behind an HAProxy load-balancer/proxy on the same machine (high-performance). The steps are:

1. Download and unpack Tomcat binaries
2. Create individual Tomcat instances
3. Start Tomcat instances and deploy GeoServer applications
4. Install and configure load balancer / proxy server

Following this walk-through, we will discuss other options for and extensions of this configuration such as high-availability, alternate proxy tools/configurations, database-backed catalogs, and triggered configuration reloads.

Download and unpack Tomcat binaries

1. Start by downloading a binary distribution of Tomcat to your machine. This example uses the latest version (7.0.39 at the time of writing) as a .tar.gz file from http://tomcat.apache.org/download-70.cgi.
2. Unpack this archive to a suitable location: In this example the entire contents of the archive are extracted to /var/tomcat. By convention, we’ll refer to this directory as $CATALINA_HOME.

Create individual Tomcat instances

Next we’ll make two directories for two separate instances of Tomcat to run from. Both instances will use the same Tomcat binaries from the directory created above; the directories created here will just hold the logs and configuration for each of our instances. In this example we’ll make two instance directories, /var/tomcat1 and /var/tomcat2. These will be referred to as $CATALINA_BASE directories. Each of these directories needs a basic structure and some initial content to host a Tomcat instance. Run the following commands (or a variation thereof) to create the basic directory structure.

# mkdir /var/tomcat1/conf
# mkdir /var/tomcat1/logs
# mkdir /var/tomcat1/temp
# mkdir /var/tomcat1/webapps
# mkdir /var/tomcat1/work
# mkdir /var/tomcat2/conf
# mkdir /var/tomcat2/logs
# mkdir /var/tomcat2/temp
# mkdir /var/tomcat2/webapps
# mkdir /var/tomcat2/work

Each Tomcat instance needs two configuration files to start with that define how the service will run (name, host(s), and ports). Copy the files server.xml and web.xml from the $CATALINA_HOME/conf directory into each of the $CATALINA_BASE/conf directories. We’ll need to edit each of the server.xml files so that each Tomcat instance runs and shuts down on different ports. In each file, look for lines like:

<Server port="8005" shutdown="SHUTDOWN">

<Connector port="8080" protocol = "HTTP/1.1"
 connectionTimeout="20000"
 redirectPort="8443" />

Change these port values. The values can typically be any unused port above 1024. Both the SHUTDOWN and the HTTP Connector ports must be different, and the values for these ports in tomcat1/conf/server.xml must be different than those in tomcat2/conf/server.xml. For example:

• On tomcat1: SHUTDOWN on port 8005, serve HTTP requests on 8085
• On tomcat2: SHUTDOWN on port 8006, serve HTTP requests on 8086

Start Tomcat instances and deploy GeoServer applications

Once configured, we’re ready to start up the servlet containers. We can do this by running a series of commands from the terminal, however it might be more pragmatic to write a small script to accomplish this since these steps often need to be repeated. Create two scripts: /var/tomcat1/tomcat1.sh and /var/tomcat2/tomcat2.sh. These scripts will be identical except for the value of the $CATALINA_BASE variable.

export CATALINA_HOME=/var/apache
export CATALINA_BASE=/var/apache1
export CATALINA_TMPDIR=$CATALINA_BASE/temp
export JRE_HOME=/usr/lib/jvm/java-6-openjre/jre
export CLASSPATH=/var/tomcat/bin/bootstrap.jar;/var/tomcat/bin/tomcat-juli.jar

$CATALINA_HOME/bin/catalina.sh start

These scripts will perform the following functions:

• Define the location of the Tomcat binaries in $CATALINA_HOME
• Set the location for the current container in $CATALINA_BASE and $CATALINA_TMPDIR
• Define the location of the JRE for Tomcat to use (we’re assuming Java is installed)
• Define the CLASSPATH to the core Tomcat JARs
• Export each environment variable so they are available to the Tomcat start-up calls
• Run the Catalina control script to start the Tomcat service

With the files created, run both /var/tomcat1/tomcat1.sh and /var/tomcat2/tomcat2.sh to configure and start the two services. Note that you’ll need to make the files executable. To confirm that the services started correctly, you can tail the catalina.out files in /var/tomcat1/logs and /var/tomcat2/logs. Next, copy the geoserver.war file (or unpacked application directory) into each of the /var/tomcat1/webapps and /var/tomcat2/webapps directories. The applications should automatically deploy. Confirm that you have two GeoServers running independently by browsing to each one on their respective port.

Two GeoServers

Install and configure load balancer / proxy server

Install a web server that is capable of acting as a load balancer in front of the cluster. In this example we use HAProxy installed on Ubuntu Linux using standard package management. That said, rhere are many other tools you can use as a front-end load balancer / proxy server other than HAProxy. One example would be Apache HTTP with mod_proxy_http and mod_proxy_balancer. Microsoft IIS can also sit in front of Tomcat instances using the Network Load Balancer (NLB) or Application Request Routing (ARR). Configure haproxy to act as a load balancer using the following sample in /etc/haproxy/haproxy.cfg. Stop haproxy, copy / edit the config file in place, and then restart.

global
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log global
    mode http
    option httplog
    option dontlogpull
    retries 3
    option redispatch
    maxconn 2000
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000
    log 127.0.0.1 local0
    log 127.0.0.1 local7 debug

frontend http-in
    bind *:80
    default_backend geoserver

backend GeoServer
    balance roundrobin
    server GeoServer1 localhost:8085 maxconn 32 check
    server GeoServer2 localhost:8086 maxconn 32 check
    ##option httpchk
    ##option forwardfor

listen admin
    bind *:8080
    stats enable

(Note that these are very basic configurations. Your system administrator will have a better idea of what is the norm for your organization.) There are a few ways you can confirm that HAProxy is working and balancing the clustered back-ends as anticipated. 1) Browse to the HAProxy admin page at http://<server>:8080/haproxy?stats:

HAProxy stats

In the figure above we see that HAProxy recognizes the GeoServer backend with two members GeoServer1 and GeoServer2. 2) You might also make a noticeable configuration to one or GeoSevers and observe that the single proxy URL is requesting data from all members of the backend. For example, a change to a single SLD file, and then requesting a layer that makes use of that style through HAProxy confirms that both of our GeoServers are handling requests.

Conflicting styles from two different GeoServers through HAProxy

A Common Data Directory

Normally, GeoServer data directories in a cluster will be identical in content. In this example we will set a common data directory for all cluster members using a context parameter in the web.xml file of each GeoServer in our cluster. You can specify the GeoServer data directory location several other ways. Make a new directory for your shared GeoServer Data Directory.

# mkdir /geoserver_data

Copy a template data directory into the new location. This step is optional. As long as the base data directory exists, GeoServer will create the basic configurations it needs if they’re not found on start-up. It’s just a bit more painless for this example if they are where we expect them.

# cp -r /var/tomcat1/webapps/geoserver/data* /geoserver_data

Stop the two GeoServer / Tomcat instances so we can reconfigure our data directory locations, either by killing the identified pids for the java processes that our GeoServers are running under, or (in a more sophisticated installation) using a service. Update the web.xml file in each web application WEB-INF directory. This file typically lives at $CATALINA_HOME/webapps/geoserver/WEB-INF/web.xml. Make four changes to each file:

• Specify the new location of the GEOSERVER_DATA_DIR
• Specify a GEOSERVER_LOG_LOCATION for this particular instance to log to. This will avoid collisions with the other GeoServer nodes in the cluster writing to the same location. For example, set to /geoserver_data/logs/geoserver_tomcat1.log
• Set GWC_DISKQUOTA_DISABLED to true. This will avoid collisions with the other GeoServer nodes’ GWCs writing disk use information to common locations.
• Set GWC_METASTORE_DISABLED to true. This will avoid collisions with the other GeoServer nodes’ GWCs writing cache status information to common locations.

Extending These Examples

This document is intended as just an example of setting up GeoServer in a clustered configuration, designed to move users towards a more scalable GeoServer installation, but might not be suitable for production in all environments. Future versions and alternate scenarios will take into consideration:

• Scaling GeoServer horizontally (on multiple machines)
• Hybrids of vertically and horizontally scaled instances
• Better ways to manage and configure multiple servlet containers (Tomcat) and web applications (GeoServer)

What sort of GeoServer clustering environments are you interested in setting up? Let us know in the comments below.

One thing I love about open source development is the ‘alpha’ release.

Last week was an exciting week of alphas for OpenGeo, both OpenLayers 3.0 and GeoGit had their first releases and launched new websites. The two websites are admittedly not very sophisticated—I made the geogit.org with GitHub’s page generator and Andreas pulled together ol3js.org with Bootstrap—but awesome websites can come later. The point of these alpha releases is to get something out in the world and widen the open source process to new users and potential contributors.

Alpha releases are rarely seen in proprietary software development since software in an alpha state is generally quite buggy. To quote Wikipedia: ”alpha software can be unstable and could cause crashes or data loss.” At this point many would turn away and run as far from the software as possible but to me it’s an awesome thing, an understood pact between the developers and the users that says: “hey, we’re not perfect, and we know our software is far from perfect, but if you understand the risks we’d be really excited to show it to you.”

The process opens up a dialog of equals—not the typical consumer relationship, but a collaborative one. The user of alpha software actually has a responsibility to communicate when (not if) things go wrong and to tell the developers how it crashes, what important option isn’t there, how the installation fails, or even how the website is confusing. In this way, responsibility can grow from being an alpha user to include helping with documentation, improving the website, debugging problems, contributing patches, and eventually building major new features as a core developer. Indeed the point of the alpha release is to put a stake in the ground and open the process to gain feedback from others, allowing users and developers to build the future together. Everyone is expected to be a true participant, in the fullest sense of the word, with responsibilities as well as privileges as opposed to just a passive consumer.

We encourage you to check out both the OpenLayers 3.0 and GeoGit alpha releases and let the teams know what you think. OpenLayers in particular has a very solid core but is looking for practical input from real users. We think the projects show a lot of potential, and we’re excited for your feedback, encouragement, and even contributions. Don’t hesitate to jump in and join us as we build the geospatial future together.

We’ll start with some clustering theory and tips before launching into the actual details of how to do it.

Background

A computing cluster consists of two or more machines working together to provide a higher level of availability, reliability, and scalability than can be obtained from a single node. Nodes in a cluster are positioned behind a proxy server and/or load balancer that delegates requests to cluster members based on any one member’s ability/availability to handle load.

Clustering

Similar to other applications with long-running in-memory states and high data I/O, GeoServer sees performance gains with two (or more) nodes clustered behind a load balancer—even with the slight overhead of the load balancer that sits in front of the cluster.

Generally, there are two complementary purposes for clustering GeoServer:

• To provide high-performance and/or throughput
• To achieve high availability

In the most demanding situations, GeoServer can be deployed in combinations of high-performance and high-availability instances.

High-Performance Clusters

A high-performance GeoServer configuration deploys several instances of GeoServer on a single machine.

High-performance cluster

Each GeoServer instance is deployed into its own servlet container (Tomcat, Jetty, etc.). Individual servlet containers are configured independently and spin up their own JVM, each with it’s own memory and processor allocations (borrowed from the pool of resources on the host machine). GeoServer’s memory and CPU runtime footprint are optimized for high throughput under heavy concurrency with such a deployment, but always consider that these different deployed units will compete for the physical server’s resources. To find the best balance we recommend, as always, to test for your particular scenario.

A load balancer or proxy fronts the cluster, and directs traffic to the member of the cluster most able to handle the current request. In this case, nodes will likely share the same server name or IP address, but listen for requests on different ports. For example:

Load Balancer @ http://<server>:80/<alias> forwards to one of:

• GeoServer 1 in Tomcat 1 @ http://<server>/geoserver:8081
• GeoServer 2 in Tomcat 2 @ http://<server>/geoserver:8082
• GeoServer 3 in Tomcat 3 @ http://<server>/geoserver:8083
• GeoServer 4 in Tomcat 4 @ http://<server>/geoserver:8084

An approach that deploys multiple instances of GeoServer into the same servlet container is not recommended. In this case, since host resource allocation (to a common JVM) will not be sequestered as neatly, competition for those resources will occur, limiting the benefits.

Users might also consider using the built-in clustering capabilities found in Enterprise Application Servers (such as Oracle Weblogic or JBoss), however this is beyond the scope of this discussion.

High-Availability Clusters

A high-availability implementation will spread several GeoServer instances across several machines (nodes) in a cluster. These nodes can be physical or virtual machines.

High-availability cluster

Nodes are normally located behind a load balancer that redirects traffic to any single GeoServer based on traffic volume and availability. In this case, nodes will likely be on different servers or IP addresses and listen for requests on the same port. For example:

Load Balancer @ http://<server>:80/<alias> forwards to one of:

• GeoServer 1 in Tomcat 1 @ http://<server1>/geoserver:8080
• GeoServer 1 in Tomcat 1 @ http://<server2>/geoserver:8080
• GeoServer 1 in Tomcat 1 @ http://<server3>/geoserver:8080

Data directory location and catalog reloads

Some important considerations to be made when clustering several instances of GeoServer concern the location of the GeoServer data directory and a strategy for reloading all cluster members’ data catalogs.

The GeoServer data directory is the location in the file system where GeoServer stores its configuration information. The configuration defines things such as what data is served by GeoServer, where it is stored, and how services such as WFS and WMS interact with and serve the data. The data directory also contains a number of support files used by GeoServer for various purposes.

The spatial data accessed by GeoServer doesn’t need to reside within the GeoServer data directory, just pointers to the data locations. This should be obvious for data stored in spatial databases, which are certainly in different locations (on disk) and often on different machines; however the same is true for file-based spatial data. (Read more about the GeoServer data directory.)

GeoServer’s catalog is an in-memory representation of the configurations in the data directory. Storing the configurations in memory means that GeoServer can access this information faster than by reading these instructions off disk. However, this sometimes requires that the in-memory catalog be refreshed when configurations changes are made to the disk-based GeoServer data directory, or to the actual data served in GeoServer.

Unless catalog (re)configurations are largely static, or some amount of catalog discrepancy or availability is acceptable, a common GeoServer data directory location for all clustered instances is highly recommended.

The location of the GeoServer data directory is stored in the GEOSERVER_DATA_DIR variable. It can be configured in one of three ways: in each instance’s web.xml file (/webapps/geoserver/WEB-INF), through a common environment variable, or through a parameter passed to the JVM in the container start-up command.

Some implementations have clustered GeoServer instances using separate data directories that are synchronized manually (low change frequency) and automatically (using rsync), but neither approach is as common or recommended as a shared data directory.

Regardless of the mechanism for synchronization, changes to the data directory and the in-memory catalog will normally be directed by one master GeoServer. This can be enforced by disabling the GeoServer user interface on all “slave” GeoServers or by configuring the front-end load balancer to only direct user interface requests to /geoserver/web to the master GeoServer.

Changes to the master GeoServer’s data catalog must be explicitly refreshed on slave instances. This can be accomplished manually through the GeoServer Admin web UI (/geoserver/web), or with some measure of automation (on a schedule, or after a trigger is fired) using GeoServer’s REST API (e.g. by sending a POST/PUT request to /geoserver/rest/reload?recurse=true).

Clustering Enhancements

Enhancements to our clustering story are coming! Specifically, in future releases of GeoServer the data directory will have the option to be database-backed. This means that a central configuration store can be queried more optimally than a file-based counterpart and doesn’t all need to be read into memory.

In the next post, we’ll go into the details on setting up a clustered instance. Remember, Enterprise: Platform clients and higher get custom clustering and deployment advice included in their maintenance agreements.

Have you been looking at deploying GeoServer in a clustered environment? Tell us about it!

Open source development of characterized by some features that differ a great deal from traditional work environments:

• the developers work asynchronously, often in different time zones, usually in different locations,
• the developers coordinate exclusively using text tools, like e-mail, issue tracking systems, and sometimes instant messaging

Because there is no need to be in the same space with other developers, either physically or even temporally, the barriers to entry to a project are lowered. More people can participate than otherwise.

However, there are disadvantages to working asynchronously and with text communications.

• asking for help when you get stuck can be time consuming, because your colleagues might be asleep at the moment when help would be most useful
• issues of subtlety or complexity take a great deal of text to describe, and any misunderstandings on the part of a reader take even more text to correct
• discussion of emotional issues can lead to conflict due to the limited emotional nuance in text communication

A code sprint is a chance to work for a time with your open source colleagues “the old fashioned way”, face to face, on the same clock.

Because everyone is together, and communications are high-bandwidth and high-fidelity, a code sprint is a great time for:

• planning and designing large scale changes to the code
• designing new APIs or new user interfaces, and
• triaging ticket lists to prepare for release

I usually spend the first half of a sprint on communication-heavy tasks like the ones above. The second half I usually spend heads down on a hard piece of code.

If the right experts are around, code sprints are an excellent time to attack a new piece of code you don’t quite understand. Learning how a module works from the expert who wrote it is far faster than doing it alone at home.

And finally, having lunch and dinner and socializing usually provide the social space for unexpected topics to slip out and get a discussion, whether they be uncomfortable issues like dealing with a difficult team member or just a crazy feature idea that turns out to be not so crazy at all when discussed with the group.

If you have a chance to participate in a code sprint on a project you contribute to, don’t pass it up!

Last year we had a pretty big hand in organizing the inaugural DC conference, the event went so well that it’s become an annual event. This year we’ve stepped back from the day-to-day planning but we’re still helping out on the program committee, and we’re happy to support the conference as gold sponsors.

We’ll be sending a pretty large contingent to Minneapolis; nine OpenGeo presentations were accepted and we’ll be teaching four workshops. We’re looking forward to an exciting (and busy!) week and hope to see you there. Don’t forget to register to attend; we hear spots are filling up quickly. And if you’ll be in Minneapolis make sure to come by our exhibition table, you never know who you’ll run into.

Interested in hearing us speak? Want to enroll in a workshop? The preliminary program has been announced and the times and dates of specific talks will be posted soon. Scroll down to see list of what we’ll be up to at FOSS4G-NA:

OpenGeo FOSS4G-NA Presentations (Marriott City Center in Minneapolis, MN)

• GeoServer CSS: David Winslow
• Say Hello to OpenLayers 3: Tim Schaub & Eric Lemoine of Camptocamp
• OpenLayers 3: Vectors Redux: Tim Schaub & Andreas Hocevar
• Scripting GeoServer with GeoScript: Tim Schaub
• LIDAR in PostgreSQL with PointCloud: Paul Ramsey
• GeoServer in Production: Juan Marin
• State of GeoServer: Justin Deoliveira
• PostGIS Feature Frenzy: Paul Ramsey
• Diversity in FOSS4G Mailing List: An Analysis: Alyssa Wright & Georgia Bullen of the Open Technology Institute

OpenGeo FOSS4G-NA Workshops (University of Minnesota in Minneapolis, MN)

• Introduction to PostGIS, Paul Ramsey
• GeoNode, the Open source Geospatial CMS, Ian Schneider
• OpenLayers Application Development, Tim Schaub
• Introduction to GeoServer, Sam Smith

The rumors are true: GeoScript is pretty awesome. How awesome? We’ll let you be the judge. In this post we’ll focus on data exploration and show you how to create a few visualizations right from the GeoScript command line, but GeoScript can do much more than that. In subsequent posts we’ll build up enough code and data to create other GeoScript products, then we’ll show how to easily refactor this code into processing web services. If you’d like to follow along with this post please install GeoScript. For the purposes of these exercises make sure that you download the latest version. For our examples we’ll be using the Python flavor of GeoScript. Make sure to keep the API doc close at hand and don’t hesitate to experiment. If you get stuck, please contact us and we’ll try to help you out.

Getting the Source Data. We’ll be using solar resource data from the National Renewable Energy Lab. The data set contains direct normal irradiance (DNI) average values for the contiguous United States and Hawaii. If you want to find out more details should be contained within the metadata. You can download the entire data set to work through the examples below. Unzip to a directory, navigate to it, and fire up the GeoScript shell. (If you see some diagnostic messages along the way, just ignore them and remember that GeoScript is a work in progress.) Let’s get started!

Loading and Exploring Data. First, let’s load our two shapefiles into GeoServer:

>>> from geoscript.layer import Shapefile
>>> solar_all_poly = Shapefile("solar_dni_polygons.shp")
>>> states = Shapefile("usa_l48.shp")

What’s the structure (schema) of the ‘solar’ dataset and how many features does it contain?

>>> print(solar_all_poly.schema)
solar_dni_polygons [the_geom: MultiPolygon, ID: long, GRIDCODE: float, ANN_DNI: float, JAN: float, FEB: float, MAR: float, APR: float, MAY: float, JUN: float, JUL: float, AUG: float, SEP: float, OCT: float, NOV: float, DEC: float]
>>> print(solar_all_poly.count())
91439

If you want, you could print the entire ‘states’ dataset, but watch out though, as the result will be a pretty big data dump.

>>> print list(states.features())
[usa_l48.1 {the_geom: MULTIPOLYGON (((-77.05354916699991 38.8816935760002, -77.05099573299992 38.87975426100007, -77.04814371899994 38.87760644600007, -77.04687615699987 38.876303672999995, -77.04721013299991 38.87585838300009, ... ... ...

How about printing only the values of the STATE attribute? (You'll have to hit Enter twice after this command.)

>>> for f in states.features(): print f["STATE"]
... 
District of Columbia
U.S. Virgin Islands
Indiana
Pennsylvania
Rhode Island
Illinois
...

Now, let’s draw the ‘states’ layer. Nothing fancy for now, but let’s ensure correct image proportions by applying the ‘state’ data’s inherent aspect ratio:

>>> from geoscript.render import draw
>>> aspect_ratio = states.bounds().aspect
>>> draw(states, size=(int(200 * aspect_ratio),200))

States layer

A Bit of Processing: Clipping. In the next few steps, we will clip our ‘solar’ layer to a single state. First, let’s decide which state we are interested in:

>>> state = "Florida"

Now, we will create a workspace to hold intermediate processing results in memory, otherwise they will be saved to disk:

>>> from geoscript.workspace import Memory
>>> memory_workspace = Memory() 
>>> state_geom_temp = memory_workspace.add(states)

And then apply a filter to select the state we want and extract its geometry:

>>> state_geom = state_geom_temp.filter("STATE='%s'"%state).features().next().geom

We’re set to run the Clip process. Before we do though, let’s take a look at all Processes available right here in the GeoScript shell. (If you want to jump ahead a little, check out how the same Clip process works as a Web Processing Service.)

>>> from geoscript.process import Process
>>> print list(Process.list())
[(u'geo', u'length'), (u'geo', u'isEmpty'), (u'geo', u'contains'), (u'geo', u'disjoint'), (u'geo', u'intersects'), (u'geo', u'isClosed'), (u'geo', u'isValid'), (u'geo', u'buffer'), (u'geo', u'getY'), (u'geo', u'getX'), ... ... ...

>>> process_clip = Process.lookup("vec:Clip")

Which parameters does the Clip process expect and what does it return?

>>> print "Clip inputs: " + str(process_clip.inputs) 
Clip inputs: {u'clip': (u'clip', <type 'com.vividsolutions.jts.geom.Geometry'>, Geometry to use for clipping (in same CRS as input features)), u'features': (u'features', <class 'geoscript.layer.layer.Layer'>, Input feature collection)}
>>> print "Clip outputs: " + str(process_clip.outputs)
Clip outputs: {u'result': (u'result', <class 'geoscript.layer.layer.Layer'>, Clipped feature collection)}

Now that we know what to feed into the Clip process and what to expect in return, let's execute it. Please be patient, this process may take a bit longer than the previous commands.

>>> solar_state = memory_workspace.add(process_clip.run(clip=state_geom, features=solar_all_poly).get("result"))

Let's check how many features were returned and draw the clipped output:

>>> print solar_state.count()
2294
>>> draw(solar_state)

Solar resource data clipped to the bounds of Florida

Creating a Histogram. Let's say we are interested in the distribution of the total solar energy over the course of a year. A histogram is a simple way to visualize this distribution. We can build out histogram for the entire extent of our solar resource data, or for the single state we clipped out in the previous step. Let's stick with the single state for now.

>>> aoi = solar_state

>>> from geoscript.plot import bar

# create a list of attributes containing months
>>> months = ['JAN', 'FEB', 'MAR', 'APR', 'MAY', 'JUN', 'JUL', 'AUG', 'SEP', 'OCT', 'NOV', 'DEC']

# compute solar power potential totals for each month
>>> month_values_per_feature = [[f.get(mo) for mo in months] for f in aoi.features()]
>>> mo_totals = map(sum, zip(*month_values_per_feature))

# assign an index to each month starting at 1
>>> mo_numbers = range(1,len(months)+1)

# build data for the histogram, print it and draw
>>> histo_xy = zip(mo_numbers, mo_totals)
>>> print histo_xy
[(1, 3016.6060161298988), (2, 3502.787616071399), (3, 3869.839870968297), (4, 3800.790388888799), (5, 3838.1435860215997), (6, 3984.095752778796), (7, 3930.639688172497), (8, 3797.606723118299), (9, 3542.7847666671996), ... ... ... 
>>> bar.xy(histo_xy).show()

Distribution of the total solar energy resource over the course of an (average) year in Florida

Creating an Animation. We'll now create an animated GIF to show how average intensity of solar resource potential varies month-by-month, over a year.

>>> from geoscript.style import * 
>>> from geoscript.render import GIF
>>> from geoscript.util import interpolate

# compute aspect ratio
>>> bbox_aspect = aoi.bounds().aspect

# compute classes based on global minmax across all months
>>> classes = 100
>>> minmax = zip(*[aoi.minmax(m) for m in months])

# interpolate using exponential function
>>> vals = interpolate(min(minmax[0]), max(minmax[1]), classes, "exp")
>>> print vals
[3.4250456989, 3.44100384308852, 3.457216649642982, 3.4736881825021397, 3.4904225704576706, 3.507424008189102, 3.5246967573152643, 3.5422451474625136, 3.560073577350009 ... ... ...

# initiate empty list for image frames and set color gradient bounds for our symbols
>>> frames = []
>>> gradient_color_min = 'yellow'
>>> gradient_color_max = 'red'

>>> for m in months:

        # generate buckets for fills and strokes
...     fills = Fill(gradient_color_min).interpolate(Fill(gradient_color_max), classes)
...     strokes = Stroke(gradient_color_min).interpolate(Stroke(gradient_color_max), classes)

        # build styles
...     styles = map(lambda x: x[0] + x[1], zip(strokes, fills))
...     x = zip(zip(vals[:-1], vals[1:]), styles)
...     parts = map(lambda t: t[1].where( m + ' BETWEEN %f and %f' % (t[0][0], t[0][1])), x) 
...     style = reduce(lambda x,y: x+y, parts)

        # draw a map, apply the style, correct aspect ratio and save it as a frame
...     frames.append(draw(aoi, style, size=(int(400*bbox_aspect), 400), format='gif'))
...

# finally, generate the animation from the frames; 
# the image will be placed in the directory with your shapefiles
>>> GIF.animated(frames, 'solar.gif', delay=50, loop=True)

Solar resource animation for Florida

Now that we've tested our logic on data for a single state, we can build an animation for the entire data set. To do this, simply reassign the aoi variable:

>>> aoi = solar_all_poly

and then re-run the animation generation steps. You should end up with something that looks like this:

Solar resource animation for the contiguous USA

Stay tuned! Next time, we'll five deeper into visualization and develop some processing services around this code and data.

Have you used GeoScript for a project? If so, please tell us about it. If not, what are you waiting for? Go give it a try! My special thanks for Justin Deoliveira and Mike Pumphrey for handholding and sanity checks respectively.