We’ve been doing a lot of data migration and new data uploads with Finder! and often times our data team runs into data and mapping headaches. One that we commonly encounter are largish shapefiles that make for really bloated KML when we convert it (for instance a 2mb shapefile for US counties becomes a 5.4 mb KML file). The end result are big files that completely kill browser based applications like Virtual Earth and Google Maps, or load really slowly in thick client applications like Google Earth and ESRI AGX.

There are three factors that constitute file bloat for any vector based geospatial data:

1) The number of attributes (how many columns)
2) The number of features (how many rows)
3) The complexity of the geometry (how much needs to be drawn)

You can do some clever things to manage the first two at a low level - although you still are going to have bloat when you convert to a standard file format. The third factor, geometry complexity, is interesting because you can also do some low level tricks whose savings can be passed along to standard file formats. Reducing the complexity of geometry is often called “map generalization” in academic circles.

The general concept is that you remove details from the map without loosing the message and context of the map. All maps have some form of generalization otherwise it would be a perfect reflection of reality. Academics have used algorithms to heuristically derive a map generalization. This is probably best explained with a few examples. Below is a map of Europe in full detail:

Next is map generalization that removes some of the detail but still keeps the context of Europe and the country boundaries:

Last a more extreme example with even greater detail removed:

To pull off these nifty computational tricks used to require some fairly sophisticated desktop software, but Matt Bloch and Mark Harrower at the University of Wisconsin figured out a clever way to enable enable real-time WYSIWIG map generalization. The resulting application is called MapShaper. You can upload a shapefile and run different generalization routines (with high level of control if you choose) then export the result back out as a shapefile or an EPS file. The shapefile export is down at the moment, but hopefully will back in action soon.

I think these kinds of technologies and mathematics are going to be increasingly important as we need to make ever larger datasets available. Especially when the receiving devices are increasingly mobile with even smaller data handling capabilities.

Popularity: 30% [?]

The Utility of Maps in Hazard Forecasting

The recent wildfires in Southern California remind of us of just how important hazard forecasting has become in helping to ensure the safety and welfare of the public and the role that mapping can play in the process. Short-term forecasts of fire direction and intensity were pivotal in containment and evacuation efforts; Mapping played a prominent role in generating forecasts and in disseminating and sharing information about potential risk.

The usefulness of maps in visualizing and and generating forecasts extends well beyond the California fire event. In the area of climate prediction, numerous sites provide regularly updated maps of long-term and short-term forecasts of a variety of conditions and in some cases, valuable watches and warnings to the public based on the forecasts.

Some Points for Discussion

While the information that is currently out there provides great utility, there are some limitations in the way that the information is is disseminated and formatted that are worth noting. The points are intended to be food for thought and to get us thinking about how we can increase the value of forecasting even further - particularly in an interactive, web-based mapping environment.

First, forecasts are scattered across multiple websites and even within websites, requiring some effort and time on the part of the consumer to find, extract and process information. The sites and links vary in terms of the information they provide. In terms of fire forecasting, some sites focus on drought conditions, others on smoke generation and yet others on combinations of factors to characteristic future fire potential. The forecasting horizons also vary considerably from site to site.

Second, much of the maps provided on the web are in a “hard copy” format and not in an interactive mode where the user can pan, zoom and perform other functions. Some sites do have map viewers however, they are currently limited in the amount and type of data that can be displayed.

Third, and related to the second point, is that the possibility for “layering” data to create custom maps with richer information relevant to the needs of the user is limited. For example, someone may be interested in seeing if an environmentally sensitive or protected area is in the path of a projected wildfire.

Fourth, there lacks a mechanism for consumers and providers of the forecasts to interact and share information. Interaction could be very useful in understanding forecasts but also in terms of improving current predictive models. In the book Making Climate Forecasts Better, Stern and Easterling write: “The utility of forecasts can be increased by systematic efforts to bring scientific output and users’ needs closer together. These efforts may include both analytic efforts to identify the climatic parameters to which particular sectors or groups are highly sensitive or vulnerable and social processes that foster continual interaction between the producers and the consumers of forecasts.”

Fifth, not all information is publicly available and perhaps it should be? In climate forecasting, having access to the “best” information is in the national interest: it can save lives. And in some cases, the private sector is the keeper of such information. A recent study by ForecastWatch, found that in terms of recent historical forecasting of next day rain and snow, government sites had a 21% greater error rate than some of the private companies that do similar projections.

What Could the Future Hold?

The new web is fertile for the development of a system by which forecasts can be provided to the public in a more usable, digestible and efficient manner. Sites like Geocommons could be a one-stop location for viewing forecasts, such as those related to hazards and climatic conditions. In such an environment, visitors could interact with each other or the producers of the forecasts, discuss the validity of the forecasts or provide additional information to augment the projections, all through a wiki or blog-style environment. They could also create custom forecast maps with overlays of additional information that is of most useful to them for solving a problem, understanding a situation or simply planning ahead.

Popularity: 21% [?]

There were two great articles that popped up in the last week or so that I’ve been trying to carve out some time to blog about. The first was an insightful overview of the GeoWeb from The Economist called “The World on Your Desktop“. One of the major points in The Economist article is the convergence of the GeoWeb and GIS, “when the analytical insights and data quality of GIS are combined with the geoweb’s visualisation (ease of use) and networking prowess, startling efficiencies emerge.” I must admit I am biased since the article mentions us as one of the companies blurring the line between GIS and the GeoWeb, although sometimes to GIS professional’s chagrin. The second article was a blog post by Nick Black of OpenStreetMaps. Nick does a great job of better defining neogeography. The conclusion being neogeography is about pragmatic solutions to geographic problems that cut through the tendency of traditional GIS to build “complexity to ensure exclusivity”. My take away from the two articles and several recent conversations; the market is moving towards convergence faster than expected and the democratization of GIS will be here sooner than we think.

In many ways the spreadsheet analogy works well. I use about 20% of the functionality in my spreadsheet program and that 20% of functionality is accessible to me with little or no training. That 20% of functionality is what about 80% of spreadsheet users utilize on a regular basis. This in turns allows a very large population of users to create data in a spreadsheet format and share it with other people. I do not need a four year degree in statistics to use the spreadsheet, although if I did there would be a larger percentage of functionality I could tap into and create more results to share.

In the world of traditional GIS there is no 20% that is open to 80% of users. Either you are a GIS wonk or you are not a GIS wonk, and the number of GIS wonks is quite low because of the training and barriers of entry to becoming one. The end result is a complex technology that ensures an exclusive user base. Why can’t or why is there not a GIS technology that is ubiquitous as spreadsheets. Google Earth and Microsoft Virtual Earth have surely demonstrated the public’s appetite for an “easy to use” technology. Can the leap be made which create “easy to use” web based technologies that bring the relevant 20% of GIS analytic/exploration functionality to the 80% of the market hungry for it?

Popularity: 13% [?]