Over two weeks have passed since the 7.9 magnitude earthquake devastated Chengdu, China. The end of this tragedy is still hiding as aftershocks continue to ripple throughout the country. News feeds continue to stream from China as the recovery process continues. The most recent reports have the death toll climbing to 50,000.

The USGS keeps daily records of recorded earthquakes worldwide, and enables us to pinpoint earthquake locations by providing latitude and longitude coordinates. Within Finder we have made this data available for use in shape, kml, and CSV formats to the public.

The following Datasets can be found on Finder, and can be used to gain a spatial perspective on the current events in China.

USGS, M 1+ Earthquakes, World, 5.5.08 through 5.12.08
USGS, M 1+ earthquakes, World, 5.12.08 through 5.19.08
USGS, M 1+ earthquakes, World, 5.20.08 through 5.27.08
USGS, Earthquake Records, World, 1998-2007

Popularity: 19% [?]

We got such useful feedback from the last metadata post I thought I would add some screen shots of how it is starting to come together. Unfortunately we were not able to get all the suggestions in because of the time crunch hitting our release date, but please keep posting the feedback and we’ll work it in as we have more time.

The first screen shot is of the data details page, which contains the metadata information for the data set. In this case 2000 US Census data at the tract level for Alabama:

Here you can see the major elements we are capturing in a user friendly graphical lay out. One of the cool new bits is the system automatically calculates statistics when you upload the data. Being able to data mine and run statistics on the fly is one of the new developments we are particularly excited about.

All the metadata on the data details page is exposed as Dublin Core elements which should make them machine readable to the rest of the world:

Also there are links to FGDC and ISO 19115 metadata mappings which take you to simple text pages with the indicated information. We probably need another pass to get these completely correct, but the infrastructure is all in place to do so.

FGDC looks like this:

ISO 19115 looks like this:

Hopefully this will help make the data in GeoCommons useful to multiple geospatial work flows. We hope having the ability to get data out in shapefile, KML, and .CSV (spreadsheets) will create more cross fertilization between GeoWeb and GIS users. With some luck it can help get more geospatial data out to the public that has been difficult to access in the past. A couple of examples below.

US Census Tract Data for Alabama

Global Maritime Shipping Lanes

Zillow Neighborhoods and Shipping Lanes (just because it looked kinda cool)

Thanks again for the feedback from folks on the metadata and we’ll keep iterating on getting it spot on.

Popularity: 37% [?]

I promised Andrew a comparison of the big three map creation applications by feature and functionality, so here it goes. The story of how lightweight web based map creation applications came to be is interesting in and of itself. I think looking at how the three applications evolved historically will provide a bit of insight.

Before the GeoWeb came into mainstream popularity both Microsoft and Yahoo! had mapping applications. Microsoft offered their browser based Terraserver which hooked up USGS imagery for the map tiles. Microsoft launched Terraserver in June of 1998 - practically prehistoric.

Microsoft had also been active in the mapping space with products like MapPoint (both desktop application and web services). Yahoo! also was an early adopter of mapping applications in conjunction with their local search destination (although I completely failed at finding a date for when they first added maps). Despite the early adoption of web based mapping applications by Yahoo! and Microsoft it was arguably the launch of Google Maps in 2005 that jump started both the GeoWeb and the mash up craze.

Shortly after Google Maps launched, Paul Radamacher hacked the application to allow it to display Craig’s List rental listings on the Google slippy map. Shortly there after Adrien Holovaty followed suit mashing up Chicago crime statistics with Google Maps. Google quickly released an API to allow developers to do the same thing seamlessly and we were off to the races. Microsoft quickly created Virtual Earth and Yahoo! pushed out Yahoo! Maps. Microsoft created compelling innovations with birds eye imagery and Yahoo! launched several popular GeoWeb services like free geocoding and Flash based mapping APIs.

Microsoft Collections

Through all these innovations there was a constant one way flow of content creation - developers could create unique maps and users could view them. Microsoft changed this when they launched Collections May 23, 2006:

Collections. Social networking functionality allows customers to create lists of favorite landmarks and locations, attach personal photos and save them to a Scratchpad. Collections can be saved, recalled later, “permalinked,” and shared with friends and community in e-mail or through their MSN® Spaces blog.

While not well publicized the “Collections” concept fundamentally changed the work flow for creating maps. No longer did you need to be a developer or GIS pro to create a basic map and share it with other people. The Virtual Earth folks even gave users a decent amount of cartographic power and options:

Customized pushpins. A pushpin is essentially a marker indicating points of interest on a map view. A customized pushpin can easily be added with a simple right click, anywhere on a map, which will display a small red dot and a pop-up menu. A pushpin title or note of up to 200 characters can be added that will appear with the pushpin whenever a mouse hovers over it. Pushpins can easily be edited or deleted. When a pushpin is removed, whether customized or standard, the remaining pushpins will be automatically renumbered.

2-D drawings in Collections. Users can add lines and drawings in a variety of colors, shapes and styles to personalize their Collection. They also can draw lines and shade areas that they want to mark on the map, such as for marking a running or bike trail, or neighborhood boundaries).

MyMaps

Despite the potential of the innovation the new functionality did not get much coverage in the press or massive levels of adoption. The TechCrunch article on it was lumped in with other new features from Yahoo! Maps.

Just short of a year later Google launched Google MyMaps on April 4th 2007 to big headlines across the blogs, including MyMaps being the death knell of popular map mashups like Platial, Frappr and Tagzania.

Fundamentally the functionality and features of MyMaps was not remarkably different than Collections, but the buzz around it was at least ten fold. So why was the attention so skewed towards Google for fundamentally the same innovation Microsoft had launched a year earlier? A few guesses:

MapMixer

Yahoo! was not too far behind launching their own map creation application, Yahoo! Mapmixer on September 13th 2007. Mapmixer took a different angle on map creation by allowing users to put static maps on top of the Yahoo! Maps applications. For instance after the Buscan oil spill in the San Francisco Bay last year I made a lot of calls trying to get the raw data on the location of the spills, for GeoCommons, but had no luck.

I did find a PDF with a map of the oil spills so I saved it as a PNG then uploaded it to Yahoo Mapmixer and they took me through three easy steps to georeference the map on Yahoo! Maps. The user experience I thought was the best of the three and there were lots of great social features for me to give a short description of the map and for other users to comment on the map. Although much like Microsoft the application did not generate lots of buzz as with Google MyMaps, and the gallery only features 38 user submitted maps today. Interestingly, in concept, it is quite similar to Microsoft’s MapCruncher, although it is a download and supports a wider variety of raster based formats that must already be georeferenced.

Since the launch of map creation applications by the three big players there have been two noticeable waves of enhancement 1) support for external data and 2) collaboration features. Microsoft put themselves out as being the first to support loading KML, “The October 07 release of Live Maps was the first to support KML viewing and import to Collections”. November 27th 2007 Google added KML, KMZ and GeoRSS support to MyMaps. Google followed this up with social features, like commenting, rating and open collaboration invitations for MyMaps.

Performance Trials

That covers features and functionality from a historic evolution stand point, but how do they perform? We did a very informal, one user, stress test. Create push pins as quickly as possible and see when the map application maxes out or gets sluggish. For Yahoo! Mapmixer this was pretty easy. You can overlay one picture or map onto the application, so you max out at one.

In the process of loading and georeferencing the image you get speedy performance and predictable response times. For MyMaps and and Collections we had a bit more to stress. We’ll start with Collections where we created 200 push pins with good response time then got the following message “You cannot add more than 200 items to a collection. To add more items, create another collection.”

When we went with the same test on MyMaps,we did high rate push pin creation and after about 30 the system got a bit sluggish, and sometimes it would create a listing for a pushpin on left hand pane but not create the push pin on the map. The caveat here is we were doing this high speed, and when we slowed down to a more deliberate pace the system handled it fine.

MyMaps also maxes out at 200 push pins on the map, but instead of providing a warning it generates a pagination for a continuing set of push pins. So when you click on the first page you get a map with the first 200 push pins and when you click on the second page you get the next 200 push pins on a new map in the same browser and tab. Oddly it stops at 820 push pins and starts back over at the number one but you can keeping adding push pins to the map.

What’s Next?

That pretty much wraps it up for a comparison of the big three, how they evolved in a competitive environment, and a very ad hoc test of their limits.

I believe the most interesting part will be where they evolve to next. What is the next set of functionality that will distinguish one from the other? Can Microsoft or Yahoo! introduce the next killer functionality that will catch up to 7 million maps that have been created with MyMaps?

Popularity: 81% [?]

I thought it would be fun to take a different angle on the virtual globe competition and look at the content and data made available by two of the players - Google and ESRI. From a technical perspective I think most would agree that ArcGIS explorer is pale emulation of Google Earth especially when it comes to user experience. I’ll put aside my gripes around difficulty in set up and version updates with Explorer and focus on content. The ESRI supporters have made the argument that ArcGIS Explorer is about access to GIS content and has a different mission than Google Earth. Actually, I would say the missions (exposing geographic data to larger audiences) are the same but the approaches are much different.

I’ll assume that some day ESRI will nail the technical side of Explorer, and look a little further down the road - to the content available for Google and ESRI’s thick client applications and how easy it is to access. As the GeoWeb evolves and we’ve rendered ever more amazing three dimensional worlds I think an increasing premium is going to be on the scope of data and content that can be delivered to these applications.

So where do the two ends of the GeoWeb spectrum sit on the topic? From my observations pretty far apart. For practical purposes I’ll break the comparison into four topics 1) data formats 2) data sources 3) data search and 4) data packaging

Data Formats

This is one category where ArcGIS Explorer ups Google Earth cleanly. Explorer allows you to load ArcGIS Explorer files (.nmf why they felt the need to create another proprietary file format that works with nothing else is beyond me), servers (WMS, IMS, file server), geodatabases, shapefile, raster, and KML. This is an impressive list, although a little bit less impressive when you consider that half are open standards KML, WMS and raster and the others are really ESRI proprietary formats at the end of the day. On the upside the raster support is quite extensive (30 or so different formats), although they require a spatial reference file.

On the Google side you have KML and KMZ (once proprietary now turned over to the OGC - jury is still out on how open it will be) for the free version of Google Earth, and within the free Google Earth 3-D warehouse support for SketchUp “.skp” and COLLADA “.dae”. In Google Earth Pro and Enterprise versions there is vector support for ESRI shapefile and MapInfo .tab, and for imagery support for TIFF (.tif), including GeoTiff, National Imagery Transmission Format (.ntf), Erdas Imagine Images (.img), Atlantis MFF Raster (.hdr), PCIDSK Database File (.pix), Portable Pixmap Format (.pnm), Device Independent Bitmap (.bmp). While this is an extensive list you have to pay to get the file support, in ArcGIS Explorer the support is currently free, so ESRI does have a distinct edge in the category.

Data Sources

In addition to supporting different file formats there is also baked in content for both applications. For Explorer you can access ArcGIS Online directly through the file server option, which allows you to access a file directory of cotnent. A second option is you can access the “Resource Center” website through the help tab, where you can download content in the .nmf file format. The “Resource Center” is definitely the more user friendly of the two with a nice user interface categorizing content into useful categories like, “imagery, street, physical features”.

On the downside the content is very limited - twenty four layers supplied by ESRI and four contributed by the community. ArcGIS Online has more content, but was a pain in the ass to access. You have to get a user name and password from ESRI, read through the incorrect direction to access it, then you get a list of UNIX style titles with abbreviations and underscores, like UNEP_WCMC_WDPA2006_2D. Not exactly user friendly, but you do get close to fifty additional layers of data once you jump through the hoops.

In Google Earth there are two sources of baked in data from the application, “Layers” including (terrain, geoweb, roads, traffic, 3-building, borders and labels, gallery, global awareness, and places of interest) and the ability to search for businesses. You can also click the “help” tab and be taken to the Google Earth Community web page. On the Google Earth Community page site alone there are 638,213 KML or KMZ files.

While finding this content (they are all file attachments to bulletin board posts) is pretty clunky and often frustrating, it is a LOT of content. Especially when you compare it to the four user generated files on the ESRI equivalent. The quality and source of this content/data varies wildly, and it is difficult to tell what is good and what is bad, but the potential is there. Actually the metadata support for both is pretty sparse. This is ESRI’s metadata for a .nmf imagery layer from the “Resource Center”:

“Displays satellite and aerial imagery at a 15m minimum resolution worldwide, and 1m resolution for the U.S. World boundaries, place names, and transportation layers are also included. Use this map to view man-made and natural features, or as a base map for overlaying associated data layers.”

For an ArcGIS Online layer this:

Layer Name: Imagery
Layer Source: ArcGIS Globe Service Layer
Layer Type: Draped
URL: http://services.arcgisonline.com/v92
Service: I3_Imagery_Prime_World
Sub Service: Imagery
LOD Tile Fetch: True
Hidden: False

Neither super useful.

Data Search

Neither application directly supports data search, but both have communities or services built around them that do. On the ESRI side there is a huge number of geospatial data repositories that have shapefiles in them and the search capabilities of them vary widely. Probably the largest is the Geospatial One-Stop that has a connection to most Federal geospatial data. Although in reality it is more often access to the metadata than the data itself. Still a large amount of content that could be conceivably viewed in ArcGIS Explorer, although largely disconnected in a large number of different repositories.

Google Earth has not only a good number of community aggregators like the official GE Community site above and unofficial like Mapufacture, GE Library and GLayers, but also the ability to search all KML files indexed on the web. I’ve heard numbers north of 10 million KML files indexed by this approach, but have nothing official. One way to search this content base is to type filetype:kml

Popularity: 51% [?]

During a late night epiphany we decided the blog had gotten a bit stale. So, to encourage a regular flow of content we figured a new look and pithy title would be just the trick. Welcome to the shiny-new, rebranded, USGS approved “Off the Map”. Now fortified with vitamins, minerals, insight, and elegant prose.

Why change the title to “Off the Map”? Well push pins seemed so 2005 and we needed another reason for an office contest. The winner you’ve now seen, but there were lots of other great entries such as:

1. Geo Me This
2. Plain to the Simple
3. MapRap (bling your map ????)
4. Map This (including middle finger to the man* graphic)
5. Libre la Data
6. The Lat and Long of It
7. Atlas Maximus
8. Adept and Disheveled

Why did we end up picking “Off the Map”. Well speaking for myself I just wanted to be able to use Kyle’s graphic with the dead push pin.

The next reason? As we’ve been developing the second generation of GeoCommons we found the big areas we were having to innovate had nothing to do with the map. The new ideas that were going to change the way people use maps - were literally “off the map”. Whether it was handling large datasets ridiculously quickly in a browser or structuring taxonomies and semantic relationships we were increasingly putting lots of resources into data management. Just so happened that data could be shown on a map.

Don’t get me wrong the map is still the single interface that ties all the data together, but increasingly I think what will make the GeoWeb matter has less to do with maps (including all sorts of crazy 3-D worlds) and more to do with delivering useful data to help people make better decisions. Which happens to be done through a map.

We should be getting a couple of posts up a week explaining this line of thought in more detail. Most likely with several side trips of randomness and entertainment. So, please stay tuned and we promise to keep a regular flurry of GeoWeb bits o’ knowledge.

* “The man” of course being all those evil cabals preventing easy public access to open data

Popularity: 23% [?]