Brett Taylor says that we need a Wikipedia for data. He realizes how hard it is for a everyday programmer to get access to even the most basic factual data, which is a barrier to innovation.

Dave Bouwman shows us the National Geographic MetaLens service with Virtual Earth. MetaLens is a geospatial content management and archival system that National Geographic uses to secure and manage its content.

Dan Catt from Geobloggers and Flickr shares the new Flickr video and geo-tagging option.

James Fee shares how to leverage the Google application engine with GIS applications. He also reviews the confusing commercial difference in licenses with Microsoft Virtual Earth Mapcruncher and the MSR edition.

According to GISUSer, General Dynamics has completed the testing for Geo-Eye, an earth imaging satellite. GeoEye-1 is part of the National Geospatial-Intelligence Agency (NGA) NextView program. The NextView program is designed to ensure that the NGA has access to commercial imagery in support of its mission to provide timely, relevant and accurate geospatial intelligence in support of national security.

GISLounge shares top causes of errors in online mapping systems, including inaccurate base data, accuracy of geocoding, lag time to incorporate newly developed areas and difficulty in interpreting variations on addresses.

Popularity: 12% [?]

Bret Taylor had a great blog post today on the need for a Wikipedia for data. In short Bret feels like the difficulty in accessing data for web development project seriously hinders innovation. Lots of people resonated with the blog post and it is already up to 51 comments and has spawned several related posts – including ReadWriteWeb’s excellent list of web application offering open source data and related services.

We’ve spent a good bit of time working on the geospatial side of producing a large repository of open data. Specifically, dealing with the ability to handled structure quantitative data. I thought this might be a good time to talk a bit about the challenges we ran into and possible solution to the bigger problem posed by Bret in his blog post.

When we launched GeoCommons just short of a year ago we had grand ambitions of creating a community around geospatial data and creating maps. As with most first attempts we got overly ambitious and tried to pull off too many features in the first go, and ran into a serious case of overall mediocrity.

That aside the big technical hurdle that eventually led us to take GeoCommons down was a data scaling issue. The constant reading and writing to the database caused performance to really drag as the overall database got bigger and bigger. In the first go we used MySQL and hit a wall with about one billion data entities and then again with PostGIS when we got up to around five billion.

This problem was only further complicated by the size of the files we were working with, which often were over 10 mb. Geospatial data files trend large and can become quite monstrous. As a result we had upload and download times that far exceeded the patience of most general web users. Also, executing operations (mathematical, query, mining etc.) with data sets that large, and producing visual results in an acceptable time, created yet another challenge.

In short when Bret said, “DataWiki seems like an extremely hard problem” he was correct and we were just tackling one data niche. The good news is that the problems are solvable given enough time and resources, and we were able to find solutions for all the barriers we ran into with the first attempt at GeoCommons.

While this is good news for us I think it illustrates a bigger point – it is unlikely that one company is going to be able to solve the whole DataWiki problem. The list of current open data projects listed in the ReadWriteWeb post illustrates this hunch. Even a project as comprehensive and as well funded as Freebase is “better equipped to handle conceptual rather than statistical information on topics”.

Given the vast amount data out there and the limited resources any one company has, it seems that connecting multiple different repositories through a federation approach would be a clever way to go forward. This of course opens up an entire can of worms on best approaches and possible standards.

There are some important prerequisites like establishing metadata and schemas as well as the use of common formats, but a practical real world implementation I think will get us a long way. We’ve been talking with some folks in the GeoWeb space about doing some federation test beds with GeoCommons and I believe there are some interesting approaches to look at.

I’m curious what other folks out there think about the possibility of federation and who out there is open to doing test bed implementations? I believe that real potential of a DataWiki and the concept of a semantic web driven by data is going to require an interconnection of all the new data resource coming on line instead of the current islands of innovation.

Popularity: 8% [?]

I promised Andrew a comparison of the big three map creation applications by feature and functionality, so here it goes. The story of how lightweight web based map creation applications came to be is interesting in and of itself. I think looking at how the three applications evolved historically will provide a bit of insight.

Before the GeoWeb came into mainstream popularity both Microsoft and Yahoo! had mapping applications. Microsoft offered their browser based Terraserver which hooked up USGS imagery for the map tiles. Microsoft launched Terraserver in June of 1998 – practically prehistoric.

Microsoft had also been active in the mapping space with products like MapPoint (both desktop application and web services). Yahoo! also was an early adopter of mapping applications in conjunction with their local search destination (although I completely failed at finding a date for when they first added maps). Despite the early adoption of web based mapping applications by Yahoo! and Microsoft it was arguably the launch of Google Maps in 2005 that jump started both the GeoWeb and the mash up craze.

Shortly after Google Maps launched, Paul Radamacher hacked the application to allow it to display Craig’s List rental listings on the Google slippy map. Shortly there after Adrien Holovaty followed suit mashing up Chicago crime statistics with Google Maps. Google quickly released an API to allow developers to do the same thing seamlessly and we were off to the races. Microsoft quickly created Virtual Earth and Yahoo! pushed out Yahoo! Maps. Microsoft created compelling innovations with birds eye imagery and Yahoo! launched several popular GeoWeb services like free geocoding and Flash based mapping APIs.

Microsoft Collections

Through all these innovations there was a constant one way flow of content creation – developers could create unique maps and users could view them. Microsoft changed this when they launched Collections May 23, 2006:

Collections. Social networking functionality allows customers to create lists of favorite landmarks and locations, attach personal photos and save them to a Scratchpad. Collections can be saved, recalled later, “permalinked,” and shared with friends and community in e-mail or through their MSN® Spaces blog.

While not well publicized the “Collections” concept fundamentally changed the work flow for creating maps. No longer did you need to be a developer or GIS pro to create a basic map and share it with other people. The Virtual Earth folks even gave users a decent amount of cartographic power and options:

Customized pushpins. A pushpin is essentially a marker indicating points of interest on a map view. A customized pushpin can easily be added with a simple right click, anywhere on a map, which will display a small red dot and a pop-up menu. A pushpin title or note of up to 200 characters can be added that will appear with the pushpin whenever a mouse hovers over it. Pushpins can easily be edited or deleted. When a pushpin is removed, whether customized or standard, the remaining pushpins will be automatically renumbered.

2-D drawings in Collections. Users can add lines and drawings in a variety of colors, shapes and styles to personalize their Collection. They also can draw lines and shade areas that they want to mark on the map, such as for marking a running or bike trail, or neighborhood boundaries).

MyMaps

Despite the potential of the innovation the new functionality did not get much coverage in the press or massive levels of adoption. The TechCrunch article on it was lumped in with other new features from Yahoo! Maps.

Just short of a year later Google launched Google MyMaps on April 4th 2007 to big headlines across the blogs, including MyMaps being the death knell of popular map mashups like Platial, Frappr and Tagzania.

Fundamentally the functionality and features of MyMaps was not remarkably different than Collections, but the buzz around it was at least ten fold. So why was the attention so skewed towards Google for fundamentally the same innovation Microsoft had launched a year earlier? A few guesses:

MapMixer

Yahoo! was not too far behind launching their own map creation application, Yahoo! Mapmixer on September 13th 2007. Mapmixer took a different angle on map creation by allowing users to put static maps on top of the Yahoo! Maps applications. For instance after the Buscan oil spill in the San Francisco Bay last year I made a lot of calls trying to get the raw data on the location of the spills, for GeoCommons, but had no luck.

I did find a PDF with a map of the oil spills so I saved it as a PNG then uploaded it to Yahoo Mapmixer and they took me through three easy steps to georeference the map on Yahoo! Maps. The user experience I thought was the best of the three and there were lots of great social features for me to give a short description of the map and for other users to comment on the map. Although much like Microsoft the application did not generate lots of buzz as with Google MyMaps, and the gallery only features 38 user submitted maps today. Interestingly, in concept, it is quite similar to Microsoft’s MapCruncher, although it is a download and supports a wider variety of raster based formats that must already be georeferenced.

Since the launch of map creation applications by the three big players there have been two noticeable waves of enhancement 1) support for external data and 2) collaboration features. Microsoft put themselves out as being the first to support loading KML, “The October 07 release of Live Maps was the first to support KML viewing and import to Collections”. November 27th 2007 Google added KML, KMZ and GeoRSS support to MyMaps. Google followed this up with social features, like commenting, rating and open collaboration invitations for MyMaps.

Performance Trials

That covers features and functionality from a historic evolution stand point, but how do they perform? We did a very informal, one user, stress test. Create push pins as quickly as possible and see when the map application maxes out or gets sluggish. For Yahoo! Mapmixer this was pretty easy. You can overlay one picture or map onto the application, so you max out at one.

In the process of loading and georeferencing the image you get speedy performance and predictable response times. For MyMaps and and Collections we had a bit more to stress. We’ll start with Collections where we created 200 push pins with good response time then got the following message “You cannot add more than 200 items to a collection. To add more items, create another collection.”

When we went with the same test on MyMaps,we did high rate push pin creation and after about 30 the system got a bit sluggish, and sometimes it would create a listing for a pushpin on left hand pane but not create the push pin on the map. The caveat here is we were doing this high speed, and when we slowed down to a more deliberate pace the system handled it fine.

MyMaps also maxes out at 200 push pins on the map, but instead of providing a warning it generates a pagination for a continuing set of push pins. So when you click on the first page you get a map with the first 200 push pins and when you click on the second page you get the next 200 push pins on a new map in the same browser and tab. Oddly it stops at 820 push pins and starts back over at the number one but you can keeping adding push pins to the map.

What’s Next?

That pretty much wraps it up for a comparison of the big three, how they evolved in a competitive environment, and a very ad hoc test of their limits.

I believe the most interesting part will be where they evolve to next. What is the next set of functionality that will distinguish one from the other? Can Microsoft or Yahoo! introduce the next killer functionality that will catch up to 7 million maps that have been created with MyMaps?

Popularity: 70% [?]

The All Points Blog points us to a story from India where Yahoo Maps has added mapping capabilities in nine languages as well as many features including "key landmarks along the route (ATMs, hospitals, etc.), the walking direction." However, this may be a moot point since much of the country has limited internet access.

The AnyGeo Blog shows us ways that people can match up their assets on Google Maps, including using information from a cemetery and an office floor plan.

Dave Bouwman has an interesting post on the gap between client expectations of the ArcGIS server, and the realities. Does the marketing match the product?

Mapufacture shares a presentation on emerging mass market geo standards. Mapufacture is a huge proponent of supporting and sharing data via standards – especially ones that encourage broad adoption.

GIS Lounge gives us an example of a non-geographic use for geographic technology in the form of Zkimmer, which uses the “visual web” to promote reading of magazines through Google Maps.

Popularity: 9% [?]