When we started the very first iteration of GeoCommons in 2005 folksonomies were all the rage and we jumped on board using tags to organize the geospatial data that was pushed into the new platform. During the time we had the prototype deployed we ran into many of the same issues other applications have found with folksonomies

1) people’s tags may be difficult for others to understand,
2) people may have tagged items inappropriately for others’ needs.

In short your users will not always implement tags in ways that are productive for the community – in the extreme resulting in Flickr’s 20 million unique tags. How many of those 20 million tags are misspelled words or so off the path they never get found.

In addition to the problems you encounter with folksonomies in general you have the further complications of geopspatial data. All geospatial data sets have location tags, but adding them in an unstructured way creates enough chaos that it is very difficult to leverage location tags in a thorough way. Secondly many potential users do not know the variety of geodata available. Put more simply they do not know what to search for, and having the ability to browse through data by topics is appealing.

Despite the downsides of folksonomies they are incredibly powerful and have been hugely effective in organizing vast amount of data on the web. So, as we worked on the next iteration of GeoCommons we started looking at possible hybrid approaches to folksonomies and hierarchies.

Specifically we looked at the two problems specific to geospatial data listed above 1) place tags and 2) organizing data for browsing. Solving the problems required both short term and long term solutions.

Fortunately we had a small advantage over many crowd sourced project in that we have a full time data team. They are a great group of folks that spend their day finding cool geodata and coming up with clever ways to organize it.

Through the data team and the other community members that contributed data to the first iteration of GeoCommons we had a big pool of data with a wide variety of tags to examine. What we found were some distinct trends in the tagging and titling of data. Across the data there were a commons set of tags that broke the data up into a useful set of distinct categories, but there were also many data sets that were tagged with elements that made them often indiscoverable. After the analysis we started to look at structures we could establish to help create self similarity in tagging that still had the flexibility to be adaptive.

The result was the creation of a location and topical taxonomy based on our existing corpus of data that has the intelligence to adapt as the content grows and evolves. I can’t go into the technical details in depth, but fundamentally the concept is to intelligently leverage the taxonomies and structures to provide suggestions to users to tag their data better.

In many cases this can be very simple – like providing tips on how to tag and title effectively to make your data more valuable to the community. For instance with titles we found across GeoCommons there were four key pieces of information used for datasets in the past.

1) Source name, 2) Original Name of Dataset from Source (or short description of dataset) 3) Geographic Area, 4) Time period of data

Examples:

Communicating this effectively to users is a great way to get better consistency across data contributions, while still allowing flexibility for users to be creative and bring in information that does fit the rigid mold of a hierarchy. Of course this is the most simple and you can get far more clever.

Del.icio.us for instance has a great feature that notifies a user they are putting in a new tag no one has used before and asking if that is what they meant to do. You can also suggest tags from your taxonomy that are semantically related to the data the user is contributing. This creates a consistency across tags that makes data easier to find as the system scales to larger volumes.

The nice thing about taxonomies as opposed to folksonomies is that they can be structured as trees, which means you can compute across them quite easily. With a solid and adaptive taxonomy in place you can go a long ways in intelligently guiding users towards creating better and more consistent tags. At least that is what we think and it will be fun to see how it works out after the launch.

Popularity: 23% [?]

First off want to thanks the folk that commented on the last post. Lots of useful feedback and it also highlighted a bit of confusion I created with the first post. The purpose of the first post was not a proposal to create a new metadata standard. Instead it was simply a proposal of how we could map the metadata we collect in GeoCommons to existing standards.

From that standpoint the proposal is for an implementation not a standard. We have just about 5,000 unique datasets and about 70,000 data layers, and it would be great to expose useful metadata for the data. The data covers the gambit, from EPA toxic release sites to the number of Facebook users by city. The system and metadata requirements needs to be flexible enough to accommodate both a user uploading Facebook data and one uploading EPA data.

While GIS users might not be intimidated by a metadata form with 75 or even 335 elements your average Web/GeoWeb user definitely will be. The goal with GeoCommons is to provide a destination where both communities can consume and share data, and I think both communities will find tools and data that are useful.

In regard to the metadata elements we proposed to map to in the last post, we were looking for those that both technical and non-technical users would understand, and also automatically trap as many additional elements as possible. To cover technical users, that have a full compliment of metadata, the plan is to have an element where you can you can provide a link to a full metadata specification.

The comments directing us to the ISO 19115 standard were very useful and we are looking to see what elements we are missing to map to that standard as we evolve. The thing we want to make sure we get right is finding to best set of metadata elements to request from users. Balancing the fact that if we have a huge number of elements, most people are going to go running for the hills.

Right now it looks like we’ll have 17-20 elements that will map to Dublin Core, FGDC, and in a next release ISO 19115. So, for each data set in Geocommons you’ll have a page that lists those 17-20 elements in the metadata format technical folks are used to seeing. This should also provide a means by which to explore federating the data with other applications and search approaches.

The goal here is to create a bridge between content being created for the GeoWeb and content created for the GIS world and make both usable and remixable by the web community as a whole. I fully respect the motivations and requirements for the GIS metadata specifications out there, and I hope we can leverage them to create an implementation that will see a high level of adoption.

Without adoption standards are pretty hollow as we’ve seen with all the work that went into GML versus the much lighter specifications for KML and GeoRSS. While both have their place it is clear what the market is supporting. As more geospatial data is created outside of the government we are not going to have the government mandate to force metadata creation and what the market accepts is going to become increasing critical – IMHO. Look forward to getting more feedback as we get ready to launch.

Popularity: 13% [?]

It seems like it is a daily dose of semantic web on the tech blogs of late. Today it was Textwise’s Million Dollar Semantic Hacker Challenge and a few days ago it was Yahoo opening their search platform to support a wide variety of semantic web standards. This has lead to a good bit of proselytizing, mostly in the comments, that this heralds the arrival of the Semantic Web, or Web 3.0 or the Next Generation Web. All of which sounds like the circling of the marketing band wagons.

Unfortunately when the wagons circle everything starts picking up the label – in this case semantic. This is especially dangerous when you have a word like “semantics” that can be defined, so many different ways. Just look at the definition tree created by Wikipedia:

*Semantics is the study of meaning in communication.
*In computer science semantics reflects the meaning of programs or functions.
*The Semantic Web refers to the extension of the World Wide Web through the embedding of additional semantic metadata

More often I see folks labeling things semantic that are really syntax. “Syntax” being the rules to construct and define something like a sentence or line of code and “semantics” the meaning of those rules or definitions. Syntax is fairly easy and semantics are fairly hard, as most folks in artificial intelligence would argue. Even going so far as saying all programming languages other than LISP are syntax and not semantic.

This is a bit more clear with an example. Lets take the Textwise announcement – a technology that will parse plain text on a website or elsewhere and categorizes it to predefined topics. One example in the Techcrunch comments was the following:

input text:
Call us crazy, but we think there are some brilliant minds out there that can find some really amazing uses for this incredibly powerful and scalable technology. Think you’re up to the Challenge? We think you are!

categories (ranked from 0 (worst) to 100 (best)):
Shopping/Health/Alternative/Hypnotherapy/Audio_and_Video 43 Business/Telecommunications/Services/Wireless/Software 33 Arts/Music/Bands_and_Artists/311/Tablature 28
Computers/Internet/Consultants/Research 26 Shopping/Health/Alternative/Meditation/Audio_and_Video 25

The output is really not telling me anything about the meaning of the text just setting up rules to provide categorization. So I would definitely put this in the syntax and not semantic category. I would also say what Yahoo! is doing is really more syntax than semantics although there is the possibility of building truly semantic technologies on top of what they are enabling. They’ve created a set of rules based on rich standards to allow applications to be built. Remains to be seen what will come of it, but in rush of market buzz I think it is easy to miss that building truly semantic technologies is quite hard. Some folks in AI (the Chinese room) would argue machines are not even capable of semantic meaning or understanding.

From this perspective I think we’ll see a lot of people building applications based on syntax that reorganize and categorize content by giving the “page web” a bit of structure. Oddly its like we’ve gone full circle back to DMOZ. While these technologies may be clever and useful I do not think they will fundamentally change the Web. In the other category I think we’ll see a few companies pushing towards something more sophisticated (call it a semantic, implicit, computational web) where new data and services are mixed with existing web content to provide answers to users questions.

Popularity: 15% [?]

It is only fitting that the day after I posted “Moving Push Pins Off the Map” I saw the post on Ogle Earth about a new geotagging icon….which is?

A GIANT PUSH PIN!

With my interest peaked we did a little digging and found another geotagging icon:

ANOTHER GIANT PUSH PIN (actually when I dug into it this icon was a first version that evolved into the red one.)

I of course blame this all on the Google monolith for perpetuating push pin mania. Last time I saw Mike Jones he even had a push pin tie tack. Joking aside the reason for creating a geotagging icon itself is worth discussing.

The stated purpose on the GeoTagIcons.com website is “The Geotag Icon is intended as a web “standard” icon for identifying geotagged content to humans.” So, if a photo or blog post has been geotagged then there is an icon on it to let you know. The thought being many times geotags are hidden in microformats or the URL, thus not visible to the user.

This seems like a straight forward approach to the problem, but also seems to have overlap with existing icons such as KML and GeoRSS. The tutorial on GeoTagIcons has examples of using it for links to both KML and GeoRSS content. This could lead to some ambiguity and confusion for users.

One of the most interesting parts of the pitch for using the GeoTagIcon is, “Reason 4: It encourages development of the semantic web”. On first blush this got me excited, but reading a bit deeper realized they meant it acts as an advertisement for linked content that could help support an evolving semantic web. This is in and of itself is a worthy cause and advertising has been directed at far less useful goals.

The link between geotagging and the semantic web does bring up a good topic for debate. How will all these geotagged objects (KML, GeoRSS, geo-microformats, GPX, etc.) be tied together in a method that creates semantic meaning? What questions will the semantic technologies answer? The GeoTagIcon site provides an example of , “Show me a plot of other bloggers in my vicinity”, or “I’d like to see a map showing which of my friends have also visited Australia”, “Who else has photographed this location?”, etc.

While these are interesting I think the examples and the direction many folks are taking geotagging misses the real potential of the semantic web. The geotagging premise is based on doing increasingly sophisticated things with geo-coded annotations – 99% of the time taking the form of a pushpin. In each of the examples above users or a screen scraper and geo-coder (most likely) have added a latitude and longitude to a piece of unstructured data (bloggers, my friends, photos). While this all useful information it is often relegated to only answering trivial questions.

There is only so much you can do with a bit of unstructured text or html that has geographic coordinates. You can measure vicinity (bloggers nearby), intersection (friends that have visited Australia) and union (show me all photos from a location). There might be a few that I am missing but it is fairly small universe of questions that can be answered, and the semantic web is all about answering questions. Hopefully a very large universe of questions.

From my limited perspective the semantic web is all about bringing vast data resources to the web in an easy and intuitive way. While turning unstructured text into geocoded annotations already on the web is important I think the bigger challenge is blending existing structured data (largely in databases and not on directly on the page web) with organized unstructured data through the web in a seamless way like we have for text, pictures and video.

Metaweb has done some compelling work with Freebase. They have even been doing some interesting geo work with their database. To date Freebase has largely been working with conceptual data, but from the look of their GIS app could be getting into more quantitative data.

As you get into quantitative data the power and tools available for asking sophisticated questions increase exponentially. Unfortunately so do the technical challenges, both computational and creating an intuitive user experience for something not intuitive to most people – numbers, math, statistics, etc. Despite the challenges I think this is where some of the greatest potential awaits for the emerging semantic web. That said I do think the new icons are quite nice and serve a useful function – despite the push pin.

Read the rest of this entry »

Popularity: 22% [?]