We’ve been playing around lately with some new data visualization techniques, so I started poking around to see what approaches have been generating buzz of late. In the process I came across Stamen Design’s very cool “Oakland CrimeSpotting” via Brady’s post on O’Reilly Radar. We’d been really impressed with what they did at Where 2.0 with Trulia and their latest further pushes the frontier.

Once I was done being wowed by the visual I found, Stamen had some thoughts similar to ourselves on the importance of public data being more accessible to…well the public. Specifically,

“We’ve found ourselves frustrated by the proprietary systems and long disclaimers that ultimately limit information available to the public. As citizens we have a right to public information. A clear understanding of our environment is essential to an informed citizenry.”

That is the biggest mission of GeoCommons and it is encouraging to see other folks feel similarly. The sad thing is that in order to gain easy access to much of this data folks pay third party providers. One of our developers, Minh, passed along a list of data you can buy from Pushpin, including ESRI supplied Census data. Census data comes from the government, all our tax dollars pay for it – why do we pay twice? I’m not intending to pick on ESRI or Pushpin – the whole industry does it, and for many free data sources other than Census.

The good news is there has been a real ground swell to change the market. Whether it is OpenStreetMaps, ShapeWiki, or OpenAerialMaps crowdsourcing geodata or Swivel and WeoGeo opening up public data for easy consumption there is some real momentum to change the market place and business model. Specifically on the Census front Bill, aka Mr Data McFindsAlot, just posted up Census tract level data for the entire US by state. You can download any of it as KML or access it through our data and mapping API’s. Census data, though, is just the tip of the iceberg of whats out there. My latest favorite “Pirate Attacks” a detailed dataset with lat long locations of real deal pirate attacks last year.

Popularity: 18% [?]

The lottery is an old hot topic of controversy with a new surge of debate. With the addition of North Carolina’s lottery and the impending implementation of video lotteries, supporters and critics are, once again, suiting up for battle over our nations lotteries and laws.
But outside of the active contributors to this conflict, what are the views of our nation regarding the lottery? For the most part, pretty positive. How can you not be, when the lottery takes all that green and turns into read…as in books and other supplies for education. After all, that is the most popular pitch for buying a lottery ticket and supporting the lottery’s establishment in a state. In fact, according to The New York Times, most people believe the majority of lottery dollars go to education and that most of the cost of education is covered by lottery dollars. With state lottery slogans like “Big Fun, Bright Futures” and “Raising billions to educate millions,” who’s to blame for these beliefs. Below is a video of one such advertisement.

Linked here is The New York Times article that addresses the misconception that the lottery heavily profits the education system: For Schools, Lottery Payoffs Fall Short of Promises. (Most of the facts and sources in this post are from this article, however here is another “Lottery Fact Sheet.”)

The original form of the lottery started in the 1700′s and was used to fund different social programs, bridges, roads, and the Continental Army. Now a days, there are thousands of different lottery games you can play with the cost of a ticket going as high as $50 dollars. You can find websites with lottery predictions or you can even be a professional lottery player.

Forty two states, plus the district, have lotteries. Twenty three of these states have dedicated funds to education. However, all lottery sourced money for education only amounts to “between one and five percent of the total revenue for K-12 education.”

Last year the lottery earned 56 billion and returned 17 billion to the state government. 460 million goes towards advertising which is one of the largest ad budgets of any marketer. Lottery officials say that since the induction of the first modern lottery, 234 billion dollars have gone toward state coffers. The majority of sales has to go back into the lottery just to sustain the games.

Here is a map showing the states who contribute at least 30% of their profits to education.
One must note that some states put their lottery dollars directly into the general pot, and can’t report exactly how many lottery dollars go directly to education.

A bad habit that is developing in certain states is the substitution of general funds for education with lottery dollars. So instead of additional money going towards education when the lottery does well, the education budget gains nothing, but the states general funds give less to meet the same mark. In fact, in Nebraska, they actually transferred lottery dollars out of education and into the general funds to make up for a deficit.

Lottery officials say that they are catching most of the flack for decisions that have nothing to do with the lottery, but are, instead, in the hands of the state legislature who decide what to do with the lottery dollars they receive. “Our job is to raise money for the things the legislators want,” said Clint Harris, director of the Minnesota lottery. “We don’t have any control over what happens to the money.”

The New York lottery director, Gardner Gurney, says, in defense of his percentage statistics, “Too much of the focus is on percentages. My focus is on dollars. You can’t spend percentages.” In 2000, New York State kept 38 percent of its lottery revenue for education. That share has dropped to 32 percent, but the dollar amount rose from $1.3 billion in 2000 to $2.2 billion last year, as reported by the NY Times.

Time and again the lottery argument comes back to money for education. According the lottery promoters, people are spending their money on tickets, it’s just a matter of whose state they are going to spend their money in. Then to pull a few heart strings, they mention all the kids in their home state who are watching those dollars go to other kids’ educations. However, taking a closer look at the facts makes it look like the only thing profiting from the lottery is the lottery. After all, we all know that you have a better chance of being struck by lightning twice than winning the lottery. Or do we? I guess the people standing in line for lottery tickets aren’t all decked out in rubber.

Popularity: 7% [?]

NPR ran a story on Monday’s Morning Edition entitled “Security Officials Seek to Block Some Online Maps”. The story centered around local government officials refusing to release electronic maps of what they call “critical infrastructure,” such as water mains and fire hydrants. Specifically the story of Steven Whitaker’s futile quest to obtain infrastructure data from the Greenwich, CT local GIS repository. As part of the story NPR came by to ask my opinion on the matter because of our history of creating security concerns using open source data.

The story has a nice quote of me saying it was an impossible task to try and control all the geodata out there and who has access to it. The part that did not air is that no one even knows what data is accessible and not accessible to the public. While we do have a good index and census of most of the web pages that exist, we have much less understanding of the databases including geospatial databases connected to the Web (often called the Deep Web). The indexes run by Google and others do a great job finding web pages but databases are a different game. A Cal Berkley study by Bergman found that, “the deep web consists of about 91,000 terabytes. By contrast, the surface web, which is easily reached by search engines, is only about 167 terabytes.” While it is uncertain how much of this data is geospatial in nature it is fair to assume it is a considerable amount of data that we largely have little clue about. Often times government agencies do not even realize what data they have online available to the public, and we definitely do not have a comprehensive way to understand the entire universe of geospatial data. What raised so much alarm with our original research were the authorities realizing that that the data was available open source. Everyone clamored the work should be classified, but the source data is all still out there hidden in myriad local, state, federal and NGO data repositories. This begs the question, how are we going to control a world of data that we have so little comprehension of?

In order to move towards greater security I believe we actually need to open up more so that the entirety of geospatial data can be indexed. We will have no true idea as to what geospatial data available to the public is potentially dangerous until know what is out there. The move towards making KML an OGC standard is a great first step as a standard geospatial data format for the Web. Although KML natively is geared towards providing a geographic framework for text, html, pictures etc., and not structured information like databases. We’ve been working on changing that by ensuring a mechanism exists by which to include feature attribute data in the schema tag of KML . Some of this work has carried over into KML 2.2 as “extended data“.

Once you begin to index the geospatial data out there you are in a much better position to have a logical debate about what data is a security threat and what data contributes to the public good. For instance you may want to know where there have been hazardous pipeline accidents, but not divulge where critical pipeline routing junctures are. By opening up geospatial data, not only do we have a foundation to better insure dangerous data stays out of the hands of bad guys, but we also have the positive externality of a whole wealth of data being made available to the public to solve a wide range of problems.

Read the rest of this entry »

Popularity: 13% [?]

In conjunction with this morning’s social media news release announcing we have reached 2500 sets of data on GeoCommons, we also created a short podcast to expand on some details. Sean discusses the exponential power of each new data set added to GeoCommons. He also provides some insights into what may be next for GeoCommons.

If you prefer to download the podcast, please visit media.libsyn.com/media/geoliv/GeoCommons2500podcast3.wav.

Popularity: 5% [?]