I’ve been working over the past few weeks on shaping the Web Foundation’s Open Data strategy and planning for upcoming projects. Talking to some of our Directors was also key to thinking more seriously about the issue I raised in a previous post and, more broadly, about the Open Data landscape we hope to enable, with the community, in a few years time.

As the Open Data community knows, Tim Berners-Lee proposed a 5-star scale for Linked Open Data back in 2010. This has since been used to rank datasets, as in the map of ODIs. Useful as it is, there are two broad issues I have identified here: Lack of metrics and  methods, and scope.

Metrics and Formal Methods

There is neither a methodology nor formal indicators stating how and to what extent an ODI complies or not with that scale. For example, can we say that the UK initiative is a 5-star initiative based on that scale? Producing 5-star linked data is not trivial.  Although the UK program is doing remarkable stuff, only a small percent of all datasets released so far could score 5-stars. And what percentage of released datasets should be 5-star for an ODI to be considered as such? Is that percentage really important? It’s easy to argue that there are datasets that have more value than others.  So even if only 10% of the data in an ODI were 5-star, that might be almost as valuable as making the remaining 90% 5-star. And what does “most valuable” mean? For whom?

We are approaching the limits of the scale as such, and trying to understand why measuring impact of ODIs is still a big (but important and interesting) issue. Other measures, such as number of downloads per dataset, are not sufficient either. We need more — much more —  than that.  We are including measurement and evaluation of datasets as part of our research agenda. Have you heard of the upcoming Web Index? Check it out, if you have not. You’ll realize how ODI issues are related to several that we are uncovering in building the Web Index, and why we’re finding increasing synergies between both initiatives at the Web Foundation that might even lead to an Open Data Index in the not so distant future.

Scope

As I mentioned in my previous post, the technical dimension is just one of several that should be taken into account. In my view and, especially when talking about Open Government Data, we must think about up to six dimensions: political, legal, organizational, technical, social and economic. TimBL’s scale mainly targets the technical aspects.

A Proposal: Building a 5-Star Scale for ODIs

So what is needed to make an ODI a 5-star ODI has yet to be stated (and proven!) I’m a firm believer on acting across all dimensions.  I’ve witnessed failures in ODIs (and eGovernment projects) for not doing so, having seen very short-term successes becoming mid-term flops. Putting data online just for the sake of doing it could result, in the not so long term, in abandoned websites and lack of supporting (and much needed) legislation requirements.

Here’s a proposal: a 5-star scale ODI is one that is 5-star on every single of the six dimensions.

TimBL’s scale is valid for the technical one but we need to set scales for the other dimensions. What is more, we need to set formal methods and indicators to measure compliance. This would allow us to target very specific steps on specific dimensions, plan ahead for what is achievable in terms of compliance and, in general, make better informed decisions. For example, we may say that country X seems to be ready to achieve 3-stars on the technical dimension and it’s already a 3-star on the political one but needs to focus and act consistently on the legal dimension because there, it’s only hitting the 1-star mark.

Don’t ask me just yet what a 5-star ODI on the (e.g.) economic dimension means. One that creates 1,000 jobs in a year? What about 10,000? Or one that contributes to the country GDP at 1% per year? Both? What else? It might well be that we need some other scale analogous to the stars scale for those other dimensions.

We are starting to work on an initial experimental model, but please consider this post as a statement and call to action for you to provide input on our strategy on these very important aspects.

It is our goal at the Web Foundation to build locally sustainable Open Data ecosystems. It’ll be our goal to target 5-stars across every dimension, and also develop free open global resources for others to use to get there. As you may imagine this is not an easy nor a short-term task so don’t expect us to get there tomorrow but maybe the day after with your much needed and appreciated help.