I’ve been working over the past few weeks on shaping the Web Foundation’s Open Data strategy and planning for upcoming projects. Talking to some of our Directors was also key to thinking more seriously about the issue I raised in a previous post and, more broadly, about the Open Data landscape we hope to enable, with the community, in a few years time.
As the Open Data community knows, Tim Berners-Lee proposed a 5-star scale for Linked Open Data back in 2010. This has since been used to rank datasets, as in the map of ODIs. Useful as it is, there are two broad issues I have identified here: Lack of metrics and methods, and scope.
Metrics and Formal Methods
There is neither a methodology nor formal indicators stating how and to what extent an ODI complies or not with that scale. For example, can we say that the UK initiative is a 5-star initiative based on that scale? Producing 5-star linked data is not trivial. Although the UK program is doing remarkable stuff, only a small percent of all datasets released so far could score 5-stars. And what percentage of released datasets should be 5-star for an ODI to be considered as such? Is that percentage really important? It’s easy to argue that there are datasets that have more value than others. So even if only 10% of the data in an ODI were 5-star, that might be almost as valuable as making the remaining 90% 5-star. And what does “most valuable” mean? For whom?
We are approaching the limits of the scale as such, and trying to understand why measuring impact of ODIs is still a big (but important and interesting) issue. Other measures, such as number of downloads per dataset, are not sufficient either. We need more — much more — than that. We are including measurement and evaluation of datasets as part of our research agenda. Have you heard of the upcoming Web Index? Check it out, if you have not. You’ll realize how ODI issues are related to several that we are uncovering in building the Web Index, and why we’re finding increasing synergies between both initiatives at the Web Foundation that might even lead to an Open Data Index in the not so distant future.
As I mentioned in my previous post, the technical dimension is just one of several that should be taken into account. In my view and, especially when talking about Open Government Data, we must think about up to six dimensions: political, legal, organizational, technical, social and economic. TimBL’s scale mainly targets the technical aspects.
A Proposal: Building a 5-Star Scale for ODIs
So what is needed to make an ODI a 5-star ODI has yet to be stated (and proven!) I’m a firm believer on acting across all dimensions. I’ve witnessed failures in ODIs (and eGovernment projects) for not doing so, having seen very short-term successes becoming mid-term flops. Putting data online just for the sake of doing it could result, in the not so long term, in abandoned websites and lack of supporting (and much needed) legislation requirements.
Here’s a proposal: a 5-star scale ODI is one that is 5-star on every single of the six dimensions.
TimBL’s scale is valid for the technical one but we need to set scales for the other dimensions. What is more, we need to set formal methods and indicators to measure compliance. This would allow us to target very specific steps on specific dimensions, plan ahead for what is achievable in terms of compliance and, in general, make better informed decisions. For example, we may say that country X seems to be ready to achieve 3-stars on the technical dimension and it’s already a 3-star on the political one but needs to focus and act consistently on the legal dimension because there, it’s only hitting the 1-star mark.
Don’t ask me just yet what a 5-star ODI on the (e.g.) economic dimension means. One that creates 1,000 jobs in a year? What about 10,000? Or one that contributes to the country GDP at 1% per year? Both? What else? It might well be that we need some other scale analogous to the stars scale for those other dimensions.
We are starting to work on an initial experimental model, but please consider this post as a statement and call to action for you to provide input on our strategy on these very important aspects.
It is our goal at the Web Foundation to build locally sustainable Open Data ecosystems. It’ll be our goal to target 5-stars across every dimension, and also develop free open global resources for others to use to get there. As you may imagine this is not an easy nor a short-term task so don’t expect us to get there tomorrow but maybe the day after with your much needed and appreciated help.
José M. Alonso
November 17, 2011
just made a few grammatical edits
5-Star Open Data Initiatives | Transparency & Accountability Initiative
November 22, 2011
[...] Source: World Wide Web Foundation [...]
November 23, 2011
Hi JoseWhy is the 5-star scale being applied to ODIs? Wasn't it conceptualised for evaluating individual datasets? Your proposal for a multi-dimensional scale sounds likes it would be fitting for evaluating whether an ODI would be successful or discovering what the obstacles to implementation would be but I'm having trouble understanding it as a metric for an ODI itself. Am i missing the point here?
November 23, 2011
Hello Jose,This is some really interesting food for thought. I was going to drop in a quick comment, but it turned into a rather long blog post:http://www.practicalparticipation.co.uk/odi/2011/11/evaluating-open-government-data-initiatives-can-a-5-star-framework-work/In summary, thinking this thought my sense was: - There is a need to distinguish organisational/institutional change aspects of ODIs, from their impacts, and look at these as distinct sides of the question - It's hard to see how a simple 'index' of ODIs could capture the diversity of them (or could do so without narrowing focus to a fairly formalistic set of concerns about the supply side, weakening your multi-dimension point)- Care needs to be taken to understand how far the 5-stars are a scale, or how far each star presents a cumulative indicator that can be used as a target for mapping and planning status and actions in an organisation.Would be really happy to explore this space more: some potential overlaps with the directions I'm scoping PhD study in.
José M. Alonso
November 24, 2011
@Adi no, you're not. Consider the existing 5-star scale as an example. Even for the technical dimension alone we will likely need to evolve it to a higher level one, beyond individual datasets. Simple example: how to evaluate the Open Data portal itself?Adi, think of the 5-star scale as an example for a baseline on the technical dimension. Anyway, as TimD mentioned in his post, even the technical dimension goes beyond that scale, which is more easily applicable to a dataset.My thinking is that a scale (yet to be defined) would be useful for both. I see 3 main phases for ODIs:1/Evaluation of readiness. Is gov X ready to start a successful ODI?2/Implementation. Toughest part. Taking previous phase as partial input.3/Monitoring and Evaluation (although the means to do M&E should be built in previous phase and embedded in the various architectural pieces) As in every single project one should know where it stands at the beginning and where it stands at the end, how to measure the difference, how to address deviations from plan, etc.
José M. Alonso
November 24, 2011
@TimYour comments are much appreciated and useful. Every time I think seriously about issues around indicators and indexes, I have to recognize I find many. People also tend to put too much trust on indexes, but having witnessed politicians pointing to them and being much concerned by them (you mentioned this relationship in your post), it's a tool we need to consider. One important takeaway from your post is that you're not questioning the dimensions, so we might be up to agreeing on an initial approach here. If you think there's something big missing, please shout!Beyond that, I agree with the points you made. My main goal here is not the index. That may come as a consequence. Let's set the index part aside for now and focus on evaluating evolution and impact of ODIs.What we are really up to here at the Web Foundation is developing shinning examples and trying to understand what that means. An example. I said numerous times that you can do Open Data without transparency and transparency without Open Data. But if you want to have a shinning Open Data example, you want to put stuff such as transparency and accountability at the core. Demand side of the equation is always asking for performance related data. This is the sort of feature you might want to add to a, so ti say, 5-star ODI, and evaluate. Whether we can build an scale or not, well, as you said, that's difficult to do.I'd be more than happy to explore these questions further with you and others as I believe they are of increasing interest and need.
Complexity and complementarity – why more raw material alone won’t neccessarily bring open data driven growth : Tim's Blog
October 28, 2012
[...] system involves paying attention to many different dimensions of the environment around data. Jose Alonso highlights “the political, legal, organisation, social, technical and economic” as all being [...]