This is a slightly modified version of a post that originally appeared on the Open Data Research Network site.

At the Web Foundation, we’re convinced that open data, accessed via a free and open Web, holds the potential to tackle some of the toughest challenges facing humanity. By empowering activists, businesses, governments, civil society groups and others to analyse and link data in new ways, open data often makes it possible to find new, cost-effective solutions to complex and opaque problems, or to spot hidden patterns of waste and corruption.

However, as the old adage goes: “You can’t manage what you can’t measure”, and so as open data initiatives have grown in scope and popularity, a range of different efforts have emerged to measure various aspects of open data readiness, implementation, outcomes and impacts. In order to explore the state of the art in measurement of open data activities, and to explore opportunities for future collaboration in their development, The World Wide Web Foundation and The Governance Lab at NYU convened a two-day workshop on May 8-9, 2014 in New York, bringing together open data assessment experts to explore the development of common methods and frameworks for the study of open data.

This two day workshop started by reviewing existing research and projects on open data measurement and looking at use cases assessment, before identifying key questions concerning open data and looking at conceptual frameworks that could connect questions to use cases. Building on this foundation, the second part of the workshop focussed on identifying common categories and indicators within an overarching framework. Meeting participants split into sub-groups to consider specific questions, and to build out the larger framework with which we can study open data.

We’ve published both the full workshop report (PDF), as well as the draft framework (Google doc), which is open to comments and feedback until July 8, 2014.

The framework at a glance

In assessing open data activities, a project may look at:

• Context/Environment – The context within which open data is being provided. This might be the national context, as in the case of central Open Government Data, or might be the context in a particular sector. Important aspects of the environment to assess include the legal and regulatory environment; organisational context; political will and leadership; technical capacity; the wider social environment, in terms of civil society and political freedoms; and the commercial environment and capacity of firms to engage with open data.
• Data – The nature and qualities of open datasets. This includes the legal, technical, practical and social openness of data, as well as issues of data relevance and quality. The framework also looks to identify core categories of data which might be evaluated in assessments.
• Use – The context of use of the open dataset. This includes the category of users accessing (or providing) the dataset, the purposes for which the data will be used, and the activities being undertaken. This part of the framework addresses the who, what and why of open data programmes.
• Impact – The benefits to be gained from using the open dataset. Potential benefits can be studied according to social, environmental, political/governance, and economic/commercial dimensions.

In the draft framework we provide suggested questions and indicators for each of these components, and look at the existing projects that have piloted relevant methods. You can add your comments to the online version of the framework on Google Docs. We’ll be looking to turn these discussions into early action by implementing these framework elements into our 2014 Open Data Barometer, due out in late 2014.

Building on a common framework

As we explored in our workshop in New York, there are many diverse reasons to assess open data activities—from checking implementation of a plan, through to critically researching what kinds of approaches to open data yield the most equitable social outcomes—and open data activities are operating at many different levels—from national open data policies, to sectoral programmes in health or education, to individual small NGOs adopting open data practices. Different assessment projects will need to pick-and-mix from the overall framework to select the kinds of questions and indicators that will be most relevant in any particular instance.

However, having a common framework to draw on allows researchers, policy makers and practitioners to share definitions, create comparable data, collaborate on the design of measurement instruments and training, and ultimately join efforts in building up a rich picture of the evolving open data landscape across the world.

In the report you will find details of the next steps we have planned for this work, including creating space for ongoing sharing and discussion amongst those working on open data measurement.

Related documents

• Towards common methods for assessing open data: workshop report & draft framework  (PDF)
• Draft framework table (Google Docs)

About the Open Data Research Network

The Web Foundation, together with the International Development Research Centre (IDRC), coordinates the Open Data Research Network—a collaborative project that works to connect open data-focused researchers from around the world, bring together information and news relating to research into the implementation and impacts of open data initiatives, and host focused research projects into open data. The network is open to all researchers interested in open data, and has a particular focus on research into open data in the global South.