The article gives helpful information about how all of these numbers break down and what they mean. The article explains that "most of the EU's money comes from member nation contributions, 108.5 billion EUR in 2010". Because of the fact that this is not completely straightforward to everyone, the Guardian's own graphic artist, Paul Scruton, visualises the data for us, so that it is easier to understand (see above). The data goes on to tell us that, "the biggest item of spending is the Common Agricultural Policy - which incorporates rural development and a small amount on fisheries."

More importantly though, is the fact that full datasets and spreadsheets of all the EU budget data is made available at the end of the article for anyone to read and make use of. This enables readers to not only get a glimpse into the EU's personal spending book, but it also allows them to take the information and run with it. The data world is your oyster. What will you make of it? (For the full article, please visit the Guardian Data Blog website).

How public funds should be spent is often controversial. Information about how that money has already been spent should not be ambiguous at all. People arguing about the future will care about the present, and if data about past or present public spending is available, many will certainly look at it. When they do, occasionally they will find errors, or believe themselves to have found errors.

OpenSpending, which aims to track every (public) government and corporate financial transaction across the world, encourages users to:

• augment the existing spending database with additional sources of data
• use that data -- e.g., to write evidence-based articles and formulate informed decisions about how their society is financed.

Spending Stories is our effort to make OpenSpending a natural way to do data journalism about public spending.

THE PROBLEM

FACT 1: Errors occur in data, no matter how official the source.

FACT 2: Data wrangling (manipulating or restructuring datasets to correct inaccuracies, remix with other datasets to augment the data, or perform calculations on the data), generally improves data quality, for example, through reconciling entities and flagging amounts that are obviously incorrect.

FACT 3: Data wrangling can also introduce errors if not tackled correctly.

Crucial to ensuring the use of this data in articles or ensuring re-use by concerned citizens is the ability to show that the data is valid. In addition, maintaining a good relationship with public bodies who are confident that they are not being misrepresented in the data is vital to ensuring the data continues to be released in the first place. In practice, this means that the provenance of the data has to be clear including:

• where the data originally came from (preferably a URL)
• whether anyone (e.g., government, community data wrangler, or OpenSpending) has worked on the data since it was published, and what steps they took to change the data (i.e., these steps should be reproducible to produce the same result)

The OpenSpending team has gone to lengths to retain enough information to say who was responsible for both of the above.

OpenSpending is a system, somewhat like a wiki, which allows you to track back through the data wrangling process and work out what changes were made to the data, when and by whom.

ERROR REPORTING IN PRACTICE

OpenSpending recently received a pointed inquiry from the U.K. Treasury disputing the claims we were making about the payment of British public money to a private company. Believing that an error had been introduced, we attempted to retrace our steps and find out where this had occurred, and who was responsible.

As we discovered, the payment had actually taken place, but the the OpenSpending descriptions used to label the transaction were not sufficiently detailed to accurately reflect the item in question.

With Spending Stories, we were able to retrace our steps because we had preserved a copy of the software tools we used for collecting the data (the data is published by about 50 public bodies, and must be downloaded, stitched together, and firmly molded into shape). These tools had been also made available to the public, so the Treasury and other concerned citizens could have checked our work themselves; the availability of this kind of check keeps all participants in the fiscal debate honest.

What had gone wrong was a problem of terminology: The transactions existed, but ambiguous language had been used to describe them, glossing over the distinction between the government department reporting what money had been spent and the government agency which actually spent the money. The bodies in question were the Department of Health and a regional health care trust; this distinction is certainly one which a concerned citizen would expect to be made clearly -- so we should make sure our system makes it easy to know which question is being asked.

CHECKPOINTS IN OPENSPENDING

In the short term, we are mitigating the problem of data errors as follows:

• Data provenance - is the source identifiable and the process reproducible? OpenSpending encourages people to add modified datasets to a "package" in the Data Hub. This allows other users to see the original document alongside any modified documents and track the chain of changes made to see clearly which points errors could have been introduced.
• Crowdsourcing feedback on spending data.
• Permitting re-use of the structured data we present, so that it can inform decisions in other fact-checking systems.

Ultimately, we will build our part of the ecosystem to provide feedback to the political process, by improving democratic discourse about the public finances.

Lucy Chambers is a community coordinator at the Open Knowledge Foundation. She works on the OKF's OpenSpending project and coordinates the data-driven-journalism activities of the foundation, including running training sessions and helping to streamline the production of a collaboratively written handbook for data journalists.

Martin Keegan is a software engineer and linguist, currently leading the Open Knowledge Foundation's OpenSpending project. He is also on the Open Knowledge Foundation's board, and has worked for SRI, Citrix, University of Cambridge and co-founded and worked for various civil society organizations.

The LRA Crisis Tracker is a real-time crisis mapping platform and data collection system, which makes information about the attacks and movement of the Lord’s Resistance Army (LRA) in Africa publicly available. The LRA is a rebel group held responsible for Africa’s longest-running armed conflict. The LRA operates in Uganda, Sudan, the Democratic Republic of Congo and the Central African Republic, and is accused of violence against civilians and forcing children to participate in hostilities.

The tool is produced by Invisible Children, an international NGO working to assist communities in LRA-affected areas of Central Africa, and Resolve, a Washington D.C.-based NGO advocating for an end to atrocities perpetrated by the LRA. It combines empirical data with storytelling, visually engaging data representation and narrative-rich media.

Why is the LRA Crisis Tracker essential?

Incidents taking place in Africa can easily go ignored when they happen in remote areas where communication is nearly impossible. According to a press release of the creators of this platform, four out of five LRA incidents have never been reported by any Western or regional news source in spite of the fact that LRA attacks on civilians occur on average every 21 hours.

The project is aimed both at informing local communities through an early-warning radio network about the activities of the LRA as well as informing the organisations that support the affected communities in order to reduce their response time to incidents. For the media the platform is a source of real-time information. Journalists can sign up with their email address to receive breaking news on the conflict.

Michael Poffenberger, Executive Director of Resolve, explains: 'Not only is this a pioneering tool for activists and policymakers, but community-run protection organisations in Central Africa will directly benefit from regular reports analysing LRA movement and attack patterns. The response time to LRA atrocities should be three hours, not three months.'

How does it work?

The LRA Crisis Tracker consists of a digital map, a breaking news feed, biweekly data-analysis reports and a mobile application. A plotted map shows incidents by type on a timeline. 

Screenshot of the LRA Crisis Tracker showing data from the beginning of 2011

With the timeline the movement of the LRA can be followed. Below the map there is a real time feed listing the most recent news. Biweekly reports are compiled on the LRA activities.

You can learn more about how the map and the database were produced in the project documentation released by the creators of this platform. A part of the raw data fed into the platform is available for public download under the Open Data Common’s Open Database License.

Explore the LRA Crisis Tracker at LRACrisisTracker.com.

Have you used the LRA Crisis Tracker in your work? Let us know your experience with it by leaving a comment.

According to Farida Vis, who initiated the project, 'growing your own has exploded in popularity in recent years and waiting lists for council allotments are soaring. But finding out how exactly long allotment waiting lists are everywhere in the country is a mammoth data task.'

Vis points out that the data was obtained from several sources, namely, the Transition Town West Kirby, which has been collecting allotment information for three years on a national basis, as well as hundreds of Freedom of Information requests sent to the different councils. This data includes the number of available lots, the number of people on the waiting lists, as well as the prices in 2008 and 2011. Other information was obtained in the form of tenancy agreements.

Sreenshot of interactive map of allotment waiting lists

‘Although it is tempting to simply highlight this problem in a different way, with additional data and accompanying visualisations, I was keen to highlight that whilst I do think there is an issue with councils not providing more sites, it is also clear to me that they are not exactly in a position to necessarily do so given the current economic climate,’ says Vis.

Read the full article on the Guardian DataBlog and explore the data yourself through the interactive map.

"If you want to understand how carbon footprints are affected by international trade flows, the paper argues, you need to consider trade not only in gadgets and garments but also in fossil fuels themselves. After all, though country X might import a television that was made in country Y, it's quite possible that country Y in turn imported some of the coal, oil or gas consumed by the television factory from country Z.

Of course, there's nothing revelatory in the idea that fossil fuels are traded between nations. We all know that, say, Saudi Arabia produces much of the world's oil. But what the academics behind the new data have done is a remarkable feat of number crunching: they've tracked the carbon flows of virtually the whole world, from the countries extracting the oil, gas and coal via the countries in which it's burned to the countries that ultimately consume the goods and services all that energy is used to create."

The article includes an interactive chart that allows us to explore the world's carbon footprint from extraction of fossil fuels and emissions of CO2 through to consumption of goods and services. The chart "Extraction to Consumption" shows the ultimate supplier of the fuels used to support the lifestyles in each country:

"Extraction to Consumption" chart. The Guardian Environment Blog

The interactive chart is based on the interactive data tool "The Supply Chain of CO2 Emissions" built by researchers at the Carnegie Institution for Science. 

Read the full article on the Guardian Environment Blog.

Teaser image credits: Jonathan W Smith. 

The Guardian Data Blog: The future of UK aid - interactive

Some of the trends highlighted in the analysis:

"Health and education remain among the top areas for UK aid. But when you look within health, for example, interesting trends become apparent. By 2015, reproductive, maternal and newborn health will consume nearly 64% of DfID's global health funding, while the share of funding for HIV and Aids will drop from 16% in 2010-11 to under 6% in 2014-15. In the Caribbean, which has the world's highest prevalence rate, the UK will not fund any further work on HIV and Aids, with the justification that this sector is 'well served by other donors'."

The interactive is built on data from the Department for International Development (DfID) and uses the BubbleTree code from the Open Knowledge Foundation and OpenSpending.org.

To get the data and read about how the Guardian analysed it visit the Guardian Data Blog.

—

The internet is bursting with information, but journalists – at least in The Netherlands – don’t get the full potential out of it. Basic questions on what data driven journalism is, and how to practise it, still have to be answered. Two Dutch regional newspapers (de Stentor and TC Tubantia) have launched RegioHack, an experiment with data driven journalism around local issues and open data.

Both newspapers circulate in the eastern and middle part of the Netherlands. In November, journalists will collaborate with local students, programmers and open data experts in a 30 hour coding event. In preparation for this hackathon, the forum on our website (www.regiohack.nl) is opened for discussion. Anyone can start a thread for a specific problem. For example, what’s the average age of each town in our region? And in 10 years, do we have enough facilities to accommodate the future population? And if not, what do we need?

The newspapers provide the participants with hot pizza, energy drink and 30 hours to find, clean up and present the data on these subjects.

After the hackathon, the projects are presented and participants will be named in the publications. That’s what RegioHack is all about: making unique stories with data, helping each other to develop new skills and finding out how to practise data driven journalism.

If you happen to be in The Netherlands on November 10th and 11th, contact me on jerry@regiohack.nl or Twitter (@JerryVermanen) for an invite to the final presentation.

We’re also searching for guest bloggers – and yes, that can be in English.

We're currently organising a session on 'Data-Driven Journalism in a Box' at the Mozilla Festival 2011, and we want your input!

In particular:

• What skills and tools are needed for data-driven journalism?
• What is missing from existing tools and documentation?

If you're interested in the idea, please come and say hello on our data-driven-journalism mailing list!

Following is a brief outline of our plans so far...

## What is it?

The last decade has seen an explosion of publicly available data sources - from government databases, to data from NGOs and companies, to large collections of newsworthy documents. There is an increasing pressure for journalists to be equipped with tools and skills to be able to bring value from these data sources to the newsroom and to their readers.

But where can you start? How do you know what tools are available, and what those tools are capable of? How can you harness external expertise to help to make sense of complex or esoteric data sources? How can you take data-driven journalism into your own hands and explore this promising, yet often daunting, new field?

A group of journalists, developers, and data geeks want to compile a 'Data-Driven Journalism In A Box', a user-friendly kit that includes the most essential tools and tips for data. What is needed to find, clean, sort, create, and visualize data -- and ultimately produce a story out of data?

There are many tools and resources already out there, but we want to bring them together into one easy-to-use, neatly packaged kit, specifically catered to the needs of  journalists and news organisations. We also want to draw attention to missing pieces and encourage sprints to fill in the gaps as well as tighten documentation.

## What's needed in the Box?

Introduction

• What is data?
• What is data-driven journalism?
• Different approaches: Journalist coders vs. Teams of hacks & hackers vs. Geeks for hire
• Investigative journalism vs. online eye candy
• Understanding/interpreting data
• Analysis: resources on statistics, etc.
• Visualization tools & guidelines - Tufte 101, bubbles or graphs?

Acquiring data

• Guide to data sources
• Methods for collecting  your own data
• FOI / open data
• Scraping

Working with data

• Guide to tools for non-technical people
• Cleaning
• Publishing data
• Rights clearance
• How to publish data openly
• Feedback loop on correcting, annotating, adding to data
• How to integrate a data story with existing content management systems

## What bits are already out there?

• Open Data Manual
• Data Patterns
• Data Journalism Manual
• List(s) of Free/Open Tools for working with data
• Data Journalism Developer Studio

## What bits are missing?

• Tools that are shaped to newsroom use
• Guide to browser plugins
• Guide to web-based tools

## Opportunities with Data-Driven Journalism:

• Reduce costs and time by building on existing data sources, tools, and expertise.
• Harness external expertise more effectively.
• Towards more trust and accountability of journalistic outputs by publishing supporting data with stories. Towards a "scientific journalism" approach that appreciates transparent, empirically- backed sources.
• News outlets can find their own story leads rather than relying on press releases.
• Increased autonomy when journalists can produce their own datasets.
• Local media can better shape and inform media campaigns. Information can be tailored to local audiences (hyperlocal journalism).
• Increase traffic by making sense of complex stories with visuals.
• Interactive data visualizations allow users to see the big picture & zoom in to find information relevant to them.
• Improved literacy. Better understanding of statistics, datasets, how data is obtained & presented.
• Towards employable skills.

Image credits: Flickr user okfn.

—

Since March, a crowdsourcing experiment collects the price of tap water throughout France. In just 4 months, over 5,000 people fed up with corporate control of the water market took the time to look for their water utility bill, scan it and upload it on prixdeleau.fr. The result is an unprecedented investigation that brought geeks, NGO and traditional media together and improved transparency.

The French water utility market consists in over 10,000 customers (cities buying water to distribute to their taxpayers) and just a handful of utility companies. The balance of power on this oligopoly is distorted in favor of the corporations, which sometimes charge different prices to neighboring towns!

The French NGO France Libertés has been dealing with water issues worldwide for the past 25 years. It now focuses on improving transparency on the French market and empowering citizens and mayors, who negotiate water utility deals. The French government decided to tackle the problem 2 years ago with a nationwide census of water price and quality. So far, only 3% of the data has been collected. To go faster, France Libertés wanted to get citizens directly involved.

Together with the OWNI team, I designed a crowdsourcing interface where users would scan their water utility bill and enter the price they paid for tap water on prixdeleau.fr (price of water). In the past 4 months, 8,500 signed up and over 5,000 bills have been uploaded and validated.

While this does not allow for a perfect assessment of the market situation, it showed stakeholders such as national water overseeing bodies that there was a genuine, grassroots concern about the price of tap water. They were skeptical at first about transparency, but changed their minds over the course of the operation, progressively joining France Libertés in its fight against opacity and corporate malpractice.

What’s there to learn for media organizations?

1. Partner with NGOs. NGOs need large amount of data to design policy papers. They will be more willing to pay for a data collection operation than a newspaper exec.
2. Users can provide raw data. Crowdsourcing works best when users do a data collection or data refining task.
3. Ask for the source. We pondered whether to ask users for a scan of the original bill, thinking it would deter some of them (especially as our target audience was older than average). While it might have put off some, it increased the credibility of the data.
4. Set up a validation mechanism. We designed a point system and a peer-review mechanism to vet user contributions. This proved too convoluted for users, who had little incentive to make repeated visits to the website. It was used by the France Libertés team, however, whose 10 or so employees did feel motivated by the points system.
5. Keep it simple. We built an automated mailing mechanism so that users could file a Freedom of Information request regarding water pricing in just a few clicks. Though innovative and well-designed, this feature did not provide substantial ROI (only 100 requests have been sent).
6. Target your audience. France Libertés partnered with consumers’ rights news magazine 60 Millions de Consommateurs, who got their community involved in a big way. It was the perfect match for such an operation.
7. Choose your KPI carefully. The project gathered only 45,000 visitors in 4 months, equivalent to 15 minutes worth of traffic on nytimes.com. What’s really important is that 1 in 5 signed up and 1 in 10 took the time to scan and upload his or her utility bill.

—

More about Nicolas Kayser-Bril at his website.