The hacking was punctuated by talks by Tom Lee of the Sunlight Foundation and Jake Porway of Data Without Borders.

Projects

Dan Nguyen scraped Florida mugshots from and used face.com's API to analyse each photo to tell you the arrestee's mood.

Michael Keller, Marc Georges et al. related the NYPD stop, question and frisk data to nine mosques referenced in an NYPD report on surveillance in order to see whether there had been unusual changes in stopping activity around these mosques.

The dataset is insanely messy, but they fortunately had access to a relatively clean version that Data Without Bordes had developed in November.

They were still going strong after the data camp. Refusing to leave, they moved to a different room after getting kicked out of the data camp space.

Mike Caprio and team cleaned a spreadsheet of 80,000 records from the New York lobbiest website to power a site on New York lobbyists based on the Chicago Lobbyists site. It appears that $120 million was spent on New York on lobbiests in 2011.

I helped one team relate contracts from Open Book New York to data that they had scraped by hand (!) from hand-written forms in order to identify pontential conflicts of interest.

I helped another team identify potential stories (outliers) in the NYC Open Data graffiti locations dataset.

Susan McGregor was "clearly hooked" because she liberated lobbyist contract details the next evening instead of watching the Superbowl.

Technical Awards  

Mike Caprio won *Best Data Liberator* for liberating the Iowa accident reports database.

Michelle Koeth won *Best Creation of an API* for scraping New York, NY hospitals from Medicare Hospital Compare.

Jeremy Baron, from UN peacekeeping team, won *Best Use of ScraperWiki* for scraping United Nations PDFs. This team also scraped peacekeeping statistics and contributions.

Honorary ScraperWikian  

Susan McGregor was awarded "Honorary ScraperWikian". We haven't decided what that means yet. 

Learning

Teaching the Learn to Scrape sessions and working with many of the project teams, I got the impression that we had opened participants to thinking more about how data can be scraped, transformed and analyzed to identify unusual subsets and potential stories. Split-applied-combined, if you will.

Our 'Learn to Scrape' sessions seemed to work as well; I found several participants who had claimed no knowledge of webscraping prior to the sessions to be creating reasonably complex scrapers by the next afternoon.

What Next?

More data camps are coming up, and several groups plan on contining to work on their projects. But in the mean time, we now have lots of data for you to analyse!

Where can I find data and how can I request access to it? What tools are available to me? How can I find useful stories within datasets? And, most importantly, how can I make a living through the practise of data journalism?

Do these questions sound familiar? They should, because these are all questions that any data journalist, who is just getting started, should be asking.

Leading data journalists from the New York Times, the Guardian, the BBC, and other top media organisations from around the world, are working together to answer all of these questions (and more) in the Data Journalism Handbook. The handbook is the first comprehensive practical guide to data journalism. As Aron Pilhofer, editor of Interactive News at the New York Times, noted at the beginning of the project: “A project like this is quite necessary. It’s kind of surprising that someone hasn’t tried to do this until now.”

Interested in contributing? We want YOU!

Work on the handbook was kick-started in November of last year at the Mozilla Festival in London. In just two days, 55 contributors drafted 60 pages (20,000 words) for six chapters. The handbook is a community project; therefore, anyone who has experience in data journalism can help draft the book. The work, in turn, goes back to the community as anyone is allowed to freely use, modify, adapt, and reuse the handbook.

Contributions have been flooding in and since Mozilla we have been editing, updating, filling in gaps, and restructuring. We are very close to a first complete draft of the book and we need your help to get there. We are looking for authors, editors and peer reviewers to draft chapters, review content, style and accuracy of the book.

In the table of contents below you can see the progress that has been made and where input is needed:

• 0. Preface (in progress)
  • 0.1 The purpose of this book
  • 0.2 Add to this book
  • 0.3 Share this book
• 1. Introduction (done)
  • 1.1 What is data journalism?
  • 1.2 Why is it important?
• 2. Introducing data journalism in the newsroom
  • 2.1 Changes in the newsroom (contributors needed)
  • 2.2 How is it done: journo-developers vs. coders for hire (in progress)
• 3. Types of outcomes/projects and case studies
  • 3.1 Data powered stories (in progress)
  • 3.2 Data served with stories (in progress)
  • 3.3 Data driven applications (contributors needed)
• 4. Working on the data story
  • 4.1. Step 1: Getting your data
    • 4.1.1 Where does data live? (in progress)
    • 4.1.2 Asking for data (in progress)
    • 4.1.3 Getting your own data (in progress)
    • 4.1.4 Crowdsourcing data (contributors needed)
  • 4.2 Step 2: Understanding your data
    • 4.2.1 Data literacy (done)
    • 4.2.2 Working with data tips (in progress)
    • 4.2.3 Tools and techniques for analysing data (contributors needed)
    • 4.2.4 Harnessing expert opinions: Annotating datasets (contributors needed)
  • 4.3 Step 3: Finding a story in your data (in progress)
    • 4.3.1  From datasets to stories - approaches 
  • 4.4 Step 4: Delivering your data project (contributors needed)
    • 4.4.1 Serving data with stories 
    • 4.4.2 Visualising data 
    • 4.4.3 Data driven applications 
• 5. Engagement, outreach and community (contributors needed)

• 6. How to make data journalism sustainable
  • 6.1 Measuring impact (contributors needed)
  • 6.2 Business models (contributors needed)
• 7. Appendix (in progress)
  • 7.1 Further resources 
  • 7.2 Glossary

Whether you are a budding data journalist putting the manual through its paces or someone with years of experience, we'd love your help to fine-tune the book and get it ready for the press (no pun intended).

If you're interested to contribute to the first Data Journalism Handbook please fill in this form as soon as possible indicating your level and area of expertise and the chapter that you would like to contribute to. We will get in touch with you as soon as we can once you've submitted the form.

Deadline

A first complete draft is planned to be ready by the end of February.

The official launch of the The Data Journalism Handbook will be at the International Journalism Festival in Perugia from 25-29 April. The book will be available online as an e-book. Participants at the festival will have the opportunity to buy a printed copy of the book and enjoy a meet and greet with the authors where they may exchange knowledge and learn helpful tips about how to successfully become data journalists. 

Questions?

For questions get in touch with the Data Journalism Handbook coordinators: bounegru@ejc.net (European Journalism Centre) or lucy.chambers@okfn.org (Open Knowledge Foundation).

Image by Kate Hudson.

Following an opening talk on upcoming developments from Wikimedia, such as Wikihadoop, five presentations were given; each covering a specific group or project concerned with the use or pursuit of open data.

First, Velichka Dimitrova introduced the Open Economics working group, which is seeking to break down the barriers between members of the public and the economic data which is increasingly influential in their daily lives. The group is currently focusing on a project aimed at modelling social progress in Italy, which it hopes to enter into the Apps4Italy competition. Ideas for future tasks were also discussed, of which two of the most interesting were an analysis of ‘author cartels’ within the economic journal publishing industry, and the proposal of an ‘open data accreditation badge’ which could be awarded to organisations and companies who seek to make open to the public any data they hold.

Next, Julian Tate, of Open Data Manchester, gave a fascinating glimpse of his group’s ongoing analysis of public transport data in the city. Somewhat perversely, competing bus companies currently refuse to publish fare information online, fearing that rivals will undercut them by tiny margins and steal their customers. According to Julian’s research, thousands of potential commuters are put off using the service because of the lack of transparency, and far more would use public transport if prices for the various journeys were openly available. Julian and his team are now attempting to do what Transport for Greater Manchester have as yet been unable to do, and compile - through a mixture of crowdsourcing and pleading with bus companies - a comprehensive and open list of fares for journeys throughout the region.

Kevin Carter then introduced his venture; the ingeniously titled Landscape Portrait. The project aims to blend art and data to produce what is effectively an ‘identity map’. The idea is that individuals provide answers to a number of questions about their aspirations, feelings and concerns, and these are then compared with statistically-generated regional stereotypes. The result is an intriguing spatial patchwork of social outlooks and aspirations.

Jo Pugh then announced a hack day at the National Archives, which will take place on 24/25 March at the Archives in Kew. Pugh described how the very existence of the Archives in only justified if the records held there are examined and shared - precisely what the hack day is intended for. Anyone who has an idea for making use of the records - some of which stretch back for centuries  - should come along.

Finally, Keiichi Matsuda outlined his project; ‘Prism: a Sculptural aggregator of live data in London’. While the name might be something of a handful, the idea itself is beautifully simplistic. Matsuda is creating an exhibition for the London Design Festival where he plans to project interesting London-centric statistics onto a series of angular sculptures in what he refers to as a display of “urban informatics”.

The Eurozone crisis has dominated headlines across the continent in recent months, and will not be leaving the spotlight any time soon. Along with Greece, Italy has been one of the biggest causes for concern, and the Italian economy provided the backdrop for the day’s main focus.

Following the success of Yourtopia, Open Economics’ main project to date, the central aim of Saturday’s activities was to develop an application to map social progress in Italy over time and across its 21 administrative regions.

Over the years, different bodies have come up with different methods of measuring social change, from the qualitative - public surveys - to the quantitative - unemployment statistics and income distribution - but none can be considered sufficiently holistic to provide a fully representative overview of the state of affairs.

This new project, Yourtopia 2, will take a wide variety of measures and combine them into a broad model, with the hope of providing an unprecedented analysis of socioeconomic welfare which can then be applied to other countries and regions. All being well, the final application will be submitted as an entry to the Apps4Italy competition.

As with Yourtopia, the aim for Yourtopia 2 is to end up with an interactive tool which users can customize depending on which factors they consider most important to achieving social progress. As a result, Saturday’s focus was on gathering data for the different variables, beginning design of the user interface, and carrying out further research into how the eventual model will be built.

In one corner, a handful of data researchers scoured the web for sources of regional unemployment data and Gini coefficients, in another, coders and designers worked tirelessly to iron out glitches in the model and create a slick user interface, while elsewhere academics debated which factors the model should incorporate in order to be most effective.

The project is ongoing. Anyone wishing to contribute in some way had the possibility to take part in the online follow-up session last Saturday. Should you wish to take part in future meetings keep an eye on the Open Economics website. 

Photo by Ilias Bartolini. 

In the past investigative reporters would suffer from a scarcity of information relating to questions they were trying to answer. While this is still the case, today journalists are also faced with an overwhelming abundance of data. In an age of information overload, to stay relevant to society journalists need to learn to separate signal from noise in order to provide valuable insights. Journalists need to be equipped with knowledge of the tools, techniques and tactics of working with data in order to derive maximum value from for their readers.

The European Journalism Centre and the Open Knowledge Foundation are pleased to invite you to the School of Data Journalism hosted at the sixth edition of Italy's leading journalism event, the International Journalism Festival. The 2012 edition takes place in the beautiful city of Perugia between 25-29 April. Entry to the School of Data Journalism panels and workshops is free. Each workshop has a limited number of places and therefore registration will be necessary. Please note that not all requests to participate in the workshops will be accepted.

What is the School of Data Journalism and who is it for?

The School consists of three panel discussions and five workshops.

The panels attempt to provide answers to crucial questions for aspiring data journalists, editors and decision-makers in newsrooms:

• What can aspiring data journalists learn from the successes of the past?
• How can data journalism save your newsroom?
• How do you start a data journalism operation?
• How can you become a data journalist and what do you need to do?

In the workshops journalists who are interested to get started with reporting with data and budding data journalists will learn from experienced data journalists and open data experts essential skills related to how to get the data you need, how to analyse it, how to get stories from data and how to present your stories.

Panels

Panel 1: News and numbers: from CAR to data journalism (Thursday, 26 April)

Journalists have always used data and numbers to produce stories…and win Pulitzers. From Philip Meyer’s coverage of the Detroit riots in 1967 to Steve Doig’s 'What Went Wrong' analysis of the damage patterns from Hurricane Andrew, data-driven reporting has brought valuable public service and won journalists recognition and prizes.

Whereas there may be distinguishing aspects about the data journalism of today and the computer-assisted reporting of the past, it is crucial to learn from successful examples, techniques and approaches of the past.

• How can we bring the data journalism community and the CAR community closer together?
• What can aspiring data journalists learn from the successes of the past?
• What is the future of data journalism?

Panel 2: How can data journalism save your newsroom? (Friday, 27 April)

Journalism is under siege. Traditional models are collapsing. Developing the know-how to use the available data more effectively, to understand it, communicate and generate stories based on it, could be a huge opportunity to breathe new life into journalism.

• What is the potential of data journalism?
• How do you start a data journalism operation?
• How can you start thinking about making money with data journalism?

Panel 3: You Too Can Be A Data Journalist! (Saturday, 28 April)

Where can I find data? How can I request data? What tools can I use? How can I find stories in data? How can I make money with data journalism?

Several leading data journalists, CAR specialists and journalism professors from the Guardian, the New York Times, Financial Times, Chicago Tribune, Medill School of Journalism and Cronkite School of Journalism worked together to answer these questions in the Data Journalism Handbook.

The handbook, the first comprehensive practical guide to data journalism, will be officially launched in this session. The session will provide the opportunity to meet and greet authors of the book, exchange knowledge and learn from them what you need to know to be a data journalist, as well as get a printed copy of the book.

Speakers

• Caelainn Barr, EU data journalist, formerly with the Bureau of Investigative Journalism in London
• Heather Brooke, journalist and FOI campaigner
• Sarah Cohen, Knight Professor of the Practice of Journalism and Public Policy, DeWitt Wallace Center for Media and Democracy, Duke University (Pulitzer prize winner)
• Steve Doig, Knight Chair in Journalism, Walter Cronkite School of Journalism, Arizona State University (Pulitzer prize winner)
• Mirko Lorenz, data journalism trainer, Deutsche Welle
• Aron Pilhofer, editor of Interactive News at The New York Times and co-founder ofDocumentCloud.org
• Dan Nguyen, news application developer, ProPublica
• Simon Rogers, editor of the Guardian Data Blog

Workshops

Workshop 1: Scraping data & cracking PDFs (Thursday, 26 April)

Hands up who knows what machine-readable data is? You will soon, and more importantly, how to get it and what you can do with it once you have it!

A workshop targeted at scraping from scratch, including:

• What PDFs and webpages look like to your computer - An introduction to machine-readable / non-machine readable data
• The Scraper Cookbook - an overview of the key things you need to know to write a scraper
• Hands on session - learning to screen-scrape. Main focus: hands on session using tools such as ScraperWiki. If sufficient interest & time, we will also touch on some of the tools & skills needed to extract data from PDFs.
• Error checking - how to check what you have makes sense, spotting the types of errors sometimes introduced if you don’t get it quite right! 

Workshop 2: Information wants to be free - Freedom of information requests and how to use them (Friday, 27 April)

Freedom of information requests are constantly evolving. Law changes and technological advancements make it increasingly easier to file and systematise FOI requests, and importantly track their progress through the system. This workshop includes demonstrations and case studies examining the current state-of-play with FOI requests in Europe and beyond and looking into what’s next for the freedom of information movement. 

Workshop 3: Making data pretty (Friday, 27 April)

Journalism is no longer just a block of prose on a page. The modern reader often demands maps, infographics and visualisations to make the story jump out at them, particularly in digital environments.

There are a vast array of free tools available on the web to allow data-journalists to quickly and easily digest, process and display the data powering their stories. This workshop aims to give a good overview of what is currently available and delve into depth on one of the most powerful: Google Fusion Tables.

Workshop 4: Getting Stories from Data (Saturday, 28 April)

Enormous datasets can often prove extremely daunting to the unfamiliar. Mistakes and crimes have historically benefited from, and triumphs and good decisions been obscured by, a mask of bewildering numbers and statistics and gone unreported.

Large datasets often hold a wealth of undiscovered stories for those willing to invest the time into exploring them. This workshop is a ‘spotters’-guide’ for things to look out for and where to look for datasets.

Workshop 5: Spending Stories (Sunday, 29 April)

To get to the bottom of a story, you need only to 'follow the money'. The same is true of government: budgeting is where policies and priorities are broken down into figures. Financial programming has a direct influence on all political areas: while other data on health or social help us understand what challenges society faces, looking at spending data allows us to see how government reacts to all of these.

There are many spending databases available on the web, some impenetrable, some accessible for analysis. We’ll show how we enable journalists and researchers to make sense of the data and what strategies can be used to investigate stories and policies.

• Overview: What datasets are out there? Where can you look for more?
• Interrogating databases, how to extract the maximum amount of data out of tricky databases
• Tools for spending analysis, how to slice and dice once you have your data

Workshop leaders

• Caelainn Barr, formerly with the Bureau of Investigative Journalism, working on the award-winning Structural Funds investigation
• Lucy Chambers, Open Knowledge Foundation, Community Coordinator for OpenSpending and the ‘Spending Stories’ project.
• Steve Doig, Knight Chair in Journalism, Cronkite School of Journalism, Arizona State University (Pulitzer prize winner)
• Friedrich Lindenberg, Open Knowledge Foundation Deutschland, Developer on OpenSpending
• Dan Nguyen, news application developer, ProPublica
• Simon Rogers, editor of the Guardian Data Blog

When and where?

The Data Journalism School takes place at the International Journalism Festival in Perugia between 25 and 29 April 2012. The schedule of the Data Journalism School, with confirmed speakers for each panel and workshop, will be posted on the festival website in early February.

How to register?

Entry to the festival and the School are free. There is no registration process to attend the festival. For the workshops there is a limited number of available seats. To secure a seat in the workshops please register via this form. The deadline for workshop registration is 20 March 2012. You will be notified by email by 25 March at the latest if we were able to confirm you a seat. The workshops are entry-level. Consideration will be given to your experience, skills and motivation to attend the workshop when making the selection.

What do you need to bring?

A lot of enthusiasm and a laptop are required for the workshop sessions. Please note for hands-on workshops tablet PC's will not be appropriate.

Questions?

If you have questions about the School of Data Journalism get in touch with the coordinators: bounegru@ejc.net or lucy.chambers@okfn.org.

I’ve been working for some time on picking apart the many processes which make up what we call data journalism. Indeed, if you read the chapter on data journalism (blogged draft) in my Online Journalism Handbook, or seen me speak on the subject, you’ll have seen my previous diagram that tries to explain those processes.

I’ve now revised that considerably, and what I’ve come up with bears some explanation. I’ve cheekily called it the inverted pyramid of data journalism, partly because it begins with a large amount of information which becomes increasingly focused as you drill down into it until you reach the point of communicating the results.

What’s more, I’ve also sketched out a second diagram that breaks down how data journalism stories are communicated – an area which I think has so far not been very widely explored. But that’s for a future post.

I’m hoping this will be helpful to those trying to get to grips with data, whether as journalists, developers or designers. This is, as always, work in progress so let me know if you think I’ve missed anything or if things might be better explained.

UPDATE: Also in Spanish.

The inverted pyramid of data journalism

Here are the stages explained:

Compile

Data journalism begins in one of two ways: either you have a question that needs data, or a dataset that needs questioning. Whichever it is, the compilation of data is what defines it as an act of data journalism.

Compiling data can take various forms. At its most simple the data might be:

1. supplied directly to you by an organisation (how long until we see ‘data releases’ alongside press releases?),
2. found through using advanced search techniques to plough into the depths of government websites;
3. compiled by scraping databases hidden behind online forms or pages of results using tools like OutWit Hub and Scraperwiki;
4. by converting documents into something that can be analysed, using tools like DocumentCloud;
5. by pulling information from APIs;
6. or by collecting the data yourself through observation, surveys, online forms or crowdsourcing.

This compilation stage is the most important – not only because everything else rests on that, but because it is probably the stage that is returned to the most – at each of the subsequent stages – cleaning, contextualising, combining and communicating – it may be that you need to compile further information.

Clean

Having data is just the beginning. Being confident in the stories hidden within it means being able to trust the quality of the data – and that means cleaning it.

Cleaning typically takes two forms: removing human error; and converting the data into a format that is consistent with other data you are using.

For example, datasets will often include some or all of the following: duplicate entries; empty entries; the use of default values to save time or where no information was held; incorrect formatting (e.g. words instead of numbers); corrupted entries or entries with HTML code; multiple names for the same thing (e.g. BBC and B.B.C. and British Broadcasting Corporation); and missing data (e.g. constituency). You can probably suggest others.

There are simple ways to clean up data in Excel or Google Docs such as find and replace, sorting to find unusually high, low, or empty entries, and using filters so that only duplicate entries (i.e. those where a piece of data occurs more than once) are shown.

Google Refine adds a lot more power: its ‘common transforms’ function will, for example, convert all entries to lowercase, uppercase or titlecase. It can remove HTML, remove spaces before and after entries (which you can’t see but which computers will see as different to the same data without a space), remove double spaces, join and split cells, and format them consistently. It will also ‘cluster’ entries and allow you to merge those which should be the same. Note: this will work for BBC and B.B.C. but not BBC and British Broadcasting Corporation, so some manual intervention is often needed.

Context

Like any source, data cannot always be trusted. It comes with its own histories, biases, and objectives. So like any source, you need to ask questions of it: who gathered it, when, and for what purpose? How was it gathered? (The methodology). What exactly do they mean by that?

You will also need to understand jargon, such as codes that represent categories, classifications or locations, and specialist terminology.

All the above will most likely lead you to compile further data. For example, knowing the number of crimes in a city is interesting, but only becomes meaningful when you contextualise that alongside the population, or the numbers of police, or the levels of crime 5 years ago, or perceptions of crime, or levels of unemployment, and so on. Statistical literacy is a must here – or at least show your work to someone who has read Ben Goldacre’s book.

Having a clear question at the start of the whole process, by the way, helps ensure you don’t lose your focus at this point, or miss an interesting angle.

Combine

Good stories can be found in a single dataset, but often you will need to combine two together. After all, given the choice between a single-source story and a multiple-source one, which would you prefer?

The classic combination is the maps mashup: taking one dataset and combining it with map data to provide an instant visualisation of how something is distributed in space: where are the cuts hitting hardest? Which schools are performing best? What are the most talked-about topics around the world on Twitter right now?

This is so common (largely because the Google Maps API was one of the first journalistically useful APIs) it has almost become a cliche. But still, cliches are often – if not always – effective.

A more mundane combination is to combine two or more datasets with a common data point. That might be a politican’s name, for example, or a school, or a location.

This often means ensuring that the particular data point is formatted in the same name across each dataset.

In one, for example, the first and last names might have separate columns, but not in the other (you can concatenate or split cells to solve this).

Or you might have local authority names in one, but local authority codes in another (find another dataset that has both together and use a tool like Google Fusion Tables to merge them).

One might use latitude and longitude; another postcodes, or easting and northing (a postcodes API and Google Refine can help). But once you’ve got them formatted right, you may find some interesting stories or leads for further questions to ask.

Communicate

In data journalism the all-too-obvious thing to do at this point is to visualise the results – on a map, in a chart, an infographic, or an animation. But there’s a lot more here to consider – from the classic narrative, to news apps, case studies and personalisation. In fact there’s so much in this stage alone that I’ve written a separate post (diagram below). Meanwhile, comments very much welcome.

Journalists and media organisations have increasingly been using data to improve reportage of complex issues and to give readers direct access to the sources behind the headlines. From the Guardian to the New York Times, La Stampa to Die Zeit, journalists and media organisations are experimenting with new ways of sourcing, delivering and enabling readers and users to engage with news. 

The DJA is the first international contest recognizing outstanding work in this field. It is organised by the Global Editors Network in collaboration with the EJC, and is sponsored by Google.

A jury of data journalism experts and editors from all over the world, including from prestigious organisations like New York Times, Reuters, and Les Echos will award a total of €45,000 to six winners. The jury is headed by Paul Steiger, founder of ProPublica.

There are three award categories awarded at both (i) national and international and (ii) local and regional levels to give a total of six prizes. The three categories are:

1. Data-driven investigative journalism
2. Data visualisation & storytelling
3. Data-driven applications

 The work submitted must have been published or aired between 11 April 2011 and 10 April 2012.

How to apply?

As of yesterday, applicants are welcome to submit their best data journalism projects at http://datajournalismawards.org/submit-your-work/ The competition is open to media companies, non-profit organisations, freelancers and individuals until 10 April 2012. The six winning projects will be announced the annual News World Summit (NEWS!) hosted by the Global Editors Network, which will take place in Paris, France, from 30 May to 01 June 2012.

Find out more about the competition and how to apply at http://datajournalismawards.org/. Get in touch with the organisers by info@datajournalismawards.org, on Twitter using the hashtag #dja or by posting a message on the DJA Facebook page.

Image: EJC Director, Wilfried Ruetten, at DJA press conference, January 19, Google, London

• What is a news app?
• Who are news apps for?
• How do you decide on when to build an app?
• What should a good news app include?
• What are the successful and not so successful examples? What can be learned from them?
• How do you package apps?

Data Best Practices from StateImpact on Vimeo.

You can read more about data practices at NPR on the StateImpact blog.

As global newspaper audiences decline, some media organisations are latching onto emerging tools and open-source technologies to make news more relevant, personalised, and 'hyper-local'. Investigative journalism is being reinvented and reporters are doing their jobs in unimagined ways. 'Data journalism' is becoming the competitive differentiator for global news. 

When Gordon Patterson released South Africa's latest Audit Bureau of Circulation data, there was bad news and more bad news for most newspapers. Patterson expects 2012 to be the toughest year yet. Readerships are declining and fragmenting and, with disposable income levels plummeting in South Africa, consumer decision making was becoming illusive. As if that wasn’t enough to put the owners of Avusa, Napers’ Media24 and Independent in the doldrums, ABC’s circulation data showed more misery for the daily and weekend press.

The aggregated data showed circulation for daily newspapers dropped by 5.1% for the third quarter of 2011, when compared with the second quarter. The bloodletting was slowing marginally, but none-the-less figures were still headed downwards. Except for The Citizen, the worst pain was felt by English dailies, while Afrikaans dailies remained stable and vernacular press showed moderate growth.

Weekend newspapers like Rapport (down 7.33%), City Press (down 6.02%) and Sunday World (down 9.37) didn’t fare much better. Sunday Times also dropped, but at 0.4% this decline was negligible. isiZulu Ilanga Langesonto showed good growth with a circulation increase of 6.45%.

Weekly newspapers, on the other hand, were smiling. Mail &Guardian continued its growth trend and reported a pleasing 2.56% increase in circulation. Ilanga, the isiZulu weekly published by Independent, spiked by a massive 31.81%, underscoring the rich success of vernacular press.

As the likes of The Star, Sunday World and Sowetan face dwindling readerships, they could be asking themselves: “What are those other newspapers doing right that are we doing wrong?” The obvious answer in a sector where news has become a commodity is there’s little distinction between competitive papers and most of these media brands have lost relevance. As these newspapers flail, there’s a sea of emerging open-source technology freely available to media owners which could help hone content, increase relevance and add value to investigative reporting skills.

But most of the press aren’t even dipping their toes in that water, despite the great interest across the continent in tech tools that enable better journalism, particularly investigative and data journalism. Ghana recently wrapped up an open data and data journalism conference which was welcomed by the state which appealed to reporters to use data to hold public officials more accountable – even as South Africa stares into the barrel of the Secrecy Bill which would do precisely the opposite.

The rest of Africa’s interest in “technology for journalism” is, in part, being stimulated by Justin Arenstein, contracted by Google to help media owners build more robust business models and diversified revenue streams. Arenstein’s helping media improve the quality of content and by showing publishers how better to engage audiences using new technologies.

Justin Arenstein (courtesy of Justin Arenstein)

"A lot of what I'm doing with Google is based on the lessons learnt from the collapse of audiences and the advertising model in the US and Europe. Africa escaped much of that disruption three years ago because of our dismal Internet penetration stats. We are, however, now starting to see some of the same seismic changes in Africa's mature media markets in SA, Kenya, and Nigeria. The causes are different, driven by mobile and not Internet usage, but the trends and results are the same," says Arenstein.

Interestingly enough Arenstein was fired from his first media management job at a local newspaper, because he insisted on equal pay for black and white journalists. The former reporter has won awards, set up Mpumalanga’s first commercial radio station and helped establish a number of organisations to support investigative work on the continent. He also built African Eye News Service from the ground up to become southern Africa’s largest independent newswire service and the first rural-based wire agency on the sub-continent.

Recently Arenstein took a year off to explore “data journalism”, “civic engagement media” and “augmented reality” as a Knight Fellow at Stanford University. The sabbatical led to his gig with Google, which sees him earning more than his fair share of frequent-flyer miles.

“There’s a huge range of brilliant open-source and non-Google tools out there just waiting to be used, and we actively help people find them,” he says.

“When the scandal broke about dodgy expense claims by members of the UK parliament, journalists were swamped by almost 500,000 pages of evidence and receipts. Instead of being overwhelmed, The Guardian recruited - or crowdsourced - thousands of volunteers to help their journalists comb through and verify the evidence. Then, they used the new tools for data analysis and visualisation to extract examples, identify trends and to identify conflicts of interest in the data,” says Arenstein.

The Guardian - Data visualisation that illustrates where tax pounds are spent

An astounding 28,800 readers helped review almost 225,000 documents. By involving readers in the investigative phase of the story, and by granting access to the raw evidence, The Guardian built widespread public interest and momentum for a story that has run for well over a year. The media’s focus on the corruption resulted in dismissals, repayments, and a reform of financial management systems in the British parliament.

“But, these new tools are not just great for mobilising readers. They also help journalists do the kind of forensic analysis that we would previously have had to use consultants for. The Guardian's use of demographic data and mapping to compare the recent public rioting in British cities with poverty data is a good example. The results have helped move the public debate beyond recriminations, to a more informed discussion. Why aren't we seeing South African media doing something similar with census and other data, to analyse the ongoing service delivery protests?" he asks.

Arenstein concedes there are pitfalls. “Journalists need to be very clear about why they're using any new tool. Does it add real, tangible value?” he says.

The Guardian – Infographic of MP Expense Claim Scandal which explains how data was visualised

The real challenge is in making information relevant by augmenting narrative reporting with “utility news” tools. With the help of these readers can calibrate decisions on personal issues like where they live to who they vote for. “Every year, the South Africa media devote massive resources and coverage to the national budget, but 99% of the reportage is boring, impenetrable or just plain irrelevant to ordinary citizens,” he says.

The knee-jerk reaction from local journalists, he adds, is often that the data doesn't exist yet in Africa for this kind of journalism. Or that the technology is too expensive, or that audiences aren't ready. "This simply isn't true. South African journalists are sitting on a goldmine of census and other socio-economic data going back decades. We could easily be using property, education, and health information to build things like SinCountyRankings to help people compare lifestyle standards in their cities or neighbourhoods," Arenstein responds. "Even when the data isn't perfect, there are free and easy-to-use tools like ScraperWiki and Google Refine to scrape and crunch data, and scores of tools to then visualise or analyse connections between different sets of data."

A few exceptions aside, South African news media are ignoring the wealth and depth of information available to them because they don’t know how to analyse the data, and aren’t au fait with the tools. Mail & Guardian’s amaBhugane and Andrew Trench’s team at Media24 investigations are among the few local “data journalism” pioneers, and the results are telling. Late last year Trench, together with Jacques Pauw and Julian Rademeyer, broke the news of the ANC’s mining rights grab through its investment front, Chancellor House.

In an environment where so much news that embarrasses the ANC comes from informants, the City Press exposé on the party’s mine grab was a direct result of putting data mining tools to good use. “A couple of months back the department of mineral resources published records of all mining and prospecting right applications in South Africa over the last couple of years,” Trench writes on his blog “Notes from the revolution”.

“As is the habit of SA government departments, the records were published in PDF format which makes them pretty useless for any useful intensive examination of the data.” Fortunately Trench had taught himself Python, a relatively easy, but incredibly powerful programming language. Trench downloaded all the PDF files and created a Python script to enable rapid searches of the documents. Once relevant information was extracted the investigative journalist and his team could start getting a better handle on what was really going on.

When Trench and his team broke the story they reported that Chancellor House, through its Chancellor House Mineral Resources subsidiary, had scored prospecting rights to 560 sites in Northern Cape, North West and Mpumalanga. The lesson here is that the Secrecy Bill’s intention to intimidate whistle-blowers will collapse in on itself as “data journalism” becomes increasingly important to expose hidden connections or connect the dots in a way that’s not immediately evident. Data tools can become an important source for stories if informants start clamming up.

"The building blocks are in place. All it takes is one data journalism project to go viral, and we could see a quantum shift in a very short period. The trick, of course, will be to build business models to support the new approach," says Arenstein. All the tools are there. Most of them are free. Now what’s required is the will to use them.

It's called the African News Innovation Challenge, and it has $1 million to award in start-up grants. Anic, which was announced in October, and had its soft launch in last week, will formally launch in January. Theresa Mallinson talked to project manager Justin Arenstein about the thinking behind the initiative.

The premise of the African News Innovation Challenge is simple: it's a fund that will enable African news organisations to experiment in the digital technologies and other newsroom innovations, at no financial risk to themselves. But it's not about simply throwing money at problems; the projects that are chosen will also receive technical support and individual mentoring.

Justin Arenstein

Anic's project manager Justin Arenstein, who currently holds a Knight International Journalism Fellowship, is a passionate advocate of news innovation and, in particular, data journalism. “People often say there are not many data sources in Africa; it's bullshit,” Arenstein says. “Even if it's not available in country, it's often available out of country.” Institutions such as the International Monetary Fund and the World Bank have detailed country data available, even where individual governments don't make it readily accessible to their citizens. Arenstein says one of the aims of Anic is about “realising the wealth of information that's available (and) the use it could have in the newsroom.

“A lot of African media... just regurgitates,” says Arensteins. “We seldom have the tools to go out and aggressively test what they're saying.” Elaborating further, he says: “The problem is we (journalists) don't have the programming and IT skills necessarily and IT geeks don't have any concept of content, so they build fantastic tools, and they sink without a trace. The idea was how do we bring together these two very different constituencies?”

Answering his own question, Arestein says that: “One of the ways that's worked internationally is to dangle a very big carrot in front of them.” And so the concept of the African News Innovation Challenge was born. Initially the Omdiyar Network put up $500,000 for the project, and this was soon matched. “It actually only took us two-and-a-half months to raise a million dollars,” says Arenstein. “There is money available for innovation and experimentation.”

The challenge will work like this: in January, at the formal launch, Anic's advisory council and judging panel will be announced. Arenstein says these two bodies already include people from the Bill & Melinda Gates Foundation, the Knight Foundation, the International Centre for Journalists, and Google. In February, a call for entries will go out. “We envision a two- to three-month window period, during which people will post their idea on a public forum, and we'll casually invite the rest of the industry to help refine that idea,” says Arenstein. “This is not really an initiative for people who want to build top-secret (projects). Ideas are cheap; it's actually execution that builds sucessful products and successful projects.”

Winners will be announced in the third quarter of 2012, and then the hard work really begins. “We've tried to learn from what people have done in the US and Europe and elsewhere,” says Arenstein. “We've looked at the innovation funds that seem to have had an impact, like the Knight News Challenge. They've learnt from what works best, and what doesn't work best. Simply throwing money at a bunch of inonvators, and giving them all that money upfront, doesn't really help.”

To this end, money given to Anic awardees will be released in tranches, with each project receiving between $12,500 to $100,000 depending on its scope. “The model that we're looking at using is helping people sit down and very critically reassess their proposal, and break it down into bite sizes,” says Arenstein. And if any of the start-ups fail, that won't be a problem; it's the nature of the business.

Innovators that are awarded funding will find themselves in an open environment. “Ideas that win (can) evolve during course of development,” says Arenstein, adding that development time will be anything between 12 and 18 months. “We're not going to hold people religiously or fanatically to the original concept,” he says. And they'll be given plenty of help and support along the way. “Where there are gaps in their own team, we'll try and plug those gaps by bringing in external people,” Arenstein says. “And right from the beginning we're going to try and give them some of the world's best media economists, to develop some of the most robust models.”

Commenting on the innovation the project intends to foster in African media, Arenstein says: “You almost need a crisis where there's real pain in a marketplace, for people to go out and invent something to solve it.” With the backing of the African Media Initiative, the Omidyar Network, the International Centre for Journalists, Google, the Knight Foundation, Konrad-Adenauer-Stiftung, the US Department of State, and the World Association of Newspapers and News Publishers – all of whom are contributing either funding or expertise – the African News Innovation Challenge has already garnered high-level support. Now it's up to the journalists and media workers of Africa to meet the challenge. We look forward to seeing what they come up with.