The hacking was punctuated by talks by Tom Lee of the Sunlight Foundation and Jake Porway of Data Without Borders.

Projects

Dan Nguyen scraped Florida mugshots from and used face.com's API to analyse each photo to tell you the arrestee's mood.

Michael Keller, Marc Georges et al. related the NYPD stop, question and frisk data to nine mosques referenced in an NYPD report on surveillance in order to see whether there had been unusual changes in stopping activity around these mosques.

The dataset is insanely messy, but they fortunately had access to a relatively clean version that Data Without Bordes had developed in November.

They were still going strong after the data camp. Refusing to leave, they moved to a different room after getting kicked out of the data camp space.

Mike Caprio and team cleaned a spreadsheet of 80,000 records from the New York lobbiest website to power a site on New York lobbyists based on the Chicago Lobbyists site. It appears that $120 million was spent on New York on lobbiests in 2011.

I helped one team relate contracts from Open Book New York to data that they had scraped by hand (!) from hand-written forms in order to identify pontential conflicts of interest.

I helped another team identify potential stories (outliers) in the NYC Open Data graffiti locations dataset.

Susan McGregor was "clearly hooked" because she liberated lobbyist contract details the next evening instead of watching the Superbowl.

Technical Awards  

Mike Caprio won *Best Data Liberator* for liberating the Iowa accident reports database.

Michelle Koeth won *Best Creation of an API* for scraping New York, NY hospitals from Medicare Hospital Compare.

Jeremy Baron, from UN peacekeeping team, won *Best Use of ScraperWiki* for scraping United Nations PDFs. This team also scraped peacekeeping statistics and contributions.

Honorary ScraperWikian  

Susan McGregor was awarded "Honorary ScraperWikian". We haven't decided what that means yet. 

Learning

Teaching the Learn to Scrape sessions and working with many of the project teams, I got the impression that we had opened participants to thinking more about how data can be scraped, transformed and analyzed to identify unusual subsets and potential stories. Split-applied-combined, if you will.

Our 'Learn to Scrape' sessions seemed to work as well; I found several participants who had claimed no knowledge of webscraping prior to the sessions to be creating reasonably complex scrapers by the next afternoon.

What Next?

More data camps are coming up, and several groups plan on contining to work on their projects. But in the mean time, we now have lots of data for you to analyse!

Where can I find data and how can I request access to it? What tools are available to me? How can I find useful stories within datasets? And, most importantly, how can I make a living through the practise of data journalism?

Do these questions sound familiar? They should, because these are all questions that any data journalist, who is just getting started, should be asking.

Leading data journalists from the New York Times, the Guardian, the BBC, and other top media organisations from around the world, are working together to answer all of these questions (and more) in the Data Journalism Handbook. The handbook is the first comprehensive practical guide to data journalism. As Aron Pilhofer, editor of Interactive News at the New York Times, noted at the beginning of the project: “A project like this is quite necessary. It’s kind of surprising that someone hasn’t tried to do this until now.”

Interested in contributing? We want YOU!

Work on the handbook was kick-started in November of last year at the Mozilla Festival in London. In just two days, 55 contributors drafted 60 pages (20,000 words) for six chapters. The handbook is a community project; therefore, anyone who has experience in data journalism can help draft the book. The work, in turn, goes back to the community as anyone is allowed to freely use, modify, adapt, and reuse the handbook.

Contributions have been flooding in and since Mozilla we have been editing, updating, filling in gaps, and restructuring. We are very close to a first complete draft of the book and we need your help to get there. We are looking for authors, editors and peer reviewers to draft chapters, review content, style and accuracy of the book.

In the table of contents below you can see the progress that has been made and where input is needed:

• 0. Preface (in progress)
  • 0.1 The purpose of this book
  • 0.2 Add to this book
  • 0.3 Share this book
• 1. Introduction (done)
  • 1.1 What is data journalism?
  • 1.2 Why is it important?
• 2. Introducing data journalism in the newsroom
  • 2.1 Changes in the newsroom (contributors needed)
  • 2.2 How is it done: journo-developers vs. coders for hire (in progress)
• 3. Types of outcomes/projects and case studies
  • 3.1 Data powered stories (in progress)
  • 3.2 Data served with stories (in progress)
  • 3.3 Data driven applications (contributors needed)
• 4. Working on the data story
  • 4.1. Step 1: Getting your data
    • 4.1.1 Where does data live? (in progress)
    • 4.1.2 Asking for data (in progress)
    • 4.1.3 Getting your own data (in progress)
    • 4.1.4 Crowdsourcing data (contributors needed)
  • 4.2 Step 2: Understanding your data
    • 4.2.1 Data literacy (done)
    • 4.2.2 Working with data tips (in progress)
    • 4.2.3 Tools and techniques for analysing data (contributors needed)
    • 4.2.4 Harnessing expert opinions: Annotating datasets (contributors needed)
  • 4.3 Step 3: Finding a story in your data (in progress)
    • 4.3.1  From datasets to stories - approaches 
  • 4.4 Step 4: Delivering your data project (contributors needed)
    • 4.4.1 Serving data with stories 
    • 4.4.2 Visualising data 
    • 4.4.3 Data driven applications 
• 5. Engagement, outreach and community (contributors needed)

• 6. How to make data journalism sustainable
  • 6.1 Measuring impact (contributors needed)
  • 6.2 Business models (contributors needed)
• 7. Appendix (in progress)
  • 7.1 Further resources 
  • 7.2 Glossary

Whether you are a budding data journalist putting the manual through its paces or someone with years of experience, we'd love your help to fine-tune the book and get it ready for the press (no pun intended).

If you're interested to contribute to the first Data Journalism Handbook please fill in this form as soon as possible indicating your level and area of expertise and the chapter that you would like to contribute to. We will get in touch with you as soon as we can once you've submitted the form.

Deadline

A first complete draft is planned to be ready by the end of February.

The official launch of the The Data Journalism Handbook will be at the International Journalism Festival in Perugia from 25-29 April. The book will be available online as an e-book. Participants at the festival will have the opportunity to buy a printed copy of the book and enjoy a meet and greet with the authors where they may exchange knowledge and learn helpful tips about how to successfully become data journalists. 

Questions?

For questions get in touch with the Data Journalism Handbook coordinators: bounegru@ejc.net (European Journalism Centre) or lucy.chambers@okfn.org (Open Knowledge Foundation).

Image by Kate Hudson.

Following an opening talk on upcoming developments from Wikimedia, such as Wikihadoop, five presentations were given; each covering a specific group or project concerned with the use or pursuit of open data.

First, Velichka Dimitrova introduced the Open Economics working group, which is seeking to break down the barriers between members of the public and the economic data which is increasingly influential in their daily lives. The group is currently focusing on a project aimed at modelling social progress in Italy, which it hopes to enter into the Apps4Italy competition. Ideas for future tasks were also discussed, of which two of the most interesting were an analysis of ‘author cartels’ within the economic journal publishing industry, and the proposal of an ‘open data accreditation badge’ which could be awarded to organisations and companies who seek to make open to the public any data they hold.

Next, Julian Tate, of Open Data Manchester, gave a fascinating glimpse of his group’s ongoing analysis of public transport data in the city. Somewhat perversely, competing bus companies currently refuse to publish fare information online, fearing that rivals will undercut them by tiny margins and steal their customers. According to Julian’s research, thousands of potential commuters are put off using the service because of the lack of transparency, and far more would use public transport if prices for the various journeys were openly available. Julian and his team are now attempting to do what Transport for Greater Manchester have as yet been unable to do, and compile - through a mixture of crowdsourcing and pleading with bus companies - a comprehensive and open list of fares for journeys throughout the region.

Kevin Carter then introduced his venture; the ingeniously titled Landscape Portrait. The project aims to blend art and data to produce what is effectively an ‘identity map’. The idea is that individuals provide answers to a number of questions about their aspirations, feelings and concerns, and these are then compared with statistically-generated regional stereotypes. The result is an intriguing spatial patchwork of social outlooks and aspirations.

Jo Pugh then announced a hack day at the National Archives, which will take place on 24/25 March at the Archives in Kew. Pugh described how the very existence of the Archives in only justified if the records held there are examined and shared - precisely what the hack day is intended for. Anyone who has an idea for making use of the records - some of which stretch back for centuries  - should come along.

Finally, Keiichi Matsuda outlined his project; ‘Prism: a Sculptural aggregator of live data in London’. While the name might be something of a handful, the idea itself is beautifully simplistic. Matsuda is creating an exhibition for the London Design Festival where he plans to project interesting London-centric statistics onto a series of angular sculptures in what he refers to as a display of “urban informatics”.

The Eurozone crisis has dominated headlines across the continent in recent months, and will not be leaving the spotlight any time soon. Along with Greece, Italy has been one of the biggest causes for concern, and the Italian economy provided the backdrop for the day’s main focus.

Following the success of Yourtopia, Open Economics’ main project to date, the central aim of Saturday’s activities was to develop an application to map social progress in Italy over time and across its 21 administrative regions.

Over the years, different bodies have come up with different methods of measuring social change, from the qualitative - public surveys - to the quantitative - unemployment statistics and income distribution - but none can be considered sufficiently holistic to provide a fully representative overview of the state of affairs.

This new project, Yourtopia 2, will take a wide variety of measures and combine them into a broad model, with the hope of providing an unprecedented analysis of socioeconomic welfare which can then be applied to other countries and regions. All being well, the final application will be submitted as an entry to the Apps4Italy competition.

As with Yourtopia, the aim for Yourtopia 2 is to end up with an interactive tool which users can customize depending on which factors they consider most important to achieving social progress. As a result, Saturday’s focus was on gathering data for the different variables, beginning design of the user interface, and carrying out further research into how the eventual model will be built.

In one corner, a handful of data researchers scoured the web for sources of regional unemployment data and Gini coefficients, in another, coders and designers worked tirelessly to iron out glitches in the model and create a slick user interface, while elsewhere academics debated which factors the model should incorporate in order to be most effective.

The project is ongoing. Anyone wishing to contribute in some way had the possibility to take part in the online follow-up session last Saturday. Should you wish to take part in future meetings keep an eye on the Open Economics website. 

Photo by Ilias Bartolini. 

In the past investigative reporters would suffer from a scarcity of information relating to questions they were trying to answer. While this is still the case, today journalists are also faced with an overwhelming abundance of data. In an age of information overload, to stay relevant to society journalists need to learn to separate signal from noise in order to provide valuable insights. Journalists need to be equipped with knowledge of the tools, techniques and tactics of working with data in order to derive maximum value from for their readers.

The European Journalism Centre and the Open Knowledge Foundation are pleased to invite you to the School of Data Journalism hosted at the sixth edition of Italy's leading journalism event, the International Journalism Festival. The 2012 edition takes place in the beautiful city of Perugia between 25-29 April. Entry to the School of Data Journalism panels and workshops is free. Each workshop has a limited number of places and therefore registration will be necessary. Please note that not all requests to participate in the workshops will be accepted.

What is the School of Data Journalism and who is it for?

The School consists of three panel discussions and five workshops.

The panels attempt to provide answers to crucial questions for aspiring data journalists, editors and decision-makers in newsrooms:

• What can aspiring data journalists learn from the successes of the past?
• How can data journalism save your newsroom?
• How do you start a data journalism operation?
• How can you become a data journalist and what do you need to do?

In the workshops journalists who are interested to get started with reporting with data and budding data journalists will learn from experienced data journalists and open data experts essential skills related to how to get the data you need, how to analyse it, how to get stories from data and how to present your stories.

Panels

Panel 1: News and numbers: from CAR to data journalism (Thursday, 26 April)

Journalists have always used data and numbers to produce stories…and win Pulitzers. From Philip Meyer’s coverage of the Detroit riots in 1967 to Steve Doig’s 'What Went Wrong' analysis of the damage patterns from Hurricane Andrew, data-driven reporting has brought valuable public service and won journalists recognition and prizes.

Whereas there may be distinguishing aspects about the data journalism of today and the computer-assisted reporting of the past, it is crucial to learn from successful examples, techniques and approaches of the past.

• How can we bring the data journalism community and the CAR community closer together?
• What can aspiring data journalists learn from the successes of the past?
• What is the future of data journalism?

Panel 2: How can data journalism save your newsroom? (Friday, 27 April)

Journalism is under siege. Traditional models are collapsing. Developing the know-how to use the available data more effectively, to understand it, communicate and generate stories based on it, could be a huge opportunity to breathe new life into journalism.

• What is the potential of data journalism?
• How do you start a data journalism operation?
• How can you start thinking about making money with data journalism?

Panel 3: You Too Can Be A Data Journalist! (Saturday, 28 April)

Where can I find data? How can I request data? What tools can I use? How can I find stories in data? How can I make money with data journalism?

Several leading data journalists, CAR specialists and journalism professors from the Guardian, the New York Times, Financial Times, Chicago Tribune, Medill School of Journalism and Cronkite School of Journalism worked together to answer these questions in the Data Journalism Handbook.

The handbook, the first comprehensive practical guide to data journalism, will be officially launched in this session. The session will provide the opportunity to meet and greet authors of the book, exchange knowledge and learn from them what you need to know to be a data journalist, as well as get a printed copy of the book.

Speakers

• Caelainn Barr, EU data journalist, formerly with the Bureau of Investigative Journalism in London
• Heather Brooke, journalist and FOI campaigner
• Sarah Cohen, Knight Professor of the Practice of Journalism and Public Policy, DeWitt Wallace Center for Media and Democracy, Duke University (Pulitzer prize winner)
• Steve Doig, Knight Chair in Journalism, Walter Cronkite School of Journalism, Arizona State University (Pulitzer prize winner)
• Mirko Lorenz, data journalism trainer, Deutsche Welle
• Aron Pilhofer, editor of Interactive News at The New York Times and co-founder ofDocumentCloud.org
• Dan Nguyen, news application developer, ProPublica
• Simon Rogers, editor of the Guardian Data Blog

Workshops

Workshop 1: Scraping data & cracking PDFs (Thursday, 26 April)

Hands up who knows what machine-readable data is? You will soon, and more importantly, how to get it and what you can do with it once you have it!

A workshop targeted at scraping from scratch, including:

• What PDFs and webpages look like to your computer - An introduction to machine-readable / non-machine readable data
• The Scraper Cookbook - an overview of the key things you need to know to write a scraper
• Hands on session - learning to screen-scrape. Main focus: hands on session using tools such as ScraperWiki. If sufficient interest & time, we will also touch on some of the tools & skills needed to extract data from PDFs.
• Error checking - how to check what you have makes sense, spotting the types of errors sometimes introduced if you don’t get it quite right! 

Workshop 2: Information wants to be free - Freedom of information requests and how to use them (Friday, 27 April)

Freedom of information requests are constantly evolving. Law changes and technological advancements make it increasingly easier to file and systematise FOI requests, and importantly track their progress through the system. This workshop includes demonstrations and case studies examining the current state-of-play with FOI requests in Europe and beyond and looking into what’s next for the freedom of information movement. 

Workshop 3: Making data pretty (Friday, 27 April)

Journalism is no longer just a block of prose on a page. The modern reader often demands maps, infographics and visualisations to make the story jump out at them, particularly in digital environments.

There are a vast array of free tools available on the web to allow data-journalists to quickly and easily digest, process and display the data powering their stories. This workshop aims to give a good overview of what is currently available and delve into depth on one of the most powerful: Google Fusion Tables.

Workshop 4: Getting Stories from Data (Saturday, 28 April)

Enormous datasets can often prove extremely daunting to the unfamiliar. Mistakes and crimes have historically benefited from, and triumphs and good decisions been obscured by, a mask of bewildering numbers and statistics and gone unreported.

Large datasets often hold a wealth of undiscovered stories for those willing to invest the time into exploring them. This workshop is a ‘spotters’-guide’ for things to look out for and where to look for datasets.

Workshop 5: Spending Stories (Sunday, 29 April)

To get to the bottom of a story, you need only to 'follow the money'. The same is true of government: budgeting is where policies and priorities are broken down into figures. Financial programming has a direct influence on all political areas: while other data on health or social help us understand what challenges society faces, looking at spending data allows us to see how government reacts to all of these.

There are many spending databases available on the web, some impenetrable, some accessible for analysis. We’ll show how we enable journalists and researchers to make sense of the data and what strategies can be used to investigate stories and policies.

• Overview: What datasets are out there? Where can you look for more?
• Interrogating databases, how to extract the maximum amount of data out of tricky databases
• Tools for spending analysis, how to slice and dice once you have your data

Workshop leaders

• Caelainn Barr, formerly with the Bureau of Investigative Journalism, working on the award-winning Structural Funds investigation
• Lucy Chambers, Open Knowledge Foundation, Community Coordinator for OpenSpending and the ‘Spending Stories’ project.
• Steve Doig, Knight Chair in Journalism, Cronkite School of Journalism, Arizona State University (Pulitzer prize winner)
• Friedrich Lindenberg, Open Knowledge Foundation Deutschland, Developer on OpenSpending
• Dan Nguyen, news application developer, ProPublica
• Simon Rogers, editor of the Guardian Data Blog

When and where?

The Data Journalism School takes place at the International Journalism Festival in Perugia between 25 and 29 April 2012. The schedule of the Data Journalism School, with confirmed speakers for each panel and workshop, will be posted on the festival website in early February.

How to register?

Entry to the festival and the School are free. There is no registration process to attend the festival. For the workshops there is a limited number of available seats. To secure a seat in the workshops please register via this form. The deadline for workshop registration is 20 March 2012. You will be notified by email by 25 March at the latest if we were able to confirm you a seat. The workshops are entry-level. Consideration will be given to your experience, skills and motivation to attend the workshop when making the selection.

What do you need to bring?

A lot of enthusiasm and a laptop are required for the workshop sessions. Please note for hands-on workshops tablet PC's will not be appropriate.

Questions?

If you have questions about the School of Data Journalism get in touch with the coordinators: bounegru@ejc.net or lucy.chambers@okfn.org.