Great lies of our time: “journalists and coders should sit together to create amazing stuff” (updated)

The Thomson Reuters newsroom. Note papers stacked all over the place. No idea if journalists and developers “sit together” here – but I’d bet they don’t. Photo by Targuman on Flickr.

I keep seeing people saying “you know how journalism and the internet can work better? Have the news org’s journalists and coders sit beside each other. Wonderful things will happen.”

Postscript, but at the top: this post generated a lot of reaction – so be sure to read the followup, which pulls together the many people saying that it can and does work./Postscript.

Let me tell you: when someone spins you this line, it’s pure unadulterated 100% bullshit. Anyone who says this has never looked at what happens when you do this, or considered the differences in work patterns between the two. (It pains me to point out that Wolfgang Blau is only the latest to suggest this – see point 4. I like and respect Wolfgang a lot and he’s really smart – he suggested adding a chapter about China to the second edition of my book – but on this, he’s wrong, as I’ll explain.)

Edit: let me be clear just before you dive all the way in: my criticism is that a seating plan will not solve this conundrum. It is possible to solve it. But not by simply putting people together. You need much more subtlety. /Edit.

Five stars for the idea..

Let’s begin with Anecdote 1. When I worked at the Guardian as technology editor, part of my remit was the games side. We had great games reviewers who turned out games reviews that people wanted to read. As time went on, though, I began to worry that these reviews (giving games between one and five stars) were being lost in the haystack of content. There were years of reviews which could all be relevant to a puzzled buyer looking to get the best stuff.

What, I wondered, about the people – parents especially – who had just got a new games console and wanted to know which were the top-rated games to get for them? Where did they go?

The obvious answer, it seemed to me, was to create a page for each console which would show the highest-ranking games – perhaps in a grid. Obviously you’d need to update it from time to time. Finding which games were for which console wouldn’t be hard because each review would be tagged “PS3” or “Xbox360” or “Wii”, but grabbing the star ranking and then ordering them was tougher. But it wouldn’t be a problem for a developer who knew their way around our CMS (content management system), I figured.

My cunning plan: make it possible to buy the games directly through the CMS, either from Amazon or a white-label seller, so that the pages would earn money.

So I went to one of the developers who I knew had some spare time and was a dab hand with the CMS. I explained the problem and the potential. He looked at me for a while. I also suggested that we might be able to get a list of upcoming games – which our reviewers could supply – and create a page where people could pre-order them. He looked at me a little longer.

Then he went away and spent a few months developing a way to display the Guardian on Google TV. After making no appreciable impact, Google TV got canned. We never got the “five-star games” pages or the “upcoming games” page.

Love ya, Excel

This leads to Anecdote 2, which followed on pretty directly from that. I’m a journalist, but I can code – PHP and Applescript, principally. Nodding acquaintance with Cobol, Fortran, Pascal, Lisp; blind date with Objective-C where I fell asleep in the wine; blind date with C where I walked out. Anyhow, I can wrangle simple-ish tasks to automate extremely boring stuff that is often the basis of work in many newspaper offices.

So, for example, every week the UK games industry puts out an Excel spreadsheet showing the top 10 games sold in the previous week by various platforms, with information such as the publisher, previous week’s position, and so on. Here’s what it looked like:

The raw UKIE games chart: an Excel file with lots of columns and spare rows. A formatting nightmare

I liked the idea of putting up a weekly chart, and making it possible for people to buy games directly from the chart. We already had a “buy” button embedded in the reviews. So you could find the relevant game, pull the “buy” button from it, embed it in a chart, and put it on the site. Money-spinner, potentially; and certainly not a money-loser.

I considered trying to get developer time for someone to write this for, oh, about two seconds. Then I decided to write it myself.

It wasn’t trivial. I had to assume access only to the tools and programs our subeditors (who would create the “story” with the chart) had, which meant a standard installation – no extras. That meant no Excel – we used OpenOffice. But that meant no Applescript, because OpenOffice has woeful Applescript support. That meant the sub had to copy the required cells out of the OpenOffice spreadsheet and paste them into TextEdit, which has fairly bad Applescript support, but at least it’s there. More wrangling (stepping through the cells to find the ones we wanted, spotting the many lines of padding in the spreadsheet and ignoring them, using Google’s API to find the link to the Guardian’s link to the game in question, pulling the source of that link to find the affiliate “buy” links) ensued. Once I had the set, we had to create another TextEdit document which could then be copied and pasted into OpenOffice, and saved as an Excel format spreadsheet, because that was the only one which our CMS recognised if you were going to include a chart.

It took me a couple of weeks of spare time, but I got it done. The result was stuff like this:

After using my script, you could buy games directly, or follow the links to reviews. (It’s a different week from the chart above.)

The outcome was that work that would have taken any subeditor about an hour, and been prone to all sorts of errors (reading the wrong column, picking the wrong game) became a two-minute task in which the toughest part was choosing which picture to use. And it earned money through affiliate buying; not huge amounts, but more than zero.

Now – you’re probably saying “hey, wouldn’t it have been easier to ask the UKIE for a link to their API so that you could get it direct, and format it?” Answer: if they’d had an API, it would have been easier. But asking and getting wouldn’t have been. That’s the sort of thing which requires lots of meetings, chinstroking, and discussions about whether it’s possible. All those things are anethema to the culture of daily newsrooms, which is to Get Stuff Done.

(Oh, but of course: there came a time when UKIE changed the formatting of the spreadsheet. And by then, I couldn’t be bothered to work back through the code to adjust for it. So the chart died.)

Update/edit: Robert Rees, who is the developer manager at Guardian News & Media, has written about this subsequently:

The central assumption that runs through Arthur’s narrative is that it is valuable to let readers pre-order computer games via Amazon. One of the pieces of work I’ve done at the Guardian is to study the value of the Amazon links in the previous generation of the Guardian website. I can’t talk numbers but the outcome was that the expense of me looking at how much money was earned resulted in all the “profits” being eaten up by cost of my time. You open the box but the cat is always dead.

Now, that’s interesting. I had seen some of the affiliate revenue numbers too; they weren’t huge by any means. My point was more that we already had the Amazon buttons (that had been set up as an automated process some time before by our development team) and so we might as well use them. But Rees makes a good point – you have to look at the opportunity cost too of the developers’ time if you go ahead with the work. (No developer time was given to my idea.) Also, this was a time – still is – when we were being encouraged to “experiment” with revenue-generating ideas; I didn’t believe that affiliate money would make us rich, but thought that if we could get a white-label deal so we got bought at wholesale and sold at retail, rather than selling at retail-affiliate, then it could be worthwhile. (The Guardian does this with books.)

Sub-update: a little discussion at Gawker threw up this gem, from its former edit honcho Joel Johnson, discussing Gawker’s finances:

Gawker Media is an advertising-based business, with revenues of around 35- to 45-million dollars a year. There are a few other sources of income: a couple of million for international licensing fees (from the companies that publish international versions, such as Kotaku Australia); and affiliate fees, largely from Amazon, that add another $5-10 million a year.

$5-10m per year? Maybe this affiliate game can be played to win if you do it right./sub-update

Rees titles his post “No one loves a bad idea”, which is a fair criticism – my idea wasn’t useful in the wider scheme of things. Equally, though, that was never pointed out. In effect, I did a little skunkworks project, which died. /update

The difference is time

It’s this key difference – that journalists tend to want to get something up there for people to read/watch/listen to, dammit – which in my experience marks the giant gulf between journalists (including those who can code a bit, like me) and “developers”. Journalism, especially daily journalism, and increasingly all journalism, is about getting a result, and doing so effectively, quickly and (optionally) thoroughly. It’s a culture where getting it done right now is what matters. And then you move right on to the next thing, because there’s always a next thing.

Update: Rees calls this (correctly) out as an example of the rush to get stuff done:

British journalism favours action and instinct and sometimes that combination generates results. Mostly however it just fails and regardless of whom is sitting next to whom, who can get inspired by a muddle-minded last-minute joyride on the Titanic except deadline-loving action junkies?

(OK, I consider myself upbraided.) /update

Developers I’ve come across, by contrast, iterate around the same thing repeatedly, just as I did in trying to get the code to work in Anecdote 2. They don’t grab an idea, use their existing knowledge to turn it into a story, and put it out there. (OK, there are some gonzo programmers who do that sort of thing, but you don’t find them in newsrooms I’ve been in.)

My coding in the newsroom was all about timesaving. I wrote scripts that wrangled InCopy so that articles which needed to be strictly formatted (Ask Jack, in its printed days) could be, in different colours, in 10 seconds rather than the 20 eyestraining minutes that it took by hand. (Ask Kate Bevan and Stuart O’Connor about that.) I wrote a script that meant the subeditors on the Media desk could get the day’s Guardian Media stories from its web page in one second rather than five minutes, ready for the daily email which had to go out before 8am. What I did was all about saving time, because that tends to be the most precious commodity for the journalist.

For the developer – in my experience – things are different. I recall Dan Catt (one of the early Flickr developers), when he worked at the Guardian, pointing out how amazing it is that the paper starts with effectively nothing every morning and has a full, complete newspaper ready by 8pm or so. And then keeps doing it again and again and again. That’s organisation. That’s deadlines.

The task of developers in a news organisation, on the other hand, isn’t on that timescale at all. It’s about building apps; building great mobile sites; building great ways for people to interact; building systems to manage and interlink content so the almighty Goooooogle indexes it better. Sometimes it’s about building new CMSs, or ways to link multiple CMSs, but in my experience (again) it’s hardly ever about spotting kinks in workflow and easing them.

And as for the idea of journalists and developers sitting beside each other and the developer saying “hey! Why don’t we..” (or vice-versa) – forget it. They just don’t have that common ground. Journalism tends to be surface-skimming, fast-moving; coding/development tends to be deep-diving, strongly focussed.

Edit: one thing that occurs to me is that many journalists who don’t have awareness of coding don’t realise that boring, repetitive tasks can be automated. So they don’t know to ask. /Edit

Update: of course as soon as I hit “publish” I could think of a counter-example: the team at the Mirror’s @ampp3d team, such as Tom Phillips and Martin Belam, who created things such as the “badgers moving the goalposts” game.

Phillips’s description of how this came about is worth reading:

October 9, 2013. Owen Paterson, the Secretary of State for Environment, Food and Rural Affairs, made a comment in the Commons that morning about the failed badger cull – he blamed the badgers for “moving the goalposts”, which was obviously hilarious. What I love is that you can trace, to the minute, the time from when my mate Francis suggested to UsVsTh3m that they do an “Owen Paterson’s Badger Penalty Shootout” game, to when they put it live. 4 hours and 21 minutes to publish a fully playable game about a news event that happened that morning. Nobody else could come close to doing that then; nobody else has got anywhere close since. They were, in the most literal sense possible, playing a different game to everybody else.

Does this disprove my argument?

You’ll be unsurprised to hear me say: I don’t think so. I see that as an example of developers who are interested in current events basically being given free rein (in a good way). The aim was to get virality, a la Buzzfeed, not to “do journalism”. Repeatable? Effective? Maybe, but the journalism element isn’t exactly blinding.

Similar for the “photo comparison” slider that you see on, I think, the WSJ and Verge (possibly others) for literally side-by-side comparison of photos taken by different phone cameras: I’m sure a journalist or developer expressed frustration at the need for it, but getting it done is an iterative process, focussed on a particular target. Deep diving, not skimming.

The data question

But wait, you say: what about data journalism? My answer: it’s not programming. It’s not coding. Most of the time, it’s great works done with Excel. And most of the time, that’s all it needs. It’s certainly not something where you want a valuable developer spending their time. Journalists are cheaper, and they know how to spot a story, or ought to.

So should journalists learn to code?

I’ve answered this before, on my personal blog – jeez, it was 2009. Here’s the posts’s title: “If I had one piece of advice to a journalist starting out now, it would be: learn to code”. See if you can figure it out.

My advice on developer/journalist seating arrangements? Keep them well apart. But also have an effective manager who listens to the two groups and can tell each to do things. Have a manager who can direct them. Don’t stop the two groups talking. But let each be aware that they’re like animals living at different speeds, which means that they simply won’t be able to comprehend why the other side reacts as it does to some things (“why can’t they understand that hunting down bugs is hard?” “Why can’t they understand that we can’t have this sort of thing cause a screwup in a live blog?”). They can live a mutually beneficial existence. They just can’t live in the same cage.

Postscript: huge amount of pushback on Twitter over this, from Wolfgang (thanks, Wolfgang), Aaron Pilhofer and many others. And lots of good points and examples raised, which I collated in a Storify linked from this followup. It would be great to know that my experience was just an outlier, though I suspect it’s more that there are good ways to do it, and rather less good ways. What we want are the good ones.