Category “rants”

Dear Meg Hillier, The Digital Economy Bill is Unjust

Dear Meg Hillier,

As a constituent whose career and majority of personal communications are conducted across the internet, I’m very worried that the Government is planning to rush the Digital Economy Bill into law without a full Parliamentary debate.

The Bill contains measures that favour the protection of commercial interests at the expense of an individual citizen’s rights – specifically measures that allow copyright holders to issue requests to limit or even terminate the internet connections of private individuals based only on the belief of the copyright holder that the individual has infringed their copyright. In effect, this creates a situation outside the bounds of a fair and just society where a person can be punished by the withdrawal of a service that the UN is proposing be considered a basic human right.

Read More »

Ten Things I’ve Learned in a Start-Up

Ten things I’ve learned as a developer at WooMe:

  1. Be clear what your direction is. Identify your key proposition and focus on it. Don’t get distracted by unrelated features – somebody else is probably already doing those better than you.
  2. Be merciless with features that don’t cut it. No matter how much you like it, if your audience doesn’t like it, either change it or kill it fast. Wasting time on your personal pet project when there’s no clear demand for it is time you should be spending on features users want.
  3. Use what’s already there rather than inventing your own. Need a message queue? Video processing? Load balancer? Email distribution? Payment system? Use something that already exists rather than waste time you should be spending on feature development.
  4. Quick and dirty is better than late to market. At the rate this industry moves, capturing your audience’s attention first is more important than having a well-engineered solution. For the most part, users don’t care about your code quality, as long as it works.
  5. Don’t optimise until you need to. Sure, you may end up chasing your tail for a bit when you need to scale, but until you hit that point you’re wasting valuable time that should be spent on feature development.
  6. Scaling is most often a data problem. When you do need to optimise, scaling won’t be about the performance of your application. Scaling web applications horizontally is easy. Scaling databases vertically is expensive. Scaling databases horizontally is very, very hard. Don’t worry about what web application framework you use, worry about what database you put behind it.
  7. Abstraction is great, as long as you understand what you’re abstracting. This has been a constant pain with Django’s ORM. Yes, it’s a powerful and useful abstraction, but without a solid understanding of what the database is doing behind it, it’s easy to get into very deep trouble, very quickly.
  8. Log everything. Nothing is harder than trying to find problems on a live environment without detailed logging.
  9. Storage is cheap. It’s better to store everything than to throw away data. You never know when you might need something.
  10. Communication is critical. Not just outside your organisation, but within it. Tell people what you’re doing, frequently. Ask them what they’re doing just as often. Not only do ideas percolate through this process much more effectively, but when things go wrong, clear and frequent communication will stop them from getting worse.

Bingo We Can Believe In

Alcohol and politics are two things that should rarely mix, but once in a blue moon there’s cause to throw caution to the wind and get a little crazy.

One such occasion is tomorrow night, when the US electorate takes to the polls and – with a little luck – elects Barack Obama their next President.

There are all sorts of reasons why this is such an important election, both in terms of America’s current standing within the world community, and a number of imminent domestic problems that the next President will find himself dealing with probably before he’s even inaugurated.

That’s perhaps a post or two for another time; once the results are in and the residents of Awesome Manor have either celebrated or commiserated until they drop.

For now, I simply offer my humble contribution to the election festivities. I call it “Bingo We Can Believe In“.

The End of the Line

About 10 days ago we had our internet connection activated at Awesome Manor. It’s Be Internet’s (up to) 24Mb all-you-can eat buffet package.

Problems, naturally, began immediately. During the normal period of line testing that the ADSL2+ products use to determine the maximum stable rate of the line, we saw massive instability and fluctuating speeds from 12Mb down to 300kb. Not ideal for a house full of internet junkies.

As it turns out, BT has been screwing around with the Barnet exchange for the past week, which has stuffed up our MSR testing well and good. It looks like Be have settled on 12Mb/1.3Mb for our line now, which at under 1km from the exchange is just not good enough.

Rather worse, however, is the quality of wiring in Awesome Manor. Judging from the amount of work I’ve just done to minimise noise on the lines (pulling the bell wire from all extension sockets, and stripping and rewiring the master) I’d guess that most of the wiring is older than I am.

After much work, and much swearing, I’ve managed to reduce line attenuation (signal loss due to the crappy quality of wiring) by more than 50%. This seems to have stabilised the line, and is a good start for improved line speed when the MSR tests finish later this week.

The worst thing is, I’m about as technical as you get. I dread to think what normal people go through in this kind of situation. I can only assume that if they reach breaking point they would fork out a few hundred quid for a BT engineer to come and rewire their house, which seems like a fairly expensive way to solve a problem that can be fixed with a screwdriver and a pair of pliers.

BBC Made of Fail? No, definitely not.

A couple of days ago I wrote an opinion piece on why I feel that being forced to duplicate existing software projects due to an antiquated and limited infrastructure is a large part of the reason the BBC is falling far behind the rest of the industry in its internet offerings.

To put this in context, until two days ago this site has had about 100 visitors total, so I was a little surprised to find that within hours the article was making its way onto the front page of Reddit, and being circulated amongst technology bloggers. In most circumstances it was accompanied by under-informed and wildly inaccurate comments and extrapolations about the BBC from people who have either never worked there, or not worked there in the better part of a decade. (I’ve also spent far too much time over the past couple of days deleting comments along the lines of “this is why we should abolish the licence fee”, strangely most of these comments coming from Americans.)

A lot of people missed the point of what I was writing about, which I’ll attribute to me not making myself sufficiently clear. I was not talking about the BBC’s web sites, about the quality of the corporation’s creative output, nor about what the situation was prior to the sell-off of BBC Technology several years ago. What I was talking about are the reasons that today, in 2007, the BBC is stuck with an archaic technological infrastructure, and how that ultimately hurts the BBC’s long-term strategy to be “part of the web” and not just a set of web pages.

You see, despite the critical handicap of its infrastructure, the BBC does amazing things with its content. It takes a little longer than the rest of the industry to get those things out there, but considering the size of the corporation, the all-encompassing nature of its audience, and its risk-averse leadership, that’s not surprising.

It’s also true that the BBC suffers from multiple personality disorder, and that within the corporation not all things are equal. As was pointed out in the comments on my article, BBC Journalism (what used to be BBC News) has its own infrastructure that lies more within the control of the BBC. The difference is clear – BBC News has to be able to respond in the most flexible and reliable manner imaginable as the changing pressures of world events and audiences demand more of their infrastructure than almost any other in the world. Waiting months for bug-fixes and code reviews to make minor changes would be the kiss of death for the BBC’s News offerings.

That’s really the crux of the matter. The BBC certainly can be slow to react to change, but over the past couple of years there has been enormous pressure from within the corporation to move forwards with this whole “internet” thing. It’s safe to say that the vast majority of the people working in on-line content at the BBC do actually “get it”, and even the traditional linear media people are catching on. There are some truly exceptional projects coming out of the BBC in the near future (especially on the CBBC and CBeebies side of things), and it’s not fair to tar everyone with the same brush.

A brilliant example of what the BBC does right is described in Curtis Poe’s response to my article. He says:

[T]oday when I got into work, I found a scathing email from one of the higher ups. He read the “BBC Fails at the Internet” post and rather than blow a gasket that internal details had been made public, he forwarded it to the responsible parties, said he agreed, and made it very clear that the problem will be fixed immediately…

Now, wouldn’t that be a fine thing?

Perl on Rails – Why the BBC Fails at the Internet

Perl on Rails is a project by the smart chaps over in BBC Audio and Music Interactive that replicates the Ruby On Rails MVC framework in Perl. They’re obviously rather proud of themselves, and I understand that internally the project is making waves. Whilst I applaud the technical achievement of the individual developers, I deplore the situation that has forced them to do this.

The problem is that the BBC doesn’t control its own technical infrastructure. In an act of staggering short-sightedness it was outsourced to Siemens as part of a much wider divesting of the BBC Technology unit. In typical fashion for the BBC, they managed to select a technology supplier without internet operations experience. We can only assume that this must have seemed like an acceptable risk to the towering intellects running the BBC at the time. Certainly the staff at ground level knew what this meant, and resigned en masse.

Several years later this puts the BBC in the unenviable situation of having an incumbent technology supplier which takes a least-possible-effort approach to running the BBC’s internet services. In my time at the BBC, critical operational tasks were known to take days or even weeks despite a contractual service level promising four hour response times. Actual code changes for deploying new applications were known to take months. An upgrade to provide less than a dozen Linux boxes for additional server capacity – a project that was over a year old when I joined the BBC – was still being debated by Siemens when I left, eighteen months later.

The BBC’s infrastructure is shockingly outdated, having changed only by fractions over the past decade. Over-priced Sun Enterprise servers running Solaris and Apache provide the front-end layer. This is round-robin load balanced, there’s no management of session state, no load-based connection pool. The front-end servers proxy to the application layer, which is a handful of Solaris machines running Perl 5.6 – a language that was superseded with the release of Perl 5.8 over five and a half years ago. Part of the reason for this is the bizarre insistence that any native modules or anything that can call code of any kind must be removed from the standard libraries and replaced with a neutered version of that library by a Siemens engineer.

Yes, that’s right, Siemens forks Perl to remove features that their engineers don’t like.

This means that developers working at the BBC might not be able to code against documented features or interfaces because Siemens can, at their sole discretion, remove or change code in the standard libraries of the sole programming language in use. It also means that patches to the language, and widely available modules from CPAN may be several major versions out of date – if they are available at all. The recent deployment of Template Toolkit to the BBC servers is one such example – Siemens took years and objected to this constantly, and when finally they assented to provide the single most popular template language for Perl, they removed all code execution functions from the language.

So talented, underpaid, and frustrated software engineers at the BBC are forced to make a decision. Either they can produce websites using static HTML, and make a few remote calls to limited Perl functions, decorating their page with SSIs, or they can fight against a reticent and incompetent technology supplier to make use of a crippled and outdated language on servers that more than likely are unable to meet the capacity requirements of a dynamic application being used by the BBC’s audience. Software engineers at the BBC must become masters of the sleight-of-hand, using every smoke and mirrors tactic they can to conjure the appearance of dynamic websites, not exactly what you would expect from one of the largest media corporations in the world. Oh, and if you’re an external agency working for the BBC and hoping to write a new application or build on technologies that the rest of the world has taken for granted for the best part of a decade, you might as well forget it. There’s only one externally available development server, and it’s not in synchronisation with the live environments.

It doesn’t have to be this way. If, instead of forcing its teams to waste valuable license fee payers’ money on duplicating existing free software, the BBC decided to take control of its technical infrastructure and provide a viable platform for complex, dynamic applications, then that creativity, effort, and time could be directed at making more of the kind of applications that make the BBC great.

Some work is already progressing in this direction. A large part of the BBC’s Creative Futures project is what the BBC calls “BBC 2.0″ (often mistakenly referred to by executives and television-types as “Web 2.0″). The last I heard this was planning to deploy an architecture based around Java, Tomcat, Hibernate, Velocity, and MySQL. Whilst I disagree with the choice of technology for many reasons, this is at least an important step in the right direction for the BBC – as long as they exert control over the infrastructure from end to end.

It’s a ridiculous situation, and I know that many talented and respected technical staff have left the BBC in the past few years citing frustration at the insufficient technical infrastructure, and the inability of both Siemens and BBC management to keep up with the pace of technological change. Unfortunately, unless something dramatic changes with the upper levels of BBC management to recognise the nature of the problem, it’s a situation that will remain the status quo for a long time to come.

Tags: , ,