Why?

Because doing so can mean the difference between a successful project and a failing (or a really stressful) one.

Isn’t this something my project manager should be doing?

Well, yes and no. Yes because they should know, and they’ll be able to communicate risks to your stake holders and potential impacts to other teams and all that stuff. I say no because risks and dependencies tend to be intrinsic to the application domain and the technical nature of the project. It’ll depend how technical or how much of a domain expert your PM is of course, but generally a business analyst, architect, tester or developer will have a much better appreciation for these risks because they’re often way more involved in the messy details.

So what are risks and dependencies?

Risks have a well known definition, they’re usually broken down into having a likelihood and an impact. A dependency is anyone or anything you depend on in the project. Dependencies are risks. The reason I’m keen to say “risks and dependencies”, probably, is that dependencies are often the most common form of risk but are rarely seen as such. At any rate, your mission is to avoid them both. Some projects will have a “risks register”, this is a good idea and reviewing it and keeping those risks in mind throughout the project is essential. But it’s not enough to just “register” risks, you need to mitigate them too, otherwise your risk register is only as good as “I told you so” when one occurs.

An example. We have a digital video ordering system that runs internally to our organisation, and we send these orders to other organisations. Orders can and do take several hours or days to be completed, and we need to know exactly when they’re completed or if they can’t be completed. Initial thoughts were that those other organisations would let our system know by making web service calls on our system to the effect of “order X completed”, or “order Y failed, because …”. This would mean that we would need to punch holes in the firewall to allow these inbound web service calls in, and all the security around who to let in and who not to, etc. The network guys are already shaking their heads, they’ve done it before but you just know it’s going to take ages (and cost a lot.) But as it stands, our project can’t go ahead without it.

This is a major dependency and all the head shaking turns it into a proper risk too. We can write the software fine, we know how to do that, but there’s a big possibility it’s just going to be sitting in source control somewhere waiting to go live because the network has been put in place yet. That stuff is depressing, and it quickly turns ugly as product owners are getting shirty, network guys are still waiting on firewall routers to be delivered and the dev team’s moved on to something else.

Ok, it’s a major risk, what can we do about it? Well, lets redesign our ordering system to periodically call theirs for any recently completed or failed orders. This means we only have outbound connections and the network guys are happy. Even better, we can go live as soon as we’ve finished without waiting for other teams to finish. (As a bonus, we’re able to debug the ordering system from our laptops too because it doesn’t matter where outbound connections are coming from.) If we really want inbound calls too, for whatever reason, we can do it later. Better to have something released and working than to have it perfect the first time.

This was just one example of many. When you look around there are simply loads of risks and dependencies, big and small, and mitigating or eliminating them will make for early stress-free releases. Just another thing to keep you Agile!

Waltzing with Bears, by Tom DeMarco and Timothy Lister is some excellent reading on the topic.

The Framework

Us geeky lot are into mathematical sensibilities, we often talk of “beautiful architectures” and “dashingly simple algorithms”, “breathtaking abstractions”. Lovely stuff – mostly what keeps us in the field is what I’d say. We like to generalise, to use simple but powerful concepts to make our lives easier, we love using frameworks that with only a few tweaks can let us change the world. Power, flexibility, generality, expressiveness. One solution for all. We’re suckers for beautiful concepts that simplify the otherwise chaotic real world.

And when we get started on a new project, our minds are constantly breaking business rules down, looking for patterns and languages and the concepts that could glue the whole thing together. And if you get it right, you can vastly simplify a complicated project, and when you do, you’re beholden to something truly beautiful. You’ll stay awake all night just thinking about it.

This brings me to my second real-life story of just such a framework. Before we get started I have to say that I wasn’t directly involved in this one, not to say that it’s particularly embarrassing, but I happened to observe it from across the room.

The big idea in this framework was a messaging and communication system for disparate and offline agents. Agents could send other agents messages whether they were online or not, and the next time the agent logged in, it could download the latest messages for it to process. Think email, but make it more reliable and put it in the control of automated applications instead of people. It was an interesting idea, not entirely revolutionary by any means but still interesting. And we had a need for it, we had several applications that required on- and off-line usability from various mobile platforms (“mobile” as in laptops, tablet PCs, PDAs, etc). But desperate need or not, the project had been going for 1.5 years by the time I’d heard of it and it still had a huge backlog of work to be done before it could be released. In the mean time, all our desperate applications had put together rudimentary communication systems to solve their own particular offline problems. Thoughts on this were of two general themes, first, that because our various application stacks had already solved the problem we didn’t especially need a single unifying framework for all, and second, that all these applications solved pretty much the same problems completely differently was precisely why we needed a single unifying framework. Both arguments were correct in their own way, and the project investment backed both at the same time.

Where this project went seriously wrong was not in concept, it was a fine concept, it was just that it never actually released anything. By the time I got involved in it, probably 2.5 years in, it was because I was designing one of those offline mobile applications myself. The scope of the framework changed immediately. Talk was of redesigning the framework to include support for my particular application’s needs, and to roll it out with all sorts of amazing features. Bumbling around adding new scope, they’d effectively extended the project by another 18 months. Meanwhile I was busy looking into quick alternatives, the last thing I wanted was for my own project to be extended by 18 months too.

The problem with the framework, as a way of delivering software, is that they’ll trick you into thinking that you have to finish it before you can start writing any real applications with it. It’s like saying, “To build this application, we’re going to write a framework that will allow us to write the application really quickly.” A perplexed product owner might ask, “And how long is it going to write this, um… framework?”

Lets say you had a 6 month project with two features, A and B. Features A and B share a lot of common functionality so you decide to put that functionality into a single unified framework. Not a bad idea by any means. But your project now looks a bit like this: you work on the framework for 4 months, then a month for feature A, that’s 5 months, and now you can release. This is the earliest release you can make. Then another month and you can release feature B, 6 months. It would be much better to split the time spent on features A and B evenly: feature A takes you 3 months, then feature B takes you the next 3. In the first 3 months, you’re building just enough framework to get feature A, then the next 3 months extend and refactor the framework just enough to get you feature B. The point is, features A and B drive out the framework and not the other way round, that way you don’t waste your time building too much framework. Also, and this can be very likely, after the first 3 months your company gets a new business opportunity and would prefer feature C instead of feature B. But that’s ok because you’ve not spent any time working on feature B yet, nothing’s wasted.

In the end, both projects (mine and the framework) were cancelled. The company lost a huge source of revenue (unrelated to either project) and was tanking pretty hard, so push came to shove and anything not making direct revenue was dropped. If only we’d released sooner, either of our projects, we might just have made the cut.

I have scant evidence for this statement, I’m only a beginner, but nevertheless, it sounds like a “true fact”. As opposed to a normal “fact”. But I’ll plough on, I’m pretty certain that scope can make or break projects, and I’ve got some stories to back up my dubious claim.

First steps though, I’m going to group these concrete examples into Anti-Patterns, as that’s now become a pattern-in-itself for describing random stuff in tech blogs and books. Ok? Here’s the first one:

The Rewrite

Back when I started out I would get dumped on some ugly looking project to fix bugs and add meagre functionality. It was fun for a couple of days because you knew the system you were working on was really “important” and you’re basking in all it’s glory. A few days later and you’re stuck in a pit of tar – even the smallest changes are like a butterfly to a hurricane and before you know it you’re shit-scared to do anything. The system’s well f£$ked, to put it bluntly. Your daily/hourly/minutely thoughts are “what on Earth?” and “Oh my God!” and the now canonical “WTF?”. If only you were given a chance to do it all over, you’ve got plenty of good ideas and you just can’t wait to be cutting clean code again. You suggest a rewrite: “based on my professional experience, which is (by now) extensive, this project is in the toilet and we’ve got to start over.”

If only I could eat those words now. All of them; there must have been dozens of times I’ve said that…

I’m sure you’ve all been on such projects and screamed out for rewrites, but in this story we rewrote for different reasons.

This project was a lovely little C++ desktop application. It had a few bugs in it (which is what I was there to fix) but it largely worked. The main problem was the C++: nobody in our C#/.Net shop knew what the hell was going on and new feature requests from users were just being avoided. Then finally good news came down from on top, we’re getting to rewrite it! This time we’re going to use the latest and greatest new tech we’d all been experimenting with in our spare time and the project was swarmed with enthusiastic hard hitters. I was in there early, writing a lovely little pull-based workflow engine, accompanied by an IronPython based DSL. Months went by and enthusiasm wained, deadlines went by and project managers were getting stuff thrown at them. Still more months went by and even the slightest reasons were given for not deploying it. Even though it was technically finished, the users couldn’t just upgrade from their C++ desktop app to a ASP.Net app over night.

It was deployed eventually and it now serves its purpose well, but the dev team did near on nothing else for 9 months! And you can imagine, all the way up to that deployment date the value we’d delivered was none. Zilcho. Had the project been pulled, or someone decided it wasn’t fit for purpose we’d have wasted some big investments. Further, we had not much idea if we were building the right application, we had little to no feedback during the whole 9 months right up until we’d released it.

The problem with rewrites is that the scope is fixed. There’s no way you can release the rewrite until it’s entirely feature complete viz-a-viz the old version. Not only that, but you have to consider data migration, user migration and dependent system migration – all things we gave very little thought to in those heady early days, discussing beautiful Ajaxian pages and cute domain-object validation. Instead we should have concentrated on getting a release out in only weeks, we should have asked questions like, “how can we add features to a C++ app with ASP.Net?”. And answered with things like “Host parts of the C++ in ActiveX in the browser.” Or, “host a browser in the C++ app for new functionality.” Neither sound sexy, but we could have delivered new features every 2 weeks and got valuable feedback from actual users and given them real control over what to release next.

This is a very common scenario and I’ve heard much worse. What’s more, in most of the cases I’ve known, the projects were all run “agile”. They ran in sprints, had standups, had reasonably good communications with product owners, and even had Kanban boards! One project in particular was well over budget and especially frantic in the end, all due to fixed scope and an inability to release early and often.

[A] Utopia is conceived as a perfect society, and anything perfect has automatically ceased growing; even [William] Godwin qualified his rash claims for the perfectibility of man by protesting that he did not mean men could be made perfect, but that they were capable of of indefinite improvement, an idea which, he remarked, ‘not only does not imply the capacity for being brought to perfection, but stands in express opposition to it.’

 George Woodcock, Anarchism.

I’ll tell you why: I’m trying to find a solution to an age old problem that is particularly prevalent in enterprise software. I mentioned this in my last post, that enterprise software can be conceptually very straightforward yet implementations can be surprisingly complex. My theory (a simple one, perhaps too simple) is that the complexity is borne out of aspects, which are often characterised as requirements that don’t affect the algorithmic nature of the software, but do affect how the software operates or is deployed.

How an algorithm operates or executes could be a separate concern, even though is most of the software I’ve written and seen over the years, the execution of an algorithm and the algorithm itself are inseperable. What do I mean by “operate” or “execute”? Lets take a simple example, to add two numbers to produce a third: c = a + b. Often this statement is executed in memory, using the CPU to add the two numbers together and pop the result into a local variable, usually found on the stack. Or perhaps a more abstract example, but no less simple: y = f(x), a function f to transform x to be assigned to y. These statements are our algorithms, it so happens that they’re so simple that not many people would actually call them algorithms, nevertheless, they define the computation that is to take place. The execution of these algorithms on the other hand, may be entirely variable. It may be the case that function f is very expensive and time consuming and we choose to distribute it over several machines. Values representing x are queued up and machines executing f just take new x values when they arrive, producing a series of ys as output.

As it turns out, a project I’ve been working on is pretty much exactly y = f(x). We’re transcoding 5,000 hours of flash video with a load of Linux boxes running ffmpeg. (ffmpeg is freakin awesome for transcoding high quality H.264 flash files.) WMV files come in, transcode magic happens, MP4 files go out. Conceptually rather simple but we have stuff to deal with like, for example, copying input files to each of the machines (provisioning) and taking the output files off (harvesting.) At some point we’ll want to know which machine is running and which not, from a monitoring point of view, logging various activities, errors etc. Nothing especially complicated, but the original y = f(x) is quickly lost in several hundred lines of Ruby scripts, running on several different machines.

My goal is slightly ambitious, I’d like to write y = f(x) first, then make decisions about how the computation is distributed, how the queues work and over what protocols, whether the need for outputs (the ys) pull the computation, or whether the presence of inputs (xs) push it. It may be fine that f is executed once for each x, or that there can be efficiency gains in processing multiple xs at once, so we batch up xs before executing. If we can separate these concerns from the basic algorithm, surely we’re better off!

A possible solution can be found in the ideas that underpin dataflow languages (see Pure Data, Mozart/Oz, Joule and the Wikipedia page). Dataflow languages make explicit the notion of processing streams of data. Processing nodes take inputs and produce outputs, and all you have to do is wire the outputs up to the inputs of other processing nodes, etc, etc, then just throw in loads of data as input. The common command line shell is a dataflow language, what with it’s pipes and all. One command pipes data to another, and the shell (Bash or whatever) will buffer the data produced by the first that hasn’t yet been consumed by the second.

The main difference between dataflow languages and normal languages is that they operate on streams of values instead of single values. The interesting for us is what happens between the processing nodes, or functions. The functions are typical, they would transform input values into output values. Some functions may be more efficient at processing several values at once, but for all intents and purposes, functions are functions like you know them to be. What happens to the data between the functions is where the fun begins.

For the sake of discussion we’ll define a process: z = f(g(x)). We could also represent it like this:

x → g() → y → f() → z

As I was saying before, run this statement in any regular programming language and it will use CPU registers and the stack to store the values represented by x and z, and also (careful not to forget) the intermediate value generated by g(), we’ll call y.

But we’ll have other considerations to worry us. Let’s mix it up a bit and suggest that g is a service that is invoked asynchronously, the input x is sent in a message to g, and the result y is sent back. (Pretend that g takes 3 hours to compute, it may make sense that it’s invoked asynchronously.) Notice that this doesn’t change the algorithm one bit, but it does change how it executes. g itself doesn’t change, but how it is invoked, how x values are ferried to it and y values are ferried back has changed compeltely. Can these concerns be added to our simple little program without having to rewrite it?

What if f is a manual process? Somebody is manually taking ys and producing zs. People tend to work only 8 hours a day but with g being automatic and mechanical it can run 24/7. We’re going to need to store the ys that g produces that the people involved in f haven’t got around to seeing yet. Ah, persistence! We’ll store in them in a database!

y used to be a CPU register or local variable in the stack frame, now it’s a database table. Ultimately, as programmers we’d like this to be a one line change in our source code, not a rewrite.

There are loads of other things you may want to do with values while they’re in this limbo zone: log them, store them, send them across networks, distribute them between load balanced services. There are plenty of things you may want to do with functions too: have them run in multiple threads, or on multiple machines, monitor their progress, or simply execute them on the same thread, synchronously and as fast as possible.

Being a programming language geek and all, I naturally think of this in terms of, well, programming languages. Ultimately then, the goal is a programming language that allows this sort of flexibility. The programmer can write algorithms that do the work, and following that, configure them for all the other requirements. The key expressiveness of this language is scalability, it scales from the smallest simplest programs that run entirely in memory using CPU registers and stack space, right out to running them across hundreds of machines spread across the Internet, all with minimal source code changes. Seriously, z = f(g(x)) might become something like:

x -> g() -> y -> f() -> zwhere    x: atom-feed http://some-atom-feed.com/incoming    y: db table_name, "connection string"    z: email some-dude@poor-suckers.co.uk

For this scaling to be possible we need to be able to make certain assumptions about these programs. We have to assume that a function f can be run in parallel, the easiest way to ensure this is for the function to have no side-effects, it is what functional programmers refer to as “pure”. We have to assume that the messages or values in queues between functions aren’t lost. I’m not sure if this wonderful language should enforce these sorts of things or not, or if we leave it to the programmer’s best judgement.

But a new programming language is a big project, and of course nobody would use it for years and bloody hell, you’d have to write the thing and who’s going to do that? Couldn’t we just use these principles when we write applications, normally, today? Sure!

First of all try to divide your application into three areas:

1. Pure functions, code that takes arguments as input and produces results as output, no side-effects!
2. Queues, code that stores or loads values from some sort of data store, either in memory or in a database, on an ATOM feed, whatever.
3. Execution environments. This is the code that drives the algorithm, picking values off queues and executing functions. An execution environment may drive functions by pushing the inputs or pulling outputs. Execution environments have some overlap with queues, for example some queues may need to be polled periodically for new values, others simply invoke some sort of event handler when new values are ready. There may be some scheduling to do, in the case of in memory function calls (as in most programming langauages) the queues are one item memory slots, and the runtime ensures that functions are executed in the right order such that each of these memory slots has exactly one value in it.

If an application has been separated into these three areas, (I’m guessing) it should be easier to swap in new configurations, and hence, new aspects. We should be able to reuse a durable queue implementation for all message types, to be plumbed between any two given functions. We should be able to reuse execution environments for different algorithms and different queue implementations. Hopefully this would lead us to a programming style or architecture or pattern that gives us flexibility to scale up or down or sideways, inserting new requirements without resulting in that dreaded hideous complexity so endemic in enterprise software.

What this is trying to do in effect, is make the marshalling of values and the execution of functions abstract. To look at a normal programming language, let’s say C#, or Ruby, they bind way too early on how data is transferred between functions and you have no option to change it. Data is always passed in memory. Functions are always called synchronously. Any time you want anything different and you basically have to program it yourself. But we do need flexibility for these concerns for enterprise software to be any easier.