Tuesday 18 November 2014

Cynefin Complexity - a new perspective on old Wickedness

The Cynefin Framework overlaps with earlier work on Wicked problems. Both have things to say about sophisticated software development, and both point towards agile techniques for delivery. DSDM seems to have a trick up its sleeve to meet the challenges of wicked problems.

Something New

The Cynefin Framework
The Cynefin Framework - wikimedia commons

In the past couple of years Dave Snowden’s Cynefin Framework has captured imaginations in the software industry. Snowden proposes that problems and activities exist in one of several domains through which they can be better understood: Simple, Complicated, Complex and Chaotic:

I'm not going to go into the framework in any great detail here. You can find out all about it on Wikipedia, on Snowden's own company, or all over the web.

Snowden recommends operating in the Complicated and Complex domains, and some commentators have suggested that this is the natural mode of software product development. Day-to-day coding sits largely in the Complicated domain, which requires specialists who can apply acquired expertise and perceive Good Practice solutions. Product and project thinking sit in the Complex domain: we make a small change, understand its effect on the product and on users, and modify our plans accordingly. It’s a nice characterisation of agile approaches.

Something Old

Wicked problems were first characterised in the late ‘60s and early ‘70s:

  1. The problem is not understood until after the formulation of a solution.
  2. Wicked problems have no stopping rule.
  3. Solutions to wicked problems are not right or wrong.
  4. Every wicked problem is essentially novel and unique.
  5. Every solution to a wicked problem is a 'one shot operation.'
  6. Wicked problems have no given alternative solutions.

This is a great description of software dev:

The problem is not understood until after the formulation of a solution.
The delivery process explores its own requirements, and invariably exposes complexities that the client hadn’t considered.
Wicked problems have no stopping rule.
There is such a thing as too little development and too much, and often a large grey zone around 'just right'.
Solutions to wicked problems are not right or wrong.
Any business requirement has multiple legitimate solutions. Some will be better than others.
Every wicked problem is essentially novel and unique.
Otherwise an off-the-shelf solution would already exist and the work would be unnecessary.
Every solution to a wicked problem is a 'one shot operation.'
If we fail, (as an internal team or a contracted consultancy) the client won’t have the money or the time to try again. And if they do, it certainly won’t be with us!
Wicked problems have no given alternative solutions.
No off-the-shelf solution.

A Cynefin perspective on wicked problems

It feels like Snowden’s Complicated/Complex boundary has things to say about wicked problems:

The problem is not understood until after the formulation of a solution.
Emergent characteristics of the Complex domain.
Wicked problems have no stopping rule.
Probe - Sense - Respond applied to stopping as well as to continuing.
Solutions to wicked problems are not right or wrong.
Beyond the domain of Best Practice, into Good Practice and Emergent Practice.
Every wicked problem is essentially novel and unique.
Hence the need to Probe - Sense - Respond.
Every solution to a wicked problem is a 'one shot operation.'
Big Design Up Front is dangerous - again why we prefer to Probe - Sense - Respond.
Wicked problems have no given alternative solutions.
Hence no established Best Practice.

A new representation of old observations can be fruitful - Mendeleev’s arrangement of the periodic table is a classic example. I generally find Cynefin frustratingly vague, and Wicked problems provide it with a concrete context that I can understand. The Probe - Sense - Respond approach partially dismantles Wicked Challenge 1 (problem not understood until after the solution) and Challenge 5 (one-shot operation). Of course this is part if the core of the Agile paradigm.

Cynefin also nicely characterises what clients hire consultancies for. Simple problems they can manage by themselves. The Complicated and Complex domains encompass activity where specialists can add expertise and value. Even the Chaotic domain has value for innovation spikes.

Wicked projects and DSDM

Agile approaches meet Wicked Challenges 2-6 very well. They generally address Challenge 1 only in iteration - The problem is not understood until after the formulation of a solution. This is fine for ongoing product development, but perhaps insufficient for projects with a discrete start (and end).

Waterfall methodologies have a partial solution. PRINCE2 has an Initiation Stage with a go/no-go gate preceding full delivery. At the end of this, the team should have given due consideration to the problem space and to alternative solutions. But PRINCE2's up-front approach to planning, with an expectation of minimal engagement, makes it fragile to unexpected change. It doesn’t Probe - Sense - Respond.

DSDM has a nice trick here - an agile method with an 'Enough Design Up Front' mentality. It directly addresses Wicked Challenge 1 with explicit Feasability and Foundations phases. At the same time, its principles of ongoing client collaboration, iterative delivery and flexible, prioritised scope retain the responsiveness required in the Complex domain.

Wicked problems and #noprojects

In my previous post I suggested that there are three equally legitimate modes of software development: projects, product dev (which I aligned with #noprojects) and support. How do these modes support Wicked problems?

When clients hire consultancies, it is almost inevitably project work. A start date, a fixed budget and therefore an end date. We'd better hope then that projects can satisfactorily address them - I've made a case above that by combining up-front preparation with ongoing agility, they can.

Product dev teams operating in the #noprojects mould are ideally placed to engage with Wicked problems. On the ground on an ongoing basis, they can continue to Probe - Sense - Respond and keep improving the solution. This may make #noprojects a better fit than a project or an external agency, if the company can afford and manage its own team devoted to the product.

Support mode is a poor fit for Wicked problems. A team focused on support is trying to address discrete issues as they are raised, and is unlikely to give a Wicked problem the all-round consideration and the iterative attention that it requires. We've all seen support teams continue to fix point problems without addressing the underlying root cause. Once they step back and recognise a Wicked problem, the team often drops out of support into one of the other modes - project or product dev, whichever suits them best.

Wednesday 5 November 2014

Projects, #noprojects, language and thought

It's a truism that when you learn a different programming language, you learn a new way of looking at code. The language we use changes our abilities to think about what we're describing. The same principle applies to Orwell's Newspeak, the linguistic battlegrounds of social equality, to the fact that arithmetic is simpler in Chinese than in English.

So it's likely that the language we use to describe our work affects how we think about it and how we perform it. Last year for example a client pointed out to me an emotive stumbling block for participants new to agile methods - he found the standard terminology of Scrum alarmingly negative - sprint, backlog, grooming.

I've had roles with an interesting range of employers, on different types of of work. Real-life examples:

  • 'Lift-and-shift' of a website from one web stack to another, with no user-visible changes
  • Ongoing feature development of a website and its custom CMS
  • Development of a social network application, segueing into product support

All of these have been software development for long-lived products, and we've described all this work as projects.

But only one is a project. And the confusion is harming us, as practitioners and as a community.

Project projects and non-project projects

PMI describes a project:

It’s a temporary group activity designed to produce a unique product, service or result.

Temporary duration to meet scoped business goals. If that annoys you because it doesn't describe your work, bear with me for a minute.

By the PMI measure, my web lift-and-shift was clearly a project.

Ongoing feature development was not a project, and while initial delivery of my social network was a project, the current support effort is not. These efforts have existing descriptors - product development and product support respectively. Together they're sometimes classed as BAU (business as usual), distinguishing them from the one-shot (and disruptive) nature of projects.

Note that I'm not suggesting that one is 'better', 'more legitimate', 'more hardcore' or 'more professional'. Clearly these are all crucial software development activities and the language to tell them apart can help us.

Why does it matter?

Simply because the different types of effort have different delivery and (whisper it) governance needs. For the dev team, BAU and support need no spinning up - there's already a product, a UX framework, an architecture, domain knowledge etc etc. And save for occasional 'major' releases there's probably no need to think much about go-live. This makes them fertile ground for a wide range of fantastic lightweight agile techniques. Projects are harder, and often mean a new team, a new set of stakeholders and business domain, new architecture, a deployment plan, handover, training and so on.

For management - the lovely people paying us to develop software - there's a predictable operating expense to BAU (OPEX - the cost of doing business) to maintain and develop an existing product with known business value. Projects are a riskier proposition. They create change in the business - that's the whole point - which means rocking the boat. They take capital expenditure (CAPEX - investment over and above OPEX). The manager has a fixed pot of money and it might go anywhere - Project X or Projects A, B and C, another server, more marketing or straight into the bank to earn interest. She has to have some reason to believe that Project X is worth the investment, which means some idea of what she's going to get, when and for how much.

How our language harms us

We throw around the 'project' to describe all of these activities, and it harms us as individual practitioners and a community.

As an individuals, the language of 'projects' blinds us to difference between the models. After all, it's all programming (or design, or system development or...). We try to apply approaches suitable for BAU to projects, techniques for projects to support, etc. Or we simply get confused why two pieces of work are shaping up very differently. This is not conjecture - I've seen extremely intelligent and skilled engineers declare 'OK but it's all just projects.'

As a community, we're engaged in a divisive debate between estimates and #noprojects / #noestimates. Each side has its impassioned advocates, and they see each other as variously as hidebound or naive.

But the clue's right there in the name. If you're not doing projects, then the #noprojects paradigm might be an fantastic fit. #noestimates might clear a swath of unnecessary overhead - even enable you to make more accurate forecasts!

If you're doing projects, this doesn't apply. Your investors still need...a project and much that comes with it.

What doesn't this mean?

This doesn't mean that projects are waterfall and #noprojects is agile. It might mean that projects sacrifice a degree of agility for a degree of predictability. How far you end up along the agile spectrum may depend on the appetite of the investor and the deftness of the team.

In similar vein, it doesn't mean that there's a hard-and-fast divide between the models. The team doing a project within a product that they support. The ongoing dev effort managed as mini-projects. There are as many ways to blur the lines as there are individual circumstances.

Where from here?

As a software delivery professional, my role is to create clarity and shared understanding. Part of that is agreeing a shared language. So I propose the following.

Within the class of Software Delivery we have three (maybe more) subclasses:

  • Projects
  • Product Development
  • Support

Because they are all Software Delivery, some aspects of all three are very similar.
And because they have different technical and management requirements, techniques that are suited to one are not appropriate for another.

And perhaps we can re-frame the #noprojects debate. Rather than pushing dogmatic points on two 'sides' that now appear illusory, we can ask when are #noprojects and #noestimates approaches beneficial? What new tools can #noprojects and #noestimates thinking add to the seasoned software pro's armoury? And framed like this, might they even find applicability in the projects domain?