Agile and Waterfall for Innovation

Agile methods are iterative and incremental. This, in theory, should prevent implementation death marches which end up with products that do not meet the customer’s needs.

Waterfall on the other hand, is all about having predictable stages with clear milestones at the end of each stage. There is no concept of iteration or increments. All of a stage (like design) are done, validated and only then is the next stage started. The concept here is that validation at the end of each stage keeps implementation aligned.

Unfortunately, both say nothing about the twin human-factor problems of over-excitement and incompetence of the people involved.

Often well understood and repeatable projects (like building a house) follow a waterfall.

Agile suits more ‘non-repeatable’ projects such as building a software product where each product will have its own challenges, risks and ‘hidden dangers’ – while being driven by changes in the ‘business’ environment. Therefore it is very important, when doing Agile for new and innovative products to:

  1. give clear guidance about what is working and what is not working back into the process 
  2. keep overall focus on the problem that the product is attempting to solve (i.e. always keep in sight that gold paved road to sales)

If Agile allows software to ‘flow’ freely, then it needs a proper pipe for it to reach its destination (i.e. the hands of the customer). If the pipe shape keeps changing, or has leaks in it there is no way the software will reach the right destination.

One thing, very easy to do (especially in a startup environment) is to get too excited about the problem domain without staking out what specific parts of that domain the product addresses. What is even worse is not sticking to it once identified! This is because Agile methods can be abused to hide (but not ultimately solve) problems with changing requirements and scope creep – resulting in big failures.

This process is made more difficult by the fact that for a new product idea one needs to find a gap in the market. The irony is: bigger the gap you find (Total Addressable Market – TAM) – more funding are you able to attract on the basis of future demand for your product – bigger the promises made – higher is the chance of getting lost in multiple interesting aspects of the ‘gap’ especially if there are conflicting views in management or if the gap area is not well understood.

Here multiple sources of tension exist: what constitutes a businesses view of the minimum viable product (MVP) and how is it different from the potential customers view (ideally both should be the same)?

The answer to the question ‘which part of the MVP should we do first’ – is the launching point for the Agile process. Ideally – a set of features are decided and then iterative and incremental development starts. As long as there is tough resistance to the business asking for massive changes to the path and clear feedback into the development from the ‘customer’, the end result should be aligned with what initial expectations.

I believe for big gaps where both investors and company owners see big $$$ signs, the so called ‘disruptive innovation’ – it may be a difficult thing to start off with Agile and maintain the discipline of clear feedback and clear definitions of done in terms of the MVP. In such a case it may be good to start of in a waterfall model – with low expectation of success, and then do Agile. Hopefully with one attempt at waterfall, one will end up with a product that can be put against the MVP concept, deltas calculated and then fed into an Agile process to be filled incrementally.

Why start with waterfall? Because waterfall imposes a strict condition of no-iteration. So it is more difficult to abuse it. It forces you to commit to requirements to do some design. To commit to design to write some code and so on. And as I said – in the end it gives you a good target to destroy when you start the Agile method. It can also give strength to push back on requirement changes and scope creep later on.

One may say that it is a waste to do waterfall. But one must remember in an Innovation, new product environment usually the target is not to ‘boil the ocean’. So it may be possible to quickly attempt waterfall to get a starting point for Agile. Also in most innovation environments, the initial team size and skill distribution does not allow for a proper Agile implementation in any case. For example it is not typical to find abundant testing or quality assurance resources.