Agile vs Waterfall: enterprise software development for financial services

Since modern humans first used cave drawings to depict game, and more than likely the process for capturing it, we have continued to document and work with processes.

Within the 21^st century the newest methodology on the block for software development is agile, and its purpose: to bring back the focus to what really matters; the customer.

Agile has become the way for so many sectors and industries, so much so that it is often mis-implemented.  The U.S. Department of Defense (DoD) even developed a document to aid in “detecting agile BS”.

How has it come to this? And how can you navigate this minefield to truly oversee proper implementation at the enterprise level across siloed teams?

Understanding if Agile is the right fit

Software Development Life Cycles (SDLC)

Software is delivered to a customer in a series of steps that can be structured into 7 phases. SDLCs came into place to help solve problems within development activities of the 1980s and 1990s. The first of such SDLCs being the waterfall methodology, with others including iterative, spiral, and more recently agile.

Waterfall takes each phase in turn before proceeding to the next step, often implemented over the course of months with little customer feedback in the early stages. Whilst this works well for projects that are mission critical and the risks involved may result in loss of life, including building space rockets, bridges, and vehicles within the automotive industry, this may not be the most suited approach for projects that want to maximise the customers’ experience.

Agile, aptly summarised by Oliver Peterson is “basically a philosophy of software development that prioritizes iterative development of working software and solutions through cross-collaboration and self-organizing teams”.

A deeper understanding of Agile

The key elements of agile, for the purpose of bettering the customers’ experience, is cross-collaboration and self-organisation within the development teams.

Agile in 2001 began as a set of 4 values and 12 principles set out in the Agile Manifesto.

Three key frameworks have been synonymous with the development of agile. These include Scrum, Kanban, and Extreme Programming. If you’re looking for an in-depth study of these areas, and how they relate to lean manufacturing, Darren Hagman does an excellent in-depth job of explaining each of these buzzwords in turn.

The litmus test you can use for detecting ‘Agile BS’

Our responsibility at the enterprise level is ensuring we have a good understanding of the available frameworks, however more importantly is the ability to facilitate true development. For this, referring back to DoD’s 6 key questions (outlined below with key elements highlighted) provides us with a simple litmus test that can be carried around and applied to any situation to determine if we are working to agile.

1. Are teams delivering working software to at least some subset of real users every iteration (including the first) and gathering feedback?
2. Is there a product charter that lays out the mission and strategic goals? Do all members of the team understand both, and are they able to see how their work contributes to both?
3. Is feedback from users turned into concrete work items for sprint teams on timelines shorter than one month?
4. Are teams empowered to change the requirements based on user feedback?
5. Are teams empowered to change their process based on what they learn?
6. Is the full ecosystem of your project agile? (Agile programming teams followed by linear, bureaucratic deployment is a failure.)

If “no” is answered to any of the above questions, then you are looking at “Agile BS”.

The simplicity of this test allows for quick analysis to be carried out. Ensuring these are culturally entwined within the organisation is the more difficult task, a task that requires a comparative study for your organisation’s needs and ambitions; a study between what’s currently being implemented, and the additional benefits from adopting agile at a level that is true to its core.

You may find from conducting such a study that adopting agile in its entirety may not be the best option immediately going forward. Choosing the most appropriate agile practices may be key as a first step to iterative changes in the organisation.

Applying Agile outside of the traditional software sectors

Over the past decade, a new wave of agile fintech companies have sprouted to tackle the challenge posed to banks failing to tackle financial crime, to the tune of $30 billion in penalties between 2009 to 2019.

Banks are needing to meet their anti-money laundering (AML) requirements, and are overwhelmed by the large know-your-customer (KYC) due diligence backlogs.

How BBVA adopted agile

Adopting an agile approach has allowed these businesses to iterate development more closely with the customer. A case study by BBVA highlighted the transitional change over a period of four years, by applying agile approaches to particular departments.

They found that having dual organisation (both traditional and agile) without fully committing to agile was not as effective, contravening DoD’s sixth question of the litmus test. By transitioning gradually to a full agile department, results in improving product quality and accelerating time to market came within a year.

Standard Bank saw positive results

Standard Bank, based in South Africa, trained their staff to follow the Scaled Agile Framework (SAFe) in late 2016 for adoption at the start of 2017. The results of their changes include:

• Time-to-market reduced from 700 to 30 days.
• Cost decreased by 77%.
• Productivity increased 50%.

Standard Bank had tried waterfall and combined team approaches in the past, but neither addressed the challenges across fragmented silos compared to cross-collaboration between their 2,000 IT systems through a change in culture.

Why Cisco moved away from Waterfall

Aside from banking, Cisco in 2015 decided to apply the agile approach to their subscription billing platform that used to follow the waterfall approach. Separate teams used to be used for each of the SDLC phases.

They used SAFe for this product, in combination with Scrum for another product, and extreme programming practices (such as test driven development and continuous integration).

The results of these changes meant a 40% decrease in critical and major defects, staff not working overtime, and delivery on time for product increments. A lesson learnt from this example is the ability to utilise different agile frameworks for different product developments within the same organisation.

When NOT to use Agile

Complex linear production

Chuck Cobb argues that agile methodologies are best suited for projects with higher levels of uncertainty surrounding creativity and innovation rather than focus on predictability, planning, and control.  And the example he cites is that of building a bridge, whereby planning is of utmost importance before getting underway.

However this notion seems not to be set in stone, as we are witnessing agile methodologies being applied to even the construction sector. Through agile, project management of construction projects could help to reduce time between identifying and resolving a problem. The only element that cannot be carried over is putting off a major design decision to a later stage. This is due to the linear nature of construction; the 3^rd floor cannot be built before the 2^nd floor is constructed.

Furthermore, with developments in technologies such as virtual reality, augmented reality, and digital twins of construction projects, clients and stakeholders can be part of the feedback process early on with virtual twins of what might be the final build. New and emerging technologies may facilitate agile in ways that were not possible not so long ago.

Product evolution

If the product you are developing is too far along the product evolution timeline, then agile is not the right fit.  The 4 phases of the product evolution starts from novelty, before progressing to stability, commodity, and utility.  Electricity in the 19^th century began as a novelty. Over time, we now expect electricity at our fingertips and use it as a utility.

Curtis Poe’s simple answer is this: “the newer the product and its market, the better fit with agile”. Curtis recommends you make this decision once you understand both agile and the product evolution. He also recommends alternative methodologies for the other phases. Use the lean methodology if your product is at the stability or commodity stage, and Six Sigma if it’s at the final utility stage; where reproducibility is key and customers do not want, nor expect change.

Adopting Agile at the enterprise level and facilitating it

Due to legacy systems existing in many banks, making a shift to agile can seem like moving a mountain. As we saw with the case studies above, exploring SAFe as a framework to adopt in the first instance could be the most suitable option going forward, as this allows for cross-collaboration at the much larger enterprise scale across organisation silos.

SAFe has 3 key elements:

1. Program Increments (PIs) – self organisation of teams to deliver against agreed goals in a collection of sprints (usually five).
2. Innovation Sprint – The final sprint of a PI that assesses the previous sprints, and identifies improvements and innovation opportunities for the next PI.
3. Release Trains – Provides structure by specifying deadlines for all teams to deliver functionality across the program.

Chris Vandersluis – in his conference paper that talks about applying agile at the enterprise level – outlines three pitfalls to watch out for when seeking to implement agile once you identify that it’s the right fit for your enterprise project:

• Do not try to adopt all of agile all at once. Pick and choose appropriate practices for particular situations. Perhaps begin applying agile to a particularly small product developments to test the waters ahead of any large scale enterprise level project.
• Do not feel the need to drop everything, including critical path schedules, risk analysis, and quality control. Retain your project management skills.
• Apply changes to the organisation iteratively. There will likely be resistance, and agile is about respecting the people working on its development, not forcing the framework and creating an agile industrial complex.

Key takeaways

Looking to apply changes at the enterprise level firstly requires you to test the waters with smaller projects.  Implement changes at a granular level for smaller product developments, and compare the results of the activity to previous benchmark results.

Once ready, if you’re a financial institution ready to tackle the challenge at the enterprise level as seen in the case studies, explore implementing agile for KYC or AML to tackle the industry’s most looming challenges.

Look for combinations of new and emerging technologies that could help to facilitate the agile approach in a way that may not have been possible in the past. Just as the construction industry is facing these major changes thanks to the advancements of a number of technologies that we are still witnessing.

And finally, have a good understanding of your product’s position in the product evolution timeline. Do not try to shoehorn agile where it will be doomed to fail.