Fix underachieving software teams in any sized enterprise!

A White Paper on Agile Methodology

by Jan Machacek of Cake Solutions Limited

This white paper is also available as a PDF document. Please click here to download.

In this paper, I will propose an alternative to the waterfall method of software development that works in any sized enterprise including major development organisations. The new methodology is called agile, and it brings much needed change to our way of working.

Agile is the proven answer. Unlike traditional projects, agile projects are ready for change. Just as importantly, agile teams focus on delivering functioning software; software of excellent quality and with features that are always aligned to the business expectations.

If you are in a position where you cannot continue using the traditional delivery methods, because you are left with software that is delivered late, does not match the clients’ requirements, and the quality is not very good, then you need change. To most teams, software delivery must become interesting and challenging again; you must bring them back from the chasm of missed deadlines and being told that the software they have implemented is not what the business wanted. The vast majority of programmers take pride in their work; most take being told that their code is not good personally. This means that your team still have the knack. Your task is to help to keep them on the right track, allow them to write software that we are certain is good and get regular [good] feedback from the business.

In this paper, I will describe the steps you need to take to help you evaluate the problem you are facing and discuss how to make that change in even very large development teams.

The reality check

Before you can mount any rescue action, you must execute a reality check. During this exercise, you will need to examine the project’s code. You need to evaluate the overall quality of the source code. You are examining every aspect of software quality. You must evaluate the architectural clarity, the implementation quality and the quality of the tests.

In addition to checking the quality of the code itself, you must also check the quality and stability of the development process. You are looking for automated build infrastructure, centralised issue tracking, code review infrastructure and evidence of collaborative working practices.

The findings may be quite worrying. It is not unusual to see even large organisations completely without any kind of automated build processes; to see code without tests or to see tests that only satisfy the formal requirement to have tests. Finally, it is not unusual to see inefficient communication between the business and the technical team; and very little communication within the teams.

This is purely a fact-finding work. Do not attempt to solve the problems as you discover them— you must instead get a full understanding of all problems before you can start tackling them. Because you are not fixing any problems, the reality check should not take more than 2 weeks.

Bringing change

Now that you know what the problems are, you can start solving them. I will now give you guides as to how we go about fixing the problems and, in my experience, there are few problems outside the ones I will cover here.

Better requirements

You must start by ensuring that the technical team is doing the right thing! You probably have complex and detailed requirements documents, yet the technical teams do not understand them fully or do not use them at all. Complex requirement documents are not very useful; the detailed analysis in those documents very quickly goes out of date as changes are introduced to the system. With every added change, the document grows as the new change request documents are being added. The key problem is in adding changes. Businesses often believe that they need the historical record of all changes. This is fallacy. It does not matter what happened in the past, what matters is what is going to happen now and in the future and how we use the past to become better in the future.

In short, stop using your old requirement documents and invite the business and the technical leaders and start creating a product backlog. The product backlog is, in its simplest form, a spreadsheet that lists every story; and it groups the stories into releases. The key difference between product backlog and requirements document is that the product backlog is a living document that will continue to change. The backlog stories include the definitions of done. The definition of done describes what the business will consider a complete work; it typically includes steps that the business will take to verify that the story is indeed done. Finally, the product backlog is often (even daily) prioritised and refined. The refinement process allows the teams to understand fully the stories they have to implement in the next 2 weeks (the current sprint); to understand very well the stories that are most likely to be implemented in the next sprint and then includes coarse-grained stories that will eventually complete the product.
How is this different from the traditional approach? Traditionally, we define the priorities and features at the start of the project. We are arrogant to assume that we know every detail upfront, that we will never have to refine our understanding of the problems; even worse, we ask the business to do the same! We then report to the business that 79 per cent of the features are implemented and ready to be tested. How do we know that there are no defects in those features; how do we know that the features we have implemented are exactly what the business wanted?
You must give the business the power to change the software, the freedom to change their mind and as much information as possible to allow the business to make the right decision.

Better teamwork

It is crucial to help the teams communicate. The technical team needs to communicate with the business and the team members need to communicate amongst themselves. Now, by communicating with the business, we don’t mean that there is an analyst who pre-chews the requirements from the business and translates them into pseudo-programming language. No; the technical team members need to talk to the business directly, without any insulation. At the beginning of the process, this is uncomfortable, but it is the only possible solution. Next, the team members need to start communicating with each other regularly. The easiest way to improve the communication is to hold morning meetings with the entire programming team and the business team. The programmers will pick tasks from the product backlog (recall that the backlog is prioritised, so there is no doubt what the programmers should work on next) and ask the business team if they have any questions. This way, everyone knows what is happening in the project.

Every week, the teams need to refine the product backlog. The purpose of this refinement session is to understand the features that will need to be implemented. Finally, every two weeks, the teams hold a sprint planning meetings and sprint retrospectives. During the planning meeting, the technical and business teams suggest the features that should be implemented in this sprint. The technical team evaluate the complexity of the stories (remember, these are the stories that the teams have refined, so there should be no major surprises); the teams will then commit to completing the selected stories. The technical team explain to the business team the stories they have picked. This gives the business team the chance to appreciate the complexity of the tasks and to finalise the plans for the next 2 weeks.

Notice the continuous feedback: the business has a say in what is being done; the technical team has the opportunity to understand the business reasons for the features they are implementing. The short iterations (2 weeks) also means that there is never a big backlog of things (be it requirements or code) to check.

Better code quality

Now you are in the grunt of the programming. You must ensure that the code is of the best possible quality; not just as good as your team can produce, but simply as good as it can get. Bring in an expert if you must, but do not accept simply OK code. Make sure that the technical team understands that the code they write must be unit and integration tested and that the team members are responsible for testing. Do not allow the team to add testing tasks; especially if they’d like to push the testing to some other team.

Configure your development environment; check that the code can be built automatically every time a developer commits a change to the version control system. Check that the automated build system runs all test and that it notifies the team if there are test failures. Do not allow failing tests to remain in the system.

Better sign-off

Traditionally, the sign-off process is an unpleasant procedure. If only bugs are uncovered, then you are indeed very lucky. In some cases, the sign-off process reveals major errors in the requirements documents, which then resulted in errors in architecture. The impact is vast. In the agile world, you should never encounter major structural problems, because the iterations are simply too short to allow a problem to grow unchecked. The sign-off also happens regularly, even during the morning meetings. When the technical team completes a story (according to their understanding of the definition of done), they tell the business team, who check the story. It is the responsibility of the business team to verify the feature and say yay or nay.

How to begin

Start today. Pick a project that is in trouble and turn it around. In doing so, you will demonstrate to the business that there is indeed a better way of managing projects. You will show that the comfortable feeling of the waterfall-style management is fundamentally flawed and that Gannt charts do not fit software development process. Not at all!—use Gantt charts and Microsoft Project to plan building a house or a bridge (a strictly sequential process that does not change or changes very little). Software is a different beast: it changes constantly and is not necessarily sequential.

Find all facts, evaluate the quality of the code, the coding practices, the development set-up, and communication amongst the team members. Next, get a buy-in from the business teams. This is usually easier than you might think. The business love being in charge; they will very quickly realise that an agile process puts them in far more control than a traditional process.

Finally, and this is usually the most difficult task, help the technical team become agile team. Chances are that the programmers will not like it: they will not like talking to the business (that’s someone else’s job, I am too smart to talk to the managers); they will not like doing the testing (why do we have testing teams?) and they will find it very hard to give estimates of the complexity of the work. However, you must persevere. Drag the technical team with you; show them the satisfaction of completing the work on time and hearing well done from the business. In six weeks (or 3 sprints), you will have a new-born agile team.

To follow-up, read the Agile methodology in large organisation whitepaper, which describes detailed steps you need to take to implement agile management in your organisation.

The agile keywords

Product Backlog (PB)

key artifact
companies new to agile need a lot of help to create useful PB
product owner (PO) coaching essential
product backlog review (PBR) – in the beginning once every week, later once a sprint.
PO, business and the whole team nee to be present, lasts at most 2-3h. It is critical to complete understanding of the project and good release plan.

Release plan

PO, business, scrum master in a meeting to decide what needs to be delivered when – group stories and set a desired date for each group(release)
good to have one as soon as product backlog is usable
review release plan after every sprint – using new knowledge of team’s velocity
review release plan after every PBR – new stories may be added, or estimates of stories
may change

Sprint Planning

product backlog review – prioritization, clarity
task planning – splitting story to tasks. if there are more them 10 tasks, the story may be
to big, consider refining
2 week suggested iteration – long enough to deliver functionality, short enough for new
teams to estimate
ownership – product owner is responsible for relevant stories to be selected, the team
owns the tasks to deliver them
budgeting – every team member needs to budget his/her time. all holidays and any
potential non-project work like meetings, emails and such need to be factored in. Honesty is important, the sprint is most likely to fail due to wrong budgeting. Team shouldn’t budget 8h a day just because their line manager expects them to – it affects the whole team.
experience is the key for good estimates – prepare for undervalued estimates in the early sprints by taking less work then the team budgeted for

Sprint execution

unit and integration tests part of every task – strictly NO ‘test’ tasks on the board
code freeze and review 2 days before end of sprint – new agile teams tend to ‘design’ for
too long, and write code too late in the sprint
define ‘sprint goal’ – the sentence that describes what the team wants to achieve, used
as a guidance to remind the team to do whats required – no gold plating (no work noone
asked for)
if team realizes that some stories need to be dropped from the sprint due to unforeseen
circumstances, communicate it to PO as soon as possible – don’t leave it for PO to find
out during demo or sprint will be a definite failure
cake strength – great technical skills in addition to agile knowledge – to guide the team to
sprint success (team velocity with cake team members is often twice as big then usual)

Sprint review

demo acceptance tests demo UI (if any) feedback from PO important – good or bad
prepare to review product backlog at the same time – things noticed in sprint review often lead to new stories not noticed before
declare sprint success or failure
Sprint retrospective
open discussion in the team only
better results if retrospective is held out of work environment (office) – park or even pub.
after retrospective list the things team wants to improve on, and any experiments team
wants to try – and do them in the next sprint
experiments could be: technical talk every sprint by team member to allow skill/knowledge growth; every team member needs to review code from previous day and report on it

Additional

introduce Sprint zero – setup development/testing environment – project, CI, automated builds and deployment. Purely technical sprint
track technical debt – any decisions taken to deliver the sprint goal that can be done better should be noted, and taken into account for refactoring in the next sprint
acceptance tests are usually seen by good value by the business – so use them as part of the demo. Try Fitnesse (fitnesse.org) – great open source tool.
code quality is important – regular code reviews, code stats – coverage, complexity, documentation. Try open source Sonar for code stats reporting (www.sonarsource.org)
there a lot of programming practices that work with agile, select as much as you can and use them: CI, TDD, DDD, automated deployment and testing…

Talk to Cake

We are confident that Cake can improve your software delivery processes by working very closely with the business and within your development teams mentoring them through the change until they are able to stand alone. Together, we can create excellent software.

Cake runs periodic Agile events where we impart our agile experiences and knowledge to audiences all over the country. If you are interested in talking to us about attending the next event or just want a chat;
Call 0161 443 2355 or send an email to enquires@cakesolutions.net