Agile methodology in large organisations!

A White Paper on Agile Methodology

by Jan Machacek of Cake Solutions Limited

Transition from traditional software development model to an agile one is not easy. In this paper, I will outline how to execute such a transition in the most efficient and painless way. Before we begin, let’s review the core concepts of agile methodologies. The foremost is that agile projects are ready for change. Just as importantly, agile teams focus on delivering functioning software; software of excellent quality and with features that are always aligned to the business expectations.

The Holy Grail of software development, verily! All this is possible because the agile process calls for continuous refinement and improvement, and does not tolerate mediocrity. This, together with the continuous involvement of the business, means that agile projects deliver exactly what the business expects in the time frame it defines; and the quality of the delivered software is truly exceptional. Nota bene that the agile projects are not necessarily faster or cheaper than their traditional counterparts; the difference is that at the end of the agile process, there are no surprises.

Bringing agile to a business

If you’re reading this paragraph, then you have decided to accept the mission. The execution, especially in large organisations, is not trivial. Big enterprises are difficult to change; usually for two reasons. Firstly, you must overcome the resistance to change. Then you have to put in a lot of effort to keep the teams on the right agile path until the teams get used to the agile processes.

Resistance to change

The first problem is to uproot the existing management methodology. In my experience, most businesses intuitively know that there must be a better way of delivering software. Unfortunately, we often do not allow the business to work with us on the better approach. We often request signed-off requirements, change documents, completed architecture, and so on, even before we start the project. To allow the business to evaluate the benefit of the new software, it requests fixed timescales and costs at the start of the project. Both the business and the technical teams are drawn into an impossible situation, where the task is essentially to predict the future.

The objections most businesses have against agile methodologies are that they are afraid that agile projects mean never-ending, ad-hoc and poorly-documented, unmanageable and unmeasurable work. I will address each of these objections.

Agile is not never-ending work

The most frequent objection is that agile projects are too fluid to pin down & control. The truth is that agile projects are in fact very well controlled.

If I were cynical, I’d ask whether stating that the work shall be completed by 01/10 makes it so. In most projects, this is not the case. There is always more work and the requirements always change. So, how do we make an agile project complete on 1st October? The key is that we involve the product owner—the representative of the business. The product owner can decide what is he or she wants to see by the delivery date. It is up to the technical team to communicate with the product owner to understand what is important; to explain to the product owner the impact of any changes.

If the project is becoming more complex, the product owner can then make an informed decision: keep the delivery date and remove some features or delay the delivery. In any case, the business now has more power than it ever had over traditional projects. Instead of ex cathedra decrees, we now have functioning feedback and control loops.

Agile is not ad-hoc and poorly-documented

Often, the business feels that agile projects are akin to anarchy: the programmers are allowed to write code freely, without any regard for proper processes and documentation.

In reality, well-executed agile projects insist on continuous quality assurance. In programming- speak, this continuous quality assurance takes form of automated test, automated test coverage measurement and frequent code reviews. The end effect is that entire teams follow the best practices of software development. Having the whole teams follow the same practices creates self-documenting software. Furthermore, the code of agile projects includes tests that serve as further documentation—think of the tests as acceptance tests in the traditional world.
Agile projects emphasise quality and the optimum design; my favourite phrase when facing technical problems in agile projects is it is still far too early for hacks, let’s do it properly. Agile is therefore not ad-hoc, undocumented anarchy; instead, it is the best software your team can produce.

Agile teams are not hierarchical

The team members in an agile project are peers; each person has different skills, but no team member is more important than the other. This is sometimes difficult pill to swallow for architects and technical leads, who are used to being important and not having to deal with the sometimes mundane programming.

A successful agile team will include a star programmer, who can solve even the most difficult technical problems; his or her role is to smooth out the road for the rest of the team. The team then needs to include another very skilled programmer and leader and three to four good programmers. Notice that the team does not necessarily include testers! Testers on the agile teams allow the programmers to abdicate all testing to the tester; wherever I have seen this happen, the overall quality of the software suffered.

Agile is not unmanageable and unmeasurable

By eradicating the Gantt charts, agile projects seem difficult to manage and measure at start. I will leave the discussion of why Gantt charts are fundamentally unsuitable for software development as an exercise for the reader. In agile environment, the product backlog is the overall view of all features we know of at any point during the development. It is clear that at the start of the project, some of the features will be poorly defined. The agile team will work with the product owner to continuously refine the product backlog. The team needs to completely understand the features they are working on in this sprint, they need to have good understanding of features coming in the next sprint. This takes you one month into the future. It makes little sense to analyse additional features at in great detail. It is sufficient to have a rough understanding of the complexity of the future features.

Recall that the agile team includes an experienced technical lead; the technical lead must be an exceptional programmer and analyst, who can imagine the complexity of the features and see any possible technical problems in the implementation and guide the team in their estimates. The lead guides the team through the obstacles, he or she does not dictate the timescales! Remember that analysing the entire project in advance means that you assume that things will never change. If change comes, then the work you have put into the analysis is wasted. Agile teams use points to represent complexity of features. Stable teams are very good at assigning these scores to the features; the scores you will get usually fall into the sequence of 1, 2, 3, 5, 8, 13, 30 and 100. These numbers represent the relative complexity of the features, so a feature with a score of 13 is thirteen times as complex as feature with a score of 1. A score of 30 or even 100 indicates that the feature is either too complex or too poorly defined to score accurately.
These points give you excellent measurement tool. Within the first two sprints, you will know your team’s best and worst velocity. Having just these two numbers allows you to predict the time it will take to complete the features. As the team works through additional sprints, this indication will become more and more accurate.

You can see that agile projects are manageable: the business owns the product backlog, the business decides what is important; it is measurable: you monitor the rate with which the team implements the features.

Keeping the agile way

The business is usually not that hard to convince. You need to show that agile methodology is a process; a process that can be managed and measured; a process that is suitable for even the most serious projects.

The difficult part is convincing the technical teams to jump on board. From experience, programmers are difficult to manage; with the best intentions, they sometimes stray from the original path.

In most situations, I found that the teams were struggling to understand that the output of the entire team matters, not the performance of the individuals. To put this another way, the team members will often use pessimistic estimates to try to make sure that they finish on time (or even early), because that’s what they have traditionally been measured on.

Draconian quality

Some teams also find it difficult to accept the strict focus on quality. It is often difficult to impress on the team that testing (unit and integration) is integral part of the code; there is never a ‘testing’ task at the end of the sprint.

Another sometimes difficult area to accept is that experienced agile teams include a perfectionist; a person who does not accept half-measures. Having this person on the team ensures that the quality and maintainability remains as good as it can be.

Velocity & getting things done

The agile process measures the team’s velocity; the rate with which they go through the points. It is this measure that allows us to predict when the team will finish. The team leader needs to focus on keeping the team’s velocity stable. Perhaps surprising aspect of maintaining the team’s velocity is that the team leader must focus on not allowing the team to write code for code’s sake. It is very difficult for traditional teams to accept that some code, however interesting, sometimes simply should not be written. The leader needs to be a skilled diplomat and drive the team to only write the code that is absolutely necessary.

Done means done

Final crucial point that is difficult to overcome for traditional teams is that there is no such thing as 90 per cent done. A story is either done or not; and it is the business—by working on the definition of done. An agile team now shares the understanding of the features with the business, but without hundreds of pages of specs.

Continuous improvement

The crucial concept in agile is that all members in the team work together to deliver the stories in the sprint. Do not measure individuals and do not measure individual stories. Instead, focus on keeping the team’s communication channels open and remove all red tape. Be sure to emphasise that it is the team’s responsibility to deliver the stories in excellent quality.
If the team does not deliver, ask why and, most importantly, ask what the team is going to do about it in the next sprint. Ensure that your agile team knows that they have the time to investigate new approaches to solving a particular problem; that it is OK to spend some time doing what may traditionally seem as unproductive research.

The agile keywords

Product Backlog (PB)

* key artifact
* companies new to agile need a lot of help to create useful PB
* product owner (PO) coaching essential
* product backlog review (PBR) – in the beginning once every week, later once a sprint.
PO, business and the whole team nee to be present, lasts at most 2-3h. It is critical to complete understanding of the project and good release plan.

Release plan

* PO, business, scrum master in a meeting to decide what needs to be delivered when – group stories and set a desired date for each group(release)
* good to have one as soon as product backlog is usable
* review release plan after every sprint – using new knowledge of team’s velocity
* review release plan after every PBR – new stories may be added, or estimates of stories
may change

Sprint Planning

* product backlog review – prioritization, clarity
* task planning – splitting story to tasks. if there are more them 10 tasks, the story may be
to big, consider refining
* 2 week suggested iteration – long enough to deliver functionality, short enough for new
teams to estimate
* ownership – product owner is responsible for relevant stories to be selected, the team
owns the tasks to deliver them
* budgeting – every team member needs to budget his/her time. all holidays and any
potential non-project work like meetings, emails and such need to be factored in. Honesty is important, the sprint is most likely to fail due to wrong budgeting. Team shouldn’t budget 8h a day just because their line manager expects them to – it affects the whole team.
* experience is the key for good estimates – prepare for undervalued estimates in the early sprints by taking less work then the team budgeted for

Sprint execution

* unit and integration tests part of every task – strictly NO ‘test’ tasks on the board
* code freeze and review 2 days before end of sprint – new agile teams tend to ‘design’ for
too long, and write code too late in the sprint
* define ‘sprint goal’ – the sentence that describes what the team wants to achieve, used
as a guidance to remind the team to do whats required – no gold plating (no work noone asked for)
* if team realizes that some stories need to be dropped from the sprint due to unforeseen circumstances, communicate it to PO as soon as possible – don’t leave it for PO to find out during demo or sprint will be a definite failure
* cake strength – great technical skills in addition to agile knowledge – to guide the team to sprint success (team velocity with cake team members is often twice as big then usual)

Sprint review

* demo acceptance tests
* demo UI (if any)
* feedback from PO important – good or bad
* prepare to review product backlog at the same time – things noticed in sprint review often
lead to new stories not noticed before
* declare sprint success or failure

Sprint retrospective

* open discussion in the team only
* better results if retrospective is held out of work environment (office) – park or even pub.
* after retrospective list the things team wants to improve on, and any experiments team
wants to try – and do them in the next sprint
* experiments could be: technical talk every sprint by team member to allow skill/knowledge growth; every team member needs to review code from previous day and report on it

Additional

* introduce Sprint zero – setup development/testing environment – project, CI, automated builds and deployment. Purely technical sprint
* track technical debt – any decisions taken to deliver the sprint goal that can be done better should be noted, and taken into account for refactoring in the next sprint
* acceptance tests are usually seen by good value by the business – so use them as part of the demo. Try Fitnesse (fitnesse.org) – great open source tool.
* code quality is important – regular code reviews, code stats – coverage, complexity, documentation. Try open source Sonar for code stats reporting (www.sonarsource.org)
* -there a lot of programming practices that work with agile, select as much as you can and use them: CI, TDD, DDD, automated deployment and testing…

Talk to us

We are confident that Cake can improve your software delivery processes by working very closely with the business and within your development teams mentoring them through the change until they are able to stand alone. Together, we can create excellent software.
Cake runs periodic Agile events where we impart our agile experiences and knowledge to audiences all over the country. If you are interested in talking to us about attending the next event or just want a chat;
Call 0161 443 2355 or send an email to enquires@cakesolutions.net