How to implement scrum on an Waterfall enterprise?

Question

Research prior to asking this questions -> When to Use Waterfall, When to Use Scrum ?

I am not focus on pure waterfall vs pure scrum conversion. I am more incline of solving the following problems that are being caused by our very rigid waterfall implementation. As everything in big corporations, there is a little of fault in red tape and additional politics, but at heart of the problem is the Waterfall methodology that we currently follow.

Here are the symptoms:

• Business Requirements have very little quality since they hand-off these requirements at the beginning meeting a date for IT to start the software analysis. This frustrates the IT teams and use some of the development time to clear some of these bad requirements.
• Because Business requirements are not completed on time or have little quality, development usually develops the wrong thing, causing re-work.
• Because rework is necessary, and changes are unavoidable the QA team usually end up without the time to do proper work.
• In order to get anything in a release we have to estimate without requirements, this causes a list of things that the business wants but is unattainable because time to develop is inflated (mainly politics issue)

This department of 40-50 IT pros, have been trying to step out of waterfall into scrum for a year now, but is causing a lot of issues. Issues like (1) people using scrum to hide on the fact that requirements are not ready (2) other IT departments that don't use scrum complain about the process.

I have decided to try a 12 weeks scrum pilot. Shoot for a strict scrum implementation where that team will have a product owner that will be responsible for what is important. I think this will be the key of having a successful implementation. Right now the business is not accountable for their deliveries, and I think they should be. I will split these 12 weeks in 3 sprints of 3 weeks with 2 weeks of planning the pilot and 1 week of releasing to production.

The questions: In your experience - is there a MUST DO before I try this pilot with a very large organization?

Answer 1

Most of the answers here are excellent. I don't think you need high-quality written requirements, though.

There are two types of things we build in software:

• Stuff that's new and differentiating
• Stuff that's similar to things we've done before and is well understood.

For the first of these, it may be the case that it's new to the business, as well. Rather than clarifying all the requirements clearly, they may simply want to try something out. Feedback is more important here than getting the requirements nailed down precisely. I often find that just getting a GUI working, even with hard-coded data, is a good idea.

For the second, a domain or business expert will be able to give the team the knowledge they need. However, trying to do this in a one-way form (like a document) is very difficult; developers can't ask questions and clarify the requirements, or even give feedback on their quality.

So here's your "Must do": get better engagement with business experts.

A product owner is a really good idea - check that they're genuinely interested in championing the product. He'll need to know what's different between that product and similar ones, because that's where the team will need to get feedback (early if possible, the end of the sprint if they can't). It's unlikely that everything will work first time, so rework is inevitable; the faster they get feedback, the quicker it will be. The team shouldn't wait until the end of the sprint if they don't have to.

Also, look out for other stakeholders whose goals need to be met too, and make sure that the product owner has good communication with them (might be legal, security, architecture, operations, etc.) - he'll need to understand their concerns too.

The best thing you can do for the team is get everyone co-located - testers, the product owner and the devs - so that they can talk to each other, ask questions, get feedback and respond to it quickly. Again, the developers don't have to wait for the sprint to show their work to testers or to the product owner; the quicker they can do this, the easier it will be to fix any rework that needs doing.

Lastly, check the KPIs in your organisation. If people are being rewarded for heroic or individual behavior - testers for the number of bugs they find, or developers for lines of code, for instance - then it's going to get in the way of them acting as a group. You may need to have a chat with HR if this is the case. These kinds of KPIs are usually pretty detrimental anyway.

This isn't by any means enough to guarantee success (nothing is) but along with the other suggestions this should get you started. Good luck!

Answer 2

Switching to scrum and implementing a tight sprint duration of, say 1 to 2 weeks might help the poor requirements get sorted out.

Since there is a functional product at the end of each sprint, the business will quickly see the fruit of their requirements. When you do your (bi-)weekly sprint kickoff, the business will set the stories for that sprint.

The danger of such a setup is that it is easy to drift off track with scope creep from hell. As the PM, it is your job to gently guide the ship away from the rocks.

To get buy-in from your developers, one thing I do is implement fast and effective standups:

• Meeting runs for 10 minutes, no exceptions
• Everyone gets an equal share of time to state
  • what they did yesterday,
  • what they're doing today and
  • any issues.
• The timer beeps and the next person goes (most of the time, we have time left over)
• Any and all discussions are not allowed (people can meet after the standup if they want). Just keep saying, "let's take that off-line."

This will make your devs happy, because they only waste 10 minutes a day (in the AM) on a short meeting. Of course the sprint kickoffs and lookbacks are longer, but we try to keep them as productive as possible.

You CAN change your local culture. You don't need much buy in*. Just start doing scrums and sprints and tell your customers that they're going to start getting a working product sooner.

*From personal experience doing this very thing at a 100+ year old crusty fortune 500 company. It can be done, and that's what politics is fer!

Answer 3

This is a tough question because the scrum framework exposes, supports and helps with many business behaviors, cultures and dependencies, but what part you struggle with and how you actually adopt it may be different than others.

My recommendation for you is to bring in an agile coach (not just a scrum master) for your pilot project that can help guide you through the new practices and even help negotiate some of those initial pain points with other stakeholders. Sometimes the internal influence can be too great for the folks already there inside the culture.

Good luck!!

Answer 4

Although I highly believe that having Agile methodology in place might solve your problem, I'd suggest to slightly change the focus here.

You might have already tried it, but still, changing the methodology (specially when reducing the rigidness of the process) in a big company has two main roadblocks:

• It needs buy-in from everyone involved (as other folks already mentioned)
• It may offer the idea that the problem is with the methodology, which is not the case. The problem is with the requirement's quality.

So, Agile may be the answer for the wrong question.

If you go for the meeting telling the stakeholders something like "hey, the approach were using here isn't working, let's split the deliverables into smaller pieces" they may be interested (because it will reduce their frustrations as well) but you'll be relying on the methodology to do the trick. Further down the road, if the root problem isn't solved, you'll be with the same scenario (poor requirements), under a different methodology.

My suggestion, in this case, would be to have a good Business Analyst capable of converting better the requirements between IT <-> Business. If happens that this person is also a Scrum Master or something like this, perfect! But I'd say that's better having a good BA with some Scrum skills rather than a Scrum Master (or any other Scrum-role) with some BA skills.

Still, if the requirements aren't good enough, agree (formally) with the stakeholders what / how the deliverables will look like. It might take most of the schedule, but as you already said, better to spend time on defining the tasks rather than on rework. We had a similar problem like this in the past that was solved with a proper BA + signoff on the functional / tech specs signoff from both IT + Stakeholders.

Success!

Answer 5

As usual, I'm going to go a slightly different way than the other answers - from your description, it's not the methodology that's the issue, but the process. Changing to Scrum isn't going to fix anything (okay, maybe it will, I'll address that at the end).

Looking at your comments, they all flow back to one specific cause - poor requirements.

"Business requirements have very little quality...", "frustrates the IT teams and use some of the development time to clear some of the bad requirements." "Because business requirements are not completed on time or have little quality, development usually develops the wrong thing, causing re-work." Because re-work is necessary, and changes unavoidable the QA team usually end up without time to do proper work." "...we have to estimate without requirements."

All of these comments go back to a single issue - poor or incomplete requirements.

So to your question of "is there a MUST DO before I try this pilot" - yes, you MUST get accountability for the requirements phase squared away and assigned."

Waterfall isn't really the problem here, nor Scrum necessarily the solution. The process is failing because there are no requirements to work from . Even Scrum needs requirements of some quality to work from. Certainly the cycles are shorter and faster, but with poor requirements, it just means you're going to fail faster, and more often.

Going back to my earlier comment - Scrum may be able to help, but only if you first get the accountability for requirements handled. Waterfall definitely needs the scope fairly well defined to work, but Scrum is also somewhat of a waterfall process (one thing has to be done before another - requirements are needed before development, which is before QA, etc.) So even switching to Scrum the requirements will need to be better defined than what you're currently getting. With Scrum you can probably get away with looser requirements that a full waterfall process, so it's probably a better methodology for your application, but until you get the other part fixed, you'll keep seeing the same results.

Answer 6

For implementing in a large organization I would say that well trained scrum masters and a shorter training on scrum for everyone that will be affected by it (from Project owner(s) to developers to QA) - mainly so that everyone knows what to deliver (and when) and what to expect.

In our 12-20 person organization (with 3-5 teams) we did it by reading the book by Kniberg and with someone with previous experience, but that was a slight uphill travel to get it working even in a organization of that size.

Also (I see it) prioritized requirement are an important part of successful scrum so make sure you are able to get that - it should be possible by education (that points out that only the most important requirements needs to be set up before each sprint).

Answer 7

To make the transition, you are going to need the full buy-in across the organisation. If you try and implement without high level support, then as soon as you start treading on the toes of those with a vested interest in staying waterfall, you're going to encounter problems.

Scrum affects so much more than developers/pm teams, so you're going to have to get as much goodwill as you can.

Maybe read Succeeding with Agile by Mike Cohn...it should give you some worthwhile pointers as it directly discusses overcoming barriers to implementation.

Also, (and you may have already considered it), what is making you choose scrum over other agile methods? Dynamic systems development method (DSDM) might be worth looking at if you encounter resistance...it's an agile approach, but it might be less stressful to organisations that are steeped in Prince2/waterfall approaches.

Answer 8

Geo, it's really, really hard to change the process used by an entire organisation, especially a large one. I work for a company with around 300 people with software delivery making up about 50% of that. For the last 18 months I have been running projects using Scrum (the rest of the organisation run an in house process that is more waterfall or iterative). My experience is that although there are problems with the current in house process and despite my Scrum based projects delivering on time and with less resources than equivalent non Scrum projects, there is still a significant lack of interest in moving more of the organisation towards Scrum. This is primarily because it is a big change and people are scared of big changes, there is never a good time to make the jump (we always have a big project in the pipe!). Also our current process has been through multiple successful audits and there is a concern around how well Scrum will audit.

Having said that, you should still try to change things to improve the process. I did and it is working well, it's just going to be a much longer process than I expected (and I am in a fairly small company!).

Although I am an advocate of Scrum and think it will ultimately help your situation, I also agree with the other answers that your issue is not the process but the requirements. I would focus initially on trying to get the requirements better quality early on. One thing that may help is to use user stories, these tend to be easier (less effort) for the business to write and that increases the chance that they will write them. Having follow-up conversations with the business to clarify the detail in the story will mean your developers and testers have some idea what needs to be delivered. You may not have a complete copy of the detail written down, but I believe 80% of the game is making sure that the people doing the work know what needs to be done, the documentation is a useful safety net for when people forget (or change their mind without telling anyone!).

Ultimately one of the goals of agile is to speed up the development process by talking rather than writing (favour working software over comprehensive documentation), so you could make steps towards this by working in a more agile way. You may decide that running a pilot strict Scrum project is the best way to approach this (at least it gives a clear differentiator between this project and others), but I would focus on getting the technical and business people to communicate requirements and features better as a starting point.

Answer 9

It sounds like your Business Analysts don't do their job properly, that they've become mere scribes. Maybe they're new, or in need of training. Why not involve a developer earlier in the process, before any requirements sign-off happens?

If the developer isn't immediately sure because they've never done this type of work before, give them time to work with the requirements as they are even up to the point of producing a prototype.

I face some of the same issues, mainly because of staff rotation among BAs, but because the work I want to be doing doesn't involve BAs I'm not following my own advice here. If all your work comes from them I think it would be useful.

Answer 10

Ask all the participants to read:
http://www.halfarsedagilemanifesto.org/ for a week.
Then schedule a 2 hour meeting to talk about it.

The above resource highlights the following points:

Manifesto for Half-Arsed Agile Software Development

We have heard about new ways of developing software by paying consultants and reading Gartner reports. Through this we have been told to value:

• Individuals and interactions over processes and tools and we have mandatory processes and tools to control how those individuals (we prefer the term ‘resources’) interact

• Working software over comprehensive documentation as long as that software is comprehensively documented

• Customer collaboration over contract negotiation within the boundaries of strict contracts, of course, and subject to rigorous change control

• Responding to change over following a plan provided a detailed plan is in place to respond to the change, and it is followed precisely

• That is, while the items on the left sound nice in theory, we’re an enterprise company, and there’s no way we’re letting go of the items on the right.