17

My team historically has averaged about 40 people over several years, often more than that. We have a pretty complex product that cuts across several problem domains. We switched from waterfall to scrum and looking back I have to say that neither worked (well). As one PM told me, our project was "always yellow or red" (meaning in state of high risk or seriously delayed).

Watefall "worked" bc once we were officially done with development and moved to testing, not many of officially done features actually worked and effective development has been done during (long) testing and tediously corrected. You could say that it was sort of official waterfall combined with underneath sort-of iterative process.

We switched to agile/scrum (in a very basic form). From frying pan into fire: we have terrible problems getting logically consistent product now. It's not strange given we have 5 major sub-teams (not all of them in one geo location) and Conway's law operates. Plus, repeated stress at the end of iteration over several years tends to wear out/exhaust people (that's my subjective feeling though, I do not have hard data to prove that). It's also has been terrible re keeping people long enough in one place/subsystem of the product so they could become experts and highly productive in their place: I get the impression that in this constant change we evolve towards being jacks of all trades and masters of nothing. In hindsight, I cannot tell that scrum has given us any definitive advantages (we have "scrum of scrums" of course and I can't detect that it has made any difference). Certainly, lack of big up-front design from my POV is a serious flaw: work gets unsynchronized, badly partitioned and needs to be redone or modified later.

It's like classic CS process deadlock, just moved to conceptual/design level: dev/subteam A waits for dev/subteam B to complete design X he/it relies so he could complete his/its design Y, while dev/subteam B waits for A to do his/their design Y so they/he could do X.

Scrum says nothing about infrastructure tasks apart from "cram it into some user story" (and so infrastructure suffers). Additional 3 smaller sub-topic teams (internal, external, content provider team using products of both previous ones) for particular area still can't even agree on common data representation and format. Tracking actual state of a zillion of tasks is a nightmare. The consensus in the team is that scrum has given us no benefits while imposing additional costs. Certainly it has not made us pay better attention to what customers/end users actually need.

I can't believe I'm saying this but for us waterfall worked (slightly) better than scrum (or to be more precise: it was "not as bad, if only slightly less bad").

Fortunately some subsystems are pretty much orthogonal, otherwise we'd get nothing done. This is pure random chance, though: a tad more internal dependencies between subsystems (re conceptual integrity, performance, etc) and we'd be toast. Being saved by dumb luck is no proof of effective organization or wisdom.

From my POV it's like this: if a project is small (grokkable by one person in entirety or close to it) or consists of a bunch of nearly independent concepts and features, any methodology will work. If a project is big (one person can't even grok an entire subsystem + there are lots of internal dependencies between subsystems in concepts, design, business logic and algorithms), no methodology works.

The only solution I personally see would be like this:

  1. do the big, up-front analysis (collecting requirements) and big waterfall-style design (principles, assumptions, business logic).
  2. treat it as draft to be modified, initial starting point to be modified on the fly.
  3. develop this "big design" iteratively, using scrum or kanban.
  4. have analysts and designers constantly re-evaluate big picture - whether the product stays consistent, and sync design and code.

If that wouldn't work, I have to say that for me my project is like a "Gordian knot from hell" that is basically unsolvable (well) from my POV.

This super-long background info gets me to my question: are there any PM methodologies that work well on big projects?


Further details:

  1. I understand that map is not the road. Sure. But I can't envision getting to Alaska from SA without the map. :-) As they say in math, methodology to me is a necessary, not a sufficient condition.

  2. Issues:

    • Deadlines: tight.
    • Budget: seriously underfunded.
    • Scope changes: constant.
    • Requirements: often changing, lots of gaps.
    • Team structure: revolving doors situation, people shifted around mindlessly.
    • Other issues: lots of (truly bad) legacy code, no appreciation of long-term situation and costs/benefits by mgmt.
    • Oh and: no (application) SW architect.
  3. I know that people will shoot down my vision as "asking for autopilot / panacea", but I tend to think that A. Whitehall is onto smth when he said that progress is measured by a number of things we can do without thinking. A perfect methodology would fit any team: just have the guts to follow the procedure. If "other factors" - the unknown factors - are necessary to make project succeed, what are those?

I'm starting to think that this is all futile. I've done some rough, example calculations: Imagine: n features, m classes (assuming OO programming), k software characteristics (reliability, performance in various contexts, security, implementation cost, defect rates, time necessary to get development done, testing cost, testing time, etc), o people. Potential complexity: every feature can be divided into p classes (p less-than m, with sum of all p for all features == m), impacts at most k characteristics and can require work or intervention of any subset of o people.

So worst-case complexity (number of combinations) is n*m*k*o. with 20 features, 500 classes, 10 aspects and 40 people that is 4000000 (4 mil). Assuming generous release time of 18 months, that is 222222 (222 thousand) combinations per month. Suppose we cut 500 (or "m") classes into 50 manageable collections with each collection modifiable/refactorable internally without impacting another collection. That's still over 22 thousand combinations per month.

It is not humanly possible to track so many potential feature/class/person impacts on k (software characteristics) and select best combinations (plan ahead). Some sci-fi style hard-AI tool would have to do it. Otherwise, we're just throwing "hail mary": maybe it will work well enough in enough cases that the project will not get delayed. Or maybe it won't.

Background info to give you better idea of the system in question. This has to be sort of "anonymized" for reason of confidentiality.


Data flow

This one is fortunately pretty simple: agents <-> server, with server consisting of following major subsystems:

  • agent comm
  • inventory
  • data aggregation
  • report generation

(there are of course many supporting subsystems, from server config console to web ui to REST api but I'm not including those here)


Implementation

Technologies: C, Java, SQL. Environments range from hundreds to tens of thousands of agents (the biggest environment: about 100K of agents).

Code size: Several thousand Java files organized in approx 500 packages. About 900 KLOC total so far (not counting the agent). So it's not that big, but it's complex in the sense of involving many problem domains.

Aggregation is the most complex and resource intensive part of the app. We're using plain SQL for performance, but even then aggregation can't finish in 24 hrs for large environments (right when the next round of aggregation should start, i.e. next day). Miscalculation can be sort of serious, since it can easily cost a customer dozens of $M.

LetMeSOThat4U
  • 379
  • 1
  • 7
  • 3
    There are techniques that make a huge project more manageable and that is to break the project down into its pieces and put the rest out for progressive elaboration. "Big" is a perception. If you want to change your perception, make it smaller. Our history shows that we have successfully completed impossible projects, eg, pretty much anything NASA has done and there are more. – David Espina Dec 19 '12 at 16:01
  • 2
    Whenever I hear about 'big projects', I remember TMMM, where Brooks discussed about his OS/2 projects with 5000+ people. It makes me feel so... amateur. – Tiago Cardoso Dec 19 '12 at 16:13
  • "Big" is not semantically equivalent to "complex." Scrum is designed to work well on complex emergent systems, but implementing any project management framework well is more of an art than a science. – Todd A. Jacobs Dec 19 '12 at 22:53
  • @David Espina, Code Gnome:

    Here is where language sort of breaks: the major problem with our project is not that it is either big in KLOC/FP or "complex". Data flow architecture is simple. The problem is that the project is more like a "bowl of paperclips" more than a bunch of independent features - you can't touch any feature or issue without impacting most of other aspects, subsystems and issues.

    – LetMeSOThat4U Dec 20 '12 at 16:02
  • @DE: if you can decompose your system neatly into a bunch of highly independent subsystems, you can do even hugely complex projects. With all due respect to Brooks and OS/2, operating system as such is just such a system: decompose into process management, virtual filesystem manager (Linux), filesystem drivers, virtual memory manager, network stack, firewall, etc. You can refactor most of those without impacting another subsystem. This is the most luxurious situation I can think of re internal dependencies ("bowl of paperclips"). – LetMeSOThat4U Dec 20 '12 at 16:05
  • OS/2 operating system is just such a system. Yeah, released in April, 1987. Their notes were taken in paper. They didn't had even a notepad. One of their biggest challenges was to keep piles and piles of paper with 'documentation' updated. – Tiago Cardoso Dec 20 '12 at 16:11
  • I've some evidence that system decomposition/partitioning can solve dependency problems in a neat if unintuitive way: my company has bought another product that historically has managed complexity much better. They've done this by extremely strict control of data collection server: whenever you throw in smth from the agent that runs your own custom tasks, you have to follow the format and rules of data collection. And then pull whatever data you need into separate analytics/mgmt servers (usage, patch mgmt, security, config, etc). This has worked incredibly well for them. – LetMeSOThat4U Dec 20 '12 at 16:11
  • @Tiago Cardoso: I do understand that. It must have been extremely tough, but for reasons other than ours I think. Elsewhere - when Bill Gates begged and blackmailed programmers to create a single rich text control for windows ui to reuse, he did not have to worry about filesystem issues. In our project another thing is hellish: if you touch anything, almost everyone else screams "no!" because it impacts the guts of what they're doing in some way (e.g. it makes data aggregation take longer time to the point of it being unacceptable). – LetMeSOThat4U Dec 20 '12 at 16:28
  • It remembers me an old project I worked with, @mrkafk ... it was a maintenance project for a VBA tool with over hundreds of thousands (really!) of (spaghetti) code. It's decommissioned now, thankfully. – Tiago Cardoso Dec 20 '12 at 18:44

4 Answers4

17

Following on Tiago's answer, the issue is not the methodology but how the methodology is being tailored on your projects.

Based on my experience working on biodefence projects for US govt, as the complexity of a project increases so to does your need for upfront planning and thorough project oversight. Regardless of your use of waterfall or agile or whatever:

  • Take your time while doing your initial planning. Make sure that you engage the necessary stakeholders. Have an open and honest conversation about capabilities and assumptions. Come up with a schedule, budget and scope that are realistic.
  • Take the time to revist the plan at regular intervals. When a key milestone has been reached don't just check it off the list. Sit the team down and celebrate the success and review what work is coming up. Find out how underlying assumptions have changed, and if there are any issues that have come up that may affect the original plan.
  • Establish a robust change control system. If someone wants a change that is OK, they just have to follow the approved process. Don't tolerate ad hoc changes from the team. These will lead to scope creep and will invalidate your plan.
  • Establish proper governance structures. You as PM should be working with a project sponsor/champion to execute the project. You should both be reporting to a project committee/board that represents supplier, customer and business interests. Making sure they all have a say will help guide the project to a positive conclusion.
  • Communicate progress in a timely and effective manner. Set up regular reports with the key stakeholders so that you can flag issues and manage expectations proactively. These could be written, through telecons or whatever is most appropriate.
Doug B
  • 8,899
  • 17
  • 40
  • @Doug B: I'm tempted to mark your answer as "accepted", but I want to wait in (vain maybe) hope that even better answer comes along.

    Looking at my project I have to say that when we switched to agile/Scrum it became much, much worse in all departments you listed. In the past BDUF enforced some sort of consistency, change control and planning. Now we have almost none of those. So making a delta between those I think you nailed a good chunk of root causes behind our problems. It looks like responsibility problem to which methodology is only indirectly related.

    – LetMeSOThat4U Dec 20 '12 at 15:57
  • @Doug B: The more I think about it, the more I like your angles of looking at the problem. In hindsight what hit us most destructively I think are failures at "Take your time while doing your initial planning" and "Establish a robust change control system". Also, I have impression that when you have scrum teams working on subsystems instead of on business-related features from start to finish, demos are near-useless to external stakeholders. They don't even try to understand it bc those do not produce direct benefits to them. So "communicating progress" has to be done in other form. – LetMeSOThat4U Dec 20 '12 at 17:51
  • 1
    @mrkafk Demos could be drawn from the scrum-of-scrums integration features rather than individual sub-projects if that's a better fit for your environment. – Todd A. Jacobs Dec 20 '12 at 20:14
10

Asking if there's a methodology that works fine for a large project sounds like asking if a map route can take someone from South America to Alaska.

Is there any (methodology or map)? Yes. Will it do it (work or travel) by itself? Definitely, no.

A methodology is a guideline. But, as the map, it's only suggestions of how to reach something or somewhere, not the destiny itself.

You mentioned Waterfall x Agile. There's a very good discussion on this topic HERE, and I believe Pawel's answer is amazing (as usual!). In your case, as a big project, it seems that a more formal methodology would fit better (i.e. Waterfall), but there's no strict rule saying that.

Bottomline: The problem is not the map. The problem is not the methodology. Having said that, the underlying question here is:

What's your project number one issue? Or, why your projects are always between amber / red?

  • Deadlines?
  • Costs?
  • Scope changes?
  • Unclear requirements?
  • Team structure?
  • Any other issue?

You'll need to nail down the root cause of your project issues and then you'll be able to assess what's the best methodology to fit your project. Not the other way round.

Tiago Cardoso
  • 8,645
  • 6
  • 29
  • 72
9

Classic X/Y Problem

In the comments, the OP elaborated on his project environment. He said:

[T]he major problem with our project is not that it is either big in KLOC/FP or "complex"...The problem is that the project is more like a "bowl of paperclips" more than a bunch of independent features - you can't touch any feature or issue without impacting most of other aspects, subsystems and issues.

As I understand it, the issue is that the project tasks, milestones, features, and objectives are so interdependent that any change or deviation requires a refactoring of the entire project plan and/or schedule.

So, this is an X/Y problem because:

  1. X: The underlying issue is either architectural or conceptual---possibly both. Perhaps the work hasn't been decomposed sufficiently, or sub-projects have not been designed to be sufficiently self-contained. Either way, the issue appears to be entanglement between tasks.
  2. Y: The OP has determined that a different methodology will solve the underlying issue, and therefore asks "[A]re there any PM methodologies that work well on big projects?"

Solve for X Instead

Based on my understanding of the root cause, I would recommend refactoring the project plan rather than redesigning the project management process. Specifically:

  1. Hold a review of the project's architectural assumptions. See if the architecture can be refactored to be more loosely-coupled.
  2. Revisit the work breakdown structure, and see if tasks can be decomposed sufficiently to avoid dependencies between milestones.
  3. Consider refactoring the project plan into sub-projects that are less co-dependent. This is non-trivial if the project has already been designed as a "bowl of paperclips," but is certainly worth at least a thought experiment to see if it can be done.
  4. Invite your technical team to discuss ways to reduce technical or process dependencies.

You might also consider treating the heavy inter-task dependencies as project risk. Document the risk appropriately, report the risks upstream to senior management, and let them either accept the risk or authorize necessary risk-mitigation measures.

Lastly, if all else fails, set a benchmark for when the project is so far out of tolerance that it should be recommended for termination. While it isn't usually the project manager's role to authorize pulling the plug, it is certainly within your scope to identify the limits of risk and capital expenditure that the organization is willing to tolerate. Once that information is known, you can manage against that metric, and recommend a timely early termination (if failure is inevitable) in order to save the company money.

Todd A. Jacobs
  • 50,264
  • 7
  • 59
  • 177
  • I suspected as much and you're completely correct on X/Y problem. (that is: we're having this problem, on top of responsibility and accountability and organization structure and lack of intra-team requirement mgmt problems).

    I have already suffered ramifications of bad problem and work partitioning on my previous scrum team so I recognized this instantly. But there are so many problems to address simultaneously I could not even manage to articulate them all in one post.

    – LetMeSOThat4U Dec 20 '12 at 19:40
  • 1
    +1 for considering the inter-task dependencies as project risk. – Tiago Cardoso Dec 21 '12 at 11:40
5

Waterfall is used successfully on the largest projects in the world e.g. 500 people over 6 years at a cost of $21 billion. These are mostly infrastructure projects (bridges, tunnels and railways), aerospace and defense projects (airplanes and weapons systems) and energy projects (pipelines, drilling rigs)

It is not the methodology that fails, but the implementation and the "vibes" of the project environment. For an example of "vibes" on the project environment, I just came back from a workshop that studied the $3.3 billion development of the FA-18 Hornet fighter jet. One of the critical success factors mentioned on the project was creating an atmosphere of trust and accurate data flow on the project.

You need to look at the organization, leadership and project environment. This is not a methodology issue.

Mark Phillips
  • 8,849
  • 4
  • 30
  • 57
  • thanks, good hints - but that does not address "how" issue that I have trouble with: organization - how? that is, how to model the structures? Leadership: not active enough I guess. I have yet to recall a single visit of subsystem owner (equivalent of "product owner" in basic scrum I think) on any of the scrum meetings in 2 teams I participated in. Project environment: pretty good actually, people are still optimistic, friendly and willing to help each other out. We just can't seem to be able to get our act together into a parsimonious "whole thing". – LetMeSOThat4U Dec 20 '12 at 16:24
  • The "how" is not something that can be answered on a site like this. It is not a recipe, checklist or quick fix. It is a process that starts with getting a deeper understanding of the organization. You need an empowered change agent or catalyst. – Mark Phillips Dec 20 '12 at 16:51