The Failure of Agile

I am proud to be one of the 17 founders/authors of the The Agile Manifesto back in 2001. I think it provided a jolt of energy, hope of a better way of doing things, of creating software and making the world work better. It was a pivotal turning point.

But in the 14 years since then, we‘ve lost our way. The word “agile” has become sloganized; meaningless at best, jingoist at worst. We have large swaths of people doing “flaccid agile,” a half-hearted attempt at following a few select software development practices, poorly. We have scads of vocal agile zealots—as per the definition that a zealot is one who redoubles their effort after they've forgotten their aim.

And worst of all, agile methods themselves have not been agile. Now there‘s an irony for you.

How did we get into this mess?

The Joy of Rules

The basis of an agile approach is to embrace change; to be aware of changes to the product under development, the needs and wishes of the users, the environment, the competition, the market, the technology; all of these can be volatile fountains of change. To embrace the flood of changes, agile methods advise us to “inspect and adapt.” That is, to figure out what‘s changed and adapt to it by changing our methods, refactoring our code, collaborating with our customers, and so on.

But most agile adopters simply can‘t do that, for a very good reason.

When you are first learning a new skill—a new programming language, or a new technique, or a new development method—you do not yet have the experience, mental models, or ability to handle an abstract concept such as “inspect and adapt.” Those abilities are only present in practitioners with much more experience, at higher skill levels (see my article Why Johnny Can't be Agile for more).

The only way for beginners to be effective is to follow simple, context-free rules; rules of the form: ”When this happens, do that.” And since agile methods conveniently provide some concrete practices to start with, new teams latch on to those, or part of those, and get stuck there.

So instead of looking up to the agile principles and the abstract ideas of the agile manifesto, folks get as far as the perceived iron rules of a set of practices, and no further.

Agile methods ask practitioners to think, and frankly, that‘s a hard sell. It is far more comfortable to simply follow what rules are given and claim you're “doing it by the book.” It’s easy, it’s safe from ridicule or recrimination; you won’t get fired for it. While we might publicly decry the narrow confines of a set of rules, there is safety and comfort there. But of course, to be agile—or effective—isn’t about comfort (see my article Uncomfortable with Agile).

And if you only pick a handful of rules that you feel like following, and ignore the hard ones, then you’ve got it made! You are ”doing agile” if anyone asks, and you can focus your energy on enforcing a small set of largely useless rules. Everyone feels better. Until it’s time to actually ship something.

Stuck in a Concrete Rut

Following a rigid set of rules limits your performance to that of a novice (see my book Pragmatic Thinking & Learning: Refactor your Wetware for more). So teams find themselves unknowingly stuck in a rut: blindly following the concrete rules, not gaining the experience they need to get to the point where they know they need to move beyond the rules.

It’s all too common these days to see arguments on Twitter or mailing lists with these rules-bound zealots arguing that ”you’re not agile” because you aren’t following the rules to their satisfaction.

What happened to the idea of inspect and adapt? What happened to the idea of introducing new practices, of evolving our practices to suit the challenges at hand? The canonical agile practices of popular methods have remained essentially unchanged for over a decade. We are stuck in a rut.

And as I’m fond of saying, the only difference between a rut and a grave is the dimensions.

A Way Forward

Now there have been many folks complaining about the agile movement, some quietly, some loudly, and none of these complaints are novel. But here‘s a new twist: let‘s fix it. Right here, right now. Together, we can fix the inherent problems in agile adoption and evolution and help move the industry forward.

First off, we need a name. A loose collection of good ideas doesn‘t generally get much traction in the world. There has to a be a name, a brand, to hang it on. Let's call it:

The GROWS™ Method.

GROWS in this case is an acronym, for GRowing Real-World Oriented Working Systems. It's an idea Jared Richardson and I (Andy Hunt) have been working on.

These seem like important concepts: software is not designed and built; that’s far too a deterministic, linear model that doesn’t work here. Growing is a better metaphor, because with growth comes change. Real-world oriented is a nod to the idea that we need to base all our decisions and direction on actual evidence: feedback from the real world, under actual conditions. Anything else is just some unfortunate combination of a fantasy and wishful thinking. And of course, working software, our ultimate deliverable. But not at the expense of a working, functional team and overall organization. The software, the team, the users, the sponsors all form one system. And we need to make sure the whole system works, for all the participants.

So let’s base our ”new” method on four foundational ideas:

1. Evidence-based
2. Dreyfus Skill Model
3. Local Adaptation
4. All-Inclusive

That is, every team should be able to inspect and adapt based on actual evidence, on real-world feedback. But to get them there, we need to use the lessons of the Dreyfus Skill Model to provide support for beginners and latitude for more advanced practitioners. With that framework in place, we can position every team to adapt themselves to local conditions, safely, based on actual evidence. And finally, whatever we attempt here has to work for the whole system, not just the developers, not just the managers, not just the testers, not just the users, not just the sponsors. There is no ”us” versus ”them.” There is only us.

In the next post, I'll take a closer look at each of these foundational ideas.

If you find this sort of thing interesting, please join us at growsmethod.com and sign up for our mailing list.

With your help, we can fix this thing.

—Andy Hunt

Now read Part 2 of this series: http://blog.toolshed.com/2015/05/its-an-experiment.html

Comments

[Keithb_b]

I suggest that before building a large structure on the foundation of the Dreyfus Model of Skill Aquisition you should go read Drefus & Dreyfus' original report. The “model” was invented out of thin air: they read a book about jazz piano, and a PhD thesis about chess and then they had a think. That's all. The model has no empirical content. The philosophical Dreyfus brother considers it a great strength of the model that it is unpolluted by data—I'm not even joking.

Before a certain subset of the IT industry latched on to the model it was thoroughly exercised by the nursing profession, in the 80's. Their conclusion was that it doesn't really help much.

[Andrew Hunt]

Keith,

All models are wrong; some may be useful. I suppose it's probably more fair to say that we are "inspired" by the Dreyfus model rather than following it slavishly, based on our experiences and direct observation. And with that in mind, it *has* helped us and has been useful.

And yes, I've actually met and spoken with Dr. Benner and others in the nursing field, so I am aware of the strengths and weaknesses of their approach to it. The bottom line is that skill, training, education, intuition, practice, governance — these are all very complex, interrelated issues, parts of which we understand well and parts of which we haven't a clue yet.

Any simple model, be it Dreyfus, or Shu-ha-ri, or any of the more academic learning models, is going to be brutally off the mark because it cannot take into account nuances of context and the full range of "yes, but.." that should be addressed.

However, you have to start somewhere, no matter how simplified or limited. And it is very possible that our interpretation and use of Dreyfus will evolve over time. In fact, I'm counting on it. Because the major thrust of GROWS is to elevate the idea of the Experiment to a first-class member of the method. It's all an experiment, we get feedback, we adjust, we move on. That's true of your project under GROWS, and it's true of GROWS itself.

More on that in the next blog post.

/\ndy

[Keithb_b]

Indeed “All models are wrong; some may be useful” but what is the Dreyfus model supposed to be a model of? It's hard to believe that it even is a model of skills acquisition, since they made no meaningful study of skill acquisition. It might be a model of the Dreyfus' prejudices about how they think skills should be acquired, particularly the phenomenological stance that it isn't possible (even in principle) to look inside expert behavior and see how it works.

Hubert Dreyfus, for reasons of his own, wants expert behavior to be, loosely speaking, mysterious and magical. I hope that GROWS isn't going to take the view that expert practice is software development is an un-analyzable, magical process where the expert “just knows” without thought or reflection what to do, and without the ability to explain it—which is what Dreyfus & Dreyfus claimed expert behavior is.

[The_chrismo]

Is GROWS agile?

[Andrew Hunt]

"Is GROWS agile?" I don't think that's actually the key question. The intent is that GROWS is a framework to help you grow and discover what works for you and your organization. We've learned a lot of what can work well from the agile movement, but we've also seen far too much "agile dogma" trump common sense.

So what we want to promote with GROWS is a return to self-determination, using what we've learned and valued from the agile movement, in a framework that promotes experimentation to a first-class methodological practice, that's inclusive and offers very concrete steps for beginners to help get them to the point where more abstract values and principles can be applied.

We're just starting, so there's a lot to write and discuss and present. But that's our goal, in a nutshell.

[Mfeathers]

Andy, do you know Greg Wilson? Among other things, he was one of the editors of the book Making Software, which has a lot to say about evidence-based approach.

I like the direction GROWS is attempting. It will be interesting to see you deal with the issue of evaluation of evidence and experimental results. I'm reminded of the book 'The Deadline' where the protagonist was able to run real world controlled experiments in way that researchers never are. Sadly, the effects of our decisions in development are often delayed and the etiology of the effects we see are elusive.

Emphasis on evidence and a good amount of argumentation about evidence would likely be an improvement over "this feels right" or "this seems to make sense." But it could be quagmire unless there's a time-bound decision process behind it and a clear recognition that most of the time we just don't know.

[Joefec]

I like this idea. I'm a big, big fan of agile and see it as a growing ecosystem of solutions, mindsets, tools, debates, and change. One core question I hope GROWS addresses is culture. I have been amazed at the effects of culture and how it responds to new ideas. So when attempting agile in a control-based culture, how do you approach it, or do you at all? Exciting !

[BuggyFunBunny]

Agile has always, and I mean always, struck me as a naive reaction to BDUF by folks who couldn't do BD. They offered up the notion that no process could be thought through a priori, so coding from the first moment, fixing in the debugger, pestering the users to offer up changes to UI or data or ..., thence rinse and repeat, was the way to go. IOW, cowboy coding with a mantra.

Agile ignores what the earliest craftsmen learned millennia ago: measure twice, cut once. But that actually means thinking through what one wishes to do. Nobody in the agile movement wants to do that.

[The_chrismo]

To me whether or not GROWS is agile is key, because it seems you are setting up GROWS as a replacement for agile - yet if it is itself agile, then there's a disconnect somewhere.

"We've learned a lot of what can work well from the agile movement, but we've also seen far too much "agile dogma" trump common sense." ... I'm curious how you intend for GROWS to not fail where agile apparently did?

I myself don't believe in the failure of agile. Any idea worth its salt will attract ignorance and corruption, and while no idea is able to keep these forces from acting, neither are those forces able to alter the original idea itself.

The presumption that agile itself failed to keep people from taking advantage of it strikes me as terribly un-agile. Agile is powerful because it directs us to manage the world as it is, not the world as we'd like it to be (BDUF). Now it seems you're writing off agile because it failed to do something it could never do in the first place. So, like BDUFers, you're going to 'just try harder' this time and do it all again.

But if GROWS is agile, and is another entry into the world of XP, SCRUM, Crystal, etc. then I have no issue with that. Principles vs. practices. One is ideal and solid, the other is fragile and needs iteration.

[The_chrismo]

Another thought: when I read "Failure of Agile" /me wonders what the postmortem of the manifesto looks like? I consider the manifesto a 'constitution' of the movement. Are you giving up on the manifesto? Do you think it itself is inadequate? Are you offering amendments?

[Dizzident]

The GROWS method. Here we go again.

Yes, I'm jaded after a decade of being slapped around the face with agile, SCRUM, KanBan etc.

[Philippeback]

Ah ah, what a good post.

I have been very active in 2001 on that Agile thing and then all of the "flaccid agile" (what a great term!) just made me sick.

So, like written in the Tao:

"Those who know do not talk. Those who talk do not know. Keep your mouth closed. Guard your senses. Temper your sharpness. Simplify your problems. Mask your brightness. Be at one with the dust of the Earth. This is primal union. He who has achieved this state Is unconcerned with friends and enemies, With good and harm, with honor and disgrace. This therefore is the highest state of man."

I have chosen to go do real work and projects in the trenches and get products out of the door.

Problem solving is problem solving, no matter what labels we stick on it.
As is execution. No amount of rha rha is ever going to help with the sweating.

Phil

[Andrew Dapkiewicz]

The failure of Agile is people not reading past the first page of the manifesto. So your solution is to create a new methodology or rebrand Agile?

You bring up the same concerns that people have been saying for years of 'Why Agile Doesn't Work' and do not even address the underlying problem that people are in environments where they inject a standup every morning and state with assertion they are now an 'Agile Shop'.

Unless that is magically addressed with GROWS, I think I'm going to be able to predict the future...

In 14 years: "The failure of GROWS".

[Kartzontech]

"How did we get into this mess?" In my observation, the real and hard answer to this question is that which everyone knows and no one would dare to come forward pointing it. And it is "Commercialisation", the root cause for the failure for genuine Agility.

So long as we park our values and try to commoditise and commercialise "anything and everything that sells", we'll remain in this crappy situation :(

Lastly, I remain your follower and have appreciations for your opinions and insights.

~KarthiK~

[Toddwprice]

Bravo!

I've always bucked at methodologies that are practiced religiously. If rules become more than a guideline for how a team or group should relate, then people begin to see the rules as flawless laws handed down from above. Any deviation from the perfect laws is heresy, and groups have various ways of dealing with heresy. This is true in business, religion, family, government, and yes, even software development.

The beauty of the original agile manifesto is that it says very little about HOW, and a lot to say about WHAT. It's a vision for where we should head, but not an exact roadmap for how to get there. Each team must decide that on their own. Granted, there are many time-tested principles that can be applied. We don't need to reinvent things all the time. But it takes a seasoned leader to recognize when patterns are useful and when they don't apply in the current context.

For this reason I'm not in favor of a new name. I think it eventually will become a new set of rules to be religious about. Focus on "inspect and adapt", and a set of principles to help you do just that. The simpler the better. If people demand more rules, don't give it to them. Demand relationship and dialogue from them instead. That's the antidote to rules, and it will keep teams working creatively rather than mechanically.

[AndyBrandt]

Are you going to create Certified Grows Professionals (CGP) and offer Certified GROW Classes led by Certified GROWS Instructors (CGI)?

[Andrew Hunt]

No. We will not promote the same old useless pay-a-fee/multiple-choice-test certification garbage that permeates our industry.

In the spirit of keeping everything based on outcomes, any sort of certification or rating should be done from real world results. If you think you have a great team, how do your users rate you? Now there are issues with that sort of a system as well, but I think it's a better direction than the old-fashioned, outdated, pass-a-test and you're certified nonsense.

[William Stone III]

Hm. There seems to be no way to delete my prior post, in which I hit too soon. Apologies.

In my experience, the blame for Agile failures -- or ANY programming or infrastructure project -- lies squarely at the feet of management.

Management doesn't understand what we do. At all. We're a giant black hole sucking cash as far as they're concerned.

I've worked for more than one company where Dev reports to Sales. I'm sure you can imagine how well that works. Sales sells things that don't exist, sign an arbitrary contract date by which it will exist, then tell Dev to drop what they're doing to make it exist. Never mind that the deadline in no way matches the real-world time to implement.

Hell, they've no idea how long it will take to implement. They just sell it, sign a date, and then the Devs are on the hook

I've seen it over and over and over. How can any developer (or sysadmin, like myself) function in environments where no one understands what you do, don't care, and expect miracles as a matter of course.

No development methodology -- Agile or otherwise -- can survive modern Management.

If you want Agile to succeed, you need to educate Management first. I've been in the game for 30 years, and I still have no idea how you do that.

[Jason Pascucci]

Question (or, maybe, a challenge):

Can you "open source"-ify the description of a development _methodology_ so that you get some of the benefits of open source (many eyes, customizability, testing).

I sort of envision specifying it via a directory structure plus documentation files, with human-editable file organization (i.e. not too much or too little text in one file) that 'compiles' into one document. Then, put that document source up on github, which would allow me (or someone) to fork it and make my changes, and offer pull requests upstream (to see if you think they or something like them would be valuable), and allow people to fork my repository downstream. Then, in an empirical fashion, you could see what modifications people feel strongly about.

I'd envision having one 'index' page in the top most upstream, which additions could be very freely pushed to (unlike other push requests that you'd or a central team would curate and review to keep the upstream up to your desired purity), which anyone could link a url to their github repository, and a paragraph about their fork of the methodology and the reasoning behind it.

(For instance, I feel strongly about "doing agile" on something more like a dual-track, one more standard agile team that responds to user stories for UI stuff and the supporting mechanisms, but then a second, separate more "waterfall" library team that standardizes central and shared implementations, responding not to user stories/feature requests but to conglomerate individual implementations that were required to do a particular user story, etc. I would totally fork agile or an agile-replacement to implement rules for this sort of bifurcation of action to produce a maintainable project.)

[Dan Schertner]

Do you have change for my paradigm?

[TromboneTiger]

Thanks for the article... I think you just described my vision of DevOps (at least partially): http://plandoing.blogspot.com

[Nicolas Vérité]

"
So let’s base our ”new” method on four foundational ideas:

1. Evidence-based
2. Dreyfus Skill Model
[...]
"

But, from Wikipedia:
http://en.wikipedia.org/wiki/Dreyfus_model_of_skill_acquisition

"[...]
4. Expertise
[...] intuitive decision [...]
5. Mastery
[...] intuitive decision [...]
"

Is there a conflict here between evidence and intuition?

[Andrew Hunt]

I don't think there's a conflict between evidence and intuition; you lead with intuition, but follow up with evidence. Intuition is extremely useful, but far from perfect. In any case, take the idea and try it. Prototype it. Trial it as an experiment with the team (more on that in the next post). Then you begin to get evidence.

[srikanth ramanujam]

Interesting article and I can understand your frustrations around Agile, or the lack of perfection around it. Well, in an imperfect world where everything is really flawed, the human race continues to want predictability and certainty, and no model is every going to fit and solve every value problem.

Agility will evolve in the next several decades, as did waterfall and other processes before it. There will be a few who will truly agile and many a company practicing so-called "Agile" will fall by the way side and be replaced by a fresh set of players who would have really jumped the competition with marginally better execution, but may or may not be due to their use of Agile.

Welcome to the walks of reality of what we call "life". The only certainty is:
- People don’t change unless they want to
- People may change only at the pace they want
- It will be impossible to make every one conscientious
- It is what it is!

[SixSigmaGuy]

Great article!! I’ve seen so many great movements go this same way, e.g., Six Sigma, CMM, Hungarian, TCO, etc.. After some new methodology/process has been proven, it becomes a buzzword and everyone wants to get on the band wagon without fully understanding it. They implement it, partially, promote that they’ve adopted it, and fly the banner. Then, some years later, they claim it doesn’t work and blame the methodology; yet, they never implemented it correctly in the first place.

[Corey Burnett]

When I boil down the descriptions of things like Agile and GROWS it always seems to be just "...do what works best for your organization. Embrace change. There are no concrete rules. If something doesn't work, stop doing it and do something else. Be flexible...". In my mind, that really isn't anything helpful. It's a motivational poster. It certainly isn't any sort of methodology. At the end of the day I work for customers with fixed budgets and tight schedules. They want to know what they are going to get (within some reasonable expectation) before they will sign an SOW. Descriptions of Agile seem so ephemeral and magical. There doesn't seem to be any substance. Yes, we need rules. If you don't have rules then it really isn't a methodology. It's just a bunch of ideas that you may want to or not want to adapt. It's like saying "I have found that if I drink coffee before 9am then I am more productive and I get more done. You may want to try that too. But if it doesn't work for you, then don't do it." Ok - got it. Very helpful. We may or may not add that to our process. Teams and companies want processes they can put in place that are helpful and repeatable, not a bunch of ideas that may or may not work that you have to work with for years to find out what is best for your company.

[Rafaelesantos]

I am curious about one thing. It seems the kanban community came to the same conclusions a long time ago and the kanban method was born as a change management framework with principles that seems to handle all your concerns.

Start with what you know
Visualize your workflow
Limit your your work in progress
Pursue incremental, evolutionary change
etc.

Why do we need GROWS if we have the kanban method? Why another methodology?

[Thomasjeffrey]

Andy,

This is a good post.

I feel it misses one important point in my experience helping clients introduce a job I've noticed that the one key success factor is the mindset of the executives running the company.

Unfortunately, there There are a few senior leaders out there who actually understand what agile is really trying to achieve and how it's trying to achieve it. To them as I was just another method or technique that can be adopted as it is to achieve better results.

If we take this and add the fact that the vast majority of agile practitioners do not have the tools to help executives change their mind set, then we are left with a problem that is only fixed when the next generation comes into leadership positions.

There are a number of thinking tools and approaches gear towards agile oriented change coming out and I think many of these will be useful, but they need to contain patterns, case studies, and perspectives that can help executives make the necessary mind shift change in a way that meets their objectives.

[Marc]

Certainly, there's a need for an evolutionary approach to frame and guide software development endeavors. Also, to divert from the obsolescence of the (still lucrative) mechanical deployment of the so-called "agile" methods. Hopefully, blind ignorance won't prevail.

Here some humble suggestions for the foundations of this method. (1) Should be handy to consider Lehman's laws of software evolution, and its associated dynamical model, in the GROWS conceptual framework. (2) Regarding education in a broad sense, since one of the fundamental concerns that you pointed out is the lack of openness to learn―"Agile methods ask practitioners to think, and frankly, that‘s a hard sell", the notion of "short-circuit" and long circuit in pedagogy might be of your interest for further reflection.

Thank you for bringing new paths to our industry.

[__lorenzeau__]

The problems I have seen have been people related. Management related, to be specific. We talk about "servant leadership" but the Agile evangelist(s) in management want "agility" done their way. "Agile dictatorship" is an oxymoron for a reason.

[Trisweb]

This is a great post with a lot of truth in it.

I hope you've studied Deming and his model of organizations, which he called the System of Profound Knowledge. It is like yours but I think it's more complete and you could learn some things from it.

The four points in GROWS are exactly analogous to Deming's system: scientific knowledge approach (evidence based), knowledge and understanding of psychology (skills method, but far more complete), knowledge of the statistical consequences of variation (local adaptation, but stats gives you the ability to do it well), and finally, knowledge of systems (all inclusive, in an even better way). Exactly one to one.

Deming said a few other things as well, important among them that management must lead the organizational system -- leadership alone can transform the system. This is a huge deficit in many agile organizations, where no true agility is possible under the umbrella of a distant or worse, malignant top management.

Finally, Deming prescribed a process of continuous improvement by which the method of work was always to change and improve, under the framework of knowledge, statistical understanding, salience of psychological impact, and of course, systems knowledge. It is all of those things that enable agility and the ability for adaptation.

If you're in this world, you should know about Deming, and all his contemporaries. This is a rebirth of an idea that has seen a lot of road time. There are even software companies out there that use his concepts wholesale: Pluralsight is one. Take a look.

[Krishan_mathis]

Great post.
It describes adequately a lot of the shortcomings of the majority of agile implementations.
If I had a Christmas wish, it would be a little broader focus: many of teams' problems stem from the outside, e.g. the lack of a culture of trust, lack of autonomy etc.
I have written down my thought about the post in: https://improuv.com/en/blog/christoph-mathis/failure-cargo-cult

[Andrew Hunt]

Part of GROWS is to involve everyone involved in the effort — especially the executives. They need to know what they're getting, and what's expected of them in return.

[Andrew Hunt]

I think there are certainly similarities, as you'll find in any Lean/Agile approach. But there are some important differences, for instance, the PDCA cycle starts at the wrong place, and uses the wrong words. And unfortunately, as we've discovered, words do make a difference.

[Jeff_vee]

We won't lose our way of we always remember the principles of the agile manifesto. It's disheartening when you compare the simplicity of the manifesto to all the spew in some Scrum presentations.

Loving this idea by the way.

[Andrew Hunt]

Sure we can still lose our way. The problem is that the manifesto can easily be mis-interpreted and mis-applied. We see that happen all the time. The folks authoring that "spew" likely think they are, in fact, honoring the manifesto. Alas, they are not.