Agile Is Hard

This is the fourth blog post in my series on the current state of Agile. In the first post, I shared my view on the changing sentiment towards Agile and some of my own history and experience with it. I also identified four factors that, in my opinion, contribute to the current state. 

In the second blog post, we looked at the first of these four factors: Agile’s rise in popularity.

In the third blog post, we looked at the second factor: “Agile” has lost its meaning. 

This is the fourth blog post in the series. We will look at the third factor: Agile is hard.

A Reason for Change

In May 2018, my wife, my then one-year-old son, and I were spending time in Canada and the US. We went out for breakfast in Philadelphia, PA, with my wife’s family and some friends. One of them remarked on the fact that I had just finished a huge breakfast burrito by saying “I can’t believe you ate that entire thing!” At that precise moment, the waiter showed up with my full stack of peanut butter caramel pancakes that I had asked him to serve after the breakfast burrito. I ate that too. It’s not like I had a light lunch after that breakfast, or a light dinner after lunch. 

Lying in bed that night, I thought about how unhealthy my behavior was. I decided that things had to change. 

That wasn’t my first commitment to weight loss. Far from it. I had actually been trying to lose weight for about a decade. It never worked. I had read all kinds of books and really knew quite a lot about nutrition. I had tried all kinds of diets and different kinds of workouts. When I had lost a few pounds they quickly came back. During that time in North America, I decided that, once home, I would try one final time to lose weight. Otherwise, I would resign myself to just being “the chubby dad”. 

Over the course of the next year and a half, I lost more than 50 pounds of body fat and have successfully kept it off ever since. In a way, my weight loss was super easy: I consumed less calories than my body burned. That’s it. And that’s also what I had tried – and failed – to do countless times before. So what changed? My relationship to food. I had a long, hard, and painful look at my underlying beliefs and how they influenced my behavior. For the first time, they didn’t always pull me back into old habits.

Agile Is Easy for Some and Hard for Many

My process of weight loss was remarkably similar to the Agile transitions I tend to see with my clients: Something that seems very easy at first glance becomes exceedingly hard for most people.

I have two sons now. If you have been paying attention, that is one more than I had in 2018. The older one is six, the younger one four. Watching them eat is absolutely fascinating to me. For one, they don’t overeat. Even if there’s something they really enjoy, they stop when they’re full. I have never heard them say “Oh man, I really shouldn’t have eaten the whole thing.” It simply doesn’t happen. They are also incredibly active, running and jumping around all day long until they go to bed (where they move around some more until they finally fall asleep). Due to these factors, it is very easy for my sons to stay lean. They simply haven’t developed unhealthy habits. Maintaining their current amount of body fat is almost effortless.

In the same way, I occasionally encounter Agile teams that make the whole process look like child’s play. Deploy to production multiple times a day, inspect live product data during the Sprint Review, make strategic product decisions together, talk to actual customers constantly. Why wouldn’t you?

These teams usually started out small and without a lot of organizational barriers around them. Direct customer interaction was there from the start. They focused on test automatization and flexible architecture from the get-go. They adapted their own process. Slowly scaling that approach, and improving it along the way, made staying agile almost effortless. For them, Agile is easy.

But most of my clients? They struggle just as much as I was struggling trying to lose weight. And just like my old habits kept pulling me back, my client’s habits are doing the same. For them, Agile is hard.

Competing Commitments

One approach that has influenced me more than others is Robert Keagan’s “Immunity to Change”. Here’s a short summary: 

We often set out to achieve specific goals. After a while, we discover that we are not achieving those goals, even though we might care deeply about them. When we stop to investigate what is going on, we discover that our actions are not in line with what we are trying to accomplish. We might act in a way that is actually opposed to what we say we want to achieve. Immunity to Change helps us look at our actions and derive so-called competing commitments from them. Those are hidden goals that we have and that guide our actions but that we are not consciously aware of. These competing commitments make us work against our actual goals. We subconsciously follow these competing commitments because we have deeply held values and beliefs that inform our actions on a deeper level than our espoused goals.

In my case, I seemed to value staying lean and eating a healthy diet. But over time, I discovered that I had tons of competing commitments. For example, I noticed that I would always finish food on the table even after I was clearly full. The thought of wasting food bothered me so much that I would rather endanger my health than throw it out. Nowadays, I know that there are other options, like eating it the next day or giving it to someone else. But those options would never occur to me back then. I was essentially blind to them. And so I would eat way too much even though I didn’t want to.

I also noticed that parties and family gatherings were always really tricky for me. Those were moments when I would overeat like mad. There is a Liberating Structure in development inspired by Immunity to Change. It’s called Talking With Pixies [the link is going to open a Word document]. There was a moment in our Liberating Structures User Group in Hamburg when we were using it and two guys were giving voice to the silent competing commitment in my head. I had told them about my issues with declining food that someone else had prepared. Behind my back those two guys were going at it, saying things like “Do you really want to break someone’s heart by rejecting food they lovingly made? What kind of person are you? Just take a bite!” I laughed but I actually wanted to cry. It was a painfully accurate manifestation of the actual voice in my head. 

Gaining awareness of those competing commitments helped me tremendously. Whenever I encountered a tricky situation I was able to identify what was going on. “Yup, that’s the voice. I would rather gain weight than endanger a relationship by declining food.” And suddenly, I had a choice. I could act on my first impulse or I could choose something else. Increasingly, I started choosing something else.

What Makes Agile So Hard

My clients make a commitment to Agile. But they are full of competing commitments. And that is the thing that makes Agile really hard. Sure, they want all the benefits of working that way. But that usually means they have to let go of old behavior, values, and beliefs. And those are not only ingrained in people’s minds, they manifest in the company’s structures. Making Agile work would mean a tremendous change. Often, those companies are not ready for that change. 

Let me illustrate what I mean. While there’s an overt commitment to Agile within an organization, the dominating paradigm optimizes structures and behaviors for predictability and efficiency. Agile optimizes for flexibility and effectiveness. And this causes friction. Without the existing paradigm, Agile would be easy. With the existing paradigm in place, it becomes hard. 

If your company is struggling with Agile, I can practically guarantee that it is related to at least one of the following points:

• Frequently producing working and fully integrated increments within a short time frame
• Transitioning from individual siloed experts or component teams to cross-functional teams
• Limiting work in progress, prioritizing opportunities or work items as a consequence of the limitation, and then actually sticking to prioritization
• Confusion regarding roles and decision-making authority caused by the (theoretically) increased autonomy of Agile teams
• Putting Agile teams in direct, frequent contact with customers or markets (note that I’m not talking about internal customers here, see our post on Stakeholders)

Why do I know that you are struggling with these issues? Because they are typical of organizations that try some form of Agile but actually optimize for predictability and efficiency. Are people within the system aware of this? Most likely not. But the structures and habits reflect the values and beliefs. When you are “certain” about what you want to release you neither have to be in touch with customers nor release often. Customer Distance doesn’t matter. You can ship large batches infrequently and don’t need to worry about dynamic reprioritization. Just cram as much as possible into the next release and then repeat the cycle. Make sure your experts are busy at least 100% of the time. Hand them fragmented pieces of user requirements but never give them any end-to-end responsibility. 

This won’t work well with Agile. But these are solvable challenges! Trivial even for a few companies. What is interesting is what makes them so challenging for most organizations. And that is where the competing commitments come into play. While there might be a goal to change the paradigm, people will get pulled into old behavior all the time. Or they won’t change organizational structures that inhibit agility, because they are unable to see how these structures are optimized for something else. 

People engaging in these behaviors are trying to be successful. However, that behavior is not aligned with the organization’s commitment to Agile. And that is precisely what makes Agile so hard. Not building it from the ground up – which is often relatively easy – but changing from one paradigm to another. 

The False Allure of Agile

Getting started seems so easy: just pick an Agile framework and get going. The Scrum Guide is only 13 pages long! Kanban only has six practices! 

It’s as easy as losing weight. Just eat less and move more! Have more vegetables and fruits! 

There’s a quote that I used to share that has been removed from the Scrum Guide:

Scrum is:

• Lightweight
• Simple to understand
• Difficult to master

Whenever I introduced Scrum to a group, this is what I would show them. I wanted them to prepare for the way ahead. I wanted them to know that it wouldn’t be easy and that struggling didn’t necessarily mean they were doing anything wrong. The same applies to Kanban, of course. Yes, it’s an evolutionary approach. No, that doesn’t make it easier. Maybe easier to get started, but probably not easier to make a significant change in the long run. 

As Agile Coaches we should give the people we work with a realistic expectation of what they are getting into. Otherwise, we get what we see now: people questioning whether Agile has ever worked anywhere.

The journey is tough. But we can help people by being honest about what awaits them. We can help them by figuring out what success they are striving for at the moment, how that is reflected in their values and structures, and what would need to change for Agile to work. We can help them identify competing commitments and prepare them for the fact that there will be forces working against their success. 

If we don’t, we will turn into the equivalent of the diet industry that has been selling magazines, ebooks, supplements, videos, and false hopes for decades. We are advertising the next flashy approach that doesn’t solve the underlying problem and leaves people dissatisfied, only to start selling the next thing. Come to think of it, maybe that is what we are doing already.