« Anyone know why trackbacks aren't working? | Main | Not with a bang.. but a whimper »
September 21, 2004
How Agile Development Ruined My Career (Sort Of)
This is a reprint of something I wrote in May. I lost most of my site, so I'm slowly trying to recover it.
This is a rather painful story to relate, but I think it needs to be told. One thing I want to make clear right off the bat is that I think that every person in this tale was doing what they thought was best. No one was acting stupidly, or maliciously. At the end of the day, it was a difference of opinions.
Introductions
I started at XXXX in March of 2003, as the Senior Director of R&D, responsible for the development group of 8 people. At that point, we were trying to wrap up the delivery of a "small" release of the system, a release that had started in January with a month of architecture and design, coding in February and March, and the goal of QA delivery at the end of April or beginning of May. As you can see, we weren't off to a great start, in terms of agility.
Agile Elements
There was a nightly build, and a smattering of unit tests, but little else that one could consider agile. Much of the system architecture was opaque to the developers, and in general people didn't know or understand how they interacted with other parts of the system. It was also pretty apparent that the team was pretty down.
I immediately established three practices to help improve the system. Borrowing a page from Scrum, I instituted short daily meetings, where everyone talked about their deliverables over the last 24 hours, and the upcoming 24 hours. To be fair, this had been partially implemented before I started, but I tried to make it rigorous. I also established architectural cross-training, trying to get the team talking and interacting. Lastly, I established a set of core tests that a developer could perform to verify the "heart" of the system - with the goal that once this heart was working properly, we could start to transition the code to QA.
The release stretched on until July before it was finally ready to be put into production. But it was a solid release - well-defined, well-tested and ready for production. It would be our core platform for the next 9 months. And, in spite of the time it took to bring the release to production, I was rewarded - I was given responsibility for 4 additional developers who had been focusing on short-term customer projects. This was great for me, since I had been pushing for the title (and associated salary/options) of VP of Development. It seemed like I was on the right track, and that things were going well.
The Long Haze of Pain
Emboldened by my success, I decided to push farther into agile practices. I started preaching the value of pair programming, and encouraged everyone to try it. I taught classes on unit testing and test-first development, and we added automated testing to the nightly build. We formalized the Scrum process, by establishing a backlog of bugs and features for a given release, and creating the Scrum burndown tracking and following 1 month releases.
And this is where I started to hit challenges. The QA group wasn't comfortable with the lack of structure. They wanted documents to be written, explaining design and architectures so they could understand how to test the system. I countered by suggesting that they test during the Sprints, but I didn't get much buy in. Eventually we decided that the developers would test to the limit of our capabilities before handing it over to QA, and in general, the quality of each release was quite good.
My second challenge was churn. I was continuously assaulted with a wide range of short-term projects and longer releases, and I had to put a lot of brainpower into getting the right people into the right project at the right time, while not jeopardizing the previous projects.
My third challenge was employee negativity. Some of my employees didn't think much of the agile approach - they were comfortable with lots of upfront planning, and felt that the lack of up-front planning was offensive, and that I was immature and unprofessional for following this approach. But they didn't tell me this - they told my boss. If my boss had believed in my approach(es), this probably wouldn't have mattered, but as it was, it became a major issue over time. But I'm getting ahead of myself.
Interactions with my boss
My boss is a fascinating individual. An ex-GE mid-level executive, he has many years experience successfully running large projects via the waterfall approach. He'd also run a small software company and been CTO of a dot-com recruiting agency. We had a number of discussions over time, and it became clear that he wasn't impressed by my push for agile practices or my management style. We talked at length about planning, about how we could avoid all these challenges I was facing if we just spent more time planning. I pointed out that give sufficient planning, you end up with a waterfall development model, and he denied wanting that. But in the end, I believe that he thinks that anything less than a fully drawn out project plan, (fully defined PERT chart with names, hours and dependencies , with every risk planned for with a contingency, with every resource optimized and precisely scheduled) is the sign of an unprofessional, immature project manager.
Were he to read this, I suspect he would deny feeling this way - but I had the opportunity to watch many projects while I worked for him, and he heaped praise on projects that followed this model, and ladled criticism (or simply said nothing at all) about projects that didn't follow this model, no matter how successful they were.
Big Stupid Mistake #1
Do not try to introduce agile development methodologies when your boss is a high-structure operations guy. In my defense, when I started introducing agile techniques, he wasn't my boss. But in retrospect, once he became my boss, I should have dropped Scrum like a hot potato and brushed up on my PERT charting and 3 - 6 month releases.
Many of you (especially the HackNot crowd) will be saying "He was right, and you got what you deserved." This may be true, but the nature of our business was that:
* there was no roadmap - new projects were decided on usually only days before commitment. In no case that I can recall did we actually work on a project with more than 2 weeks of notice that it was important and needed to be done.
* customer projects were the highest priority, and we had about 20 we had to complete, sometimes with dates specified by sales, instead of by the development team, and often with 1 or 2 days notice.
* We had a bonus pool that was defined by the number of customer projects we completed, and everyone scrambled to pull in as many projects as possible to get them done in the shortest amount of time.
And so I worked very hard to keep everything going - delivering projects, getting the team to build unit tests, setting up the rhythym of daily meetings and monthly iterations. And the whole time, my boss was consistent in his disappointment and criticism with my methods and my results - always in a very nice way - he is by no means an ogre. But in some ways, the differences are striking. For example, on at least one occassion, I pointed out to him that adding the additional structure he wanted would reduce the number of projects that could be delivered. His response was - "sometimes you need to tell the customer 'no'." But I wasn't about to go for that, not when the entire company's bonus depended on my team's delivery of X projects in 6 months. I was going to do whatever it took to make sure that we delivered X+ projects. So I worked even harder, and at the same time tried very hard to internalize my boss's criticisms, which only made my life harder because I started to second guess every decision that I made. Which leads me to:
Big Stupid Mistake #2:
If you're in a no win situation, don't try to win, at least not by conventional means. I was sure that when the dust had settled from the 6 month bonus plan, my boss would recognize the effort and value that my approach had brought to the team. In retrospect, I was a fool for letting myself believe that. And now that some time has passed, and with some feedback from the team here, even if I had done everything just the way he wanted it done, I still would have failed, because other parts of the company would have been disappointed in the slower delivery and excessive structure for what should have been an early stage company's desperate focus for a sustainable business model.
Some positive results, too little too late
My boss did have some good suggestions - I adopted a Risk Census, partially at his suggestion and it worked great. By February of 2004, we had gotten pretty good at delivering these iterations on time and in scope. Unfortunately, it was too late for me. The last project I was in charge of was delivered on time and in scope. On the day it went to QA, I went to the senior management team, excited that we were finally getting proof of the value of agility, but the response I got from management can best be described as "tepid."
Three days later, I was demoted and reassigned to a different line of business, although I was still reporting to the same guy. To be sure, I made a number of mistakes outside of my push for agile methodologies, and it is certain that my demotion was not simply "because I advocated using agile development". On the day my reassignment was announced, I received my only public compliment from my boss, a grudging, lukewarm acknowledgement that the last release was on time and in scope.
You could wrap the story here, but I've learned that by redefining the endpoint, you can make everything turn out well in the end.
Second Try
My new job was actually quite interesting, and since I was working for a particular LOB in this 40 person company (heh), my boss pretty much left me alone, and I was able to accomplish a lot of good things. That is, until my boss decided to get involved again, and started up with the planning and the roadmap and the resource optimization, etc, etc. This, in spite of the fact that my customer thought I was doing an outstanding job, had been very vocal about the great things I had accomplished, and were able to work together to solve his problems in whatever priority and balance he felt was necessary. All the old anxiety flooded back in, and I decided over that weekend that I would rather be unemployed than continue to work for him. Again, not because he is a bad man, or mean or evil or stupid - but simply because he and I fundamentally do not believe in each other's approaches to software development.
Big Stupid Mistake #3
Don't make the same mistake twice (Luckily, I didn't)
New Beginnings
In the end, I didn't have to quit and go home. I landed a great job at a much smaller company (I'll be employee #1), where my boss (an entrepreneur and mid-size company CTO) believes strongly in agile approaches and has had great success with them.
And they lived happily ever after (I hope)
UPDATE: I changed some confusing text in the description of my boss's background.
What I did wrong:
Well, I can spot a few obvious things:
1) I wasn't very good at defending agile practices, which probably made me seem less competent and less professional than I would have if I had been able to rattle off responses to challenges immediately. That's partially a matter of education, an area where I have gotten better, and partially a matter of my personality (INTP) where I almost effortlessly try to adopt the frame of reference of my opponent.
2) I should have been more gradual in my introduction - I should have started with more up front planning and structure, if for no other reason than it would have made the agile thing easier to swallow.
3) I didn't adapt well to the corporate culture, try as I might (and again, because I am an INTP, I tried _very_ hard).
4) I was in a situation where I had very little control over the inputs to my team (in terms of work, etc), but I was expected to keep everything flowing smoothly and sedately down the river of delivery. This was essentially impossible, and rather than ruin my mental health, my car and my home life, I should have walked away sooner.
Anything else that anyone spots that's worth noting?
Posted by jb at September 21, 2004 11:34 AM
Trackback Pings
TrackBack URL for this entry:
http://www.undefined.com/cgi-bin/mt/mt-tb.cgi/26