When (not) to document?

Documentation, it’s a boring job, but someone has to do it, right?  That’s what process advocates say, but maybe that view is overstated.  To think about that question, consider the reasons and flaws of documentation.  On the plus side, it’s hard to argue documentation is bad; it’s additional data.  In theory, documentation helps others understand something non-intuitive.  Managers hope documentation will allow replacement of one employee with another.  They say in case of accident (“hit by a bus”), but we know the reason is in case one employee quits or demands more money.  In another case, it educates users, and reduces the need for hands on training.

Documentation doesn’t always live up to those goals, but it at least partially accomplishes them.  But I see two primary problems with over-documentation.  The first, it is better not to need documentation, then to have it.  A goal of 100% documentation not only distracts from the quest to reduce it’s need, but as I’ll explain, it’s often an obstruction as well.  The second problem, is time.  Creating, maintaining, organizing and disseminating documentation is time consuming.

The second problem is obvious, and there is little to be said.  Each project must make it’s own decisions on when and how much documentation is appropriate.  Whatever that decision is, insure the entire lifecycle of documentation cost is considered.

The idea of fulfilling the goals of documentation through intuitive design is much more interesting.  It’s a commonly accepted practice in the development world that the design of classes, method names and other elements of code should strive to not need documentation. 

If you have the choices of creating a method void Execute(byte[] array, string input), and then providing documentation stating, “The Execute(ref byte[] array, string input) method reads the contents of a file with the input path input into a byte array, array.”, or simply creating a method byte[] ReadFromFile(string filePath) with no documentation, any rational developer would choose the second.

It’s not impossible to do both, especially in the case above, but it can be difficult.  I already pointed out it’s time consuming, but documenting what doesn’t require documenting is most tedious.  Repetition is the fuel of tedium, so don’t repeat yourself.

It’s not just code that benefits from the rule of eliminating the need for documentation.  The highest goal of any user interface designer should be an application that is intuitive, an application that trains the user through use.  RTFM is a nice phrase, but it’s been repeated so many times we should question it’s wisdom.

Interaction vs. Stimulation

I’ve read in On Intelligence by Jeff Hawkins, that a significant feature of the human brain is the way the neocortex is connected to the thalamus (I think I have the regions right, but the books on loan so I can’t check right now).  Generally, there is a more direct connection between motor control and the “learning” regions of the human brain.

Since reading this I’ve theorized this as an explanation of why humans early development of human babies is so slow in comparison to other species.  Elephant babies walk within minutes of birth, yet it’s a year or more before human babies walk.  Gorillas are closer to humans but still quicker.

But humans reach levels other species never do, and one reason might be that a more direct connection to motor skills produces more valuable feedback.  In a sense, it would make life more interactive.  Rather than interacting with the world through a instinct based filter, actions and reactions would have a direct loop.

So it was reading GristMill about educational TV’s lack of positive effect, that I was thinking back to this.  Why is TV such a poor tool for learning?  It can be a fairly rich experience, but it’s never interactive.  And interactivity is key to learning, especially in humans.

But the study doesn’t just suggest that TV is no help, it suggests it is harmful.  It really states that a child may have been better off starting into space than watching TV.  That’s right, it’s not just a problem of substituting TV for valuable parent time, it’s just plain bad.

The researches theorize that the babies are simply over stimulated and unable to process important interactive information when it’s available.  That may be, but there may be even more going on.  If children are trained by TV not to interact, because the TV does not respond, or that they don’t need to interact, because the TV responds the same either way, then they won’t be trying as hard during parent time.

I think that is a lesson that could be applied much more broadly.  I find one of the major differences between good and great programmers is their “research” instinct.  When a bad programmer is stumped, he resorts to direct questioning.  That may seem more interactive than a Google or MSDN search, but in reality it’s not.  The person you ask either knows the answer or not (often not).  With the other options you’re required to find the answer.  Do this enough and you’ll learn a lot more than if you had the answer spoon fed.

That topic brings me to my last thought, TV vs. computers.  Computers are often grouped with TV by parents as two of the same.  The glow, they make noises, and they suck time in great quantities, yes.  The similarity doesn’t go much farther though.  Computers are interactive in a way TV’s are not.  It’s possible to find non-interactive material on a computer, but unless trained otherwise most humans opt for the interactive option.  Actually, maybe that applies to animals as well, just think of the cat playing with a mouse.  It’s fun till it stops moving…

The Flux

Continuing a chain of thought, Cosine posted an idea for Process Teams.  He concludes:

A company may end up with a situation where every department ends up with its own version of almost every policy. But then again, is that really a bad thing? I do not think so.

I understand what he’s looking for but I have to dissent here.  Every department with a separate policy would be a bad thing.  Fortunately, that’s not the probably outcome of his idea, with a little good management.

All you need to do is, in a non-sadistic manner, monitor the different departments.  When one begins to diverge, that’s okay.  But if after a reasonable time if their policy isn’t producing appreciably better results internally you should steer them back to the standard.  If it is appreciably better you should start steering other departments in that direction.  If it’s outstandingly better results, you have a new standard.

I should caveat this with one thing to watch out for.  It’s possible the new policy a department has adopted only works better for that department.  If it’s outstandingly better then you might have a case for divergent paths, rather than a new standard.

The problem with a central policy isn’t generally the sub-optimal results of the subservient departments, but the static nature evolved from having too many simultaneous inputs.  If you rely upon a central policy team to evolve you’re polices it will fail to keep up with the changing world.  Then you have bad policy.

Shepherding is an important role and technique in many situations.  From the outside it can look a bit frightening because it’s appears more disordered.  In reality, you’re simply allowing the systems natural flux to maximize.  Reducing flux will reduce the apparent disorder, but at the cost of the systems energy.  The best managers redirect energy, rather than suppressing it.

Agile Architects and Agile Developers

I’ve been experimenting with agile development practices for the past year. Actually it’s longer than that, but I never called them agile practices before, and some I had never tried before. It’s been slow going because I get to do less and less development all the time.

I have however been happy with the results. Test Driven Development emerged somewhat naturally from a greater focus on unit tests, and more coding in unit test friendly languages, rather than the unit test agnostic C++.

Ironically, my only opportunity to conduct pair programming has not been the two-coders in a room together situation, but one with me in Chicago and the other half in California. It did however work extremely well despite the separation. My second monitor was invaluable, and the biggest regret was my California compatriot was stuck with only one monitor. A more hands free audio setup would have helped too.

The reason I’m losing my development time is because I have more and more responsibilities as an architect, and while I don’t intend to ever let go of coding entirely, I do have to approach it with a different purpose these days. I code to support the architecture; to educate, to demonstrate and to investigate. I unfortunately must leave delivery up to others. It’s not just a matter of insuring I don’t have a 48/7 job, but also a vexing realization that allowing delivery to depend upon me is not wise. I can’t readily abandon an architectural duty, but just as much cannot allow delivery to be held up by my other duties.

So I must settle for helping out when possible, hoping the developers will respect my talents enough that they won’t go to waste. That’s difficult because it’s not something that comes easy to most developers. When it works some of my experience is added to that of the developers but delivery remains non-dependent upon me. When it doesn’t it’s a waste, but that’s life eh?

My approach to architecture so far has been very common sense based. Some organizations seem to think they can improve upon this through a “framework” or process like FEAF or Zachman. Truthfully I find none of these appealing. They’re rigid in a way that doesn’t relate well to any successful architect I know. I think they are worthy of reading and understanding, but I don’t think they’re worthy of practicing. They attempt to take everything an architect might ever want to do, and sequence it. This means there’s plenty of good ideas. The bad idea is trying to do all of them all of the time, and in a specific order.

Maybe I’d like them better if they were called Enterprise Architecture Grab Bags.

I see the general advice from the agile architecture group as far more reasonable. An EA framework in my opinion is at best a reference tool, where the principles of agile architecture are better daily guides. Many match with my own common sense approach, and approaches of successful architects.

1. Value People
2. Communicate!
3. Less is More
4. Embrace Change: Plan It, Manage It!
5. Choose the Right Solution for the Enterprise
6. Deliver Quality
7. Model and Document in an Agile Fashion

My daily conundrum is the balance of the first three principles. On one hand I’m constantly pressured to include everyone and to communicate to all of them. On the other hand I believe less is more applies to meetings, not just models. You want to give everyone the opportunity to contribute, but the process of evolving solutions happens more effectively when a core group can follow a specific path without interruption or diversion.

That is a tough balance, but possibly the most important one because if you can find it, and you have the right people, you’ll be able to plan for change better, choose the appropriate solutions and deliver quality.

The right people is a key point. It’s not just that they need to be smart either. They need the right attitude. Although there may be a love-hate relationship between agile architects and agile developers, I think the pair is ideal, at least better than EA architects and waterfall developers.

There is significantly higher conflict between agile architects and waterfall developers. The architect will be pushing for change, pushing for open communication, and pushing for participation. True waterfall developers won’t participate, they’ll wait. That is what they are accustomed to being forced to do, so why not? Developers a little less than true waterfall will participate some, but they’ll doggedly fight against change. They’re still thinking in the time boxed fashion of waterfall development, so the future value won’t align with their value system.

I’m unsure of the interaction of EA architects and agile developers. I think there are a few possibilities. In one case the EA architects employ a full governance process. I can’t even fathom how a developer who complies with such a process could still be considered agile. The only way I see it possible to practice agilism under such a regime is to rebel, which is likely to end in early termination.

In a slightly less restrictive version of EA I expect developers would push on with early development. At some point the architects would get involved and either kill the development, re-task it as a prototype, or let it slide. If they let it slide it won’t comply with any of the EA principles, goals or designs. It’s not a totally dysfunctional system, but either most of the work of the EA architects is wasted, or a great deal of developers work is wasted.

Systems with more open communication would have developers participate in the creation of the architecture, and architects contribute to the direction of development.

Getting into the "Zone"

For a long time all programmers preferred an isolated, sound insulated office.  It may not have been quiet but if it wasn’t it was full of music.  More recently some programmers prefer group work environments.

Different people have different preferences, but as different as these two practices appear on the surface, I think they’re a lot alike in ways that keep them totally separate from cubicle land.

Both environments offer predictability. 

• Total quiet and solitude, or a consistent group of nearly always available companions.  In cube farms half the people are at meetings and there are so many coming and going.
• In cube farms people sometimes invite people to their desks to talk about things you don’t care about or sometimes they’re on the phone.   In a group environment conversation is either social and inclusive, or relevant.  In a solo office it’s the same.  People only butt in when it’s relevant.

Predictability allows you to get to and stay in the zone.  A solo office is best for the straight to the keyboard thought for sure, but a group environment doesn’t do too bad.  Cubes are the worst because it’s one little thing after another, all unrelated.

Group environment excel at the bounce.  Work, think, talk, work, think, talk.  If you have a question answered, or need to answer one the process goes quick, there’s no thinking about a meeting, or stopping for a meeting, or wondering when your next meeting is.  Get the question done and back to work.  That’s a bit harder with solo offices, but at least there’s no conference rooms to worry about.  You can go by and (if they exist) peek through the window to see if someone’s busy.

In cubes, stopping by is higher cost to you, to the person you’re approaching, and to everyone in the surrounding area.  Worst of all, it’s unpredictable.  And thus comes the meeting to false rescue.  The lead life saver.  Meetings are supposed to be predictable, but their the exact opposite.  With a bunch of people, or even a few, coming from diverse locations their start times are bound to be vary by 5 minutes.  If you want them to start on time everyone needs to try to get there 5 minutes early.

Once there, you might be lucky enough to have functioning equipment, all the necessary materials, and someone who still remembers why you’re meeting in a clear enough form to start the discussion.  Worst of all, the only generally reliable thing, the end time, is no good.  It’s no good because it eliminates the predictability of leaving a conversation with answers.  But since someone has booked the conference room right after you …

P.S. This post was inspired by reading Geek Management @ killthemeeting

On Simplicity

In many camps, especially anti-Microsoft camps, the clarion call of simplicity has become highly popular. It’s hard to argue with the natural value of simplicity when confronted with the enormous complexity developers encounter daily. However, I believe the movement has simply gone too far, and has lost sight of the true principles. Actually, I’m not sure they were ever well known in the first place.

The Simple Hedgehog

In the business world, there is a book “Good to Great”. The book outlines a number of practices, but one, the “Hedgehog Concept” applies well to the place of simplicity in software. A Hedgehog Concept, is a single simple and unifying concept that fulfills three qualities. The first two qualities, are that the concept must be based upon your key economic driver and be something your passionate about. While both of these are valuable principles for choosing a software concept, the third quality, that you must be able to be the best at this thing, applies more broadly.

The success of the Hedgehog Concept is heavily reliant upon being the best at the one simple thing that defines your Hedgehog Concept. With software, it’s much the same. Features are valuable, but without the best core in the world, your software will suffer. Your software will never be great, no matter how many features you add.

It is however possible to add features without neglecting your core once your product reaches a certain degree of maturity. In fact, simplicity will often run counter to your efforts to make the best core possible. Once your core is implemented in the best way possible, the only way to improve upon it is to add additional layers. Admittedly, few things ever reach the point of “best way possible”, but certainly many reach the point of “best so far”.

Simple products, like Google Docs can only excel by being better at the core functionality, not just by being simpler. In general, I don’t think these products succeed at this goal. Is there really anything in Google Docs that makes it better than Word? I don’t think so. At the very immediate core, a word processor is about helping you write. Google Docs captures keystrokes almost as good as Word, unless you have an internet connection problem. Google Docs provides a spell checker, which you have to manually operate (In fact it even prevents the FireFox 2.0 spellchecker from working properly). Is there a grammar checker? Nope.

With these failings, Google Docs would need to do something truly extraordinary, but it doesn’t. Simple just isn’t good enough. It has to be the best core. If that requires additional features that add complexity, then those features are absolutely more valuable than simplicity gained from their omission.

The Upper Limit

Why then, is simplicity so highly prized in software architecture and user interfaces? Finding the simplest design possible to accomplish a given task is certainly valuable, but I think extreme minimalist designs are overemphasized. I contend that the biggest benefit of the simplest possible design is that it gives you room to add additional features before you pass the “complexity” limit.

The human brain has an upper limit upon the amount of complexity that it can handle simultaneously. Once beyond this point, things start to slow down considerably. So without a very strong core UI concept you’ll eventually reach a limit where the value of additional features is mitigated by their difficulty or obscurity. A better UI concept can enable absorption and effective use of a much larger quantity of useful features, and allow each of those features to reach their true potential. This is the beauty of the changes to Office 2007. So many primary features have been made orders of magnitude simpler, that the value of the secondary features is multiplied greatly.

It hasn’t helped me a great deal in Word, but in PowerPoint the difference is quite striking in the depth of my usage patterns between the 2003 and 2007 versions.

In software architectures, the same principle applies, but in an even more absolute manner. A complex architecture isn’t a problem until it breaks or you attempt to change it. If the additional complexity exists to reduce breakages, as is often the case, then the only disadvantage left is future modification. An architecture with an unsuitably complex design will not be able to evolve effectively past a certain point until the core complexity is reduced.

In other words, it is only possible to support additional essential complexity after you have removed accidental complexity from your core. Adding additional features before addressing core design issues will have diminishing returns. It’s even possible to reach a point of negative returns. But in the end, at least in software design, the goal is not fewer features, but additional space for more features.

Mike Champion calls this process engineering.

The Long Fight

The amazing thing about all this is the realization that software design is attempting to simultaneously add and remove complexity at the same time. The result is a precipitous balance, which goes a long way to explaining the volatile nature of software schedules. A religious adherence to simplicity may produce some interesting ideas, but unfortunately it will produce relatively few great applications. The true path seeks to remove the unnecessary or accidental complexity, while simultaneously adding essential features and complexity.

How do you manage this process? Cutting 80% of your features isn’t how. Neglecting your architecture for the sake of feature releases doesn’t work either. You do it with a cool head, patience, and unwavering focus to not fall off the thin edge. Sound easy? Sure…

Good Ideas Taken Badly

It appears other people had a similar reaction to Steve Yegge’s Agile rant. The funny thing is that as wrong as I think the basic argument is, it has a good core of thought. Somehow it went astray. It’s a bit scary how thin that line really is. You know, how a set of good ideas and good principles can appear to be justifying a bad argument. That, ironically enough, is exactly what Yegge attacks Agile for, when he points out how the short cycles can be used to make a project more date focused, rather than less as Agile is intended to.

It is a risk, when you talk about short release cycles, that managers will treat them the same as the old release cycles, but expect them more often. But when properly used, an iteration is more of a goal than a make or break moment. Sure, you put out a build, but there are no guarantees any specific feature will be part of it. Whatever is done is what’s done. And you use what you learned about your velocity to predict the future. But predictions should remain just that, predictions. Not commitments. And they aren’t that way just because Agile says so. No, it’s because the incremental releases relieve pressure and present clear alternatives.

High Tech Agile

But the possibility of misuse isn’t his only criticism of Agile. He also attacks many of the “low-tech” solutions, such as pair programming, whole team, and story cards. These arguments actually have a ring of truth to them, because it does feel like there should be a better way. A way that doesn’t require all of the trade-offs, or at least not as many as these introduce. But that feeling is irrelevant, at least in an immediate sense because these low-tech solutions are better than the pre-existing alternatives.

Yegge does offer up a high-tech alternative to story cards though, and as I’ve written it’s very interesting. The only problem is it’s inaccessible to most of us for the moment. I’m sure that will change, but in an immediate sense, the best options available are the low-tech variations. Thought into how to evolve these practices further through technology is certainly justified.

For example, I’m starting to believe the concept of an office will go the way of the dodo, at least for programmers. Despite its sensibility, sitting together has always bothered me because of the lack of privacy. Sure, you can provide private spaces, but then you have to switch spots, and deal with all the associated hassles. Plus, what if the team next door is just too loud? Or what about people with bad hygiene or simply annoying habits. That kind of environment is extremely fragile, and unless someone is intentionally obnoxious or inconsiderate, rather than just dealt an unfair deck (smelly, sneezy, etc.), it’s hard to justify their removal from a moral or legal sense.

My solution is to use technology to create a virtual shared space. The beauty of this is that you can turn it on or off, and selectively include or exclude parts of the team. It’s dynamic, and possibly best of all tackles one of the largest hurdles, the remote worker issue. Unfortunately the tech just isn’t quite here yet. It’s coming, no doubt, and pieces and parts are here today in various forms of quality and incompleteness, but it’s not yet complete. And considering how difficult it is to argue the case for a second monitor for every developer, it’s probable that when it is here, it will be difficult to argue the case for the proper equipment.

Pair programming is another piece that would benefit from this tech, and the greater dynamism. But pairing could also benefit from several other techniques that are doable today. Another blogger writes an interesting way of doing parallel pair programming (read the end). I’ve often thought about just plopping down a few extra monitors on a developer’s desk dedicated to watching what his pair is doing, thus allowing them to collaborate without one of them being relegated to the stone ages of paper and pencil (or more commonly marker and plastic).

Back to the Point

Agile is definitely improvable, definitely full of flaws, but rejecting it in favor of the old way isn’t right. I don’t think this is what Yegge meant, but it’s what more than a handful unfortunately read. It’s not an uncommon problem either. We often make the mistake of attacking the solution to one problem, because it’s not quite perfect, while ignoring the real problem. Why are people more concerned with rewriting Java programs when they have a heap of COBOL code lying around? Both have merit, but one is a far bigger problem. I’m sure part of the issue is that rewriting the Java code is less difficult, but COBOL code isn’t going to update itself, and the number of people willing to even look at it is shrinking faster than the amount of deployed code.

Unfortunately there’s no general solution to this problem, because you can’t always say one end is right, and the other end is wrong. A few issues gradate evenly from black to white, but many others are a mixed bag. The best examples here I can think of are the static/dynamic debate, and the browser/client debate. Personally, I believe that static is the “right” way. But there are a lot of great things about dynamic languages like Ruby, and if I took a black and white view, I’d ignore that. Learning and using Ruby has opened me up to these advantages. But for every advantage so far I see a counter.

For example, unit testing is beautifully done because of the flexibility afforded by the dynamic nature. But on the other hand, I find some of the unit tests I’m writing implement a hand-coded custom version of static analysis. Not typing variable types can speed things up a bit, but I find I lose much or all of that to extra syntax debugging due to a misspelled variable name or such, and the problem that I may need several test cycles to find consecutive syntax errors, rather than one compile cycle (or a squiggly).

For this last part, syntax debugging, better Ruby IDE’s can certainly help, but there isn’t any reason a static language can’t use type inference to remove 95% of the extra typing. Likewise for the first part, unit testing, there’s no reason static languages couldn’t make a special exception to type checking, encapsulation, etc. in unit tests. The JVM and CLR have all the necessary pieces and parts to support this. The only downside might be speed, but how fast your unit tests run is not your number one performance concern. (And a static language faking dynamism through reflection is still likely about the same speed as the natively dynamic language.

Clearly this is an area where both sides have a lot to learn from each other, and the best solution lies somewhere in between, not at one end or the other. My belief is that it will be easier to add dynamic features to static languages, than the other way around, but other than that, I see value in both arguments. My real point here is not whether static or dynamic is better, it is that a dynamic or a static language advocate both have a set of good arguments that can easily be used to justify the wrong conclusion. That conclusion is that everything the other side believes is wrong because static and dynamic are opposites and in their minds, wholly incompatible. It’s not true, but it’s a case where the truth is easily and often ignored.

Browsers and clients are another great example of this same problem. Since I’m not going to take the time to explain my belief of which is ultimately the “right” way, I’ll simply direct you to a similar viewpoint. Curiously another entry from Yegge.

Agile is a path (to trust (among other things))

I think I figured out why Steve Yegge was so inclined to bash Agile yesterday. It’s actually rather obvious when you think about it. I don’t think it’s some recruiting scam, like some do. No, there’s a much simpler explanation. Steve likes Google’s practices, and is trying to warn other Google employees away from Agile. It has nothing to do with the rest of the world, unless of course you’re lucky enough to have practices like Google, which work.

He went astray by thinking what’s bad for a group of Google newbies is bad for everyone. That’s an easy mistake to make since he probably spent at least 20 times more time with those young Google employees, and other Google employees then with other programmers, at least recently. I don’t really know anything more about the Google process than what Steve told us, but it sounds like it has promise. More importantly it’s working. It might be better than Agile. I say might because I don’t really know. To be honest, I’ve never had the opportunity to participate in an “Agile” project, but I’ve employed bits and pieces in different places over the years. And from those experiences one thing has been clear: anything is better than waterfall.

I can understand how, given his pre-existing skepticism, and observing a few Googlbies having problems with Agile, he might lash out. But Steve is wrong. Those Googlbies might actually be ditching a great process for a merely good process, but for the majority of us out here, the question is completely different. For us, for right now, Agile is the way to go. It’s not easy to convince management to try it, but I think it might be easier than asking to become Google. For example, if you wanted to be more like Google, the first thing you would do is fire 90% of management. That might not be such a bad thing because most bad agile is the result of bad managers who can’t adapt to the principles of agile, who keep trying to force old principles and values into the system. Principles like overwork, whipping and unrealistic date-driven development. But managers have most of the power at most companies, and it seems unlikely they’ll sign on for that plan.

The next items on the checklist are basically developer freedoms, which are hard to do if you skipped the first step and still have bad managers hovering around. Maybe you could get away with stripping those managers of their authority, so that hovering isn’t so destructive. But, I have a feeling they won’t sign up for that plan either. I mean the majority of the bad ones became managers because they crave authority, so they aren’t going to give it up easy.

Like many people have mentioned, these are people problems. What Agile is good at doing is creating an environment where management actually starts to understand what developers do. After that they’ll start to delegate more authority to the developers. Maybe one day everyone will end up like Google, but that’s not an easy change, and before anything like that happens management will have to first learn to trust development. Google probably works because management IS development. And I don’t just mean the guys at the top; I mean almost all of it.

However, the point behind all of it is trust. And trust is a tricky thing. A lot of people forget trust has two parts. The one most people focus upon, is a belief in honesty or integrity. But there is another part, a belief in capability. This division is the source of many arguments. People take it very personally if you say you don’t trust them. Much more seriously than if you simply said you didn’t think they were capable. It’s not very flattering to be told you’re not capable, but it’s a lot less insulting then to be told you’re not trusted, even though usually they mean the same thing.

In the workplace, you’ll hear people say they trust so and so, but only in the most confidence will you ever hear someone mention who they don’t trust, no matter which type of trust is involved. This is unfortunate, because when it’s a lack of trust in capability, people should know so they can respond rationally, either by proving their worthy of trust, or by improving. But trying to do this is like walking through a minefield. You never know when someone is going to blow up. Even if there is a pot of gold on the other side, is it really worth the risk?

Another part of the problem is that management often does not trust themselves, though they won’t easily admit this. They don’t trust themselves to spot the bad apples, or the incompetent. To deal with bad apples, they resort to paranoia. In their defense, part of the problem is that they are hindered by legalities put in place to protect minorities, disabled and other protected groups, they have a secondary effect of making it much more difficult to fire (or at least not promote) the incompetent. That’s a tough issue because at least for now, those groups do need some protection, and no one has found a better way of insuring it. Maybe sometime in the distant future we’ll fix that problem, but for now it’s a bit harder than any of our development quibbles.

To bring this back on track, Agile development is intended to encourage communication that promotes trust. It’s intended not only to help the developers understand the customer’s needs better, but also to help the customer understand the developer’s role better. Waterfall on the other hand attempts to squeeze every ounce of risk out of any decision. Unfortunately it makes the mistake of forgetting that by avoiding a decision, you have made a decision; a decision to stall.

And stall is what happens to project after project under a waterfall development methodology. By the time a decision is made, there may be no chance of misstep, but you’ve taken a considerable penalty for that piece of mind. Even worse you often find that things change so rapidly, that this point is unreachable. Organizations try to make up for lost time by cramming the follow up to the decision in unreasonable ways, and well.. we know what happens then, don’t we?

This has been quite a ramble, even for me, but overall what did I say? We need trust, Agile is a path from the here and now to trust, once we have mutual trust we can tackle the next stage, and according to Steve, Google already has done all of this. Wow, can that really be true?

Try your best Agile, and why we can’t all be Google (yet).

Steve Yegge is definitely one of my favorite writers, because he makes some really good points, and in interesting ways. But I have to admit it’s usually full of a lot of blind spots. If he were asking me (he’s not) whether he should mellow out, I’d say no because in despite the blind spots, his rants are probably the most effective route from point a to b, for him. I think that’s because he’s not a zealot, and is willing to flip-flop his opinions several times while trying to find the right solution. Either way, when you see of these blind spots, you gotta yell “car!”

So why am I writing all this crap about someone else’s writing? I mean, in all likeliness I’m probably just as bad or worse in some other way, or maybe the same way. Well today, it’s because of his most recent post, Good Agile, Bad Agile. On the good side, he reminds anyone who forgot that there is a bad version of almost everything. He also points out many of the more risky Agile practices, like abuse of iterations as a recurring death march tool, or as an excuse to leave refactoring for time boxed iterations.

XP actually cover both worries, with the principles “Sustainable Pace“, and “Design Improvement”. Of course, those can be ignored, and according to Steve often are. He goes on to compare Agile to Google’s process, which certainly sounds very interesting. But Google’s not your average company, and just like Agile itself, there is no guarantee that if someone tried to export those practices to another company, that company wouldn’t screw it all up. There are several parts of those practices, at least today, are impossible, highly difficult or outrageously risky for any organization not in Google’s class.

Take the work queue he wraps up the description of good Agile with. Sounds cool, but I don’t see it on the market, and until it is, I don’t see any company without lots of “hand-wavy” math types putting a similar internal app into service. And I think that piece is critical to the whole system to provide some balance to the chaos factor of the other freedoms.

It sounds a lot better than index cards, but index cards in many ways area better than the project management software on the market today. I’ve seen a number of these and amazingly, they haven’t even tackled what should have been the first task: making it easy to get data into the system. It takes like a whole hour to enter in a few tasks for the week, and even then, it’s an inaccurate jumbled mess.

Beyond that visibility is generally very low, so that only the managers have a clue what’s going on. Or at least they think they do, because they don’t realize how bad the data is. Beyond being wrong, it leaves out huge gaping holes; and since non-managers never see the data, they never have cause to point out the deficiencies. And so managers make decisions as if that non-measured data didn’t even exist.

The index cards ideas are just a simple way to try to resolve some of these problems, like visibility and ease of data collection. If you resolved them in software then you’d gain the benefit of being able to use all that hand-wavy math, but today the rest of us get the worst of both worlds: no math, bad visibility, and painful data entry.

So, I’ve made a case for “all the bad stuff he described is the fault of the team’s execution of the process”, so I might as well totally ignore Steve’s caution and move on to “all the good stuff he described is really Agile”. To be truthful, not all of it is, but I have made the case that at least some of it would be very hard for any non Google class company to do alone. And a lot of the rest just might not work without those other pieces, or without the natural enthusiasm Google has from being able to be very selective, very rich, and very famous. It might work, but it’s hardly proven.

A lot of it is Agile, however. And Agile is potentially good. And, yes, if 90% it screws up it’s bad.. unless of course your comparing it to a process that screws up 95% of the time. In that case, it’s the most amazing thing since sliced bread. It does however mean that you need to pay attention to what you’re doing, don’t take things for granted, and always try to improve your process. And as far as bad Agile goes, well I’d say even the most questionable practices, like pair programming have value in moderation.

Even if taken to obsessive extremes they’re better than the other extreme. It might seem wasteful to tell half your staff “thou shalt not code”, but it’s a lot better than telling half your staff, “thou shalt analyze”, and the other half “thou shalt copy badly written documentation into badly written code, with no understanding of the real problem.”

So, for most organizations, it’s not a choice of good Agile vs. bad Agile, but bad waterfall vs. try your best Agile.

I can identify with Steve’s desire to separate out some of the technology aspects, like unit testing, continuous integration, refactoring, out from the process part of Agile. In many cases they are a lot more difficult to screw up so it’s nice to show that distinction, but a lot of the process practices are intended to help adoption of those practices, and to keep them in place, so you can’t really get the Agile people to drop em. At best, you could come up with a new group, like “The Workable Code Movement™”, that’s a subset of practices if that’s what you want.

Update: I wrote two more follow up posts on the same topic. Over the top? You decide.
See “Agile is a path (to trust (among other things))“, and the more liked “Good Ideas taken Badly“

The "What isn’t easily measurable, doesn’t exist" Rule

In one of my earlier posts, What Is a Software Architect, I touched upon the effect of governance in software organizations. Specifically, IT governance promises more than it delivers.

In principle, it sounds great. The reality is today few if any organizations can accurately track enough information to deliver these promises. Governance requires a great deal of information, and taking action without the necessary information is likely to have negative effects, rather than positive. Every decision made at a higher level negates the decision making power of those closer to the reality of the situation. Those closer to ground usually have masses of data that are not trivially quantifiable or reducible.

Gathering information is still valuable, but it requires a flexible process to make use of it. The data is not accurate enough that it can replace hands on techniques. At best, the data can be useful to augment hands on techniques, but this requires the data to be available to those individuals with the best position for decision-making. Contrary to popular belief, these individuals are not always those highest in an organizational structure. Except for some extraordinary decisions, those in the best position are actually those lowest in the organizational structure.

One example of misapplication is when IT governance drives performance reviews. While unbiased, measurable performance reviews are attractive, attempting to perform one using incomplete or irrelevant data is a mistake. For example, IT governance often uses metrics like process compliance as a key measurement, because they are easy to measure. However, since the relationship between process compliance and performance is very weak, performance reviews will often reward poor performance, and fail to reward stellar performance.

At organizations treating process compliance as a metric of performance, the opposite is likely true. Processes always have flaws, and individuals with a drive for performance will use processes when they help achieve results, and avoid them when they harm results. Individuals strictly interested in a favorable performance review, will strictly apply processes with a degree of paranoia that harms the performance of the entire organization.

The desire to drive up process compliance, and thus increase information accuracy is a natural instinct among those involved in governance, because these are the metrics upon which they are judged. It is easy for them to ignore the side effects, such as wasted bureaucratic efforts, delays to real progress, and other general dysfunctions. The side effects are outside of their purview, and highly difficult to measure. These two factors lead to application of the “What isn’t easily measurable, doesn’t exist” rule.

This simple mistake is responsible for a huge quantity of the bad business decisions. Many trends, such as poor computing resources, poor work environments, adversarial management-employee relationships. Quite possibly, it is second only to general short sightedness as the most common source of business mistakes. Quite often the two are related as the more distant a future event becomes, the less measurable it becomes.

So, what can you do about this? You could try to use the arguments of this essay with management, in order to gain greater control, but that is likely to be a difficult battle, because that requires managers to place more trust in their employees. Trust is not an easily won quality. It is especially hard when you figure that trust is not based solely upon your qualities, but is heavily based upon managers’ trust in themselves to pick the good and bad apples.

Another tactic is to try your best to quantify the unquantifiable. You’ll likely have some very imprecise results. But even those extremely rough values are more accurate than the “zero”, assigned to them today. It’s important when you do this that you make the imprecision clear. Unfortunately, even when you take all reasonable measures to communicate this there are likely to be misinterpretations, and this more than anything is the reason why most people are afraid to attempt to quantify these values.

Neither path is without risk, and it’s quite possible that successfully following either will gain little or no recognition, so it’s no wonder that it’s uncommon to find champions who try to address these problems. More often than not, they progressively decline until the point where only natural selection can fix them. But since the issues are endemic to the average corporation, natural selection can take a very long time.