Wednesday, December 1, 2010

Scrum Alliance vs. Scrum.org vs. Scrum.

I just finished reading the following article: http://borisgloger.com/en/2010/11/29/scrum-alliance-and-scrum-org-yesterdays-concepts-do-not-solve-todays-problems/ I was going to comment there, and then decided it was really worth its own blog post.

Full disclosure: I am a CSM and a CSPO, and (while not a CST) have co-taught CSM courses in the past. I firmly believe that good education is essential to getting started well with Scrum, and that having some way to identify baseline knowledge and expertise in a field is a boon both to knowledge workers and to potential employers.

I believe that we need a crowd-sourced, community-based Scrum education/certification organization.

Some meaningful alternative to the Scrum Alliance's strategies and methods has been needed for a long time. I had hopes when I first read about the split between the Scrum Alliance and Scrum.org that Scrum.org could be that alternative, but this is plainly not the case.

We have seen, in venues ranging from YouTube to Wikipedia to OkCupid, the power of crowd-sourcing knowledge and creativity. Let's brainstorm on what a crowd-sourced Scrum organization might look like:

1) It needs to have *no* financial interest in owning its ideas, teaching its ideas, etc. The profit motive has arguably distorted the judgment of the professionals at the Scrum Alliance, and seems highly likely to distort the judgment of the professionals at Scrum.org in the same ways.

2) It needs to grow, adapt, and change over time. It needs to take feedback from its users, alter with the increasing knowledge of the community, and always be focused on inspecting and adapting to improve itself.

3) It needs to be small, lightweight, and agile (not "Agile"). We don't need a huge organization with pages of bylaws and a board of directors, we need a website, a few programmers, and a bunch of Scrum practitioners sharing their knowledge to improve the way everyone builds software.

There are already wikis out there that capture and share crowd-sourced knowledge about different aspects of Agile (e.g. http://agileretrospectivewiki.org/ ). Without re-inventing the wheel, how could such a crowd-sourced Scrum organization draw from the community as a mass of creative, experienced individuals, to give back to the community in the form of useful, free educational resources, or a meaningful certification process?

I'm much enamored of the OkCupid model, which combines mathematical analysis with user-provided data-gathering mechanisms. Something as simple as a correlational model which accepts questions from the community, correlates individuals' answers to them with those same individuals' Scrum experience (number of years doing Agile, number of years doing Scrum, number of Scrum projects, number of successful Scrum projects), and then allows community members to take a selection of the questions and receive a "Scrum Experience Score", might be all that is needed.

Thoughts?

Saturday, October 9, 2010

The Brick-Layer and the Doctor

There is a long-standing belief among the planners and architects of the software world that coding is like the work of a brick-layer. Misled, perhaps, by the fact that the elements of computer languages are few and identical, they model the writing of software as a simple, linear, step-by-step process where the same fundamental operations are repeated over and over again. Slap down some mortar, lay a brick, true up the edges, lather, rinse, repeat.

They could not be more wrong.

Why do I say that with such confidence?

Because anything which is actually simple, linear, and step-by-step gets automated. Not all at once, admittedly--it took us more than a decade to get from the first general-purpose electronic computer (ENIAC, 1946) to the first object-oriented programming language (Sketchpad, 1960).

Stop and think about that. From plugging and unplugging a large web of electrical cables to writing object-oriented code...in just 14 years.

It is not possible for competent programmers to be brick-layers. Competent programmers stop after laying down a couple of rows of bricks, say "A machine could do this!" and proceed to write one that does.

What, then, is the role of a programmer?

The world presents us with a complex, ill-defined, fuzzy set of problems. Most of them overlap. Some of them have a common cause. Some of them look very much like other problems with completely different causes. Millions of dollars -- even lives -- hang in the balance.

Programmers are doctors. Businesspeople present us with a problem: "I want a billing system." "This old system is too slow and it keeps crashing when we try to run payroll." "Why can't it just work the way I want it to?!?" Our work starts with diagnosis, and continues through selecting an appropriate treatment, tracking our patient's progress to make sure they comply with it, and following up with additional treatments if the first one doesn't work.

Over the next couple of weeks, I'll be writing a series of articles based in this metaphor. My goal is to help programmers and managers alike break free from the misconception that programming is unskilled, repetitive labor, that programmers are interchangeable "resources", and that the best way to write good code is to follow strict recipes with rigor and precision.

The world needs more good code. I want to help you write it.

Next...Diagnosing Sick Code.

Wednesday, August 11, 2010

Another excellent quote from the same source...

I propose that the real issue is that design is not really a beneficial activity in software development, and to say "The Source Code Is The Design" is trying to use semantics to gloss over the issue.

I feel this is an important distinction if the goal is to remove the "design" stage from the software development process. Rather than being afraid of being accused of "not doing design", we need to turn the debate around to be "Why should we do design?"

For some tasks, it may be much more cost effective to create a design and evaluate the design before building the actual product. For software, this is not the case. For years, software has struggled to come up with something to use for "design." We had flow charts, PDL, Data Flow Diagrams, prose descriptions, and now UML. With software, however, it takes as much time to create the design as the actual software; the design is more difficult to validate than the actual software; and the simplifying assumptions made in the design are often the critical issues to evaluate in the software. For these reasons, it simply is not cost effective to design, iteratively correct the design, then write the software (and then iteratively correct the software). It is better to start with the software and iteratively correct it.

I believe it is time to explicitly state the long held secret of software, we do not need to do design; design is ineffective and costly. -- WayneMack

From http://c2.com/cgi/wiki?TheSourceCodeIsTheDesign

Best quote ever on software specifications.

Walden Mathews, talking about specifications documents and functional specs and the like, says:

"Specification" is interesting nomenclature, because it's really the lack of specific-ness that keeps it from being the product.

From: http://c2.com/cgi/wiki?TheSourceCodeIsTheProduct, which I got to from http://c2.com/cgi/wiki?TheSourceCodeIsTheDesign.

He's absolutely right. The source code is the first thing specific enough to actually constitute the system—and in a well-written object-oriented system in a high-level language, it’s often just as readable as (if not more readable than) the original “specifications” document.

Monday, May 24, 2010

Acceptance Testing and the Testing Pyramid

For the past couple of months, I've been working with a client who is seeking to get the best value they can out of their testing automation efforts. One of the big opportunities I've seen for them to increase the value of their tests is to adhere to the "Testing Pyramid" -- the idea that you should have lots of unit tests, fewer integration tests, even fewer functional tests, and very few acceptance tests at the top of the pyramid.

But in the process of working this out with them, I've been noticing that how people define acceptance tests varies widely. And then I came across this interesting review of an article by Jim Shore on whether to automate acceptance tests at all:

http://www.infoq.com/news/2010/04/dont-automate-acceptance-tests

(the original article is here: http://jamesshore.com/Blog/The-Problems-With-Acceptance-Testing.html )

Having read the original article, I don't think the reviewer quite fairly represents Shore's position. Shore is not so much arguing against automating acceptance tests, as arguing that automating acceptance tests, by itself, doesn't buy the business value that he once thought it did.

Me? I think that the key value lies in defining "acceptance tests" as literally that--the automated reification of the Product Owner's Acceptance Criteria.

As any Agile developer knows, the five-to-seven Acceptance Criteria on a particular story represent only a fraction of the expected functionality--but they represent, ideally at least, the core business value and core expectation of the PO. By automating those expectations in a "visible execution" testing tool like Selenium, we gain the dual benefits of both securing those core expectations against regressions, and creating a built-in "known working demo" for the customer of the functionality the customer most desires. But what about the rest of the functionality, the part that the delivery team fills in, that isn't explicitly mandated by the acceptance criteria?

That's where you can (and should) drop down a level, into functional testing--headless browsers like HtmlUnit for web applications, and behind-the-UI testing tools like FIT for desktop applications. Less brittle than automated acceptance tests and considerably faster-running, they pay for these features by being opaque to the PO. But we've already written the PO's core expectations in a visible, user-comprehensible form.

The problem comes when people take "acceptance tests" to mean "system tests done through the UI", and then attempt to test their entire application via this type of test. I think that's what Shore is getting at when he says "plus a full set of business-facing TDD tests derived from the example-heavy design". I've worked on projects like this, when the legacy technology we were using (ColdFusion) gave us literally no entry points between the web page and the database. Yes, we were able to build a good application this way--but by the time we were done, the entire test suite took over seven hours to run, and we were only actually executing the "most relevant" roughly 10% of the test suite until the end of each Sprint. That's not really a recipe for good iterative development or TDD.

As long as "acceptance tests" are kept at the level (and scope) of the acceptance criteria, they become a powerful tool both for communicating with the PO and for securing the PO's core business value against regressions. Then, testers and developers are free to use all the fast, efficient, lower-level tests they want to provide test-driven design and a refactoring safety net.

Thursday, September 11, 2008

Practical Refactoring via Code Smells

Refactor, Refactor, Refactor! (um…thanks?)

Just as "location, location, location" is the mantra of real estate, "refactor, refactor, refactor" is the mantra of Agile development. If you're new to Agile development, people will be constantly telling you to refactor your code, to make it "simpler" and “better” and "more elegant".

But those aren't instructions--they're goals. It's all too easy to define "refactor" in terms of its goals, and think you've told people what to do. But all you've told them is the destination, not how to get there.

In this post, I'll be discussing some simple, practical instructions for refactoring: rules that, if you follow them, will reliably make your code simpler, better, and more elegant. These instructions aren't the result of programming theory (although I’ll take a minute at the end to discuss the theory behind why they work).They're rules that I've seen empirically proven, day after day, in real software development situations.

Refactor from Code Smells

They all follow the same basic pattern:
  1. Observe a Code Smell in your code.
  2. Remove the code smell.
It’s really that simple. When you’re refactoring, you shouldn’t be thinking about functionality, or testing, or Big Architecture. You should be looking at the code you have (which already has correct functionality, and automated tests that all run green…right?), and making simple improvements to it, over and over. Code Smells are the best way I’ve found of identifying where there’s a simple improvement to be made, and eliminating Code Smells is the best forcing function for making refactoring actually happen.

There are lots of Code Smells, but only a few of them have the key attributes that make them optimal refactoring drivers: they have simple, direct fixes; and when you fix them, you end up fixing problems rather than symptoms.

The Code Smells I’ve found most useful for driving the process of refactoring are:
  1. Long methods.
  2. Bad names.
  3. Breaking the Law of Demeter.
  4. Comments.
Sniff out and eliminate every instance of these four key Refactoring Smells in your application, and your code will reliably improve in ways that go far beyond the cosmetic changes the Code Smells seem to require.

Keep Your Methods Short

Methods should never be longer than seven or eight lines. I know some developers who can hold 30-line methods in their heads and reason about them (guys, have pity, and keep it to a length we can manage!). For the rest of us, anything longer than about eight lines requires us to think about it in smaller chunks anyway; the method might as well be written in these chunks in the first place, saving everyone who has to edit or maintain it from re-doing that parse.
But there’s a more profound benefit that short methods provide. They require you to really think about the hierarchical decomposition of the task. What are you trying to accomplish here? If you can’t express it simply in terms of smaller tasks, you probably haven’t thought it out enough. Capping your method length at eight lines will naturally highlight for you the places where more analysis will improve your code.

It’s easy to give people instructions like “Just look for separable concepts in your method, and pull them out into a separate method.” But nobody actually does it, because it’s not easy to identify a “separable concept” at a glance. If you follow the simple rule “no methods over eight lines long”, you’ll be forced to restructure your methods along clear, conceptual breakdown lines, and you’ll end up identifying the underlying concepts almost as a side effect.

Be a Name Nazi

If you insist on naming concepts precisely in your code, one of the nicer things people will call you is a “name Nazi”. Don’t let them fool you—there is vast benefit in accurate naming.

Code should read like English. Remember that 80% of software costs are maintenance: your code is going to be read over and over again, hopefully for years or decades to come. It should be as easy to read as possible.

If your code is full of lines like
entity.Leg.Hand.Shake ( friend.Leg.Hand )
you can be pretty sure that you haven’t clearly thought out the underlying concept. And while it may take a little longer the first time to type out
entity.HandshakeAppendage.Shake ( friend.HandshakeAppendage )
the first time, you’ll more than make up that time difference the first time someone isn’t confused by the Leg object having a Hand property.

And, you’ll have improved the object oriented abstractions in your code. Instead of a multi-part if-statement that looks like
if ( entity is Dog && friend is Human ) {
entity.Leg.Hand.Shake ( friend.Arm.Hand );
} else if ( entity is Human && friend is Dog ) {
entity.Arm.Hand.Shake ( friend.Leg.Hand );
} else if ( entity is Human && friend is Human ) {
entity.Arm.Hand.Shake ( friend.Arm.Hand );
} else if ( entity is Dog && friend is Dog ) {
Debug.Fail(“Dogs cannot shake hands with other Dogs.”);
}
you’ll have a simple bit of code that says
if (entity is Dog && friend is Dog ) {
Debug.Fail (“Dogs cannot shake hands with other Dogs.”);
} else {
entity.HandshakeAppendage.Shake ( friend.HandshakeAppendage );
}
By being religious about naming precision (“there must be something wrong with this code, because Leg.Hand doesn’t make any sense!”), you’ve successfully extracted one of the key abstractions of your application: that different species will offer different parts of their bodies for handshakes.

Obey the Law of Demeter

The first step to fixing violations of the Law of Demeter is knowing what it is. Go read about it (this step is optional). Confused yet? Read some debates about it, and a technical definition (this step, also, is optional).

Now forget everything you just read (see? Toldja they were optional).

The Law of Demeter boils down to, “Don’t do someone else’s job.” An object should deal with its own business, and ask other objects to deal with their own business. If you’ve got a method in your class that does nothing but poke at other objects’ internals, pull out that functionality and put it into the class you’re poking at.

Instead of
if ( myNeighbor.Children != null ) {
myNeighbor.Children.Remove ( neighborChildToRemove );
} else {
myNeighbor.Children = new List();
}
write
myNeighbor.RemoveChild ( neighborChildToRemove );
and, in the Neighbor class,
private List children;
private List Children {
get {
if (children == null) {
children = new List();
}
return children;
}
}

public void RemoveChild ( NeighborChild childToRemove ) {
Children.Remove ( childToRemove );
}
It’s now infinitely easier, when you suddenly discover that Neighbor needs to listen to events from all of its children NeighborChild objects, to add that kind of constraint reliably:
public void RemoveChild ( NeighborChild childToRemove ) {
childToRemove.InformParentEvent -= myChildListener;
Children.Remove ( childToRemove );
}
The responsibility for managing the list of NeighborChild children lives entirely where it belongs: in the Neighbor object.

Destroy Comments

“But, but, but,” I hear some of you splutter, “comments are good things, right?”

No.

As Alex Chaffee said, “Every comment is a lie waiting to happen.” Comments are almost never updated when the code they document is changed, and thus are more likely to be actively misleading than helpful and explanatory.

Worse yet, the very existence of a comment indicates that the original developer didn’t invest the resources, or simply wasn’t able, to make the code or algorithm clear.

We want to get rid of comments, to remove the likely divergence between the comment and the code. But it’s not sufficient simply to delete the comment; in fact, it’s often actively harmful. By deleting the comment, you’re removing the invaluable marker that someone failed to write clear code here.

In order to safely remove comments, and thus eliminate the “lie waiting to happen”, you must first eliminate the bad code that inspired the comment in the first place. Often, simply applying the first three rules will result in the clear code that should have been written in the first place, and deleting the comment will be the natural final step in the process of refactoring. Sometimes, however, especially in code written by an expert programmer, the comment reflects an authentic difficulty with making the code clear. Maybe it accesses a poorly documented interface, or is solving a problem with the wrong tool (“pounding a nail with a wrench”). Maybe it’s a quick hack that somehow made it into the production code unrevised. Whatever the original cause of the bad code, take the time now to fix it, and remove the comment.

Fixing this code smell also synergizes nicely with the first fix (keep your methods short). If you have a 20-line method, where the successive sets of five lines are commented, respectively:
// Prepare data structures for Foobarizing
// Foobarize prepared data structures
// Verify Foobarized data
// Write out newly Foobarized data
it’s very simple simultaneously to convert it into a four-line method and to eliminate the four lines of comments.

This rule is also great for getting rid of comments designed to “clarify” massive, multi-line conditionals. Instead of
// Only shake hands if both “entity” and “friend” can shake hands.
if ( ( ( entity is Dog && ((Dog)entity).HasBothFrontLegs() )
|| (entity is Human && ((Human)entity).HasARightHand()) )
( ( friend is Dog && ((Dog) friend).HasBothFrontLegs() )
|| (friend is Human && ((Human) friend).HasARightHand()) ) ) {
entity.HandshakeAppendage.Shake ( friend.HandshakeAppendage );
}
write
if ( entity.CanShakeHands() && friend.CanShakeHands() ) {
entity.HandshakeAppendage.Shake ( friend.HandshakeAppendage );
}
Note how I sneakily snuck in a fix to a Law of Demeter violation, there? Now, instead of having to check the type of the object and then poke around in its internals to figure out for it whether it can shake hands, I just ask it “Can you shake hands?” And it decides for itself, and lets me know.

Be Not Afraid

Refactoring can look complex and scary when you’re just given the general goals. But by choosing Code Smells with simple resolutions and profound consequences, you can drive your refactoring efficiently and with minimal stress. Don’t let the general “make your code more elegant” instructions scare you off; all they’re saying is “make your code smell less.” Your bug count will go down, your customers will be happier, and your code—voila—will be the magical “better”!