Developing good software well

From Months to Minutes

2011-07-22T05:06:00.002-05:00

I found another presentation on InfoQ, this time by Dan North. In it, he puts many Agile practices through the wringer, and turns them inside out and on their head. Dan questions why we do them, what they do for us, and offers pointers on how to intelligently apply them to become a high performing team.

It's a great and very fast-paced discussion, and it got me thinking a lot about what I do and what I recommend to my teams.

My favorite parts, which are also the most important steps to effectively delivering software, is about the need for one team. In his case, he talks of developers working side by side with traders. He talks about how the developers know as much about trading as the traders do, having learned this by spending time talking with the traders while they work making trades. There is no difference between the business and the engineers. It's one team, consisting of both engineers and traders.

He also talks about minimizing boundaries and hand-offs, particularly within the team. The team does not have members who are analysts, QA's, DBA's, etc. Those roles exist, and are filled by the team members as needed. Particularly, the developers are the ones talking to the traders about features. There is no handoff from customer to analyst to designer to developer. The developers talk directly with the traders, and then implement the features. There is no loss of fidelity; the message does not have a chance to be misunderstood. This is helped by the fact that the traders and developers work together.

Craft and Software Engineering

2011-07-21T16:08:00.008-05:00

Glenn Vanderburg has a great presentation available on InfoQ titled Craft and Software Engineering. In it, he describes where Software Engineering came from, some of the fallacies, and a direction to make it more like other Engineering disciplines. It's a great talk and moves quite well through the content. I'd recommend it to anyone involved in software development.

One of the things he talks about is feedback, and how the different XP provide feedback at different levels of granularity, and at different timescales. He also discusses this in his 2005 paper XP Annealed. Here's what this looks like in a table:

XP Practice	Feedback	Timeframe
short releases	solutions	months
planning game	priorities	weeks
acceptance testing	features
collective ownership	architecture	days
on-site customer
continuous integration	design	hours
system metaphor	classes and interfaces	minutes
unit testing	statements and methods	seconds
pair programming

In his paper, he goes on to describe how the nested levels of feedback are effective and efficient because the practices support each other, and cover for each other. Each practice, at each level of granularity and timeframe, covers and is covered by nearby practices.

Observation: The feedback can be broadly classified as technical at the lower levels, and business at the higher levels. Taken broadly, there are only two types of roles needed to implement the project. Technical (developers) to work out the design and testing, and business (non-technical) to provide the vision, the priorities and acceptance criteria.

Taking it one step further, putting these people together, as one collocated team, would be the most effective way to facilitate this communication. Nothing original here, but it seems to be a natural progression. It minimizes the number of hops that need to happen for any communication.

That's not to say there is no room for specialists, for things such as the environment, deployment, acceptance testing, UI design, and so on. However, they should all be part of the same collocated team.

Learning Apache and Linux

2011-05-13T12:01:00.001-05:00

A co-worker asked me if I knew any good books where he could read up on using Apache and Linux. I thought for a second, and quickly answered “Nope, not a one.”

“Oh!”, he said, looking somewhat dejected. Well, neither of us were looking for that response.

Then I had an idea. “Why a book? Just get a slice and just practice. It’ll probably be cheaper than a book, and you’ll learn more.”

After planting that idea, I went looking at what it would really take. I was close.

Get a slice

I checkout out what Slicehost was offering. For $20/month, you can get a linux instance with 256MB of memory, 10GB of disk. Not bad, but that seemed a little pricey to me. I kept looking.

I didn’t have to go far. Slicehost is owned by Rackspace, which offers the same instance for about $10.95/month. You’re only charged for the time your server is running, so you could bring it offline to save some dough.

Not very original

The idea of learning as you go on your own virtual instance wasn’t very original — Slicehost was promoting the same idea right on their faq:

I’m kind of new to this, will it be too hard?

Maybe. Our Slices are full distros with root access. Many people coming from shared hosting are used to a cPanel type control system and limited shell access. We’re geared for people who want to start from scratch and tune a system for their purposes. However, everyone has to start somewhere. A 256slice is a great place to cut your teeth on building production boxes from scratch. Tweak, explore, learn. If you screw up, login to SliceManager and restart the box. If you really screw up, login and reinstall ;). You’ll be a Linux guru in no time.

Now what?

So you got a slice. How do you start? This is where the options (and opportunities) are endless. The key point would be to start playing and see how far you can get. And maybe O’Reilly’s Lamp could help as an initial resource.

Some ideas:

Install Apache and get a static website deployed.
Install PHP, Perl or Python and get a dynamic website going.
Install a database, maybe Postgres or MySQL. Develop an deploy a small application.
Install Ruby and deploy a Sinatra or Rails application.
For the advanced — install and use another datastore, like redis or mongoDB.
For extra credit — deploy several applications together on the same host.

Thin, valuable features

2010-12-02T23:47:00.010-06:00

You're on a team developing an e-commerce site for a traditional retailer. The product owner has a great idea about the Store Locator feature. As you start to discuss this story, you start to wonder how many ITERATIONS it will take you to develop the functionality.

Users can search for a store by zip code or city and state. The five closest stores will be displayed with their address, phone number and store hours. A map will be displayed showing the five closest stores, with the ability to zoom in and out. Finally, the search results will also show the regional managers name, phone number, email and picture. Oh, and if the user is logged in, lets pre-fill their address to search.

Too much at once

After thinking about this "story" or "feature", you decide that it's too much work for one pair to complete in a week, and might even be too much to complete in two weeks. What can you do? You think about breaking it up so that you complete the domain models and services in the first week, and then the user interface during the second week. This could work, but there are a few problems with this approach.

First, will the models that you develop during the first week be sufficient to complete the user interface without changes? Will they have enough functionality, and will the functionality be correct?

Second, what will tell your product owner when he asks how the story is going? Will you give say that you're 40% done? If you are 40% done, when will you be all done? More importantly, will you be able to show any functionality along the way?

Finally, if someone offers to help with this story, will you be able to hand-off any tasks? Can any of the work be done in parallel?

Breaking up your story in this way, focusing on models and services first, and then on user interface, leads to alot of difficult questions. The answers, unfortunately are either "no" or "I don't know". So, what can you do to improve the situation?

Strands of functionality

What if you take this "story", and instead re-word it into several end-to-end features? Each piece of functionality is much smaller. Once complete, each feature can be demonstrated to the product owner. Once some critical features are complete, other features can be developed in parallel. Finally, once the product owner starts to see this story built out he may have other ideas, or want to change direction altogether. So, what do we need to do to give our team all of these options?

First, lets break this story down into features. The difference is subtle, and development teams may use these words interchangeably. I like to think about a feature as a single, end to end, piece of functionality, that adds value to the user. So how can we break the Store Locator story down into some constituent features? Here's one way:

User searches by zip code, and sees a list of stores with address and hours.
User searches by address.
Results are displayed with a static map.
Map can zoom in, zoom out, and pan.
Display the regional managers name, phone and email.
Display the regional managers picture.
Pre-fill the search when the user is logged in.

Good for everyone

When we think about this story by features this way, alot of opportunities open up. Each feature is small and focus, and therefore easier to develop. After the initial search with results is developed, features can be developed in parallel. The product manager will be able to see real progress much easier. The individual features can be prioritized amongst each other, as well as among other stories, so that only the most valuable features are developed.

Though I initially looked at breaking up the stories from the product owners perspective, there are also benefits for the developer. The biggest benefit is that development can focus on just one slice of functionality at time. This is much easier to think about than the original, full story. You can focus on search without worrying about maps or regional managers. Often when working on a large sotry, it's hard to know where to start. Breaking it down into thin features makes it much easier to focus.

Working on thin slices is also alot of fun to develop. The combination of quick wins and rapid feedback energizes developers. The laser focus of the features keeps their mental stack light.

Nuances

It's not always easy, or straightforward, to break a large story down into small, thin slices. The key is to think about what brings the user value, and what can be demonstrated to the user. You don't want your stories too thin, so that there is nothing to demonstrate. You want them going end to end in your system, from the user interface, into your domain logic, persistence and even collaborating systems.

This example isn't made up or theoretical. It came from a real project I recently worked on. We refer back to it often as a good way to add functionality to our system.

Take-away

Once you have demonstrated working functionality, you are in a position to get the feedback and iteration needed to make a really spectacular system. This is very fertile ground for controlled innovation and growth, because it all starts with demonstrating value. Focus on this slices of functionality, and continuously stay in this sweet spot of value and innovation.

Fun development

2010-11-23T22:58:00.001-06:00

Several things came together today, and though not evident at first, they are all somewhat related.

FIRST

My pair kept asking why I was so adamant about not writing any code without specifying it first. I explained how specifying kept us focused on what we needed to do, helped us drive out the overall design, and eventually provided a safety net for testing and refactoring. In particular, we were practicing outside-in development in Rails, using Cucumber and RSpec. The goals we were achieving, in order, were:

understanding our problem
designing and developing our solution
building up tests for the future.

The cucumber scenarios help us focus on understanding the requirements, from the end-user's point of view. They also drive out the high-level interface of the app, particularly the URL's. The URL's then lead us to create routes, and the routes lead us to create controllers. The cucumber scenarios also drive out any expectations we may have on any models, which the controller will usually set as instance variables.

Once we know we need a particular controller, we jump in to specify the controller, and the detailed behavior we expect there. Fleshing out the controller may drive out any additional functionality needed on the model. Finally, we spec out the model. At this point, we work through any additional functionality in the cucumber scenario. Once all scenarios and specs are passing, and we can't think of any more to write, we're done.

Obviously, my description is quite simplified. There is alot of subtlety here. That's why I suggested that I send some links to some good videos online, which talk about outside in development, specification, integration tests, and design in general. My friend Tom had done something similar earlier this year.

Here is my list, in suggested viewing order. The first is fairly concrete, and then they get more abstract and theoretical.

Joseph Wilk's Working Outside-in with Cucumber and RSpec
J.B. Rainsbergers Integration Tests are a Scam
Michael Feathers' The Deep Synergy Between Testability and Good Design

SECOND

On the way home, I saw that my coworker Mike tweeted about a blog post he wrote today. This struck a chord with me, especially when I got to the end. The entire article rings true in the Enterprise Java space as it does in .NET. Then he clinched it with this:

If I do ever go back to working on a .NET project, I'll bring a lot of new knowledge and ideas that will likely make that a more enjoyable experience.

Over the last few months, I've been having tremendous amounts of fun developing with Cucumber, RSpec and Rails. There seems to be a huge sweet spot, where it's very easy to get into the flow. You start writing the scenarios and specifications, and the application starts to write itself.

I had practiced acceptance testing and unit testing in the java world, and it was alot of hard work! With these tools, it's great fun, and it practically does the work for you. There is still alot of thinking involved, but it is much more focused on the problem domain.

THIRD

Andy talked about a powerful technique to write and grow your specs. In addition, he talks about how the developer specifications, when printed up pretty, made great documentation for a non-technical client. Consider how much stronger his collaboration was because he could share his specifications with his client. Shared language and shared understanding lead to better communication and increased collaboration.

FINALLY

Syntactically, the Ruby language is not very large. You can start reading ruby code very quickly. And, you can read most any ruby code that you want. As an interpreted language, all libraries are shipped in very readable files. In addition, the community seems very open to keeping things open and readable. I have yet to see any library that is obfuscated.

Which brings us to Colin, and his tweet at the end of the day:

pro-tip: every week read up on a new class in the #ruby stdlib. i don't think enough people know what's available to them.

You could wield alot of power when you are given the chance to understand every piece of every library that you are using. This power comes in the form of understanding what you're using.

TAKEAWAY

Learn powerful tools and techniques. Read as much code as you can. Understand the guts of what you're working with. Go out and have fun developing!!!

Your days will be alot more enjoyable, and your work will be alot more professional.

A Perfect Programming Day

2010-11-12T20:23:00.007-06:00

I had what I would consider a nearly perfect programming day today. It really started yesterday before I left, when Jim (tech lead) reviewed an upcoming task with me. I was in the middle of another task, so I suggested we pick up in the morning.

The api team was processing a product file, and we needed to adapt that process so that the web team could also consume it. The next day, Jim and I reviewed the current implementation of the process, as well as the goals of the updates. The process took a file, filtered and translated it, and created a new file. We wanted to add to that process, to create a second file to be used by the web team.

Our BA came by to discuss the details of the fields and the formats we would be getting, as well as how we wanted to filter the records. We all agreed to an approach. Our BA would do the write-up for the sake of sharing the understanding with QA and other team members. We also agreed not to wait for a detailed write-up before starting development.

I was going to work with Matt, who had written the initial process. When Matt arrived, he asked what we needed to do. I gave him an overview of what I understood. He immediately knew where to jump into the code and start. We were working at a station that had not been used for this project before, so first we synched with source control. Then we ran the test suite. After installing a couple of gem dependencies, the tests passed.

Matt showed me the classes involved in the process, and gave me a detailed walk through of how the input was filtered and transformed. This took about 10 minutes. We then talked about what parts were common and could be re-used, and what would change to support the creation of a second file. At that point, instead of digging in, we decided to refactor the current implementation. This would allow us to more easily add the code to create the second file.

This process consisted of two classes: a Processor and a Line. The Processor looped through source file, passed each line to a Line instance to be filtered and transformed, and wrote the output file. The Line class was responsible for the actual filtering and transforming.

We spent a few minutes creating a superclass, and moving common processing up into the superclass. We ran all tests, and they passed. We didn't change any tests along the way. We only changed the implementation by introducing a superclass. This was pure refactoring. We were ready for an initial local commit.

We spent a few minutes trying different names for the superclass and the subclass with the current implementation. Initially, the Line class was called SocLine, named after the file which was being filtered and transformed. We tried Line for the superclass, but that was too generic. We finally went with ProductFileLine as the superclass, representing the source of the data, and ApiLine for the subclass.

In afterthought, these classes' responsibilities were filtering and transforming. Maybe better names would have been ProductFileLineTransformer and ApiLineTransformer.

Next, we looked at the code that was looping through the input file, filtering and transforming, and creating the output file. We added acceptance tests to check that a second file was being created. We modified the loop to create a second file, and our tests passed.

Now we wanted to implement the new filtering and transformation for the web file. This we decided would be better served through unit tests. First we wrote the specs for the new filtering rules, and then the code to get those to pass. Then we wrote the specs to transform each line to the correct format.

At this point, we were done with the happy path for this story. What remained were a couple of specific transformations, and any final cleanup. We also wanted to look at an example input file for any special data considerations. We continued after lunch.

We transformed several columns to change their data types from text flags to booleans. Our specs failed as expected. We updated them to pass, and continued.

We reviewed the order of the data we were exporting, and arranged the fields in a more logical order. We also reviewed the BA's write-up (which was done at this point) for anything that we may have missed. We identified a small item, fixed it, and continued.

At this point, we squashed our local commits down to one, pushed our changes up to source control, and called it a day.

Why perfect?

The obvious point is that the day flew by, and it was incredibly fun. But there is more to it than just fun.

Working with Jim and our BA, we were able to start development quickly with a minimal amount of ceremony. I was able to start developing while the BA was detailing the story. At the end of the day, we were able to do a final check before moving the story to QA. We went with a light a process as we could, and we worked in parallel once we had a good common understanding.

We never strayed too far from all tests passing. We kept our changes small. We worked in small increments, and committed (locally) whenever we were at a stable point. Though there was no need today, had we needed to go back and take a different approach, it would have been as easy as reverting a commit or two.

The code that we started with was already well-factored. This made it easy to make our changes.

There were tests in place when we started, so that refactoring was safe. When we finished refactoring, we were quite confident that we hadn't broken anything.

Finally, we were pairing. Both Matt and I were at the keyboard, and taking turns driving and thinking throughout the episode.

PS: Towards a stronger design

The extension to the design that we did today was sufficient. We could have taken it further, having our Processor class follow the Open-Close Principle. That way, it's core functionality would not need to change when we needed to add another output file. We would simply need to add a new Line class with it's own filters and transformations.

A little learned, a little desired

2010-09-13T22:17:00.003-05:00

You can't use jQuery to dynamically set the title of your web page, even though the syntax (and the documentation) would lead you to believe that you could. I spent about an hour today tracking this one down, before coming across this gem.

Instead of saying: $('title').html('My new page title'); what you really want to do is go back to simple javascript: document.tile = 'My new page title';

To really put a wrench in the whole matter, the jQuery call works in Firefox and Safari, but not in IE.

FloorSweepings: Efficiency and Trust

2010-05-17T22:46:00.001-05:00

My friend recently went on this tirade at FloorSweepings: Efficiency and Trust about ceremony and dress code. Though I should probably be chopping it up about ceremony with him, the dress code part ignited some emotions.

I've always thought that people shouldn't judge you based on what you wear. Instead, they should look at the work you perform, and the value that you bring to your team and your organization. Unfortunately, people aren't so rational.

I've been reading Influence by Robert Cialdini. It's chock full of anecdotes, backed up by real research, about why people act the way they do. It's been quite eye-opening. In particular, one area of research deals with how people respond to others "in uniform". His examples include people posing as security guards, priests, and businessmen in pinstripes. In every case, people are able to exert significantly more influence when the look the part. That is to say, "customers" and "suckers" are more likely to believe your message when your clothes match your role.

Back to my friends rant, I don't think that his consulting manager is out to get him four days a week by asking him to dress no jeans, no sneakers, with dress shirt. Nor is it declaring itself "Old Biz" or non-"Agile" by requesting a certain level of dress. This company understands that most people aren't so enlightened as to focus on the work you do and the value, and is playing it safe by dressing "one-level up". Add to it that most consultancies should be called what they are -- staff augmentation firms -- and you can see why consulting managers want to do everything they can to exert any influence possible on an organization.

So, do you prefer your shirts on hangers or boxed?

Good enough software

2010-05-11T00:18:00.006-05:00

Around 2000-2001, the Pragmatic Programmers introduced me to this term. At the time, I took it to mean doing just enough design to get your feature working. I didn't really take to heart the longer-term trade-offs that lingered around, including technical debt.

Fast-forward to the spring of 2010...I've run across some great articles by Ron Jeffries and Josh Kerievsky addressing this topic, and framing it against product value and productivity.

Take-aways:

Extremely poor quality slows us down. Striving for perfection also slows us down. We need to find the sweet-spot somewhere short of perfection, where we are not slowed by neither defects nor perfection.
We need to design towards perfection on the more important, more valuable parts of the system. Here, value is what the customer perceives, or areas of code where we spend much of our time and cannot afford slow development.
We should back away from perfection, and spend less time designing the areas of the software that are not as critical.
These decisions should be based on the value of the features we are developing, and on how much the design gets in the way of future work.

There is alot of subtlety here, and getting good at this comes with mistakes and experience.

Unit testing and your level of focus

2010-04-28T23:26:00.004-05:00

I ran across this little gem recently. It's about 40 minutes long, and full of great insights into unit testing.

To unjustly summarize, there are several points along the spectrum of unit testing:

No testing
Manual testing
Test-after unit testing
Test-first unit testing
TDD Test-driven-development
TDD Test-driven-design

The sweet spot seems to be in going back and forth between TDD and TDD. However, it takes a good amount of time and effort to be able to recognize where you are on this spectrum, or that there even is a spectrum of testing.

Test-driven-design is a tool to help you work out the classes involved in your solution, and how they will work together. This can be likened to the old school technique of designing your interface.

Test-driven-development is a tool to bullet-proof your code; to test the edge-cases; to test for boundary values; and generally to make sure that your code works in as many situations as possible. This can be likened to the old school technique of fleshing out your implementation.

It takes great skill to work back and forth between these two flavors of of TDD. Yet by getting to that point, in addition to a solid design and a solid implementation, you'll have the safety net in place to be aggressive, make changes, and respond to your clients needs.

You may even quickly deliver valuable features. What could be better than that?

Focus

2010-04-17T00:07:00.000-05:00

I'm working on an application that plays video, and displays some widgets and graphics on top of that, depending on what actions the user makes. Some of the widgets have a degree of transparency to them, others do not, and cover up the entire area of video behind them.

I was working on getting the widgets to display the right information, and didn't pay as much attention to how each layer was interacting with those below. When I demoed my work to my colleagues, one immediately noticed that when the widgets overlapped, the transparency of the top was was being passed on to the one below, making the overlapping areas transparent and displaying the video below.

I was focused on displaying the correct information, and didn't even notice the transparency problem. My colleague, a graphics expert, immediately noticed the problem, and started thinking of ways to solve the issue. That's when our other colleague, also a graphics expert, chimed in with a very pragmatic idea:

Let's move the widgets around, and put off fixing the problem until we have to.

Takeaways: While focusing on a specific task, continue to keep the entire application (project) in mind.

In my case, while I was working on presenting the correct data, I should have paid attention to other details in the application.
As graphics experts, my colleagues could have jumped at the chance to fix the transparency problem. Instead, they stepped back to consider this problem in regards to other priorities within our project, and decided to delay the fix.

Software Engineering?

2010-04-13T10:39:00.000-05:00

Tom DeMarco is "gradually coming to the conclusion that software engineering is an idea whose time has come and gone." He says so here. He concludes:

Consistency and predictability are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

Go out and build great software. You should take pride in doing it well -- so that others can enjoy reading and maintaining your code. But your most important focus should be on building great software for your users. Focus on rocking their world. Change the way they do things, and change it significantly.

What Do We Know about Agile Software Development?

2010-04-06T11:53:00.000-05:00

Dybå, Tore and Dingsøyr, T. "What Do We Know about Agile Software Development?" IEEE Software, vol. 26, no. 5, 2009, pp. 6-9.

The authors surveyed the academic literature, to summarize what is known (academically, scientifically) about agile software development. Based on my experience in the field, there seems to be a disconnect between academic research and practices in the trenches. Below are some of the findings from this research, and my responses.

Some studies saw pair programming as inefficient, and some studies claimed that XP works best with experienced development teams. One study reported the further limitation, which the literature repeatedly mentions, of lack of attention to design and architectural issues.

Any development method or development process will work best with experienced development teams, particularly when compared against itself. What I have seen is that XP is great at taking a mixed team, and improving the abilities of everyone on the team. The practices of open communication and collaboration, collective code ownership, pair programming and emergent architecture all contribute. As the project progresses, each team member, no matter how junior, will have worked on many aspects of the system, each full of opportunities for learning.

Using a prescriptive development process could still lead to a lacking design and a lacking architecture. XP provides practices of refactoring, collective code ownership, test driven development and emerging architecture. These practices together allow the team to constantly improve the design, while building out only as much architecture as necessary. Diligence and attention to detail can be used or abused in any development process.

However, the on-site customer’s role can be stressful and unsustainable for long periods.

There is more than enough here for another full post. Without an on-site customer, we can regress to having long feedback cycles, waiting for emails or phone calls to be answered. We may even regress back to writing a requirements document, usually by a business analyst, and throwing it over the wall to development. The document is then ripe for mis-interpretation and other abuses by the readers. To have the right software developed, the right person needs to be fully involved and committed.

The findings regarding pair programming’s effectiveness were mixed. Several developers regarded it as an exhausting practice because it requires heavy concentration.

Pair programming provides many benefits: automatic code review, information exchange, focus, efficiency, learning. XP has another practice, sustainable pace, which offsets the intensity. The authors even note in their findings that agile team members believe they are more efficient.

One limitation that came up was that team members are less interchangeable in agile teams, which has consequences for how projects are managed.

By pairing and through collective code ownership, the team is spreading the knowledge around, so that more and more of the team members can work on each part of the system. Spreading this knowledge decreases your dependency on a particular developer for a particular area of the code.

Pair programming also makes it much smoother to introduce new developers to the team. The new developer will always be working with someone who is familiar with the code base, the design, and the entire project. This makes it easier to bring on new team members.

If the point of interchangeable developers is to maximize resource utilization, this seems like a case of sub-optimizing. The goal of a project manager should not be to make sure that each resource is 100% efficient, but instead that the team (of people) is meeting its goal: delivering the right software.

Our review identified 1,996 studies from literature searches, 36 of which we found to be research studies of acceptable rigor, credibility, and relevance to include in the review.

I may be biased by successful agile projects in which I have participated over the last ten years. I may also be biased by all of the pairs I have worked with, who taught me new things and made each task much more fun.

Improving your team's code

2010-04-05T11:16:00.000-05:00

Peer code reviews are a great way to catch software defects early and often. This practice has been around since at least the mid-seventies [Fagan '76].

Extreme programming (XP) has a couple of practices built-in to support reviews. Collective Ownership encourages everyone to contribute new ideas to all segments of the project. Pair programming increases software quality without impacting time to deliver. All code to be sent into production is created by two people working together at a single computer.

Having experienced both traditional code reviews, and full-blown pair-programming, I believe pairing to be a very effective and efficient step towards good, clean code.

Yet, I've had team-mates that didn't want to be part of the process, and I've had team leaders who didn't know how to constructively make it work. I have come across some good practices for peer code review to pass on to them. Even though this is written from the point of view of traditional code reviews, I think that the principles can be applied to Agile environments as well. Below are some excerpts I particularly enjoyed:

8. Managers must foster a good code review culture in which finding defects is viewed positively.
Code review can do more for true team building than almost any other technique we’ve seen – but only if managers promote it at a means for learning, growing, and communication. It’s easy to see defects as a bad thing – after all they are mistakes in the code – but fostering a negative attitude towards defects found can sour a whole team, not to mention sabotage the bug-finding process.
9. Beware the “Big Brother” effect.
Managers must continue to foster the idea that finding defects is good, not evil, and that defect density is not correlated with developer ability. Remember to make sure it’s clear to the team that defects, particularly the number of defects introduced by a team member, shouldn’t be shunned and will never be used for performance evaluations.
10. The Ego Effect: Do at least some code review, even if you don’t have time to review it all.
The “Ego Effect” drives developers to write better code because they know that others will be looking at their code and their metrics. And no one wants to be known as the guy who makes all those junior-level mistakes. The Ego Effect drives developers to review their own work carefully before passing it on to others.

Managing ideas, ala Weinberg

2010-02-07T22:16:00.000-06:00

This is a list of ways to get ideas flowing when trying to solve a problem, particularly in a team, based on Jerry Weinberg's Becoming a Technical Leader. Each of his ideas is great on its own, and they work well together, but there are also subtleties that you should be aware of.

Contribute a clever idea to the team. When it works, and when it's really novel, this is really cool. Most of the time, there are no new ideas, but re-hashed ideas from other sources. Instead of trying to get the perfect solution right away, spend your energy creating an environment where ideas can be exchanged and the "right" ideas can be recognized when they come along.

Encourage copying of useful ideas. Look for inspiration in your own branch, or in another knowledge area as well. And remember problems that you've already solved, in case this problem is similar.

Elaborate on an idea that a teammate contributed. Iterate, and make it better. Keep following through. Follow up the inspiration with enough perspiration to see it through.

Drop one's own idea in favor of an idea the team wants to develop. Strike a balance between stubbornly holding on to your ideas, and letting go of your ideas just for the sake of keeping the group happy and moving along.

Refuse to let an idea drop until everyone understands it. Like the previous example; this will come with time and experience.

Resist time pressure, and take the time to listen when other people explain their ideas. Take extra care to know why an idea is being dropped.

Test ideas contributed by other people.Look for the initial idea anywhere and everywhere, and then make it better.

Withhold quick criticism of teammates' ideas, in order to keep the ideas flowing.This is particularly important at the beginning, such as in a brainstorming session. Make the tradeoff to accept more ideas, which will be dropped later, than to drop early and not generate enough ideas. This improves with experience.

When you must criticize an idea, make clear that you are criticizing the idea, not the person who offered the idea.In addition to maintaining a good relationship with your teammates, this let's you explain which parts of the idea work, and which ones don't. It also let's your teammates in on how you come to your conclusions.

Test your own ideas before offering them.Take the extra time to scrutinize your ideas the same way you would the ideas of your teammates.

When time and labor are running short, stop working on new ideas and just pitch in.

Encourage the team to drop ideas that had succeeded earlier, but cannot be extended to the new situation. Make sure that the ideas fit in with what you are trying to accomplish.

Revive a dropped idea later, when it has value for another part of the problem. Remember your old ideas and solutions, because they may become useful again.

Reusable classes

2009-12-15T22:22:00.000-06:00

Today, my pair commented on how we have gotten a lot of use from our test-helper classes. How true, how true. Over the course of the last year, we have created the classes needed in by our application, and the tests to support those classes. But we didn't stop there. We also created many small utility classes to support our tests. These classes are barely worth writing, or so it would seem. They are actually priceless.

We have gotten a lot of re-use from these classes because they are small and concise. They each address and encapsulate one sole responsibility. The classes usually consist of two or three methods, and are no longer than about twenty lines. It's in this precision that they find their utility. Being small and to the point, it is very easy to pick the right class when needed.

We've developed classes to set and reset system properties; to set and reset logging levels; to read and write private fields; to set id's on our domain models; and many other small responsibilities. At first, when developing these helpers, we would wait for a duplicate need to appear, and then factor out a re-usable component. As we got better at this, we wouldn't wait for a duplicate need, but instead would factor out responsibilities as we saw them appear. This helped us get even more utility from these classes. As soon as the classes were created, uses for them would appear.

By encapsulating concepts, keeping concepts small and concise, and identifying concepts early, you will increase your chances for re-use, and get a lot of mileage from your classes.

My takeaways from this experience:

encapsulate responsibilities
create small classes
factor responsibility early

Hindsight is 20-20, even several years later

2009-12-05T00:31:00.000-06:00

I ran into an old colleague, Alex, on the train today. We went through the usual catch up conversations, and got into talking about some of the old projects we had worked on together.

Alex mentioned that an old application that I had worked on along time ago was still being used, and the code still maintained. Our project was a banking web site, and this application was a supporting application that was used to verify and enroll users into the main application, which was a java web application. In typical hindsight is 20-20 fashion, we started to comment on how we might have approached that application differently.

To appreciate what we would have done differently, I guess I owe a little background on the application. The original enrollment application was a Lotus Notes application, and it verified the enrollee information against the account database. When lookups against the account database started having performance problems, we decided to "quickly" fix the problem by replacing the lookup with a PowerBuilder application. After some modest success improving performance, we decided to fully re-write the enrollment application in PowerBuilder. The application was written along-side the main application, with little design, little visibility and little testing. Despite that, it was deployed on time, and it worked.

Looking back, I would have made this a full-fledged and visible project. The enrollment application, by its nature, has a lifespan that matches that of the main application. This alone should have been consideration enough to give the enrollment application more visibility.

Alex mentioned that maintenance of the application was passed around from team to team. In hindsight, I would have implemented the enrollment application using the same technologies as the main application. The enrollment application was written in PowerBuilder, by a couple of team members, while the rest of the team was working on a Java web application. This would bring the technical skill set in line with the main application, improving the chances of maintaining the application.

I mentioned increasing the visibility. To me this means two things: more project management visibility, and more testing. Over the years, I've seen that detailed visibility into the details of a project, and into the true status of tasks, leads to more successful projects. In part, because developers tell it like it is, and in part because managers can see the details for themselves. This outcome of the original project may not have changed, since the application worked, but the process of getting there would have been much smoother. The same goes with more testing; the outcome may not have changed, but the project would have gone smoother.

Finally, when the PowerBuilder application was first developed, it was thought that this would be a short-lived application, to help with performance for a short time, and then the application would be retired. Of course, it is still not retired. In hindsight, I would have approached even the smallest effort with the same rigor as an application planned for a long lifespan. You never know when an application will take on a life of its own, and outlive everyone's expectations.

Where have I been?

2009-12-02T23:59:00.000-06:00

I need to thank my friend Tom for getting me back to writing. He's doing a craftsman spike to refresh his learning process. Talking to him got me back on track about documenting my own learning. I haven't been documenting lately because I've been studying. I'm 3 classes away from getting my Master's in Software Engineering at DePaul University.

I started looking into this a couple of years ago while in a rut. I wasn't inspired by my work. The problems were interesting, but I wasn't. Instead of solving problems, I spent a lot of energy trying to muster up the inspiration to get into the problem. Up to that time, all of my programming experience had been on the job, so I thought that some formal study would jump-start me. A year and a half later, that was a good decision. I've learned things about the how software is developed, as well as how software can be developed. Best of all, I again look forward to getting up and going to work; to solving problems; and to learning a little more each day.

To give myself a best chance to succeed, I will need to combine all of what I've learned on the job, everything I've read along the way, and what I've learned in my course work. There are some great SE practices that can help move your project forward, as there are some great practices and trends from the field, along the lines of agile methods. It's to cross-pollinate some of these ideas, to mix and match practices and techniques, to upgrade our toolbox for development. I plan to share some of my ideas on this....soon.

Keeping it simple

2008-07-01T14:44:00.000-05:00

A teammate asked for help on a problem he was having. He was using Hibernate to retrieve data from a table, and wanted to retrieve a character column (Y,N) as a Boolean. His first thought was to the @Type annotation. He also considered setting the hibernate.query.substitutions property.

When he asked me for help, both of these solutions seemed quite complicated for his problem. I asked about retrieving the data as a String, but he said he tried and that it did not work. He was stumped. I was skeptical. I was sure there was a simple way to get this to work.

He changed his model to represent the data as a string, reloaded the system, and showed me the error he was getting. When we followed the stacktrace back to the line that was causing an error, I noticed that the model being retrieved was not the one he as changing. He had two similarly-named models, and was changing one while testing the behavior of the other.

These questions came to mind:

Why wasn't he pairing while working on this task?
Why were the models similarly named?
Why did he need to reboot the system? Why wasn't there a quick integration test written?

Respect your unit tests

2007-12-27T15:25:00.000-06:00

Our team just completed a major release of our software. It's been three months since the last release. This is a very long time based on our usual pace of 2 to 3 weeks, so we could expect to have some issues after deployment. Had we shown some respect for our unit tests, we would have fewer issues.

One of the problems we had was related to some external libraries we were using, which required our models to use conform to the JavaBeans spec ( having both a getX() and a setX(X x) method). In some cases, the getX() was the only meaningful method, and the setX(X x) was only put there as a placeholder to conform to the API. If the setX(X x) had a nice comment explaining it's purpose in life, we were safe. But sometimes, we had the stub setX(X x) method, without an explanation of why it was there. And upon doing some reference checks, we would quickly determine that it was not used by any code, and could be removed.

That's just what happened in this case. A developer noticed the dead code, asked around if it was used, got concensus that it wasn't and proceeded to remove it. He ran the unit tests, fixed a couple of broken tests, and checked in his changes.

He removed un-used code, and then had to fix some unit tests.

And when we released the code to production, we had problems.

The fact that the unit tests failed and needed fixing should have been the indication that the code was used, even if not explicitly referenced by any of our code. The fact that we had production problems confirmed it. The unit tests were in place to prevent this error. But the developer bypassed that warning.

The unfortunate fact is that it is very easy to learn from a mistake. But it's these mistakes that we need to file away as experience, so that we get better at what we do. Take care in writing unit tests, so that they are meaningful and accurate. And then take greater care when modifying them. When in doubt, trust that the unit tests are right. Grab a pair; get a second opinion; get a third opinion. Let the unit tests serve as one of your safety nets when making changes to code. And respect your unit tests. They don't lie.

Back to the basics of Refactoring

2007-12-04T13:25:00.000-06:00

I ran into a couple of separate bugs the last couple of days, that were caused because of some refactoring. How can that be, you ask? I asked myself the same thing! By definition, refactoring should change the structure of the code, without changing the behavior. Yet, in both cases, the behavior did change. Let's see if there is a way to prevent this from happening, and keep with the true nature of refactoring.

Refactoring defined

Martin Fowler defines refactoring like this:

Refactoring is a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior. Its heart is a series of small behavior preserving transformations.

The key point I see is "small behavior preserving transformation." A refactoring is a small change, and it does not change the behavior.

Preserving Behavior with Unit Tests

The best way to ensure that you are preserving the behavior, is to have a suite of fast and accurate unit tests. Yes, I am asking alot here.

The unit tests need to be fast, so that developers do not feel bogged down by having to run them. When they are fast, developers can run all unit tests many times throughout their development cycles.

The unit tests need to be accurate so that they go so far as to specify the intended external behavior of a module (or class, or method, or whatever your "unit" is), yet they are flexible enough so that they do not specify the internal structure. This is part science and part art, and is well beyond the scope of this discussion. But the goal is to specify behavior, not implementation.

So why did these refactorings go south?

We all know the theories of modern development (everything is tested, pair programming, the simplest thing that works), yet it can be hard to put them into practice.

One of the refactorings I encountered was very subtle in how it broke down. The developer ran all units tests, they passed. Then he made the refactoring. Then he ran all tests again, and they passed again. So he was done, and went on to the next task. However, the tests passed because the code in question was not covered by tests.

So let me restate my thesis. The best way to ensure that you are preserving the behavior, is to have a suite of fast, accurate, and complete unit tests. There, that should do it.

When unit tests are not enough

The other refactoring was a bigger change. The goal was to put the soft into the software. We wanted to take some data that was hardcoded into the application, and move it to the database, so that changes became configurable. The problem was that we had three scenarios, but two of them shared the same hardcoded implementation. So when we went to a configurable implementation, we lost one of the scenarios.

This problem could also have been avoided by tests, though maybe not what is considered a "unit" tests. To catch this problem as soon as it happened, a broader suite of test would be needed: integration tests, functional tests. The key, though, is that these tests should be broad enough to cover the scope of the refactoring, fast enough to be able to be run by the developer, and complete enough to have tested (and exposed) all scenarios.

One last time. The best way to ensure that you are preserving the behavior of your code while refactoring, is to have a fast, accurate, complete and broad suite of tests. Fast enough to run often, accurate enough to specify the behavior, complete enough to test what you are changing, and broad enough to cover your code base at different levels of granularity.

Conclusion

Refactoring improves the code base. Refactoring simplifies the code, makes it easier to comprehend, and easier to change. No arguments there. The key, however, is that refactoring preserves behavior, and the only way to ensure that behavior is preserved is to have a broad and accurate suite of tests around the code.

Stated another way, the end user should not be able to tell immediately that you have completed a refactoring. Instead, they should be somewhat impressed when you quickly deliver the next feature that they request.

A quick review of debugging Struts applications

2007-11-29T08:31:00.000-06:00

I had worked on a Struts web application a couple of years ago, and within our team I am still considered the "expert" on that application. So yesterday, when something wasn't working correctly, a teammate approached me and asked for help. As I walked her through what was going on, I made it a point NOT to rely on any knowledge of the application (after all, it has been two years and several maintenance programmers since I have worked on the application.) Here's what we did:

Based on the URL, track down the action mapping
Look in the struts-config.xml file and find the JSP that is rendered
Examine the JSP and see where the data is coming from; identify the form object that holds the data
Back in struts-config.xml, find the action that does the work
In the action, look at how the form is populated

In finding and resolving the problem, I didn't have to use any tacit knowledge of application. Instead, we just ran the application, identified the problem, and tracked it back to the code that was causing the problem.

A quick and easy way to minimize java.lang.NullPointerExceptions

2007-11-20T17:51:00.000-06:00

In Java, we often see code that compares a particular value to some know constant. Often this is written like this:

someObject.getSomeValue().equals("SomeConstant");

This works ok, assuming that someObject is not null, and as long as you are sure that getSomeValue() will never be null.

If you aren't so sure, or if you just want to develop a good habits that will minimize the number of NullPointerExceptions you run into, you may try to write the same comparison this way:

"SomeConstant".equals(someObject.getSomeValue());

You are ensuring that you will not run into the dreaded java.lang.NullPointerException, because your constant value will never be null. And you are improving your own productivity, because you and your teammates will spend less time tracking down and fixing NullPointerExceptions.

Glimmer to ECLIPSE RubyOnRails?

2007-11-16T12:00:00.000-06:00

When I hear about Ruby, the first thought that comes to mind is Ruby on Rails and Web 2.0 applications. I would have never made the association from Ruby to desktop application. Until now. About a year ago, it was suggested that JRuby and SWT might be a viable combination for Ruby on the desktop. After all, SWT is the performant, native desktop library available from Eclipse, and Ruby gives you many productivity advantages. There was even a SWeeTgui project at the time, though it doesn't seem like there was much traction. Fast-forward one year, and we now have Glimmer: "a JRuby DSL that enables easy and efficient authoring of user-interfaces". What advantages are there with Glimmer? Here's what I see:

A compact api that allows Ruby developers to write native destop applications
A clean wrapper around the SWT libraries, that takes a minimalist approach by exposing the most important features and applying smart defaults everywhere
An API that is based on Ruby's programming paradigms, not Java's.
The ability to implement complex SWT desktop applications with only 25% of the code.

That last point is what brought me over. Being able to write the same functionality with just a quarter of the code (and time). I've been developing Java applications for nearly a decade now, and using SWT for two years, and I feel very comfortable. .

When I first saw Glimmer, I didn't believe that I needed it for my Java SWT applications, because I know Java, and I know SWT. But as we discussed the merits of this API, and I saw a demonstration of some complicated user interfaces, I got a "glimmer" in my eye. I could see alot of productivity benefits here.

Take a look for yourself, and consider it for your next desktop project.

A REAL Onsite Customer

2007-11-02T00:40:00.000-05:00

I'm currently working on a project that is using an agile process to manage development. We use most of the XP practices, but are mising what I would consider the most important one: an onsite customer. Though our customer is a manager, and the final decision-maker, she doesn't participate directly in the daily development activities. Instead, she sends a proxy.

This works out great about 75% of the time, but doesn't work so well the other 25%, when we need a tough question answered quickly. And that often introduces some long waiting. Here are a couple of examples.

Last Friday, the manager had her proxy call a meeting with the developers to discuss a featuer we had just completed. She wanted to improve the flow of the feature, and make sure that it was as easy as possible for the users. We got together with the proxy, brainstormed, offered ideas and estimated the different steps. He then went back to her, she had some other ideas, and some questions. So he came back to us and .....

All this going back and forth was costing precious time. We are still answering questions and going over ideas, though we never really meet with our customer. There are several problems with this approach. Information is lost, intent is misunderstood with each link in the chain, and delays are introduced. And this happens both ways. The proxy is an intelligent guy, but all the intelligence in the world doesn't help here. Its the nature of communication, similar to the game we played in grade school, where we each whispered something to the next person, and what came out of the chain is not what went in. Extra energy is expended going back and forth. It would be much more convenient, and direct, to be talking directly to the decision-maker.

Contrast this to what happened yesterday. We were discussing a feature that the developers thought must be included, but the manager thought wasn't necessary. Of course, she sent the proxy to discuss this with us. However, we didn't feel she had a strong argument, and she didn't think we had a strong argument. By chance, one of our teammates saw her walking by, and we asked her to explain her point. Then we explained our. We were quickly able to come up with some options that would meet her requirements and our own. I walked out of this meeting energized and excited about our application. We were going to deliver a valuable feature that met the needs of our users, and our developers, and we were able to get to that point very quickly.

All modern books preach the need for an onsite customer. This experience of two extremes solidified the idea in my mind. Your team can definitely work more effectively with an onsite customer. Not a proxy, nor a proxy to the proxy, but the real decision-maker, in the trenches with your team, day in and day out.

RTFM

2007-10-31T00:45:00.000-05:00

Yesterday, a colleague sent out an email to the group, asking for help on a certain topic. Over IM, I asked if he was doing this for a particular scenario, which he was. So I suggested he use a feature of the tool we are using. He said he wasn't aware of that feature, so I said "RTFM" (read the frickin' manual).

This didn't go over very well with him. His first reaction was to lecture me, never to do something like this again. I didn't need a lecture, but I apologized, trying to quell an escalating situation. Later, after he calmed down, he explained that this triggered memories of bad relationships with past colleagues, and that he didn't want our rapport to get out of hand. Fine, I can accept that. But this situation got me to thinking about two things: why I said RTFM to begin with, and how I could have said the same thing in a more gentle manner.

As a consultant, I know alot is expected of me by the client. There is a reason they are paying more for my services than they pay their employees, and I have an obligation to keep up my end of the deal. I expect the same of the other consultants on the project. In this case, I expected my colleague to know the tools that we are working with, and to have at least gone through the manuals and have been familiar with the concepts. Staff employees often expect to be sent to training. As consultants, we need to go out of our way to quickly learn as much as we can about the tools and technologies we are using. We also need to be able to pass that knowledge on to our teammates, whether they are consultants or employees. It's by this transfer of knowledge that the entire team gets better, and the entire project can go smoother.

Which brings me to my next thought: how could I have been more effective in that communication. After thinking about it, I should have said something along these lines:

You can use this feature. You should be able to find information about this in the manual. Take a look, and let me know if you have any questions.

It would have gotten the point across much better. It also would have been in line with the kind of response my colleague expected. Even though I don't have any particular title, I am often perceived as a leader on our team. To fulfill my part of being a leader, I should have used the gentler statement to get my point across.

The work we do executing projects and developing software is difficult enough. We don't need to make it any more difficult by creating problems among our colleagues. As a team leader, whether by title or by perception, we need to excel at increasing the knowledge of our entire team (and organization), and working towards project success.