Thin, valuable features

You're on a team developing an e-commerce site for a traditional retailer. The product owner has a great idea about the Store Locator feature. As you start to discuss this story, you start to wonder how many ITERATIONS it will take you to develop the functionality.

Users can search for a store by zip code or city and state. The five closest stores will be displayed with their address, phone number and store hours. A map will be displayed showing the five closest stores, with the ability to zoom in and out. Finally, the search results will also show the regional managers name, phone number, email and picture. Oh, and if the user is logged in, lets pre-fill their address to search.

Too much at once

After thinking about this "story" or "feature", you decide that it's too much work for one pair to complete in a week, and might even be too much to complete in two weeks. What can you do? You think about breaking it up so that you complete the domain models and services in the first week, and then the user interface during the second week. This could work, but there are a few problems with this approach.

First, will the models that you develop during the first week be sufficient to complete the user interface without changes? Will they have enough functionality, and will the functionality be correct?

Second, what will tell your product owner when he asks how the story is going? Will you give say that you're 40% done? If you are 40% done, when will you be all done? More importantly, will you be able to show any functionality along the way?

Finally, if someone offers to help with this story, will you be able to hand-off any tasks? Can any of the work be done in parallel?

Breaking up your story in this way, focusing on models and services first, and then on user interface, leads to alot of difficult questions. The answers, unfortunately are either "no" or "I don't know". So, what can you do to improve the situation?

Strands of functionality

What if you take this "story", and instead re-word it into several end-to-end features? Each piece of functionality is much smaller. Once complete, each feature can be demonstrated to the product owner. Once some critical features are complete, other features can be developed in parallel. Finally, once the product owner starts to see this story built out he may have other ideas, or want to change direction altogether. So, what do we need to do to give our team all of these options?

First, lets break this story down into features. The difference is subtle, and development teams may use these words interchangeably. I like to think about a feature as a single, end to end, piece of functionality, that adds value to the user. So how can we break the Store Locator story down into some constituent features? Here's one way:

1. User searches by zip code, and sees a list of stores with address and hours.
2. User searches by address.
3. Results are displayed with a static map.
4. Map can zoom in, zoom out, and pan.
5. Display the regional managers name, phone and email.
6. Display the regional managers picture.
7. Pre-fill the search when the user is logged in.

Good for everyone

When we think about this story by features this way, alot of opportunities open up. Each feature is small and focus, and therefore easier to develop. After the initial search with results is developed, features can be developed in parallel. The product manager will be able to see real progress much easier. The individual features can be prioritized amongst each other, as well as among other stories, so that only the most valuable features are developed.

Though I initially looked at breaking up the stories from the product owners perspective, there are also benefits for the developer. The biggest benefit is that development can focus on just one slice of functionality at time. This is much easier to think about than the original, full story. You can focus on search without worrying about maps or regional managers. Often when working on a large sotry, it's hard to know where to start. Breaking it down into thin features makes it much easier to focus.

Working on thin slices is also alot of fun to develop. The combination of quick wins and rapid feedback energizes developers. The laser focus of the features keeps their mental stack light.

Nuances

It's not always easy, or straightforward, to break a large story down into small, thin slices. The key is to think about what brings the user value, and what can be demonstrated to the user. You don't want your stories too thin, so that there is nothing to demonstrate. You want them going end to end in your system, from the user interface, into your domain logic, persistence and even collaborating systems.

This example isn't made up or theoretical. It came from a real project I recently worked on. We refer back to it often as a good way to add functionality to our system.

Take-away

Once you have demonstrated working functionality, you are in a position to get the feedback and iteration needed to make a really spectacular system. This is very fertile ground for controlled innovation and growth, because it all starts with demonstrating value. Focus on this slices of functionality, and continuously stay in this sweet spot of value and innovation.

Fun development

Several things came together today, and though not evident at first, they are all somewhat related.

FIRST

My pair kept asking why I was so adamant about not writing any code without specifying it first. I explained how specifying kept us focused on what we needed to do, helped us drive out the overall design, and eventually provided a safety net for testing and refactoring. In particular, we were practicing outside-in development in Rails, using Cucumber and RSpec. The goals we were achieving, in order, were:

1. understanding our problem
2. designing and developing our solution
3. building up tests for the future.

The cucumber scenarios help us focus on understanding the requirements, from the end-user's point of view. They also drive out the high-level interface of the app, particularly the URL's. The URL's then lead us to create routes, and the routes lead us to create controllers. The cucumber scenarios also drive out any expectations we may have on any models, which the controller will usually set as instance variables.

Once we know we need a particular controller, we jump in to specify the controller, and the detailed behavior we expect there. Fleshing out the controller may drive out any additional functionality needed on the model. Finally, we spec out the model. At this point, we work through any additional functionality in the cucumber scenario. Once all scenarios and specs are passing, and we can't think of any more to write, we're done.

Obviously, my description is quite simplified. There is alot of subtlety here. That's why I suggested that I send some links to some good videos online, which talk about outside in development, specification, integration tests, and design in general. My friend Tom had done something similar earlier this year.

Here is my list, in suggested viewing order. The first is fairly concrete, and then they get more abstract and theoretical.

1. Joseph Wilk's Working Outside-in with Cucumber and RSpec
2. J.B. Rainsbergers Integration Tests are a Scam
3. Michael Feathers' The Deep Synergy Between Testability and Good Design

SECOND

On the way home, I saw that my coworker Mike tweeted about a blog post he wrote today. This struck a chord with me, especially when I got to the end. The entire article rings true in the Enterprise Java space as it does in .NET. Then he clinched it with this:

If I do ever go back to working on a .NET project, I'll bring a lot of new knowledge and ideas that will likely make that a more enjoyable experience.

Over the last few months, I've been having tremendous amounts of fun developing with Cucumber, RSpec and Rails. There seems to be a huge sweet spot, where it's very easy to get into the flow. You start writing the scenarios and specifications, and the application starts to write itself.

I had practiced acceptance testing and unit testing in the java world, and it was alot of hard work! With these tools, it's great fun, and it practically does the work for you. There is still alot of thinking involved, but it is much more focused on the problem domain.

THIRD

Andy talked about a powerful technique to write and grow your specs. In addition, he talks about how the developer specifications, when printed up pretty, made great documentation for a non-technical client. Consider how much stronger his collaboration was because he could share his specifications with his client. Shared language and shared understanding lead to better communication and increased collaboration.

FINALLY

Syntactically, the Ruby language is not very large. You can start reading ruby code very quickly. And, you can read most any ruby code that you want. As an interpreted language, all libraries are shipped in very readable files. In addition, the community seems very open to keeping things open and readable. I have yet to see any library that is obfuscated.

Which brings us to Colin, and his tweet at the end of the day:

pro-tip: every week read up on a new class in the #ruby stdlib. i don't think enough people know what's available to them.

You could wield alot of power when you are given the chance to understand every piece of every library that you are using. This power comes in the form of understanding what you're using.

TAKEAWAY

Learn powerful tools and techniques. Read as much code as you can. Understand the guts of what you're working with. Go out and have fun developing!!!

Your days will be alot more enjoyable, and your work will be alot more professional.

A Perfect Programming Day

I had what I would consider a nearly perfect programming day today. It really started yesterday before I left, when Jim (tech lead) reviewed an upcoming task with me. I was in the middle of another task, so I suggested we pick up in the morning.

The api team was processing a product file, and we needed to adapt that process so that the web team could also consume it. The next day, Jim and I reviewed the current implementation of the process, as well as the goals of the updates. The process took a file, filtered and translated it, and created a new file. We wanted to add to that process, to create a second file to be used by the web team.

Our BA came by to discuss the details of the fields and the formats we would be getting, as well as how we wanted to filter the records. We all agreed to an approach. Our BA would do the write-up for the sake of sharing the understanding with QA and other team members. We also agreed not to wait for a detailed write-up before starting development.

I was going to work with Matt, who had written the initial process. When Matt arrived, he asked what we needed to do. I gave him an overview of what I understood. He immediately knew where to jump into the code and start. We were working at a station that had not been used for this project before, so first we synched with source control. Then we ran the test suite. After installing a couple of gem dependencies, the tests passed.

Matt showed me the classes involved in the process, and gave me a detailed walk through of how the input was filtered and transformed. This took about 10 minutes. We then talked about what parts were common and could be re-used, and what would change to support the creation of a second file. At that point, instead of digging in, we decided to refactor the current implementation. This would allow us to more easily add the code to create the second file.

This process consisted of two classes: a Processor and a Line. The Processor looped through source file, passed each line to a Line instance to be filtered and transformed, and wrote the output file. The Line class was responsible for the actual filtering and transforming.

We spent a few minutes creating a superclass, and moving common processing up into the superclass. We ran all tests, and they passed. We didn't change any tests along the way. We only changed the implementation by introducing a superclass. This was pure refactoring. We were ready for an initial local commit.

We spent a few minutes trying different names for the superclass and the subclass with the current implementation. Initially, the Line class was called SocLine, named after the file which was being filtered and transformed. We tried Line for the superclass, but that was too generic. We finally went with ProductFileLine as the superclass, representing the source of the data, and ApiLine for the subclass.

In afterthought, these classes' responsibilities were filtering and transforming. Maybe better names would have been ProductFileLineTransformer and ApiLineTransformer.

Next, we looked at the code that was looping through the input file, filtering and transforming, and creating the output file. We added acceptance tests to check that a second file was being created. We modified the loop to create a second file, and our tests passed.

Now we wanted to implement the new filtering and transformation for the web file. This we decided would be better served through unit tests. First we wrote the specs for the new filtering rules, and then the code to get those to pass. Then we wrote the specs to transform each line to the correct format.

At this point, we were done with the happy path for this story. What remained were a couple of specific transformations, and any final cleanup. We also wanted to look at an example input file for any special data considerations. We continued after lunch.

We transformed several columns to change their data types from text flags to booleans. Our specs failed as expected. We updated them to pass, and continued.

We reviewed the order of the data we were exporting, and arranged the fields in a more logical order. We also reviewed the BA's write-up (which was done at this point) for anything that we may have missed. We identified a small item, fixed it, and continued.

At this point, we squashed our local commits down to one, pushed our changes up to source control, and called it a day.

Why perfect?

The obvious point is that the day flew by, and it was incredibly fun. But there is more to it than just fun.

Working with Jim and our BA, we were able to start development quickly with a minimal amount of ceremony. I was able to start developing while the BA was detailing the story. At the end of the day, we were able to do a final check before moving the story to QA. We went with a light a process as we could, and we worked in parallel once we had a good common understanding.

We never strayed too far from all tests passing. We kept our changes small. We worked in small increments, and committed (locally) whenever we were at a stable point. Though there was no need today, had we needed to go back and take a different approach, it would have been as easy as reverting a commit or two.

The code that we started with was already well-factored. This made it easy to make our changes.

There were tests in place when we started, so that refactoring was safe. When we finished refactoring, we were quite confident that we hadn't broken anything.

Finally, we were pairing. Both Matt and I were at the keyboard, and taking turns driving and thinking throughout the episode.

PS: Towards a stronger design

The extension to the design that we did today was sufficient. We could have taken it further, having our Processor class follow the Open-Close Principle. That way, it's core functionality would not need to change when we needed to add another output file. We would simply need to add a new Line class with it's own filters and transformations.

A little learned, a little desired

You can't use jQuery to dynamically set the title of your web page, even though the syntax (and the documentation) would lead you to believe that you could. I spent about an hour today tracking this one down, before coming across this gem.

Instead of saying: $('title').html('My new page title'); what you really want to do is go back to simple javascript: document.tile = 'My new page title';

To really put a wrench in the whole matter, the jQuery call works in Firefox and Safari, but not in IE.

FloorSweepings: Efficiency and Trust

My friend recently went on this tirade at FloorSweepings: Efficiency and Trust about ceremony and dress code. Though I should probably be chopping it up about ceremony with him, the dress code part ignited some emotions.

I've always thought that people shouldn't judge you based on what you wear. Instead, they should look at the work you perform, and the value that you bring to your team and your organization. Unfortunately, people aren't so rational.

I've been reading Influence by Robert Cialdini. It's chock full of anecdotes, backed up by real research, about why people act the way they do. It's been quite eye-opening. In particular, one area of research deals with how people respond to others "in uniform". His examples include people posing as security guards, priests, and businessmen in pinstripes. In every case, people are able to exert significantly more influence when the look the part. That is to say, "customers" and "suckers" are more likely to believe your message when your clothes match your role.

Back to my friends rant, I don't think that his consulting manager is out to get him four days a week by asking him to dress no jeans, no sneakers, with dress shirt. Nor is it declaring itself "Old Biz" or non-"Agile" by requesting a certain level of dress. This company understands that most people aren't so enlightened as to focus on the work you do and the value, and is playing it safe by dressing "one-level up". Add to it that most consultancies should be called what they are -- staff augmentation firms -- and you can see why consulting managers want to do everything they can to exert any influence possible on an organization.

So, do you prefer your shirts on hangers or boxed?

Good enough software

Around 2000-2001, the Pragmatic Programmers introduced me to this term. At the time, I took it to mean doing just enough design to get your feature working. I didn't really take to heart the longer-term trade-offs that lingered around, including technical debt.

Fast-forward to the spring of 2010...I've run across some great articles by Ron Jeffries and Josh Kerievsky addressing this topic, and framing it against product value and productivity.

Take-aways:

• Extremely poor quality slows us down. Striving for perfection also slows us down. We need to find the sweet-spot somewhere short of perfection, where we are not slowed by neither defects nor perfection.
• We need to design towards perfection on the more important, more valuable parts of the system. Here, value is what the customer perceives, or areas of code where we spend much of our time and cannot afford slow development.
• We should back away from perfection, and spend less time designing the areas of the software that are not as critical.
• These decisions should be based on the value of the features we are developing, and on how much the design gets in the way of future work.

There is alot of subtlety here, and getting good at this comes with mistakes and experience.

Unit testing and your level of focus

I ran across this little gem recently. It's about 40 minutes long, and full of great insights into unit testing.

To unjustly summarize, there are several points along the spectrum of unit testing:

• No testing
• Manual testing
• Test-after unit testing
• Test-first unit testing
• TDD Test-driven-development
• TDD Test-driven-design

The sweet spot seems to be in going back and forth between TDD and TDD. However, it takes a good amount of time and effort to be able to recognize where you are on this spectrum, or that there even is a spectrum of testing.

Test-driven-design is a tool to help you work out the classes involved in your solution, and how they will work together. This can be likened to the old school technique of designing your interface.

Test-driven-development is a tool to bullet-proof your code; to test the edge-cases; to test for boundary values; and generally to make sure that your code works in as many situations as possible. This can be likened to the old school technique of fleshing out your implementation.

It takes great skill to work back and forth between these two flavors of of TDD. Yet by getting to that point, in addition to a solid design and a solid implementation, you'll have the safety net in place to be aggressive, make changes, and respond to your clients needs.

You may even quickly deliver valuable features. What could be better than that?

Focus

I'm working on an application that plays video, and displays some widgets and graphics on top of that, depending on what actions the user makes. Some of the widgets have a degree of transparency to them, others do not, and cover up the entire area of video behind them.

I was working on getting the widgets to display the right information, and didn't pay as much attention to how each layer was interacting with those below. When I demoed my work to my colleagues, one immediately noticed that when the widgets overlapped, the transparency of the top was was being passed on to the one below, making the overlapping areas transparent and displaying the video below.

I was focused on displaying the correct information, and didn't even notice the transparency problem. My colleague, a graphics expert, immediately noticed the problem, and started thinking of ways to solve the issue. That's when our other colleague, also a graphics expert, chimed in with a very pragmatic idea:

Let's move the widgets around, and put off fixing the problem until we have to.

Takeaways: While focusing on a specific task, continue to keep the entire application (project) in mind.

1. In my case, while I was working on presenting the correct data, I should have paid attention to other details in the application.
2. As graphics experts, my colleagues could have jumped at the chance to fix the transparency problem. Instead, they stepped back to consider this problem in regards to other priorities within our project, and decided to delay the fix.

Software Engineering?

Tom DeMarco is "gradually coming to the conclusion that software engineering is an idea whose time has come and gone." He says so here. He concludes:

Consistency and predictability are still desirable, but they haven’t ever been the most important things. For the past 40 years, for example, we’ve tortured ourselves over our inability to finish a software project on time and on budget. But as I hinted earlier, this never should have been the supreme goal. The more important goal is transformation, creating software that changes the world or that transforms a company or how it does business.

Go out and build great software. You should take pride in doing it well -- so that others can enjoy reading and maintaining your code. But your most important focus should be on building great software for your users. Focus on rocking their world. Change the way they do things, and change it significantly.

What Do We Know about Agile Software Development?

Dybå, Tore and Dingsøyr, T. "What Do We Know about Agile Software Development?" IEEE Software, vol. 26, no. 5, 2009, pp. 6-9.

The authors surveyed the academic literature, to summarize what is known (academically, scientifically) about agile software development. Based on my experience in the field, there seems to be a disconnect between academic research and practices in the trenches. Below are some of the findings from this research, and my responses.

Some studies saw pair programming as inefficient, and some studies claimed that XP works best with experienced development teams. One study reported the further limitation, which the literature repeatedly mentions, of lack of attention to design and architectural issues.

Any development method or development process will work best with experienced development teams, particularly when compared against itself. What I have seen is that XP is great at taking a mixed team, and improving the abilities of everyone on the team. The practices of open communication and collaboration, collective code ownership, pair programming and emergent architecture all contribute. As the project progresses, each team member, no matter how junior, will have worked on many aspects of the system, each full of opportunities for learning.

Using a prescriptive development process could still lead to a lacking design and a lacking architecture. XP provides practices of refactoring, collective code ownership, test driven development and emerging architecture. These practices together allow the team to constantly improve the design, while building out only as much architecture as necessary. Diligence and attention to detail can be used or abused in any development process.

However, the on-site customer’s role can be stressful and unsustainable for long periods.

There is more than enough here for another full post. Without an on-site customer, we can regress to having long feedback cycles, waiting for emails or phone calls to be answered. We may even regress back to writing a requirements document, usually by a business analyst, and throwing it over the wall to development. The document is then ripe for mis-interpretation and other abuses by the readers. To have the right software developed, the right person needs to be fully involved and committed.

The findings regarding pair programming’s effectiveness were mixed. Several developers regarded it as an exhausting practice because it requires heavy concentration.

Pair programming provides many benefits: automatic code review, information exchange, focus, efficiency, learning. XP has another practice, sustainable pace, which offsets the intensity. The authors even note in their findings that agile team members believe they are more efficient.

One limitation that came up was that team members are less interchangeable in agile teams, which has consequences for how projects are managed.

By pairing and through collective code ownership, the team is spreading the knowledge around, so that more and more of the team members can work on each part of the system. Spreading this knowledge decreases your dependency on a particular developer for a particular area of the code.

Pair programming also makes it much smoother to introduce new developers to the team. The new developer will always be working with someone who is familiar with the code base, the design, and the entire project. This makes it easier to bring on new team members.

If the point of interchangeable developers is to maximize resource utilization, this seems like a case of sub-optimizing. The goal of a project manager should not be to make sure that each resource is 100% efficient, but instead that the team (of people) is meeting its goal: delivering the right software.

Our review identified 1,996 studies from literature searches, 36 of which we found to be research studies of acceptable rigor, credibility, and relevance to include in the review.

I may be biased by successful agile projects in which I have participated over the last ten years. I may also be biased by all of the pairs I have worked with, who taught me new things and made each task much more fun.

Improving your team's code

Peer code reviews are a great way to catch software defects early and often. This practice has been around since at least the mid-seventies [Fagan '76].

Extreme programming (XP) has a couple of practices built-in to support reviews. Collective Ownership encourages everyone to contribute new ideas to all segments of the project. Pair programming increases software quality without impacting time to deliver. All code to be sent into production is created by two people working together at a single computer.

Having experienced both traditional code reviews, and full-blown pair-programming, I believe pairing to be a very effective and efficient step towards good, clean code.

Yet, I've had team-mates that didn't want to be part of the process, and I've had team leaders who didn't know how to constructively make it work. I have come across some good practices for peer code review to pass on to them. Even though this is written from the point of view of traditional code reviews, I think that the principles can be applied to Agile environments as well. Below are some excerpts I particularly enjoyed:

8. Managers must foster a good code review culture in which finding defects is viewed positively.

Code review can do more for true team building than almost any other technique we’ve seen – but only if managers promote it at a means for learning, growing, and communication. It’s easy to see defects as a bad thing – after all they are mistakes in the code – but fostering a negative attitude towards defects found can sour a whole team, not to mention sabotage the bug-finding process.

9. Beware the “Big Brother” effect.

Managers must continue to foster the idea that finding defects is good, not evil, and that defect density is not correlated with developer ability. Remember to make sure it’s clear to the team that defects, particularly the number of defects introduced by a team member, shouldn’t be shunned and will never be used for performance evaluations.

10. The Ego Effect: Do at least some code review, even if you don’t have time to review it all.

The “Ego Effect” drives developers to write better code because they know that others will be looking at their code and their metrics. And no one wants to be known as the guy who makes all those junior-level mistakes. The Ego Effect drives developers to review their own work carefully before passing it on to others.

Managing ideas, ala Weinberg

This is a list of ways to get ideas flowing when trying to solve a problem, particularly in a team, based on Jerry Weinberg's Becoming a Technical Leader. Each of his ideas is great on its own, and they work well together, but there are also subtleties that you should be aware of.

Contribute a clever idea to the team. When it works, and when it's really novel, this is really cool. Most of the time, there are no new ideas, but re-hashed ideas from other sources. Instead of trying to get the perfect solution right away, spend your energy creating an environment where ideas can be exchanged and the "right" ideas can be recognized when they come along.

Encourage copying of useful ideas. Look for inspiration in your own branch, or in another knowledge area as well. And remember problems that you've already solved, in case this problem is similar.

Elaborate on an idea that a teammate contributed. Iterate, and make it better. Keep following through. Follow up the inspiration with enough perspiration to see it through.

Drop one's own idea in favor of an idea the team wants to develop. Strike a balance between stubbornly holding on to your ideas, and letting go of your ideas just for the sake of keeping the group happy and moving along.

Refuse to let an idea drop until everyone understands it. Like the previous example; this will come with time and experience.

Resist time pressure, and take the time to listen when other people explain their ideas. Take extra care to know why an idea is being dropped.

Test ideas contributed by other people.Look for the initial idea anywhere and everywhere, and then make it better.

Withhold quick criticism of teammates' ideas, in order to keep the ideas flowing.This is particularly important at the beginning, such as in a brainstorming session. Make the tradeoff to accept more ideas, which will be dropped later, than to drop early and not generate enough ideas. This improves with experience.

When you must criticize an idea, make clear that you are criticizing the idea, not the person who offered the idea.In addition to maintaining a good relationship with your teammates, this let's you explain which parts of the idea work, and which ones don't. It also let's your teammates in on how you come to your conclusions.

Test your own ideas before offering them.Take the extra time to scrutinize your ideas the same way you would the ideas of your teammates.

When time and labor are running short, stop working on new ideas and just pitch in.

Encourage the team to drop ideas that had succeeded earlier, but cannot be extended to the new situation. Make sure that the ideas fit in with what you are trying to accomplish.

Revive a dropped idea later, when it has value for another part of the problem. Remember your old ideas and solutions, because they may become useful again.