User Experience in Agile

User Experience (UE) is probably one of the least understood fields in todays software creation community. Yet it is arguably the most important thing we humans look for in an application. One of the companies that understands UE is Google. That’s one of the reasons they are building a religious user base. Truly yours is a fan of their applications including Reader and Maps and although they seem to have blooped with Desktop Search, they are still topping my and many other users’ most likable applications lists. In fact it is not a coincidence that many of us have come to expect the next cool technology or cool app to come from no one else, but Google. So, how does Google do its magic? By understanding human behavior and creating human-friendly applications.

Agile methods are particularly well suited for applying good UE principle in your application and achieving Google-like UE designs. I’d like to list ten practical recommendations from my personal experiences that could help better understand where UE fits and how it can be used to add value to the application and make its users feel good (…yes, I feel good every time I use Reader).

Hire the right people

Start with recruiting the right talent for the job. Do not underestimate this step, it is both difficult to locate good people in this fields and also to tell the good from the bad. Look for people with social sciences backgrounds and experience working in fields such as cognitive psychology, interaction design, information systems design, semantic user interface design, context-based interactions. Shy away from engineers and graphic designers [ok, this is sooooo general, that perhaps is worthless] because we tend to be too analytical, systematic and predictable [read:boring]

Make the UE part of the Onsite Customer

Because UE is such an important task it goes hand in hand with the business requirements. Make the Product Manager (PM) and the UE expert share the responsibility of getting the right requirements to the User Interface (UI) team, the business analysts and the programmers. Let the UE participate in the story prioritization and iteration-end demos. Take their feedback seriously and treat is a real business requirement, equal to the ones coming from the PM.

Plan for UE

If you want users to love your software then take UE seriously. Plan for it. Allocate a few points each iteration for UE. Start with the lowest possible unit (point) and find-tune as you go. If your app is UE-centric, you may find yourself spending up to 10% of each iteration on UE. That’s quite normal and it shows you are dedicating enough time for the cool factor, besides the useful stuff. You will find yourself spending those points on clarifications, UE acceptance testing and reworking the UI based on valuable feedback from UE.

Automate the UE testing

UE testing is the activity of making sure that the users have a pleasant time using your application. So if humans and pleasure is involved how can an automated test help? Easy. Concentrate on parameters such as how much time it takes for a particular screen to render and capture that data. Fail the test if it is more than a threshold representing the “patience coefficient” of a user. Look for misaligned elements in your screen and fail the test when they happen. A common failure in AJAX-based systems is the connection and when that happens parts of the screen remain blank for a certain period of time. Try to automate as much as you can using tools such as Selenium and Watir. Make the automated UE testing part of your continuous integration loop and raise hell when it breaks just as you’d do if my JSP fails to compile.

Internal UE releases

When does UE start and end? It does not. It’s there all the time, from beginning until the end. One of the advantages of the Agile methods is that working software is produced at the end of each iteration, so make it a habit to release a working copy to the UE team to play with while the programmers are working on the next set of stories. This will give the UE team a chance to test the application in a more relaxed environment and therefore they can spend more time evaluating its usefulness than the limited demo at the end of each iteration.

Keep UE independent

They are the people who should be able to design and evaluate the software as an outsider, so keep the UE outside of the core team. If this means exclusion from the common team area, do it. Let them stay independent and give them enough time and space to evaluate your interim product on a regular basis. This independence will eventually manifest itself as great features, likable user experience and fresh new ideas. You can occasionally invite them to your parties, that is OK; we all need well-dressed people around us khaki-knowing-and-loving engineers.

UE Acceptance Testing

Make the UE experts the ones who decide whether your iteration was successful in the UE department or you missed the mark. Invite them to the iteration-end demo and let them evaluate the software based on independent evaluation criteria. Do not postpone UE rework as a result of the feedbacks and if possible tackle failed features in the next immediate iteration. Do not allow for the accumulation of UE debt.

UE mockups are not UI mockups

Make sure you and your team understand that what UE creates as UE mockups (wireframes) are not screenshots the future application. That is the UI team’s responsibility which starts by them getting the UE mock and working with the PM and UE to create a UI mockup. UE is said to create low-fidelity wireframes, sometimes drawn by hand, and the UI converts them into high-fidelity screen mockups. This is an important distinction as many team members seem confused about the various artifacts produced. Think of the UE mocks as the principles and the UI mockups as the prototypes that implement those principles.

UE and Architecture go hand in hand

UE works closely with the Agile architect. What will eventually become the domain model starts as an idea in the UE’s brains. UE and Architects should share brainstorming sessions and use whiteboard design, and loose notation [read:non-UML] to capture their artifacts.

Get creative

Just because you have not heard about it does not mean it’s not possible. Get creative and use other disciplines to interact with UE and allow them better means to share their non-technical ideas [and passions too]. For example import the concept of “storyboards” from the movie industry. Treat each screen of your software as a scene from a movie. Sketch your players, mark your cameras and lights and shout “Action”. See what happens. Map the paths a user can traverse to get in and out of the particular screen and try to imagine how he/she feels.

UML has been particularly difficult for non-engineers to understand. Use other, less-formal means to communicate with the creatives of the team. For example use mind maps whenever possible to draft user paths and behaviors. Use the mind maps to work on a domain model. The relaxed notation of the mind map makes it the ideal choice to put UE, PM, business analysts and architects on the same drawing board.

Agile Monkeys Gone Wild

One of my favorite readings on Agile is Barry Boehm and Richard Turner’s now classic work “Balancing Agility and Discipline” [1]. In fact I made it a habit to revisit the book every once in a while. The prelude section features an interesting story about the elephant and the monkey. The story has a happy ending where the trusted elephant and the agile monkey join forces to serve their villages better. That’s really cool I say, but what if the monkey does not want to play? In other words, what if your village’s agile monkey wants to stay on the wild side and continue doing what iit is doing? What can your village do?

Extreme Agilists can present a real risk to any company developing software in an Agile manner. Working with people set on a particular model and unwilling to bend for the greater good of the whole could be particularly challenging for teams in the process of transitioning to an Agile method.

From my experience there are at least two major drivers for an extreme Agilist behavior. The first is the failure to link technical and methodology objectives to the business objectives of the company. This scenario usually happens when the Agilist takes the team into a pre-cooked method or solution with little regard to the real objectives of the company, such as markets, competitors, product positioning and overall strategy.

The second and perhaps more important driver is the over-emphasis on Agile methods rather than fundamental software engineering practices. A typical example of the latter is the way upfront design and architecture is handled in certain Agile methods. It is easy to fall into the trap of assuming that very little or no upfront design is going to work, just because of an Agile method. The reality is that in most cases the architecture does not evolve on its own and it needs a certain babysitting from the architects and the team. There needs to be balance of upfront architecture plus evolutionary design to achieve a middle way of delivering solid architecture in the final product.

So, what can the village do if they have to deal with a rogue agile monkey? They can start by talking about the real purpose of the project and the team’s existence. They can make it clear that most companies in today’s world cannot afford to subscribe solely to any particular platform, technology, or methodology. All of these are tools in the company’s disposal to achieve its final objective. Companies today pick their tools based on their short and long term business vision and the toolset should reflect that.

Finally if nothing else works I recommend the village elders get together with the monkeys and tell them the story of Babel. Per Frederick P. Brooks Jr. [2] despite that the project had a clear vision, plenty of manpower, adequate technology, lots of materials and enough time, it still failed. The project was a big initiative for its time. In fact, according to the Genesis account, the tower of Babel was man’s second major engineering undertaking, after Noah’s ark. But Babel was the first engineering failure.

Babel failed in two respects—communication and organization. The man were unable to talk with each other; hence they could not coordinate. The lack of communication led to disputes, bad feelings, and group jealousies. Shortly the clans began to move apart, preferring isolation to wrangling.

The village elders should talk to the agile monkeys and try to persuade them to play with the elephant. The alternative would be isolation and that could ultimately lead to the failure of the project.

References:

[1] Balancing Agility and Discipline: A Guide for the Perplexed, Barry Boehm, Richard Turner - http://www.amazon.com/Balancing-Agility-Discipline-Guide-Perplexed/dp/0321186125

[2] Mythical Man-Month, The: Essays on Software Engineering, Anniversary Edition, Frederick P. Brooks Jr. - http://www.amazon.com/Mythical-Man-Month-Software-Engineering-Anniversary/dp/0201835959

Continuous Refactoring and ROI

“So, just to make sure I understand correctly, you’re saying you would like the entire development team for one month to work on cleaning technical issues and this has nothing to do with our latest set of requirements. So, we are not adding new features…, right?” asked the VP, sinking in the sea of technical people surrounding him. “Yes, that’s our plan…“, responded the Director of Development. “…I consulted with my team leads and we are all in agreement that adding more features to the system without cleaning the technical debt will contribute to the mounting cost of fixing defects and changing various parts of the system. We as a team feel embarrassed to say to customers that even some small changes will take weeks.” “Show me the ROI!” impatiently shouted the VP of Engineering, “…that’s the question I’ll get from upstairs. So what’s our answer?”. “If we are asking to spend a whole month allocating the entire team to cleaning this technical debt, then we’d better have the right answer to these two questions: first, why is this much debt accumulated anyway - why can’t we get it right the first time? And then, let’s assume I am able to buy you this time, is this going to solve all of our issues once and for all?“

This dialog is all too common in software development teams today. It reflects a feature-driven brainset where product management is anxious to get as many new features to the market as soon as [mostly] humanly possible. In this marathon of features, usually no time is allocated-or-left for architecting a proper solution. Similarly, after a release gets out no time is spared to clean-up leftovers.

Company executives fear of refactoring, because they think it has no positive effect on the marketability of the product. But it hurts the bottom line and so they develop severe fear for changes to the software that have no proven ROI, which I’ll simply call refactophobia [define: obsessive, persistent, unrealistic fear of an external object or situation, in this case of changes to a software system with no prospects of increase in revenue]. They like to think of it as a frivolous luxury; something the tech guys enjoy doing every once in a while. Plus, there are no new features and since customers do not pay for a beautiful code, what’s the point?

I’ll try to explain the point. Agile development promotes the practice of continuous refactoring, a practice which plans small refactoring into each iteration and therefore does not allow for the accumulation of any technical debt. There are significant cost savings to the teams following this practice. Let’s compare this approach to other alternatives.

Figure 1 - Software Products: Cost of Change vs. Time

Let’s take a look at Figure 1. The black line in the chart represents a traditional software product lifecycle. Complexity and code smells (badly designed or coded code) mounts as the product matures until it reaches a point of infeasibility. There are two major factors making the product infeasible: cost of change and cost of adding new features and cost of fixing defects. In fact, inability to maintain their product is one of the major reasons for many software companies’ demise. Linked to this is the falling behind competitors due to the lack of new features. The total cost of this strategy can be thought of as the area under the black line in the chart.

The pink line is a representative of the above conversation. Some companies manage to squeeze once-a-year technical debt refactoring marathons which are aimed to reduce complexity and clean-up the code. These marathons are costly, but effective. They solve pressing issues and let the team take a breath until the next string of features. This strategy, while better than no refactoring, is still costly as it lets the smells live too long in the system and mutate into more complex, nastier smells. The area under the pink line in the chart is smaller in size than the black line which is an indication of the cost savings of this method.

The last and most effective way of dealing with accumulated complexity in software products is continuous refactoring. Agile teams which practice short iterations are familiar with the concept of Iterative Incremental Design (IID). Another important practice is continuous refactoring, aka. fearless refactoring, aka. courageous refactoring. In the latter, teams plan ahead for the refactoring which will take place in each iteration by adding a few extra points for that iteration. In my experience the ratio of refactoring points to the whole has been in the 5-10% range. In reality it always goes the opposite to the planned, however the agile project plans have the magic of evening out after a certain period of time, so starting with the 5-10% figure may not be a bad idea. In the chart, continuous refactoring is represented by the green line, where as a result of small and incremental changes to the internals of the code with no effect on its external behaviors, the cost of change is kept under control.

“…so this chart looks sleek, but how am I going to explain that to the people in the board room?” …the confused VP was still mumbling …Well, there is not much any developer can do to fix that, except maybe reminding the managers of the simple values that most Agile developers are driven by: trust, courage, empowerment, accountability, respect, communication and simplicity.

Code Quality in Eclipse

Paul Duvall published a new article in developerWorks where he talks about a set of Eclipse plugins focussed on helping developers in key areas such as coding standards, code duplication, code coverage, dependency analysis and complexity monitoring. He outlines steps to download and install the plugins and provides visual step-by-step instructions on how to use them.

Paul writes, “Using Eclipse plugins shouldn’t preclude you from using these inspection tools as part of the build process. In fact, you want to ensure that the rules followed using the Eclipse plugins are the same rules applied to your build process.” I would add to this that each Eclipse project file should be checked-in to your code repository and special care should be taken to make the project file machine and platform independent. The classpath in the project should reflect the same set of JARs as the ones used by your build system. This is not that difficult to maintain, but it is crucial that each developer uses both Eclipse and the build system before checking-in his/her changes.

One more note. In addition to the plugins outlined in Paul’s article there are a few more interesting plugins that can help you get a different view of your code. One such plugins is Simian. You can follow the following steps to download and install Simian:

1. From the Help menu in Eclipse, select Software Updates | Find and Install.
2. Select Search for new features to install and click Next
3. Click New Remote Site. In the dialog enter the name Integility and the URL http://www.integility.com/eclipse then click OK
4. Ensure that the Integility entry in the list is ticked then click Next.
5. Select the latest version of the Simian UI feature and click Next.
6. Review the license agreement and (if you agree to) accept it and click Next.
7. Choose the install location, this will usually be your main Eclipse directory. Click Finish.
8. After downloading, Eclipse will show a warning about unsigned code. Click Install if you wish to proceed with installation.
9. Restart the workbench when prompted.

Another interesting plugin is the Code Analysis Plugin (CAP). It features:

On the Way to Agile Transparency: Climbing the Big Wall

You may have heard of Reinhold Messner. He is one of the world’s top climbers and he did not gain fame due to his personality (some argue he has lost his mind) or his PR skills, but because of his improvised, one-of-a-kind climbing style. He is the mountaineer who developed the agile climbing model. Ascend solo. Use no oxygen. Improvise routes. He wasn’t modest about his achievement: “As far as the public is concerned, since 1978, my sensational climbs - Everest without oxygen and Nanga Parbat solo - are unsurpassed” (1). Indeed. Agility is a good way to get there from here: whether you are climbing a mountain, or building software.

Agile development methodologies mean a lot of things to a lot of people these days, and most of these have to do with making life easier: as a business partner, you may want faster time to value and lower development costs. As a Director of Software Development, you may dream of a people-centric model, minimal methods and maximum collaboration (2).

Transparency is a major dynamic associated with agile development. At the roots of the people-centric model advocated by agile development is a philosophy of collaborative, non-punitive accountability that can be defined as transparency. Thanks to their social methodology, agile projects offer better transparency to clients, business partners and corporate decision makers. When broken down, this concept consists of management components such as individual responsibility, commitment, and accountability. Peter Drucker’s definition of individual responsibility in his book “A Functioning Society” aligns smoothly with the transparency model agile supporters believe in: “Responsibility is both external and internal. Externally it implies accountability to some person or body and accountability for specific performance. Internally it implies commitment. The Responsible Worker is not only a worker who is accountable for specific results but also who has authority to do whatever is necessary to produce these results and who, finally, is committed to these results as a personal achievement.” To paraphrase, Agile methods advocate transparency, not to be confused with lack of accountability. As Stephan Haeckel (1999), director of strategic studies at IBM’s Advanced Business Institute, has observed, “Organizational responsiveness comes from giving individuals and groups the freedom to behave in ad hoc ways to respond to unforeseen circumstances. For this reason, organizational roles are defined in terms of accountability for commitments to particular outcomes, rather than in terms of activities.” If we are going to build a hub organizational structure and manage outcomes rather than activities, then there needs to be a mechanism for teams within the project structure to manage their commitments to each other. Rather than have the project manager keep up with and manage all inter-team dependencies, the teams need to accept that responsibility themselves, just as development teams commit to customer teams on features.” Team leads should continue to exercise traditional management, leadership and coaching practices and perform reviews based on each employee’s contributions and performance. In the lack of such understanding and team protection an environment of fear and witch hunting may become unavoidable, and that’s a sure way to kill any well-intended process.

An organizational philosophy of transparency, when correctly implemented, can make an impact. Recently a friend shared with me a neat little solution they’ve developed to track Agile projects – they call it the Agile Project Management Dashboard. The system in place is ingeniously simple yet very powerful. Each developer, analyst or tester involved with a particular project is asked to enter the status of their tasks to a shared spreadsheet located on a network drive. Later on a Share Point process picks up the updated files and generates a neat project dashboard bundling important information such as:

• Color coded project health status
• Iteration and release estimates and current percent done
• Percentage deviations from original estimates
• Completed and remaining stories

Acknowledging that one of the key concepts of agile methods is transparency, my friend set a system where the root cause of every problem is exposed early and decision makers get a sense of the life cycle of each project an any given instant. In highly collaborative and transparent teams like this, it is important to take the time to educate top management about the implications of constant visibility. In the case of the dashboard, it is paramount to convey the right message to upper management: namely, that this dashboard is an internal status tracking tool and nothing more. Many organization fall into the trap of attaching accountability to simple tracking tools such as this one, and hence risk reducing the level of collaboration and individual responsibility. Some places go even further to tie employee performance reviews based on some metrics generated from transparent tools such as this. In an open dialog environment such as an Agile project, top management should be well aware that increased transparency should not come at the expense of holding individual team members accountable of things they may have little control over. Honesty seems to go a long way of establishing credibility and contributing to the transparency of any team. This was emphasized by my friend and also once expressed by management guru Jack Welch. Welch asserts that “…leaders establish trust with candor, transparency and credit. When leaders do not do this, they foster an environment of suspicion. No one really knows what is happening. As a result, the network of people within the organization begin to work at cross purposes from one another. It is important that critical candor be constructive. That it is not just to put someone in their place, but provide them a way to move to a different place. When you have that kind of communication stream going on, the sharing of credit becomes much more motivational. Honesty begets motivation to improve and perform at your best. The leader though must model this genuinely for it to be effective.”

It may be argued that agile is to software development what globalization is to Ford and GM – agile development promises to shrink traditional boundaries by advocating a social, people-centric model that derives its power from commitment and dedication. It seeks to enhance productivity by introducing faces into what used to be an anynomous code production chain. Transparency and trust coupled with governance and accountability increase visibility and encourage collaboration. When special care is taken to balance the newly increased visibility with non-punitive accountability, great results will be achieved. In the end, anyone can climb that big wall, if they chose the path of trust and transparency.

(1) Retrieved from: http://www.jerberyd.com/climbing/climbers/messner/. Messner is also cited at The Agile Web Design Manifesto, An Introduction, by Emily Chang and Max Kiesler.
(2) A definition of agile development can be found here