Project and Technology Development Methodologies

Rather than be the trillionth blogger writing on the subject of Agile, refactoring, Test-Driven-Development, domain-driven-design, and other practices and paradigms, I'll lay out some pragmatic principles to keep in mind when executing projects, no matter which development practices you adopt. In the rush to absorb the hottest trends, we should keep in mind the tried-and-true principles that still work:

Start with iterative and incremental. Project phases are well and good, but in order to complete a project successfully, the path to completion needs to be traversed. And yes, you must have an idea of your destination. But will you wait until all requirements are specified to the last detail before moving on to any architecture or design concerns? Will you have a design project phase that takes into account the current state but fails to connect with the changes in the world three months later when the design phase is completed?

If you break down your project phases into iterations, with a guaranteed feedback session at the end of each iteration, you will be much more likely to keep up with changing requirements and conditions. You will also be able to identify failing efforts sooner, before they cause damage later.

Start as soon as you can. One of the common reasons for project slippage is that some critical component or project dependency was not available earlier in the project cycle. Was it because the need for this component was not forseen? No, it was just not available at the appropriate time. One way to mitigate this is to start working on these components after just enough baseline requirements have been gathered. You may spend a little more up front to support this initial development, but it will cost you far more in the long run should you end up in an ill-timed slippage scenario. The ways it may cost you range from longer development cycles to mis-timed market entry to project personnel turnover. These are big-ticket costs.

Military leaders are trained to make decisions with 40% to 70% of the information necessary, and regularly provide feedback as more information is available. You can do the same on projects and be very effective.

Do your Risk Management. As stated so eloquently in the fantastic treatise on software project management Waltzing With Bears, "Risk Management is Project Management for adults." Listing and valuing project risks is something that can be done to 100% completion up front. But the great (and sometimes painfully realized) thing about Risk Management is that 100% completion is often not enough. Here is a great place to be iterative and incremental in providing feedback loops on the state of risks, mitigating circumstances, and changing requirements. If you build the iterations and feedback loops into your risk management on a project, the rest of your project practice will need to follow suit just to keep up.

You'll also be more inclined to tackle project deliverables that resolve the greatest risks first. The sooner these risks are resolved, the more accurately you can publish a date range for delivery.

Test early and continuously. Your project exists in some form from start to finish. It may start on paper, and live in hardware and software at the finish. But in whatever form it exists, it can be tested. As new requirements are gathered and formulated, test cases can be drafted and pitted against the project's assumptions. As new hardware is installed, its images and monitoring agents can be configured and proven. As new code is written, test cases can be written before or alongside the code.

Don't wait for a project to be 50% or 75% completed to start drafting up test cases and setting up a test facility. Automate much of your tests if you can, so that they can be run at least once a day. Automation is again a case of a little more effort up front, saving much more cost and headache later.

Does Our Technology Equate To Lies?

I recently had a conversation with a CTO with experience heading large and global teams. His recent work had concentrated on installing SOA-based systems into his organizations. He brought a viewpoint of his to my attention that related custom software development to a lie. The lie that you tell yourself, he said, was that a custom or one-off module or block of code solves your business problem the way you want it to. You're lying to yourself because the problem wasn't solved in an configurable, and maintainable, and service-oriented way.

He went on to support this statement by relating a story about lining up the dates in a fiscal calendar of a system that did not lend itself easily to change. A custom solution working around the constraints of an existing system was needed to meet the demands of the business, and he was not quite satisfied that the solution had to be of a custom nature. In his view, he preferred the solution to be service-oriented and configurable.

While I understand the CTO's view towards a service-oriented and configurable architecture, I don't think that his lie is attached to the appropriate concepts. Service-oriented or not, you can effectively eliminate the lie he speaks of by managing the lifecycle of the customized solutions.

If your business is sufficiently pained by the problem where you need a customized solution implemented and deployed urgently, and the solution has been appropriately constructed and tested, then by all means deploy it. But if the custom solution is not the most optimal, configurable, or maintainable, then put an expiration date on it and schedule the time and resources to implement the optimal solution by the expiration date. This way you would provide urgently-needed relief of the pain your business is experiencing, while having an outlook toward the optimal future.

But it is crucial that the follow-up happens. The common occurrence, and the real fear here, is that once a solution is implemented and deployed, people move on to solving the next problem or working on the next great project without coming back to readdress that sub-optimal solution. This custom development often ends up being yet another buried non-catalogued nugget of logic.

In the CTO's anecdote above, the custom solution is necessary to work around the larger system's constraints. But what if this custom development was configurable, maintainable, and service-oriented from the start? It is still a separate custom component, one more component to be maintained in the catalog of assets. In maintaining a service-oriented architecture, having an up-to-date and discoverable catalog of components, the contracts they satisfy, and their development artifacts is key to performing effective maintenance. SOA implementations typically have more components than non-service solutions, not less. SOA implementations without these catalogs can quickly become burdensome and error-prone to maintain.

So I think that the real ways organizations lie about custom software development are:

- not assigning an expiration date to sub-optimal solutions, and not managing the transition process from sub-optimal to optimal

- not maintaining an effective catalog of solutions, contracts, and components

- expecting a silver bullet or all-inclusive solution to eliminate the need for custom development

- not actually solving the organization's problems because no custom (or market-leading) solution is considered to be the optimal solution for the business

Unlike the CTO's organization, I've seen organizations with very painful problems put off implementing solutions for years because the solution candidates don't fit into the "optimal" category, service-oriented or not. If the solutions are not considered to be the holy grail, then nothing will be implemented at all. And the business continues to limp along in pain without further technology assistance.

Wrapping up my conversation with the CTO, he was considering implementing one of the larger off-the-shelf tools that can effectively assist in service orientation. He left me with the impression that these solutions, their price tag, and their large deployment footprint offered him great comfort in removing the custom development lies from his organization. But alas, that is a lie for another day.