Category: Get.Rounds, Tech Talk
Leave a Comment
Our CEO makes the rounds with other CEOs, entrepreneurs and investors. One of the themes that gets batted around is the concept of Continuous Development, Continuous Integration (CDCI). At the CEO level, it is taken to mean that the company releases products more or less continuously and not on a fixed schedule every few weeks or months or even years.
In the good old days of Brook’s book “The Mythical Man Month”, software projects followed that same cycle that you would see in a major construction project. First, someone draws up a list of requirements (Make a bridge from point A to point B that can handle Y traffic per hour). An architecture creates the overall design and architecture plans. (Civil) engineers draw up a complete design on paper that is then reviewed to see if it meets the written and unwritten requirements (Don’t break the bank, Don’t make it look like a Picasa painting). The construction company reviews the plans for implementation. The project owner arranges for the right permissions from the appropriate agencies. The parts and materials are ordered. The builders build. Test and review the result. Open for traffic.
These projects follow what is called the Waterfall method. The project starts at the top and all work flows down towards the goal line. Going back to the start is expensive and is to be avoided whenever possible. For major construction projects, it works. You very rarely hear about a skyscraper that was torn down because the project failed to produce. You do hear about cost overruns, but the project somehow converges to a finished whole.
Software projects were run this way from the 1960′s through the turn of the century, and the majority of large scale projects failed to deliver. There have been many attempts to figure out why software development is not like building a bridge. My personal belief is that requirements are too easy to change and that developers overbuild. Let’s explore each of these points in turn.
The worst thing you can do as a customer is to order a custom project and then change your mind during the process. Each change request means that the process “should” be restarted from the beginning, but in practice, no-one is willing to do that. Instead, they try to shoehorn the changes into the existing design, with the result that the project becomes more and more inconsistent. The software customer has been taught that software is easy to change. In the customer’s view, “minor” additions or subtractions should not affect the price or the schedule.
At the same time, developers don’t build what they need, they build to idealized standards of completeness. For example, in object oriented programming, we can produce an object that represents time. In a complete, idealized world, this method would have hundreds of methods, ranging from manipulating time objects (add, subtracts, get_hour, etc.) to handling daylight savings time, to converting to sunrise and sunset values and to calculating the next coming of the messiah. Ok, so we don’t usually go that far, but in many cases, developers build too much. Every method needs to be tested and verified. Every piece of code should be reviewed for correctness, performance, readability, maintainability. The original object went from a simple “represent the hours in a day” to a monster with thousands of lines of code and inherent complexity, few of which will be used in the actual product.
There have always been skunk works projects in software development where a small group of star programmers create a miracle seemingly overnight. They start with an idea and together, create a prototype or working model within a few days or weeks. I’ve been a party to a few of these projects and they are uniformly fun and exciting. Each member contributes their skills to the pot and without any manager telling them what to do, the project converges to some result.
In the past ten years, the software development community has been trying to convert the skunkworks approach to larger, more formal projects. The names have changed, from Extreme programming to Agile programming to Scrum, CDCI and other terms, but the basic concept remains the same. First, don’t’ try to build the entire project on day one. Aim for small incremental sub-projects that can be completed in a short time span. Verify that each sub-project performs as expected. That does not mean that each sub-project is complete, only that it does what the developers claimed that it would do. Roll out each sub-project as soon as it is complete and move on to the next sub-project.
In this approach, each “waterfall” lasts for one or two weeks. Changes are expected and accepted. Instead of building a complete architecture, design or model, we start with something “close” and incrementally improve it. Occasionally we get it wrong and then we need to throw away some work and start again. The hope is that we recognize these problems long before we have invested too much into them.
This approach seems to work in software development, but it would fail miserably in civil engineering projects. Part of the problem and the opportunity is that in civil engineering all of the building blocks are well known and construction techniques have been proven time and time again. In software development, we have not yet settled on a well known set of modules and components that can be reused time and again. There is improvement, but the field is just not as mature as any other engineering field.
At Rounds, we started with a waterfall method. When I arrived, the last release took two months to complete. The system was rolled out as a monolith, everything was deployed at once and all of the versions were intertwined, tested together, released together. It took some time, but we broke the monolith. We now have almost twenty separate projects, any of which can be modified, tested and installed at a moments notice. This allows us to have some people focusing on short term efforts, fixing user interface issues, or adding a new feature, while other developers are working on longer term projects that may take up to a month or more to complete. Each effort moves at its own pace, but the product incrementally improves each week, sometimes each day.
In a startup, it is more important to try something than to wait until its perfect. We can rarely predict actual customer behavior. Only by watching what they actually do can we learn what they want and how to provide it. Frequently minor releases give us that flexibility to try new things. In the worst case, we make a mistake (which does happen). Because the changes are small, we just unroll that change and start again from a previous state.
Projects that run this way put much more responsibility on their developers. No longer do we have a long, drawn out testing and release cycle to catch mistakes. Each and every developer can roll out changes on a daily basis and must be responsible in case those changes screw up the system. I’m proud of our team at Rounds. They understood this model and stepped up their quality. We do occasionally rollback a change because it was just not ready yet, but in more than 90% of the cases, our releases go smoothly, frequently without an affect on our customers. What more could we ask for!