Cynefin's Complex Context

A real life example

In prior articles, I described Cynefin's Obvious and Complicated contexts by providing real life examples. In this article I will continue that story and introduce the Complex context.

As the number of PCs in the firm grew, the cost to license our menu system was becoming excessive. We were planning to transition to Microsoft Windows, but had at least a year before we'd begin the project and possibly several years before it was complete. Many of the computers in the firm were not capable of running Windows and the cost to upgrade them all was additionally prohibitive.

I'd been programming in Assembler, writing low-level disk utilities and other tools to help us with PC diagnosis and repair. I'd recently written a program to provide simple ascii based menus and pop-up windows to improve the interface of some of the more involved batch files we were running on PCs. I decided it would not be too much of a stretch to create an actual menu system and replace the costly one we were using. It took a bit of time and I learned a good deal as I went along, but within a few weeks, I'd coded a complete replacement for the existing menu system. Within another couple of weeks, I'd added features not previously available such as the ability to easily add new items and sub-menus, some basic security settings, and the ability to upgrade the system via a network file copy while maintaining the local settings.

The new menu system was a hit. Not only was it better than our previous system, but it allowed us to opt-out of a $50k renewal with our current provider. This was a savings more than twice my salary (remember, this was the early 90s and I was still officially a junior PC repair technician).

With the launch of the menu system, I was promoted to junior PC programmer and moved off of support desk.

I was assigned to a small team responsible for the re-write of the company's legal docketing system. We were in a similar, but more dire situation here. The Docketing software was not only proprietary, but it ran on proprietary hardware. The company that originally created it had folded. We'd contracted with a third party for hardware support, but the software had been out of support for years and we were hitting the limits of the system's capabilities. As we grew, the number of cases handled by the firm grew drastically and the docketing system was straining under the load. Searches were taking far too long. Simple data entry was taking twice as long as it used to. The docketing department had to hire extra help to get through the load, which meant more people doing data entry at the same time, slowing the system even more. We estimated we had less than one year to completely replace the system. The replacement needed to provide all the same functionality along with several new features. At the same time, we had to ensure we didn't lose any of the historical data.

We spent weeks working with the docketing team. We observed their work, taking copious notes. We discussed workflows with them, talking at length about how the existing system could be improved. It soon became obvious that we were not re-writing the existing system. We were writing a new docketing system. One that satisfied the needs of a firm our size and was up to date with the latest capabilities for electronic filing and email integration. They had been suffering with the old system for so long, they had a litany of new features and improvements that ranged from items they thought would be nice to have to items that were absolutely critical.

Probe, Sense, Respond

Joe, my new manager and lead of the development team for the docketing system, announced that we were going to work on the application in small deliverable pieces, starting with basic functionality and adjusting as we went along.

On one hand, this made sense to me, as it was pretty much how I wrote everything. When I was a kid working with friends on Asteroids in TR-BASIC, we started off with just a space ship that could spin around and shoot. It could not fly. The bullets did nothing. But we'd play with that ship anyway. Trying every possible move we could think of. While it couldn't fly, we learned that we needed to code some kind of transition when the direction of rotation changed. Say the ship was moving clockwise and we pressed and held the left arrow, indicating it should move counter-clockwise. An abrupt change in direction felt unnatural. This wasn't how it would work in the "real world". Once we introduced forward motion and acceleration, we learned that we needed to put a limit to the ships speed or we couldn't control it after a certain point. We learned that the speed of the bullet needed to be some constant plus the speed of the ship or a bullet could be overtaken by the ship that just fired it. We then added asteroids, which were squares that moved in straight lines. We then added the ability to detect when the ship was hit; first by asteroids and then by its own bullets. With asteroids in the way and bullets flying around, we made adjustments to the steering to allow for better control and we worked out how the ship behaved with and without thrusters turned on. We kept building one small piece after another until we had a pretty good version of the game, complete with flying ship, bullets, and breaking asteroids.

On the other hand, part of a program is no good to anybody, right? It is an all or nothing proposition. What good is a half finished program? We're not kids writing silly video games. This is real software for real adults in a real business context. As another developer on the team used to say, "You don't want to show them the sausage until it is completely ready. Nobody wants to see how their sausage is made." I mean, I didn't give anyone the menu system until it was complete. How could we expect them to use a docketing system that didn't do everything they needed?

Outcomes cannot be predicted

Joe assured me that small incremental steps was the right way to go about this. "Were not going to cut over until they say it is ready, but if we build an application this big without their feedback while we go, we'll get it wrong. We need to let them see it now and then. We need to let them kick the tires and take it for a test drive. Only then can they tell us what they really want."

I was confused. "What they really want? How could they not know?", I asked. "I mean come on... they have a whole list of features and requirements they gave us on top of all the stuff the original system does. I just don't see how showing them a half-done system is going to help. I mean, we can build it in pieces, sure. But they don't need to see it until we're done."

"No.", Joe said sternly, letting me know the discussion was ending here. "We're going to show it to them as we go. They don't know what they want and neither do we. They think they do, but all they know is the solution they have and the features they wish it had. When we show them what is possible, they'll start to think about it differently. And so will we. Trust me."

"I want you to think about how you actually wrote and delivered the menus.", Joe said. "You started with the basics. You had the abiity to navigate a single menu before you introduced sub-menus. You had sub-menus before you added the ability to edit menus. You worked on it in pieces all along, right?"

"Well sure", I replied, "but I didn't deliver it until it was done."

"Oh really?", said Joe with a turned up brow, "Didn't you just add a new security feature last week? In fact, didn't you write it so that you could easily update it on people's machines? Why would you do that if it was 'done'?"

This was, of course, a rhetorical question. Joe was right. I did write it in small steps. And I was still adding features to it after it was "done".

When we started on the Docketing system, much to my surprise, we started with basic data maintenance screens. It was the fastest way to get to something they could review and give feedback on. And it allowed us to get a hands-on look at our attempts to import the legacy data. What we all learned during those first few weeks was incredibly valuable.

The first couple of screens seemed easy enough. They were straight-forward, simple data maintenance screens to manage court locations [and other stuff and things]. We built the court location screen based on the existing system and the requirements the docketing department provided. We added a couple of additional fields and made the screen a little easier to use by switching up the tab order. When we showed it to them, they were delighted to see the progress. After a couple of minutes playing around with the screen, Cleo, one of the clerks asked, "Do we have to fill in this data every time?", pointing at the city, state, and zipcode fields.

"I mean we know what city we're in. Most of these are going to be right here. Can we have City and State default? We can type over them if they're wrong." That was an easy enough change. We could add City and State to some default settings and have it auto fill. If they tabbed to the field, it would highlight the text so they could type over it. If they kept tabbing, the defaults would remain.

We added a couple of fields to the defaults table, plunked them on the settings screen and wired up the court location screen. In reviewing these new features, the manager commented, "It would be nice to put all these court records together."

"What do you mean by together?", Joe asked.

"Well, she said. In the old system, every room in the building was a separate entry. The County court, for example, has several courtrooms. Today, we put every one of them in the system separately. It would be nice if we could just key in the building address and then for each room, indicate where it is in the building."

"Huh. Why didn't any of us think of that before?", asked Cleo, "If we did that, we wouldn't even need the fields filled in for us."

High Collaboration

And so it went. We'd discuss features and functionality. We'd talk about what screens would look like and we'd come up with a plan. We'd build to that plan and then we'd change it. The more they saw of the system being built, the more they came up with new and different ways to achieve what they needed.

I struggled with the amount of work we had to do in building something and then getting rid of it as we discovered new ways. Joe assured me, this was normal. "Just imagine if we'd written the whole thing first.", he said, "Imagine how much re-work we'd have then."

Joe worked with me and showed me more flexible ways to write code. He showed me ways to test my code as I went along. We wrote tests that made sure key aspects of the system did what we wanted them to do. We were particular about what code went where. We moved code outside of screens into separate files. When we found code that was common between screens or reports, we moved it yet again into separate files, calling the same code from everywhere that required the same functionality. We broke functions into smaller functions, making them easier to read. "Code is for humans.", Joe would say to me. "The compiler writes the code for the machine." "Write your code so that future version of you doesn't hate the present version of you when he has to fix it." "Don't be clever. Keep it simple."

While a lot of this work seemed like a bit of overkill at first, I soon began to see the advantages. When we realized we needed to make a significant change to the application logic, there was only one place we needed to make the change.

In the end, the system we wrote was significantly different from the system we were replacing. "I never liked the way they approached the problem.", the docket manager explained one day, "It's like they wrote the whole system without ever talking to an actual docket department. Maybe they worked with a filing room that handled docketing. I don't know. But is seems the entire system was based on records management, not dates, filings, and proceedings. This one is way better." We couldn't have known going in what the final product would look like. We realized new possibilities as we were writing the system. The docketing department didn't know what was possible. They didn't actually know what they wanted. We'd deliver a screen or a report or a complete workflow and after using it for while, they'd come up with a new idea or an alternate approach. As we built more features, the code changed. We'd pull functionality out into new files so it could be shared on multiple screens and reports. As those screens and reports changed, we'd restructure the code again. Sometimes breaking it out into additional separate pieces of code. Sometimes collapsing two or more separate pieces of code back into one chunk. Even after we were "done" with the system, we were still making adjustments. New legal requirements, one docketing department handling two locations, new ideas for reports, electronic records transfer to differing courts and filings with each state.

This is the nature of emergent work. As you cannot know the outcome up front, you must be adaptive and able to respond to change. The pragmatic approach is not to try to plan everything up front and control it, but to run small safe tests to see what happens and respond accordingly. Probe, sense, respond is the appropriate course. This isn't to say planning has no value. In the words of General Dwight D. Eisenhower, "Plans are worthless, but planning is everything." Here managers need to rely on the expertise of the people involved. This is a highly collaborative environment where people are sharing their perspective and experience with one another in efforts to figure out the best course of action. While management may be one of the experts, often they are not. It is the team that needs to make the majority of the decisions. Management's role is to reinforce the objective, facilitate conversation, keep everyone focused, and remove impediments.

Summary

In a complex context cause and effect are discernible after the fact. Confidence in our ability to predict outcomes is folly. The "right" solution cannot be determined up front as working on or in the system actually changes the system. Our preferred course of action is probe, sense, respond.

Management needs to be more of a facilitator. Adherence to hierarchy in decision making or establishing procedure is detrimental here. Teams of experienced workers collaborate together, executing small tests and getting rapid feedback to better inform their next step.

The key risks here is a desire for determinism and failure to learn. A complex context is unpredictable. This introduces risk, especially in environments where people are evaluated on their ability to be right. There may be a desire to demand fail-safe plans with defined outcomes; hoping that with enough rigor, we can control the environment. As we try to control the environment, we impede the opportunity to learn. Without that learning; without an opportunity through testing, observing, and reflecting, we cannot determine an informed path. Results are therefore unforeseen and often undesirable, increasing our want for control and predictability.