Vita Rara: A Life Uncommon

Replacement Systems are Really Hard

Categories: | |

Create a list of the hardest things to accomplish in software development and I'm sure near, if not at the top of the list, will be replacing an existing system. Especially a system that runs the lion's share of a business' operations. Especially when you can't look at the source of the system you're replacing.

For the past year I've been working on my Quadran project. Quadran is a drop in replacement for a system my client has been using for about five years. Completing Quadran has been just about the hardest things I've ever done. It has felt like an IT death march.

Why is it so hard? We have a working system to look at. We can see what it does. We can talk to people about what it does. Why is it so hard?

  • We don't have access to source code.
  • The old system has been running smoothly for a few years now and people have forgotten what it does under the covers.
  • Because the old system has been running for so long no one remembers the details on what it does and especially they don't remember why. We've lost tribal knowledge.

The reader should also know that I was involved in the development of the system that is being replaced. I was the lead developer on a team of five or six developers, and I maintained the system for four years after it went live. So, it's not like I'm walking into a problem domain in which I have no experience. I know this area well, but apparently not well enough. It has been a year long process of reeducation.

Some would ask why we're even doing this? Why not maintain the exiting code? The short answer is there were intellectual property and technical issues that precluded the continued usage of the old code. Maybe another day I'll tell the long version of the story.

So, what are the lessons to be learned? There are a lot of them. Some for the developer. Some for the client.

Client Lessons:

  • Never ever bet your enterprise on a closed source solution from a small vendor. EVER!
  • Don't confuse sourceware with open source software. Just because the vendor will provide you with source code that doesn't mean someone else can really maintain it. Never enter into an agreement that requires a third party to sign a non-disclosure agreement to maintain the code in your system.
  • If you violate rule number one insist upon documentation for all of the business rules that are embedded in the system, or preferably an interface in the system itself to view and edit them in a scripting language.

Developer Lessons:

  • Access to a data schema will only get you just so far. The rubber really hits the road when you try to ferret out the business rules.
  • Identifying every process in the system being replaced is paramount, then figure out the forgotten side effects of those processes. They are there. Go find them.
  • Mock up early and often. Release processes to the end users as soon as possible, and make them review them. Tell them you are stopping all development until they review the work if you have to. The sooner you get them thinking about how the new system should work, the sooner you will find out about the invisible business rules.
  • Dependent business rules are one of the largest source of bugs in large business systems. To address this we created a DependencyManager class that handles the updating of dependent properties of an entity based on its dependent entities, then recursively calls the tree of dependent entities in a specific order to have them update their dependent properties.It's not the best solution, but at least when some says, "we received this shipment, but the related purchase order didn't move to a COMPLETE status," we know where to go look.
  • Test, test, test.... Unit tests are great, but the largest benefits for us in this project have derived from integration test running over the clients actual data. This has highlighted bugs in their old system, kept our data model in line, and helped us clean up the validity of our data migration.
  • If you're migrating from a legacy data model to new data model look for entities/tables that multiple "status" type fields in the old model. Each status probably represents a different entity, and should be broken out into its own entity/table with a separate status.

That's all for now. If you're considering a clean sheet rewrite of an existing system think long and hard about it before plunging in. More on that later.