Executive Overview

This is the final part of a 3 part series on Legacy Applications, visit Part 1 and Part 2 to learn more.

All of us in our Careers as Executives have encountered Software Applications and systems that over time have degraded in their capabilities, can no longer support the business in its current state, becomes too slow and costly to maintain, or worse is mission critical but very few people know how to support it.

The question that always comes up is, do we:

• Fix It?
• Replace It?
• Re-Architect It?
• Or “Pray” that nothing Bad happens… and that we can hold out just one more year.
  

Of course, any of the above Options may work and they all have some costs and risks associated with them, especially if the system suddenly stops working at an inopportune time. In this Blog Article, we will discuss the primary Business Drivers, Technical Considerations, Decision Criteria, as well as several real world Examples.

Real World Examples

Here are just a few of my own personal experiences during my career as a CIO & Software Executive:

Garvin Promotion Group:  Rebate Claims Processing

Garvin Promotion Group is a marketing promotion management company that creates and processes Consumer Rebates for a variety of industries, primarily in the Alcoholic Beverages, Tire & Auto Parts, Firearms, and Pharmaceutical industries.

We had a Legacy application which handled all of the Rebate Claims Processing for every Promotion for each Client.  In our case, the biggest problem was Technical in nature, which directly caused multiple problems for the Business and was preventing us from scaling the application to support a larger Client base.

The core problems that our ArkusNexus Team uncovered were:

• The original Application Designer did not have an IT background, so he built a structure that worked like a Spreadsheet, just using a MS SQL Database instead.
• Every major processing step was done in a “Batch Mode”:‍
• If batches were processed in the wrong order, then things would break
• When creating a Batch, it would literally delete all of those records from the Database
• Batches were then stored on the local PC, so if someone accidentally deleted one, if was gone and could not be recovered
• The Business would have to go back to the original paper claims and re-enter everything into the system from the very beginning
• In addition, this meant that our Customer Service Department never had up to date information on a specific Claim when a Consumer called

• There were no centralized Reports, all Reporting was done locally on an individual’s PC
• This meant that frequently numbers didn’t match, as they weren’t using the same exact queries
• End-Users were actually using SQL Enterprise Manager to build and run their SQL Queries, and because they didn’t know any better would run a query that locked up tables, records, or an infinite query that brought down the entire system.
  

Needless to say, this was causing a lot of pain, frustration and even in a few cases loss of business with a particular Client, as we were not able to process Rebate Claims timely or with a high level of customer satisfaction. In addition, the owner Sam Garvin was frustrated that none of his key reports from different departments matched, so he was never sure exactly what the state of the business was.

ArkusNexus Team Approach

We worked with our Arkus Team and the rest of the Key Business Executives on the following plan of action:

• Define a Product Roadmap to Fix / Re-Architect the Application, based on which Modules / Functions were causing the most “Pain”
• Then Module by Module we completely redesigned it, such that‍
• The Database had a more standard Normalized structure
• Implemented a Message Queuing Service for processing Transactions - eliminating the need for Batches
• Provided a standard set of Reports that could be run from within the Application
• Provided near real-time updates on the Status of Claims Information and processing‍
• We also migrated the entire system to a Cloud Virtual environment on RackSpace, which allowed us to easily scale the system, take snapshots on a regular basis, and easily recover everything if anything happened
  

It took nearly 2 years to accomplish this, but overall this effort was an enormous success:

• Eliminated Application Locks and Outages due to “Bad” SQL Queries
• Provided dramatically improved Custom Service, and ultimately Client Retention
• Eliminated the need for Batch Processing, which increased Performance, Reliability and Scalability of the Application.  We could now easily support 1,000x the number of Consumers and Clients that we did previously.
• And we now had “One Version of the Truth” for Sam, the Owner.

‍

Incorporate Massage:  Booking Application

Incorporate Massage provides professional Massage Therapists for on-site Corporate Events, Special Events, Conferences and Seminars, throughout the United States. The core application that is used by both Clients (Corporations & Organizations) as well as our network of licensed and certified Massage Therapists is our Custom Booking Application. This allows:

• Corporate Users to‍
• Schedule Upcoming Events
• Decide the details:  how many participants, how many Therapists, length and type of Massage, etc.
• Update or change Billing information

• Massage Therapists to‍
• Receive notifications of Events in their area
• Accept / Reject assignment to an Event
• View all of the details for the Event - parking instructions, security information, address, length of the event, start / end time, break times, etc.
  

Initially, our original Offshore Development Team in India was struggling with a number of issues in supporting the Business Operations and our Massage Therapists:

• Significant Performance Issues at busy times
• Data Issues - specifically with Events and Massage Therapists being assigned incorrectly, causing a lot of Customer Service problems
• Billing Problems
• Scheduling Problems
• Support Availability during normal business operations - as they were based in India, it could take 2 to 3 days before an issue was identified and fixed
• Lack of visibility in terms of what they were working on and their progress
• Lack of quality - often new Releases included numerous Defects / Bugs that made it into Production
  

Because of all of these challenges we made the decision to change our Software Development Partner to ArkusNexus.

Almost immediately, our new ArkusNexus Team uncovered a number of serious Technical issues as well:

• Passwords were being passed in the “Clear” and could easily be intercepted.
• SSL Authentication was not configured correctly.
• Virtual Server configuration was not set up correctly, so there were numerous Security holes
• Backups of the core Database, Code Repository, and System was not configured correctly - and would not work.
• The Database Structure could not scale going forward, and was more of a flat hierarchy structure
• Most of the SQL Queries were poorly written and did not use common techniques like, “No Lock”, or optimization, to prevent database table locks or long running queries from affecting the overall performance
• The Co-Location Facility hosting our Booking Application was fairly expensive and did not provide any standard monitoring systems, and support was lacking
• And we uncovered several Memory Leaks within the Booking Application, such that overtime it would Fail and need to be restarted
  

Based on all of these considerations, in conjunction with our ArkusNexus Team we put together a plan to slowly Migrate the Booking Application to AWS Cloud, and then component by component start to Re-Architect the entire Application. We started naturally with the areas of the application that were causing the greatest amount of pain to the business or represented the greatest amount of risk.

ArkusNexus Team Approach

Our ArkusNexus Team executed the following Plan and Approach:

• Initial Analysis
• Develop a Plan of Action - which was tweaked a little by the Business Executives and the CEO

• Migrate the Booking Application to AWS Cloud‍
• Included migrating the full Production System (Web App, SQL Db, Integrations, etc.)
• Creating a new QA / STAGE Environment
• Configuring the Backups of the environments
• Etc.

• Fix all of the Security Issues
• Re-Architect the Client Event Scheduling system to use Micro-Services, this eliminated some of the broken SQL Locks
• Re-Architect the Therapists Mobile & SMS Text experience, making it easier and more reliable for them to accept Events
• Re-Architect other parts of the Database Structure to better support existing and planned features
• Implement a Message Queue (MSMQ) for Transactions and Integrations, which fixed many of the Billing issues
• Eliminated other Memory Leaks and SQL Queries that cause Locks and Application Slowness
  

In addition to all of this our ArkusNexus Team was also able to add some new and unique features, as well as features that were required from a State Regulatory perspective. This included:

1. Implementation of Paid Breaks for all States - every State is slightly different in their specific rules, so we had to create a complex rule system to handle this, which also needed to interact with our Event Scheduling system.
2. Heat Map - for determining the availability of Massage Therapists within a given area of the Country.  This was hugely beneficial for the Scheduling Coordinators.
3. Bonus Payments to Massage Therapists - for either high volume times (Holidays), or remote locations.
4. SMS Text Message acceptance / rejection of Events, as well as the ability to get all of the key information in a SMS Text at any time.
   

All of these enhancements and changes dramatically affected the core business in a very positive way.

Later, the Board of Directors decided to wind down the Custom Booking Application and Replace it with SalesForce. That is a story for another day, but as you can expect took longer and cost more than expected; and then the Covid-19 Pandemic hit, which dramatically changed the entire business.

‍

Universal Technical institute (UTI - NYSE:  UTI)

UTI is the largest Technical Training education facility for Automotive, Motorcycle and Marine repair Technicians in the US. They have 13 campuses in the US, are a mid-sized firm (roughly $400 M in annual revenue), and approximately 20,000 students.

In addition to their apprentice Training Programs, they also have very strong industry partnerships with many of the manufacturer’s within this Industry. In many cases the Advanced Programs allow a student to work on the latest models of the vehicles, and upon graduation are offered a job as a Journeyman Technician at a dealership. This includes organizations like:

• BWM
• Cummins
• Daimler Truck
• Ford
• General Motors
• Harley-Davidson
• Honda
• International Trucks
• Kawasaki
• Mercedes
• Mercury Marine
• NASCAR
• Nissan
• Peterbilt
• Porsche
• Suzuki
• Toyota
• Volvo
• Yamaha
  

In 1999, UTI’s IT Department convinced the Executive Management Team and Board that they needed to implement a Student Information Management System.  Unfortunately, UTI’s model did not fit into the common University model at that time, as they had a new cohort of Students every 3 weeks. They couldn’t find a COTS Solution that fit their needs, so they opted to build a custom Student Information Management System from scratch.

The initial process took over 3 years, with continuous upgrades and enhancements after the “SMART” Application finally went live in Production. Unfortunately, the IT Director at the time selected a fairly new 4GL Programming Language known as Progress Software. The “promise” was that using Progress would make it easier and faster to develop an application, maintain it, and enhance it in the future.  

The HUGE drawback is that the technology never really took off that much within the US, so it was very challenging to find and hire new Developers who knew this programming language. We often had to hire them, wherever they happened to live and have them work remotely.

By 2009 this was starting to cause problems both for the Business as well as from a Technology (IT) perspective. So, IT in conjunction with our Business Executives, started an Initiative to replace SMART with a COTS Solution, as they had evolved since the late 1990’s. Our challenge was that everytime we found a potential possibility, someone on the core team would find a fatal flaw to it. The running “Joke” was that it was going to take 5 years for us to replace SMART… and that is what we would say every year.

Finally, it just had to be done, the Business simply had enough. UTI found a collection of COTS products built on modern technologies, Cloud based, and highly scalable; and over roughly 3 years replaced core modules one at a time, while creating an Integration Framework that could support the system into the future. But this was not easy or cheap, as it ended up being a multi-million dollar replacement project.

‍

Conclusion

In this Blog Article we’ve covered the main Business Drivers, Technology Considerations, and Options that you might take as an Executive or Organization. Which approach you take in either Fixing, Replacing, or Re-Architecting your System / Application, will depend greatly upon your individual situation and business needs.

In Part 3, we have also provided you with several real world examples, including the challenges and considerations which we took into account. We hope that this Blog Article has been valuable to you.

Thank you, David Annis.