The ERP system you tested is not the one you go live on

Your programme did UAT properly. The business wrote out the scenarios, the team ran the cycles, defects got raised and fixed, the scenarios were run again and then the business signed it off. That is real work done by people who already had a day job, and the sign-off does mean something. What it means is that the configuration worked on the day it was signed, which is not quite the same thing as the configuration working on the day you go live.

After that day the configuration keeps moving for various reasons including defect fixes, a late requirement from finance, a pricing table that turned out wrong during testing, or the business realising the approval limit was set too low for the way they actually buy so they asked for the config to be updated. On a mid-market rollout that is dozens of changes at the low end and a few hundred at the high end, and each one of those changes means that some part of your signed-off evidence no longer matches the system you are about to go live on.

Nobody re-runs the full scenario list after each of those changes, and I do not think anyone should be blamed for that, because the scenarios need your best business people and those are the same people who are closing the month and getting product out of the door. Two full cycles is normally all that the budget and the goodwill will carry, and everyone on the programme knows it.

So what happens instead is that someone looks at each change and works out what it touches, and they will say something along the lines of this is only a purchasing change so we will re-run the purchasing scenarios. That call is normally made by one experienced consultant who is under time pressure, and it is normally the right call. The difficulty is that it is not written down anywhere, nobody goes back and reviews it, and it happens something like a hundred times over the course of a programme.

No date on your test evidence

By the time you get to the go-live decision your evidence is not one single thing, it is a mixture. Some of it was proven last week and some of it was proven three months ago against a configuration that has changed several times since, but on the sign-off document those two look exactly the same, because nobody writes the date on proof.

So when the steering committee asks whether the system is ready, nobody in the room can separate the parts that are still true from the parts that were true in April. The problems are already there at that point, they just do not surface until someone hits them in production, which is normally a few weeks after go-live when the business is trying to close its first month.

No requirements to test coverage

Your requirements and your test scenarios are two separate lists that were put together at different times by different people. The requirements were signed early and they are a few hundred statements of what the business needs, whereas the scenarios were written later and are usually grouped by process area rather than by requirement.

In most programmes there is no map between the two, so if somebody asks which of the signed requirements have no scenario covering them at all, there is no answer available anywhere. A requirement with no scenario against it never shows up as a failure, it simply does not appear on any report, which is why that kind of gap normally gets found after go-live when somebody tries to do their job and finds the system will not let them do it.

Cloud releases and config changes after go-live

This gets very little attention while the programme is running, but it is where a lot of the pain ends up.

If you are on CloudSuite or QAD Cloud then Infor and QAD ship updates into your environment on their schedule rather than yours, and even if you are running on-premise you control when an upgrade lands but not whether you eventually take it. On top of that your own team will keep changing configuration in production because the business keeps asking for changes, and if you are rolling out to more than one plant then sites two and three will be getting configured on the same instance while site one is already running the business on it.

Every one of those things changes a system that was only ever proven once, before any of them had happened. Most programmes do not re-prove anything at all after go-live, so the first indication that something has moved is a support ticket, or a customer ringing up to tell you.

How we fixed this: Infomind’s Agentic ERP Lifecycle Platform  

All of this traces back to one thing. Running your scenarios costs real money and takes your business people away from their day job, so a programme can only afford to do it a couple of times, and everything above follows from that.

That is what we built Infomind to change. Every scenario runs automatically overnight and again as soon as the configuration changes, and that means all of them rather than just the ones somebody judged to be affected.

The usual reason test automation gets abandoned on ERP projects is that maintaining it turns into a job in itself. Configuration changes move fields around, rename screens and add steps, the automated scenarios start failing for reasons that have nothing to do with the business process, and someone has to go and fix them one at a time. Infomind works out what changed and updates the scenario itself, so it carries on running rather than joining a backlog and being switched off three months later.

What that gives you is worth setting out plainly:

• • Every scenario carries a date, and nothing you are relying on is older than yesterday.
  • Nobody has to work out which scenarios a change might have affected, because they all get re-run regardless.
  • Your business people spend their time on the judgement calls and the awkward edge cases, which is the part that only they can do.
  • Requirements and evidence are held together as one thread, so any gap in coverage has a name attached to it.
  • Any scenario that has been automated can be turned into a training video and an assessment in a few clicks, in over 140 languages, so the same work that proves the system also trains the people who have to use it.
  • Updates from Infor and QAD get re-proven before they reach your users.
  • Site two starts from the proof that was built at site one instead of starting with a solution assumed to be validated.

Five questions for your next steering committee

You do not need to know anything technical to ask any of these.

1. How many configuration changes have we made since UAT sign-off?
2. What percentage of our scenarios have been re-proven since the most recent one?
3. What is the oldest piece of proof we are still relying on, and who decided it was safe to leave alone?
4. Which of the signed requirements have no test scenario validating them?
5. Once we are live, what validates the system when Infor ships an update?

If you get those answers back inside the hour then your programme is in better shape than most and somebody should say so out loud. If they take a week to put together then the answers do not currently exist anywhere, and that is not your team failing at anything, it is simply the way this has always had to be done, and it is the part that is now optional.

What I would ask for

Your UAT sign-off was honest work and the problem was never the sign-off itself. What is worth doing is asking for the date on it, asking what has changed since that date, and asking who decided which scenarios were safe to leave alone. If you are comfortable with those three answers then you are in good shape, and if you are not then you have a lot more time to do something about it now than you will have in six weeks.