PLANT REPLAY

As an extension to Inductive Automation’s Ignition platform, Tamaki has developed Plant Replay. This allows operators, managers, or plant engineers to go to any HMI client in a plant and replay exactly what was happening at any time – days, weeks, or months in the past.

It is important to emphasize that this is more than just a screen replay, it is the operator client which has been switched to Replay. While time is running forwards/backwards, fast or slow, the operators change screens, call up trends, build new ones, open device detail popups – in other words use their client exactly as they would in real time – and everything they see is in replay.

To get a free trial, go to our Downloads page

June 2015: Plant Replay first installed in a Colorado dairy processing plant

2015-2016: Installed and qualified on Tamaki projects in the US, UK, Ireland, China, New Zealand

Feb 2017: Released for general use

We have posted the following “Ignition-style” training videos. Please let us know what more you need:

Introduction and Installation

Simple Example

More Complicated (Indirection)

In 2014, a small team of Tamaki engineers were working on a reporting project at one of our customer’s sites – a little PLC code to track plant states, snapshots of product levels at the start and end of runs, instantaneous and average flows and temperatures, all of which needed to be reported in a format that was useful to the plant managers. As often happens when an exercise goes well, the customer kept wanting more and reports were extended into one plant area after another.

The exercise came to an end however, when we hit the maximum number of tags licensed for our customer’s plant historian and could add no more points into history. Extending the license required an up-front purchase of more history tags as well as an ongoing fee for annual support, and this was deemed too costly for a “mere” reporting project.

I’m sure this is a situation familiar to many of us who are working in the automation industry. We have grown to accept that what can be reported in our factories is often limited by the size of our plant historian licenses. But some of Tamaki’s engineering team on this project had only recently graduated from college and instead of accepting the situation they did what came naturally, and typed “HMI unlimited history” into Google. And that was how Tamaki discovered Inductive’s Ignition.

Fast forward two years and the site we were working on is now historizing 180,000 tags, and the HMI that was a constraint in 2014 is now itself in history.

Not long after discovering Ignition and using it for reporting, we were working at a dairy processing site which had a couple of small HMIs and a few panel displays, but no site-wide HMI. Neither did it have a large budget for an HMI, so we looked at the options.

By now, in addition to unlimited history we had verified a few more “unlimiteds” in Inductive’s Ignition, for example unlimited tags, clients, and designers, so we decided that it was time to use it for more than data logging and reporting. This wasn’t a large project so we started by putting all tags into history. After all, if three years of data from 180,000 tags fits on what is now the standard size harddrive for a laptop, why wouldn’t we.

The project was a success – screens were built, machinery was programmed and commissioned, and it was all over rather quickly.

At this point, the HMI project we had built was much the same as those we and everyone else have been building for decades. An operator, an engineer, or a manager could go to any screen in the plant, call up mimic and selection displays, and very easily see exactly what was happening in the plant NOW.

There were indications as to what the plant was going to do next; timers counting down through wash steps and messages saying for example “waiting for 12.5 gallons to dose, now at 8.3” and counting. There were reports to summarize what had happened in past days, and trends that could be called up to show what a pressure or a flow had been doing between 2am and 3am the previous morning.

In fact, because everything was in history, the trends were rather good. You could combine a level in a tank with the open state of the product valve that was feeding it, the run state of the pump that was emptying it and the on state of the operator selection that controlled it and configure a custom built “super-trend”. But even though these were pretty good, they were still only traces on a graph and needed to be analyzed in order to understand what had happened.

It seems so simple in retrospect that we are almost ashamed that we hadn’t thought of it before. At this point we asked ourselves, “instead of looking at trends and trying to work out what the plant was doing yesterday, last week, or last month, why don’t we just play it back on the operator’s HMI?”.

So this is what we did: we added a field with a calendar for date and time entry, we added a slider for time, along with fast/slow/rewind/pause controls, and we linked the points on our screens back into Ignition’s unlimited history, and allowed Plant Replay from our HMI clients.

It has only been 18 months since then, but we have rolled this out as standard on Ignition projects across the US, UK, Ireland, China and New Zealand. We have installed it in large plants with 70 operator clients and with more than 300 mimics, and on small two operator blending plants. In process plants and on machine lines. In every case the benefits have exceeded what we could have imagined.

When you think about it, a good HMI tells the operators exactly what they need to know about their plant – but only while they are looking at it, and only if they are looking at the right screen. Much of the time, when something happens the operator will be out of the control room, looking at a different screen, or looking at a trend which is over the top of what happened.

Useless. And in retrospect, how did we ever think that was acceptable?

Now at our factories, if something is missed the operators click “Replay”, change to the screen they should have been looking at, call up the selection with the message they needed to see, and slide time back to look at what happened. In fact, it wasn’t altogether what we expected. We give operators the ability to Replay everything that has happened over the past two years and they are mostly interested in the last 10 minutes.

It is important to emphasize that this is more than just some sort of screen replay, it is the operator client which has been switched to Replay. While time is running forwards/backwards, fast or slow, the operators change screens, call up trends, build new ones, open device detail popups etc – in other words use their client exactly as they would in real time – and everything they see is in replay.

Since the HMI now plays the past, maintenance can track down anything that happened provided only that they get to hear about it. To make sure this happens we have added an “Operator Feedback” function to our HMIs. From any operator client, anyone can report an issue. “Outload Oper05 – 2:14 am – noticed that the flow was low and had to boost the pump speed 20%. Contact Joe”

In the morning, maintenance look through the feedback reports, and when they find this one they change to the outload mimic, dial in 2:13 am and play forward at 5 times speed. Sure enough, at 2:14 am they see the control mode for the pump speed changes to “Operator”, and see that the pump speed was raised to 80% (noting in passing that this was up by 16% rather than the 20% which the operator reported). At the same time the flow picks up and a few minutes later the transfer finishes normally. They know that the outload happens at the same time every night, so quickly change to 2:15 the night before, and 64% is giving the correct flow as it did the day before that. But here’s the thing – change the time to 02:45 and call up the trend for tank level and you see that the tank emptied in 20 minutes previously, but only took 14 minutes last night.

Conclusion – it’s the flowmeter. Recalibrate and close the report with “Joe – thank you. We traced this to the flowmeter which was reading low last night. All fixed now.”

One of the most revealing trends you can build on a site which has Plant Replay is a “system” trend showing the number of connected clients, and the number of clients which are running in Replay. Most of Tamaki’s customers operate their factories 24/7, with shift change for example at 6am and 6pm. Ten minutes either side of shift change we see a surge in the number of clients as Team Leads meet in their offices and review the past shift. If there have been issues during the night, we see these clients flick in and out of Replay while an incident is reviewed. Half an hour later, the number of connected clients has dropped back to normal, but we see operator clients flick briefly over to replay as the Team Lead goes around the control rooms, and shows operators what occurred.

The next surge in the number of client connections occurs closer to 8am, as management checks on the plant status and reviews the night. Numbers settle while meetings are held then typically hold steady through the remainder of the day. If the site has problems though, more clients are connected, and this time they stay in Replay longer as Engineering hunts down root cause.

We can’t prove that Plant Replay makes a site run smoother and more profitably, but we can see that it is being used.

We really developed Plant Replay for ourselves, because it seemed like a pretty neat trick, and because we could.

Compared to the examples of daily use cited above, we are only occasional users of our own system. The exception to this is during plant commissioning when we are installing new code into a factory. Plant Replay introduces transparency. The time when commissioning was a somewhat blurred process during which the programmer knew more than anyone else about what was going on and could perhaps smooth some rough edges without others noticing, is now over.

But transparency is a fixer in automation, as it is for many other things. “On the first three runs when we lost level you can see that this happened, but after the changes we made on the 14^th, even though we dip below 9% three times, we can show you here, here and here, that the small amount of feed forward we have added….”