The Function to be Tested First

Yesterday I went to GE Hungary again to introduce the industry to my brother attending to mechanic engineer course (that will be a separate post as soon as censorship done). During the visit we had several technical discussions, and we talked about the “Emergency Stop” button among many other topics. A colleague we met in GE and I have a joint adventure with an emergency button from the time we worked together 10 years ago. Let me share the lesson we have learnt that time.

The system we worked on was a calibration and validation equipment for water meters. It consisted of many pipes, valves, boilers, pumps, tanks and sensors. My task was to develop the software to control the valves, and it gathered the measured data too. I needed to handle the “Emergency Stop” signal as well. Of course the project was in a large delay, we needed to demonstrate the base functionality within a few days. I created the function to control the emergency stop process. It looked like this:

void EmergencyStop()
 {
 //ToDo
 }

Yes, you are right! Of course I had no time to finish the code, we have started the equipment without having any emergency process implemented. When we needed to demonstrate the emergency stop functionality I just used the control interface to open and close valves one by one and stop the pumps as required. The customer didn’t even notice that the actions were not automatic. Then we completely forgot the issue.

Next time we went onsite to perform some testing. That was an unlucky day. First we tested the pump control algorithm, setting extreme low (few l/h) and extreme high flows (15 m³/h), and in parallel set the diameter of the pipes narrow or wide (pipe organ with different diameters driven with valves). The test was passed… almost. When we were at the pressure limit (higher flow with a bottleneck), suddenly a water flow with 8 cm diameter flew across the room horizontally hitting the customer in his chest. He immediately laid down behind a control panel, what was beamed by water. No injury was made, fortunately. We pressed the Emergency Stop button and … nothing happened! The water heavily sprinkled to the high-voltage control panel, the only equipment that would have been able to stop the pumps.

Finally we stopped the system and found ourselves in the middle of a dark room, which is fled by 10 cm water. All of us were wet, especially the customer, who had spent the last few minutes laying in the water under an operating high voltage pump control cabinet, what is continuously hit by high pressure water… That situation really looked like a failed test case.

We learnt a lot. After cleaning the room the customer requested that the next function we needed to test was the emergency stop process. We agreed.

Then I implemented the following code (or something like this) and we decided to have another attempt.

void EmergencyStop()
 {
 Valves=e_state_EmergencyStop
 Pump.RPM=0
 Horn(e_state_EmergencyStop)
 }

We knew that we would not be able to stop a few tons of water only by closing valves; therefore we defined a pipe ring, where the water could circulate consuming its motion energy after all the pumps are off. This valve setting was represented by e_state_EmergencyStop parameter. The valves had much delay, so we started with switching them first, and then stopped the pumps, finally turn on horns and flashing emergency signals. This was exactly the same I did at the first demonstration, so we thought that would have taken the job.

We went on-site again, started a (medium) flow circulating. And – with a bit of fear in our heart – I pressed the emergency stop button. We heard the valves turning, the pumps stopping, and we were really happy to see and hear the visual and audible emergency signals. We were so relieved and the customer seemed to be less upset.

Two minutes later water began to appear everywhere, especially at the corner of the room. It was terrifying! None of the pumps worked, no water intake was open, but there was a lot of water coming from the ceiling!

After cleaning the room again, we investigated the issue. Last time when I emulated the emergency stop manually I switched the valves one by one. This time it was done in the same time by the emergency stop algorithm. Almost all large valves were pneumatic ones, and the supply pipe of the compressed air was too narrow. When all valves were activated in the same time the air pressure decreased, and some of the valves were not able to open or close accurately. The pumps were stopped immediately, but the water remained circulating due to its movement energy, and one of the valves directed the flow into a tank two floors above us. This tank was the part of the system; it stood in the corner of the local library and computer room on the 2^nd floor. (It’s a good design to have an open tank in the library…) The capacity was only 400 l, so it was full in a few seconds. The sensor signaled the overload and activated the emergency stop, but we were still in emergency stop state, so nothing was to be done. Then in the corner of the library the water started to come down, through the high voltage laboratory on the 1^st floor. Fortunately there was nothing operating actually in that laboratory, otherways it would have given some unforgettable moments to the operators there.

We learnt a lot again. I modified the valve control algorithm to know which pumps are pneumatic ones, and I delayed the parallel switches to let 0.5 second the pressure to recover.

In later testing and calibration we needed to use the emergency stop many times, we knew that the procedure was reliable and well designed. As finally it was.

Pics:
http://www.gostarcn.com/product_list.asp?ClassID=34
http://www.jatekcsillag.hu/nkv/product_info.php?products_id=2731