Yesterday I went to GE Hungary again to introduce the industry to my brother attending to mechanic engineer course (that will be a separate post as soon as censorship done). During the visit we had several technical discussions, and we talked about the “Emergency Stop” button among many other topics. A colleague we met in GE and I have a joint adventure with an emergency button from the time we worked together 10 years ago. Let me share the lesson we have learnt that time.

The system we worked on was a calibration and validation equipment for water meters. It consisted of many pipes, valves, boilers, pumps, tanks and sensors. My task was to develop the software to control the valves, and it gathered the measured data too. I needed to handle the “Emergency Stop” signal as well. Of course the project was in a large delay, we needed to demonstrate the base functionality within a few days. I created the function to control the emergency stop process. It looked like this:

void EmergencyStop()
 {
 //ToDo
 }

Yes, you are right! Of course I had no time to finish the code, we have started the equipment without having any emergency process implemented. When we needed to demonstrate the emergency stop functionality I just used the control interface to open and close valves one by one and stop the pumps as required. The customer didn’t even notice that the actions were not automatic. Then we completely forgot the issue.

Next time we went onsite to perform some testing. That was an unlucky day. First we tested the pump control algorithm, setting extreme low (few l/h) and extreme high flows (15 m3/h), and in parallel set the diameter of the pipes narrow or wide (pipe organ with different diameters driven with valves). The test was passed… almost. When we were at the pressure limit (higher flow with a bottleneck), suddenly a water flow with 8 cm diameter flew across the room horizontally hitting the customer in his chest. He immediately laid down behind a control panel, what was beamed by water. No injury was made, fortunately. We pressed the Emergency Stop button and … nothing happened! The water heavily sprinkled to the high-voltage control panel, the only equipment that would have been able to stop the pumps.

Finally we stopped the system and found ourselves in the middle of a dark room, which is fled by 10 cm water. All of us were wet, especially the customer, who had spent the last few minutes laying in the water under an operating high voltage pump control cabinet, what is continuously hit by high pressure water… That situation really looked like a failed test case.

We learnt a lot. After cleaning the room the customer requested that the next function we needed to test was the emergency stop process. We agreed.

Then I implemented the following code (or something like this) and we decided to have another attempt.

void EmergencyStop()
 {
 Valves=e_state_EmergencyStop
 Pump.RPM=0
 Horn(e_state_EmergencyStop)
 }

We knew that we would not be able to stop a few tons of water only by closing valves; therefore we defined a pipe ring, where the water could circulate consuming its motion energy after all the pumps are off. This valve setting was represented by e_state_EmergencyStop parameter. The valves had much delay, so we started with switching them first, and then stopped the pumps, finally turn on horns and flashing emergency signals. This was exactly the same I did at the first demonstration, so we thought that would have taken the job.

We went on-site again, started a (medium) flow circulating. And – with a bit of fear in our heart – I pressed the emergency stop button. We heard the valves turning, the pumps stopping, and we were really happy to see and hear the visual and audible emergency signals. We were so relieved and the customer seemed to be less upset.

Two minutes later water began to appear everywhere, especially at the corner of the room. It was terrifying! None of the pumps worked, no water intake was open, but there was a lot of water coming from the ceiling!

After cleaning the room again, we investigated the issue. Last time when I emulated the emergency stop manually I switched the valves one by one. This time it was done in the same time by the emergency stop algorithm. Almost all large valves were pneumatic ones, and the supply pipe of the compressed air was too narrow. When all valves were activated in the same time the air pressure decreased, and some of the valves were not able to open or close accurately. The pumps were stopped immediately, but the water remained circulating due to its movement energy, and one of the valves directed the flow into a tank two floors above us. This tank was the part of the system; it stood in the corner of the local library and computer room on the 2nd floor. (It’s a good design to have an open tank in the library…) The capacity was only 400 l, so it was full in a few seconds. The sensor signaled the overload and activated the emergency stop, but we were still in emergency stop state, so nothing was to be done. Then in the corner of the library the water started to come down, through the high voltage laboratory on the 1st floor. Fortunately there was nothing operating actually in that laboratory, otherways it would have given some unforgettable moments to the operators there.

We learnt a lot again. I modified the valve control algorithm to know which pumps are pneumatic ones, and I delayed the parallel switches to let 0.5 second the pressure to recover.

In later testing and calibration we needed to use the emergency stop many times, we knew that the procedure was reliable and well designed. As finally it was.

Pics:
http://www.gostarcn.com/product_list.asp?ClassID=34
http://www.jatekcsillag.hu/nkv/product_info.php?products_id=2731

As I expressed exactly a year ago I think sometimes being fired is the hugest appreciation a manager may get. Do you think I have gone mad? No, I have not.

When a manager appears on an interview, wins the competition and accepts the offer, he is not able to see where he has been dropped. It’s like marrying without knowing the bride before. And then, when the manager starts to do his work he has to realize that the changes he has to make cross other’s interests. Interests of people, who have enormous informal power: they can easily make the manager out of the door. In such case what can be done? One option is to change your mind, give up your plan to achieve success and hold your tongue to get your money as long as you can. The other option is to enter into tough negotiations and receive nasty attacks. Both ways may lead into your dismissal sooner or later, but I really admire managers, who have been fired due to their enthusiasm and stamina in a lost situation.

Counter_of_managers_I_really_admire_both_personally_and_professionally++;
Previously: EPF Composer

BPMN – Bizagi Process Modeler

I had to continue seeking ways to show processes visually. I knew UML2 together with its limitation, and I looked for other standard modeling languages to visualize process flows. I found BPMN, Business Process Modeling Notations. First I read the standard, and I found that very useful. I found BPMN support in Visio, but it was really cumbersome to draw process models. I gave up. Then I started to browse the internet systematically to find support tools for BPMN. Sourceforge did not provide solution, but I found Bizagi Process Modeler. This application is a (free) part of a huge system, what is able to compile and execute the models you create in BPMN. It looked very impressive. Of course I downloaded it immediately and started to use. It was awesome! It knew everything I missed before with other tools! It fully supports BPMN, but lets you to deviate a bit from the standard. Drawing is easy and fast. I really recommend.

So, I found the best tool ever to visualize our processes. The question was how I can integrate the pictures with the text, and how the user can browse it easily. I used one of the oldest solution: HTML. I put HTML imagemaps on the pictures I exported from Bizagi Process Modeler and linked them to the appropriate parts of the text. It works well, easy to browse, easy to maintain. I’m satisfied so far, but I’m sure I’ll try other tools if I find any. I’ll let you know if I see anything better than Bizagi Process Modeler.

Previously: Enterprise Architect and StarUML

EPF Composer

We were coding our products in Java Eclipse, and when we looked for add-ins to Eclipse we accidentally bumped into EPF Composer. This is an XML based tool for Eclipse that is able to handle all types of objects necessary to describe processes. First I did not want to get rid of UML, but I had to change my mind. EPF Composer is able to contain both text and process flow diagrams, and maintain the links between the objects. I really loved working in EPF Composer! It had a template for processes, containing fields for all interesting aspects, like summary, description, roles, WBS, and even best practices, templates and guidelines. It’s a really powerful tool! Generally I was really satisfied with EPF Composer and its features. The designed process definitions appeared in an HTML based report what we were able to share on our HTTP server. But, of course there were issues. It works well only if you use the same structure that has been defined in the software. This structure is too complex, because it needs to be able to store everything. We wanted to simplify that, but when you skip an object between two others (like process – roles – role definition path you want to get rid of role level, as role definition is enough) you cannot hide the intermediate object, therefore users need to click twice to find the appropriate object. In other words: you have to use and maintain all objects, which were defined in the application. We gave up using EPF Composer.

Will be continued…

Previously: Visio

UML – Enterprise Architect

We used Enterprise Architect to maintain our software designs, so it was straightforward to try its Business Process extension to draw our processes. Enterprise Architect did the work, it was quite easy to show who does what. It provided solution to keep our document templates just next to their definition, and it was able to track the changes made during continuous improvement. There was only two drawbacks: its price and the lack of acceptable and readable export.

UML – StarUML

UML had led us to success in Enterprise Architect, so we looked for another tool supports UML2, for free. I found StarUML, which is a quite good UML modeling tool, and it is open source. I started to redraw our process models in it based on the Enterprise Architect models. I could draw only the first few boxes to realize that StarUML supports connections only according to the UML standard, while EA was more flexible. It was impossible to create drawings with the same information content.

Will be continued…

On 1st January 2010 new taxation rules have been applied in Hungary. In a nutshell this means that the 18-36% graded personal income tax has been reduced to be 16% direct tax, and families having children may achieve further significant decrement. The change will increase the net salaries by 10-100% (!) in the field of IT, but in special cases it can give more. On lower wages it is in the other way around due to other changes, but now I’m focusing only on IT sector in Budapest.

So, there will be a huge increase on the net salaries. Fine, we are happy.

But.

The tax changes are applied only on staff employed on the lawful way. You may say that most of the international companies employ their workforce legally, but it is not true. I don’t know how it works in Western Europe, but in Hungary companies outsource tax evasion by using leased resources.

Normally leased resources are used to cover peaks in your resource needs. They are a bit more expensive than others, but no long-term contract is made, they may be laid off even today. Multinational companies often bring their strict “no tricky business and bribery” policies from their headquarters, but locally they lease resources from (Hungarian) companies, which have tricky business and bribery as core strategy. They don’t employ the people they rent, or they employ them on a reduced salary, and give them the rest as “zsebi” (“pocket-money”), what can be up to 80%.

Let’s assume you have both employees and leased resources in your specialist team.

Employees make gross HUF 600k, pays 300k tax, so the net is 300k (roughly). Leased ones make gross HUF 150k, pays 70k tax, and somehow they find another 220k in their pocket every month. This is also net 300k. The wages in your team is according to your pay grades, and very well balanced.

Then 1st January arrives. Ceteris paribus the employee will make 600k, but pays no tax, because he has 3 children, so his net salary will be 600k, what is 100% increase. Your leased resource will make 150k, pays no tax as he has a half kindergarten at home, and receives the 220k pocket. So he earns 370k, that means 23% increase.

Both of them have had a large increase, but your pay grade system has exploded. 600k >> 370k

You will need to be prepared to resolve this conflict. I’m sure that the body-leasing firm won’t handle the difference. The strategy is up to you, but I’m sure companies need to be more flexible in headcounts if they want to keep their long term leased resources. Or, according to the Hungarian way, you need to fire all, who have children. Both ways work, and probably many others.

The choice is yours, but you have to build your strategy rapidly, because people get they pay checks in early February, and the bomb starts the destruction.

Previously: RoboHelp

Visio

The first tool I wanted to use to create graphical overview of the Quality System was Microsoft Visio. It is able to draw anything you want, but sometimes the standard license is not enough. Professional one costs a fortune. I entered an overview of the system, and then one of the most complicated processes: Software Development Life Cycles. The initial drawing was not an issue, but later maintenance was a pain. Visio handles the graphical elements as graphical elements – how surprising -, and all of them has its own isolated life. Arrows crosses boxes, overlaps, and it is cumbersome to align boxes. Since process development normally means hundreds of changes in the first period, I gave up after a few days. I really do hate Visio.

Will be continued…

 

Previously: PDF

Robohelp

When we failed with PDF files, we initiated a project to look for possible tools to migrate the Quality System in. Since the main lack was the integration of the different processes, tools were sought that support one click reference between document parts. HTML seemed to be the most appropriate solution. The documentation team began seeking a tool, which is able to generate HTML or any other browser supported content. They evaluated Frame Maker and RoboHelp among some other free tools. RoboHelp has been selected with Flash content generation.

The documentation team migrated the entire quality system to RoboHelp. It took a week only, and then we started to revise and close the gaps between the independent processes. In half year the quality system was better than ever before. Some people, who were a bit more familiar with the content started to read and refer to clauses in the Quality System. In the years before there was only a few change requests, after the migration almost a hundred arrived in 5 months. The Quality System has got alive.

There was still an issue: newcomers were not really able to find the things in the QS. There were no pictures in that, only text, text and text. And as we used the documents, we realized that there are still gaps and mismatch between processes, and the description of some processes was still confusing. I decided to look for ways to show the system graphically.

Will be continued…

 

Before: Paper based

PDF

When I joined, they had the quality manual in electronic form: PDF files. It was more suitable to handle. Furthermore they did not distribute the manual to every employee at every change: the latest version was only available on the intranet, and it was forbidden to create hardcopies.

Worked, but still, there were problems. Every process was in separate PDF file and the links between them was not successfully maintained. The consistency has gone as changes were made. To avoid of frequent changes the processes were designed on too high level, only providing requirements to the daily work, not best practices or a real workflow with advices. Therefore staff did not feel that is helpful enough; they were not able to overview that. They were not allowed to copy and store the documents, so they needed to download them every time. This would have been fine, if the server had not been 2000 km away, behind an internet access with extreme narrow bandwidth…

This was an acceptable way to handle these documents, until the company was taken over by another one. The acquisition required the complete revision of the quality system, and that required a lot of changes. These changes made PDF files confused, full of interfering processes, undefined roles and inconsistent terminology. A change was necessary.

Will be continued…

Here I’m going to start a series to show how I tried to find an acceptable solution for documenting the Quality Management System. During this evolution I worked for several companies. Of course names of them cannot be mentioned here, so I’ll refer to them as my clients. I’ll also mention techniques, what I’ve seen in organizations, where I was invited to look around. All of them were software and / or hardware development firms; of course systems in other areas may require different approaches. Let me start…

Paper based

At my very first client, the Quality System was paper-based. It was around 180 pages. The Quality Manager distributed the Quality Manual after every revision. Changes were not too frequent, because every time thousands of pages needed to be printed on the only bubble jet printer of the company. I did not understand why they did so. All staff worked on computers. The most important processes were supported by excel macros. The company designed software, so all people were familiar with handling electronic content. The documented processes existed in electronic format, since the hardcopies were printed from Microsoft Word. Why was the Manual still on paper?

Then, as I spent a few time there I realized what the reason was. The most significant reason was that the management doesn’t really understood the idea behind the ISO 9000 / 14000 standard, they did not see the benefit of that. This was very common that time: there were no affordable management trainings available in Hungary. ISO 9000 was only a paper to put on the wall to be able to receive subsidy and orders from the government. Therefore most of the quality systems followed the standard, word-by-word, without any tailoring. And of course the processes were not followed, since they were really ineffective. When we talked about changing the documents to electronic form, the quality manager was reluctant; he insisted even talk about that, because – he said – the standard did not allow electronic form.

The paper-mill began its work just before the audit to counterfeit all the required documents, which have not been prepared during the year. Then the auditor arrived, received a coffee in a cup, and HUF 180k (EUR 650) in an envelope, and he left. This is the way how many Hungarian companies comply with ISO 9000 and ISO 14000. Still, unfortunately.

Will be continued…