Being able to quickly and safely maintain your instrumented safeguards is essential to maximizing process uptime. Maintenance facilities and procedures should always be factored into front-end loading estimates for instrumented safeguards. However, it is likely that many sites have encountered projects, where provisions for maintenance, test and repair were treated as an afterthought.

The failure rates assumed during the verification of the risk reduction are based on the timely performance of the routine planned preventive maintenance (PPM). The reliability of an instrumented safeguard also depends on the scope and timing of periodic inspections and the timeliness of any needed repair when a fault is detected. Procedures and maintenance facilities can be needed shortly after start-up, as early failures are found and planned maintenance work begins.

An effective plan for instrumented safeguard maintenance can impact:

• equipment and piping – taps, bleeds, valves, maintenance bypass lines, spare parts strategies
• operating strategy – whether to commit to always coming down to safe state for test and repair or alternatively to design for on-line maintenance and repair activities
• organizational plans – need sufficient numbers of competent personnel to execute the activities, capture the relevant performance data, and investigate abnormal behavior

The following case study highlights an incident where there was plenty of evidence that the existing equipment was not performing as needed. A change in the instrument specification was also not carried forward into the maintenance procedures, so the criticality of a maintenance task was unrecognized by the technicians executing the test. Changes in equipment technology often impact maintenance procedures, but if the management of change process does not drive personnel to examine the maintenance procedures, the disconnect between the field and the procedure can have significant safety impact, as illustrated by the Hemel Hempstead incident.

Case Study Example:

Fuel storage; Hemel Hempstead, England; December 11, 2005

Impact: Explosion and fire; 43 injuries; 2,000 evacuated, commercial and residential damage

Summary

Gasoline was being delivered to the tank on the day before the incident. Early the next morning, the Automatic Tank Gauging (ATG) system displays an unchanging level, although the tank continued to fill. By practice, the operator controlled level by terminating transfer upon receipt of the user alarm. However, the ‘user’, ‘high’, and ‘high high’ level alarms used the same transmitter, so the failure of the shared transmitter rendered these alarms inoperative. Since the alarm never activated, the operator did not take action to terminate transfer.

An independent high-level switch, set above the ATG high-high level, was designed to close inlet valves and activate an audible alarm, but it also failed. The high level switch had been disabled when the maintenance organization, due to lack of understanding of the relatively new technology and to insufficiently detailed procedures, did not reinstall a lock on the switch test arm. Without the lock, the level switch was not activated when the float was lifted. By late afternoon, the tank overfilled and contents spilled out of tank roof vents. A vapor cloud was formed and noticed by tanker drivers and by people outside the facility. The fire alarm was activated and firewater pumps were started. An explosion occurred a short time later, likely ignited by the startup of the firewater pumps.

Instrumentation and Controls Gaps

• Analog level had 14 dangerous failures (stuck) in preceding 3.5 months
• Safety implications of frequent analog level dangerous failures not noted or logged
• 3 level alarms did not activate due to same analog level failure
• High level switch interlock failed due to undermanaged instrument technology change performed by maintenance group ~18 months earlier

Key Automation Learning Points

It is critical to train maintenance staff to properly test equipment and to verify that the equipment has been properly returned to service. The fault response and repair procedures should include a check for unacceptably high failure rates. Written instructions should be provided on how to escalate these situations to maintenance and facility leadership for investigation and correction.

In the above case, a control instrument with abnormally frequent failures (a maintenance “bad actor”) went unreported. The only independent safeguard failed due to an instrument technology change that was not effectively incorporated into the maintenance program.

Sustainability Actions

• Work checks for abnormal failures and to whom among leadership to escalate this information to should be incorporated into the detailed written maintenance procedures
• A change to make, model, or electronic version number consider the maintenance strategy and ensure that PPM and test procedures are updated and maintenance personnel retrained if needed.

References:

1. 2007. Buncefield Standards Task Group (BSTG) Final Report. UK: Health and Safety Executive.
2. 2017. Guidelines for Safe Automation in Chemical Processes-2ed. New York: AICHE.
3. Summers, Angela E., E. Roche, H Jin, M Carter. 2015. “Incidents That Define Safe Automation.” Presented at 61st International Instrumentation Symposium, Huntsville, Alabama, May.

Myth 1: Application programming for modern safety PLCs is so easy that anyone can do it.

With drag-and-drop interfaces, function blocks and some training, you would think that almost anyone could program a PLC. But translating critical safety logic into a PLC application program requires close cooperation among programmers, process control engineers, and operations and maintenance personnel to ensure that the application program supports plant operation, as well as safe operation.

A program specification should be developed and approved prior to PLC programming. Some PLCs have tools that translate the program as written into an engineering document. This is great for debugging the program, but it is not the same as programming and validating the system operation per an approved specification. Through a debugging exercise, you can obtain a program that under certain test conditions will do what you intend, but this program can have significant hidden problems that will not be seen until conditions change. A program with latent bugs can negatively impact system reliability and risk reduction. Disorganized and complex programming can yield an application program that’s difficult to understand, properly test and safely modify.

SIS-TECH has experience with all major brands of PLCs.  Whether Emerson, Honeywell, Siemens, Allen-Bradley, Yokogawa, HIMA, Triconex, or any other PLC, SIS-TECH can execute turn-key projects in critical control, alarm, interlock and SIS applications. For more information on SIS-TECH’s independent engineering services, contact Eric Randecker, erandecker@sis-tech.com or 832-434-7307.

Step 1. Confirm Hazard and Risk Analysis (H&RA) assumptions

The Hazard and Risk Analysis establishes the preliminary risk management strategy. The supporting documentation contains assumptions related to automation reliability and the capability of on-site staff to sustain the system. Examples of common assumptions include:

• Well trained operations who reliably use written procedures
• Facility only operates under the conditions evaluated in the H&RA
• Facility does not intentionally violate the safe operating limits (throughput, process conditions, etc.)
• Adequate staffing levels of operations and maintenance technicians to address faults and discrepancies in a timely manner
• An effective alarm management program is in place
• Instruments are fit-for-use for the process applications
• Low demand rate on the instrumented safeguards
• Independence between causes and safeguards exists

Functional safety practices dictate that the site verify the  H&RA assumptions. IEC-61511:2016 requires the determination that the design, operation, maintenance, and testing are sustaining the safety integrity as required by the site risk management strategy. ANSI/ISA-18.2 requires that alarm events be monitored to ensure effectiveness of the alarm system. Gaps between the assumptions and reliability can have both direct and indirect impact on actual site risk.

Case Study Example:

Natural gas processing; Longford, Australia; September 25, 1998.

Impact: Explosion and fire; 2 fatalities; 8 injuries; Plant 1 destroyed, Plants 2 and 3 shutdown, 5% loss of supply, 250,000 workers sent home.

Control instrumentation for absorber bottoms condensate

 Summary

The LPG Plant 1 separated methane from LPG in a pair of absorber towers using lean oil. During the night before the accident, the level increased in the knockout section of Absorber B. Since the disposal route to Plant 2 was not available, an alternate route to a Condensate Flash Tank was used. The normal procedure of increasing absorber bottom temperature was not done. As a result, the flash tank protected itself from excessively cold temperatures by decreasing incoming flow, which in turn caused absorber condensate level to continue to increase. Eventually, condensate mixed with rich stripping oil. This mixture flashed across the level control valve and lowered the temperature in the Rich Oil Flash Tank. Temperatures throughout the plant were lowered as rich oil flowed through the process. A low temperature trip of the lean oil pumps resulted, and the trip was not communicated to the plant supervisor for over an hour. A hand switch was actuated to decrease flow through exchanger GP905 in an attempt to restart the pumps. The heat exchanger ruptured due to cold temperature embrittlement, releasing a vapor cloud of gas and oil. The cloud traveled 170 meters to fired heaters before ignition occurred.

Instrumentation and Controls Gaps

• 100s-1000s of alarms happened per day, many regarded as nuisance
• Critical alarms were not prioritized
• Operators desensitized, alarm system ineffective
• Operators and supervisors did not understand the consequences of their manual actions and experienced engineers had been moved off-site

Key automation learning points

Alarm management reduces the number of alarms to only those requiring operator action. The risk reduction that normally results from a robustly managed safety alarm program is fully dependent on timely and correct operator response to the alarms. Having chronically high process alarm rates in a facility or a significant number of alarms which do not require action will promote the development of ineffective alarm response habits. [ANSI/ISA 18.2]

As in most large events, there were many contributing factors. Among them were two significant deviations from common H&RA automation assumptions:

• The facility was operating in an alternate mode in which operations and the front-line supervisors, in the absence of experienced engineers, did not correctly understand how the process conditions would respond to the manual actions they took.
• The alarm management system was ineffective, with chronically high levels of unprioritized alarms, fostering desensitization to and basic distrust of alarms by the operators.

With these deviations from standard H&RA assumptions, any Operator Response to Alarm protection layer would be very unlikely to succeed.

Sustainability action

Audit to confirm that the assumptions made during hazard and risk analysis are actually reflected in plant operation and that these assumptions remain valid over time.

References:

1. Hopkins A. 2000. Lessons from Longford: The ESSO Gas Plant Explosion. CCH Australia Limited.
2. 2017. Guidelines for Safe Automation in Chemical Processes-2ed. New York: AICHE.
3. Summers, Angela E., E. Roche, H Jin, M Carter. 2015. “Incidents That Define Safe Automation.” Presented at 61st International Instrumentation Symposium, Huntsville, Alabama, May.