Document pmGG6g3a1QNw9o9w5kznzyLn6

< V ' Deer Park Plant {NARRATIVE REPORT Power Failure and Zebra Poly Relief Valve VCM Release July 14, 1997 Summary At 0406 on 7/14/97, a plant power failure occurred due to a voltage dip on both electrical feeders supplying the plant, with subsequent failure of the plant control system uninterruptible power supply (UPS). All eight polys were emergency short-stopped with AMS, but Zebra poly, one of five polys in reaction at the time, overpressured and the relief valve lifted three times for approximately one second each time. Total VCM released was 57 pounds. All agencies were notified within 15 minutes of the first discharge. It is not conclusive from the data whether the cause of the overpressure was a hydrofull poly or continued reaction. There were no injuries or exposures. Board of Inquiry A meeting was held at 7:30 am on 7/14/97 to discuss the incident. The following were in attendance: Manuel Araujo Steve Johnson Bill Lindstrom Mark Reynolds Byron McWhirter Val O'Keefe Bob DesJardins Brion Cann Joe Schwarz Alex DeSantis Charlie Franks Shift leader Geon operator Geon operator Sr.Process Control Engineer Supervisor, Process Reliability Environmental engineer Plant Manager Sr.Process Engineer Sr.Safety Engineer "A" Line engineer "B" Line engineer Utilities engineer Sr. Engineering Specialist Corrective Actions 1. Install startup instruction plaque on the UPS panel and review with all shift leaders and head operators. (S. Johnson by 8/1/97). 2. Replace three low voltage batteries on the UPS. (Complete 7/17/97). 3. Replace all 30 UPS batteries with a new 30 battery set. (L. Henderson by 8/1/97). 4. Recommend rewiring the UPS bypass source to a motor control center on a different feeder (S3 or S5). (C. Franks, plan by 8/15/97, due by 12/31/97). 5. Install a second on-line 30-battery set to minimize exposure to a battery failure. (C. Franks, plan by 8/30, due by 12/31/97). N:\Environmental\BOI Report 71497.doc redacted NGC 13604 6. Review PM scheduling and content. Add individual battery check to PM in addition to overall set. (L. Henderson by 8/1/97). 7. Investigate UPS internal revision to change fail state. Requires manufacturer agreement and re-design. (S. Johnson by 8/30/97). 8. Evaluate timing allowance on overdue PM work orders based on PM frequency. (B. Lindstrom by 8/30/97). 9. Request follow-up report on transformer failure from Oxychem-Deer Park. (Complete). 10. Develop a procedure for start-up of critical utilities after a power failure. (E. McKinley by 8/30/97). 11. Refine procedures for dealing with excess AMS after plant power failure of this magnitude. (B. Cann by 8/30/97). 12. Review AMS shortstop effectiveness with regard to the catalyst package, timing, and agitation. (B. Cann by 9/30/97) 13. Evaluate radio base station back-up power needs. (McKinley by 9/30/97) Narrative At 0406 on 7/14/97, a power failure occurred at the Deer Park plant. All screens in the control room went dark, and virtually all equipment stopped running. All polys were shortstopped automatically with AMS, and verified in the field by Geon operator. (At about 0420, a call was received from Oxychem's rectifier control room asking if Geon experienced a power failure. Oxychem at that time thought there had been a momentary blip from Houston Lighting & Power). During the course of manually restarting critical equipment, the relief valve on Zebra poly lifted at approx. 0445 for 1 second or less. Subsequent to this first release, Zebra poly relief valve lifted for two more times, one second each, at about 0505 and 0520. . Two operators noticed the local pressure gauge reading on Zebra was 200+ psig just before the second release. The laboratory technician, Jimmy Mitchell, reported the release to all appropriate agencies at 0500. The emergency generator to supply power for critical controls and equipment did not start automatically on initial loss of power. Although most of the power to plant equipment appeared to be restored by 0500, there was still no power in the control room, and all computer screens were inoperative. At 0526, power was restored to the control room by restarting the inverter on the UPS panel for the control room. At that time the poly pressure was seen for the first time since 0400 on the DCS and the reading was 192 psig on Zebra poly. No personnel were injured during the incident. All systems were back in operation by approx. 0545, and procedures were initiated to deal with the excessive amounts of shortstop added to the system as a result of the power failure. Facts Surrounding the Incident 1. Geon operator, verified that all 24 AMS pot levels were OK at 1930 on 7/13/97, the start of the night shift. 2. Three polys had low AMS pot pressures at the start of the shift but were topped up to 350 psig per procedure. N;\Environtnental\BOI Report 7l497.doc NGC 13605 3. Power to the plant was not actually lost, but only a power "dip" occurred, so the emergency generator was not called upon to start. 4. The emergency AMS quick-kill system switches, valves, etc., are PM'd on an annual basis. 5. The rupture discs on Zebra poly are rated for 211 psig; the relief valves are set at 215 psig. 6. When Zebra poly was opened after the incident, the direct steam injection valves were checked and found not leaking by. 7. A meeting with Oxychem engineering and maintenance was held on 7/18/97. Their investigation revealed the power dip was caused by the internal fault of "A2" transformer, which affects the high voltage feeders supplying Geon. The transformer was on a regular and appropriate PM schedule, and is only 12 years old. A bushing failure allowed the transformer to leak oil and create an internal fault. 8. Response from Geon operations personnel, maintenance and technical staff was timely and effective. 9. The base radio station in the control room was inoperative during the event and no backup was available, hampering communications. 10. See Addendum I, UPS Failure Investigation 7/15/97, and Addendum II, Zebra poly #828 Hydrofull Analysis 7/21/97 for additional details. Conclusion A voltage dip on both primary feeders to the Deer Park plant shut down virtually all critical equipment on July 14, 1997. Due to the nature of this event, the UPS system supplying power to the control room computers, instruments, and displays shut down power and, due to failure of the battery set, did not supply backup power as designed. Although incoming line power was restored rather quickly, the UPS "fail state" did not allow it to switch back to line power after it saw the power dip. The requirement at that point is to manually restart the UPS inverter, something the control room personnel were unaware of and not trained for. Without electronic instrumentation and control of computer operated devices, the control operators were "blind" for almost 90 minutes, during which time Zebra poly overpressured, the rupture disc failed, and the relief valve lifted three times. Although the theoretical calculations do not predict either a hydrofull condition nor a runaway reaction, the nature of the overpressure and the behavior of the relief devices during the event indicate a hydrofull condition, but at this time, the data does not conclusively support either cause. N:\EnviromrientaI\B01 Report 71497.doc NGC \3606 From: To : CC: Subj : DPK100: rMCWHIRTERB MCWHIRTERB MCWHIRTERB boi AbD-GTMbUM X "Byron McWhirter" 27-JUL-1997 17:24:56.43 DEER PARK UPS FAILURE INCIDENT - JULY 14, 1997 REVIEW/INVESTIGATIVE MEETING - JULY 15, 1997 MEETING PARTICIPANTS: C FRANKS D HRUSKA S JOHNSON J WHITESIDE S MCGINLEY L HENDERSON G HUBERT - DELTECH REP{UPS MFGR) INCIDENT DESCRIPTION At -0406 on July 14,1997, a power dip was experienced on both primary and secondary power feed to the UPS. In addition, there were three batteries out of the 30 battery backup set that had low voltage, making the total battery voltage too low to operate the UPS.The result of this three point failure was that the UPS was in a zero output mode requiring a manual restart. There was and normally is no one on site during off hours qualified or knowledgeable in starting the UPS. At ~ 0515, S Johnson arrived on site and started the UPS. The consequence of this UPS shutdown was total loss of instrument power to the plant computer systems including Provox as well as business systems. During the Provox outage, the ability to see and control the following items was lost(most valves fail closed and motors fail to off): virtually all electronic instrumentation most electric motors over 25hp all computer controlled valves A NESHAPS release occurred due to the inability to see and react to a pressure problem in one reactor. BACKGROUND NGC 13607 The UPS was installed -3.5 years ago with the Provox computer installation. The UPS has had no previous failure, switching properly during numerous source power problems. UPS preventative maintenance is done on 12 month intervals for system internal checks(last one in January 1997) and battery checks are part of the monthly pm (last done on May 21, 90 days currently allowed for overdue PMs) . The batteries were scheduled for replacement at year end, per manufacturers recommendation (mfg recommendation for PM is annual, we check quarterly). At UPS installation time, the plant substation operated with tie breakers closed between motor control center pairs. Expansions and additions have added load beyond 50% of each MCC requiring the tie breakers to be normally open. This is standard practice at GEON plants. F Zo-fZ. CONCLUSIONS 1. The primary cause of the UPS failure was the battery failure. Battery power is the only way the UPS can stay online in this situation and all others except battery failure. 2. The problem extended due to lack of startup knowledge of off-hour personnel. The need for this was unforseen due to a misunderstanding of the UPS state resulting from a three point failure such as this. It was thought that the the UPS would stay in BYPASS and startup when bypass source power was restored; this was not correct, a manual restart of the inverter is required. 3. The change in normal tie breaker state probably did not affect this incident but does make the UPS more suseptable to loss of both 480v sources as they are both on the same incoming power feeder from Oxy. 4. Addition of an external automatic bypass switch was considered and is not recommended due to complexity and functionality would not cover all possible failures.(cost ~$35k) 5. When the UPS loaded up, one battery failed and prevented the whole set from operating since they are wired in series. RECOMMENDATIONS/CORRECTIVE ACTION See Narrative Report NOC 13608 ADDENDUM 11 p.\ of 5 Zebra # 828 Hydrofull Analysis PRELIMNARY REVIEW - 7/21/1997 On Monday, July 14th at approximately 04:06, there was a critical power failure, which lead to the AMS quick kill system being triggered on all 8 reactors. At approximately 04:45, Richard Rayon and James Peters heard what they thought to be a relief valve that had "popped" and reseated immediately afterwards. Subsequent reliefs were also heard at approximately 05:05 and 05:20. After analysis of the events and data surrounding the incident, it is the general consensus that the Zebra reactor rupture disc blew and relief valve lifted as the result of a hydrofull condition in the reactor. Although the possibility of an uncontrolled reaction does exist, the nature of the release does not support this theory. The following observations were made before, during and after the incident: 1. The AMS pots were confirmed to be full earlier in the shift. 2. The AMS pots were confirmed to be empty after the power failure had triggered the system to activate. 3. There was no residual resin found in the Relief Valve after it was removed from the reactor. 4. There was no slurry found in the stack. 5. The period of relief was approximately! second. 6. The first DSI valve was not leaking (there is a double block). 7. The amount of AMS was sufficient to kill the reaction. 8. The AMS worked sufficiently in the other reactors. 9. Instrumentation was unavailable for approximately 1 hour and 20 minutes. 10. Zebra is equipped with 3 temperature probes - each showed a significantly different temperature after the agitator was restarted. 11. Zebra was not burped after the initial R/V lift. 12. The R/V for slurry or residual VCM at 0615. 13. The AMS pots fired immediately. 14. Calculations show that we had enough residual agitation to properly mix the AMS and subsequently kill the reaction. Ji/cjm/zebra.doc NGC 13609 14 July 1997 Zebra #828 Hydroful] Calculation Poly Input VCM (Fresh and Reclaim) Water Solution Make-up Water Emusifiers/lnitiators Solution Tank Flush Solution Line Flush Seal Water Injection Water DSI Steam Condensate Displacement from Pofy Build-up. Nitroqen AMS Seal Water after Pump Restart Total Volume of Inputs Reaction Shrinkage Actual Pofy Volume Provox Tag FQI102 FQI103 i Time ! Start i End 2:23:15 2:40:15 2:21:45 2:32:45 FI48303Z FIC48301Z FI48007 2:21:45 2:21:45 2:47:30 N/A N/A N/A 4:30:00 4:06:30 4:06:30 3:00:45 N/A N/A N/A 4:45:00 N/A N/A 4:06:30 SmSS SK^EHS Total 0:17:00 0:11:00 0:00:00 0:00:00 0:00:00 0:00:00 1:44:45 1:44:45 0:13:151 N/A N/A N/A 0:15:00 N/A 1:05:30 Rata I Metered Amount Unit Quantity Unit Adjusted Volume j Quantity Volume Notes: , 8500.0 gallons 9220.6 gallons Temp Compensaled from 7QF to 141F 5097.1 gallons 8213-0 gallons Temp Compensated from 7QF to 141 ~ 2100.0 pounds 255.9 gallons Weigh Tank Measured 15.0; gallons gallons Estimated Volume 1 500.0 gallons 505.4 gallons Temp Compensated from 7DF to 141F 200.0 1 gallons 1 202.2 gallons Temp Compensated from 70F to 141F 3.34 gpm 349.9 gallons! 354.9 gallons Temp Compensated from 70F to 141F 5.39 gpm 564.6 gallons | 572.7 gallons Temp Compensated from 70F to 141F 17680 ,N/A N/A N/A #/hr 3904.3 N/A N/A . ~ . w* pounds] n/a! n/a| N/A 475.8 -m>) ii gallons Assumes a baseline of 37800 Ib/lir gallons. Estimated Volume gallons] Estimated Volume gallons Estimated Volume ' :'iK gpm WCTSBiff gallons! gallons Estimated Volume 11 N/A N/A[ N/A 20033.6 gallons ' N/A * " ' N/AJ N/A n/a| --ii n/aT (510.9) gallons N/Aj 20525.0 gallons B&boi&feBSSHBfll 18% Conversion based on Model -H! '1- -- f1- NGC 13610 90 ZEBRA Zebra Poly Trend PROUOX TREND PLOT fl=addtrace 'E=erasetrace' P=preutrace N=nexTtrace ' 5-saue U=`uiew Cun1 j. I1. m, ; (ts i - I M, ' 191.843 62.9395 3 13f- -**=-- 1 TIC503Z 2 PI314Z-1 3 TIC504Z 4 FIC4367Z 5 TIC504Z 6 TIC503Z PU PU PU PU 2IUP SP 0 NUL DGF 0 NUL PSI 0 NUL DGF 0 NUL LBM 0 NUL DGF 0 NUL DGF U+-H UH UH U-H U-H UH Z POLV TEMP POLV PRESS Z JACKET OUTL Z NH3 TO BFL Z JACKET OUTL Z POLV TEMP NGC 13611 5 Jo 93 ZEBRA A6IT Zebra Agitator Anps PROUOX TREND PLOT A=addtrace 'E=erasetface__P=preOtrace N=nexttrace ~S=saue~ U=ui6u" --- cn IK f, ju. .. i 92.8588 150.422 1 i1 IIV i*- 3 II916Z-1 4 TI48036Z-2 5 SIC100Z 6 TI48036Z-1 PU PU PU PU La 0\ K) NUL AMP NUL DGF NUL RPM NUL DGF UH UH UH UH Z AGITATOR BOTTOMS TEMP Z AGIT SPEED BACKUP TEMP 5 From4: TO: CC: Subj: p. 5 f DPK100: :CANWJ "J Brion Cann MCWHIRTERB, MCKINLEYE SCHWARZJ, ALGVAX::SCHONW,CANNJ AMS Quantity 281-930-9426'' 18-JUL-1997 09:53:05.62 Byron and Ed: Bill Schon and I discussed the Zebra Poly incident this morning. In response to the question - Did we have enough AMS to kill the reaction? After running through some numbers, the conclusion was - YES. I estimate that we add approximately 15 gallons (3 pots @ 5 gallons each) when this happens. According to Bill, this is equivalent to 432 moles of AMS compared to 52 moles of initiator (104 moles of free radicals). We could have killed this poly 4 times over even with NO previous initiator consumption. I think it is safe to say that poly was DEAD - this conclusion assumes that all the AMS got into the poly (it did...) and that it was true AMS that we're using (samples are on the way to verify............) . Let me know if you have any questions. Thanks, J.Brion NGC 13613