Requirement to submit a test plan for control system upgrades

This case study involves a situation where Meridian Energy Limited (Meridian) was installing multiple frequency keeping (MFK) software into its SCADA control system at Benmore, when Benmore’s generation suddenly dropped by 200MW. Meridian had not submitted a test plan to the system operator.

Meridian has consented to the Electricity Authority publishing the details of this case study.

Code provisions

Clause 2(6)(a)(iii) of Technical Code A of Schedule 8.3 provides:

2 General requirements …

(6) Each asset owner must provide a commissioning or test plan in accordance with subclauses (7) or (8) (as the case may be) in the following situations:

(a) when changes are made to assets that alter any of the following at the grid interface: …

(iii) a control system, including a change to a control system setting: …

Part 1 definition:
 control system means equipment that adjusts the output voltage, frequency, MW or reactive power (as the case may be) of an asset in response to certain aspects of common quality such as voltage, frequency, MW or reactive power, including speed governors and exciters

Circumstances

On 10 June 2013, Meridian was installing MFK software scripts into its SCADA control system at Benmore. Meridian undertook testing of the software change and did a risk analysis using its plant control process and did not identify any potential risk of changes to its generation whilst implementing this software change. Therefore, Meridian did not submit a test plan to the system operator, and did not inform the system operator of this software implementation. However, as these changes did alter the control system, the Code required Meridian to submit a test plan to the system operator.

At 12:20, Meridian’s implementation team was part way through switching to the updated software on one of two parallel servers, when Benmore's generation unexpectedly dropped by 200MW.

Meridian’s generation controller quickly re-issued a new set point in SCADA to return the station back to its previous generation level.

At this point, Meridian was part way through the software implementation and was concerned that proceeding with the second server would result in another loss of generation. Meridian decided to abort implementing the software switch and return to the original code on the first server.

In executing the return to the first server, generation dropped by 200MW again. Again, the generation controller re-issued the set point to return the generation back up to the previous level and called the implementation team to investigate.

The system operator then called the generation controller, requesting that work cease. As mentioned above, the system operator was not aware of this SCADA software implementation.

Meridian analysed the event and discovered that the station control logic software had issued a set point error of zero upon takeover of the new software. Therefore, the momentary reduction of generation experienced was due to the unanticipated operation of the control system software when switching to the updated code.

Analysis

The system operator believed that if Meridian had submitted a test plan, it would have helped the system operator to plan to comply with its principal performance obligations. However, Meridian did not anticipate that generation drops would result from this work. Meridian believed that even if it had submitted a test plan, it would not have highlighted the potential risk to the system, instead it would only have identified that work was occurring.

The system operator does not receive many test plans related to software changes. This was the first reported breach of this provision.

Meridian sought guidance from the system operator to resolve this issue. Both parties agree that not every SCADA system change will be a change to a control system and therefore warrant a test plan. Submitting a test plan for every SCADA system change could result in an over-submission of test plans which could be counter-productive for the system operator.

Therefore, it is not necessary for participants to submit test plans for all changes. Clause 2(6)(a)(iii) only requires participants to submit test plans where these changes “alter” the control system at the grid interface. This means it only applies to alterations to equipment that adjusts the output voltage, frequency, MW or reactive power (as the case may be) of an asset.

Steps taken to prevent recurrence

Meridian identified several areas to improve compliance in similar situations:

  • Meridian identified that its plant control process could be further improved to ensure that software changes are subject to a robust engineering, information, and communications technology review to check that it has identified all foreseeable risks and put effective risk mitigation measures in place.
  • Meridian also employs a roll-out process to progressively switch individual generating units one at a time onto any new SCADA code to minimise the MW affect of any unanticipated consequence.
  • Further, Meridian’s now uses its hydro test plan template for SCADA system changes. Meridian has modified the template to include check boxes to ensure SCADA control system operational engineering signoff.
  • Meridian has also take steps to communicate the learnings from this event across personnel in operations and project teams.

Compliance Committee decision

The Compliance Committee decided to take no further action on the breach of clause 2(6)(a)(iii) of Technical Code A of Schedule 8.3 of the Code and issued a warning letter.

Lessons learned

It is not necessary for participants to submit test plans for all changes. However, each participant that makes changes to control systems needs to carefully assess whether it is required to submit a test plan to the system operator.

If an asset owner is unsure about the possible implications of the planned change it should contact the system operator for advice or test the change in a safe environment, before live implementation.