March, 2021: In continuation of our earlier press statements dated February 24, 25, and 26th, 2021, further clarifications are given below:
With respect to the incident that occurred on February 24, 2021, NSE wishes to state the following:
NSE’s primary data centre is in BKC, a Near Disaster Recovery (NDR) site is maintained in Kurla, and the disaster recovery (DR) site is in Chennai. There is synchronous data replication between our primary site in BKC and NDR site to ensure no data loss in case of primary site failure, and asynchronous replication to our DR site in Chennai which is designed to take over with zero data loss in case of disaster at the primary site.
Between our primary and NDR sites, NSE has multiple telecom links with two service providers to ensure redundancy. On February 24, 2021 we had instability in links from both service providers primarily due to digging and construction activity along the path between the two sites. The replication to NDR is designed such that in the event of the links between primary and NDR getting cut, the primary continues operations without any direct effect. Post earlier link failures in February 2021, operations continued without any interruption.
However, on February 24th, post link failure, we saw unexpected behaviour of the Storage Area Network (SAN) system, with the primary SAN becoming inaccessible to the host servers. This resulted in the risk management system of NSE Clearing and other systems such as clearing and settlement, index and surveillance systems becoming unavailable.
While there was no impact on the trading system, given that the risk management system was unavailable, allowing trading to continue on NSE posed an unacceptable risk, and hence trading had to be halted.
The SAN is a fault tolerant system that was designed to function seamlessly even in the event of telecom link failures between primary and NDR copies. One of the features of SANthat was deployed in October 2020 was designed to provide not just zero data loss but also zero down time. Before deployment, the system was tested against various scenarios including link failures and functioned properly. However, on 24th February, post link failure, the SAN system at the primary data centre stopped functioning, which was completely unexpected. Subsequent incident analysis showed that the problem was caused by failover logic implemented by the vendor which did not conform to NSE’s stated design requirements, coupled with issues in the configuration done by the SAN vendor that triggered the failover logic. We note that the specific failure logic used by the vendor is not documented, was not communicated to NSE, and was not appropriate for NSE’s setup. The resultant SAN failure led to the incident on February 24th.
Post halting of trading on NSE, we considered all the available alternatives on hand, including invocation of DR, to decide on the course of action that would bring up the market at the earliest with least disruption to market participants and post evaluation, a decision was taken to bring up the systems at the primary site. NSE regularly tests its DR readiness in line with SEBI regulations wherein quarterly drills are conducted and live trading sessions from DR site are conducted twice a year. Further, SEBI has also discussed and agreed with all MIIs on the decision-making process and timeline for the DR move going forward.
NSE was continuously working on resolution of the problem and once the same was resolved, NSE made an announcement with respect to re-opening of the markets to its members at 3:17pm. This communication was done only after there was visibility and clarity on resumption of services and any prior communication would not have been appropriate. We reiterate that we could not have communicated sooner because we did not have the clarity and visibility that was important for making an announcement.
There are various steps that have already been taken and others that are under implementation to address the SAN and telecom link issues. We had already placed orders in January for two additional telecom provider links and have removed the SAN software that caused the incident. We are also exploring alternate solutions to de-risk dependency of critical applications to a single storage device.
The interoperability framework was put in place in 2019 for providing capital and operational efficiencies by enabling market participants to consolidate their clearing and settlement under one clearing corporation while trades could be executed on any exchange. The interoperability design was done by all MIIs under the guidance and supervision of SEBI. As part of the interoperability design, all the 3 clearing corporations have set up their slave systems of the primary systems in other exchanges to facilitate risk management.
On February 24th, NSE Clearing’s (NCL) risk management system (RMS) at BSE and MSEI was functioning and cleared trades executed on BSE and MSEI within the collateral levels available at the time NCL’s RMS at the primary site became unavailable. Updation of collateral was not available as part of the design which is being addressed as part of the strengthening of certain aspects of interoperability that all MIIs are collectively working on with SEBI.
Although trading at NSE was halted, trading continued on BSE and MSEI and volumes on BSE were twice the average daily volumes on BSE after excluding any block trades. Also brokers were able to close out open positions on NSE at BSE. On February 24th, 97% of trades in the cash market and 87% of the trades in equity derivatives on BSE were cleared by NSE Clearing. Out of the BSE trades that were cleared by NSE Clearing, 64% of the regular cash market trades (excluding block deals) and 56% of the equity derivatives trades took place between 11:40 am and 3:30 pm when trading on NSE was closed. Also, on February 24th, between 10:06 AM and 3:30 PM when NSE Clearing’s primary risk management system was not operational, approximately Rs. 8,100 crores of cash market trades, Rs. 258,000 crores of notional value of equity derivatives trades and Rs. 16,700 crores of currency derivatives trades on BSE and approximately Rs. 700 crores of currency derivatives trades on MSEI were risk managed by NSE Clearing’s slave risk management system at BSE and MSEI.
