Digging for WORM (2/2)

In the previous blog entry, (/blog-post/digging-for-worm/) we introduced the conflict of legislative language and the technology used to fulfill it. Specifically regarding MIFID2, the confusion around the use of WORM (“Write Once, Read Many”) storage to fulfill the record-keeping requirements of MIFID2. We also established that the term “WORM” was purposefully left out of the legislation as ESMA (the pan-EU regulator for MIFID2) wanted record-keeping to evolve with storage technology and not be limited to any type by specific terms.

With this in mind, what is compliant storage and how do we get to it?

Let’s look at the basic requirements for electronic records:

1. Keep the records for five years at a minimum.
2. Ensure the data is kept on a durable medium that prevents deletion or editing.

For Part 1, the answer is very easy. Simply keep track of the create date of a financial record or communication for five years. All electronic file systems are capable of this as it is essential to their own file management.

Part 2 is where things get tricky. Typically, any user with the proper rights can delete or edit the data in a file system. For the management of financial and legal risk, this cannot be tolerated by regulators for the simple reason being that if questioned, individuals and entities may delete or alter data to hide information that may be damaging to them. (Referred to in legal circles as “spoliation”)

How do technology storage vendors get to a state where they can prevent users (IT or Business) from deleting or altering their very own records? The answer lies in hardware and software protocols that prevent data deletion or edit.

Early in enterprise computer technology history, data was simply stored on a medium that could not be altered or deleted, thus the “Write Once” part of WORM. This was for example, certain magnetic tapes and optical disks. From the 1960’s to early 2000’s – this was manageable but with the tech boom of the late 1990’s and 2000’s – storage became problematic as increasing digitalization of business processes required ever increasing amounts of storage with a non-linear growth rate. Further, as business records became ever more digital, especially with the advent of e-mail replacing typical paper-based business correspondence, the need for the collection of documents from digital media became a problem as older media-based systems were slow to produce data when required for audit, regulatory or legal discovery. (Because they were on manual or robotic media management systems and often had no or poor text and data indexing)

From the slow retrieval of newly digitized records – the “Read Many” part of WORM began to dominate the storage development lifecycle with the introduction of new WORM systems that used standard magnetic hard drives in large arrays. With their use, data could be retrieved many times and much more quickly than with the older physical media systems. Data deletion and edit of data on the magnetic hard drives were prevented with a mix of specialized “firmware” and the hard drives and software that controlled the arrays. Along with the new storage on magnetic drives, software vendors built “archiving” software that indexed the data that was being stored so that it could be searched for and quickly retrieved from the arrays.

With “WORM” technology firmly developed and in place by the close of the first decade of the new century, regulators began to see it as the default solution for digital records management for financial services companies. But as noted in the first part of this blog post, it was never often mentioned specifically.

So now we are in the second year of MIFID2 and many EU firms are still questioning why they need WORM based storage when they feel that backup tapes and auditable file and database records are available. The answer is that these methodologies don’t hold up to the Write Once, Read Many principal in its entirety.

File and Database Records with Audit and Access Controls

While you can store files on hard drives or arrays of hard drives with robust file access auditing and access control to prevent deletion or edit, someone, somewhere in the organization will have access to do both. Typically the administrator of the system. For regulatory purposes, this level of access is not enough. What if the System Administrator’s account were compromised or worse, they are asked to spoliate data and do so?

Backup Systems

In modern enterprises, backups consist of physical copies on tape, optical or other hard drive arrays. Optical systems can indeed use write-once media but are hindered in term of scale for large enterprises and lack rich content indexing for multiple content types making access and retrieval difficult and slow. Tape based systems share the same scaling and indexing problems as Optical media and being a primary cold storage mechanism, “cycling” is often used to manage the amount of media a firm must manage. Tape cycling is a method that reuses the oldest tapes for newer data thus overwriting older data.

While the combination of the above methods can make for robust enterprise data and backup management, neither individually or together do they create a robust WORM system that prevents data spoliation for legal and regulatory purposes. When pressed, regulators while not specifying the technology directly, prefer to see financial firms use WORM based systems. Evidence for this is strongest in the United States where WORM based systems are the de-facto requirement as outlined in numerous regulatory filings called “AWC’s” (Letter of Acceptance, Waiver and Consent) by the USA financial regulator, FINRA.

The Road Ahead

The requirement for WORM like storage has been in place longer in the USA than in the EU but MIFID2 changed that on January 4, 2018. It’s almost hard to believe that it was an option for EU financials to keep data in WORM storage under MIFID1. (To be fair, MIFID2 addressed this because of the financial crisis of 2008-2010.)

Now that we’ve established that WORM really is the proper way for EU financials to archive their electronic business records and communications, Arkivy Cloud is an ideal WORM solution for those that are required to store regulated data as it provides cost efficient and indexed WORM storage via Amazon’s cloud as a Service thus saving companies the headaches and costs of setting up and managing their own solutions. For an inquiry or sales consultation, please feel free to contact us by phone or email.