Back Up and Recovery Testing

Backup and recovery testing are critical to ensuring data integrity and business continuity for critical computerized systems. They are also a hard regulatory requirement in our computer system lifecycle.

Part 11 (21 CFR 11.10 and 11.30) requires that:
“For the availability of computerized systems supporting critical processes, provisions should be made to ensure continuity of the systems in the event of an incident or system failure. This includes implementing adequate backup and recovery measures, as well as providing sufficient system redundancy and failover mechanisms.”

Part 11 also requires that “The backup and recovery processes must be validated in order to ensure that they operate in an effective and reliable manner.”

Similarly, Annex 11 requires that backup and recovery processes be validated to ensure they operate reliably and effectively. Annex 11 also requires that the validation process be documented and includes a risk assessment of the system’s critical processes.

Similar requirements can be found across the GxP data integrity requirements.

The regulatory requirements require that backup and recovery processes be validated to ensure they can reliably recover the system in case of an incident or failure. This validation process must be documented, including a risk assessment of the system’s critical processes.

Backup and recovery testing:

  1. Verifies Backup Integrity: Testing backups lets you verify that the backup data is complete, accurate, and not corrupted. It ensures that the backed-up data can be reliably restored when needed, maintaining the integrity of the original data.
  2. Validates Recovery Procedures: Regularly testing the recovery process helps identify and resolve any issues or gaps in the recovery procedures. This ensures that the data can be restored wholly and correctly, preserving its integrity during recovery.
  3. Identifies Data Corruption: Testing can reveal data corruption that may have gone unnoticed. By restoring backups and comparing them with the original data, you can detect and address any data integrity issues before they become critical.
  4. Improves Disaster Preparedness: Regular backup and recovery testing helps organizations identify and address potential issues before a disaster strikes. This improves the organization’s preparedness and ability to recover data with integrity in a disaster or data loss incident.
  5. Maintains Business Continuity: Backup and recovery testing helps maintain business continuity by ensuring that backups are reliable and recovery procedures are adequate. Organizations can minimize downtime and data loss, ensuring the integrity of critical business data and operations.

To maintain data integrity, it is recommended that backup and recovery testing be performed regularly. This should follow industry best practices and adhere to the organization’s recovery time objectives (RTOs) and recovery point objectives (RPOs). Testing should cover various scenarios, including full system restores, partial data restores, and data validation checks.

LevelDescriptionKey ActivitiesFrequency
Backup TestsEnsures data is backed up correctly and consistently.– Check backup infrastructure health
– Verify data consistency
– Ensure all critical data is covered
– Check security settings
Regularly (daily, weekly, monthly)
Recovery TestsEnsures data can be restored effectively and within required timeframes.– Test recovery time and point objectives (RTO and RPO)
– Define and test various recovery scopes
– Schedule tests to avoid business disruption
– Document all tests and results
Regularly (quarterly, biannually, annually)
Disaster Recovery TestsEnsures the disaster recovery plan is effective and feasible.– Perform disaster recovery scenarios
– Test failover and failback operations
– Coordinate with all relevant teams and stakeholders
Less frequent (once or twice a year)

By incorporating backup and recovery testing into the data lifecycle, organizations can have confidence in their ability to recover data with integrity, minimizing the risk of data loss or corruption and ensuring business continuity in the face of disasters or data loss incidents.

AspectBackup TestsRecovery Tests
ObjectiveVerify data integrity and backup processesEnsure data and systems can be successfully restored
FocusData backup and storageComprehensive recovery of data, applications, and infrastructure
ProcessesData copy verification, consistency checks, storage verificationFull system restore, spot-checking, disaster simulation
ScopeData-focusedBroader scope including systems and infrastructure
FrequencyRegular intervals (daily, weekly, monthly)Less frequent but more thorough
Testing AreasBackup scheduling, data transfer, storage capacityRecovery time objectives (RTO), recovery point objectives (RPO), failover/failback
ValidationBackup data is complete and accessibleRestored data and systems are fully functional

Business Continuity Planning

The pharmaceutical regulations call, repeatedly for business continuity plans. For example, the FDA calls for fairly significant requirements for Medically Necessary Products:

Medically necessary drug products and their components are manufactured all over the world. An emergency situation anywhere in the world thus might affect the availability of drug products in the United States and result in drug shortages. Emergency preparedness for situations that could result in high employee absenteeism is an important goal for manufacturers of drug products and their components. For example, in an influenza pandemic, widespread human outbreaks of illness would be expected in the United States and around the world, resulting in widespread high absenteeism that could hinder normal production activities and cause shortages in the supply of drug products, packaging materials, and drug components. It is therefore vital for industry to prepare before an emergency situation occurs and to develop plans to ensure continuity of operations during emergencies (including, for example, an influenza pandemic, natural disaster, or personnel issue) that would prevent a significant portion of the work force from reporting. It is especially important for manufacturers of finished drug products to be aware of their suppliers’ and contractors’ responses to personnel shortages and, when appropriate, work with them to ensure the availability of high quality materials and services that contribute to the manufacture of MNPs.

FDA, Guidance for Industry Planning for the Effects of High Absenteeism to Ensure Availability of Medically Necessary Drug Products

You can find less definitive requirements throughout the various health authorities’ regulations and guidances.

So what do we mean by business continuity?

Business continuity is the holistic management process that ensures operations continue and that products and services are delivered at predefined levels (e.g. no shortages, no halt to an ongoing clinical trial). This approach is aligned with ISO 22301 Business Continuity Management Systems.

Business continuity management is an ongoing process based on the plan-do-check-act methodology that is made up of 4 key elements:

  • Emergency Action and Response Plans
  • Disaster Recovery Plans
  • Crisis Management Plans
  • Business Continuity Plans

Emergency Action Plans

An emergency action plan is designed to respond to an emergency with mitigating procedures to protect, secure and evacuate people to safety. This is more an OSHA thing; chances are your average Quality unit doesn’t end up owning it. Unless you have no HS&E unit, and then you write one.

This plan includes procedures for detecting, warning, and responding to specific potential emergencies such as fire, severe weather, earthquake, medical emergencies, workplace violence, and other potential threats.

Disaster Recovery Plan

Disaster recovery plans are designed to recover from a disaster, usually related to equipment, infrastructure, and information technology. Something big goes boom, how do you restore this vital support system or equipment as soon as possible and minimize downtime and loss of data. Very important for computer system lifecycle, disaster recovery plans should include specific plans for recovery functions, resumption strategies, critical personnel, equipment, services, and external and internal communications.

Crisis Management Plans

Crisis management is all about planning and mitigating situations that have risk, and are usually a lot of management of communications internally and externally. This includes with regulators, health care providers, etc. When we implement SOPs for health authority notifications we are engaging in crisis management planning.

Business Continuity Plans

Business continuity planning identifies and plans for disasters to events that could negatively an organization’s business functions, objectives, income, reputation, and ultimate survival. This planning takes place in advance of the potential disasters or events that could harm an organization. It takes potential disasters and events into consideration with their effects on suppliers, vendors customers, and the organization’s other stakeholders.

In a GxP environment, we are looking at the potential impact of disasters on drug supply and clinical study outcomes (amongst other key activities).

The BCP is all about minimizing the effects of the disaster or event on the organization and returning to normal operations as soon as possible.

These Plans are Interrelated

All four plans are interrelated and should be coordinated. The plans can be combined, but as there are usually very different owners they are often separated.

Documented Plans

The business continuity planning process should result in formal, documented plans that serve as a reference guide in the event of a disaster or event. The existence of the business continuity plans should be well communicated, with individuals with responsibilities having ready access and additional training.

Applying the Risk Management Process

The Business Continuity process should leverage existing risk assessments and sit around it.

Select Team

The team should be multifunctional and very knowledgeable about the organization’s business and the risks it faces. This should be a permanent team, not ad hoc, as this is a living process. You can always bring in ad hoc members for specific questions.

Define Context, Purpose, Scope

At a minimum you are tackling the disruption to product supply and cessation of critical GxP data but there may be other business requirements to tackle. Make sure everyone agrees on these.

Define Terminology

Make sure everyone is on the same page with just what disaster, event, crisis, stakeholder, and business continuity plan (and other important concepts) are.

Agree on the scales for likelihood and severity.

Critical Function Assessment

Identify the business functions that are sensitive to downtime, fulfill regulatory obligations and are vital for maintaining product supply.

Threat Assessment

Identify the threats to the performance of the critical functions.

Identify Hazards and Risks

There are three major categories of hazards:

  • Natural Hazards
    • Meteorological
    • Geological
    • Biological
  • Human-Caused Hazards
    • Accidents
    • Intentional acts
  • Technological Hazards
    • Information technology
    • Utility
    • Fire/explosion
    • Hazardous material
    • Supply Chain interruption

Utilize a risk matrix to assess the likelihood and severity of the identified hazards and risks.

Develop Business Continuity Plan(s)

After the hazards and risks have been identified, the impact understood and the risks assessed it is time to develop the business continuity plan (BCP). The BCP allows the organziation to survive the event or disaster with minimal disruption. The BCP focuses on mitigating the consequences of the event or disaster that could not be prevented. Recovery strategies for these cosnequences are determined, developed and become part of the BCP.

When many potential risks have been identified, use the risk score to prioritize.

BCPs cover management commitment, team ientification, team responsibilities, mitigation plans, recovery strategies, training, testing and evaluation and continious improvement. Basically the same thing any good plan does.

Mitigation plans are intended to lessen the negative effectis of an event or disaster.

Provide appropriate awareness training to everyone impacted, with more substantial trining to the BCP team.

Verify it periodically and ensure it is continues to be relevant.

Whenever relevant, procceduralize these BCP instructions.