Computer Software Assurance Draft

The FDA published on 13-Sep-2022 the long-awaited draft of the guidance “Computer Software Assurance for Production and Quality System Software,” and you may, based on all the emails and posting be wondering just how radical a change this is.

It’s not. This guidance is just one big “calm down people” letter from the agency. They publish these sorts of guidance every now and then because we as an industry can sometimes learn the wrong lessons.

This guidance states:

Determine intended use
Perform a risk assessment
Perform activities to the required level

I wrote about this approach in “Risk Based Data Integrity Assessment,” and it has existed in GAMP5 and other approaches for years.

So read the guidance, but don’t panic. You are either following it already or you just need to spend some time getting better at risk assessments and creating some matrix approaches.

Warning Letter for Aurolife demonstrates failures in process validation

The FDA commented in a Warning Letter to Aurolife Pharma that the manufacturer lacked data showing that the process was in “state of control” before batch release. The 483 pointed to the FDA’s guidance document Process Validation: General Principles and Practices, and found the company lacks a state of control of the process, which comes back to change control.

They also found major deficiencies in their OOS and cleaning programs.

ASQ Audit Conference – Day 2 Morning

Jay Arthur “The Future of Quality”

Starts with our “Heroes are gone” and “it is time to stand on our two feet.”

Focuses on the time and effort to train people on lean and six sigma, and how many people do not actually do projects. Basic point is that we use the tools in old ways which are not nimble and aligned to today’s needs. The tools we use versus the tools we are taught.

Hacking lean six sigma is along a similar line to Art Smalley’s four problems.

Applying the spirit of hacking to quality.

Covers valuestream mapping and spaghetti diagrams with a focus on “they delays in between.” Talks about how control charts are not more standard. Basic point is people don’t spend enough time with the tools of quality. A point I have opinions on that will end up in another post.

Overcooked data versus raw data – summarized data has little or no nutritional value.

Brings this back to the issue of lack of problem diagnosis and not problem solving. Comes back to a need for a few easy tools and not the long-tail of six sigma.

This talk is very focused on LSS and the use of very specific tools, which seems like an odd choice at an Audit conference.

“Objectives and Process Measures: ISO 13485:2016 and ISO 9001:2015” by Nancy Pasquan

I appreciate it when the session manager (person who introduces the speaker and manages time) does a safety moment. Way to practice what we preach. Seriously, it should be a norm at all conferences.

Connects with the audience with a confession that the speaker is here to share her pain.

Objective – where we are going. Provide a flow chart of mission/vision (scope) ->establish process -> right direction? -> monitor and measure

Objectives should challenge the organization. Should not be too easy. References SMART. Covers objectives in very standard way. “Remember the purpose is to focus the effort of the entire organization toward these goals.” Links process objectives to the overall company objectives.

Process measures are harder. Uses training for an example. Which tells me adult learning practice is not as much as the QBOK way of thinking as I would like. Kilpatrick is a pretty well-known model.

Process measures will not tell us if we have the right process is a pretty loaded concept. Being careful of what you measure is good advice.

“Auditing Current Trends in Cleaning Validation” by Cathelene Compton

One of the trends in 2019 FDA Warning letters has been cleaning. While not one of the four big ones, cleaning validation always seems relevant and I’m looking forward to this presentation.

Starting with the fact that 15% if all observations on 483 forms related to leaning validation and documentation.

Reviews the three stages from the 2011 FDA Process Validation Guidance and then delvers into a deeper validation lifecycle flowchart.

Some highlights:

Stage 1 – choosing the right cleaning agent; different manufacturers of cleaning agents; long-term damage to equipment parts and cleaning agent compatibility. Vendor study for cleaning agent; concentration levels; challenge the cleaning process with different concentrations.

Delves more into cleaning acceptance limits and the importance of calculating in multiple ways. Stresses the importance of an involvement of a toxicologist. Stresses the use of Permitted Daily Exposure and how it can be difficult to get the F-factors.

Ensure that analytical methods meet ICHQ2(R1). Recovery studies on materials of construction. For cleaning agent look for target marker, check if other components in the laboratory also use this marker. Pitfall is the glassware washer not validated.

Trends around recovery factors, for example recoveries for stainless tell should be 90%.

Discusses matrix rationales from the Mylan 483 stressing the need to ensure all toxicity levels are determined and pharmaceological potency is there.

Stage 2 all studies should include visual inspection, micro and analytical. Materials of construction and surface area calculations and swabs on hard to clean or water hold up locations. Chromatography must be assessed for extraneous peaks.

Verification vs verification – validation always preferred.

Training – qualify the individuals who swab. Qualify visual inspectors.

Should see campaign studies, clean hold studies and dirty equipment hold studies.

Stage 3 – continuous is so critical, where folks fall flat. Do every 6 months, no more than a year or manual. CIP should be under a periodic review of mechanical aspects which means requal can be 2-3 years out.

Considerations when validating and auditing algorithms

The future is now. Industry 4.0 probably means you have algorithms in your process. For example, if you aren’t using algorithims to analyze deviations, you probably soon will.

And with those algorithms come a whole host of questions on how to validate and how to ensure they work properly over time. The FDA has indicated that ““we want to get an understanding of your general idea for model maintenance.” FDA also wants to know the “trigger” for updating the model, the criteria for recalibration, and the level of validation of the model.”

Kate Crawford at Microsoft speaks about “data fundamentalism” – the notion that massive datasets are repositories that yield reliable and objective truths, if only we can extract them using machine learning tools. It shouldn’t take much to realize the reasons why this trap can produce some very bad decision making. Our algorithm’s have biases, just as human beings have biases. They are dependent on the data models used to build and refine them.

Based on reported FDA thinking, and given where European regulators are in other areas, it is very clear we need to be able to explain and justify our algorithmic decisions. Machine learning in here now and will only grow more important.

Ask an Interesting Question

The first step is to be very clear on why there is a need for this system and what problem it is trying to solve. Having alignment across all the stakeholders is key to guarantee that the entire team is here with the same purpose. Here we start building a framework

Get the Data

The solution will only be as good as what it learns from. Following the common saying “garbage in, garbage out”, the problem is not with the machine learning tool itself, it lies with how it’s been trained and what data it is learning from.

Explore the Data

Look at the raw data. Look at data summary. Visualize the data. Do it all again a different way. Notice things. Do it again. Probably get more data. Design experiments with the data.

Model the Data

The only true way to validate a model is to observe, iterate and audit. If we take a traditional csv model to machine learning, we are in for a lot of hurt. We need to take the framework we built and validate to it. Ensure there are emchanisms to observe to this framework and audit to performance over time.

Computer verification and validation – or what do I actually find myself worrying about every day

voting_software — xkcd “Voting Software” https://xkcd.com/2030/

This xkcd comic basically sums up my recent life. WFI system? Never seems to be a problem. Bioreactors? Work like clockwork. Cell growth? We go that covered. The list goes on. And then we get to pure software systems, and I spend all my time and effort on them. I wish it was just my company, but lets be frank, this stuff is harder than it should be and don’t trust a single software company or consultant who wants to tell you otherwise.

I am both terrified and ecstatic as everything moves to the cloud. Terrified because these are the same people who can’t get stuff like time clocks right, ecstatic because maybe when we all have the exact same problem we will see some changes (misery loves company, this is why we all go to software conferences).

So, confessional moment done, let us turn to a few elements of a competent computer systems validation program (csv).

Remember your system is more than software and hardware

Any system is made up of Process, Technology, People and Organization. All four need to be evaluated, planned for, and tested every step of the way. Too many computer systems fall flat because they focus on technology and maybe a little process.

Utilize a risk based approach regarding the impact of a computer system impact on product quality, patient and consumer safety, or related data integrity.

Risk assessments allow for a detailed, analytical review of potential risks posed by a process or system. Not every computer system has the same expectations on its data. Health authorities recognize that, and accept a risk based approach. This is reflected across the various guidances and regulations, best practices (GAMP 5, for instance) and the ISOs (14971 is a great example).

Some of the benefits of taking this risk based approach include:

Help to focus verification and validation efforts, which will allow you to better focus of resources on the higher-risk items
Determine which aspects of the system and/or business process around the system, require risk mitigation controls to reduce risk related to patient safety, product quality, data integrity, or business risk
Build a better understanding of systems and processes from a quality and risk-based perspective

Don’t short the user requirements

A good user requirement process is critical. User requirements should include, among other things:

Technical Requirements: Should include things like capacity, performance, and hardware requirements.
System Functions: Should include things like calculations, logical security, audit trails, use of electronic signature.
Data: Should describe the data handling, definition of electronic records, required fields.
Environment: Should describe the physical conditions that the system will be required to operate in.
Interface: What and how will this system interface with other systems
Constraints: discuss compatibility, maximum allowable periods for downtime, user skill levels.
Lifecycle Requirements: Include mandatory design methods or special testing requirements.

Evaluate each of people, process, technology and organization.

This user requirement will be critical for performing a proper risk assessment. Said risk assessment is often iterative.

Build and test your system to mitigate risk

Eliminating risk through process or system redesign
Reduce risk by reducing the probability of a failure occurring (redundant design, more reliable solution)
Reduce risk by increasing the in-process detectability of a failure
Reduce risk by establishing downstream checks or error traps (e.g., fail-safe, or controlled fail state)
Increased rigor of verification testing may reduce the likelihood by providing new information to allow for a better assessment

After performing verification and validation activities, return to your risk assessment.

Apply a lifecycle approach once live

Apply proper change management
Perform periodic reviews of the system. This should include: current range of functionality, access and training, process robustness (do the current operating procedures provide the desired outcome), incident and deviation review, change history (including upgrades), performance, reliability, security and a general review of the current verified/validated state.
Ensure the risk assessment is returned to. On a periodic basis return to it and refresh based on new knowledge gained from the periodic review and other activities.

Do not separate any of this from your project management and development methodology

Too many times I’ve seen the hot new development lifecycle consider all this as an after thought to be done when the software is complete. That approach is expense, and oh so frustrating