Layering metrics

We have these quality systems with lots of levers, with interrelated components. And yet we select one or two metrics and realize that even if we meet them, we aren’t really measuring the right stuff nor are we driving continious improvement.

One solution is to create layered metrics, which basically means drill down your process and identify the metrics at each step.

Lots of ways to do this. An easy way to start is to use the 5-why process, a tool most folks are comfortable with.

So for example, CAPA. It is pretty much agreed upon that CAPAs should be completed in a timely manner. That makes this a top level goal. Unfortunately, in this hypothetical example, we are suffering a less than 100% closure goal (or whatever level is appropriate in your organization based on maturity)

Why 1Why was CAPA closure not 100%
Because CAPA tasks were not closed on time.

Success factor needed for this step: CAPA tasks to be closed by due date.

Metric for this step: CAPA closure task success rate
Why 2Why were CAPA tasks not closed on time?
Because individuals did not have appropriate time to complete CAPA tasks.

Metric for this step: Planned versus Actual time commitment
Why 3Why did individuals not have appropriate time to complete CAPA tasks?
Because CAPA task due dates are guessed at.

Metric for this step: CAPA task adherence to target dates based on activity (e.g. it takes 14 days to revise a document and another 14 days to train, the average document revision task should be 28 days)
Why 4Why are CAPA task due dates guessed at?
Because appropriate project planning is not completed.

Metric for this step: Adherence to Process Confirmation
Why 5Why is appropriate project planning not completed?
Because CAPAs are always determined on the last day the deviation is due.

Metric: Adherence to Root Cause Analysis process

I might on report on the top CAPA closure rate and 1 or 2 of these, and keep the others in my process owner toolkit. Maybe we jump right to the last one as what we report on. Depends on what needs to be influenced in my organization and it will change over time.

It helps to compare this output against the 12 system leverage points.

Donella Meadows 12 System Leverage Points

These metrics go from 3 “goals of the system” with completing CAPA tasks effectively and on time, to 4 “self organize” and 5 “rules of the system.” It also has nice feedback loops based on the process confirmations. I’d view them as potentially pretty successful. Of course, we would test these and tinker and basically experiment until we find the right set of metrics that improves our top-level goal.

Conducting an ACORN test on my mission statement

Here is my first draft at a mission statement developed after reviewing my SWOT and doing a quick brainstorming exercise asking myself some what, why and how questions:

The mission of my blog is to encourage a thoughtful life as a quality professional. I want to ask and explore questions related to building a quality culture and utilizing system thinking to spark a dialogue with my peers. I am to become a leader in quality both internal to my own company and externally to the wider profession.

An acorn

The ACORN test is a check on a mission or project charter goal to determine if it well defined. I am drawing from The Quality Toolbox, but this is a tool you can find all over the internet.

QuestionThoughts
AAccomplishment. Does the goal describe results rather than behaviors? The goal here is to write. Secondary to that I want to spark a dialogue and also be invited to conferences and other speaking opportunities.

The mission statement as is could use some tightening here.
CControl. Does the team’s actions determine whether not the goal is accomplished? If the way the mission is articulated primarily depends on others outside the team, consider rewriting it.
I am a team of one. Everything within my mission statement is in my own power. I can write, and publish and share. Publicizing my posts is within my control.
OOnly Objective. If this and only this was accomplished would it be enough?

O is sometimes also phrased as “Overall Objective” — Make sure that the mission truly captures the organization’s reason for being. Avoid writing a mission that is simply a subgoal of that overall purpose.
My only goal is thinking aloud. I am not trying to sell anything, nor am I concerned about finding a job (I work in pharmaceutical quality in Boston, getting a new job involves going to Kendall square and waving a resume around)
RReconciliation. Will accomplishing this goal prevent another group within the organization from accomplishing its goal? Does anyone else have this goal?
Various units should be working in harmony to achieve the overall organization mission.
As an individual, I do not need to worry about other groups. However it is important for my mission statement for this blog not to interfere or conflict with any of my other goals.

This mission statement is aligned to my personal and professional goals. In fact it helps further several of them.
NNumbers. Can this goal be measured?Blogging has several measurements built in – views, visitors, likes, shares and comments.

I can also measure other things like invitations to speaking opportunities, questions directed to me, and others.

An overall mission statement must pass all give parts of the ACORN test in order to be well defined. In my case I pass but need some tightening in accomplishment.

Data, and all that jazz

As  we all try to figure out just exactly what Industry 4.0 and Quality 4.0 mean it is not an exaggeration to say “Data is your most valuable asset. Yet we all struggle to actually get a benefit from this data and data integrity is an area of intense regulatory concern.

To truly have value our data needs to be properly defined, relevant to the tasks at hand, structured such that it is easy to find and understand, and of high-enough quality that it can be trusted. Without that we just have noise.

Apply principles of good master data management and data integrity. Ensure systems are appropriately built and maintained.

Understand why data matters, how to pick the right metrics, and how to ask the right questions from data. Understand correlation vs. causation to be able to make decisions about when to act on analysis and when not to is critical.

In the 2013 article Keep Up with Your Quants, Thomas Davenport lists six questions that should be asked to evaluate conclusions obtained from data:

1. What was the source of your data?

2. How well do the sample data represent the population?

3. Does your data distribution include outliers? How did they affect the results?

4. What assumptions are behind your analysis? Might certain conditions render your assumptions and your model invalid?

5. Why did you decide on that particular analytical approach? What alternatives did you consider?

6. How likely is it that the independent variables are actually causing the changes in the dependent variable? Might other analyses establish causality more clearly?

Framing data, being able to ask the right questions, is critical to being able to use that data and make decisions. In the past it was adequate enough for a quality professional to have a familiarity with a few basic tools. Today it is critical to understand basic statistics. As Nate Silver advises in an interview with HBR. “The best training is almost always going to be hands on training,” he says. “Getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth.”

Understanding data is a key ability and is necessary to thrive. It is time to truly contemplate the data ecosystem as a system and stop treating it as a specialized area of the organization.

Contamination Control, Risk Management and Change Control

Microbiologists won’t be sequestered in the laboratory, running samples and conducting environmental testing, once the revisions proposed for Annex 1 of the EU and Pharmaceutical Inspection Cooperation Scheme (PIC/S) GMP guides take effect, Annex 1 rapporteur Andrew Hopkins said Oct. 15.

They will have a broader role that includes conducting risk assessments to ensure that sterile products are made as contamination-free as possible, said Hopkins, who is an inspector for the UK Medicines and Healthcare products Regulatory Agency.

Pink Sheet “EU GMP Annex 1 Would Give Microbiologists A Greater Role In Sterility Assurance, Rapporteur Says

Contamination Control is a fairly wide term used to mean “getting microbiologists out of the lab” and involved in risk management and compliance. Our organization splits that function off from the QC Microbiology organization but there are many models for making it work.

Risk Management is a major part of the new Annex 1, and what they are driving at are good risk assessments with good risk mitigation that involve the microbiologists.

living risk assessments

This is really what is meant by a contamination control strategy which considers the product and process knowledge and skills in pharmaceutical product manufacturing and GMP/ cGMP compliance under the auspices of a Pharmaceutical Quality System (Q10) together with initiatives of Quality by Design (Q8) and Quality Risk Management (Q9).

From this strategy comes:

  • Targeted/ risk based measures of contamination avoidance
  • Key performance indicators to assess status of contamination control
  • A defined strategy for deviation management (investigations) and CAPA

environmental monitoring

When it comes to change management, one of the easiest places to go wrong is to forget to bring the microbiologist in to changes. Based on your strategy you can determine change changes require their assessment and include it in the tool utilized to determine SMEs, for example:

Department Required if the change meets any of the following criteria:
Contamination Control The change impacts environment integrity, conditions or monitoring, including:

  • Changes to a controlled room or area that impact integrity
  • Changes in sampling methodology
  • Construction activities
  • Changes in personnel or material flow
  • The change will result in or modify exposure of product to the environment.

The change can impact microbiological control within a process stream, raw material or process equipment

The changes are to water systems

Release control process sipoc

Google does not share search terms for privacy reasons, so it is rare that a search term stands out on my blog.

So here you go.

SIPOC for disposition

This is purposefully a high level process.

Quality Review of Records: Batch Record, Packaging Record, the like

Lot Assessment: Evaluation of deviations, of change control and test results; but also of other inputs such as Critical Utilities and Environmental Monitoring Review. Ideally a holistic view.

Lot Disposition: Decision that the product meets all requirements of the GMPs and the market authorization.

Some important regulatory requirements:

  • United States: 21 CFR 211.22(a); 211.22(d)
  • EU: 1.4(xv); 1.9(vii); EU Annex 16
  • World Health Organization: Annex 3-GMP 1.2(g); Annex 3-GMP 9.11, 9.13, 9.15

 

One of the drivers for digital transformation, and a concept at route of the ICHs, is the idea of release by exception. Our systems will be tight enough, our design space robust enough, that most products are automatically released and sent into the market.