While focused on medical devices, this proposal is interesting read for folks interested in applying machine learning and artificial intelligence to other regulated areas, such as manufacturing.
We are seeing is the early stages of consensus building around the concept of Good Machine Learning Practices (GMLP), the idea of applying quality system practices to the unique challenges of machine learning.
My day 2 at WCQI is Day 1 of the conference proper. I’m going to try to live blog.
Morning keynote
Today’s morning keynote is the same futurist as at the LSS in Phoenix last month, Patrick Schwerdtfeger, and not only was I dismayed that it was they exact same I was reminded yet again how much I dislike futurists. I’m all about thinking of the future, but futurists seem to be particularly bad at it. It is all woowoo and bro-slapping and never ever a serious consideration of the impact of technology. Futurists are grifters.
These grifters profit by obscuring facts for personal gain.
They are working an angle, all of them: the health gurus and the life hackers
peddling easy solutions to difficult problems, the futurists who basically
state current trends as revelations. They are all trying to pull off the ultimate
con – persuading people they really matter.
They are selling themselves: their books, their podcasts,
their websites, their supplements, their claims to some secret knowledge about
how the world works. But I fundamentally doubt that anyone who gave 40 talks in
the last year has the bandwidth to do anything that really matters. It is all
snake-oil. And as quality professionals, individuals who are dedicated to
process and transparency and continuous improvement, we deserve better.
I’m not sure how these keynotes are selected but I think we
need to holistically view just what we want to be as a society and the pillars
we want for our conferences.
Anyone know a good article that evaluates futurists and life
hackers with the prosperity gospel? Seem like they are coming from a similar
place in the American psyche.
Ooh, artificial intelligence. Don’t get me wrong there is
real potential (maybe not the potential people feel like there is) but most
discussions on artificial intelligence is hype and bluster, and this
presentation is no different. Autonomous vehicles block chains. Hype and
bluster.
There is definitely people thinking this seriously, offering real insights and tools. We’re just not there. The speaker admits he gets 90% of his income from speaking. Pretty sure he isn’t actually doing that much. He might be an aggregator (as most of his slides with real content were attributed) but I keep struggling to see value here.
Gratuitous Steve Jobs picture.
After this there is some white space for vendor stuff.
At least I could multi-task and did some work.
Quality 4.0 Talks
I didn’t attend many of these last year because they were all standing and had a thrown together feeling. This year the ASQ seems to have upped the game. Shorter sessions can be good if the presentation is tight. At 15 minutes that is a hard bar to set.
First up we have Nicole Radziwill on “Mapping Quality Problems to Machine Learning Solutions” Nicole’s very active in the software section, which under the new membership model I’ll start paying a lot more attention to.
In this short presentation Nicole focuses on hitting the points of her 2018 Quality Progress article. Talking about quality 4.0’s path from Taylorism as “discovery & learning.”
Hit on big data hubris and the importance of statistics and analysis. The importance of defining models before we use them.
From ” Let’s Get Digital” in Quality Progress
Covers machine learning problem types at a high level.
Prediction
Classification
Pattern Identification (Clustering)
Data Reduction
Anomaly Detection
Pathfinding
High level recommendations were domain expertise, statistical expertise, data quality and human bias.
Next up is Beverly Daniels on “Risk and Industry 4.0”
It is telling that so many of the talkers make Millennial jokes. As a Gen-Xer I am both annoyed because no one ever made Gen-Xer jokes (the boomers never even noticed we were at the conference) AND frustrated because this is something telling about the graying of the ASQ.
Quick review of risk as more than probability. Hit on human beings as eternally optimistic and thus horrible at determining probability. As this was a quick talk it left more questions than answers. Getting rid of probability from risk is something I need to think about more, but her points are aligned to my thoughts on uncertainty.
Focusing on impact and mitigation is interesting. I liked the line “All it does is make your management feel good about not making a decision.”
A critical skill of a quality professional (of any professional), and a fundamental part of Quality 4.0, is managing data — knowing how to acquire good data, analyze it properly, follow the clues those analyses offer, explore the implications, and present results in a fair, compelling way.
As we build systems, validate computer systems, create processes we need to ensure the quality of data. Think about the data you generate, and continually work to make it better.
Have the tools to decide what data stands out, use control charts and regression analysis. These tools will help you understand the data. “Looks Good To Me: Visualizations As Sanity Checks” by Michael Correll is a great overview of how data visualization can help us decide if the data we are gathering makes sense.
Then root cause analysis (another core capability) allows us to determine what is truly going wrong with our data.
Throughout all your engagements with data understand statistical significance, how to quantify whether a result is likely due to chance or from the factors you were measuring.
In the past it was enough to understand a pareto chart, and histogram, and maybe a basic control chart. Those days are long gone. What quality professionals need to bring to the table today is a deeper understanding of data and how to gather, analyze and determine relevance. Data integrity is a key concept, and to have integrity, you need to understand data.
As we all try to figure out just exactly what Industry 4.0 and Quality 4.0 mean it is not an exaggeration to say “Data is your most valuable asset. Yet we all struggle to actually get a benefit from this data and data integrity is an area of intense regulatory concern.
To truly have value our data needs to be properly defined, relevant to the tasks at hand, structured such that it is easy to find and understand, and of high-enough quality that it can be trusted. Without that we just have noise.
Understand why data matters, how to pick the right metrics, and how to ask the right questions from data. Understand correlation vs. causation to be able to make decisions about when to act on analysis and when not to is critical.
2. How well do the sample data represent the population?
3. Does your data distribution include outliers? How did they affect the results?
4. What assumptions are behind your analysis? Might certain conditions render your assumptions and your model invalid?
5. Why did you decide on that particular analytical approach? What alternatives did you consider?
6. How likely is it that the independent variables are actually causing the changes in the dependent variable? Might other analyses establish causality more clearly?
Framing data, being able to ask the right questions, is critical to being able to use that data and make decisions. In the past it was adequate enough for a quality professional to have a familiarity with a few basic tools. Today it is critical to understand basic statistics. As Nate Silver advises in an interview with HBR. “The best training is almost always going to be hands on training,” he says. “Getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth.”
Understanding data is a key ability and is necessary to thrive. It is time to truly contemplate the data ecosystem as a system and stop treating it as a specialized area of the organization.
A ton of sensors (cheap, reliable sensors for everyone)
Data everywhere! (So much data. Honest data is good. Trust us.)
Collaboration (Because that never happened before technology)
Machine learning (this never ends well in the movies)
However, Quality 4.0 is really a lot more than the technology, it is all about using that technology to improve our quality management systems. So Quality 4.0 is really all about understanding that the world around us, and thus the organizations we work in, is full of complex and interconnected challenges and increasingly open systems of communication, and that we can no longer afford to address complex issues as we have in the past. The very simple idea behind Quality 4.0 is that current and future challenges requires thinking that is consistent with a living world of complexity and change.
As such there is nothing really new about Quality 4.0; it is just a consolidation of a lot of themes of change management, knowledge management and above all system thinking.
System Thinking requires quality professionals to develop the skills to operate in a paradigm where we see our people, organizations, processes and technology as part of the world, a set of dynamic entities that display continually emerging patterns arising from the interactions among many interdependent connecting components.
There are lots of tools and methodologies for managing systems. Frankly, a whole lot of them are the same that have been in use in quality for decades; others are new tools. The crucial thing to remember about Quality 4.0 is that it is an additive and transformative way to look at quality, and quite frankly one can go back and read Deming and see the majority of this there.
When I work on systems (which is according to my job description my core function), I keep some principles always in mind.
Principle
Description
Balance
The system creates value for the multiple stakeholders. While the ideal is to develop a design that maximizes the value for all the key stakeholders, the designer often has to compromise and balance the needs of the various stakeholders.
Congruence
The degree to which the system components are aligned and consistent with each other and the other organizational systems, culture, plans, processes, information, resource decisions, and actions.
Convenience
The system is designed to be as convenient as possible for the participants to implement (a.k.a. user friendly). System includes specific processes, procedures, and controls only when necessary.
Coordination
System components are interconnected and harmonized with the other (internal and external) components, systems, plans, processes, information, and resource decisions toward common action or effort. This is beyond congruence and is achieved when the individual components of a system operate as a fully interconnected unit.
Elegance
Complexity vs. benefit — the system includes only enough complexity as is necessary to meet the stakeholder’s needs. In other words, keep the design as simple as possible and no more while delivering the desired benefits. It often requires looking at the system in new ways.
Human
Participants in the system are able to find joy, purpose and meaning in their work.
Learning
Knowledge management, with opportunities for reflection and learning (learning loops), is designed into the system. Reflection and learning are built into the system at key points to encourage single- and double-loop learning from experience to improve future implementation and to systematically evaluate the design of the system itself.
Sustainability
The system effectively meets the near- and long-term needs of the current stakeholders without compromising the ability of future generations of stakeholders to meet their own needs.
In order to be successful utilizing these principles when designing systems and processes we need to keep user at the forefront — striving to be sensitive to the user, to understand them, their situation and feelings: to be more empathetic.
We leverage both the affective component and the cognitive component of empathetic reasoning, in short we need to both share and understand.
We are in short asking 5 major questions:
What is the purpose of the system? What happens in the system?
What is the system? What’s inside? What’s outside? Set the boundaries, the internal elements and elements of the system’s environment.
What are the internal structure and dependencies?
How does the system behave? What are the system’s emergent behaviors and do we understand their causes and dynamics?
What is the context? Usually in the terms of bigger systems and interacting systems.
Think holistically, think empathetically with the user, and ask questions about system behavior. Everything else falls into place from there.
