Folks,
Let’s check in on Patty…
Patty was intensely preparing a lecture on Bayes’ Theorem. She always felt that this theorem was the most profound in probability and statistics. She remembered a real application, when her best friend took the Tine test for tuberculosis before she got married – and tested positive. The test claimed to be 99.9% accurate in identifying someone with TB. Her friend was devastated to find out that she apparently had this ancient, dreaded disease. Further investigation uncovered that the 99.9% number was more accurately stated as, “if you have the disease, this test will pick it up 99.9% of the time.” There was an important number not told: false positives. This rate was 5%. With so few people having TB, a 5% false positive rate would indicate that almost everyone that tested positive for TB, would be a false positive, hence not have TB. So it was, much to the relief of many, with her friend. This situation is an example of the false positive paradox.
While Patty was deep in thought, she was startled by the sound of her phone ringing. She looked at the area code and exchange and knew it was from her old company, ACME. She picked up the phone.
“Professor Coleman,” Patty answered. She liked the sound of that.
“Hey, Patty! It’s Reggie Pierpont!” the cheery voice declared.
Patty’s heart sank. Reggie was an OK guy, but he always got involved in things he didn’t understand and often convinced management to pursue expensive and ineffective strategies. He was that persuasive.
“Reggie, what’s up?” Patty said half-heartedly.
“Well, Madigan insisted I call you before we order some new testers. I think it is a waste of your time, but I’m following orders,” Pierpont said.
“What are the details?” Patty asked.
“We have a contract to produce one hundred thousand Druid mobile phones a week. We are confident our first pass yield is greater than 99%,” he began.
“Impressive,” Patty said with sincerity.
“I want to order some testers that identify a defective phone in a rapid functional test with 99.9% certainly. The testers are very expensive, so Madigan wants a sanity check before buying them. The other important info is that we get a huge penalty from the customer for any defected phone we ship,” Reggie continued.
“Well, with a large penalty, 99.9% is the right number. What do you do with the units the tester determines are defective?” Patty asked.
“Well, it is a good thing yields are high. The phones are so complex that we have quite a drawn out process to find the defect and fix it. Just finding a defect can cost $5 to $10 dollars in burdened labor, but, considering the value of a phone, it’s worth it. Like I said, it’s a good thing yields are high so we don’t have too many units needing this procedure,” Pierpont continued.
“What about false positives by the tester?” Patty asked.
“Shouldn’t be a problem, remember the tester is 99.9% accurate,” Pierpont answered.
Patty knew that Pierpont was missing her point, but she didn’t want to embarrass him……too much.
“Reggie, from what you told me, if a unit is defective the tester will catch it 99.9% of the time. What I am asking is, if a unit is good, how often does the tester say it is bad? This situation is usually called a ‘false positive’,” Patty responded.
“Well, it would be 100 – 99.9 or 0.1%,” Pierpont replied.
“That’s the percentage of bad units that would be called good. These units are often called ‘escapes.’ The only way to determine false positive rate is by a test, you can’t determine it from the 99.9% number,” Patty went on.
There was silence at the other end of the phone.
“What do I need to do to get the false positive number?” Reggie asked.
“You need to test about a 1,000 known good units and see how many the tester says are bad,” Patty said.
“I’ll do that with the loaner tester the tester company is letting us use and get back to you,” Pierpont replied.
Patty hung up the phone. She thought it interesting that Pierpont’s problem was so closely related to both Bayes’ Theorem and her friend’s false positive with the Tine test.
Two days went by and Patty, Rob, and Pete had just returned from lunch with the Professor. They would all meet with him quite often to discuss technical problems they were having. So, they offered to treat him to lunch.
As she walked into her office, Pete spoke up.
“Did Reggie Pierpont ever get back to you?” Pete asked.
“No, maybe I’m off the hook,” Patty chuckled.
At that instant, her phone rang. It was Pierpont.
“Hey, Reggie! What’s up?” Patty asked with more enthusiasm than she felt.
“Well, the tester says 5% of the good units are bad, I think you are going to tell me this is a problem,” Peirpont began.
“What if you run them through the tester again?” Patty asked.
“That IS running them through two or more times! If we run them through just once, it was 7%,” Reggie sighed.
“Well, let’s look at the numbers. You are making 100,000 units a week, with a 5% false positive rate that’s 5,000 units. Your yield loss is 1% or 1,000 units. So, you will have about 6,000 units the tester will declare as bad when only 1,000 really are. These numbers are off a little bit. Bayes’ Theorem would give us the precise numbers, but these are very close. Since your process to analyze fails after the tester costs at least $5 per unit, you will be losing $25K per week due to false positives,” Patty elaborated.
“Time for a new strategy,” Pierpont sighed.
Patty and Pete agreed to help Pierpont work with the tester vendors to develop a better strategy.
Epilogue
Patty and Pete helped Pierpont develop an effective test strategy working with a tester vendor. Neither Patty nor Pete had known Reggie well before… but, after this joint effort, they grew quite close. Reggie became quite engaged in the process and seemed to learn quite a bit. Patty was able to use some of the data in her classes.
A few weeks later she got a beautiful card in the mail. She opened it. It read, “Dear Patty, Thanks for all of your help. We wouldn’t have made it without you and Pete helping us with our testing strategy. Best Regards, Your faithful student, Mike Madigan.”
Patty got a little choked up.
Cheers,
Dr. Ron