In my day job, I’m a test engineer. Specifically, I create electronics to test other electronics. At work recently, I have had an ongoing… debate. Yes, I think debate is the right word. Because my boss told me to look at a test and think of some ways to improve it. And my initial response was, this is a horrible test, and you need to rethink it entirely.
That was not the answer he wanted, of course, but he’s a good boss, and an engineer, so he asked me to explain. Without getting technical, the short answer is: you are using the system to test the system.
From a test engineering standpoint, this is not only wrong, it invalidates the test. You cannot verify that a component works by saying that it works in the system, because the next question is usually, how do you know the system works? And the answer is, because it works with components we previously tested. With the system.
There are several reasons why this is a dangerous way to approach testing. First, you have no outside verification. There is no external standard that you can point to and say, “When I test it this way, I get the same results as when I test in the system.”
But you also have the issue of verifying something is good because it hasn’t really failed yet. Not enough to notice. Oh, sure, the whole process may have drifted over time, but it still works, right? We just want to verify it works.
Now imagine your customer starts complaining that their system isn’t working correctly. You bring in the defective device, and you test it against your own system, and it works fine. How do you determine where the fault lies? What authority do you appeal to? Your system works. Theirs doesn’t. But you are also saying the systems are the same, because they were tested in the same way.
If you want to know how well something works, you have to tell me how you measure success. Larry Correia is famous for saying that the only measurement of a book’s success if how many dollars people give you or it. But we have the whole Hugo debacle claiming that books are measured by the gender/race/sexuality of the authors and the characters. Who is correct? They both are, because they’ve set up different standards, and then they test against those standards.
An SJW does not care how well a book does in the market, because that’s not how they determine value. Likewise, Larry doesn’t care about what the authors or characters look like, he just wants to know how many people are buying it. And it’s not like those are the only ways of looking at it.