It is no accident that the routine scientific paper test report plays down such factors. It is the absence of these discussions which makes science testing look like a special activity; scientists testers become merely mediators or passive observers of Nature. Because the establishment of skill and competencies becomes important during a controversy we start to see better what goes into the making of science a good testing approach. Processes which are normally hidden become visible.

The quote above (with some creative license applied) is one of the conclusions drawn in Collins and Pinch’s book, The Golem. I was reminded of the book while reviewing an upcoming article for the TestRail blog the other day, in which the author was speaking about (amongst other things) the skills involved in experiment design.

If you haven’t read The Golem and have some interest in the field of testing, then I can heartily recommend it. Testing is all about experiment design, and so is The Golem book. Except, when you read The Golem (which is mainly filled with stories of historic and highly influential experiments by professional scientists) what you realise is, experiment design is a very messy business. Especially when you don’t necessarily know what the outcome of the experiment is supposed to be.

But we don’t know if we have built a good [experiment] until we have tried it and obtained the correct outcome. But we don’t know what the correct outcome is until… and so on ad infinitum. – the ‘experimenter’s regress’.

And that’s not the end of it. Once you’ve carried out the experiment and published your results, people within your community (your team, your organisation etc) may not agree with the result, or be able to reproduce it, or agree with the way in which the experiment was carried out, or even whether it should have been carried out in the first place. It can all get incredibly complicated. And so it is with testing.

I was reminded of this particular book because the article I was reading was about Things Your Manager Needs to Know About Testing. But one of the problems with what managers get told is that it tends to arrive on their desk in a highly sanitised format. A report with some metrics — tests created, tests run, tests passed, tests failed, tests not run etc., etc., probably with some pie charts — doesn’t tell your manager the story of your testing work. It doesn’t convey the experience, of the work. It doesn’t communicate the messiness of it all. Which brings me back to the opening quote.

If we really want people to understand what good testing and experiment design look like, let’s not hide behind metrics. Let’s tell them a story about what we learned, how we carried out the testing and how good (or not) the testing was. As my colleague Michael Bolton says (and from whom I learned of The Golem) — “To test is… to construct, edit, narrate and justify a story.” Whether your testing results in a controversy or not, tell your stakeholders a compelling story about the work you did, the problems you had, what you observed and how — and be sure to highlight the skills and competencies involved, so they can see how important the work you do is.

The Experimenters Regress

The problem of the experimenters regress is a challenge that exploratory testers will be intimately familiar with, and one that we typically try and solve with reference to oracles or heuristics that enable us to determine whether or not the result of a test, experiment or charter is within an expected range.

Thanks for reading. Feel free to reach out via a comment or on the socials if anything resonates.



Leave a Reply