When I was writing my critique of modern science I intended to fit this point in there somewhere but for whatever reason it didn't make it in there, regardless I still believe that the point I will argue here is one that I feel is important for us to understand when approaching "scientific studies" in all fields.
This other point is one that (like many others) was inspired by Joel Salatin. There is one point in his book Everything I Want to do is Illegal where he talks about some conference he went to where some guy with a PhD was promoting, in Salatin's words, "spring when the new high-tech snake oil [that] was an herbicide that retarded senescence in grasses." Senescence is a state that grass can reach when a heard of cattle graze on it for too long, when grass reaches senescence it becomes harder for cows to eat and it contains much less nutrients than it did before. I guess this must be some sort of natural defense grass has against predators. Anyways, grass in this state is much less healthy for the cows than grass in its normal state so it is within the best interest of a responsible farmer to make sure that their cows have grass to graze on that is not in this state.
Of course the modern solution to this problem was to use this chemical this scientist was being paid to advertise. The scientist's graphs clearly showed that cows that were kept on a field that used this snake oil were much healthier than cows that were kept on fields without it. If we were to trust science we'd be making sure every cattle rancher used this product, but should we? Salatin doesn't think so. Humans have been practicing animal husbandry since the dawn of civilization, it would be quite foolish to think that after thousands of years innovative farmers (not scientists) would have come up with a solution to this senescence problem. Salatin's simple solution is one that has likely been practiced for hundreds, if not thousands, of years, if the grass goes bad when cattle grazes on it for to long just move the cattle to fresh grass every day. Salatin calls this a controlled-grazing program, I call it common sense.
Salatin eventually asked the scientist who was schilling this snake oil whether or not it was tested against the effectiveness of a controlled-grazing program. The scientist answered him by saying, "no, that was not part of the parameters of the research," which is a fancy way of saying, "no, if we had done that we likely would not have gotten the results that we wanted from this pseudo-experiment and would not be able to show you these biased graphs which the company that paid us wanted you to see so that you could be tricked into buying their product." This experiment was testing the effectiveness of doing something over doing nothing, it did not prove whether or not that something was the best thing to do at all, in fact it is quite reasonable to assume that a controlled-grazing pattern would be better than using whatever these people were trying to sell, it would also be much cheaper.
Unfortunately the dishonest unscientific practice of testing something against noting has become increasingly common in our modern world where science is more interested in selling products than it is in finding the truth. Does taking a multivitamin give you any benefit if you are on the terrible diet (or lack of one) that most people are on? Yes, of course it does. Does taking a multivitamin give you any benefit if you are eating a healthy diet? I don't know, maybe it does, maybe it doesn't I haven't seen a scientific study about it.
This practice is truly ridiculous. When presented as a joke we think it is a joke, but it is not a joke. No honest scientist would test something against noting and say that the results of their experiment prove the definitive best course of action. To quote Salatin once more, "if you’re constantly testing something against nothing, lots of times the something will win, even if it isn't a very good something."