Data dredging
#1
Posted 11 July 2007 - 07:16 PM
This is a familiar story---so much so that in Bristol we set our medical students the exercise of examining the "health scare of the week" that appears each Friday, generally from a study reported in the BMJ or Lancet.
http://www.bmj.com/c...l/325/7378/1437
In preparation for the publication of his book Sorry, Wrong Number! Brignell spent two years collecting relevant articles from The Times. He found "hundreds of contradictions in published claims", that is, reports of studies that contradicted each other. (Brignell, Sorry, Wrong Number!) He suggests that when two studies come to mutually contradictory conclusions, one of them must be false. Others have come to the same conclusion, and it has been suggested that a large number of published studies are false.[9]
Brignell suggests that the cause of such contradictions is the reliance on statistical significance alone as an indication of truth. He suggests that this reliance is undermined by a combination of four main factors:
1. the use of 95% confidence levels ('the one in twenty lottery');
2. the acceptance of low relative risks (RR<2.0 or RR>0.5);
3. the absence of randomisation;
4. small studies.
http://www.answers.c...c/john-brignell
Defenders of the status quo are always stronger than reformers seeking change,
UNTIL the status quo self-destructs from its own corruption, and the reformers are free to build on its ashes.
#2
Posted 15 September 2007 - 11:33 PM
Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true." . . .
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.
He further argues that only a fraction of incorrect studies are ever corrected or retracted, meaning that there are many published studies still "on the books" that support erroneous findings. It's an interesting and provocative thesis.
http://volokh.com/po...189863895.shtml
Defenders of the status quo are always stronger than reformers seeking change,
UNTIL the status quo self-destructs from its own corruption, and the reformers are free to build on its ashes.
#3
Posted 16 September 2007 - 08:20 AM
#4
Posted 16 September 2007 - 10:36 AM
In addition to the excellent points made be maineman above, I'd like to add something important. Almost ANY journal article with an important new finding published in a major peer-reviewed journal will have several important sections after presenting the methods and results.
First there is a DISCUSSION section which reviews the implications of the findings in the context of previously known findings and offers reconciliation and clarity that advance knowledge and suggest practice trends, or alternatively, in some cases, points of conflicting information that need further study.
That section is followed by another all important section called LIMITATIONS whereby the authors explain the shortcomings of their study and point out which types of patients their study might apply to and more importantly which other types of patients NOT to extrapolate to. They may also point out that their study only examines an effect on some lab test or parameter, and not necessarily OUTCOMES in broader endpoints such as death, heart attack, kidney failure, etc.
Finally, once any study gets published, it is out there for the brightest minds to dissect and discuss including statisticians. Any important study will have editorials usually in the same issue and then be discussed at hundreds of subsequent medical meeting throughly the world, where experts in that field will have at it if there are problems with the study.
As with any walk of life, some may misapply the data in a study. But at least the medical community continues to strive towards evidence based medicine rather than folklore and hear-say.
Echo
Edited by Echo, 16 September 2007 - 10:37 AM.
#5
Posted 16 September 2007 - 11:22 AM
Long ago, wiser heads prevailed and separated faith and dogma from scientific endeavor
Sorry Doc, but I believe that can only happen when scientists are "Spock" like, non-emotional, totally logical...and honest, without motive.
Taint gonna happen, IMHO.
BIGGEST SCIENCE SCANDAL EVER...Official records systematically 'adjusted'.
#6
Posted 16 September 2007 - 02:01 PM
Sorry, nothing new there. See today's New York TImes magazine for a long, excellent review of epidemiological studies.
Science is ultimately great. Peer review journals are great. The search for "truth" inches along.
mm
Doc,
Yes, I think the excellent Times article echoes my post from 7-11-2007.
A quote from the Times article:
"Richard Peto, professor of medical statistics and epidemiology at Oxford University, phrases the nature of the conflict this way: “Epidemiology is so beautiful and provides such an important perspective on human life and death, but an incredible amount of rubbish is published,” by which he means the results of observational studies that appear daily in the news media and often become the basis of public-health recommendations about what we should or should not do to promote our continued good health."
I wish you would spend more time commenting on the sloppy work in your profession; you do an admirable job exposing quackery elsewhere. But then again, maybe that is not your role and quackery outside the profession is a bigger problem that sloppiness in it.
Can we trust any profession to police itself? Not if they are filled with human beings.
Defenders of the status quo are always stronger than reformers seeking change,
UNTIL the status quo self-destructs from its own corruption, and the reformers are free to build on its ashes.
#7
Posted 16 September 2007 - 02:31 PM
quackery outside the profession is a bigger problem that sloppiness in it.
Can we trust any profession to police itself? Not if they are filled with human beings.
Very true and important points, stocks.
"Quackery" outside the profession is like the wild west. You gotta protect yourself cuz you might not have anybody watching yer back.
Sloppiness inside the profession is a fact of life and human nature. Never going to eliminate it. Gotta do your best to minimize it.
Can you trust the medical profession to police itself? Well to a certain extent, limited by the same complex human factors affecting any profession policing itself.
Echo
#8
Posted 16 September 2007 - 02:41 PM
#9
Posted 17 September 2007 - 06:04 PM
Do you really think there are any serious scientists on the board of the "Creation Museum"? Or writing high school textbooks about "Intelligent Design"?
The "Piltdown Man" is one of the most famous "scientific frauds" of all time.
It was accepted for decades because it "fit" what the scientists wanted to prove.
Why resort to fraud to refute "wolves in sheep's clothing" if there is any actual solid evidence?
Maybe there is a bit of "Dogma" in science.
Adapatation and mutation are evident all around us.
But I'm still looking for something to evolve.
Why do these "serious scientists" panic when some species is near extenction?
Will it not come back again?
Isn't evolution so dynamic and vibrant that new stuff will be popping up all the time everywhere?
Isn't that what the fossel record shows?
Anyway, I'm teaching cats to fly.
But it's going to take lots of cats.
Edited by Rogerdodger, 17 September 2007 - 06:12 PM.
BIGGEST SCIENCE SCANDAL EVER...Official records systematically 'adjusted'.
#10
Posted 03 May 2012 - 01:14 PM
Statistics Proves Same Drug Both Causes And Does Not Cause Same Cancer"Richard Peto, professor of medical statistics and epidemiology at Oxford University, phrases the nature of the conflict this way: “Epidemiology is so beautiful and provides such an important perspective on human life and death, but an incredible amount of rubbish is published,” by which he means the results of observational studies that appear daily in the news media and often become the basis of public-health recommendations about what we should or should not do to promote our continued good health."
In 2010, two research teams separately analyzed data from the same U.K. patient database to see if widely prescribed osteoporosis drugs [such as fosamax] increased the risk of esophageal cancer. They came to surprisingly different conclusions.
One study, published in the Journal of the American Medical Association, found no increase in patients’ cancer risk. The second study, which ran three weeks later in the British Medical Journal, found the risk for developing cancer to be low, but doubled.
... to do statistics the right way. .. we would see about a 99 percent drop in papers published. Sociology would slow to a crawl. Tenure decisions would be held in semi-permanent abeyance. Grants would taper to a trickle. Assistant Deans, whose livelihoods depend on overhead, would have their jobs at risk
http://wmbriggs.com/blog/?p=5583
Defenders of the status quo are always stronger than reformers seeking change,
UNTIL the status quo self-destructs from its own corruption, and the reformers are free to build on its ashes.