17 replies to this topic

[stocks]

On 4 October 2002, women who were moderate drinkers received good news: their risk of breast cancer was not raised, according to a report in the Lancet that was widely covered by the British media.1 The bad news was that smoking at an early age was now implicated as a risk factor for breast cancer. However, after they had enjoyed guilt-free drinks (without cigarettes) for only a few days, on 13 November the message was reversed: alcohol did increase the risk of breast cancer after all, but smoking was declared innocent.2 The press release proclaimed "Alcohol, tobacco and breast cancer: the definitive answer." A reader was driven to complain in the letters page of the Guardian (14 November 2002): "So let me get this right---alcohol's no good anymore, and if you smoked within five years of getting your periods, that's bad news too. Oh no, that was a couple of weeks ago; smoking's okay now . . . Do things stop being bad for us if we just forget about them for a bit, do you think?"

This is a familiar story---so much so that in Bristol we set our medical students the exercise of examining the "health scare of the week" that appears each Friday, generally from a study reported in the BMJ or Lancet.

http://www.bmj.com/c...l/325/7378/1437

In preparation for the publication of his book Sorry, Wrong Number! Brignell spent two years collecting relevant articles from The Times. He found "hundreds of contradictions in published claims", that is, reports of studies that contradicted each other. (Brignell, Sorry, Wrong Number!) He suggests that when two studies come to mutually contradictory conclusions, one of them must be false. Others have come to the same conclusion, and it has been suggested that a large number of published studies are false.[9]

Brignell suggests that the cause of such contradictions is the reliance on statistical significance alone as an indication of truth. He suggests that this reliance is undermined by a combination of four main factors:

1. the use of 95% confidence levels ('the one in twenty lottery');
2. the acceptance of low relative risks (RR<2.0 or RR>0.5);
3. the absence of randomisation;
4. small studies.

http://www.answers.c...c/john-brignell

[stocks]

This week's "Science Journal" column in the W$J (link for subscribers) reports on the interesting and provocative research of John Ioannidis, who argues that the results of most published scientific studies are wrong.

Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true." . . .

Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.

He further argues that only a fraction of incorrect studies are ever corrected or retracted, meaning that there are many published studies still "on the books" that support erroneous findings. It's an interesting and provocative thesis.

http://volokh.com/po...189863895.shtml

[maineman]

Sorry, nothing new there. See today's New York TImes magazine for a long, excellent review of epidemiological studies. Morality, or honest religion, thrives when its adherents understand that doubt is of the essence. Once the "church" stopped insisting they were 100% correct, they finally stopped massacring innocents. We hope someday this will be so for current "doubtless" religions, like radical Islam. Long ago, wiser heads prevailed and separated faith and dogma from scientific endeavor. Slowly over time we collectively learn from each other and are still developing ever-critical ways of analyzing data in order to make sense of measurable phenomena. Bit by bit we hone the craft. We've come a long way from Koch's postulates (which determines how an infectious agent causes an infection). Modern science is deep into statistical analysis currently. To those of us who care to wait for the results we look forward to ever better data. It is extremely difficult for those in "the trench" i.e. those of us who are faced with sick or worried patients every day to be able to offer valid advice - Does he need a blood pressure pill? Based on what? Which one? What is the risk? What is the benefit? But we try to do our best with the best information we have. And we stay educated and are willing to change once new and correct data is receieved. Ever wonder what goes (or not) through the mind of a person who is self diagnosing and self prescribing off the shelves of the health/vitamin shoppes? How critically are they examing the "evidence" and performing rigorous statistical analysis? Never. Science is ultimately great. Peer review journals are great. The search for "truth" inches along. mm

[Echo]

Stocks,



In addition to the excellent points made be maineman above, I'd like to add something important. Almost ANY journal article with an important new finding published in a major peer-reviewed journal will have several important sections after presenting the methods and results.

First there is a DISCUSSION section which reviews the implications of the findings in the context of previously known findings and offers reconciliation and clarity that advance knowledge and suggest practice trends, or alternatively, in some cases, points of conflicting information that need further study.

That section is followed by another all important section called LIMITATIONS whereby the authors explain the shortcomings of their study and point out which types of patients their study might apply to and more importantly which other types of patients NOT to extrapolate to. They may also point out that their study only examines an effect on some lab test or parameter, and not necessarily OUTCOMES in broader endpoints such as death, heart attack, kidney failure, etc.

Finally, once any study gets published, it is out there for the brightest minds to dissect and discuss including statisticians. Any important study will have editorials usually in the same issue and then be discussed at hundreds of subsequent medical meeting throughly the world, where experts in that field will have at it if there are problems with the study.



As with any walk of life, some may misapply the data in a study. But at least the medical community continues to strive towards evidence based medicine rather than folklore and hear-say.

Echo

Edited by Echo, 16 September 2007 - 10:37 AM.

[Rogerdodger]

Long ago, wiser heads prevailed and separated faith and dogma from scientific endeavor

Sorry Doc, but I believe that can only happen when scientists are "Spock" like, non-emotional, totally logical...and honest, without motive.
Taint gonna happen, IMHO.

[stocks]

Sorry, nothing new there. See today's New York TImes magazine for a long, excellent review of epidemiological studies.

Science is ultimately great. Peer review journals are great. The search for "truth" inches along.

mm

Doc,
Yes, I think the excellent Times article echoes my post from 7-11-2007.

A quote from the Times article:

"Richard Peto, professor of medical statistics and epidemiology at Oxford University, phrases the nature of the conflict this way: “Epidemiology is so beautiful and provides such an important perspective on human life and death, but an incredible amount of rubbish is published,” by which he means the results of observational studies that appear daily in the news media and often become the basis of public-health recommendations about what we should or should not do to promote our continued good health."

I wish you would spend more time commenting on the sloppy work in your profession; you do an admirable job exposing quackery elsewhere. But then again, maybe that is not your role and quackery outside the profession is a bigger problem that sloppiness in it.

Can we trust any profession to police itself? Not if they are filled with human beings.

[Echo]

quackery outside the profession is a bigger problem that sloppiness in it.

Can we trust any profession to police itself? Not if they are filled with human beings.

Very true and important points, stocks.

"Quackery" outside the profession is like the wild west. You gotta protect yourself cuz you might not have anybody watching yer back.

Sloppiness inside the profession is a fact of life and human nature. Never going to eliminate it. Gotta do your best to minimize it.

Can you trust the medical profession to police itself? Well to a certain extent, limited by the same complex human factors affecting any profession policing itself.

Echo

[maineman]

Roger, I refer to the great separation of dogma and science. This has already occurred. Scientists are human. They will be able to separate personal ideology as best as they can. In the vast arena of modern science personal ideology will be found out to be fraud and dismissed. Do you really think there are any serious scientists on the board of the "Creation Museum"? Or writing high school textbooks about "Intelligent Design"? Those are wolves in sheep's clothing and the truth will out. Many an excellent scientist spends his days poring over miniscule bits of data, adhering to strict scientific guidelines. He may still go home and say a blessing over wine or pack up the kids for Sunday School. Some people get this.... some don't. Stocks, It is important to understand that I am "just" a doctor. And at the risk of sounding silly, let me say I am ME. I am not "my profession". That is a naive statement. I am one of may hundreds of thousands of doctors. My opinions and my actions are my own. What can I say when someone says to me, "You doctors...." ? For what its worth each one of us is responsible for our own actions. I choose to study a lot. I choose to question my knowledge. I try to do the best I can. I also respond to those around me when I feel they are off course. I will challenge colleagues openly if I feel they are practicing bad medicine, or ordering a test or treatment I feel is wrong. I was chief of medicine at a large hospital for many years and was involved in disciplinary actions against my peers. We have systems in place to monitor the adherence to established guidelines. We read each other's notes. We review all hospital deaths and unexepected outcomes. We have back up teams of radiologists to double read x-rays and mammograms. Our labs have very high standards to ensure that test results are accurate. Our offices are wide open to second party insurers who can come in at any time and pull patient charts off the shelf and review them. Our prescribing information is available. The HMOs, etc. know exactly what pills we write prescriptions for. They know from reveiwing our charts if the treatments are justified. They even know if the patients are compliant - do they refill prescriptions? Get annual mammograms? Etc. Is it enough? No. Is what we are doing now adequate to assess "proper care"? No. But we sit together and hammer out new ways of policing ourselves. Those of us who take pride in medicine are glad to be involved in this. That's all I can say about ME and what I DO. I can assure you that this type of self-policing and self-reflection does not take place in the world of Chiropractics, herbalists, macrobiotics, vitamin sellers, and fish-oil salesmen, to name a few. But it is not my "job" to go after "them". The best I can do is be the best medical doctor I can be and help educate my patients about what is known, what is not known, what is safe and what is not safe. So yes, we can expect decent people to police themselves. Contrary to what you may believe, there are many of us who care about truth, about people's health and the pursuit of decent scientific knowledge. And there are some patients, believe it or not, who are glad to hear that. mm

[Rogerdodger]

Do you really think there are any serious scientists on the board of the "Creation Museum"? Or writing high school textbooks about "Intelligent Design"?

The "Piltdown Man" is one of the most famous "scientific frauds" of all time.
It was accepted for decades because it "fit" what the scientists wanted to prove.
Why resort to fraud to refute "wolves in sheep's clothing" if there is any actual solid evidence?
Maybe there is a bit of "Dogma" in science.

Adapatation and mutation are evident all around us.
But I'm still looking for something to evolve.
Why do these "serious scientists" panic when some species is near extenction?
Will it not come back again?
Isn't evolution so dynamic and vibrant that new stuff will be popping up all the time everywhere?
Isn't that what the fossel record shows?

Anyway, I'm teaching cats to fly.
But it's going to take lots of cats.

Edited by Rogerdodger, 17 September 2007 - 06:12 PM.

[stocks]

"Richard Peto, professor of medical statistics and epidemiology at Oxford University, phrases the nature of the conflict this way: “Epidemiology is so beautiful and provides such an important perspective on human life and death, but an incredible amount of rubbish is published,” by which he means the results of observational studies that appear daily in the news media and often become the basis of public-health recommendations about what we should or should not do to promote our continued good health."

Statistics Proves Same Drug Both Causes And Does Not Cause Same Cancer

In 2010, two research teams separately analyzed data from the same U.K. patient database to see if widely prescribed osteoporosis drugs [such as fosamax] increased the risk of esophageal cancer. They came to surprisingly different conclusions.
One study, published in the Journal of the American Medical Association, found no increase in patients’ cancer risk. The second study, which ran three weeks later in the British Medical Journal, found the risk for developing cancer to be low, but doubled.

... to do statistics the right way. .. we would see about a 99 percent drop in papers published. Sociology would slow to a crawl. Tenure decisions would be held in semi-permanent abeyance. Grants would taper to a trickle. Assistant Deans, whose livelihoods depend on overhead, would have their jobs at risk


http://wmbriggs.com/blog/?p=5583