Cutting-edge Research: JoVE partners with US Government

Posted on May 15, 2012 by Neal Moawed

Today marks an important day in JoVE’s history. Today JoVE, The Journal of Visualized Experiments, published two articles in partnership with the United States government’s Defense Threat Reduction Agency (DTRA). JoVE is proud to present the work from Temple University’s Dr. Chris Schafmeister and State University of New York Buffalo’s Dr. David Pawlowski and Dr. Richard Karalus.

The support of scientists conducting research for DTRA has significant ramifications for identifying, treating, and preventing the outbreak of defense threats. DTRA exists to safeguard America and its allies from weapons of mass destruction (WMD), including chemical, biological, radiological, nuclear weapons and high-yield explosives (CBRNE), by providing capabilities to reduce, eliminate, and counter the threat and mitigate its effects. Because the techniques and technologies developed in DTRA sponsored laboratories promise a safer world and will benefit from distribution in a highly visual format, DTRA has sponsored the research and publication of their scientists in JoVE.

Learn about the research here!

Data Vs. Methods: How to Increase Reproducibility?

Posted on May 15, 2012 by Moshe Pritsker

Scientists learn from video and text.

Recently, I wrote about low reproducibility of science articles, a problem plaguing every experimental scientist and confirmed by massive studies taken by biotech/ pharma companies. There is a certain agreement that this phenomenon is due to the inadequate description of methods and materials in science articles, and the obsession of science journals and funding bodies with data, often paying little attention to methods. There are no winners in the data vs. methods battle as data is not useful if it cannot be reproduced. In the end, science funding is wasted as scientists are in the endless cycle of learning and relearning techniques instead of innovating the field.

It is time to be productive and take initiative to fix the problem instead of adding to the cacophony of complaints. Let’s talk about some possible practical solutions:

Science journals should not limit the space devoted to materials and methods section. While this was a problem in the pre-Internet era, such limitations were justified by printing costs. They no longer make no sense for online publishing where printing costs are not as significant.
Methods sections should be required to include a step-by-step description of research procedures such as those available in methods journals (e.g. Nature Protocols, Current Protocols, JoVE, and others). When I co-founded JoVE, this was a priority and something I yearned for in my own research.
Funding bodies such as NIH should change the funding criteria to include a requirement for reproducibility of the research financed by the taxpayer money. This will further increase transparency associated with these grants.
Scientists should publish special articles on methods developed in their lab in a visualized format, similar to one applied by JoVE or, in a more limited fashion by medical journals New England Journal of Medicine and Journal of Bone & Joint Surgery. Visual demonstration is a better way to teach a new technique than just reading the text alone. Scientists mostly learn in the lab visually and by demonstration, for example they ask their more experienced colleagues to show them a new experimental technique

Click here for more

Publication Bias and the Scientific Process

Posted on May 14, 2012 by Robert Ainslie

The future of the scientific process is now open for debate. On May 10^th, Daniel Sarewitz published a column in Nature (Nature 485,149 (10 May 2012) doi:10.1038/485149a) decrying the recent trend towards positive publishing bias in the biomedical sciences. This means that more often than not, studies that fail to find positive results are not reported, while any hint of a statistically significant positive result is published. This positive publication bias is substantiated by the now famous Amgen study (C. G. Begley & L. M. Ellis Nature 483, 531–533; 2012) that reported the inability to reproduce 47 of 53 landmark cancer studies.

Calle et al. (2011) - 'A Procedure for Lung Engineering' -a novel technique published in a visualized format

One possible explanation for this lack of reproducibility is that positive results are actually false positives, (i.e. not actually true). Another explanation is supported by a 2005 paper by John Ioannidis (also cited by Sarewitz), titled ‘Why Most Published Research Findings Are False’ (PLoS Med. 2, e124; 2005). Here, Ioannidis (2005) asserts that research results are less likely to be true when the studies have greater ‘flexibility in designs, definitions, outcomes, and analytical modes’. Simply put, your method of study is extremely important in producing un-biased, true research. However, in the current state of most scientific publishing, transparency of methods is embarrassingly poor. If you open any of the highest impact journals, methods sections are relegated to the end of the paper, often in trimmed down and difficult to decipher versions. We exist in a scientific culture that places exceeding importance on positive results, without fully describing, or showing, how these results were obtained.

If Ioannidis (2005) is correct, then reducing variability amongst methods for similar experiments by increasing transparency is one key solution for reducing bias and providing faster verification and falsification of findings. This self-correction is at the heart of the scientific method. By keeping methodology in a diminished form, and not highlighting verified, accurately executed techniques, we fail to control for one highly significant variable in producing unbiased research. The scientific process is at stake here, so we must ask ourselves: how are we going to move forward? We can begin by publishing highly transparent and visible methodology, both novel and ‘gold standard’.

Lee et al. (2012) showing proper technique for loading and running agarose gels

Here, it’s important for novel methods to garner high visibility so that the scientific process can hone, correct and apply them in the appropriate circumstances. For example, in March 2011, Dr. Laura Nicklason published her laboratory’s tissue engineering technique (Calle, E. A., Petersen, T. H., Niklason, L. E. J. Vis. Exp. (49), e2651, DOI: 10.3791/2651 (2011)), which is difficult to reproduce without visualization of the apparatus and methodology. Presented in the multimedia format of a JoVE publication, this complicated technique is now more easily reproduced. Concurrently, it’s critical that the time honored, ‘gold standard’ techniques continue to be executed correctly. For example, a recent JoVE publication, Agarose Gel Electrophoresis for the Separation of DNA Fragments (Lee, P. Y., Costumbrado, J., Hsu, C., Kim, Y. H. J. Vis. Exp. (62), e3923, DOI: 10.3791/3923 (2012), shows best practices for a standard biological technique. In both of these cases, visualizing techniques allows for better execution of experiments, which will improve reproducibility and, hopefully, decrease publication bias.

Evaluating Techniques to assess Brain Function

Posted on May 10, 2012 by Neal Moawed

We all fear brain trauma. Be the trauma in ourselves or in our loved ones, the concept of partial or full loss of function or responsiveness proves a reality many do not prepare for or understand. In light of recent news chronicling the effects of brain trauma in football, hockey and other sports, and spurred most recently by the apparent suicide of football player Junior Seau, many people find themselves asking questions about traumatic brain injuries, their effects, and how we monitor and diagnose the brains of these patients. There is a real necessity for scientists and the public alike to understand and learn about these afflictions and the tools available to doctors.

Scientist demonstrating TMS

It is prudent that a new review article titled “Brain Connectivity in Disorders of Consciousness,” found in the journal Brain Connectivity, evaluates three different procedures used to evaluate brain function in unconscious, brain injured patients. The methods studied are functional magnetic resonance imaging f(MRI), transcranial magnetic stimulation (TMS) combined with electroencephalography, and response to neuronal perturbation, measuring sensory evoked potentials (ERP). The review focuses on using techniques to monitor a patient’s disorders of consciousness (DOC), a condition commonly divided into coma, vegetative state, and minimally conscious state. The review finds that neuroimaging should be used in addition to clinical diagnosis to help determine a patient’s condition.

The review is illuminating in how technology helps with diagnosis of hard to evaluate conditions. It also raises questions about the use of these high technologies in brain damage diagnosis. Also, by monitoring and recording the brains of patients with DOC we may be able to help predict when patients are on the road to DOC, or otherwise correlate inactive brain regions with presenting symptoms. It seems like the first step in the right direction.

Mayo Clinic Makes Recommendations to Journals, Are they enough?

Posted on May 9, 2012 by Neal Moawed

As a scientist, one frustrating perception of the industry held by many of my friends and family is their mistrust of research and medicine. I hear statements such as this far too often: “Research is funded by pharmaceuticals that conspire with doctors to conduct research on expensive drugs to drive their profit margins up and don’t actually care about curing or helping people.” Alternatively, I hear things like “Scientists are out of touch with the needs of the public and only conduct research on self-serving topics to keep their jobs.” These statements illuminates a common problem with science: a lack of transparency and clarity of research and the disclosure of who sponsors it has caused the public to mistrust scientists and their efforts.

This image came from info.biotech-calendar.com

In an effort to combat a lack of transparency, the Medical Publishing Insights and Practices (MPIP) initiative, and the Mayo Clinic have joined forces to develop a call to action for journals, scientists, and industry sponsors to close the “credibility gap” in Industry-sponsored research. Found here, the recommendations include items to ensure clinical relevancy of studies, increasing protocol information and length, publicly reporting all results and describe statistical methods used to make a conclusion. In theory, these suggestions could prove valuable to increasing public support if scientists and sponsors adhere to the guidelines.

While the call to action is valuable and a definite step in the right direction, if the “credibility gap” between industry-sponsored research, scientists and the public truly needs to be closed, more steps must be taken. As I mentioned in a previous article on transparency, reporting raw data is far from tamper proof and does not mean that the public will understand what they’re reading. Would my friends and family be more trusting of industry sponsored research if the industry clearly stated their reasons for funding the work, including possible commercial applications? Or is the impetus on the researcher and scientists to promote their research and explain clearly and openly their work with the public? Either way, I thank the Mayo Clinic for their efforts and hope to see what will come from MPIP in the future. Do you think they went far enough in closing the credibility gap? Leave your responses in the comments, and look to @JoVEJournal on twitter as the conversation progresses.

Who is killing scientific progress?

Posted on May 9, 2012 by Beth Hovey

We all know that the advancement of science is dependent on the accessibility of information. Today, many claim that the best way to address this is for authors to publish in open access journals, otherwise their peers will not be able to review the published information. I believe this is a false hypothesis – most scientific information is available if you know where to go or whom to ask. Rather, I propose that the real culprits to inaccessibility of information are the authors themselves.

: Cell culture experiment.

I know this is a controversial statement to make, but let me explain. In general, authors fulfill their ethical mandate to communicate their work. However, when publishing, good intentions do not necessarily guarantee reproducibility or transparency of research. Information accessibility suffers due to four basic issues: 1) methods are considered an afterthought when publishing; 2) important information about the methodology is often omitted – intentional or not; 3) only results fitting the authors’ hypothesis are being published; and 4) rarely are negative results published or discussed. In subsequent blogs, I will explain my ideas on these topics.

In the meantime, feel free to share your thoughts. Please comment with other ideas on who or what is killing scientific progress.

Mapping of a Flutter: A “How-To” from One M.D. to Another

Posted on May 8, 2012 by Alexa Meehan

A healthy adult heart beats 50-100 times per minute in a regular steady rhythm. However, persons with cardiac arrhythmias experience an upset that alters this rhythm. One such type of arrhythmia is atrial flutter. Atrial flutter, which causes cardiac output to decrease, can occur in otherwise healthy hearts, and are most commonly associated with tachycardia – a rapid heart rate.

Historically, electrocardiography (ECG) has been the go-to method used to monitor and assess the occurrence of such flutters. But advances in electrophysiological mapping have brought other forerunners in, such as: electromagnetic mapping, fluoroscopic mapping, and intracardiac catheter mapping. These techniques are used standalone and in concert to create electroanatomical maps of a subject.

At the Heart Rhythm 2012 conference in Boston, May 9-12, a new section is being introduced, “How-To” sessions. “How-To” includes titles such as, “How to Trouble-shoot Pacemakers” and “How to Perform Specific Techniques in AF Ablation.” In a third session, “How to Manage Atypical Atrial Flutter Following AF Ablation,” conference attendees will have the opportunity to see two schools of thought on how to map and ablate left atrial flutters: The Bordeaux Approach vs. The Michigan Approach. It seems that we are not the only ones to recognize how beneficial it is to have a technique demonstrated for you by an expert in the field. In this case, participants at Heart Rhythm 2012 get two.

Data vs. Methods – Why Science Articles are so Difficult to Reproduce

Posted on May 7, 2012 by Moshe Pritsker

The growing debate on reproducibility in science articles (10-30% according to the last studies) makes me think again about my days in the lab. As a grad student at Princeton in the early 2000s, I remember spending weeks trying again and again to reproduce a method to culture embryonic stem (ES) cells in a serum-free media, which was published in Cell (one of the most prestigious and selective journals in biological sciences). In the end I had to travel to the lab outside of USA that published the article to see how they do the experiment so I can learn.

Then, for my project on genetic screening in ES cells, I had to learn how to isolate a plasmid (a circular piece of DNA) from these cells. Getting plasmids out of bacteria is a common technique, but getting plasmids out of eukaryotic cells such as ES cells is a very specific case and different problem. There were a lot of references in the literature; however, most of them were giving partial information with further references to articles in obscure, hard to find journals. Compiling pieces of information together and repeating the experiment for a few weeks, I was able to master this method and get my experiment going.

Looking around myself I saw other scientists, post-docs and Ph.D. students suffering from the same problem. They came to science to solve great problems (cancer, stem cell therapy, etc…) and instead they were spending their time to reproduce experiments that sometimes were published 10-20 years ago.

What are the reasons for this phenomenon? Based on my own experience of working in different university labs for nearly 10 years and articles published on this subject elsewhere, I think the roots of the problem are in the current structure of scientific research and scientific publishing.

Click here to continue reading!

Crowd Sourcing for Malaria? Game On!

Posted on May 4, 2012 by Neal Moawed

The internet strikes again! UCLA researchers have called upon the user resources of the world wide web to allow the public to diagnose patients with malaria-, accurately and, with almost no training. Assuming that large groups of people can learn the rules of a game, and thus be trained to recognize infectious diseases with the accuracy of medical experts, UCLA’s Henry Samueli School of Engineering and Applied Science and the David Geffen School of Medicine at UCLA have built an on-line game where players distinguish malaria-infected blood cells from healthy ones by looking at images obtained from microscope slides.

This image came from the game's website

Found here, the game asks players to decide if a cell is positive or negative for malaria. There is a performance bar on the top of the screen, as well as an aggregate score for the game, showing players their progress and accuracy. After the first couple of rounds, the player learns what earns points and what doesn’t, in terms of cell’s appearance. UCLA has found that when using this approach, allows players to collectively diagnose malaria-infected blood cells with an accuracy within 1.25 percent of the accuracy of a trained medical professional.

Taking a page from Fold.it, UCLA taps into what game developers have known for a long time: there are people who will do anything repetitively for hours to get a high score. The game is available cross- platform, allowing for mobile access and play. After the first couple of rounds, I recognized the pattern and quickly realized what to look for in an infected cell.

This approach is in great contrast to the current gold-standard for malaria diagnosis: trained pathologists using a conventional light microscope to view images of cells and count the number of malaria-causing parasites. While this is effective, the game allows people who have not been trained to diagnose individual cells, and aggregate the results at no cost. On a large enough scale, a diagnosis can be delivered more quickly, which benefits resource-poor countries and with a high degree of collective accuracy.

I tried the game myself and after the first few rounds, I recognized the pattern and quickly realized what to look for in an infected cell. My current high score is 1102, which I fully intend to decimate during my lunch break. Play the game here, and let us know how you do in the comments.

Content for this post came from this press release. Images came from the UCLA page found here.

Studies show only 10% of published science articles are reproducible. What is happening?

Posted on May 3, 2012 by Moshe Pritsker

Studies show a very low reproducibility for articles published in scientific journals, often as low as 10-30%. Here is a partial list:

The biotech company Amgen had a team of about 100 scientists trying to reproduce the findings of 53 “landmark” articles in cancer research published by reputable labs in top journals.
Only 6 of the 53 studies were reproduced (about 10%).
Scientists at the pharmaceutical company, Bayer, examined 67 target-validation projects in oncology, women’s health, and cardiovascular medicine. Published results were reproduced in only
14 out of 67 projects (about 21%).
The project, PsychFileDrawer, dedicated to replication of published articles in experimental psychology, shows a
replication rate 3 out of 9 (33%) so far.

My hair is standing on end as I read these numbers! Unbelievable! The reproducibility of published experiments is the foundation of science. No reproducibility – no science. If these numbers are true, or even half-true, it means there is something fundamentally wrong in today’s system of scientific research and education.

On a practical level, the US government gives nearly
$31 billion every year in science funding through NIH only, which is mainly distributed in research grants to academic scientists. The 10% reproducibility rate means that 90% of this money ($28 billion) is wasted. That’s a lot. How are the tax-payers supposed to respond to the scientist plight for more research funding given these numbers? Would you give more of your own money to someone who delivered you such a result?

Beyond the practicalities, there is an interesting philosophical question. Since the middle of the 20-th century, life science research concepts and technologies have rapidly grown from the discovery of DNA to sequencing of genomes. Amazing technologies like microarrays, mass spectrometry, high-throughput assays, imaging, and robotic surgeries were introduced, making biology a data-rich science. One would expect that all these new tools would make science more rigorous and precise, but something opposite is happening.

Any ideas what could be the reason? Who is the main culprit? Write here in comments or write me directly at moshe.pritsker@jove.com.

JoVE Blog

This is the official blog of The Journal of Visualized Experiments (JoVE), the first and only video journal for biological, medical, chemical and physical research indexed in PubMed.

Visit JoVE