According to two studies published in the journal Health Affairs, health policy research is too often flawed and poorly targeted toward the questions that are most important to decisionmakers.
"The potential for a disconnect between the information needs of decision makers and the research being produced for them may stem, in large part, from fundamental differences in the priorities and cultures of researchers and decisionmakers, whether from the public or private sector," said Claudia Schur, vice president and director of the Center for Health Research and Policy, Social and Scientific Systems, in Silver Spring, Md., and coauthors in a study supported by the California HealthCare Foundation.
Schur and her colleagues point to the need for "decision-based evidence making," defined by CHCF President Mark Smith as seeking "to figure out which decisions need making, and what evidence is necessary in order to make them in the real world, in our lifetime."
The researchers say more attention has been paid to the research needs of public-sector health decision makers than to the needs of their private-sector counterparts. They designed a multi-step inquiry culminating in an online survey of respondents from California's larger health plans and employers.
Respondents were asked to rank 20 issues in terms of importance to the decisions they are required to make in their current positions. "Engaging consumers in their own care management" was ranked the highest, followed closely by "understanding underlying drivers of healthcare costs."
For more than half of the 20 issues – and three of the five issues ranked highest in importance – fewer than 10 percent of respondents reported that existing research was adequate for their needs.
There is a need for a long-term, "systematic approach to linking the needs of private-sector decisionmakers, funders of research and those who conduct the research," say Schur and coauthors Marc Berk of SSS, former SSS research analyst and current Emory University doctoral student Lauren Silver, Jill Yegian of CHCF, and Michael O'Grady of NORC in Bethesda, Md.
Two such strategies garnered majority support from survey respondents: 58 percent "strongly supported both the sharing of data by employers and health plans in order to create new sources of information and the creation of a clearinghouse for compiling, assessing and disseminating research, with the clearinghouse sponsored by a foundation or another nonprofit organization."
Because the designs of many health policy research studies are fundamentally flawed, policymakers and the public do not always know if they can believe even work appearing in the highest-impact journals, Sumit Majumdar and Stephen Soumerai say in a second study published in Health Affairs.
Majumdar and Soumerai reject the argument that only randomized controlled trials (RCTs) produce trustworthy evidence. They say RCTs are rarely possible in health policy research, and they defend the use of valid non-randomized or quasi-experimental study designs that seek to learn from the natural experiments created by health policy interventions.
Insisting on the exclusive use of RCTs "inappropriately lumps together valid evidence from strong nonrandomized designs (that is, before-and-after studies with concurrent controls or the interrupted time series study in which a policy causes a sudden, visible change in trend) with evidence from weak designs that permit little causal influence (that is, the commonly conducted cross-sectional analysis that looks at outcomes only after a policy has been implemented)," write Majumdar, an associate professor in the Department of Medicine at the University of Alberta in Edmonton, and Soumerai, a professor in the Department of Population Medicine at Harvard Medical School and the Harvard Pilgrim Health Care Institute in Boston, Mass.
The problem, say Majumdar and Soumerai, is that researchers too often use weak study designs. For example, according to a recent systematic review of 13 studies of financial incentives for physicians to improve the quality of pharmaceutical care, not one study "met minimal methodological standards to ensure validity. "Most consisted of cross-sectional analyses or longitudinal studies comparing providers' activities before and after financial incentives were introduced, without controlling for pre-existing trends in study outcome or important factors ('confounders') that might have accounted for the policy 'effects,'" they said.
Similarly, under economic stimulus legislation passed earlier this year, the government is spending tens of billions of dollars on nationwide implementation of predominately commercial health information technology systems, despite the fact that this has not yet been proven to increase the quality or quantity of life, the two researchers say.
Majumdar and Soumerai suggest a number of potential steps "to improve the health of health policy research," including providing researchers, clinicians, and policymakers "with a more solid grounding in the comparative rigor of various quasi-experimental policy designs." They also call for better visual displays of time trends before and after policy implementation, and other strategies.
In addition, Majumdar and Soumerai suggest the creation of "secondary review" health policy journals, akin to highly trusted publications such as ACP Journal and Journal Watch that exist in the clinical world. These publications "apply quality filters and review many articles; summarize the results of important studies; and comment on the validity and broader context of the work," the two researchers explain.