Any research findings, including health-related ones, are only as good as the design and execution of the underlying research. Being an informed user of research findings requires some ability to understand methodological issues relevant to the research. A good research writeup will include a discussion of those issues. I am always interested in research or articles that discuss these methodological concerns. The recent edition of Health Services Research contains a couple of articles on survey methodologies. (HSR Articles) In the first the authors sought to understand whether phone surveys still had sufficient validity. At a time when most households had a phone, such surveys were a standard way to gather general health information from a large population. But in the last decade response rates have dropped, to below 50% in most states, and as people rely exclusively on cell phones, less than half of all homes now have a landline. Cell phone numbers are not necessarily tied to a particular geography which complicates health research attempting to understand differences in health issues among areas. The authors created a random sample of addresses and attempted to match them to phone numbers. If they could, they contacted the resident by phone for a survey. For those addresses that couldn’t be matched to a phone, a letter was sent saying that an in-person interviewer would be coming and offering a small incentive to complete the survey. A similar follow-up was done with the people who did have phones but didn’t respond to the initial survey attempt by phone. Overall, 20% of people with a matched phone number responded, 43% of those who initially didn’t respond by phone did respond to an in-person interview, and 41% of households without a matched phone responded to the in-person interview. The demographic characteristics and health status of those who responded by phone, those who didn’t respond by phone and those who did not have a phone were analyzed. The authors wanted to see if the results were affected by differences in these groups. As you would expect, there were large differences. Those with a phone who responded to a phone survey were on average much more likely to be over 65, to be a non-hispanic white, to be married and to be college graduates. Those households with a phone but which did not respond to a phone survey, but did to an in-person one, shared some characteristics but were more likely to be younger, male and a minority. The group without a matched phone in the household looked like the population as a whole. There were significant differences in health conditions among the groups. These differences were significant enough, and varied in ways that were not predictable, to indicate that a survey based solely on phone contacts is likely to have flawed findings. The primary cause of the variance appears to be the gap between those who respond to a telephone or in-person survey and those people who don’t respond to either, rather than just phone versus in-person results. In other words, people who don’t respond via any channel to a survey have a health status and other health care behaviors that differ in unpredictable ways from those people who do respond. So survey findings need to be used cautiously. Tomorrow we will cover the other methodology article.
✅ Subscribe via Email
About this Blog
The Healthy Skeptic is a website about the health care system, and is written by Kevin Roche, who has many years of experience working in the health industry. Mr. Roche is available to assist health care companies through consulting arrangements through Roche Consulting, LLC and may be reached at [email protected].
Healthy Skeptic Podcast
This is an outstanding report on total global drug spending and trends, with projections out to 2025. It helps you understand this important area of health care, which does much...
June 1, 2021
MedPAC 2019 Report to Congress
June 18, 2019
Now here is a tale from the glory days of epidemic investing, as the Feds pumped trillions into the economy and a lot of it went to investing, some to...
January 10, 2023
Another merger of two struggling “digital” health companies, which use virtual reality as a product design. Apparently you are less likely to drown in high seas when you are holding...
December 18, 2022
Mercer, a large benefits consulting firm, confirms that health plan premiums are rising in 2023, publishing survey results showing that employers expect an increase of costs by at least 5%....
December 9, 2022
Access ACO Care Management Chronic Disease Comparative Effectiveness Consumer Directed Health Consumers Devices Disease Management Drugs EHRs Elder Care End-of-Life Care FDA Financings Genomics Government Health Care Costs Health Care Quality Health Care Reform Health Insurance Health Insurance Exchange HIT HomeCare Hospital Hospital Readmissions Legislation M&A Malpractice Meaningful Use Medicaid Medical Care Medicare Medicare Advantage Mobile Pay For Performance Pharmaceutical Physicians Providers Regulation Repealing Reform Telehealth Telemedicine Wellness and Prevention Workplace