Survey Research Methodology

Any research findings, including health-related ones, are only as good as the design and execution of the underlying research.  Being an informed user of research findings requires some ability to understand methodological issues relevant to the research.  A good research writeup will include a discussion of those issues.  I am always interested in research or articles that discuss these methodological concerns.  The recent edition of Health Services Research contains a couple of articles on survey methodologies.    (HSR Articles)   In the first the authors sought to understand whether phone surveys still had sufficient validity.  At a time when most households had a phone, such surveys were a standard way to gather general health information from a large population.  But in the last decade response rates have dropped, to below 50% in most states, and as people rely exclusively on cell phones, less than half of all homes now have a landline.  Cell phone numbers are not necessarily tied to a particular geography which complicates health research attempting to understand differences in health issues among areas.  The authors created a random sample of addresses and attempted to match them to phone numbers.  If they could, they contacted the resident by phone for a survey.  For those addresses that couldn’t be matched to a phone, a letter was sent saying that an in-person interviewer would be coming and offering a small incentive to complete the survey.  A similar follow-up was done with the people who did have phones but didn’t respond to the initial survey attempt by phone.  Overall, 20% of people with a matched phone number responded, 43% of those who initially didn’t respond by phone did respond to an in-person interview, and 41% of households without a matched phone responded to the in-person interview.  The demographic characteristics and health status of those who responded by phone, those who didn’t respond by phone and those who did not have a phone were analyzed.  The authors wanted to see if the results were affected by differences in these groups.  As you would expect, there were large differences.  Those with a phone who responded to a phone survey were on average much more likely to be over 65, to be a non-hispanic white, to be married and to be college graduates.  Those households with a phone but which did not respond to a phone survey, but did to an in-person one, shared some characteristics but were more likely to be younger, male and a minority.  The group without a matched phone in the household looked like the population as a whole.  There were significant differences in health conditions among the groups.  These differences were significant enough, and varied in ways that were not predictable, to indicate that a survey based solely on phone contacts is likely to have flawed findings.  The primary cause of the variance appears to be the gap between those who respond to a telephone or in-person survey and those people who don’t respond to either, rather than just phone versus in-person results.  In other words, people who don’t respond via any channel to a survey have a health status and other health care behaviors that differ in unpredictable ways from those people who do respond.  So survey findings need to be used cautiously.  Tomorrow we will cover the other methodology article.