More on Survey Methodology

Yesterday we reviewed one article in Health Services Research on survey methodology; today we review a second study.  (link in yesterday’s post)   The authors sought to identify the effects of administration method on the CAHPS survey.  CAHPS stands for Consumer Assessment of Healthcare Providers and Systems and is a survey used by Medicare and other payers to determine patient satisfaction with providers and health plans.  The results often have financial ramifications, so accuracy and validity is important.  The researchers compared use of web survey with a portal invitation, use of a web survey with an email invitation, use of a web survey with a mail invitation and a mail only survey.  The objectives were to assess if there were differences in the characteristics of the groups who responded or didn’t respond to each method, or in the groups’ substantive responses to the survey.  Three primary care practices in the Boston area were the study sites.  Patients in these practices had access to a web portal.  In all branches of the study initial non-respondents were sent reminders about the survey.  The survey itself was the standard CAHPS instrument.

The mail protocol alone, or when mail was used as the reminder method, produced twice the response rate of email only methods.  There was no significant difference in response rate from being able to access the survey directly by a link in an email versus an email that told the respondents that the survey was available in the portal.  There was no difference in demographic characteristics between those who responded to these two methods of accessing the survey on the internet; and there was no significant difference in responses among these two groups.  Those patients for whom there was not an email address and therefore were in the mail only group, were somewhat less likely to respond to the survey.  This group was also much more likely to be older and less educated and somewhat more likely to be male.  But there was little difference in their substantive responses to the survey.  In general, despite differences in response rates among the study arms, there was almost no difference in substantive responses.  The results would suggest that if surveying organizations want to maximize response rates, and minimize the costs of doing the surveys, they should use mail.  But if an internet-based method is used, the survey results would not be significantly different.