The dilemma of survey nonresponse(3)

244J.Burkell/Library&InformationScienceResearch25(2003)239–263

levelsofnonresponseifitcanbearguedthattherespondentsarerepresentativeofthepopulation.Forexample,CookandThompson(2001)arguedthat,despitearesponserateofapproximately14%(Thompson,2000),theresultsofaWeb-basedLibQUAL+studycouldbegeneralizedtothelargerpopulationbecausetherespondentswereshownnottodiffersignificantlyfromthepopulationonvariousmeasuredvariables(Thompson,2000).TheapproachusedbyCookandThompsonisonestrategyfordealingwithnonresponse.Thenextsectionofthisarticlepresentsthisandothersuggestionsforaddressingthedilemmaofnonresponse.

3.Dealingwiththedilemmaofnonresponse

Asafirststrategyfordealingwithnonresponse,theresearchershouldaddresstheproblematthesourcebytakingadvantageofmethodologicalvariationsproventoreducenonresponse.Althoughtheseeffortsareimportant,inmostcasestheywillnotresultina100%responserate.Theresearcheristhereforeleftwiththeproblemofinterpretingsurveyresultsinthecontextofsomeremaininglevelofnonresponse.

Reportsofsurveyresultsshouldalwaysindicatethedegreetowhichinitiallyselectedrespondentsparticipateinthesurveybyreportingtheresponserate(responserate=1ànonresponserate).Thereareseveraldefinitionsofresponserate,whichdifferinthewaythatincompletesurveys,noncontacts,refusals,andotheroutcomesaretreated(AmericanAssociationforPublicOpinionResearch,2000).Thisarticleusesthe‘‘maximumresponserate’’definedbytheAmericanAssociationforPublicOpinionResearch:responserate=(completeresponses+partialresponses)/totalnumberintheeligiblesample.Whentheresponserateislessthan100%,theresearcherhasthreechoices:(1)ignorethenonresponse,preferablyonasubstantivebasis(e.g.,alowlevelofnonresponseordocumentedreasontobelievethatnonresponsewouldnotaffectresearchresults);(2)limitthesurveyconclusionsbasedontherateandpresumedimpactofnonresponse;or(3)attempttoassessand,ifnecessary,correctforthesamplingbiasintroducedbynonresponse.3.1.Addressingtheproblematthesource:Methodstoreducenonresponse

Themethodofsurveyadministrationisanimportantdeterminantofresponserates.Fourmethodsthatarewidelyusedinsurveysaremail,telephone,face-to-face,ande-mailadministration(somesurveysarealsodistributedbyfax,butthesearefew,andthemethodhasquicklybeensupplantedbye-mailadministration).SurveyshavealsobeendeliveredovertheWeb(e.g.,Perkins&Yuan,2001),butWebsurveysofprobabilitysamplesgenerallyrequirerecruitmentofrespondentsusingoneoftheothermeansofadministration(Couper,2000).Ofthefourmethodswidelyusedwhenarepresentativesampleisdesired,face-to-faceadministrationtendstoresultinthehighestresponserates(Hox&Deleeuw,1994),althoughthiseffectmaydifferacrossdemographicgroups(Krysan,Schuman,Scott,&Beatty,1994).Telephonesurveysdemonstratethenexthighestresponserates,followedbyregularmailandthene-mail(Hox&Deleeuw,1994;Kettleson,1995).

J.Burkell/Library&InformationScienceResearch25(2003)239–263245

Althoughresponseratesarelowestformailande-maildelivery,thesemethodsarealsotheleastexpensiveandthereforearewidelyusedinsurveyresearch.Dillman(2000)hasdevelopeda‘‘tailoreddesignmethod’’toreducetheerrorassociatedwithnonresponse(amongotherfactors)inmailande-mailsurveys.Hesuggestedthatsurveyresponsecanbemaximizedbythefollowing:(1)establishingtherespondent’strust,(2)increasingtheexpectedrewardsofparticipation,and(3)reducingthesocialcostsofparticipation.SomeofthespecificstrategiessuggestedbyDillmanarerelativelylow-costandsimpletoimplementinanysurvey.Thesestrategiesincludeprenotification,personalizedcoverletters,theuseofreminders,incentiveswiththeinvitationtoparticipate,andstamped,self-addressedenvelopesforthereturnofmailsurveys.Researchsuggeststhat,ofthesemethods,smallmonetaryincentives(i.e.,$1to$2)includedwiththeinitialsurveymailoutarethesinglemosteffectivestrategythatresearcherscanusetoincreaseresponserates(Church,1993;Fox,Crask,&Kim,1988;Gajraj,Faria,&Dickinson,1990;Helgesen,Voss,&Terpening,2002;Hopkins&Gullickson,1992;Singer,VanHoewyk,&Maher,2000).Meta-analysesofstudiesofmailsurveyresponserateshaveofferedsupportforthestrategiessuggestedbyDillman(Foxetal.,1988;Hopkins&Gullickson,1992;Yammarino,Skinner,&Childers,1991),indicatingthattheimpactofinterventionstoincreaseresponseratesisconsistentacrosspopulations(Green,Boser,&Hutchinson,1998).Hart(1998)usedmanyofDillman’ssuggestionsforasurveyoftherelationshipsbetweenworkrolesandinformationgatheringforcollegefacultyandachievedaresponserateof84%.

Inadditiontothesestrategies,researchersshouldconsiderselectingasmallerinitialsampleandconcentratingeffortsandresourcesonachievingahighresponserate(Wayne,1975–76).Researchersmaychoosetouseoneofthemoreexpensivemethodstocollectdatafromthesmallersample(e.g.,telephoneinterviews),theymaychoosetousemultiplemethodsinsolicitingresponse(Roselle&Neufeld,1998),ortheymayprovideincentivestothoseinvitedtoparticipate.Althoughasmallersamplelimitstheprecisionwithwhichresultscanbestated,thisdisadvantageisoffsetbythereductioninbiasassociatedwithanincreasedresponserate.3.2.Ignoringnonresponse

Whencanyousimplyignorenonresponse?Theshortanswer,correctinthemostabsolutesense,isnever.Althoughlowresponseratesdonotnecessarilycompromisesurveyresults(Keeteretal.,2000;Krosnick,1999,Visseretal.,1996),anylevelofnonresponsecouldbiasconclusions(Jones,1996).Therealproblemisthattheresearchercannotknowwhethernonresponseaffectssurveyconclusions,becausethisrequiresinformationaboutthediffer-encesbetweenrespondentsandnonrespondents.Nonetheless,someconditionscanjustifyignoringthenonresponse,andthefactorsthatshouldbetakenintoaccountareoutlinedinthissection.Researchersshouldnot,however,seethisasapanaceaordefaultstrategyfordealingwithnonresponse.Ignoringtheproblemisappropriateonlyinasmallproportionofcases,andtheresearchermustensurethattheuseofthestrategyisjustified.