The dilemma of survey nonresponse(2)

Nonresponsepresentstwoproblemsfortheinterpretationofresearchresults.First,itreducesthesamplesizeand,thus,decreasestheprecisionwithwhichresultscanbestated.Second,andmoreimportant,isthefactthatitintroduceserrorintothesamplingprocessbyexcludinganonrandomsubsetofthepopulation.Iftheexcludedsubsetisdifferentfromthoseremainingwithregardtothesurveydata,theresultsofthesurveywillbebiased.Thefirstissue,ifitoccursalone,isneitherparticularlyseriousnorparticularlydifficulttoremedy.Inmostcases(exceptwhereacensushasbeenattempted),additionalrespondentscanberecruitedinnumberssufficienttocompensateforexpectedrefusals.Ifthisstrategyisnotfeasible,statisticaltechniquesautomaticallyaccountforsmallersamplesizebyincreasingtheconfidenceintervalsaroundestimatesofpopulationparametersorbydecreasingthesignificanceofinferentialstatistics.However,bothoftheseresponsestoasmallersamplearevalidonlyiftheresultsareunbiased,thatis,respondentsarerepresentativeoftheentirepopulation.Whennonrespondentsdiffersignificantlyfromsurveyrespondents,increasingthesizeofthesampleorusingstatisticsthatreflectsamplesizewithoutaddressingthebiascausedbynonresponseservesonlytobolsterreaderconfidenceinpotentiallyinaccu-rateresults.

J.Burkell/Library&InformationScienceResearch25(2003)239–263243

Nonresponsealwaysintroducessomebiasintothesample,evenifthatbiascannotbedetectedbystatisticaltestsonavailablecomparisonmeasures.Nonresponsearisesdirectlyasaresultofanindividual’schoice(e.g.,thechoicenottocompletethesurvey)oranindividual’scharacteristics(e.g.,thecharacteristicofhavingmovedandthereforehavinganinvalidmailingaddress).Inthesechoicesand/orcharacteristics,nonrespondentsbydefinitiondifferfromrespondents.Thegroupsalsodifferinthequalitiesthatdeterminethechoicesand/orarecorrelatedwiththecharacteristics.Thus,publiclibrarypatronswhochoosenottocompleteasurveyregardingtheimportanceofInternetaccessarealsolikelytousetheserviceless,andthegroupoflibrarianswhodonotreceiveajobsatisfactionsurveybecausetheyhavechangedpositionsislikelytoincludemanywhochangedjobsbecausetheyweredissatisfiedwiththeirwork.Occasionally,researchersarguethatiftheycannotidentifyanysystematicdifferencebetweenrespondentsandnonrespondents,thereisnobiascausedbynonresponse.Statisticalcomparisonsofrespondentsandnonrespondentsonavailabledescriptors,however,canonlyeliminatethepossibilitythatthetwogroupsdiffersignificantlyintermsofthemeasuredqualities.Itremainspossible,andevenlikely,thattherearedifferencesbetweenthegroups:eitherrealdifferencesonmeasuredvariablestoosmalltoreachstatisticalsignificanceordifferencesonothervariablesforwhichdataareunavailable.

Thequestion,therefore,isnotwhethernonresponsehasresultedinabiasedsample:thatanswerisalways‘‘yes.’’Theimportantissueiswhetherthebiasinfluencessurveyresults.Thereis,however,adearthofdirectresearchonthisquestion,foragoodreason:thedegreetowhichnonresponseaffectssurveyconclusionsdependsontherelationshipbetweenthevariablesofinterestandthecausesofnonresponse(Brehm,1993),andthisrelationshipislikelytodifferfromsurveytosurvey.Ifthereisnorelationship,nonresponsewillnotaffectthesurveyresults.If,however,thereisarelationship(eitherdirectorthroughathirdvariablethatinfluencesbothnonresponseandthedependentvariablesofinterest),nonresponsewillcompromisethesurveyconclusions.Ifresearcherswereabletomeasuretherelationshipbetweenselectionmechanismsandsurveyresultsfortheirparticularsurvey,itwouldbesimpletodeterminewhethernonresponserepresentedathreat.Thenatureofthatrelationship,however,isvirtuallyimpossibletodetermine,becauseassessingitrequirestheverythingthatnonresponseprecludes:analysisofdatafromnonrespondents.

Thisisthecentraldilemmaofnonresponse:theimpactofnonresponseonsurveydatacannotbedeterminedwithoutdata(eitheractualorestimated)fromnonrespondents.Nonresponseraisesthespectre(butnotthecertainty)ofbiasedresults;furthermore,thehigherthelevelofnonresponse,thegreaterthepotentialbias(Alexander,Alliger,&Hanges,1984;Chen,1996;Cochran,1963;vanGoor&Stuiver,1998;Viswesvaran,Barrick,&Ones,1993).Inisolatedcases,researchershavedemonstratedthatlowresponseratesdonotnecessarilycompromisesurveyresults(Keeter,Miller,Kohut,Groves,&Presser,2000;Visser,Krosnick,Marquette,&Curtin,1996).Thisfindingshouldnotbetakenasgeneralevidencethatnonresponsecanbeignored,however.Theimportantissueisrepresentative-ness:whethertherespondentsresemble(withregardtosurveyresults)thepopulationfromwhichtheyweredrawn(Cook,Heath,&Thompson,2000;Cook&Thompson,2001;Krosnick,1999;Thompson,2000).Itispossibletointerpretsurveyresultsevenwithhigh