International Journal of Document Analysis (2006) DOI 10.100

Abstract Feature selection for ensembles has shown to be an effective strategy for ensemble creation due to its ability of producing good subsets of features, which make the classifiers of the ensemble disagree on difficult cases. In this paper we present

InternationalJournalofDocumentAnalysis(2006)DOI10.1007/s10032-005-0013-6

ORIGINALPAPER

LuizS.Oliveira·MarisaMorita·RobertSabourin

Featureselectionforensemblesappliedtohandwritingrecognition

Received:8October2004/Revised:19September2005/Accepted15November2005cSpringer-Verlag2006

AbstractFeatureselectionforensembleshasshowntobeaneffectivestrategyforensemblecreationduetoitsabilityofproducinggoodsubsetsoffeatures,whichmaketheclas-si ersoftheensembledisagreeondif cultcases.Inthispa-perwepresentanensemblefeatureselectionapproachbasedonahierarchicalmulti-objectivegeneticalgorithm.Theun-derpinningparadigmisthe“overproduceandchoose”.Thealgorithmoperatesintwolevels.Firstly,itperformsfeatureselectioninordertogenerateasetofclassi ersandthenitchoosesthebestteamofclassi ers.Inordertoshowitsro-bustness,themethodisevaluatedintwodifferentcontexts:supervisedandunsupervisedfeatureselection.Intheformer,wehaveconsideredtheproblemofhandwrittendigitrecog-nitionandusedthreedifferentfeaturesetsandmulti-layerperceptronneuralnetworksasclassi ers.Inthelatter,wetookintoaccounttheproblemofhandwrittenmonthwordrecognitionandusedthreedifferentfeaturesetsandhid-denMarkovmodelsasclassi ers.Experimentsandcompar-isonswithclassicalmethods,suchasBaggingandBoosting,demonstratedthattheproposedmethodologybringscom-pellingimprovementswhenclassi http://doc.guandang.netparisonshavebeendonebycon-sideringtherecognitionratesonly.

KeywordsEnsembleofclassi ers·Featureselection·Handwritingrecognition·Multi-objectiveoptimization·Geneticalgorithms

L.S.Oliveira(B)Pont´ ciaUniversidadeCat´olicadoParan´a(PUCPR),ProgramadeP´os-Graduac¸a oemInform´aticaAplicada(PPGIA),RuaImaculada

o1155,PradoVelho,80215-901,Curitiba,Pr,BrazilConceic¸a

E-mail:soares@ppgia.pucpr.br

M.Morita·R.Sabourin´EcoledeTechnologieSup´erieure(ETS),Laboratoired’Imagerie,deVisonetd’IntelligenceArti cielle1100,rueNotreDameOuest,Montreal,Canada,H3C1K3

Presentaddress:M.Morita

HSBCBankBrazil,IT,Curitiba,PR,BrazilE-mail:marisa.e.morita@http://doc.guandang.net.br

1Introduction

Ensembleofclassi ershasbeenwidelyusedtoreducemodeluncertaintyandimprovegeneralizationperformance.Developingtechniquesforgeneratingcandidateensemblemembersisaveryimportantdirectionofensembleofclassi- ersresearch.Ithasbeendemonstratedthatagoodensem-bleisonewheretheindividualclassi ersintheensemblearebothaccurateandmaketheirerrorsondifferentpartsoftheinputspace(thereisnogainincombiningidenticalclas-si ers)[1–4].Inotherwords,anidealensembleconsistsofgoodclassi ers(notnecessarilyexcellent)thatdisagreeasmuchaspossibleondif cultcases.

Theliteraturehasshownthatvaryingthefeaturesub-setsusedbyeachmemberoftheensembleshouldhelptopromotethisnecessarydiversity[4–7].Traditionalfeatureselectionalgorithmsaimat ndingthebesttrade-offbe-tweenfeaturesandgeneralization.Ontheotherhand,en-semblefeatureselectionhastheadditionalgoalof ndingasetoffeaturesetsthatwillpromotedisagreementamongthecomponentmembersoftheensemble.TheRandomSub-spaceMethod(RMS)proposedbyHoin[5]wasoneearlyalgorithmthatconstructsanensemblebyvaryingthesubsetoffeatures.Morerecentlysomestrategiesbasedongeneticalgorithms(GAs)havebeenproposed[4,8,9].Allthesestrategiesclaimbetterresultsthanthoseproducedbytra-ditionalmethodsforcreatingensemblessuchasBaggingandBoosting.InspiteofthegoodresultsbroughtbyGA-basedmethods,theystillcanbeimprovedinsomeaspects,e.g.,avoidingclassicalmethodssuchastheweightedsumtocombinemultipleobjectivefunctions.Itiswellknownthatwhendealingwiththiskindofcombination,oneshoulddealwithproblemssuchasscalingandsensitivitytowardstheweights.

Ithasbeendemonstratedthatfeatureselectionthroughmulti-objectivegeneticalgorithm(MOGA)isaverypower-fultoolfor ndingasetofgoodclassi ers[10,11],sinceGAisquiteeffectiveinrapidglobalsearchoflarge,non-linearandpoorlyunderstoodspaces[12].Besides,itcanovercomeproblemssuchasscalingandsensitivitytowards

Abstract Feature selection for ensembles has shown to be an effective strategy for ensemble creation due to its ability of producing good subsets of features, which make the classifiers of the ensemble disagree on difficult cases. In this paper we present

theweights.KudoandSklansky[13]havecomparedsev-eralalgorithmsforfeatureselectionandconcludedthatGAsaresuitablewhendealingwithlarge-scalefeatureselection(numberoffeaturesisover50).Thisisthecaseofmostoftheproblemsinhandwritingrecognition,whichisthetestprobleminthiswork.

Inthislight,weproposeanensemblefeatureselec-tionapproachbasedonahierarchicalMOGA.Theunder-lyingparadigmis“overproduceandchoose”[14,15].Thealgorithmoperatesintwolevels.Theformerisdevotedtogenerateasetofgoodclassi ersbyminimizingtwocrite-ria:errorrateandnumberoffeatures.Thelattercombinestheseclassi ersinorderto ndanensemblebymaximizingthefollowingtwocriteria:accuracyoftheensembleandameasureofdiversity.

Recently,theissueofusingdiversitytobuildensembleofclassi ershasbeenwidelydiscussed.Severalworkshavedemonstratedthatthereisaweakcorrelationbetweendi-versityandensembleperformance[16,17].Inlightofthis,someauthorshaveclaimedthatdiversitybringsnobene tsinbuildingensembleofclassi ers[18],ontheotherhand,otherssuggestthatthestudyofdiversityinclassi ercombi-nationmightbeoneofthelinesforfurtherexploration[19].Inspiteoftheweakcorrelationbetweendiversityandperformance,wearguethatdiversitymightbeusefultobuildensemblesofclassi ers.Wedemonstratedthroughexperi-mentationthatusingdiversityjointlywithperformancetoguideselectioncanavoidover ttingduringthesearch.Inor-dertoshowrobustnessoftheproposedmethodology,itwasevaluatedintwodifferentcontexts:supervise …… 此处隐藏：50994字，全部文档内容请下载后查看。喜欢就下载吧 ……