A Novel Approach to On-Line Handwriting Recognition Based on

ANovelApproachtoOn-LineHandwritingRecognitionBasedon

BidirectionalLongShort-TermMemoryNetworks

MarcusLiwicki1

1

AlexGraves2

J¨urgenSchmidhuber2,3

HorstBunke1

Inst.ofComputerScienceandAppliedMathematics,

UniversityofBern,Neubr¨uckstr.10,3012Bern,Switzerland2

IDSIA,Galleria2,6928Manno-Lugano,Switzerland

3

TUMunich,Boltzmannstr.3,85748Garching,Munich,GermanyAbstract

Inthispaperweintroduceanewconnectionistapproachtoon-linehandwritingrecognitionandaddressinpartic-ulartheproblemofrecognizinghandwrittenwhiteboardnotes.Theapproachusesabidirectionalrecurrentneu-ralnetworkwiththelongshort-termmemoryarchitecture.Weusearecentlyintroducedobjectivefunction,knownasConnectionistTemporalClassi cation(CTC),thatdirectlytrainsthenetworktolabelunsegmentedsequencedata.Ournewsystemachievesawordrecognitionrateof74.0%,comparedwith65.4%usingapreviouslydevelopedHMM-basedrecognitionsystem.

1.Introduction

Althoughtheproblemofhandwritingrecognitionhasbeenconsideredformorethan30years[1,12,16],therearestillmanyopenissues,especiallyinthetaskofun-constrainedhandwrittensentencerecognition.Handwritingrecognitionistraditionallypidedintoon-lineandoff-linerecognition.Inon-linerecognitionatimeorderedsequenceofcoordinates,representingthemovementofthetipofpen,iscaptured,whileintheoff-linemodeonlytheimageofthetextisavailable.

Inthispaperweconsideranon-linerecognitionprob-lem,namelytherecognitionofnoteswrittenonawhite-board.Thisisarelativelynewtask.Aspeoplestand,ratherthansit,duringwritingandthearmdoesnotrestonata-ble,handwritingrenderedonawhiteboardisdifferentfromhandwritingproducedwithapenonawritingtablet.De-spitesomeadditionaldif culty,thewhiteboardmodalityisimportantinseveralapplications,suchasthedocumenta-tionoflecturesormeetings.Intheparticularapplication

underlyingthispaperweaimatdevelopingahandwritingrecognitionsystemtobeusedinasmartmeetingroomsce-nario[17],inourcasethesmartmeetingroomdevelopedintheIM2project[11].Smartmeetingroomsusuallyhavemultipleacquisitiondevices,suchasmicrophones,cam-eras,electronictablets,andawhiteboard.Inordertoallowforindexingandbrowsing[18],automatictranscriptionoftherecordeddataisneeded.

Inthispaper,weintroduceanovelapproachtoon-linehandwritingrecognition,usingasinglerecurrentneuralnet-work(RNN)totranscribethedata.ThekeyinnovationisarecentlyintroducedRNNobjectivefunctionknownasCon-nectionistTemporalClassi cation(CTC)[5].Whereaspre-viousobjectivefunctionsonlytrainRNNstolabelinpid-ualdatapointswithinasequence,CTCtrainsthenetworktolabeltheentireinputsequenceatonce.Thismeansthenet-workcanbetrainedwithunsegmentedinputdata(animpor-tantrequirementforon-linehandwriting,wherecorrectseg-mentationofinpiduallettersisoftendif culttoachieve),andthe nallabelsequence(inthiscase,thecharacterleveltranscription)isgivendirectlybythenetworkoutput.

InourwriterindependentexperimentsontheIAM-OnDB[9]1,awordrecognitionrateofupto74.0%hasbeenachieved.Theseresultsaresigni cantlyhigherthenthosefrompreviousexperimentswithanHMM-basedsys-tem[10].

Therestofthepaperisorganizedasfollows.Section2givesanoverviewoftheproposedsystem.InSection3themainstepsforpreprocessingthedataandextractingthefeaturesarepresented.Section4introducesthenewclassi cationapproachforhandwritingrecognition.Exper-imentsandresultsarepresentedinSection5,and nallySection6drawssomeconclusionsandgivesanoutlooktofuturework.

1http://www.iam.unibe.ch/ fki/iamondb/

Figure1.Illustrationoftherecording

2.SystemOverview

TheeBeaminterface2isusedforrecordingthehandwrit-ing.Itallowstheusertowriteonawhiteboardwithanor-malpeninaspecialcasing,whichsendsinfraredsignalstoatriangularreceivermountedinoneofthecornersofthewhiteboard.Theacquisitioninterfaceoutputsasequenceof(x,y)-coordinatesrepresentingthelocationofthetipofthepentogetherwithatimestampforeachlocation.Theframerateoftherecordingsvariesfrom30to70framespersecond.AnillustrationisshowninFig.1.

Thesystemdescribedinthispaperconsistsofthreemainmodules:theon-linepreprocessing,wherenoiseintherawdataisreducedandthetextlineisnormalizedwithre-specttoskew,slant,widthandheight;thefeatureextrac-tion,wherethesequenceofpointsistransformedintoase-quenceoffeaturevectors;andtherecognition,whereanASCIItranscriptionofthehandwritingisgenerated.

3.Preprocessing

Beforefeatureextractioncanbeapplied,therecordeddatahastobenormalized.Thisisaveryimportantstepinhandwritingrecognitionsystems,becausethestylesofthewritersdifferwithrespecttoskew,slant,heightandwidthofthecharacters.Ifwedonotapplyanypreprocessingandusetherawfeatures,therecognitionrateissigni cantlylower.Thepreprocessingstepsappliedinthecurrentsystemhavebeenintroducedin[10],butforthepurposeofcomplete-ness,wegiveashortoverviewbelow.

Therecordedon-linedatausuallycontainnoisypointsandgapswithinstrokes,whicharecausedbylossofdata.Hence,weapplysomenoise lteringoperations rst.Thecleanedtextdataisthenautomaticallypidedintolinesus-ingsomesimpleheuristics.Astheskewoftensigni cantly

2eBeam

SystembyLuidia,Inc.-

http://doc.guandang.net

Figure2.Splittingatextlineintosubpartsandskew

correction

Figure3.Baselineandcorpuslineofanex-amplepartofatextline

varieswithinthesameline,wesplitlinesintosubparts.AnexampleofsplittingisshowninFig.2(upperline).

Nextthesubpartsarecorrectedwithrespecttotheirskewusingalinearregression.ThisprocessisillustratedinFig.2withtheresultingtextlineshowninthelowerpart.Forslantnormalization,wecomputethehistogramoverallanglesbetweenthelinesconnectingtwosuccessivepointsofthetrajectoryandthehorizontalline[8].Subsequently,thehis-togramisprocessedtorecovertheskewangle.Aftertheseoperations,weremovedelayedstokes,e.g.thecrossingofa“t”orthedotofan“i”,usingsimpleheuristics.Thenextim-portantstepisthecomputationofthebaselineandthecor-puslinebycomputingtwolinearregressionlinesthroughtheminimaandmaximaofthey-coordinatesofthestrokes.Figure3illustratestheestimatedbaselineandthec …… 此处隐藏：15233字，全部文档内容请下载后查看。喜欢就下载吧 ……