Improved representations and hardware implementation of the

Probabilistic computing forms a relatively new computational style, of significant practical interest because stochastic behaviour is common and must be taken into accountin in biological and other real-world processes. We examine a particular stochastic A

ImprovedrepresentationsandhardwareimplementationoftheHelmholtzMachine

AlexanderAstaras,RyanDalzell,AlanMurray,RobinWoodburn

Dept.ofElectronicsandElectricalEngineering

TheUniversityofEdinburgh

TheKing’sBuildings,May eldRd,EdinburghEH93JL,Scotland

aa,rwd,afm,martin@ee.ed.ac.uk

April6,2000

Probabilistic computing forms a relatively new computational style, of significant practical interest because stochastic behaviour is common and must be taken into accountin in biological and other real-world processes. We examine a particular stochastic A

theBoltzmannMachine.Howeverthesemeth-odssufferfromverylongsettlingtimesinthesamplingwhichcannotbecompensatedforbyahardwareimplementationduetothecomplexityofthealgorithms.TheHelmholtzMachine(HM)isavariantofExpectationMaximisationthatfea-turessimplebinarystochasticunitsandalearn-ingrulewhichissimpleandpurelylocal.Hence,theHMisveryamenabletoimplementationinhardware.2..1.

AdescriptionoftheHelmholtzMa-chine(HM)

TheHM[2,3]isanunsupervised,stochasticneuralnetworkwhichattemptstobuildaprob-abilisticmodelofdatainahierarchicalfashion.Byaprobabilisticmodel,wemeanthatthehid-denunitsofthenetworkchoosestatesaccordingtoaprobabilitydistributionratherthandeterm-inisticallyasinanMLP.TheHMcomprisestwocomplementarynetworks,abottom-up,recogni-tionnetworkandatop-down,generativenetwork(seeFigure1).

TheHMistrainedbythewake-sleepal-gorithm,wherebytwountrainednetworkscaneachprovidethetargetsnecessaryfortheothertolearn;overtimetherecognitionnetworklearnstoextractlatentvariablesfromthedatainitshid-denunits,andthegenerativenetworklearnstoreproducethedatadistributionfromthesevari-ables.Asensor-fusionapplicationwouldapplyasimpleclassi ertothehiddenunitsofthere-cognitionnetworktoextractinterestinginforma-tionfromthesystem.Furthermore,examinationoftheconditionalprobabilitiesinthegenerativemodelmayprovideinformationonhowthedata-fusiontaskisbeingperformedandshowifanysensordriftoccursovertime.2..2.

TheapplicationoftheHelmholtzMa-chinetosensor-tracking

TotestthehypothesisthattheHMcanbeusedfortrackingnon-stationaritiesinanenvironmentwithoutseriousalterationtoitsoriginalinternalmodel,severalsimulationswereperformed.Thedatasetusedwasamixturemodelwithfourmix-tures,eachofwhichspeci edaprobabilitydis-tributionovernineindependentbinaryvariables.Thedetailsofthedistributionsusedarethesameasin[1].AtwolayerHMwithnineinputunitsandsixhiddenunitswastrainedonthedata.TheperformancemeasureusedwasHelmholtzFreeEnergy(HFE),whichisthecostfunctionminimisedbytheHelmholtzMachineandisno-tionallyequivalenttothenegative-logprobabil-

ityofthedataunderthemodel[2].Wake-sleeplearningwasperformeduntiltheHFEstabilisedandthenoneoftheinputunit’sdistributionswasslowlydrifted.Twoexperimentalrunswereper-formed,onewithlearningduringdriftandan-otherwithout.TheresultsarepresentedinFig-ures2and3.Ascanbeseen,thedriftoccurredbetweenepoch120000and150000.

Drift without wake-sleep learning

9.298.8y

8.6grenE8.4 eerF8.2 ztloh8mleH7.87.67.47.2

02000040000

6000080000100000180000200000

Number of Epochs

Figure2:Simulationswithoutcontinuouslearn-ing.

Figure2plotstheHFEduringthesimulationrunwhentrainingwasstoppedoncetheHFEhadstabilisedonthestaticdata(epoch100000).Itclearlyshowsthedetrimentaleffectofasingledriftinginputontheperformanceofthenetwork.

Drift with wake-sleep learning

98.8

8.6

y

gr8.4

enE e8.2

erF ztl8

ohmleH7.8

7.6

7.4

7.2

020000400006000080000100000180000200000

Number of Epochs

Figure3:Simulationswithcontinuouslearning.Figure3isidenticalexceptthattrainingoccursthroughoutthesimulation.Heretheonsetofdriftisclear,butthewake-sleepalgorithmquicklybe-ginstoaccountforitandtocorrecttheinternalmodeltomatchthedata,keepingtheHFElowandeventuallyreturningittoitsoriginalvalue.

Probabilistic computing forms a relatively new computational style, of significant practical interest because stochastic behaviour is common and must be taken into accountin in biological and other real-world processes. We examine a particular stochastic A

2..3.

De cienciesoftheBinaryHelmholtzMachine

Althoughtheaboveresultsareencouraging,thedatasetisconstructedtobesuitableforabinary-valuednetwork.Thedistributionofeachunitisindependentofanyotherandtheinputdataisinherentlybinary.Althoughitmaybepossibletohaveanarrayofbinaryvaluedsensors,eachactingeffectivelyasfeaturedetectors,indicatingwhetheravalueisaboveorbelowaparticularthreshold,orindicatingthepresenceorabsenceofaparticularsubstance,thisislikelytobeun-common.HencesomethoughthasbeengiventoexpandingtherepresentationalpoweroftheHM,andinparticularofbeingabletousecontinuousdata,ascommonlyavailablefromasensor.2..4.

ImprovingtheRepresentationswithaDiscreteHelmholtzMachine

Theobviousenhancementtorepresentationalpoweristouseunitswhichhavealargernum-berofstates,eitherdiscreteorcontinuous.Inthissectionsomeworkonusingadiscretenumberofstatesforeachunitispresented.TheHMcanbegeneralisedquiteeasilytothiscaseandtheequa-tionsfortheoperationandlearningofitareverysimilartothebinaryvaluedHelmholtzMachine.Howeversomenewproblemsalsoappear.Wedenotethenumberofstatesforunitsaswithindexwithandindexthenumber.Thenofeachstatesunitforunitsasnowhasabiasforeachstateandanumberofweightsequalto,denotedfortherecognitionweightsandforthegener-ativeweights(seeFigure1(b)).Theprobabilitydistributionoverstatesforaunitisnowgivenbythegeneralisedlogisticfunction,orSoftmaxfunction:

(1)

(2)

where:istheinputactivationforstate;isthebiasweightforunitandstate;

istheweightfromunitwhenitisinstatetounit …… 此处隐藏：11234字，全部文档内容请下载后查看。喜欢就下载吧 ……