hadoop 分布式 系统 存储 数据库 云计算 (7)

hadoop 分布式 系统 存储 数据库 云计算

Whirlwind tour of Hadoop

Inspired by Google's GFS

Clusters from 1-10000 systemsBatch Processing High Throughput Partition-able problems Fault Tolerance in storage Fault Tolerance in processing

hadoop 分布式 系统 存储 数据库 云计算

HDFS

Distributed File System (HDFS)

Centralized “INODE” store NameNodeDataNodes provide storage 1-10000 Replication set Per File Block Size normally large 128 MB Optimized for throughput

hadoop 分布式 系统 存储 数据库 云计算

Map Reduce

Jobs are typically long running

Move processing not dataAlgorithm uses map() and optionally reduce() Source Input is split Splits processed simultaneously Failed Tasks are retried

hadoop 分布式 系统 存储 数据库 云计算

Things you do not have

Locking

BlockingMutexes wait/notify POSIX file semantics

hadoop 分布式 系统 存储 数据库 云计算

Problem: Large Scale Log Processing

Challenges

Large volumes of DataStore it indefinitely Process large data sets on multiple systems Data and Indexes for short periods (one week) do NOT fit in main memory

hadoop 分布式 系统 存储 数据库 云计算

Log Format

hadoop 分布式 系统 存储 数据库 云计算

Getting Files into HDFS

hadoop 分布式 系统 存储 数据库 云计算

Hadoop Shell

Hadoop has a shell ls, put, get, cat..

hadoop 分布式 系统 存储 数据库 云计算

Sample Program Group and Count

Source data looks like

jan 10 2009:.........:200:/index.htm jan 10 2009:.........:200:/index.htm jan 10 2009:.........:200:/igloo.htm jan 10 2009:.........:200:/ed.htm

hadoop 分布式 系统 存储 数据库 云计算

In case your a Math 'type'

(input) <k1, v1> → map -> <k2, v2> -> combine -> <k2, v2> -> reduce -> <k3, v3> (output)

Map(k1,v1) -> list(k2,v2) Reduce(k2, list (v2)) -> list(v3)

hadoop 分布式 系统 存储 数据库 云计算

Calculate Splits

Input does not have to be a file to be “splittable”

hadoop 分布式 系统 存储 数据库 云计算

Splits → Mappers

Mappers run the map() process on each split

hadoop 分布式 系统 存储 数据库 云计算

Mapper runs map function

For each row/line (in this case)

hadoop 分布式 系统 存储 数据库 云计算

Shuffle sort, NOT the hottest new dance move

Data Shuffled to reducer on key.hash()

Data sorted by key per reducer.

hadoop 分布式 系统 存储 数据库 云计算

Reduce

One Reducer gets data that looks like

”index.htm”,<1,1> “ed,htm”,<1> //igloo.htm went to another reducer

hadoop 分布式 系统 存储 数据库 云计算

Reduce Phase