华盛顿大学公开课 Introduction to Data Science 037_pig_evalua

ExampleA= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’ OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; LOAD

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleA= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’; OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; LOAD

GROUP

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleLOAD A= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’; OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; GROUP

FOREACH

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleLOAD A= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’; OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; GROUP

FOREACH FILTER

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleLOAD A= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’; OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; GROUP

FOREACH FILTER STORE

Algebraic Optimization!5/18/2013 Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo 5

ExampleLOAD A= LOAD‘traffic.dat’ AS (ip, time, url); B= GROUP A BY ip; C= FOREACH B GENERATE group AS ip, COUNT(A); D= FILTER C BY ip IS‘192.168.0.1’; OR ip IS‘192.168.0.0’; STORE D INTO‘local_traffic.dat’; FILTER

GROUP FOREACH STORE

Lazy Evaluation: No work is done until STORE

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleLOAD Create a MR job for each COGROUP

FILTER Map Reduce

GROUP FOREACH STORE

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

ExampleLOAD 1) Create a MR job for each COGROUP

FILTER Map Reduce

GROUP FOREACH STORE

2) Add other commands where possible

Certain commands require their own MR job (e.g., ORDER)

5/18/2013

Howe, UW Adapted from slides by Oliver Bill Kennedy, U of Buffalo

Review NoSQL–“NoSchema”,“NoTransactions”,“NoLanguage”– A“reboot” of data systems focusing on just high-throughput reads and writes– But: A clear trend towards re-introducing schemas, languages, transactions at full scale Google’s Spanner system, for example

Pig– An RA-like language layer on Hadoop– But not a pure relational data model–“Schema-on-Read” rather than“Schema-on-write”

5/18/2013

Bill Howe, UW