Example of Hadoop Python Streaming job script
May 21, 2010
here is a sample job script I got running to test out some hadoop mapreduce jobs for our new cluster. You can put this in the same directory with the map/reducer files. the -file parameter will package up those files and send them to the tasknodes in the cluster so you don't have to install them yourself.
#!/bin/sh
# remove local output data
rm -rf /data/out/insights-output-traffic
# remove dfs output data
/data/hadoop/bin/hadoop dfs -rmr output-traffic*
# start hadoop job
/data/hadoop/bin/hadoop jar /data/hadoop/contrib/streaming/hadoop-0.20.1-streaming.jar \
-jobconf mapred.reduce.tasks=9 \
-mapper mapper.py \
-reducer reducer.py \
-file mapper.py \
-file reducer.py \
-input daytest/smallday/* \
-output output-traffic
# move output to local dir
/data/hadoop/bin/hadoop dfs -copyToLocal output-traffic /data/out
#!/bin/sh
# remove local output data
rm -rf /data/out/insights-output-traffic
# remove dfs output data
/data/hadoop/bin/hadoop dfs -rmr output-traffic*
# start hadoop job
/data/hadoop/bin/hadoop jar /data/hadoop/contrib/streaming/hadoop-0.20.1-streaming.jar \
-jobconf mapred.reduce.tasks=9 \
-mapper mapper.py \
-reducer reducer.py \
-file mapper.py \
-file reducer.py \
-input daytest/smallday/* \
-output output-traffic
# move output to local dir
/data/hadoop/bin/hadoop dfs -copyToLocal output-traffic /data/out
Biber hapi says:
May 29, 2010 @ 00:51 — Reply
Comment pending moderation
links of london says:
June 7, 2010 @ 18:18 — Reply
Comment pending moderation
Anonymous says:
June 8, 2010 @ 17:20 — Reply
Comment pending moderation
cheap duffel bags says:
June 10, 2010 @ 19:05 — Reply
Comment pending moderation
abaya says:
June 11, 2010 @ 10:42 — Reply
Comment pending moderation
handbags online says:
June 11, 2010 @ 18:10 — Reply
Comment pending moderation
billiards says:
June 11, 2010 @ 20:56 — Reply
Comment pending moderation
cheryl says:
June 15, 2010 @ 04:04 — Reply
Comment pending moderation
fake designer handbags says:
June 15, 2010 @ 20:54 — Reply
Comment pending moderation
Louis Vuitton handbags says:
June 15, 2010 @ 23:56 — Reply
Comment pending moderation
Sign Up for Gmail says:
June 16, 2010 @ 00:13 — Reply
Comment pending moderation
Louis Vuitton handbags says:
June 16, 2010 @ 01:07 — Reply
Comment pending moderation
christian louboutin says:
June 16, 2010 @ 05:13 — Reply
Comment pending moderation
christian louboutin says:
June 16, 2010 @ 05:24 — Reply
Comment pending moderation
sweetvo@yahoo.com.cn says:
June 16, 2010 @ 05:26 — Reply
Comment pending moderation
christian louboutin says:
June 16, 2010 @ 05:27 — Reply
Comment pending moderation
christian louboutin in uk says:
June 16, 2010 @ 07:02 — Reply
Comment pending moderation
who called me says:
June 16, 2010 @ 19:26 — Reply
Comment pending moderation
coach outlet says:
June 17, 2010 @ 02:56 — Reply
Comment pending moderation
abaya says:
June 17, 2010 @ 16:38 — Reply
Comment pending moderation
evening dresses says:
June 18, 2010 @ 08:20 — Reply
Comment pending moderation
mobile phones says:
June 18, 2010 @ 19:16 — Reply
Comment pending moderation
Microsoft Office 2007 says:
June 19, 2010 @ 02:18 — Reply
Comment pending moderation
Jared Doto says:
June 20, 2010 @ 05:45 — Reply
Comment pending moderation
hajj 2010 says:
June 20, 2010 @ 05:48 — Reply
Comment pending moderation
Replica Jewelry says:
June 21, 2010 @ 00:37 — Reply
Comment pending moderation
Sara says:
June 21, 2010 @ 23:58 — Reply
Comment pending moderation
Tall ugg shoes says:
June 22, 2010 @ 17:54 — Reply
Comment pending moderation
Ugg Boots UK says:
June 22, 2010 @ 17:56 — Reply
Comment pending moderation
uggs shoes says:
June 22, 2010 @ 17:57 — Reply
Comment pending moderation
ugg boots says:
June 22, 2010 @ 17:59 — Reply
Comment pending moderation
uggs boots says:
June 22, 2010 @ 18:00 — Reply
Comment pending moderation
vibram shoes says:
June 22, 2010 @ 18:03 — Reply
Comment pending moderation
Cheap EdHardy says:
June 22, 2010 @ 18:04 — Reply
Comment pending moderation
hollister says:
June 22, 2010 @ 18:06 — Reply
Comment pending moderation
birkenstock says:
June 22, 2010 @ 20:05 — Reply
Comment pending moderation
Anonymous says:
June 23, 2010 @ 15:51 — Reply
Comment pending moderation
Camping Accessories says:
June 23, 2010 @ 19:33 — Reply
Comment pending moderation
Finland says:
June 24, 2010 @ 12:43 — Reply
Comment pending moderation
replica watches says:
June 25, 2010 @ 01:07 — Reply
Comment pending moderation
cheap photo albums says:
June 27, 2010 @ 18:16 — Reply
Comment pending moderation