Hadoop and Python Streaming

May 21, 2010

I've been starting to write some hadoop and python streaming jobs and there isn't all that much documentation regarding it out there. Things like, how do I pass environment variables, how do I pass along modules that my scripts might need, etc...

here's a couple of quick tips... to pass environment variables to your tasknodes use this command line param when launching a hadoop job:
  1.  
  2. /Users/Hadoop/hadoop/bin/hadoop jar /Users/Hadoop/hadoop/contrib/streaming/hadoop-0.20.2-streaming.jar \
  3. -mapper /Users/hadoop/code/traffic/mapper.py \
  4. -reducer /Users/hadoop/code/traffic/reducer.py \
  5. -input insights-input-small/* \
  6. -output insights-output-traffic \
  7. -cmdenv PYTHONPATH=$PYTHONPATH:/Users/jim/Code \
  8. -cmdenv MYAPP__PATH=/Users/jim/Code \
  9. -cmdenv MYAPP_ENVIRONMENT=development


if you want to distribute your modules to the tasknodes instead of having them installed on the target task nodes then you can zip up your module file, rename it to mymodule.mod and use this command line param

-file /Users/jim/Code/mymodule.mod

then in your script you can unzip it and import it as usual

  1.  
  2. import zipimport
  3. importer = zipimport.zipimporter('mymodule.mod')
  4. insights = importer.load_module('mymodule')


hope that helps someone :)



Comments

RSS feed for comments on this post.

  1. virbram five fingers says:
    June 4, 2010 @ 23:03 — Reply

    Comment pending moderation

  2. tiffany jewellery says:
    June 7, 2010 @ 18:27 — Reply

    Comment pending moderation

  3. Womens Perfume says:
    June 8, 2010 @ 17:33 — Reply

    Comment pending moderation

  4. LOUIS VUIttON Outlet says:
    June 9, 2010 @ 22:54 — Reply

    Comment pending moderation

  5. fake designer handbags says:
    June 15, 2010 @ 22:16 — Reply

    Comment pending moderation

  6. Louis Vuitton handbags says:
    June 15, 2010 @ 23:58 — Reply

    Comment pending moderation

  7. christian louboutin in uk says:
    June 16, 2010 @ 07:31 — Reply

    Comment pending moderation

  8. christian louboutin shoes says:
    June 16, 2010 @ 07:45 — Reply

    Comment pending moderation

  9. christian louboutin shoes says:
    June 16, 2010 @ 07:45 — Reply

    Comment pending moderation

  10. referencement chambery says:
    June 17, 2010 @ 16:44 — Reply

    Comment pending moderation

  11. evening dresses says:
    June 18, 2010 @ 08:22 — Reply

    Comment pending moderation

  12. Microsoft Office 2007 says:
    June 19, 2010 @ 02:20 — Reply

    Comment pending moderation

  13. replica watches says:
    June 21, 2010 @ 01:43 — Reply

    Comment pending moderation

  14. birkenstock says:
    June 22, 2010 @ 20:09 — Reply

    Comment pending moderation

  15. gocasino.com says:
    June 24, 2010 @ 00:47 — Reply

    Comment pending moderation

  16. replica watches says:
    June 25, 2010 @ 01:08 — Reply

    Comment pending moderation

Leave a Comment

Line and paragraph breaks automatic, HTML allowed: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <code> <em> <i> <strike> <strong>

Comments disabled due to spammers being losers that lead sad lives.