Jim Plush's Programming Paradise

I got this instant msg from a buddy today at Panasonic:

MyBud
1:07
today i got spanked for putting out issues of rolling stone that i was done with; apparently some female around here didn't appreciate the provocative pose of shakira on a previous cover
1:07 went to HR to complain


*sigh*

here is a sample job script I got running to test out some hadoop mapreduce jobs for our new cluster. You can put this in the same directory with the map/reducer files. the -file parameter will package up those files and send them to the tasknodes in the cluster so you don't have to install them yourself.

I've been starting to write some hadoop and python streaming jobs and there isn't all that much documentation regarding it out there. Things like, how do I pass environment variables, how do I pass along modules that my scripts might need, etc...

here's a couple of quick tips... to pass environment variables to your tasknodes use this command line param when launching a hadoop job:

 
/Users/Hadoop/hadoop/bin/hadoop jar /Users/Hadoop/hadoop/contrib/streaming/hadoop-0.20.2-streaming.jar \
-mapper /Users/hadoop/code/traffic/mapper.py \
-reducer /Users/hadoop/code/traffic/reducer.py \
-input insights-input-small/* \
-output insights-output-traffic \
-cmdenv PYTHONPATH=$PYTHONPATH:/Users/jim/Code \
-cmdenv MYAPP__PATH=/Users/jim/Code \
-cmdenv MYAPP_ENVIRONMENT=development
 

if you want to distribute your modules to the tasknodes instead of having them installed on the target task nodes then you can zip up your module file, rename it to mymodule.mod and use this command line param

-file /Users/jim/Code/mymodule.mod

then in your script you can unzip it and import it as usual

 
 import zipimport
 importer = zipimport.zipimporter('mymodule.mod')
 insights = importer.load_module('mymodule')
 

hope that helps someone :)

Exciting times at the new job. We're releasing a handful of super secret invites to the new site. Crazy days to come. Wish I could say more at this time.

After 5+ years at Panasonic I've decided to accept a new position with Blue Rover Labs. It will be an exciting change back to start up life and I'm looking forward to getting started. I bought a Civic Hybrid with carpool stickers this morning so I'm all set for the commute :)

It was a pleasure to work with everyone at Panasonic. They really do take care of their employees and that really made the decision difficult.

We'll see what happens in the future!

So a few months ago I took the leap and spent some time getting down and dirty with Zend_Form. The elements, decorators, display sub groups, validations, filtering, etc...
In our new project we are now creating a single form class and are able to use that for our edit form, creation form and also a method for "read only" that will just display the contents properly all using the same form code. We have custom validators plugging into the elements. It's just a beautiful thing. Once you get the hang of it whipping out quality forms is a snap.

Nice job Framework devs

I'm currently set to speak at ZendCon 2009 on the topic of "How to run an enterprise PHP Shop" (http://zendcon.com/)

I'd love to see what people may be most interested in talking about. I've be a lead developer at McAfee and Panasonic, two large, very different corporations.

Here's what my blurb on the Zend site says:

"How to run an enterprise PHP Shop

Speaker: Jim Plush

In this session we'll take a look at how to successfully manage and lead PHP teams. We'll cover finding and interviewing PHP developers, the processes involved with producing products (build, qa, release, continuous integration, and development processes), building the team, team standards (coding, documentation, unit testing, reviews), providing ongoing education, and preventing developer burn out. We will also touch on the balacing act between process and productivity.

22.10.2009 | 09:15 bis 10:15 Uhr "


I'm currently working on the outline for the talk. It's only an hour and I have a lot to cover so if there's anything you want me to focus on just drop me a comment. Here is my super rough outline so far:

How to find, interview and hire PHP developers
How to manage teams
Coding Standards
Testing standards
Build Processes
Release Processes
Bug reporting and mitigation processes (bug scrub meetings with steering team)
Requirements gathering
Deploying Code
Training the team
Unit Testing
Continuous Integration Servers
System Testing
Weekly team meetings
Pre-release Bug Parties, prizes for most show stoppers, most bugs found
Tools we use at Panasonic (svn, why not to use telelogic change, phing, phpunit, zend framework, twiki wiki)
Team Building
Importance of certifications
Importance of the balance between process and performance
Wikis for collaboration
Why we chose Zend Framework

I've gone ahead and added the createuserAction to my indexController to illustrate how we could reuse all the code previously written to validate and save creating a new user. If you want to view the change log to see the code I added to support this, you can view it here: http://code.google.com/p/plushcode/source/detail?r=23


Here's what our createuserAction code looks like now

 
 public function createuserAction()
    {
    	// we're going to be creating a user
    	$userRepo = new user_UserRepository();
    	$user = $userRepo->getNewUser();
    	
    	// pass the form the validator it needs to work with
    	$form = new forms_CreateUserForm(array('Validator' => new user_validation_Default($user)));
    	
 
		// find a user with id 1 in the DB
    	if ($this->getRequest()->isPost()) {
	            if($form->isValid($_POST)) {
	   
	    		$values = $form->getValues();
 
		        // let's update the user's email address to what was posted from the form
		        $user->setEmailAddress($values['email']);
		        
		        // update the profile's nickname
		        $user->Profile->setNickname($values['nickname']);
        
		        
		    	// check to make sure the whole user object is valid with these new changes
		        if(!$user->isValid()) {
		        	$this->view->errors = $user->getMessages();
		        } else {
		        	$userRepo->save($user);
		        	$this->view->message = "USER CREATED!";
		        }
	    	}
    	
        }
    	
    	$this->view->form = $form;
    	
    }
 
 

I've gone ahead and added validation to our test project: http://code.google.com/p/plushcode/source/browse/#svn/trunk/ZendModel/application/default/models
If you take a look at the indexController you'll now see I've added an updateAction that uses a Zend Form component.

The goal of the validation was to provide context sensitive validation on a model. A model by itself might not know if it's valid or not so you'll want to apply different contexts depending on what the client operation is. Saving an order might have a different validation need than shipping an order. Instead of hacking a bunch of if/else statements you can simple say

 
$order->setValidator(new order_validation_NewOrder());
 
// OR
 
$order->setValidator(new order_validation_ShipOrder());
 
if(!$order->isValid()) {
    $messages = $order->getMessages();
}
 

and $order->isValid() will operate on that context object. The form object get's pass a validator so it can also operate on the same validator chains as the model so you don't have to duplicate that logic.

Anyway, I think it can be improved so take a look!
If you've downloaded and installed the code you'll be able to find the example on your local machine: http://youraddress/index/update

As a follow up to my previous post I'm going to post some live examples of model layer with Zend framework based on Domain Driven Design concepts. I've spent the past few days studying up on domain driven design as well as a lot of Martin Fowler's work in relation to domain modeling and I think I have at least some code to start with that could get some conversations rolling. The issue is Zend Framework doesn't have a formal model layer and it shouldn't. Models are very much domain specific to what you're working on. As I've mentioned in my previous post sometimes you need complex business logic that will need to rapidly change and adapt to growing business needs. Active record can only get you so far when complexity expands.

I've posted a working Zend Framework project that has a basic working set of models so far. The current concept has a User who in turn owns a profile object. That profile object holds a nickname attribute that we can play around with. I have two tables users and profiles that are joined with a foreign key (user_id). Normally when you want profile information you'd just do a join and work with the result. What if that profile becomes context sensitive? What if a user can have a different profile based on the location he's in? Maybe he's a CIA agent and if he's overseas he needs a different profile to be covert. You'd now have to start adding conditions to your logic to account for this. Having the user own a profile object that can be injected at runtime allows for these contexts to be more manageable.

What if your user is going to be used in a workflow engine, a piece of marketing business logic that operates on users, it will be nice to be able to pass around a complete User object that has well documented properties and APIs instead of forcing each business logic service to know the details of your database.

Here is the google code repository you can checkout and play around with: Google Code Repository for Zend Model project
It's all working, just create a database and dump in the zendmodel_DB.sql file, change your config.ini file settings and you're all set assuming you handle your apache settings



Take a look at the default/controllers/IndexController.php file for how the client code will be used.

 
	// instantiate the user repository who will be in charge of building a proper user
        $userRepo = new user_UserRepository();
        
        // find a user with id 1 in the DB and grab their current nickname from the profile
        $user = $userRepo->find(1);
        $this->view->nickname = $user->Profile->getNickname();
        
		// set their nickname in their profile to something new
        $user->Profile->setNickname('tommy tom');
        
        // find the user again, it should pull the same object reference and not create a brand new object (Identity Map)
        $user = $userRepo->find(1);
        $this->view->nickname2 = $user->Profile->getNickname(); 
        
      
        
        // a fictional web service to check to see how popular your new nickname is
        $profileService = new services_ProfileService();
        $popularity = $profileService->howCommonIsMyNickname($user);
        $this->view->popularity = $popularity;
        
        
        // let's update the user's email address
        $user->setEmailAddress('newdude@yahoo.com');
        
        
        // you can manually commit the save to the database like this or in the destruct method of ObjectManager 
        // it will do it automatically for you, this will save all the changes we've made to all our objects
        // using the unit of work pattern in ObjectManager, the profile and the user will get updated
        ObjectManager::performOperations();
 

We have a User model and we have some supporting files around it.
in models/user/ you'll see some files:
/User.php
/UserMapper.php
/UserRepository.php

Each one plays a role within the domain User.php is the model itself that models a user in our system. It contains no SQL or knowledge of the persistence layer and can be passed around to any app on ths system. UserMapper.php is our basic DB Mapper that is in charge of doing the dirty work of putting things into our database. UserRepository is the default User Factory for creating a valid User object along with it's dependencies. Because this could be context sensitive we can contain that logic in a central place in there or add another Repository.

if you look in models/infrastructure/ObjectManager.php you'll see this file is a combination identity map and unit of work. It's job is to make sure if we have already instantiated a User object for example that we get back the same exact one and don't try to recreate it. It's basically an object cache.

Anyway, take a look and let's talk about what's right and what's wrong and how we can improve the complex domain model with Zend Framework. I'd like to start building on this example with everyday issues like logging, authentication, transactions, validation, unit tests, etc...