Bump dates in packaging files for release.

md5 allow copy if file is missing

CHANGELOG.md

@@ -1,20 +1,88 @@
 Ansible Changes By Release
 ==========================
 
-0.4 "Unchained" ------- in progress, ETA May 2012
+0.5 "Amsterdam" ------- July XX, 2012
 
-* custom facts provided by the setup module mean no dependency on Ruby, facter, or ohai
+* Service module gets more accurate service states when running with upstart
+* Jinja2 usage in playbooks (not templates), reinstated, supports %include directive
+* support for --connection ssh (supports Kerberos, bastion hosts, etc), requires ControlMaster
+* misc tracebacks replaced with error messages
+* various API/internals refactoring
+* vars can be built from other variables
+* support for exclusion of hosts/groups with "!groupname"
+* various changes to support md5 tool differences for FreeBSD nodes & OS X clients
+* "unparseable" command output shows in command output for easier debugging
+* mktemp is no longer required on remotes (not available on BSD)
+* support for older versions of python-apt in the apt module
+* a new "assemble" module, for constructing files from pieces of files (inspired by Puppet "fragments" idiom)
+* ability to override most default values with ANSIBLE_FOO environment variables
+* --module-path parameter can support multiple directories seperated with the OS path seperator
+* with_items can take a variable of type list
+* ansible_python_interpreter variable available for systems with more than one Python
+* BIOS and VMware "fact" upgrades
+* cowsay is used by ansible-playbook if installed to improve output legibility (try installing it)
+* authorized_key module
+* SELinux facts now sourced from the python selinux library
+* removed module debug option -D
+* added --verbose, which shows output from successful playbook operations
+* print the output of the raw command inside /usr/bin/ansible as with command/shell
+* basic setup module support for Solaris
+* ./library relative to the playbook is always in path so modules can be included in tarballs with playbooks
+
+0.4 "Unchained" ------- May 23, 2012
+
+Internals/Core
+* internal inventory API now more object oriented, parsers decoupled
+* async handling improvements
+* misc fixes for running ansible on OS X (overlord only)
 * sudo improvements, now works much more smoothly
-* some support for reading some default settings from the ssh config file (key pairs, ports)
-* OS X support in progress for ansible on the host and clients (modules still TBA)
+* sudo to a particular user with -U/--sudo-user, or using 'sudo_user: foo' in a playbook
+* --private-key CLI option to work with pem files
+
+Inventory
+* can use -i host1,host2,host3:port to specify hosts not in inventory (replaces --override-hosts)
+* ansible INI style format can do groups of groups [groupname:children] and group vars [groupname:vars]
+* groups and users module takes an optional system=yes|no on creation (default no)
 * list of hosts in playbooks can be expressed as a YAML list in addition to ; delimited
-* tweaks to SELinux implementation for file module
-* first_available_file feature, see selective_file_sources.yml in examples/playbooks for info
-* fixes for yum module corner cases on EL5
-* --extra-vars="a=2 b=3" etc, now available to inject parameters into playbooks from CLI
-* file module now correctly returns the mode in octal
-* modules can no longer include stderr output (paramiko limitation from sudo)
+
+Playbooks
+* variables can be replaced like ${foo.nested_hash_key.nested_subkey[array_index]}
+* unicode now ok in templates (assumes utf8)
+* able to pass host specifier or group name in to "hosts:" with --extra-vars
+* ansible-pull script and example playbook (extreme scaling, remediation)
+* inventory_hostname variable available that contains the value of the host as ansible knows it
+* variables in the 'all' section can be used to define other variables based on those values
 * 'group_names' is now a variable made available to templates
+* first_available_file feature, see selective_file_sources.yml in examples/playbooks for info
+* --extra-vars="a=2 b=3" etc, now available to inject parameters into playbooks from CLI
+
+Incompatible Changes
+* jinja2 is only usable in templates, not playbooks, use $foo instead
+* --override-hosts removed, can use -i with comma notation (-i "ahost,bhost")
+* modules can no longer include stderr output (paramiko limitation from sudo)
+
+Module Changes
+* tweaks to SELinux implementation for file module
+* fixes for yum module corner cases on EL5
+* file module now correctly returns the mode in octal
+* fix for symlink handling in the file module
+* service takes an enable=yes|no which works with chkconfig or updates-rc.d as appropriate
+* service module works better on Ubuntu
+* git module now does resets and such to work more smoothly on updates
+* modules all now log to syslog
+* enabled=yes|no on a service can be used to toggle chkconfig & updates-rc.d states
+* git module supports branch=
+* service fixes to better detect status using return codes of the service script
+* custom facts provided by the setup module mean no dependency on Ruby, facter, or ohai
+* service now has a state=reloaded
+* raw module for bootstrapping and talking to routers w/o Python, etc
+
+Misc Bugfixes
+* fixes for variable parsing in only_if lines
+* misc fixes to key=value parsing
+* variables with mixed case now legal
+* fix to internals of hacking/test-module development script
+
 
 0.3 "Baluchitherium" -- April 23, 2012
 

README.md

@@ -27,10 +27,9 @@ Get Involved
 Branch Info
 ===========
 
-   * The master branch corresponds to release 0.3 "Baluchitherium".
-   * Bug fixes WILL be periodically made to the master branch.
-   * The devel branch corresponds to release 0.4 "Unchained".
-   * Note: sudo support works much better in 0.4/devel, consider trying it.
+   * The master branch corresponds to release 0.4 "Unchained".
+   * Major bug fixes will be periodically made to the master branch.
+   * The devel branch corresponds to release 0.5 "Amsterdam".
    * See CHANGELOG.md for release notes to track each release.
 
 Patch Instructions

VERSION

@@ -1 +1 @@
-0.4
+0.5

bin/ansible

@@ -22,7 +22,7 @@
 import sys
 import getpass
 
-import ansible.runner
+from ansible.runner import Runner
 import ansible.constants as C
 from ansible import utils
 from ansible import errors
@@ -52,7 +52,6 @@ class Cli(object):
         parser.add_option('-m', '--module-name', dest='module_name',
             help="module name to execute (default=%s)" % C.DEFAULT_MODULE_NAME, 
             default=C.DEFAULT_MODULE_NAME)
-
         options, args = parser.parse_args()
         self.callbacks.options = options        
 
@@ -71,8 +70,8 @@ class Cli(object):
         inventory_manager = inventory.Inventory(options.inventory)
         hosts = inventory_manager.list_hosts(pattern)
         if len(hosts) == 0:
-             print >>sys.stderr, "No hosts matched"
-             sys.exit(1) 
+            print >>sys.stderr, "No hosts matched"
+            sys.exit(1)
         
         sshpass = None
         sudopass = None
@@ -80,25 +79,29 @@ class Cli(object):
             sshpass = getpass.getpass(prompt="SSH password: ")
         if options.ask_sudo_pass:
             sudopass = getpass.getpass(prompt="sudo password: ")
-
+            options.sudo = True
+        if options.sudo_user:
+            options.sudo = True
+        options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
         if options.tree:
             utils.prepare_writeable_dir(options.tree)
 
-        runner = ansible.runner.Runner(
+        runner = Runner(
             module_name=options.module_name, module_path=options.module_path,
             module_args=options.module_args,
             remote_user=options.remote_user, remote_pass=sshpass,
             inventory=inventory_manager, timeout=options.timeout, 
+            private_key_file=options.private_key_file,
             forks=options.forks, 
             pattern=pattern, 
             callbacks=self.callbacks, sudo=options.sudo, 
-            sudo_pass=sudopass,
-            transport=options.connection, debug=options.debug
+            sudo_pass=sudopass,sudo_user=options.sudo_user,
+            transport=options.connection, verbose=options.verbose
         )
 
         if options.seconds:
             print "background launch...\n\n"
-            results, poller = runner.runAsync(options.seconds)
+            results, poller = runner.run_async(options.seconds)
             results = self.poll_while_needed(poller, options)
         else:
             results = runner.run()

bin/ansible-playbook

@@ -33,8 +33,6 @@ def main(args):
     # create parser for CLI options
     usage = "%prog playbook.yml"
     parser = utils.base_parser(constants=C, usage=usage, connect_opts=True, runas_opts=True)
-    parser.add_option('-O', '--override-hosts', dest="override_hosts", default=None,
-        help="run playbook against these hosts regardless of inventory settings")
     parser.add_option('-e', '--extra-vars', dest="extra_vars", default=None,
         help="set additional key=value variables from the CLI")
 
@@ -50,25 +48,25 @@ def main(args):
         sshpass = getpass.getpass(prompt="SSH password: ")
     if options.ask_sudo_pass:
         sudopass = getpass.getpass(prompt="sudo password: ")
-    override_hosts = None
-    if options.override_hosts:
-        override_hosts = options.override_hosts.split(",")
+        options.sudo = True
+    if options.sudo_user:
+        options.sudo = True
+    options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
     extra_vars = utils.parse_kv(options.extra_vars)
 
     # run all playbooks specified on the command line
     for playbook in args:
 
         stats = callbacks.AggregateStats()
-        playbook_cb = callbacks.PlaybookCallbacks()
-        runner_cb = callbacks.PlaybookRunnerCallbacks(stats)
+        playbook_cb = callbacks.PlaybookCallbacks(verbose=options.verbose)
+        runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=options.verbose)
 
         pb = ansible.playbook.PlayBook(
             playbook=playbook,
             module_path=options.module_path,
             host_list=options.inventory, 
-            override_hosts=override_hosts,
             forks=options.forks, 
-            debug=options.debug, 
+            verbose=options.verbose, 
             remote_user=options.remote_user,
             remote_pass=sshpass, 
             callbacks=playbook_cb, 
@@ -77,14 +75,16 @@ def main(args):
             timeout=options.timeout, 
             transport=options.connection,
             sudo=options.sudo,
+            sudo_user=options.sudo_user,
             sudo_pass=sudopass,
-            extra_vars=extra_vars
+            extra_vars=extra_vars,
+            private_key_file=options.private_key_file            
         )
         try:
 
             pb.run()
             hosts = sorted(pb.stats.processed.keys())
-            print "\n\nPLAY RECAP **********************\n\n"
+            print callbacks.banner("PLAY RECAP")
             for h in hosts:
                 t = pb.stats.summarize(h)
                 print "%-30s : ok=%4s changed=%4s unreachable=%4s failed=%4s " % (h, 

bin/ansible-pull

@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+
+# ansible-pull is a script that runs ansible in local mode
+# after checking out a playbooks directory from git.  There is an
+# example playbook to bootstrap this script in the examples/ dir which
+# installs ansible and sets it up to run on cron.
+#
+# usage: 
+#   ansible-pull -d /var/ansible/local -U http://wherever/content.git -C production
+#
+# the git repo must contain a playbook named 'local.yml'
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import subprocess
+import sys
+from optparse import OptionParser
+
+DEFAULT_PLAYBOOK = 'local.yml'
+
+def _run(cmd):
+    cmd = subprocess.Popen(cmd, shell=True,
+                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    print out
+    if cmd.returncode != 0:
+        print err
+    return cmd.returncode
+
+def main(args):
+    """ Set up and run a local playbook """
+    usage = "%prog [options]"
+    parser = OptionParser()
+    parser.add_option('-d', '--directory', dest='dest', default=None,
+                      help='Directory to checkout git repository')
+    parser.add_option('-U', '--url', dest='url',
+                      default=None,
+                      help='URL of git repository')
+    parser.add_option('-C', '--checkout', dest='checkout',
+                      default="HEAD",
+                      help='Branch/Tag/Commit to checkout.  Defaults to HEAD.')
+    options, args = parser.parse_args(args)
+
+    git_opts = "repo=%s dest=%s version=%s" % (options.url, options.dest, options.checkout)
+    cmd = 'ansible all -c local -m git -a "%s"' % git_opts
+    print "cmd=%s" % cmd
+    rc = _run(cmd)
+    if rc != 0:
+        return rc
+
+    os.chdir(options.dest)
+    cmd = 'ansible-playbook -c local %s' % DEFAULT_PLAYBOOK
+    rc = _run(cmd)
+    return rc
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except KeyboardInterrupt, e:
+        print >>sys.stderr, "Exit on user request.\n"
+        sys.exit(1)

docs/man/man1/ansible-playbook.1

@@ -2,12 +2,12 @@
 .\"     Title: ansible-playbook
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: 04/26/2012
+.\"      Date: 06/19/2012
 .\"    Manual: System administration commands
-.\"    Source: Ansible 0.4
+.\"    Source: Ansible 0.5
 .\"  Language: English
 .\"
-.TH "ANSIBLE\-PLAYBOOK" "1" "04/26/2012" "Ansible 0\&.4" "System administration commands"
+.TH "ANSIBLE\-PLAYBOOK" "1" "06/19/2012" "Ansible 0\&.5" "System administration commands"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -34,9 +34,9 @@ The names of one or more YAML format files to run as ansible playbooks\&.
 .RE
 .SH "OPTIONS"
 .sp
-\fB\-D\fR, \fB\-\-debug\fR
+\fB\-v\fR, \fB\-\-verbose\fR
 .sp
-Print any messages the remote module sends to standard error to the console
+Verbose mode, more output from successful actions will be shown
 .PP
 \fB\-i\fR \fIPATH\fR, \fB\-\-inventory=\fR\fIPATH\fR
 .RS 4
@@ -82,13 +82,6 @@ Connection timeout to use when trying to talk to hosts, in
 \fISECONDS\fR\&.
 .RE
 .PP
-\fB\-O\fR \fIOVERRIDE_HOSTS\fR, \fB\-\-override\-hosts=\fR\fIOVERRIDE_HOSTS\fR
-.RS 4
-Ignore the inventory file and run the playbook against only these hosts\&. "hosts:" line in playbook should be set to
-\fIall\fR
-when using this option\&.
-.RE
-.PP
 \fB\-s\fR, \fB\-\-sudo\fR
 .RS 4
 Force all plays to use sudo, even if not marked as such\&.

docs/man/man1/ansible-playbook.1.asciidoc.in

@@ -34,9 +34,9 @@ The names of one or more YAML format files to run as ansible playbooks.
 OPTIONS
 -------
 
-*-D*, *--debug*          
+*-v*, *--verbose*          
 
-Debug mode
+Verbose mode, more output from successful actions will be shown
 
 *-i* 'PATH', *--inventory=*'PATH'::
 
@@ -71,12 +71,6 @@ Prompt for the password to use for playbook plays that request sudo access, if a
 Connection timeout to use when trying to talk to hosts, in 'SECONDS'.
 
 
-*-O* 'OVERRIDE_HOSTS', *--override-hosts=*'OVERRIDE_HOSTS'::
-
-Ignore the inventory file and run the playbook against only these hosts.  "hosts:" line
-in playbook should be set to 'all' when using this option.
-
-
 *-s*, *--sudo*::
 
 Force all plays to use sudo, even if not marked as such.

docs/man/man1/ansible.1

@@ -2,12 +2,12 @@
 .\"     Title: ansible
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: 04/26/2012
+.\"      Date: 06/19/2012
 .\"    Manual: System administration commands
-.\"    Source: Ansible 0.4
+.\"    Source: Ansible 0.5
 .\"  Language: English
 .\"
-.TH "ANSIBLE" "1" "04/26/2012" "Ansible 0\&.4" "System administration commands"
+.TH "ANSIBLE" "1" "06/19/2012" "Ansible 0\&.5" "System administration commands"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -70,11 +70,6 @@ The
 to pass to the module\&.
 .RE
 .PP
-\fB\-D\fR, \fB\-\-debug\fR
-.RS 4
-Print any messages the remote module sends to standard error to the console
-.RE
-.PP
 \fB\-k\fR, \fB\-\-ask\-pass\fR
 .RS 4
 Prompt for the SSH password instead of assuming key\-based authentication with ssh\-agent\&.
@@ -129,6 +124,13 @@ Use this remote
 instead of root\&.
 .RE
 .PP
+\fB\-U\fR \fISUDO_USERNAME\fR, \fB\-\-sudo\-user=\fR\fISUDO_USERNAME\fR
+.RS 4
+Sudo to
+\fISUDO_USERNAME\fR
+instead of root\&. Implies \-\-sudo\&.
+.RE
+.PP
 \fB\-c\fR \fICONNECTION\fR, \fB\-\-connection=\fR\fICONNECTION\fR
 .RS 4
 Connection type to use\&. Possible options are

docs/man/man1/ansible.1.asciidoc.in

@@ -60,10 +60,6 @@ The 'DIRECTORY' to load modules from. The default is '/usr/share/ansible'.
 
 The 'ARGUMENTS' to pass to the module.
 
-*-D*, *--debug*::
-
-Debug mode
-
 *-k*, *--ask-pass*::
 
 Prompt for the SSH password instead of assuming key-based authentication with ssh-agent.
@@ -101,6 +97,10 @@ Poll a background job every 'NUM' seconds. Requires *-B*.
 
 Use this remote 'USERNAME' instead of root.
 
+*-U* 'SUDO_USERNAME', *--sudo-user=*'SUDO_USERNAME'::
+
+Sudo to 'SUDO_USERNAME' instead of root.  Implies --sudo.
+
 *-c* 'CONNECTION', *--connection=*'CONNECTION'::
 
 Connection type to use.  Possible options are 'paramiko' (SSH) and 'local'.

examples/playbooks/ansible_pull.yml

@@ -0,0 +1,51 @@
+# ansibple-pull setup 
+#
+# on remote hosts, set up ansible to run periodically using the latest code
+# from a particular checkout, in pull based fashion, inverting Ansible's
+# usual push-based operating mode.  
+#
+# This particular pull based mode is ideal for:
+#
+# (A) massive scale out
+# (B) continual system remediation
+#
+# DO NOT RUN THIS AGAINST YOUR HOSTS WITHOUT CHANGING THE repo_url
+# TO SOMETHING YOU HAVE PERSONALLY VERIFIED
+#
+# 
+---
+
+- hosts: pull_mode_hosts
+  user: root
+
+  vars:
+
+    # schdule is fed directly to cron
+    schedule: '*/15 * * * *'
+
+    # User to run ansible-pull as from cron
+    cron_user: root
+
+    # Directory to where repository will be cloned
+    workdir: /var/lib/ansible/local
+
+    # Repository to check out -- YOU MUST CHANGE THIS
+    # repo must contain a local.yml file at top level
+    #repo_url: git://github.com/sfromm/ansible-playbooks.git
+    repo_url: SUPPLY_YOUR_OWN_GIT_URL_HERE
+
+  tasks:
+
+    - name: Install ansible
+      action: yum pkg=ansible state=installed
+
+    - name: Create local directory to work from
+      action: file path=$workdir state=directory owner=root group=root mode=0751
+
+    - name: Copy ansible inventory file to client
+      action: copy src=/etc/ansible/hosts dest=/etc/ansible/hosts
+              owner=root group=root mode=0644
+
+    - name: Create crontab entry to clone/pull git repository
+      action: template src=templates/ansible-pull.j2 dest=/etc/cron.d/ansible-pull owner=root group=root mode=0644
+

examples/playbooks/intermediate_example.yml

@@ -4,7 +4,7 @@
 # because of the comments it's less concise than it normally is.  But feel
 # free to comment your playbooks if you like.
 
-- hosts: dbservers
+- hosts: all
 
   # we can define variables the normal way...
 
@@ -34,7 +34,7 @@
 
   # we could also have done something like:
   # - include: wordpress.yml user=timmy
-  # and had access to the template variable {{ user }} in the 
+  # and had access to the template variable $user in the 
   # included file, if we wanted to.  Variables from vars
   # and vars_files are also available inside include files
 

examples/playbooks/selective_file_sources.yml

@@ -13,13 +13,13 @@
     first_available_file:
 
        # first see if we have a file for this specific host
-       - /srv/whatever/{{ ansible_hostname }}.conf
+       - /srv/whatever/${ansible_hostname}.conf
 
        # next try to load something like CentOS6.2.conf
-       - /srv/whatever/{{ ansible_distribution }}{{ ansible_distribution_version }}.conf
+       - /srv/whatever/${ansible_distribution}${ansible_distribution_version}.conf
 
        # next see if there's a CentOS.conf
-       - /srv/whatever/{{ ansible_distribution }}.conf
+       - /srv/whatever/${ansible_distribution}.conf
 
        # finally give up and just use something generic
        - /srv/whatever/default

examples/playbooks/templates/ansible-pull.j2

@@ -0,0 +1,2 @@
+# Cron job to git clone/pull a repo and then run locally
+{{ schedule }} {{ cron_user }} ansible-pull -d {{ workdir }} -U {{ repo_url }} >/var/log/ansible-pull.log 2>&1

hacking/test-module

@@ -49,7 +49,7 @@ if len(sys.argv) > 1:
 else:
     args = ""
 
-argspath = os.path.expanduser("/.ansible_test_module_arguments")
+argspath = os.path.expanduser("~/.ansible_test_module_arguments")
 argsfile = open(argspath, 'w')
 argsfile.write(args)
 argsfile.close()

lib/ansible/__init__.py

@@ -14,5 +14,5 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-__version__ = '0.4'
+__version__ = '0.5'
 __author__ = 'Michael DeHaan'

lib/ansible/callbacks.py

@@ -20,6 +20,8 @@
 import utils
 import sys
 import getpass
+import os
+import subprocess
 
 #######################################################
 
@@ -74,6 +76,18 @@ class AggregateStats(object):
 
 ########################################################################
 
+def banner(msg):
+    res = ""
+    if os.path.exists("/usr/bin/cowsay"):
+        cmd = subprocess.Popen("/usr/bin/cowsay -W 60 \"%s\"" % msg, 
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
+        (out, err) = cmd.communicate()
+        res = "%s\n" % out 
+    else:
+        res = "%s ********************* \n" % msg
+    return res
+  
+
 class DefaultRunnerCallbacks(object):
     ''' no-op callbacks for API usage of Runner() if no callbacks are specified '''
 
@@ -118,16 +132,14 @@ class CliRunnerCallbacks(DefaultRunnerCallbacks):
         self._async_notified = {}
 
     def on_failed(self, host, res):
-        invocation = res.get('invocation','')
-        if not invocation.startswith('async_status'):
-            self._on_any(host,res)
+        self._on_any(host,res)
 
     def on_ok(self, host, res):
-        invocation = res.get('invocation','')
-        if not invocation.startswith('async_status'):
-            self._on_any(host,res)
+        self._on_any(host,res)
  
     def on_unreachable(self, host, res):
+        if type(res) == dict:
+            res = res.get('msg','')
         print "%s | FAILED => %s" % (host, res)
         if self.options.tree:
             utils.write_tree_file(self.options.tree, host, utils.bigjson(dict(failed=True, msg=res)))
@@ -164,28 +176,23 @@ class CliRunnerCallbacks(DefaultRunnerCallbacks):
 class PlaybookRunnerCallbacks(DefaultRunnerCallbacks):
     ''' callbacks used for Runner() from /usr/bin/ansible-playbook '''
 
-    def __init__(self, stats):
+    def __init__(self, stats, verbose=False):
         self.stats = stats
         self._async_notified = {}
+        self.verbose = verbose
 
     def on_unreachable(self, host, msg):
         print "fatal: [%s] => %s" % (host, msg)
 
     def on_failed(self, host, results):
-        invocation = results.get('invocation',None)
-        if not invocation or invocation.startswith('setup ') or invocation.startswith('async_status '):
-            print "failed: [%s] => %s\n" % (host, utils.smjson(results))
-        else: 
-            print "failed: [%s] => %s => %s\n" % (host, invocation, utils.smjson(results))
+        print "failed: [%s] => %s\n" % (host, utils.smjson(results))
 
     def on_ok(self, host, host_result):
-        invocation = host_result.get('invocation','')
-        if invocation.startswith('async_status'):
-            pass
-        elif not invocation or invocation.startswith('setup '):
+        # show verbose output for non-setup module results if --verbose is used
+        if not self.verbose or host_result.get("verbose_override",None) is not None:
             print "ok: [%s]\n" % (host)
         else:
-            print "ok: [%s] => %s\n" % (host, invocation)
+            print "ok: [%s] => %s" % (host, utils.smjson(host_result))
 
     def on_error(self, host, err):
         print >>sys.stderr, "err: [%s] => %s\n" % (host, err)
@@ -214,8 +221,8 @@ class PlaybookRunnerCallbacks(DefaultRunnerCallbacks):
 class PlaybookCallbacks(object):
     ''' playbook.py callbacks used by /usr/bin/ansible-playbook '''
   
-    def __init__(self):
-        pass
+    def __init__(self, verbose=False):
+        self.verbose = verbose
 
     def on_start(self):
         print "\n"
@@ -224,7 +231,7 @@ class PlaybookCallbacks(object):
         pass
 
     def on_task_start(self, name, is_conditional):
-        print utils.task_start_msg(name, is_conditional)
+        print banner(utils.task_start_msg(name, is_conditional))
 
     def on_vars_prompt(self, varname, private=True):
         msg = 'input for %s: ' % varname
@@ -233,10 +240,10 @@ class PlaybookCallbacks(object):
         return raw_input(msg)
         
     def on_setup_primary(self):
-        print "SETUP PHASE ****************************\n"
+        print banner("SETUP PHASE")
     
     def on_setup_secondary(self):
-        print "\nVARIABLE IMPORT PHASE ******************\n"
+        print banner("VARIABLE IMPORT PHASE")
 
     def on_import_for_host(self, host, imported_file):
         print "%s: importing %s" % (host, imported_file)
@@ -245,4 +252,4 @@ class PlaybookCallbacks(object):
         print "%s: not importing file: %s" % (host, missing_file)
 
     def on_play_start(self, pattern):
-        print "PLAY [%s] ****************************\n" % pattern
+        print banner("PLAY [%s]" % pattern)

lib/ansible/constants.py

@@ -23,17 +23,21 @@ DEFAULT_HOST_LIST      = os.environ.get('ANSIBLE_HOSTS',
     '/etc/ansible/hosts')
 DEFAULT_MODULE_PATH    = os.environ.get('ANSIBLE_LIBRARY',
     '/usr/share/ansible')
+DEFAULT_REMOTE_TMP     = os.environ.get('ANSIBLE_REMOTE_TMP', 
+    '$HOME/.ansible/tmp')
 
 DEFAULT_MODULE_NAME    = 'command'
 DEFAULT_PATTERN        = '*'
-DEFAULT_FORKS          = 5
-DEFAULT_MODULE_ARGS    = ''
-DEFAULT_TIMEOUT        = 10
-DEFAULT_POLL_INTERVAL  = 15
-DEFAULT_REMOTE_USER    = 'root'
+DEFAULT_FORKS          = os.environ.get('ANSIBLE_FORKS',5)
+DEFAULT_MODULE_ARGS    = os.environ.get('ANSIBLE_MODULE_ARGS','')
+DEFAULT_TIMEOUT        = os.environ.get('ANSIBLE_TIMEOUT',10)
+DEFAULT_POLL_INTERVAL  = os.environ.get('ANSIBLE_POLL_INTERVAL',15)
+DEFAULT_REMOTE_USER    = os.environ.get('ANSIBLE_REMOTE_USER','root')
 DEFAULT_REMOTE_PASS    = None
+DEFAULT_PRIVATE_KEY_FILE    = os.environ.get('ANSIBLE_REMOTE_USER',None)
 DEFAULT_SUDO_PASS      = None
+DEFAULT_SUDO_USER      = os.environ.get('ANSIBLE_SUDO_USER','root')
 DEFAULT_REMOTE_PORT    = 22
-DEFAULT_TRANSPORT      = 'paramiko'
-DEFAULT_TRANSPORT_OPTS = ['local', 'paramiko']
+DEFAULT_TRANSPORT      = os.environ.get('ANSIBLE_TRANSPORT','paramiko')
+DEFAULT_TRANSPORT_OPTS = ['local', 'paramiko', 'ssh']
 

lib/ansible/inventory.py

@@ -1,327 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-import fnmatch
-import os
-import subprocess
-
-import constants as C
-from ansible import errors
-from ansible import utils
-
-class Inventory(object):
-    """ Host inventory for ansible.
-
-    The inventory is either a simple text file with systems and [groups] of 
-    systems, or a script that will be called with --list or --host.
-    """
-
-    def __init__(self, host_list=C.DEFAULT_HOST_LIST, global_host_vars={}):
-
-        self._restriction = None
-        self._variables = {}
-        self._global_host_vars = global_host_vars # lets us pass in a global var list
-                                                 # that each host gets
-                                                 # after we have set up the inventory
-                                                 # to get all group variables
-        
-        if type(host_list) == list:
-            self.host_list = host_list
-            self.groups = dict(ungrouped=host_list)
-            self._is_script = False
-            return
-
-        inventory_file = os.path.expanduser(host_list)
-        if not os.path.exists(inventory_file):
-            raise errors.AnsibleFileNotFound("inventory file not found: %s" % host_list)
-
-        self.inventory_file = os.path.abspath(inventory_file)
-
-        if os.access(self.inventory_file, os.X_OK):
-            self.host_list, self.groups = self._parse_from_script()
-            self._is_script = True
-        else:
-            self.host_list, self.groups = self._parse_from_file()
-            self._is_script = False
-
-    # *****************************************************
-    # Public API
-
-    def list_hosts(self, pattern="all"):
-        """ Return a list of hosts [matching the pattern] """
-        if self._restriction is None:
-            host_list = self.host_list
-        else:
-            host_list = [ h for h in self.host_list if h in self._restriction ]
-        return [ h for h in host_list if self._matches(h, pattern) ]
-
-    def restrict_to(self, restriction):
-        """ Restrict list operations to the hosts given in restriction """
-        if type(restriction)!=list:
-            restriction = [ restriction ]
-
-        self._restriction = restriction
-
-    def lift_restriction(self):
-        """ Do not restrict list operations """
-        self._restriction = None
-
-    def get_variables(self, host):
-        """ Return the variables associated with this host. """
-
-        variables = {
-            'inventory_hostname': host,
-        }
-
-        if host in self._variables:
-            variables.update(self._variables[host].copy())
-
-        if self._is_script:
-            variables.update(self._get_variables_from_script(host))
-
-        variables['group_names'] = []
-        for name,hosts in self.groups.iteritems():
-            if host in hosts:
-                variables['group_names'].append(name)
-
-        return variables
-
-    def get_global_vars(self):
-        return self._global_host_vars
-        
-    # *****************************************************
-
-    def _parse_from_file(self):
-        ''' parse a textual host file '''
-
-        results = []
-        groups = dict(ungrouped=[])
-        lines = file(self.inventory_file).read().split("\n")
-        if "---" in lines:
-            return self._parse_yaml()
-        group_name = 'ungrouped'
-        for item in lines:
-            item = item.lstrip().rstrip()
-            if item.startswith("#"):
-                # ignore commented out lines
-                pass
-            elif item.startswith("["):
-                # looks like a group
-                group_name = item.replace("[","").replace("]","").lstrip().rstrip()
-                groups[group_name] = []
-            elif item != "":
-                # looks like a regular host
-                if ":" in item:
-                    # a port was specified
-                    item, port = item.split(":")
-                    try:
-                        port = int(port)
-                    except ValueError:
-                        raise errors.AnsibleError("SSH port for %s in inventory (%s) should be numerical."%(item, port))
-                    self._set_variable(item, "ansible_ssh_port", port)
-                groups[group_name].append(item)
-                if not item in results:
-                    results.append(item)
-        return (results, groups)
-
-    # *****************************************************
-
-    def _parse_from_script(self):
-        ''' evaluate a script that returns list of hosts by groups '''
-
-        results = []
-        groups = dict(ungrouped=[])
-
-        cmd = [self.inventory_file, '--list']
-
-        try:
-            cmd = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
-            out, err = cmd.communicate()
-        except Exception, e:
-            raise errors.AnsibleError("Failure executing %s to produce host list:\n %s" % (self.inventory_file, str(e)))
-
-        rc = cmd.returncode
-        if rc:
-            raise errors.AnsibleError("%s: %s" % self.inventory_file, err)
-
-        try:
-            groups = utils.json_loads(out)
-        except:
-            raise errors.AnsibleError("invalid JSON response from script: %s" % self.inventory_file)
-
-        for (groupname, hostlist) in groups.iteritems():
-            for host in hostlist:
-                if host not in results:
-                    results.append(host)
-        return (results, groups)
-
-    # *****************************************************
-
-    def _parse_yaml(self):
-        """ Load the inventory from a yaml file.
-
-        returns hosts and groups"""
-        data = utils.parse_yaml_from_file(self.inventory_file)
-
-        if type(data) != list:
-            raise errors.AnsibleError("YAML inventory should be a list.")
-
-        hosts = []
-        groups = {}
-        ungrouped = []
-
-        
-        # go through once and grab up the global_host_vars from the 'all' group
-        for item in data:
-            if type(item) == dict:
-                if "group" in item and 'all' == item["group"]:
-                    if "vars" in item:
-                        variables = item['vars']
-                        if type(variables) == list:
-                            for variable in variables:
-                                if len(variable) != 1:
-                                    raise errors.AnsibleError("Only one item expected in %s"%(variable))
-                                k, v = variable.items()[0]
-                                self._global_host_vars[k] = utils.template(v, self._global_host_vars, {})
-                        elif type(variables) == dict:
-                            self._global_host_vars.update(variables)
-
-        for item in data:
-            if type(item) == dict:
-                if "group" in item:
-                    group_name = item["group"]
-
-                    group_vars = []
-                    if "vars" in item:
-                        group_vars.extend(item["vars"])
-
-                    group_hosts = []
-                    if "hosts" in item:
-                        for host in item["hosts"]:
-                            host_name = self._parse_yaml_host(host, group_vars)
-                            group_hosts.append(host_name)
-
-                    groups[group_name] = group_hosts
-                    hosts.extend(group_hosts)
-
-                elif "host" in item:
-                    host_name = self._parse_yaml_host(item)
-                    hosts.append(host_name)
-                    ungrouped.append(host_name)
-            else:
-                host_name = self._parse_yaml_host(item)
-                hosts.append(host_name)
-                ungrouped.append(host_name)
-
-        # filter duplicate hosts
-        output_hosts = []
-        for host in hosts:
-            if host not in output_hosts:
-                output_hosts.append(host)
-
-        if len(ungrouped) > 0 :
-            # hosts can be defined top-level, but also in a group
-            really_ungrouped = []
-            for host in ungrouped:
-                already_grouped = False
-                for name, group_hosts in groups.items():
-                    if host in group_hosts:
-                        already_grouped = True
-                if not already_grouped:
-                    really_ungrouped.append(host)
-            groups["ungrouped"] = really_ungrouped
-
-        return output_hosts, groups
-
-    def _parse_yaml_host(self, item, variables=[]):
-        def set_variables(host, variables):
-            if type(variables) == list:
-                for variable in variables:
-                    if len(variable) != 1:
-                        raise errors.AnsibleError("Only one item expected in %s"%(variable))
-                    k, v = variable.items()[0]
-                    self._set_variable(host, k, v)
-            elif type(variables) == dict:
-                for k, v in variables.iteritems():
-                    self._set_variable(host, k, v)
-
-
-        if type(item) in [str, unicode]:
-            set_variables(item, variables)
-            return item
-        elif type(item) == dict:
-            if "host" in item:
-                host_name = item["host"]
-                set_variables(host_name, variables)
-
-                if "vars" in item:
-                    set_variables(host_name, item["vars"])
-
-                return host_name
-        else:
-            raise errors.AnsibleError("Unknown item in inventory: %s"%(item))
-
-
-    def _get_variables_from_script(self, host):
-        ''' support per system variabes from external variable scripts, see web docs '''
-
-        cmd = [self.inventory_file, '--host', host]
-
-        cmd = subprocess.Popen(cmd,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE,
-            shell=False
-        )
-        out, err = cmd.communicate()
-
-        variables = {}
-        try:
-            variables = utils.json_loads(out)
-        except:
-            raise errors.AnsibleError("%s returned invalid result when called with hostname %s" % (
-                self.inventory_file,
-                host
-            ))
-        return variables
-
-    def _set_variable(self, host, key, value):
-        if not host in self._variables:
-            self._variables[host] = {}
-        self._variables[host][key] = value
-
-    def _matches(self, host_name, pattern):
-        ''' returns if a hostname is matched by the pattern '''
-
-        # a pattern is in fnmatch format but more than one pattern
-        # can be strung together with semicolons. ex:
-        #   atlanta-web*.example.com;dc-web*.example.com
-
-        if host_name == '':
-            return False
-        pattern = pattern.replace(";",":")
-        subpatterns = pattern.split(":")
-        for subpattern in subpatterns:
-            if subpattern == 'all':
-                return True
-            if fnmatch.fnmatch(host_name, subpattern):
-                return True
-            elif subpattern in self.groups:
-                if host_name in self.groups[subpattern]:
-                    return True
-        return False

lib/ansible/inventory/__init__.py

@@ -0,0 +1,172 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import fnmatch
+import os
+
+import subprocess
+import ansible.constants as C
+from ansible.inventory.ini import InventoryParser
+from ansible.inventory.yaml import InventoryParserYaml
+from ansible.inventory.script import InventoryScript
+from ansible.inventory.group import Group
+from ansible.inventory.host import Host
+from ansible import errors
+from ansible import utils
+
+class Inventory(object):
+    """ 
+    Host inventory for ansible.
+    """
+
+    def __init__(self, host_list=C.DEFAULT_HOST_LIST):
+
+        # the host file file, or script path, or list of hosts
+        # if a list, inventory data will NOT be loaded
+        self.host_list = host_list
+
+        # the inventory object holds a list of groups
+        self.groups = []
+ 
+        # a list of host(names) to contain current inquiries to
+        self._restriction = None
+
+        # whether the inventory file is a script
+        self._is_script = False
+
+        if type(host_list) in [ str, unicode ]:
+            if host_list.find(",") != -1:
+               host_list = host_list.split(",")
+
+        if type(host_list) == list:
+            all = Group('all')
+            self.groups = [ all ]
+            for x in host_list:
+                if x.find(":") != -1:
+                    tokens = x.split(":",1)
+                    all.add_host(Host(tokens[0], tokens[1]))
+                else:
+                    all.add_host(Host(x))
+        elif os.access(host_list, os.X_OK):
+            self._is_script = True
+            self.parser = InventoryScript(filename=host_list)
+            self.groups = self.parser.groups.values()
+        else:
+            data = file(host_list).read()
+            if not data.startswith("---"):
+                self.parser = InventoryParser(filename=host_list)
+                self.groups = self.parser.groups.values()
+            else:         
+                self.parser = InventoryParserYaml(filename=host_list)
+                self.groups = self.parser.groups.values()
+      
+    def _match(self, str, pattern_str):
+        return fnmatch.fnmatch(str, pattern_str)
+
+    def get_hosts(self, pattern="all"):
+        """ Get all host objects matching the pattern """
+        hosts = {}
+        patterns = pattern.replace(";",":").split(":")
+
+        groups = self.get_groups()
+        for pat in patterns:
+            if pat.startswith("!"):
+                pat = pat[1:]
+                inverted = True
+            else:
+                inverted = False
+            for group in groups:
+                for host in group.get_hosts():
+                    if group.name == pat or pat == 'all' or self._match(host.name, pat):
+                        #must test explicitly for None because [] means no hosts allowed
+                        if self._restriction==None or host.name in self._restriction: 
+                            if inverted:
+                                if host.name in hosts:
+                                    del hosts[host.name]
+                            else:
+                                hosts[host.name] = host
+        return sorted(hosts.values(), key=lambda x: x.name)
+
+    def get_groups(self):
+        return self.groups
+
+    def get_host(self, hostname):
+        for group in self.groups:
+           for host in group.get_hosts():
+               if hostname == host.name:
+                    return host
+        return None
+
+    def get_group(self, groupname):
+        for group in self.groups:
+            if group.name == groupname:
+                return group
+        return None
+
+    def get_group_variables(self, groupname):
+        group = self.get_group(groupname)
+        if group is None:
+            raise Exception("group not found: %s" % groupname)
+        return group.get_variables()
+
+    def get_variables(self, hostname):
+
+        if self._is_script:
+            # TODO: move this to inventory_script.py 
+            host = self.get_host(hostname)
+            cmd = subprocess.Popen(
+                [self.host_list,"--host",hostname], 
+                stdout=subprocess.PIPE, 
+                stderr=subprocess.PIPE
+            )
+            (out, err) = cmd.communicate()
+            results = utils.parse_json(out)
+            results['inventory_hostname'] = hostname
+            groups = [ g.name for g in host.get_groups() if g.name != 'all' ]
+            results['group_names'] = sorted(groups)
+            return results
+
+        host = self.get_host(hostname)
+        if host is None:
+            raise Exception("host not found: %s" % hostname)
+        return host.get_variables()
+
+    def add_group(self, group):
+        self.groups.append(group)
+
+    def list_hosts(self, pattern="all"):
+        return [ h.name for h in self.get_hosts(pattern) ]
+
+    def list_groups(self):
+        return [ g.name for g in self.groups ] 
+
+    def get_restriction(self):
+        return self._restriction
+
+    def restrict_to(self, restriction, append_missing=False):
+        """ Restrict list operations to the hosts given in restriction """
+   
+	if type(restriction) != list:
+	    restriction = [ restriction ]
+	self._restriction = restriction
+
+    def lift_restriction(self):
+        """ Do not restrict list operations """
+
+        self._restriction = None

lib/ansible/inventory/group.py

@@ -0,0 +1,76 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+# from ansible import errors
+
+class Group(object):
+    """ 
+    Group of ansible hosts
+    """
+
+    def __init__(self, name=None):
+        self.name = name
+        self.hosts = []
+        self.vars = {}
+        self.child_groups = []
+        self.parent_groups = []
+        if self.name is None:
+           raise Exception("group name is required")
+
+    def add_child_group(self, group):
+        if self == group:
+            raise Exception("can't add group to itself")
+        self.child_groups.append(group)
+        group.parent_groups.append(self)
+
+    def add_host(self, host):
+        self.hosts.append(host)
+        host.add_group(self)
+
+    def set_variable(self, key, value):
+        self.vars[key] = value
+
+    def get_hosts(self):
+        hosts = []
+        for kid in self.child_groups:
+            hosts.extend(kid.get_hosts())
+        hosts.extend(self.hosts)
+        return hosts 
+
+    def get_variables(self):
+        vars = {}
+        # FIXME: verify this variable override order is what we want
+        for ancestor in self.get_ancestors():
+           vars.update(ancestor.get_variables())
+        vars.update(self.vars)
+        return vars
+
+    def _get_ancestors(self):
+        results = {}
+        for g in self.parent_groups:
+            results[g.name] = g
+            results.update(g._get_ancestors())
+        return results
+
+    def get_ancestors(self):
+        return self._get_ancestors().values()
+
+
+
+

lib/ansible/inventory/host.py

@@ -0,0 +1,62 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+from ansible import errors
+import ansible.constants as C
+
+class Host(object):
+    """ 
+    Group of ansible hosts
+    """
+
+    def __init__(self, name=None, port=None):
+        self.name = name
+        self.vars = {}
+        self.groups = []
+        if port and port != C.DEFAULT_REMOTE_PORT:
+            self.set_variable('ansible_ssh_port', int(port))
+
+        if self.name is None:
+            raise Exception("host name is required")
+
+    def add_group(self, group):
+        self.groups.append(group)
+
+    def set_variable(self, key, value):
+        self.vars[key]=value;
+
+    def get_groups(self):
+        groups = {}
+        for g in self.groups:
+            groups[g.name] = g
+            ancestors = g.get_ancestors()
+            for a in ancestors:
+                groups[a.name] = a                
+        return groups.values()
+
+    def get_variables(self):
+        results = {}
+        for group in self.groups:
+            results.update(group.get_variables())
+        results.update(self.vars)
+        results['inventory_hostname'] = self.name
+        groups = self.get_groups()
+        results['group_names'] = sorted([ g.name for g in groups if g.name != 'all'])
+        return results
+

lib/ansible/inventory/ini.py

@@ -0,0 +1,151 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import fnmatch
+import os
+import subprocess
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+
+class InventoryParser(object):
+    """ 
+    Host inventory for ansible.
+    """
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        fh = open(filename)
+        self.lines = fh.readlines()
+        self.groups = {}
+        self.hosts = {}
+        self._parse()
+ 
+    def _parse(self):
+
+        self._parse_base_groups()
+        self._parse_group_children()
+        self._parse_group_variables()
+        return self.groups
+
+    # [webservers]
+    # alpha
+    # beta:2345
+    # gamma sudo=True user=root
+    # delta asdf=jkl favcolor=red 
+
+    def _parse_base_groups(self):
+
+        ungrouped = Group(name='ungrouped')
+        all = Group(name='all')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        active_group_name = 'ungrouped'
+
+        for line in self.lines:
+            if line.startswith("["):
+                active_group_name = line.replace("[","").replace("]","").strip()
+                if line.find(":vars") != -1 or line.find(":children") != -1:
+                    active_group_name = None
+                else:
+                    new_group = self.groups[active_group_name] = Group(name=active_group_name)
+                    all.add_child_group(new_group)
+            elif line.startswith("#") or line == '':
+                pass
+            elif active_group_name:
+                tokens = line.split()
+                if len(tokens) == 0:
+                    continue
+                hostname = tokens[0]
+                port = C.DEFAULT_REMOTE_PORT
+                if hostname.find(":") != -1:
+                   tokens2  = hostname.split(":")
+                   hostname = tokens2[0]
+                   port     = tokens2[1]
+                host = None
+                if hostname in self.hosts:
+                    host = self.hosts[hostname]
+                else:
+                    host = Host(name=hostname, port=port)
+                    self.hosts[hostname] = host
+                if len(tokens) > 1:
+                   for t in tokens[1:]:
+                      (k,v) = t.split("=")
+                      host.set_variable(k,v)
+                self.groups[active_group_name].add_host(host)
+
+    # [southeast:children]
+    # atlanta
+    # raleigh
+
+    def _parse_group_children(self):
+        group = None
+
+        for line in self.lines:
+            line = line.strip()
+            if line is None or line == '':
+                continue
+            if line.startswith("[") and line.find(":children]") != -1:
+                line = line.replace("[","").replace(":children]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                    group = self.groups[line] = Group(name=line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif group:
+                kid_group = self.groups.get(line, None)
+                if kid_group is None:
+                    raise errors.AnsibleError("child group is not defined: (%s)" % line)
+                else:
+                    group.add_child_group(kid_group)
+
+
+    # [webservers:vars]
+    # http_port=1234
+    # maxRequestsPerChild=200
+
+    def _parse_group_variables(self):
+        group = None
+        for line in self.lines:
+            line = line.strip()
+            if line.startswith("[") and line.find(":vars]") != -1:
+                line = line.replace("[","").replace(":vars]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                   raise errors.AnsibleError("can't add vars to undefined group: %s" % line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif line == '':
+                pass
+            elif group:
+                if line.find("=") == -1:
+                    raise errors.AnsibleError("variables assigned to group must be in key=value form")
+                else:
+                    (k,v) = line.split("=")
+                    group.set_variable(k,v)
+
+

lib/ansible/inventory/script.py

@@ -0,0 +1,58 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import os
+import subprocess
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+
+class InventoryScript(object):
+    """ 
+    Host inventory parser for ansible using external inventory scripts.
+    """
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        cmd = [ filename, "--list" ]
+        sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (stdout, stderr) = sp.communicate()
+        self.data = stdout
+        self.groups = self._parse()
+
+    def _parse(self):
+        groups = {}
+        self.raw = utils.parse_json(self.data)
+        all=Group('all')
+        self.groups = dict(all=all)
+        group = None
+        for (group_name, hosts) in self.raw.items():
+            group = groups[group_name] = Group(group_name)
+            host = None
+            for hostname in hosts:
+                host = Host(hostname)
+                group.add_host(host)
+                # FIXME: hack shouldn't be needed
+                all.add_host(host)
+            all.add_child_group(group)
+        return groups  
+        
+  

lib/ansible/inventory/yaml.py

@@ -0,0 +1,117 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+
+class InventoryParserYaml(object):
+    """ 
+    Host inventory for ansible.
+    """
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        fh = open(filename)
+        data = fh.read()
+        fh.close()
+        self._hosts = {}
+        self._parse(data)
+
+    def _make_host(self, hostname):
+        if hostname in self._hosts:
+            return self._hosts[hostname]
+        else:
+            host = Host(hostname)
+            self._hosts[hostname] = host
+            return host 
+
+    # see file 'test/yaml_hosts' for syntax
+
+    def _parse(self, data):
+
+        all = Group('all')
+        ungrouped = Group('ungrouped')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        grouped_hosts = []
+
+        yaml = utils.parse_yaml(data)
+
+        # first add all groups
+        for item in yaml:
+            if type(item) == dict and 'group' in item:
+                group = Group(item['group'])
+
+                for subresult in item.get('hosts',[]):
+
+                    if type(subresult) in [ str, unicode ]:
+                        host = self._make_host(subresult)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+                    elif type(subresult) == dict:
+                        host = self._make_host(subresult['host'])
+                        vars = subresult.get('vars',{})
+                        if type(vars) == list:
+                            for subitem in vars:
+                               for (k,v) in subitem.items():
+                                   host.set_variable(k,v) 
+                        elif type(vars) == dict:
+                            for (k,v) in subresult.get('vars',{}).items():
+                                host.set_variable(k,v)
+                        else:
+                            raise errors.AnsibleError("unexpected type for variable")
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                vars = item.get('vars',{})
+                if type(vars) == dict:
+                    for (k,v) in item.get('vars',{}).items():
+                        group.set_variable(k,v) 
+                elif type(vars) == list:
+                    for subitem in vars:
+                        if type(subitem) != dict:
+                            raise errors.AnsibleError("expected a dictionary")
+                        for (k,v) in subitem.items():
+                            group.set_variable(k,v)
+
+                self.groups[group.name] = group
+                all.add_child_group(group)
+
+        # add host definitions
+        for item in yaml:
+            if type(item) in [ str, unicode ]:
+                host = self._make_host(item)
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+            elif type(item) == dict and 'host' in item:
+                host = self._make_host(item['host'])
+                vars = item.get('vars', {})
+                if type(vars)==list:
+                    varlist, vars = vars, {}
+                    for subitem in varlist:
+                        vars.update(subitem)
+                for (k,v) in vars.items():
+                   host.set_variable(k,v)
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)

lib/ansible/playbook.py

@@ -1,563 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-import ansible.inventory
-import ansible.runner
-import ansible.constants as C
-from ansible import utils
-from ansible import errors
-import os
-
-# used to transfer variables to Runner
-SETUP_CACHE={ }
-
-#############################################
-
-class PlayBook(object):
-
-    '''
-    runs an ansible playbook, given as a datastructure
-    or YAML filename.  a playbook is a deployment, config
-    management, or automation based set of commands to
-    run in series.
-
-    multiple plays/tasks do not execute simultaneously,
-    but tasks in each pattern do execute in parallel
-    (according to the number of forks requested) among
-    the hosts they address
-    '''
-
-    # *****************************************************
-
-    def __init__(self,
-        playbook         = None,
-        host_list        = C.DEFAULT_HOST_LIST,
-        module_path      = C.DEFAULT_MODULE_PATH,
-        forks            = C.DEFAULT_FORKS,
-        timeout          = C.DEFAULT_TIMEOUT,
-        remote_user      = C.DEFAULT_REMOTE_USER,
-        remote_pass      = C.DEFAULT_REMOTE_PASS,
-        sudo_pass        = C.DEFAULT_SUDO_PASS,
-        remote_port      = C.DEFAULT_REMOTE_PORT,
-        transport        = C.DEFAULT_TRANSPORT,
-        override_hosts   = None,
-        debug            = False,
-        callbacks        = None,
-        runner_callbacks = None,
-        stats            = None,
-        sudo             = False,
-        extra_vars       = None):
-
-        """
-        playbook:         path to a playbook file
-        host_list:        path to a file like /etc/ansible/hosts
-        module_path:      path to ansible modules, like /usr/share/ansible/
-        forks:            desired level of paralellism
-        timeout:          connection timeout
-        remote_user:      run as this user if not specified in a particular play
-        remote_pass:      use this remote password (for all plays) vs using SSH keys
-        sudo_pass:        if sudo==True, and a password is required, this is the sudo password
-        remote_port:      default remote port to use if not specified with the host or play
-        transport:        how to connect to hosts that don't specify a transport (local, paramiko, etc)
-        override_hosts:   skip the inventory file, just talk to these hosts
-        callbacks         output callbacks for the playbook
-        runner_callbacks: more callbacks, this time for the runner API
-        stats:            holds aggregrate data about events occuring to each host
-        sudo:             if not specified per play, requests all plays use sudo mode
-        """
-
-        if playbook is None or callbacks is None or runner_callbacks is None or stats is None:
-            raise Exception('missing required arguments')
-
-        if extra_vars is None:
-            extra_vars = {}
-       
-        self.module_path      = module_path
-        self.forks            = forks
-        self.timeout          = timeout
-        self.remote_user      = remote_user
-        self.remote_pass      = remote_pass
-        self.remote_port      = remote_port
-        self.transport        = transport
-        self.debug            = debug
-        self.callbacks        = callbacks
-        self.runner_callbacks = runner_callbacks
-        self.override_hosts   = override_hosts
-        self.stats            = stats
-        self.sudo             = sudo
-        self.sudo_pass        = sudo_pass
-        self.extra_vars       = extra_vars
-        self.global_vars      = {}
-
-        if override_hosts is not None:
-            if type(override_hosts) != list:
-                raise errors.AnsibleError("override hosts must be a list")
-            self.global_vars.update(ansible.inventory.Inventory(host_list).get_global_vars())
-            self.inventory = ansible.inventory.Inventory(override_hosts)
-
-        else:
-            self.inventory = ansible.inventory.Inventory(host_list)
-            self.global_vars.update(ansible.inventory.Inventory(host_list).get_global_vars())
-
-        self.basedir          = os.path.dirname(playbook)
-        self.playbook         = self._parse_playbook(playbook)
-
-    # *****************************************************
-
-    def _get_vars(self, play, dirname):
-        ''' load the vars section from a play '''
-
-        if play.get('vars') is None:
-            play['vars'] = {}
-        if type(play['vars']) not in [dict, list]:
-            raise errors.AnsibleError("'vars' section must contain only key/value pairs")
-        vars = self.global_vars
-        
-        # translate a list of vars into a dict
-        if type(play['vars']) == list:
-            for item in play['vars']:
-                k, v = item.items()[0]
-                vars[k] = v
-        else:
-            vars.update(play['vars'])
-
-        vars_prompt = play.get('vars_prompt', {})
-        if type(vars_prompt) != dict:
-            raise errors.AnsibleError("'vars_prompt' section must contain only key/value pairs")
-        for vname in vars_prompt:
-            # TODO: make this prompt one line and consider double entry
-            print vars_prompt[vname]
-            vars[vname] = self.callbacks.on_vars_prompt(vname)
-
-        results = self.extra_vars.copy()
-        results.update(vars)
-        return results
-
-    # *****************************************************
-
-    def _include_tasks(self, play, task, dirname, new_tasks):
-        ''' load tasks included from external files. '''
-
-        # include: some.yml a=2 b=3 c=4
-        play_vars = self._get_vars(play, dirname)
-        include_tokens = utils.template(task['include'], play_vars, SETUP_CACHE).split()
-        path = utils.path_dwim(dirname, include_tokens[0])
-        include_vars = {}
-        for i,x in enumerate(include_tokens):
-            if x.find("=") != -1:
-                (k,v) = x.split("=")
-                include_vars[k] = v
-        inject_vars = play_vars.copy()
-        inject_vars.update(include_vars)
-        included = utils.template_from_file(path, inject_vars, SETUP_CACHE)
-        included = utils.parse_yaml(included)
-        for x in included:
-            if len(include_vars):
-                x["vars"] = include_vars
-            new_tasks.append(x)
-
-    # *****************************************************
-
-    def _include_handlers(self, play, handler, dirname, new_handlers):
-        ''' load handlers from external files '''
-
-        path = utils.path_dwim(dirname, handler['include'])
-        inject_vars = self._get_vars(play, dirname)
-        included = utils.template_from_file(path, inject_vars, SETUP_CACHE)
-        included = utils.parse_yaml(included)
-        for x in included:
-            new_handlers.append(x)
-
-    # *****************************************************
-
-    def _parse_playbook(self, playbook):
-        ''' load YAML file, including handling for imported files '''
-        
-        dirname  = os.path.dirname(playbook)
-        playbook = utils.parse_yaml_from_file(playbook)
-
-        for play in playbook:
-            tasks = play.get('tasks',[])
-            handlers = play.get('handlers', [])
-
-            # process tasks in this file as well as imported tasks
-            new_tasks = []
-            for task in tasks:
-                if 'include' in task:
-                    self._include_tasks(play, task, dirname, new_tasks)
-                else:
-                    new_tasks.append(task)
-
-            # now new_tasks contains a list of tasks, but tasks may contain
-            # lists of with_items to loop over.  Do that.
-            # TODO: refactor into subfunction
-            new_tasks2 = []
-            for task in new_tasks:
-                if 'with_items' in task:
-                    for item in task['with_items']:
-                        produced_task = {}
-                        name    = task.get('name', task.get('action', 'unnamed task'))
-                        action  = task.get('action', None)
-                        only_if = task.get('only_if', None)
-                        if action is None:
-                            raise errors.AnsibleError('action is required')
-                        produced_task = task.copy()
-                        produced_task['action'] = utils.template(action, dict(item=item), SETUP_CACHE)
-                        produced_task['name'] = utils.template(name, dict(item=item), SETUP_CACHE)
-                        if only_if:
-                            produced_task['only_if'] = utils.template(only_if, dict(item=item), SETUP_CACHE)
-                        new_tasks2.append(produced_task)
-                else:
-                    new_tasks2.append(task)
-
-            play['tasks'] = new_tasks2
-
-            # process handlers as well as imported handlers
-            new_handlers = [] 
-            for handler in handlers:
-                if 'include' in handler:
-                    self._include_handlers(play, handler, dirname, new_handlers)
-                else:
-                    new_handlers.append(handler)
-            play['handlers'] = new_handlers
-
-        return playbook
-
-    # *****************************************************
-        
-    def run(self):
-        ''' run all patterns in the playbook '''
-
-        # loop through all patterns and run them
-        self.callbacks.on_start()
-        for pattern in self.playbook:
-            self._run_play(pattern)
-
-        # summarize the results
-        results = {}
-        for host in self.stats.processed.keys():
-            results[host] = self.stats.summarize(host)
-        return results
-
-    # *****************************************************
-
-    def _async_poll(self, poller, async_seconds, async_poll_interval):
-        ''' launch an async job, if poll_interval is set, wait for completion '''
-
-        results = poller.wait(async_seconds, async_poll_interval)
-
-        # mark any hosts that are still listed as started as failed
-        # since these likely got killed by async_wrapper
-        for host in poller.hosts_to_poll:
-            reason = { 'failed' : 1, 'rc' : None, 'msg' : 'timed out' }
-            self.runner_callbacks.on_failed(host, reason)
-            results['contacted'][host] = reason
-
-        return results
-
-    # *****************************************************
-
-    def _run_module(self, pattern, module, args, vars, remote_user, 
-        async_seconds, async_poll_interval, only_if, sudo, transport, port):
-        ''' run a particular module step in a playbook '''
-
-        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
-        self.inventory.restrict_to(hosts)
-
-        if port is None:
-            port=self.remote_port
-
-        runner = ansible.runner.Runner(
-            pattern=pattern, inventory=self.inventory, module_name=module,
-            module_args=args, forks=self.forks,
-            remote_pass=self.remote_pass, module_path=self.module_path,
-            timeout=self.timeout, remote_user=remote_user, 
-            remote_port=port, module_vars=vars,
-            setup_cache=SETUP_CACHE, basedir=self.basedir,
-            conditional=only_if, callbacks=self.runner_callbacks, 
-            debug=self.debug, sudo=sudo,
-            transport=transport, sudo_pass=self.sudo_pass, is_playbook=True
-        )
-
-        if async_seconds == 0:
-            results = runner.run()
-        else:
-            results, poller = runner.runAsync(async_seconds)
-            self.stats.compute(results)
-            if async_poll_interval > 0:
-                # if not polling, playbook requested fire and forget
-                # trust the user wanted that and return immediately
-                results = self._async_poll(poller, async_seconds, async_poll_interval)
-
-        self.inventory.lift_restriction()
-        return results
-
-    # *****************************************************
-
-    def _run_task(self, pattern=None, task=None, 
-        remote_user=None, handlers=None, conditional=False, sudo=False, transport=None, port=None):
-        ''' run a single task in the playbook and recursively run any subtasks.  '''
-
-        # load the module name and parameters from the task entry
-        name    = task.get('name', None) 
-        action  = task.get('action', None)
-        if action is None:
-            raise errors.AnsibleError("action is required for each item in tasks: offending task is %s" % name if name else "unknown")
-        if name is None:
-            name = action
-        only_if = task.get('only_if', 'True')
-        async_seconds = int(task.get('async', 0))  # not async by default
-        async_poll_interval = int(task.get('poll', 10))  # default poll = 10 seconds
-
-        tokens = action.split(None, 1)
-        module_name = tokens[0]
-        module_args = ''
-        if len(tokens) > 1:
-            module_args = tokens[1]
-
-        # include task specific vars
-        module_vars = task.get('vars', {})
-        if 'first_available_file' in task:
-            module_vars['first_available_file'] = task.get('first_available_file')
-        
-        # tasks can be direct (run on all nodes matching
-        # the pattern) or conditional, where they ran
-        # as the result of a change handler on a subset
-        # of all of the hosts
-
-        self.callbacks.on_task_start(name, conditional)
-
-        # load up an appropriate ansible runner to
-        # run the task in parallel
-        results = self._run_module(pattern, module_name, 
-            module_args, module_vars, remote_user, async_seconds, 
-            async_poll_interval, only_if, sudo, transport, port)
-
-        # add facts to the global setup cache
-        for host, result in results['contacted'].iteritems():
-            if "ansible_facts" in result:
-                for k,v in result['ansible_facts'].iteritems():
-                    SETUP_CACHE[host][k]=v
-
-        self.stats.compute(results)
-
-        # if no hosts are matched, carry on, unlike /bin/ansible
-        # which would warn you about this
-        if results is None:
-            results = {}
- 
-        # flag which notify handlers need to be run
-        # this will be on a SUBSET of the actual host list.  For instance
-        # a file might need to be written on only half of the nodes so
-        # we would only trigger restarting Apache on half of the nodes
-
-        subtasks = task.get('notify', [])
-        if len(subtasks) > 0:
-            for host, results in results.get('contacted',{}).iteritems():
-                if results.get('changed', False):
-                    for subtask in subtasks:
-                        self._flag_handler(handlers, subtask, host)
-
-    # *****************************************************
-
-    def _flag_handler(self, handlers, match_name, host):
-        ''' 
-        if a task has any notify elements, flag handlers for run
-        at end of execution cycle for hosts that have indicated
-        changes have been made
-        '''
-
-        # for all registered handlers in the ansible playbook
-        # for this particular pattern group
-
-        found = False
-        for x in handlers:
-            name = x.get('name', None)
-            if name is None:
-                raise errors.AnsibleError('handler is missing a name')
-            if match_name == name:
-                found = True
-                self.callbacks.on_notify(host, name)
-                # flag the handler with the list of hosts it needs to be run on, it will be run later
-                if not 'run' in x:
-                    x['run'] = []
-                x['run'].append(host)
-        if not found:
-            raise errors.AnsibleError("change handler (%s) is not defined" % match_name)
-
-    # *****************************************************
-
-    def _do_conditional_imports(self, vars_files):
-        ''' handle the vars_files section, which can contain variables '''
-        
-        # FIXME: save parsed variable results in memory to avoid excessive re-reading/parsing
-        # FIXME: currently parses imports for hosts not in the pattern, that is not wrong, but it's 
-        #        not super optimized yet either, because we wouldn't have hit them, ergo
-        #        it will raise false errors if there is no defaults variable file without any $vars
-        #        in it, which could happen on uncontacted hosts.
- 
-        if type(vars_files) != list:
-            raise errors.AnsibleError("vars_files must be a list")
-        for host in self.inventory.list_hosts():
-            cache_vars = SETUP_CACHE.get(host,{}) 
-            SETUP_CACHE[host] = cache_vars
-            for filename in vars_files:
-                if type(filename) == list:
-                    # loop over all filenames, loading the first one, and failing if # none found
-                    found = False
-                    sequence = []
-                    for real_filename in filename:
-                        filename2 = utils.path_dwim(self.basedir, utils.template(real_filename, cache_vars, SETUP_CACHE))
-                        sequence.append(filename2)
-                        if os.path.exists(filename2):
-                            found = True
-                            data = utils.parse_yaml_from_file(filename2)
-                            SETUP_CACHE[host].update(data)
-                            self.callbacks.on_import_for_host(host, filename2)
-                            break
-                        else:
-                            self.callbacks.on_not_import_for_host(host, filename2)
-                    if not found:
-                        raise errors.AnsibleError(
-                            "%s: FATAL, no files matched for vars_files import sequence: %s" % (host, sequence)
-                        )
-
-                else:
-                    filename2 = utils.path_dwim(self.basedir, utils.template(filename, cache_vars, SETUP_CACHE))
-                    if not os.path.exists(filename2):
-                        raise errors.AnsibleError("no file matched for vars_file import: %s" % filename2)
-                    data = utils.parse_yaml_from_file(filename2)
-                    SETUP_CACHE[host].update(data)
-                    self.callbacks.on_import_for_host(host, filename2)
-
-    # *****************************************************
-
-    def _do_setup_step(self, pattern, vars, user, port, sudo, transport, vars_files=None):
-        ''' push variables down to the systems and get variables+facts back up '''
-
-        # this enables conditional includes like $facter_os.yml and is only done
-        # after the original pass when we have that data.
-        #
-
-        if vars_files is not None:
-            self.callbacks.on_setup_secondary()
-            self._do_conditional_imports(vars_files)
-        else:
-            self.callbacks.on_setup_primary()
-
-        host_list = [ h for h in self.inventory.list_hosts(pattern) 
-                        if not (h in self.stats.failures or h in self.stats.dark) ]
-        self.inventory.restrict_to(host_list)
-
-        # push any variables down to the system
-        setup_results = ansible.runner.Runner(
-            pattern=pattern, module_name='setup',
-            module_args=vars, inventory=self.inventory,
-            forks=self.forks, module_path=self.module_path,
-            timeout=self.timeout, remote_user=user,
-            remote_pass=self.remote_pass, remote_port=port,
-            setup_cache=SETUP_CACHE,
-            callbacks=self.runner_callbacks, sudo=sudo, debug=self.debug,
-            transport=transport, sudo_pass=self.sudo_pass, is_playbook=True
-        ).run()
-        self.stats.compute(setup_results, setup=True)
-
-        self.inventory.lift_restriction()
-
-        # now for each result, load into the setup cache so we can
-        # let runner template out future commands
-        setup_ok = setup_results.get('contacted', {})
-        if vars_files is None:
-            # first pass only or we'll erase good work
-            for (host, result) in setup_ok.iteritems():
-                if 'ansible_facts' in result:
-                    SETUP_CACHE[host] = result['ansible_facts']
-
-    # *****************************************************
-
-    def _run_play(self, pg):
-        ''' run a list of tasks for a given pattern, in order '''
-
-        # get configuration information about the pattern
-        pattern = pg.get('hosts',None)
-        name = pg.get('name', pattern)
-        if isinstance(pattern, list):
-            pattern = ';'.join(pattern)
-        if self.override_hosts:
-            pattern = 'all'
-        if pattern is None:
-            raise errors.AnsibleError('hosts declaration is required')
-
-        vars       = self._get_vars(pg, self.basedir)
-        vars_files = pg.get('vars_files', {})
-        tasks      = pg.get('tasks', [])
-        handlers   = pg.get('handlers', [])
-        user       = pg.get('user', self.remote_user)
-        port       = pg.get('port', self.remote_port)
-        sudo       = pg.get('sudo', self.sudo)
-        transport  = pg.get('connection', self.transport)
-
-        self.callbacks.on_play_start(name)
-
-        # push any variables down to the system # and get facts/ohai/other data back up
-        self._do_setup_step(pattern, vars, user, port, sudo, transport, None)
-         
-        # now with that data, handle contentional variable file imports!
-        if len(vars_files) > 0:
-            self._do_setup_step(pattern, vars, user, port, sudo, transport, vars_files)
-
-        # run all the top level tasks, these get run on every node
-        for task in tasks:
-            self._run_task(
-                pattern=pattern,
-                task=task, 
-                handlers=handlers,
-                remote_user=user,
-                sudo=sudo,
-                transport=transport,
-                port=port
-            )
-
-        # handlers only run on certain nodes, they are flagged by _flag_handlers
-        # above.  They only run on nodes when things mark them as changed, and
-        # handlers only get run once.  For instance, the system is designed
-        # such that multiple config files if changed can ask for an Apache restart
-        # but Apache will only be restarted once (at the end).
-
-        for task in handlers:
-            triggered_by = task.get('run', None)
-            if type(triggered_by) == list:
-                self.inventory.restrict_to(triggered_by)
-                self._run_task(
-                   pattern=pattern, 
-                   task=task,
-                   handlers=[],
-                   conditional=True,
-                   remote_user=user,
-                   sudo=sudo,
-                   transport=transport,
-                   port=port
-                )
-                self.inventory.lift_restriction()
-
-        # end of execution for this particular pattern.  Multiple patterns
-        # can be in a single playbook file
-
- 
-

lib/ansible/playbook/__init__.py

@@ -0,0 +1,299 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import ansible.inventory
+import ansible.runner
+import ansible.constants as C
+from ansible import utils
+from ansible import errors
+import os
+from play import Play
+
+#############################################
+
+class PlayBook(object):
+
+    '''
+    runs an ansible playbook, given as a datastructure
+    or YAML filename.  a playbook is a deployment, config
+    management, or automation based set of commands to
+    run in series.
+
+    multiple plays/tasks do not execute simultaneously,
+    but tasks in each pattern do execute in parallel
+    (according to the number of forks requested) among
+    the hosts they address
+    '''
+
+    # *****************************************************
+
+    def __init__(self,
+        playbook         = None,
+        host_list        = C.DEFAULT_HOST_LIST,
+        module_path      = C.DEFAULT_MODULE_PATH,
+        forks            = C.DEFAULT_FORKS,
+        timeout          = C.DEFAULT_TIMEOUT,
+        remote_user      = C.DEFAULT_REMOTE_USER,
+        remote_pass      = C.DEFAULT_REMOTE_PASS,
+        sudo_pass        = C.DEFAULT_SUDO_PASS,
+        remote_port      = C.DEFAULT_REMOTE_PORT,
+        transport        = C.DEFAULT_TRANSPORT,
+        private_key_file = C.DEFAULT_PRIVATE_KEY_FILE,
+        verbose          = False,
+        callbacks        = None,
+        runner_callbacks = None,
+        stats            = None,
+        sudo             = False,
+        sudo_user        = C.DEFAULT_SUDO_USER,
+        extra_vars       = None):
+
+        """
+        playbook:         path to a playbook file
+        host_list:        path to a file like /etc/ansible/hosts
+        module_path:      path to ansible modules, like /usr/share/ansible/
+        forks:            desired level of paralellism
+        timeout:          connection timeout
+        remote_user:      run as this user if not specified in a particular play
+        remote_pass:      use this remote password (for all plays) vs using SSH keys
+        sudo_pass:        if sudo==True, and a password is required, this is the sudo password
+        remote_port:      default remote port to use if not specified with the host or play
+        transport:        how to connect to hosts that don't specify a transport (local, paramiko, etc)
+        callbacks         output callbacks for the playbook
+        runner_callbacks: more callbacks, this time for the runner API
+        stats:            holds aggregrate data about events occuring to each host
+        sudo:             if not specified per play, requests all plays use sudo mode
+        """
+
+        self.SETUP_CACHE = {}
+
+        if playbook is None or callbacks is None or runner_callbacks is None or stats is None:
+            raise Exception('missing required arguments')
+
+        if extra_vars is None:
+            extra_vars = {}
+       
+        self.module_path      = module_path
+        self.forks            = forks
+        self.timeout          = timeout
+        self.remote_user      = remote_user
+        self.remote_pass      = remote_pass
+        self.remote_port      = remote_port
+        self.transport        = transport
+        self.verbose          = verbose
+        self.callbacks        = callbacks
+        self.runner_callbacks = runner_callbacks
+        self.stats            = stats
+        self.sudo             = sudo
+        self.sudo_pass        = sudo_pass
+        self.sudo_user        = sudo_user
+        self.extra_vars       = extra_vars
+        self.global_vars      = {}
+        self.private_key_file = private_key_file
+
+        self.inventory = ansible.inventory.Inventory(host_list)
+        
+        if not self.inventory._is_script:
+            self.global_vars.update(self.inventory.get_group_variables('all'))
+
+        self.basedir    = os.path.dirname(playbook)
+        self.playbook  = utils.parse_yaml_from_file(playbook)
+
+        self.module_path = self.module_path + os.pathsep + os.path.join(self.basedir, "library")
+
+    # *****************************************************
+        
+    def run(self):
+        ''' run all patterns in the playbook '''
+
+        # loop through all patterns and run them
+        self.callbacks.on_start()
+        for play_ds in self.playbook:
+            self.SETUP_CACHE = {}
+            self._run_play(Play(self,play_ds))
+
+        # summarize the results
+        results = {}
+        for host in self.stats.processed.keys():
+            results[host] = self.stats.summarize(host)
+        return results
+
+    # *****************************************************
+
+    def _async_poll(self, poller, async_seconds, async_poll_interval):
+        ''' launch an async job, if poll_interval is set, wait for completion '''
+
+        results = poller.wait(async_seconds, async_poll_interval)
+
+        # mark any hosts that are still listed as started as failed
+        # since these likely got killed by async_wrapper
+        for host in poller.hosts_to_poll:
+            reason = { 'failed' : 1, 'rc' : None, 'msg' : 'timed out' }
+            self.runner_callbacks.on_failed(host, reason)
+            results['contacted'][host] = reason
+
+        return results
+
+    # *****************************************************
+
+    def _run_task_internal(self, task):
+        ''' run a particular module step in a playbook '''
+
+        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
+        self.inventory.restrict_to(hosts)
+
+        runner = ansible.runner.Runner(
+            pattern=task.play.hosts, inventory=self.inventory, module_name=task.module_name,
+            module_args=task.module_args, forks=self.forks,
+            remote_pass=self.remote_pass, module_path=self.module_path,
+            timeout=self.timeout, remote_user=task.play.remote_user, 
+            remote_port=task.play.remote_port, module_vars=task.module_vars,
+            private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, basedir=self.basedir,
+            conditional=task.only_if, callbacks=self.runner_callbacks, 
+            verbose=self.verbose, sudo=task.play.sudo, sudo_user=task.play.sudo_user,
+            transport=task.play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        )
+
+        if task.async_seconds == 0:
+            results = runner.run()
+        else:
+            results, poller = runner.run_async(task.async_seconds)
+            self.stats.compute(results)
+            if task.async_poll_interval > 0:
+                # if not polling, playbook requested fire and forget, so don't poll
+                results = self._async_poll(poller, task.async_seconds, task.async_poll_interval)
+
+        self.inventory.lift_restriction()
+        return results
+
+    # *****************************************************
+
+    def _run_task(self, play, task, is_handler):
+        ''' run a single task in the playbook and recursively run any subtasks.  '''
+
+        self.callbacks.on_task_start(task.name, is_handler)
+
+        # load up an appropriate ansible runner to run the task in parallel
+        results = self._run_task_internal(task)
+
+        # add facts to the global setup cache
+        for host, result in results['contacted'].iteritems():
+            if "ansible_facts" in result:
+                for k,v in result['ansible_facts'].iteritems():
+                    self.SETUP_CACHE[host][k]=v
+
+        self.stats.compute(results)
+
+        # if no hosts are matched, carry on
+        if results is None:
+            results = {}
+ 
+        # flag which notify handlers need to be run
+        if len(task.notify) > 0:
+            for host, results in results.get('contacted',{}).iteritems():
+                if results.get('changed', False):
+                    for handler_name in task.notify:
+                        self._flag_handler(play.handlers(), utils.template(handler_name, task.module_vars), host)
+
+    # *****************************************************
+
+    def _flag_handler(self, handlers, handler_name, host):
+        ''' 
+        if a task has any notify elements, flag handlers for run
+        at end of execution cycle for hosts that have indicated
+        changes have been made
+        '''
+
+        found = False
+        for x in handlers:
+            if handler_name == x.name:
+                found = True
+                self.callbacks.on_notify(host, x.name)
+                x.notified_by.append(host)
+        if not found:
+            raise errors.AnsibleError("change handler (%s) is not defined" % handler_name)
+
+    # *****************************************************
+
+    def _do_setup_step(self, play, vars_files=None):
+
+        ''' push variables down to the systems and get variables+facts back up '''
+
+        # this enables conditional includes like $facter_os.yml and is only done
+        # after the original pass when we have that data.
+        #
+
+        if vars_files is not None:
+            self.callbacks.on_setup_secondary()
+            play.update_vars_files(self.inventory.list_hosts(play.hosts))
+        else:
+            self.callbacks.on_setup_primary()
+
+        host_list = [ h for h in self.inventory.list_hosts(play.hosts) 
+            if not (h in self.stats.failures or h in self.stats.dark) ]
+
+        self.inventory.restrict_to(host_list)
+
+        # push any variables down to the system
+        setup_results = ansible.runner.Runner(
+            pattern=play.hosts, module_name='setup', module_args=play.vars, inventory=self.inventory,
+            forks=self.forks, module_path=self.module_path, timeout=self.timeout, remote_user=play.remote_user,
+            remote_pass=self.remote_pass, remote_port=play.remote_port, private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, callbacks=self.runner_callbacks, sudo=play.sudo, sudo_user=play.sudo_user, 
+            verbose=self.verbose, transport=play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        ).run()
+        self.stats.compute(setup_results, setup=True)
+
+        self.inventory.lift_restriction()
+
+        # now for each result, load into the setup cache so we can
+        # let runner template out future commands
+        setup_ok = setup_results.get('contacted', {})
+        if vars_files is None:
+            # first pass only or we'll erase good work
+            for (host, result) in setup_ok.iteritems():
+                if 'ansible_facts' in result:
+                    self.SETUP_CACHE[host] = result['ansible_facts']
+        return setup_results
+
+    # *****************************************************
+
+    def _run_play(self, play):
+        ''' run a list of tasks for a given pattern, in order '''
+
+        self.callbacks.on_play_start(play.name)
+
+        # push any variables down to the system # and get facts/ohai/other data back up
+        rc = self._do_setup_step(play) # pattern, vars, user, port, sudo, sudo_user, transport, None)
+
+        # now with that data, handle contentional variable file imports!
+        if play.vars_files and len(play.vars_files) > 0:
+            rc = self._do_setup_step(play, play.vars_files)
+
+        # run all the top level tasks, these get run on every node
+        for task in play.tasks():
+            self._run_task(play, task, False)
+
+        # run notify actions
+        for handler in play.handlers():
+            if len(handler.notified_by) > 0:
+                self.inventory.restrict_to(handler.notified_by)
+                self._run_task(play, handler, True)
+                self.inventory.lift_restriction()
+

lib/ansible/playbook/play.py

@@ -0,0 +1,190 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+from ansible import utils
+from ansible import errors
+from ansible.playbook.task import Task
+import shlex
+import os
+
+class Play(object):
+
+    __slots__ = [ 
+       'hosts', 'name', 'vars', 'vars_prompt', 'vars_files', 'handlers', 'remote_user', 'remote_port',
+       'sudo', 'sudo_user', 'transport', 'playbook', '_ds', '_handlers', '_tasks' 
+    ]
+
+    # *************************************************
+
+    def __init__(self, playbook, ds):
+        ''' constructor loads from a play datastructure '''
+
+        # TODO: more error handling
+
+        hosts = ds.get('hosts')
+        if hosts is None:
+            raise errors.AnsibleError('hosts declaration is required')
+        elif isinstance(hosts, list):
+            hosts = ';'.join(hosts)
+        hosts = utils.template(hosts, playbook.extra_vars, {})
+
+        self._ds         = ds
+        self.playbook    = playbook
+        self.hosts       = hosts 
+        self.name        = ds.get('name', self.hosts)
+        self.vars        = ds.get('vars', {})
+        self.vars_files  = ds.get('vars_files', [])
+        self.vars_prompt = ds.get('vars_prompt', {})
+        self.vars        = self._get_vars(self.playbook.basedir)
+        self._tasks      = ds.get('tasks', [])
+        self._handlers   = ds.get('handlers', [])
+        self.remote_user = ds.get('user', self.playbook.remote_user)
+        self.remote_port = ds.get('port', self.playbook.remote_port)
+        self.sudo        = ds.get('sudo', self.playbook.sudo)
+        self.sudo_user   = ds.get('sudo_user', self.playbook.sudo_user)
+        self.transport   = ds.get('connection', self.playbook.transport)
+        self._tasks      = self._load_tasks(self._ds, 'tasks')
+        self._handlers   = self._load_tasks(self._ds, 'handlers')
+
+        if self.sudo_user != 'root':
+            self.sudo = True
+
+    # *************************************************
+
+    def _load_tasks(self, ds, keyname):
+        ''' handle task and handler include statements '''
+
+        tasks = ds.get(keyname, [])
+        results = []
+        for x in tasks:
+            task_vars = self.vars.copy()
+            if 'include' in x:
+                tokens = shlex.split(x['include'])
+                for t in tokens[1:]:
+                    (k,v) = t.split("=", 1)
+                    task_vars[k]=v
+                include_file = tokens[0]
+                data = utils.parse_yaml_from_file(utils.path_dwim(self.playbook.basedir, include_file))
+            elif type(x) == dict:
+                data = [x]
+            else:
+                raise Exception("unexpected task type")
+            for y in data:
+                items = y.get('with_items',None)
+                if items is None:
+                    items = [ '' ]
+                elif isinstance(items, basestring):
+                    items = utils.varLookup(items, task_vars)
+                for item in items:
+                    mv = task_vars.copy()
+                    mv['item'] = item
+                    results.append(Task(self,y,module_vars=mv))
+        return results
+
+    # *************************************************
+
+    def tasks(self):
+        ''' return task objects for this play '''
+        return self._tasks      
+
+    def handlers(self):
+        ''' return handler objects for this play '''
+        return self._handlers
+
+    # *************************************************
+
+    def _get_vars(self, dirname):
+        ''' load the vars section from a play, accounting for all sorts of variable features
+        including loading from yaml files, prompting, and conditional includes of the first
+        file found in a list. '''
+
+        if self.vars is None:
+            self.vars = {}
+
+        if type(self.vars) not in [dict, list]:
+            raise errors.AnsibleError("'vars' section must contain only key/value pairs")
+
+        vars = self.playbook.global_vars
+    
+        # translate a list of vars into a dict
+        if type(self.vars) == list:
+            for item in self.vars:
+                k, v = item.items()[0]
+                vars[k] = v
+        else:
+            vars.update(self.vars)
+
+        if type(self.vars_prompt) != dict:
+            raise errors.AnsibleError("'vars_prompt' section must contain only key/value pairs")
+        for vname in self.vars_prompt:
+            vars[vname] = self.playbook.callbacks.on_vars_prompt(vname)
+
+        results = self.playbook.extra_vars.copy()
+        results.update(vars)
+        return results
+
+    # *************************************************
+
+    def update_vars_files(self, hosts):
+        ''' calculate vars_files, which requires that setup runs first so ansible facts can be mixed in '''
+        for h in hosts:
+            self._update_vars_files_for_host(h)
+
+    # *************************************************
+
+    def _update_vars_files_for_host(self, host):
+
+        if not host in self.playbook.SETUP_CACHE:
+            # no need to process failed hosts or hosts not in this play
+            return
+
+        for filename in self.vars_files:
+
+            if type(filename) == list:
+
+                # loop over all filenames, loading the first one, and failing if # none found
+                found = False
+                sequence = []
+                for real_filename in filename:
+                    filename2 = utils.template(real_filename, self.playbook.SETUP_CACHE[host])
+                    filename2 = utils.template(filename2, self.vars)
+                    filename2 = utils.path_dwim(self.playbook.basedir, filename2)
+                    sequence.append(filename2)
+                    if os.path.exists(filename2):
+                        found = True
+                        data = utils.parse_yaml_from_file(filename2)
+                        self.playbook.SETUP_CACHE[host].update(data)
+                        self.playbook.callbacks.on_import_for_host(host, filename2)
+                        break
+                    else:
+                        self.playbook.callbacks.on_not_import_for_host(host, filename2)
+                if not found:
+                    raise errors.AnsibleError(
+                        "%s: FATAL, no files matched for vars_files import sequence: %s" % (host, sequence)
+                    )
+
+            else:
+
+                filename2 = utils.template(filename, self.playbook.SETUP_CACHE[host])
+                filename2 = utils.template(filename2, self.vars)
+                fpath = utils.path_dwim(self.playbook.basedir, filename2)
+                new_vars = utils.parse_yaml_from_file(fpath)
+                if new_vars:
+                    self.playbook.SETUP_CACHE[host].update(new_vars)
+                #else: could warn if vars file contains no vars. 

lib/ansible/playbook/task.py

@@ -0,0 +1,70 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+from ansible import errors
+from ansible import utils
+
+class Task(object):
+
+    __slots__ = [ 
+        'name', 'action', 'only_if', 'async_seconds', 'async_poll_interval',
+        'notify', 'module_name', 'module_args', 'module_vars', 'play', 'notified_by',
+    ]
+
+    def __init__(self, play, ds, module_vars=None):
+        ''' constructor loads from a task or handler datastructure '''
+
+        # TODO: more error handling
+        # include task specific vars
+
+        self.module_vars = module_vars
+
+        self.play        = play
+        self.name        = ds.get('name', None)
+        self.action      = ds.get('action', '')
+        self.notified_by = []
+
+        if self.name is None:
+            self.name = self.action
+
+        self.only_if = ds.get('only_if', 'True')
+        self.async_seconds = int(ds.get('async', 0))  # not async by default
+        self.async_poll_interval = int(ds.get('poll', 10))  # default poll = 10 seconds
+        self.notify = ds.get('notify', [])
+        if isinstance(self.notify, basestring):
+            self.notify = [ self.notify ]
+
+        tokens = self.action.split(None, 1)
+        if len(tokens) < 1:
+            raise errors.AnsibleError("invalid/missing action in task")
+
+        self.module_name = tokens[0]
+        self.module_args = ''
+        if len(tokens) > 1:
+            self.module_args = tokens[1]
+
+
+        self.name = utils.template(self.name, self.module_vars)
+        self.action = utils.template(self.name, self.module_vars)
+
+
+        if 'first_available_file' in ds:
+            self.module_vars['first_available_file'] = ds.get('first_available_file')
+
+

lib/ansible/runner/__init__.py

@@ -29,12 +29,14 @@ import tempfile
 import time
 import base64
 import getpass
+import codecs
 
 import ansible.constants as C 
-import ansible.connection
 import ansible.inventory
 from ansible import utils
 from ansible import errors
+import poller
+import connection
 from ansible import callbacks as ans_callbacks
     
 HAS_ATFORK=True
@@ -67,6 +69,37 @@ def _executor_hook(job_queue, result_queue):
  
 ################################################
 
+class ReturnData(object):
+
+    __slots__ = [ 'result', 'comm_ok', 'executed_str', 'host' ]
+
+    def __init__(self, host=None, result=None, comm_ok=True, executed_str=''):
+        self.host = host
+        self.result = result
+        self.comm_ok = comm_ok
+        self.executed_str = executed_str
+
+        if type(self.result) in [ str, unicode ]:
+            self.result = utils.parse_json(self.result)
+
+        if host is None:
+            raise Exception("host not set")
+        if type(self.result) != dict:
+            raise Exception("dictionary result expected")
+
+    def communicated_ok(self):
+        return self.comm_ok
+
+    def is_successful(self):
+        if not self.comm_ok:
+            return False
+        else:
+            if 'failed' in self.result:
+                return False
+            if self.result.get('rc',0) != 0:
+                return False
+            return True
+
 class Runner(object):
 
     def __init__(self, 
@@ -75,10 +108,11 @@ class Runner(object):
         forks=C.DEFAULT_FORKS, timeout=C.DEFAULT_TIMEOUT, 
         pattern=C.DEFAULT_PATTERN, remote_user=C.DEFAULT_REMOTE_USER, 
         remote_pass=C.DEFAULT_REMOTE_PASS, remote_port=C.DEFAULT_REMOTE_PORT, 
-        sudo_pass=C.DEFAULT_SUDO_PASS, background=0, basedir=None, 
-        setup_cache=None, transport=C.DEFAULT_TRANSPORT, conditional='True', 
-        callbacks=None, debug=False, sudo=False, module_vars=None, 
-        is_playbook=False, inventory=None):
+        private_key_file=C.DEFAULT_PRIVATE_KEY_FILE, sudo_pass=C.DEFAULT_SUDO_PASS, 
+        background=0, basedir=None, setup_cache=None, 
+        transport=C.DEFAULT_TRANSPORT, conditional='True', callbacks=None, 
+        verbose=False, sudo=False, sudo_user=C.DEFAULT_SUDO_USER,
+        module_vars=None, is_playbook=False, inventory=None):
 
         """
         host_list    : path to a host list file, like /etc/ansible/hosts
@@ -91,6 +125,8 @@ class Runner(object):
         remote_user  : connect as this remote username
         remote_pass  : supply this password (if not using keys)
         remote_port  : use this default remote port (if not set by the inventory system)
+        private_key_file  : use this private key as your auth key
+        sudo_user    : If you want to sudo to a user other than root.
         sudo_pass    : sudo password if using sudo and sudo requires a password
         background   : run asynchronously with a cap of this many # of seconds (if not 0)
         basedir      : paths used by modules if not absolute are relative to here
@@ -99,7 +135,7 @@ class Runner(object):
         conditional  : only execute if this string, evaluated, is True
         callbacks    : output callback class
         sudo         : log in as remote user and immediately sudo to root
-        module_vars  : provides additional variables to a template.  FIXME: just use module_args, remove
+        module_vars  : provides additional variables to a template.  FIXME: factor this out
         is_playbook  : indicates Runner is being used by a playbook.  affects behavior in various ways.
         inventory    : inventory object, if host_list is not provided
         """
@@ -115,8 +151,9 @@ class Runner(object):
 
         self.generated_jid = str(random.randint(0, 999999999999))
 
+        self.sudo_user = sudo_user
         self.transport = transport
-        self.connector = ansible.connection.Connection(self, self.transport)
+        self.connector = connection.Connection(self, self.transport, self.sudo_user)
 
         if inventory is None:
             self.inventory = ansible.inventory.Inventory(host_list)
@@ -135,10 +172,11 @@ class Runner(object):
         self.module_args = module_args
         self.module_vars = module_vars
         self.timeout     = timeout
-        self.debug       = debug
+        self.verbose     = verbose
         self.remote_user = remote_user
         self.remote_pass = remote_pass
         self.remote_port = remote_port
+        self.private_key_file = private_key_file
         self.background  = background
         self.basedir     = basedir
         self.sudo        = sudo
@@ -147,43 +185,17 @@ class Runner(object):
 
         euid = pwd.getpwuid(os.geteuid())[0]
         if self.transport == 'local' and self.remote_user != euid:
-            raise Exception("User mismatch: expected %s, but is %s" % (self.remote_user, euid))
+            raise errors.AnsibleError("User mismatch: expected %s, but is %s" % (self.remote_user, euid))
         if type(self.module_args) not in [str, unicode, dict]:
-            raise Exception("module_args must be a string or dict: %s" % self.module_args)
+            raise errors.AnsibleError("module_args must be a string or dict: %s" % self.module_args)
+        if self.transport == 'ssh' and self.remote_pass:
+            raise errors.AnsibleError("SSH transport does not support remote passwords, only keys or agents")
 
         self._tmp_paths  = {}
         random.seed()
 
     # *****************************************************
 
-    @classmethod
-    def parse_hosts(cls, host_list, override_hosts=None):
-        ''' parse the host inventory file, returns (hosts, groups) '''
-
-        if override_hosts is None:
-            inventory = ansible.inventory.Inventory(host_list)
-        else:
-            inventory = ansible.inventory.Inventory(override_hosts)
-
-        return inventory.host_list, inventory.groups
-
-    # *****************************************************
-
-    def _return_from_module(self, conn, host, result, err, executed=None):
-        ''' helper function to handle JSON parsing of results '''
-
-        try:
-            result = utils.parse_json(result) 
-            if executed is not None:
-                result['invocation'] = executed
-                if 'stderr' in result:
-                    err="%s%s"%(err,result['stderr'])
-            return [host, True, result, err]
-        except Exception, e:
-            return [host, False, "%s/%s/%s" % (str(e), result, executed), err]
-
-    # *****************************************************
-
     def _delete_remote_files(self, conn, files):
         ''' deletes one or more remote files '''
 
@@ -192,7 +204,7 @@ class Runner(object):
         for filename in files:
             if filename.find('/tmp/') == -1:
                 raise Exception("not going to happen")
-            self._exec_command(conn, "rm -rf %s" % filename, None)
+            self._low_level_exec_command(conn, "rm -rf %s" % filename, None)
 
     # *****************************************************
 
@@ -200,7 +212,7 @@ class Runner(object):
         ''' transfers a module file to the remote side to execute it, but does not execute it yet '''
 
         outpath = self._copy_module(conn, tmp, module)
-        self._exec_command(conn, "chmod +x %s" % outpath, tmp)
+        self._low_level_exec_command(conn, "chmod +x %s" % outpath, tmp)
         return outpath
 
     # *****************************************************
@@ -213,7 +225,7 @@ class Runner(object):
 
         afd, afile = tempfile.mkstemp()
         afo = os.fdopen(afd, 'w')
-        afo.write(data)
+        afo.write(data.encode("utf8"))
         afo.flush()
         afo.close()
 
@@ -257,30 +269,30 @@ class Runner(object):
                 if self.remote_user == 'root':
                     args = "%s metadata=/etc/ansible/setup" % args
                 else:
-                    args = "%s metadata=$HOME/.ansible/setup" % args
+                    args = "%s metadata=%s/setup" % (args, C.DEFAULT_REMOTE_TMP)
         else:
             if not 'metadata' in args:
                 if self.remote_user == 'root':
                     args['metadata'] = '/etc/ansible/setup'
                 else:
-                    args['metadata'] = "$HOME/.ansible/setup"
+                    args['metadata'] = "%s/setup" % C.DEFAULT_REMOTE_TMP
         return args   
  
     # *****************************************************
 
     def _execute_module(self, conn, tmp, remote_module_path, args, 
         async_jid=None, async_module=None, async_limit=None):
+
         ''' runs a module that has already been transferred '''
 
-        inject = self.setup_cache.get(conn.host,{})
-        conditional = utils.double_template(self.conditional, inject, self.setup_cache)
-        if not eval(conditional):
-            return [ utils.smjson(dict(skipped=True)), None, 'skipped' ]
-
+        inject = self.setup_cache.get(conn.host,{}).copy()
         host_variables = self.inventory.get_variables(conn.host)
         inject.update(host_variables)
+        inject.update(self.module_vars)
 
         if self.module_name == 'setup':
+            if not args:
+                args = {}
             args = self._add_setup_vars(inject, args)
             args = self._add_setup_metadata(args)
 
@@ -296,9 +308,11 @@ class Runner(object):
         else:
             cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module, argsfile]])
 
-        res, err = self._exec_command(conn, cmd, tmp, sudoable=True)
-        client_executed_str = "%s %s" % (module_name_tail, args.strip())
-        return ( res, err, client_executed_str )
+        res = self._low_level_exec_command(conn, cmd, tmp, sudoable=True)
+
+        executed_str = "%s %s" % (module_name_tail, args.strip())
+
+        return ReturnData(host=conn.host, result=res, executed_str=executed_str)
 
     # *****************************************************
 
@@ -340,7 +354,15 @@ class Runner(object):
 
     # *****************************************************
 
-    def _execute_normal_module(self, conn, host, tmp, module_name):
+    def _execute_raw(self, conn, tmp):
+        ''' execute a non-module command for bootstrapping, or if there's no python on a device '''
+        stdout = self._low_level_exec_command( conn, self.module_args, tmp, sudoable = True )
+        data = dict(stdout=stdout)
+        return ReturnData(host=conn.host, result=data)
+
+    # ***************************************************
+
+    def _execute_normal_module(self, conn, tmp, module_name):
         ''' transfer & execute a module that is not 'copy' or 'template' '''
 
         # shell and command are the same module
@@ -349,22 +371,17 @@ class Runner(object):
             self.module_args += " #USE_SHELL"
 
         module = self._transfer_module(conn, tmp, module_name)
-        (result, err, executed) = self._execute_module(conn, tmp, module, self.module_args)
-
-        (host, ok, data, err) = self._return_from_module(conn, host, result, err, executed)
-
-        if ok:
-            self._add_result_to_setup_cache(conn, data)
-
-        return (host, ok, data, err)
+        exec_rc = self._execute_module(conn, tmp, module, self.module_args)
+        if exec_rc.is_successful():
+            self._add_result_to_setup_cache(conn, exec_rc.result)
+        return exec_rc
 
     # *****************************************************
 
-    def _execute_async_module(self, conn, host, tmp, module_name):
+    def _execute_async_module(self, conn, tmp, module_name):
         ''' transfer the given module name, plus the async module, then run it '''
 
-        # hack to make the 'shell' module keyword really be executed
-        # by the command module
+        # shell and command module are the same
         module_args = self.module_args
         if module_name == 'shell':
             module_name = 'command'
@@ -372,17 +389,16 @@ class Runner(object):
 
         async  = self._transfer_module(conn, tmp, 'async_wrapper')
         module = self._transfer_module(conn, tmp, module_name)
-        (result, err, executed) = self._execute_module(conn, tmp, async, module_args,
+
+        return self._execute_module(conn, tmp, async, module_args,
            async_module=module, 
            async_jid=self.generated_jid, 
            async_limit=self.background
         )
 
-        return self._return_from_module(conn, host, result, err, executed)
-
     # *****************************************************
 
-    def _execute_copy(self, conn, host, tmp):
+    def _execute_copy(self, conn, tmp):
         ''' handler for file transfer operations '''
 
         # load up options
@@ -390,12 +406,12 @@ class Runner(object):
         source = options.get('src', None)
         dest   = options.get('dest', None)
         if (source is None and not 'first_available_file' in self.module_vars) or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
+            result=dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, result=result)
 
         # apply templating to source argument
         inject = self.setup_cache.get(conn.host,{})
         
-        # FIXME: break duplicate code up into subfunction
         # if we have first_available_file in our vars
         # look up the files and use the first one we find as src
         if 'first_available_file' in self.module_vars:
@@ -407,8 +423,12 @@ class Runner(object):
                     found = True
                     break
             if not found:
-                return (host, True, dict(failed=True, msg="could not find src in first_available_file list"), '')
+                results=dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(host=conn.host, is_error=True, results=results)
         
+        if self.module_vars is not None:
+            inject.update(self.module_vars)
+
         source = utils.template(source, inject, self.setup_cache)
 
         # transfer the file to a remote tmp location
@@ -421,17 +441,16 @@ class Runner(object):
 
         # run the copy module
         args = "src=%s dest=%s" % (tmp_src, dest)
-        (result1, err, executed) = self._execute_module(conn, tmp, module, args)
-        (host, ok, data, err) = self._return_from_module(conn, host, result1, err, executed)
+        exec_rc = self._execute_module(conn, tmp, module, args)
 
-        if ok:
-            return self._chain_file_module(conn, tmp, data, err, options, executed)
+        if exec_rc.is_successful():
+            return self._chain_file_module(conn, tmp, exec_rc, options)
         else:
-            return (host, ok, data, err) 
+            return exec_rc
 
     # *****************************************************
 
-    def _execute_fetch(self, conn, host, tmp):
+    def _execute_fetch(self, conn, tmp):
         ''' handler for fetch operations '''
 
         # load up options
@@ -439,54 +458,70 @@ class Runner(object):
         source = options.get('src', None)
         dest = options.get('dest', None)
         if source is None or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
+            results = dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, result=results)
 
+        # apply templating to source argument
+        inject = self.setup_cache.get(conn.host,{})
+        if self.module_vars is not None:
+            inject.update(self.module_vars)
+        source = utils.template(source, inject, self.setup_cache)
+
+        # apply templating to dest argument
+        dest = utils.template(dest, inject, self.setup_cache)
+       
         # files are saved in dest dir, with a subdir for each host, then the filename
-        dest   = "%s/%s/%s" % (utils.path_dwim(self.basedir, dest), host, source)
+        dest   = "%s/%s/%s" % (utils.path_dwim(self.basedir, dest), conn.host, source)
         dest   = dest.replace("//","/")
 
         # compare old and new md5 for support of change hooks
         local_md5 = None
         if os.path.exists(dest):
-            local_md5 = os.popen("md5sum %s" % dest).read().split()[0]
-        remote_md5 = self._exec_command(conn, "md5sum %s" % source, tmp, True)[0].split()[0]
+            local_md5 = os.popen("/usr/bin/md5sum %(file)s 2> /dev/null || /sbin/md5 -q %(file)s" % {"file": dest}).read().split()[0]
+        remote_md5 = self._low_level_exec_command(conn, "/usr/bin/md5sum %(file)s 2> /dev/null || /sbin/md5 -q %(file)s" % {"file": source}, tmp, True).split()[0]
 
         if remote_md5 != local_md5:
             # create the containing directories, if needed
-            os.makedirs(os.path.dirname(dest))
+            if not os.path.isdir(os.path.dirname(dest)):
+                os.makedirs(os.path.dirname(dest))
+
             # fetch the file and check for changes
             conn.fetch_file(source, dest)
-            new_md5 = os.popen("md5sum %s" % dest).read().split()[0]
+            new_md5 = os.popen("/usr/bin/md5sum %(file)s 2> /dev/null || /sbin/md5 -q %(file)s" % {"file": dest}).read().split()[0]
             if new_md5 != remote_md5:
-                return (host, True, dict(failed=True, msg="md5 mismatch", md5sum=new_md5), '')
-            return (host, True, dict(changed=True, md5sum=new_md5), '')
+                result = dict(failed=True, msg="md5 mismatch", md5sum=new_md5)
+                return ReturnData(host=conn.host, result=result)
+            result = dict(changed=True, md5sum=new_md5)
+            return ReturnData(host=conn.host, result=result)
         else:
-            return (host, True, dict(changed=False, md5sum=local_md5), '')
+            result = dict(changed=False, md5sum=local_md5)
+            return ReturnData(host=conn.host, result=result)
         
         
     # *****************************************************
 
-    def _chain_file_module(self, conn, tmp, data, err, options, executed):
+    def _chain_file_module(self, conn, tmp, exec_rc, options):
+
         ''' handles changing file attribs after copy/template operations '''
 
-        old_changed = data.get('changed', False)
+        old_changed = exec_rc.result.get('changed', False)
         module = self._transfer_module(conn, tmp, 'file')
         args = ' '.join([ "%s=%s" % (k,v) for (k,v) in options.items() ])
-        (result2, err2, executed2) = self._execute_module(conn, tmp, module, args)
-        results2 = self._return_from_module(conn, conn.host, result2, err2, executed)
-        (host, ok, data2, err2) = results2
-        if ok:
-            new_changed = data2.get('changed', False)
-            data.update(data2)
-        else:
-            new_changed = False
+        exec_rc2 = self._execute_module(conn, tmp, module, args)
+
+        new_changed = False
+        if exec_rc2.is_successful():
+            new_changed = exec_rc2.result.get('changed', False)
+            exec_rc.result.update(exec_rc2.result)
+
         if old_changed or new_changed:
-            data['changed'] = True
-        return (host, ok, data, "%s%s"%(err,err2))
+            exec_rc.result['changed'] = True
+
+        return exec_rc
 
     # *****************************************************
 
-    def _execute_template(self, conn, host, tmp):
+    def _execute_template(self, conn, tmp):
         ''' handler for template operations '''
 
         # load up options
@@ -495,7 +530,8 @@ class Runner(object):
         dest     = options.get('dest', None)
         metadata = options.get('metadata', None)
         if (source is None and 'first_available_file' not in self.module_vars) or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
+            result = dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, comm_ok=False, result=result)
 
         # apply templating to source argument so vars can be used in the path
         inject = self.setup_cache.get(conn.host,{})
@@ -511,11 +547,16 @@ class Runner(object):
                     found = True
                     break
             if not found:
-                return (host, True, dict(failed=True, msg="could not find src in first_available_file list"), '')
+                result = dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(host=conn.host, comm_ok=False, result=result)
+
+
+        if self.module_vars is not None:
+            inject.update(self.module_vars)
 
         source = utils.template(source, inject, self.setup_cache)
 
-        (host, ok, data, err) = (None, None, None, None)
+        #(host, ok, data, err) = (None, None, None, None)
 
         if not self.is_playbook:
 
@@ -526,61 +567,76 @@ class Runner(object):
                     metadata = '/etc/ansible/setup'
                 else:
                     # path is expanded on remote side
-                    metadata = "~/.ansible/setup"
+                    metadata = "~/.ansible/tmp/setup" 
             
             # install the template module
             slurp_module = self._transfer_module(conn, tmp, 'slurp')
 
             # run the slurp module to get the metadata file
             args = "src=%s" % metadata
-            (result1, err, executed) = self._execute_module(conn, tmp, slurp_module, args)
-            result1 = utils.json_loads(result1)
-            if not 'content' in result1 or result1.get('encoding','base64') != 'base64':
-                result1['failed'] = True
-                return self._return_from_module(conn, host, result1, err, executed)
-            content = base64.b64decode(result1['content'])
+            result1  = self._execute_module(conn, tmp, slurp_module, args)
+            if not 'content' in result1.result or result1.result.get('encoding','base64') != 'base64':
+                result1.result['failed'] = True
+                return result1
+            content = base64.b64decode(result1.result['content'])
             inject = utils.json_loads(content)
 
+
         # install the template module
         copy_module = self._transfer_module(conn, tmp, 'copy')
 
         # template the source data locally
-        source_data = file(utils.path_dwim(self.basedir, source)).read()
-        resultant = ''            
         try:
-            resultant = utils.template(source_data, inject, self.setup_cache)
+            resultant = utils.template_from_file(self.basedir, source, inject, self.setup_cache)
         except Exception, e:
-            return (host, False, dict(failed=True, msg=str(e)), '')
+            result = dict(failed=True, msg=str(e))
+            return ReturnData(host=conn.host, comm_ok=False, result=result)
+
         xfered = self._transfer_str(conn, tmp, 'source', resultant)
             
         # run the COPY module
         args = "src=%s dest=%s" % (xfered, dest)
-        (result1, err, executed) = self._execute_module(conn, tmp, copy_module, args)
-        (host, ok, data, err) = self._return_from_module(conn, host, result1, err, executed)
+        exec_rc = self._execute_module(conn, tmp, copy_module, args)
  
         # modify file attribs if needed
-        if ok:
-            executed = executed.replace("copy","template",1)
-            return self._chain_file_module(conn, tmp, data, err, options, executed)
+        if exec_rc.comm_ok:
+            exec_rc.executed_str = exec_rc.executed_str.replace("copy","template",1)
+            return self._chain_file_module(conn, tmp, exec_rc, options)
         else:
-            return (host, ok, data, err)
+            return exec_rc
+
+    # *****************************************************
+
+    def _execute_assemble(self, conn, tmp):
+        ''' handler for assemble operations '''
+        module_name = 'assemble'
+        options = utils.parse_kv(self.module_args)
+        module = self._transfer_module(conn, tmp, module_name)
+        exec_rc = self._execute_module(conn, tmp, module, self.module_args)
+
+        if exec_rc.is_successful():
+            return self._chain_file_module(conn, tmp, exec_rc, options)
+        else:
+            return exec_rc
 
     # *****************************************************
 
     def _executor(self, host):
         try:
-            (host, ok, data, err) = self._executor_internal(host)
-            if not ok:
-                self.callbacks.on_unreachable(host, data)
-            return (host, ok, data)
+            exec_rc = self._executor_internal(host)
+            if type(exec_rc) != ReturnData:
+                raise Exception("unexpected return type: %s" % type(exec_rc))
+            if not exec_rc.comm_ok:
+                self.callbacks.on_unreachable(host, exec_rc.result)
+            return exec_rc
         except errors.AnsibleError, ae:
             msg = str(ae)
             self.callbacks.on_unreachable(host, msg)
-            return [host, False, msg]
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
         except Exception:
             msg = traceback.format_exc()
             self.callbacks.on_unreachable(host, msg)
-            return [host, False, msg]
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
 
     def _executor_internal(self, host):
         ''' callback executed in parallel for each host. returns (hostname, connected_ok, extra) '''
@@ -588,86 +644,93 @@ class Runner(object):
         host_variables = self.inventory.get_variables(host)
         port = host_variables.get('ansible_ssh_port', self.remote_port)
 
+        inject = self.setup_cache.get(host,{}).copy()
+        inject.update(host_variables)
+        inject.update(self.module_vars)
+
+        conditional = utils.template(self.conditional, inject, self.setup_cache)
+        if not eval(conditional):
+            result = utils.smjson(dict(skipped=True))
+            self.callbacks.on_skipped(host)
+            return ReturnData(host=host, result=result)
+
         conn = None
         try:
             conn = self.connector.connect(host, port)
         except errors.AnsibleConnectionFailed, e:
-            return [ host, False, "FAILED: %s" % str(e), None ]
+            result = dict(failed=True, msg="FAILED: %s" % str(e))
+            return ReturnData(host=host, comm_ok=False, result=result)
 
-        cache = self.setup_cache.get(host, {})
-        module_name = utils.template(self.module_name, cache, self.setup_cache)
+        module_name = utils.template(self.module_name, inject, self.setup_cache)
 
-        tmp = self._get_tmp_path(conn)
+        tmp = self._make_tmp_path(conn)
         result = None
 
         if self.module_name == 'copy':
-            result = self._execute_copy(conn, host, tmp)
+            result = self._execute_copy(conn, tmp)
         elif self.module_name == 'fetch':
-            result = self._execute_fetch(conn, host, tmp)
+            result = self._execute_fetch(conn, tmp)
         elif self.module_name == 'template':
-            result = self._execute_template(conn, host, tmp)
+            result = self._execute_template(conn, tmp)
+        elif self.module_name == 'raw':
+            result = self._execute_raw(conn, tmp)
+        elif self.module_name == 'assemble':
+            result = self._execute_assemble(conn, tmp)
         else:
             if self.background == 0:
-                result = self._execute_normal_module(conn, host, tmp, module_name)
+                result = self._execute_normal_module(conn, tmp, module_name)
             else:
-                result = self._execute_async_module(conn, host, tmp, module_name)
+                result = self._execute_async_module(conn, tmp, module_name)
 
         self._delete_remote_files(conn, tmp)
         conn.close()
 
-        (host, connect_ok, data, err) = result
-        if not connect_ok:
-            self.callbacks.on_unreachable(host, data)
+        if not result.comm_ok:
+            # connection or parsing errors...
+            self.callbacks.on_unreachable(host, result.result)
         else:
-            if 'failed' in data or 'rc' in data and str(data['rc']) != '0':
-                self.callbacks.on_failed(host, data)
-            elif 'skipped' in data:
-                self.callbacks.on_skipped(host)
+            data = result.result
+            if 'skipped' in data:
+                self.callbacks.on_skipped(result.host)
+            elif not result.is_successful():
+                self.callbacks.on_failed(result.host, result.result)
             else:
-                self.callbacks.on_ok(host, data)
-
-            if err:
-                if self.debug or data.get('parsed', True) == False:
-                    self.callbacks.on_error(host, err)
+                self.callbacks.on_ok(result.host, result.result)
 
         return result
 
     # *****************************************************
 
-    def _exec_command(self, conn, cmd, tmp, sudoable=False):
+    def _low_level_exec_command(self, conn, cmd, tmp, sudoable=False):
         ''' execute a command string over SSH, return the output '''
-
-        stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudoable=sudoable)
-        err=None
+        sudo_user = self.sudo_user
+        stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudo_user, sudoable=sudoable)
         out=None
-        if type(stderr) != str:
-            err="\n".join(stderr.readlines())
-        else:
-            err=stderr
+
         if type(stdout) != str:
             out="\n".join(stdout.readlines())
         else:
             out=stdout
-        return (out,err) 
 
+        # sudo mode paramiko doesn't capture stderr, so not relaying here either...
+        return out 
 
     # *****************************************************
 
-    def _get_tmp_path(self, conn):
-        ''' gets a temporary path on a remote box '''
+    def _make_tmp_path(self, conn):
+        ''' make and return a temporary path on a remote box '''
 
-        # The problem with this is that it's executed on the
-        # overlord, not on the target so we can't use tempdir and os.path
-        # Only support the *nix world for now by using the $HOME env var
+        basefile = 'ansible-%s-%s' % (time.time(), random.randint(0, 2**48))
+        basetmp = os.path.join(C.DEFAULT_REMOTE_TMP, basefile)
+        if self.remote_user == 'root':
+            basetmp = os.path.join('/var/tmp', basefile)
 
-        basetmp = "/var/tmp"
+        cmd = 'mkdir -p %s' % basetmp
         if self.remote_user != 'root':
-            basetmp = "$HOME/.ansible/tmp"
-        cmd = "mktemp -d %s/ansible.XXXXXX" % basetmp
-        if self.remote_user != 'root':
-            cmd = "mkdir -p %s && %s" % (basetmp, cmd)
+            cmd += ' && chmod a+x %s' % basetmp
+        cmd += ' && echo %s' % basetmp
 
-        result, err = self._exec_command(conn, cmd, None, sudoable=False)
+        result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
         cleaned = result.split("\n")[0].strip() + '/'
         return cleaned
 
@@ -679,12 +742,29 @@ class Runner(object):
 
         if module.startswith("/"):
             raise errors.AnsibleFileNotFound("%s is not a module" % module)
-        in_path = os.path.expanduser(os.path.join(self.module_path, module))
-        if not os.path.exists(in_path):
-            raise errors.AnsibleFileNotFound("module not found: %s" % in_path)
 
-        out_path = tmp + module
-        conn.put_file(in_path, out_path)
+        # Search module path(s) for named module.
+        for module_path in self.module_path.split(os.pathsep):
+            in_path = os.path.expanduser(os.path.join(module_path, module))
+            if os.path.exists(in_path):
+                break
+        else:
+            raise errors.AnsibleFileNotFound("module %s not found in %s" % (module, self.module_path))
+
+        out_path = os.path.join(tmp, module)
+
+        # use the correct python interpreter for the host
+        host_variables = self.inventory.get_variables(conn.host)
+        if 'ansible_python_interpreter' in host_variables:
+            interpreter = host_variables['ansible_python_interpreter']
+            with open(in_path) as f:
+                module_lines = f.readlines()
+            if '#!' and 'python' in module_lines[0]:
+                module_lines[0] = "#!%s" % interpreter
+            self._transfer_str(conn, tmp, module, '\n'.join(module_lines))
+        else:
+            conn.put_file(in_path, out_path)
+
         return out_path
 
     # *****************************************************
@@ -728,11 +808,13 @@ class Runner(object):
             return None
 
         for result in results:
-            (host, contacted_ok, result) = result
-            if contacted_ok:
-                results2["contacted"][host] = result
+            host = result.host
+            if host is None:
+                raise Exception("internal error, host not set")
+            if result.communicated_ok():
+                results2["contacted"][host] = result.result
             else:
-                results2["dark"][host] = result
+                results2["dark"][host] = result.result
 
         # hosts which were contacted but never got a chance to return
         for host in self.inventory.list_hosts(self.pattern):
@@ -760,89 +842,10 @@ class Runner(object):
             results = [ self._executor(h[1]) for h in hosts ]
         return self._partition_results(results)
 
-    def runAsync(self, time_limit):
+    def run_async(self, time_limit):
         ''' Run this module asynchronously and return a poller. '''
         self.background = time_limit
         results = self.run()
 
-        return results, AsyncPoller(results, self)
+        return results, poller.AsyncPoller(results, self)
 
-class AsyncPoller(object):
-    """ Manage asynchronous jobs. """
-
-    def __init__(self, results, runner):
-        self.runner = runner
-
-        self.results = { 'contacted': {}, 'dark': {}}
-        self.hosts_to_poll = []
-        self.completed = False
-
-        # Get job id and which hosts to poll again in the future
-        jid = None
-        for (host, res) in results['contacted'].iteritems():
-            if res.get('started', False):
-                self.hosts_to_poll.append(host)
-                jid = res.get('ansible_job_id', None)
-            else:
-                self.results['contacted'][host] = res
-        for (host, res) in results['dark'].iteritems():
-            self.results['dark'][host] = res
-
-        if jid is None:
-            raise errors.AnsibleError("unexpected error: unable to determine jid")
-        if len(self.hosts_to_poll)==0:
-            raise errors.AnsibleErrot("unexpected error: no hosts to poll")
-        self.jid = jid
-
-    def poll(self):
-        """ Poll the job status.
-
-            Returns the changes in this iteration."""
-        self.runner.module_name = 'async_status'
-        self.runner.module_args = "jid=%s" % self.jid
-        self.runner.pattern = "*"
-        self.runner.background = 0
-
-        self.runner.inventory.restrict_to(self.hosts_to_poll)
-        results = self.runner.run()
-        self.runner.inventory.lift_restriction()
-
-        hosts = []
-        poll_results = { 'contacted': {}, 'dark': {}, 'polled': {}}
-        for (host, res) in results['contacted'].iteritems():
-            if res.get('started',False):
-                hosts.append(host)
-                poll_results['polled'][host] = res
-            else:
-                self.results['contacted'][host] = res
-                poll_results['contacted'][host] = res
-                if 'failed' in res:
-                    self.runner.callbacks.on_async_failed(host, res, self.jid)
-                else:
-                    self.runner.callbacks.on_async_ok(host, res, self.jid)
-        for (host, res) in results['dark'].iteritems():
-            self.results['dark'][host] = res
-            poll_results['dark'][host] = res
-            self.runner.callbacks.on_async_failed(host, res, self.jid)
-
-        self.hosts_to_poll = hosts
-        if len(hosts)==0:
-            self.completed = True
-
-        return poll_results
-
-    def wait(self, seconds, poll_interval):
-        """ Wait a certain time for job completion, check status every poll_interval. """
-        clock = seconds - poll_interval
-        while (clock >= 0 and not self.completed):
-            time.sleep(poll_interval)
-
-            poll_results = self.poll()
-
-            for (host, res) in poll_results['polled'].iteritems():
-                if res.get('started'):
-                    self.runner.callbacks.on_async_poll(host, res, self.jid, clock)
-
-            clock = clock - poll_interval
-
-        return self.results

lib/ansible/runner/connection/__init__.py

@@ -0,0 +1,54 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+################################################
+
+import warnings
+import traceback
+import os
+import time
+import re
+import shutil
+import subprocess
+import pipes
+import socket
+import random
+
+import local
+import paramiko_ssh
+import ssh
+
+class Connection(object):
+    ''' Handles abstract connections to remote hosts '''
+
+    def __init__(self, runner, transport,sudo_user):
+        self.runner = runner
+        self.transport = transport
+        self.sudo_user = sudo_user
+    def connect(self, host, port=None):
+        conn = None
+        if self.transport == 'local':
+            conn = local.LocalConnection(self.runner, host)
+        elif self.transport == 'paramiko':
+            conn = paramiko_ssh.ParamikoConnection(self.runner, host, port)
+        elif self.transport == 'ssh':
+            conn = ssh.SSHConnection(self.runner, host, port)
+        if conn is None:
+            raise Exception("unsupported connection type")
+        return conn.connect()
+

lib/ansible/runner/connection/local.py

@@ -0,0 +1,81 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+################################################
+
+import warnings
+import traceback
+import os
+import time
+import re
+import shutil
+import subprocess
+import pipes
+import socket
+import random
+
+from ansible import errors
+
+class LocalConnection(object):
+    ''' Local based connections '''
+
+    def __init__(self, runner, host):
+        self.runner = runner
+        self.host = host
+
+    def connect(self, port=None):
+        ''' connect to the local host; nothing to do here '''
+
+        return self
+
+    def exec_command(self, cmd, tmp_path,sudo_user,sudoable=False):
+        ''' run a command on the local host '''
+        if self.runner.sudo and sudoable:
+            cmd = "sudo -s %s" % cmd
+        if self.runner.sudo_pass:
+            # NOTE: if someone wants to add sudo w/ password to the local connection type, they are welcome
+            # to do so.  The primary usage of the local connection is for crontab and kickstart usage however
+            # so this doesn't seem to be a huge priority
+            raise errors.AnsibleError("sudo with password is presently only supported on the paramiko (SSH) connection type")
+
+        p = subprocess.Popen(cmd, shell=True, stdin=None,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        return ("", stdout, stderr)
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to local '''
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        try:
+            shutil.copyfile(in_path, out_path)
+        except shutil.Error:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to copy: %s and %s are the same" % (in_path, out_path))
+        except IOError:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
+
+    def fetch_file(self, in_path, out_path):
+        ''' fetch a file from local to local -- for copatibility '''
+        self.put_file(in_path, out_path)
+
+    def close(self):
+        ''' terminate the connection; nothing to do here '''
+
+        pass

lib/ansible/runner/connection/paramiko_ssh.py

@@ -36,34 +36,6 @@ with warnings.catch_warnings():
     warnings.simplefilter("ignore")
     import paramiko
 
-
-################################################
-
-class Connection(object):
-    ''' Handles abstract connections to remote hosts '''
-
-    _LOCALHOSTRE = re.compile(r"^(127.0.0.1|localhost|%s)$" % os.uname()[1])
-
-    def __init__(self, runner, transport):
-        self.runner = runner
-        self.transport = transport
-
-    def connect(self, host, port=None):
-        conn = None
-        if self.transport == 'local' and self._LOCALHOSTRE.search(host):
-            conn = LocalConnection(self.runner, host)
-        elif self.transport == 'paramiko':
-            conn = ParamikoConnection(self.runner, host, port)
-        if conn is None:
-            raise Exception("unsupported connection type")
-        return conn.connect()
-
-################################################
-# want to implement another connection type?
-# follow duck-typing of ParamikoConnection
-# you may wish to read config files in __init__
-# if you have any.  Paramiko does not need any.
-
 class ParamikoConnection(object):
     ''' SSH based connections with Paramiko '''
 
@@ -76,27 +48,7 @@ class ParamikoConnection(object):
             self.port = self.runner.remote_port
 
     def _get_conn(self):
-        credentials = {}
         user = self.runner.remote_user
-        keypair = None
-
-        # Read file ~/.ssh/config, get data hostname, keyfile, port, etc
-        # This will *NOT* overrides the ansible username and hostname " , getting the port and keyfile only.
-	
-        try:
-            ssh_config = paramiko.SSHConfig()
-            config_file = ('~/.ssh/config')
-            if  os.path.exists(os.path.expanduser(config_file)):
-                ssh_config.parse(open(os.path.expanduser(config_file)))
-                credentials = ssh_config.lookup(self.host)
-
-        except IOError,e:
-                raise errors.AnsibleConnectionFailed(str(e))
-
-        if 'port' in credentials:
-            self.port = int(credentials['port'])
-        if 'identityfile' in credentials:
-            keypair = os.path.expanduser(credentials['identityfile'])
 
         ssh = paramiko.SSHClient()
         ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
@@ -107,16 +59,21 @@ class ParamikoConnection(object):
                 username=user,
                 allow_agent=True,
                 look_for_keys=True,
+                key_filename=self.runner.private_key_file,
                 password=self.runner.remote_pass,
-                key_filename=keypair,
                 timeout=self.runner.timeout,
                 port=self.port
             )
         except Exception, e:
-            if str(e).find("PID check failed") != -1:
+            msg = str(e)
+            if "PID check failed" in msg:
                 raise errors.AnsibleError("paramiko version issue, please upgrade paramiko on the machine running ansible")
+            elif "Private key file is encrypted" in msg:
+                msg = 'ssh %s@%s:%s : %s\nTo connect as a different user, use -u <username>.' % (
+                    user, self.host, self.port, msg)
+                raise errors.AnsibleConnectionFailed(msg)
             else:
-                raise errors.AnsibleConnectionFailed(str(e))
+                raise errors.AnsibleConnectionFailed(msg)
 
         return ssh
 
@@ -126,20 +83,12 @@ class ParamikoConnection(object):
         self.ssh = self._get_conn()
         return self
 
-    def exec_command(self, cmd, tmp_path, sudoable=False):
+    def exec_command(self, cmd, tmp_path,sudo_user,sudoable=False):
 
         ''' run a command on the remote host '''
-
         bufsize = 4096
-        # reusing runner's TCP connect timeout as command progress timeout (correct?)
-        timeout_secs = self.runner.timeout 
         chan = self.ssh.get_transport().open_session()
-        chan.settimeout(timeout_secs)
-        stdin = chan.makefile('wb', bufsize) 
-        stdout = chan.makefile('rb', bufsize)
-        stderr = chan.makefile_stderr('rb', bufsize) 
         chan.get_pty() 
-        chan.set_combine_stderr(False)    
 
         if not self.runner.sudo or not sudoable:
             quoted_command = '"$SHELL" -c ' + pipes.quote(cmd) 
@@ -150,11 +99,13 @@ class ParamikoConnection(object):
             # tells sudo that this is the end of sudo options and the command
             # follows.  Passing a quoted compound command to sudo (or sudo -s)
             # directly doesn't work, so we shellquote it with pipes.quote() 
-            # and pass the quoted string to the user's shell.
-
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
             randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
             prompt = '[sudo via ansible, key=%s] password: ' % randbits
-            sudocmd = 'sudo -k -p "%s" -- "$SHELL" -c %s' % (prompt, pipes.quote(cmd))
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s -- "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
             sudo_output = ''
             try:
                 chan.exec_command(sudocmd)
@@ -170,7 +121,7 @@ class ParamikoConnection(object):
 
         stdin = chan.makefile('wb', bufsize)
         stdout = chan.makefile('rb', bufsize)
-        stderr = chan.makefile_stderr('rb', bufsize)
+        stderr = ''  # stderr goes to stdout when using a pty, so this will never output anything.
         return stdin, stdout, stderr
 
     def put_file(self, in_path, out_path):
@@ -199,54 +150,3 @@ class ParamikoConnection(object):
 
         self.ssh.close()
 
-############################################
-# add other connection types here
-
-class LocalConnection(object):
-    ''' Local based connections '''
-
-    def __init__(self, runner, host):
-        self.runner = runner
-        self.host = host
-
-    def connect(self, port=None):
-        ''' connect to the local host; nothing to do here '''
-
-        return self
-
-    def exec_command(self, cmd, tmp_path, sudoable=False):
-        ''' run a command on the local host '''
-        if self.runner.sudo and sudoable:
-            cmd = "sudo -s %s" % cmd
-        if self.runner.sudo_pass:
-            # NOTE: if someone wants to add sudo w/ password to the local connection type, they are welcome
-            # to do so.  The primary usage of the local connection is for crontab and kickstart usage however
-            # so this doesn't seem to be a huge priority
-            raise errors.AnsibleError("sudo with password is presently only supported on the paramiko (SSH) connection type")
-
-        p = subprocess.Popen(cmd, shell=True, stdin=None,
-                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        stdout, stderr = p.communicate()
-        return ("", stdout, stderr)
-
-    def put_file(self, in_path, out_path):
-        ''' transfer a file from local to local '''
-        if not os.path.exists(in_path):
-            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
-        try:
-            shutil.copyfile(in_path, out_path)
-        except shutil.Error:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to copy: %s and %s are the same" % (in_path, out_path))
-        except IOError:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
-
-    def fetch_file(self, in_path, out_path):
-        ''' fetch a file from local to local -- for copatibility '''
-        self.put_file(in_path, out_path)
-
-    def close(self):
-        ''' terminate the connection; nothing to do here '''
-
-        pass

lib/ansible/runner/connection/ssh.py

@@ -0,0 +1,140 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+################################################
+
+import os
+import time
+import subprocess
+import shlex
+import pipes
+import random
+import select
+import fcntl
+
+from ansible import errors
+
+class SSHConnection(object):
+    ''' ssh based connections '''
+
+    def __init__(self, runner, host, port):
+        self.runner = runner
+        self.host = host
+        self.port = port
+
+    def connect(self):
+        ''' connect to the remote host '''
+        self.common_args = []
+        extra_args = os.getenv("ANSIBLE_SSH_ARGS", None)
+        if extra_args is not None:
+            self.common_args += shlex.split(extra_args)
+        else:
+            self.common_args += ["-o", "ControlMaster=auto",
+                                 "-o", "ControlPersist=60s",
+                                 "-o", "ControlPath=/tmp/ansible-ssh-%h-%p-%r"]
+        self.common_args += ["-o", "StrictHostKeyChecking=no"]
+        if self.port is not None:
+            self.common_args += ["-o", "Port=%d" % (self.port)]
+        if self.runner.private_key_file is not None:
+            self.common_args += ["-o", "IdentityFile="+self.runner.private_key_file]
+        self.common_args += ["-o", "User="+self.runner.remote_user]
+
+        return self
+
+    def exec_command(self, cmd, tmp_path,sudo_user,sudoable=False):
+        ''' run a command on the remote host '''
+
+        ssh_cmd = ["ssh", "-tt", "-q"] + self.common_args + [self.host]
+        if self.runner.sudo and sudoable:
+            # Rather than detect if sudo wants a password this time, -k makes 
+            # sudo always ask for a password if one is required. The "--"
+            # tells sudo that this is the end of sudo options and the command
+            # follows.  Passing a quoted compound command to sudo (or sudo -s)
+            # directly doesn't work, so we shellquote it with pipes.quote() 
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
+            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
+            prompt = '[sudo via ansible, key=%s] password: ' % randbits
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s -- "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
+            sudo_output = ''
+            ssh_cmd.append(sudocmd)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            if self.runner.sudo_pass:
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL,
+                            fcntl.fcntl(p.stdout, fcntl.F_GETFL) | os.O_NONBLOCK)
+                while not sudo_output.endswith(prompt):
+                    rfd, wfd, efd = select.select([p.stdout], [],
+                                                  [p.stdout], self.runner.timeout)
+                    if p.stdout in rfd:
+                        chunk = p.stdout.read()
+                        if not chunk:
+                            raise errors.AnsibleError('ssh connection closed waiting for sudo password prompt')
+                        sudo_output += chunk
+                    else:
+                        stdout = p.communicate()
+                        raise errors.AnsibleError('ssh connection error waiting for sudo password prompt')
+                p.stdin.write(self.runner.sudo_pass + '\n')
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) & ~os.O_NONBLOCK)
+        else:
+            ssh_cmd.append(cmd)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+        # We can't use p.communicate here because the ControlMaster may have stdout open as well
+        p.stdin.close()
+        stdout = ''
+        while p.poll() is None:
+            rfd, wfd, efd = select.select([p.stdout], [], [p.stdout], 1)
+            if p.stdout in rfd:
+                stdout += os.read(p.stdout.fileno(), 1024)
+        # older versions of ssh generate this error which we ignore
+        stdout=stdout.replace("tcgetattr: Invalid argument\n", "")
+
+        if p.returncode != 0 and stdout.find('Bad configuration option: ControlPersist') != -1:
+            raise errors.AnsibleError('using -c ssh on certain older ssh versions may not support ControlPersist, set ANSIBLE_SSH_ARGS="" before running again')
+            
+        return ('', stdout, '')
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to remote '''
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("put %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file to %s:\n%s\n%s" % (out_path, stdout, stderr))
+
+    def fetch_file(self, in_path, out_path):
+        ''' fetch a file from remote to local '''
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("get %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file from %s:\n%s\n%s" % (in_path, stdout, stderr))
+
+    def close(self):
+        ''' terminate the connection '''
+        pass

lib/ansible/runner/poller.py

@@ -0,0 +1,101 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import time
+
+from ansible import errors
+
+class AsyncPoller(object):
+    """ Manage asynchronous jobs. """
+
+    def __init__(self, results, runner):
+        self.runner = runner
+
+        self.results = { 'contacted': {}, 'dark': {}}
+        self.hosts_to_poll = []
+        self.completed = False
+
+        # Get job id and which hosts to poll again in the future
+        jid = None
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started', False):
+                self.hosts_to_poll.append(host)
+                jid = res.get('ansible_job_id', None)
+            else:
+                self.results['contacted'][host] = res
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+
+        if jid is None:
+            raise errors.AnsibleError("unexpected error: unable to determine jid")
+        if len(self.hosts_to_poll)==0:
+            raise errors.AnsibleErrot("unexpected error: no hosts to poll")
+        self.jid = jid
+
+    def poll(self):
+        """ Poll the job status.
+
+            Returns the changes in this iteration."""
+        self.runner.module_name = 'async_status'
+        self.runner.module_args = "jid=%s" % self.jid
+        self.runner.pattern = "*"
+        self.runner.background = 0
+
+        self.runner.inventory.restrict_to(self.hosts_to_poll)
+        results = self.runner.run()
+        self.runner.inventory.lift_restriction()
+
+        hosts = []
+        poll_results = { 'contacted': {}, 'dark': {}, 'polled': {}}
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started',False):
+                hosts.append(host)
+                poll_results['polled'][host] = res
+            else:
+                self.results['contacted'][host] = res
+                poll_results['contacted'][host] = res
+                if 'failed' in res:
+                    self.runner.callbacks.on_async_failed(host, res, self.jid)
+                else:
+                    self.runner.callbacks.on_async_ok(host, res, self.jid)
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+            poll_results['dark'][host] = res
+            self.runner.callbacks.on_async_failed(host, res, self.jid)
+
+        self.hosts_to_poll = hosts
+        if len(hosts)==0:
+            self.completed = True
+
+        return poll_results
+
+    def wait(self, seconds, poll_interval):
+        """ Wait a certain time for job completion, check status every poll_interval. """
+        clock = seconds - poll_interval
+        while (clock >= 0 and not self.completed):
+            time.sleep(poll_interval)
+
+            poll_results = self.poll()
+
+            for (host, res) in poll_results['polled'].iteritems():
+                if res.get('started'):
+                    self.runner.callbacks.on_async_poll(host, res, self.jid, clock)
+
+            clock = clock - poll_interval
+
+        return self.results

lib/ansible/utils.py

@@ -21,6 +21,7 @@ import sys
 import os
 import shlex
 import re
+import codecs
 import jinja2
 import yaml
 import optparse
@@ -48,25 +49,19 @@ def exit(msg, rc=1):
 
 def bigjson(result):
     ''' format JSON output (uncompressed) '''
-    # hide some internals magic from command line userland
     result2 = result.copy()
-    if 'invocation' in result2:
-        del result2['invocation']
     return json.dumps(result2, sort_keys=True, indent=4)
 
 def smjson(result):
     ''' format JSON output (compressed) '''
-    # hide some internals magic from command line userland
     result2 = result.copy()
-    if 'invocation' in result2:
-        del result2['invocation']
     return json.dumps(result2, sort_keys=True)
 
 def task_start_msg(name, conditional):
     if conditional:
-        return "\nNOTIFIED: [%s] **********\n" % name
+        return "NOTIFIED: [%s]" % name
     else:
-        return "\nTASK: [%s] *********\n" % name
+        return "TASK: [%s]" % name
 
 def regular_generic_msg(hostname, result, oneline, caption):
     ''' output on the result of a module run that is not command '''
@@ -137,7 +132,7 @@ def host_report_msg(hostname, module_name, result, oneline):
     ''' summarize the JSON results for a particular host '''
     buf = ''
     failed = is_failed(result)
-    if module_name in [ 'command', 'shell' ] and 'ansible_job_id' not in result:
+    if module_name in [ 'command', 'shell', 'raw' ] and 'ansible_job_id' not in result:
         if not failed:
             buf = command_success_msg(hostname, result, oneline)
         else:
@@ -181,7 +176,12 @@ def parse_json(data):
         # not JSON, but try "Baby JSON" which allows many of our modules to not
         # require JSON and makes writing modules in bash much simpler
         results = {}
-        tokens = shlex.split(data)
+        try :
+            tokens = shlex.split(data)
+        except: 
+            print "failed to parse json: "+ data
+            raise; 
+            
         for t in tokens:
             if t.find("=") == -1:
                 raise errors.AnsibleError("failed to parse: %s" % data)
@@ -198,7 +198,35 @@ def parse_json(data):
             return { "failed" : True, "parsed" : False, "msg" : data }
         return results
 
-_KEYCRE = re.compile(r"\$(\w+)")
+_LISTRE = re.compile(r"(\w+)\[(\d+)\]")
+
+def _varLookup(name, vars):
+    ''' find the contents of a possibly complex variable in vars. '''
+    path = name.split('.')
+    space = vars
+    for part in path:
+        if part in space:
+            space = space[part]
+        elif "[" in part:
+            m = _LISTRE.search(part)
+            if not m:
+                return
+            try:
+                space = space[m.group(1)][int(m.group(2))]
+            except (KeyError, IndexError):
+                return
+        else:
+            return
+    return space
+
+_KEYCRE = re.compile(r"\$(?P<complex>\{){0,1}((?(complex)[\w\.\[\]]+|\w+))(?(complex)\})")
+#                        if { -> complex     if complex, allow . and need trailing }
+
+def varLookup(varname, vars):
+    m = _KEYCRE.search(varname)
+    if not m:
+        return None
+    return _varLookup(m.group(2), vars)
 
 def varReplace(raw, vars):
     '''Perform variable replacement of $vars
@@ -220,8 +248,9 @@ def varReplace(raw, vars):
 
         # Determine replacement value (if unknown variable then preserve
         # original)
-        varname = m.group(1).lower()
-        replacement = str(vars.get(varname, m.group()))
+        varname = m.group(2)
+
+        replacement = unicode(_varLookup(varname, vars) or m.group())
 
         start, end = m.span()
         done.append(raw[:start])    # Keep stuff leading up to token
@@ -230,21 +259,33 @@ def varReplace(raw, vars):
 
     return ''.join(done)
 
-def template(text, vars, setup_cache):
+def _template(text, vars, setup_cache=None):
     ''' run a text buffer through the templating engine '''
     vars = vars.copy()
-    text = varReplace(str(text), vars)
     vars['hostvars'] = setup_cache
-    template = jinja2.Template(text)
-    return template.render(vars)
+    text = varReplace(unicode(text), vars)
+    return text
 
-def double_template(text, vars, setup_cache):
-    return template(template(text, vars, setup_cache), vars, setup_cache)
+def template(text, vars, setup_cache=None):
+    ''' run a text buffer through the templating engine 
+        until it no longer changes '''
+    prev_text = ''
+    while prev_text != text:
+        prev_text = text
+        text = _template(text, vars, setup_cache)
+    return text
 
-def template_from_file(path, vars, setup_cache):
+def template_from_file(basedir, path, vars, setup_cache):
     ''' run a file through the templating engine '''
-    data = file(path).read()
-    return template(data, vars, setup_cache)
+    environment = jinja2.Environment(loader=jinja2.FileSystemLoader(basedir), trim_blocks=False)
+    data = codecs.open(path_dwim(basedir, path), encoding="utf8").read()
+    t = environment.from_string(data)
+    vars = vars.copy()
+    vars['hostvars'] = setup_cache
+    res = t.render(vars)
+    if data.endswith('\n') and not res.endswith('\n'):
+        res = res + '\n'
+    return template(res, vars, setup_cache)
 
 def parse_yaml(data):
     return yaml.load(data)
@@ -263,7 +304,7 @@ def parse_kv(args):
         vargs = shlex.split(args, posix=True)
         for x in vargs:
             if x.find("=") != -1:
-                k, v = x.split("=")
+                k, v = x.split("=", 1)
                 options[k]=v
     return options
 
@@ -277,8 +318,8 @@ def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, asyn
     ''' create an options parser for any ansible script '''
 
     parser = SortedOptParser(usage)
-    parser.add_option('-D','--debug', default=False, action="store_true",
-        help='debug mode')
+    parser.add_option('-v','--verbose', default=False, action="store_true",
+        help='verbose mode')
     parser.add_option('-f','--forks', dest='forks', default=constants.DEFAULT_FORKS, type='int',
         help="specify number of parallel processes to use (default=%s)" % constants.DEFAULT_FORKS)
     parser.add_option('-i', '--inventory-file', dest='inventory',
@@ -286,10 +327,12 @@ def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, asyn
         default=constants.DEFAULT_HOST_LIST)
     parser.add_option('-k', '--ask-pass', default=False, dest='ask_pass', action='store_true',
         help='ask for SSH password')
+    parser.add_option('--private-key', default=None, dest='private_key_file',
+        help='use this file to authenticate the connection')
     parser.add_option('-K', '--ask-sudo-pass', default=False, dest='ask_sudo_pass', action='store_true',
         help='ask for sudo password')
     parser.add_option('-M', '--module-path', dest='module_path',
-        help="specify path to module library (default=%s)" % constants.DEFAULT_MODULE_PATH, 
+        help="specify path(s) to module library (default=%s)" % constants.DEFAULT_MODULE_PATH,
         default=constants.DEFAULT_MODULE_PATH)
     parser.add_option('-T', '--timeout', default=constants.DEFAULT_TIMEOUT, type='int',
         dest='timeout', 
@@ -304,6 +347,8 @@ def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, asyn
     if runas_opts:
         parser.add_option("-s", "--sudo", default=False, action="store_true",
             dest='sudo', help="run operations with sudo (nopasswd)")
+        parser.add_option('-U', '--sudo-user', dest='sudo_user', help='desired sudo user (default=root)',
+            default=None)   # Can't default to root because we need to detect when this option was given
         parser.add_option('-u', '--user', default=constants.DEFAULT_REMOTE_USER,
             dest='remote_user', 
             help='connect as this user (default=%s)' % constants.DEFAULT_REMOTE_USER)

library/apt

@@ -25,6 +25,7 @@ import os
 import sys
 import shlex
 import subprocess
+import syslog
 import traceback
 
 APT_PATH = "/usr/bin/apt-get"
@@ -73,15 +74,23 @@ def package_status(pkgname, version, cache):
     except KeyError:
         fail_json(msg="No package matching '%s' is available" % pkgname)
     if version:
-        return pkg.is_installed and pkg.installed.version == version, False
+        try :
+            return pkg.is_installed and pkg.installed.version == version, False
+        except AttributeError: 
+            #assume older version of python-apt is installed
+            return pkg.isInstalled and pkg.installedVersion == version, False
     else:
-        return pkg.is_installed, pkg.is_upgradable
+        try :
+            return pkg.is_installed, pkg.is_upgradable
+        except AttributeError: 
+            #assume older version of python-apt is installed
+            return pkg.isInstalled, pkg.isUpgradable
 
 def install(pkgspec, cache, upgrade=False, default_release=None):
     name, version = package_split(pkgspec)
     installed, upgradable = package_status(name, version, cache)
     if not installed or (upgrade and upgradable):
-        cmd = "%s -q -y install '%s'" % (APT, pkgspec)
+        cmd = "%s --option Dpkg::Options::=--force-confold -q -y install '%s'" % (APT, pkgspec)
         if default_release:
             cmd += " -t '%s'" % (default_release,)
         rc, out, err = run_apt(cmd)
@@ -113,6 +122,8 @@ if not os.path.exists(APT_PATH):
 argfile = sys.argv[1]
 args    = open(argfile, 'r').read()
 items   = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
     fail_json(msg='the module requires arguments -a')

library/assemble

@@ -0,0 +1,123 @@
+#!/usr/bin/python
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+import os
+import os.path
+import sys
+import shlex
+import shutil
+import syslog
+import tempfile
+try:
+    import hashlib
+    HAVE_HASHLIB=True
+except ImportError:
+    import md5
+    HAVE_HASHLIB=False
+
+# Since hashlib is only available in 2.5 and onwards, this module
+# uses md5 which is available in 2.4.
+
+# ===========================================
+# Support methods
+
+def exit_json(rc=0, **kwargs):
+   print json.dumps(kwargs)
+   sys.exit(rc)
+
+def fail_json(**kwargs):
+   kwargs['failed'] = True
+   exit_json(rc=1, **kwargs)
+
+def assemble_from_fragments(path):
+    ''' assemble a file from a directory of fragments '''
+    assembled = []
+    for f in sorted(os.listdir(path)):
+        fragment = "%s/%s" % (path, f)
+        if os.path.isfile(fragment):
+            assembled.append(file(fragment).read())
+    return "".join(assembled)
+
+def write_temp_file(data):
+    fd, path = tempfile.mkstemp()
+    os.write(fd, data)
+    os.close(fd)
+    return path
+
+def file_digest(path):
+    if HAVE_HASHLIB:
+        digest = hashlib.md5(file(path).read()).hexdigest()
+    else:
+        digest = md5.new(file(path).read()).hexdigest()
+    return digest
+
+# ===========================================
+
+if len(sys.argv) == 1:
+   fail_json(msg="the assemble module requires arguments (-a)")
+
+argfile = sys.argv[1]
+if not os.path.exists(argfile):
+   fail_json(msg="Argument file not found")
+
+args = open(argfile, 'r').read()
+items = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
+
+if not len(items):
+   fail_json(msg="the assemble module requires arguments (-a)")
+
+params = {}
+for x in items:
+    (k, v) = x.split("=")
+    params[k] = v
+
+changed = False
+pathmd5 = None
+destmd5 = None
+src  = params.get('src', None)
+dest = params.get('dest', None)
+
+if src:
+    src = os.path.expanduser(src)
+if dest:
+    dest = os.path.expanduser(dest)
+
+if not os.path.exists(src):
+    fail_json(msg="Source (%s) does not exist" % src)
+
+if not os.path.isdir(src):
+    fail_json(msg="Source (%s) is not a directory" % src)
+
+path = write_temp_file(assemble_from_fragments(src))
+pathmd5 = file_digest(path)
+
+if os.path.exists(dest):
+    destmd5 = file_digest(dest)
+
+if pathmd5 != destmd5:
+    shutil.copy(path, dest)
+    changed = True
+
+exit_json(md5sum=pathmd5, changed=changed)

library/async_status

@@ -28,13 +28,18 @@ import subprocess
 import sys
 import datetime
 import traceback
+import syslog
 
 # ===========================================
 
 # FIXME: better error handling
 
 argsfile = sys.argv[1]
-items = shlex.split(file(argsfile).read())
+args = open(argsfile, 'r').read()
+items = shlex.split(args)
+
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 params = {}
 for x in items:

library/async_wrapper

@@ -30,6 +30,7 @@ import datetime
 import traceback
 import signal
 import time
+import syslog
 
 def daemonize_self():
     # daemonizing code: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
@@ -76,6 +77,9 @@ wrapped_module = sys.argv[3]
 argsfile = sys.argv[4]
 cmd = "%s %s" % (wrapped_module, argsfile)
 
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % " ".join(sys.argv[1:]))
+
 # setup logging directory
 logdir = os.path.expanduser("~/.ansible_async")
 log_path = os.path.join(logdir, jid)

library/authorized_key

@@ -0,0 +1,201 @@
+#!/usr/bin/env python
+"""Ansible module to add authorized_keys for ssh logins.
+
+(c) 2012, Brad Olson <brado@movedbylight.com>
+
+Results: Makes sure the public key line is present or absent in the user's .ssh/authorized_keys.
+
+Arguments
+=========
+    user = username
+    key = line to add to authorized_keys for user
+    state = absent|present (default: present)
+
+Command Line Example 
+====================
+
+ansible somehost -m authorized_key -a user=charlie key="ssh-dss AAAABUfOL+8BTwaRYr/rycsBF1D8e5pTxEsXHQs4iq+mZdyWqlW++L6pMiam1A8yweP+rKtgjK2httVS6GigVsuWWfOd7/sdWippefq74nppVUELHPKkaIOjJNN1zUHFoL/YMwAAAEBALnAsQN10TNGsRDe5arBsW8cTOjqLyYBcIqgPYTZW8zENErFxt7ij3fW3Jh/sCpnmy8rkS7FyK8ULX0PEy/2yDx8/5rXgMIICbRH/XaBy9Ud5bRBFVkEDu/r+rXP33wFPHjWjwvHAtfci1NRBAudQI/98DbcGQw5HmE89CjgZRo5ktkC5yu/8agEPocVjdHyZr7PaHfxZGUDGKtGRL2QzRYukCmWo1cZbMBHcI5FzImvTHS9/8B3SATjXMPgbfBuEeBwuBK5EjL+CtHY5bWs9kmYjmeo0KfUMH8hY4MAXDoKhQ7DhBPIrcjS5jPtoGxIREZjba67r6/P2XKXaCZH6Fc= charlie@somemail.org 2011-01-17"
+
+Playbook Example 
+================
+
+---
+# include like this:
+# - include: tasks/logins.yaml users=charlie,sue
+- name: create user charlie
+  action: user name=charlie shell=/bin/bash createhome=yes groups=www-data 
+  only_if: "'charlie' in '$users'"
+- name: add public key for charlie
+  action: authorized_key user=charlie key="ssh-dss AAAABUfOL+8BTwaRYr/rycsBF1D8e5pTxEsXHQs4iq+mZdyWqlW++L6pMiam1A8yweP+rKtgjK2httVS6GigVsuWWfOd7/sdWippefq74nppVUELHPKkaIOjJNN1zUHFoL/YMwAAAEBALnAsQN10TNGsRDe5arBsW8cTOjqLyYBcIqgPYTZW8zENErFxt7ij3fW3Jh/sCpnmy8rkS7FyK8ULX0PEy/2yDx8/5rXgMIICbRH/XaBy9Ud5bRBFVkEDu/r+rXP33wFPHjWjwvHAtfci1NRBAudQI/98DbcGQw5HmE89CjgZRo5ktkC5yu/8agEPocVjdHyZr7PaHfxZGUDGKtGRL2QzRYukCmWo1cZbMBHcI5FzImvTHS9/8B3SATjXMPgbfBuEeBwuBK5EjL+CtHY5bWs9kmYjmeo0KfUMH8hY4MAXDoKhQ7DhBPIrcjS5jPtoGxIREZjba67r6/P2XKXaCZH6Fc= charlie@somemail.org 2011-01-17"
+  only_if: "'charlie' in '$users'"
+
+
+This file is part of Ansible
+
+Ansible is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+Ansible is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+
+import sys, os, shlex, pwd, syslog
+from os.path import expanduser, exists, isfile, join
+
+params = {}
+msg=""
+
+def exit_json(rc=0, **kwargs):
+    if 'name' in kwargs:
+        add_user_info(kwargs)
+    print json.dumps(kwargs)
+    sys.exit(rc)
+
+def fail_json(**kwargs):
+    kwargs['failed'] = True
+    exit_json(rc=1, **kwargs)
+
+def get_params():
+    """Startup tasks and read params.
+
+    :return: parameters as dictionary.
+    """
+    global msg
+
+    msg = "reading params"
+    argfile = sys.argv[1]
+    try:
+        f = open(argfile,"r")
+        args = f.read()
+    finally:
+        f.close()
+
+    msg = "writing syslog."
+    syslog.openlog('ansible-%s' % os.path.basename(__file__))
+    syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
+
+    msg = "parsing params"
+    params = dict(                            # make a dictionary of...
+        [ arg.split("=", 1)                 # assignment pairs 
+          for arg in shlex.split(args)  # using shell lexing
+          if "=" in arg                     # ignoring tokens without assignment
+        ])
+
+    return params
+
+def keyfile(user, write=False):
+    """Calculate name of authorized keys file, optionally creating the 
+    directories and file, properly setting permissions.
+
+    :param str user: name of user in passwd file
+    :param bool write: if True, write changes to authorized_keys file (creating directories if needed)
+    :return: full path string to authorized_keys for user
+    """
+
+    global msg
+    msg = "Reading system user entry."
+    user_entry = pwd.getpwnam(user)
+    msg = "Calculating special directories"
+    homedir = user_entry.pw_dir
+    sshdir = join(homedir, ".ssh")
+    keysfile = join(sshdir, "authorized_keys")
+    if not write: return keysfile
+
+    #create directories and files for authorized keys
+    msg = "Reading user and group info."
+    uid = user_entry.pw_uid
+    gid = user_entry.pw_gid
+    msg = "Making ~/.ssh."
+    if not exists(sshdir): os.mkdir(sshdir, 0700)
+    os.chown(sshdir, uid, gid)
+    os.chmod(sshdir, 0700)
+    msg = "Touching authorized keys file."
+    if not exists( keysfile):
+        try:
+            f = open(keysfile, "w") #touches file so we can set ownership and perms
+        finally:
+            f.close()
+    os.chown(keysfile, uid, gid)
+    os.chmod(keysfile, 0600)
+    return keysfile
+
+def readkeys( filename):
+    global msg
+    msg = "Reading authorized_keys."
+    if not isfile(filename): return []
+    try:
+        f = open(filename)
+        keys = [line.rstrip() for line in f.readlines()]
+    finally:
+        f.close()
+    return keys
+
+def writekeys( filename, keys):
+    global msg
+    msg = "Writing authorized_keys."
+    try:
+        f = open(filename,"w")
+        f.writelines( (key + "\n" for key in keys) )
+    finally:
+        f.close()
+
+def enforce_state( params):
+    """Add or remove key.
+
+    :return: True=changed, False=unchanged
+    """
+    global msg
+
+    #== scrub params
+    msg = "Invalid or missing param: user."
+    user = params["user"]
+    msg = "Invalid or missing param: key."
+    key = params["key"]
+    state = params.get("state", "present")
+
+    #== check current state
+    params["keyfile"] = keyfile(user, write=False) #just get the filename, don't create file
+    keys = readkeys( params["keyfile"])
+    present = key in keys
+
+    #== handle idempotent state=present
+    if state=="present":
+        if present: return False #nothing to do
+        keys.append(key)
+        writekeys(keyfile(user,write=True), keys)
+    elif state=="absent":
+        if not present: return False #nothing to do
+        keys.remove(key)
+        writekeys(keyfile(user,write=True), keys)
+    else:
+        msg = "Invalid param: state."
+        raise StandardError(msg)
+    return True
+
+#===== MAIN SCRIPT ===================================================
+
+try:
+    params = get_params()
+    changed = enforce_state( params)
+    msg = ""
+except:
+    msg = "Error %s" % msg
+
+# Don't do sys.exit() within try...except 
+if msg:
+    fail_json(msg=msg)
+else:
+    exit_json( user=params["user"], changed=changed)
+    

library/command

@@ -29,9 +29,12 @@ import datetime
 import traceback
 import shlex
 import os
+import syslog
 
 argfile = sys.argv[1]
 args = open(argfile, 'r').read()
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 shell = False
 

library/copy

@@ -21,18 +21,37 @@
 import sys
 import os
 import shlex
+import syslog
 
 # ===========================================
 # convert arguments of form a=b c=d
 # to a dictionary
 # FIXME: make more idiomatic
 
+def dump_kv(vars):
+    return " ".join("%s='%s'" % (k,v) for (k,v) in vars.items())
+
+def exit_kv(rc=0, **kwargs):
+    if 'path' in kwargs:
+        add_path_info(kwargs)
+    print dump_kv(kwargs)
+    sys.exit(rc)
+
+def md5_sum(f):
+    md5sum = os.popen("/usr/bin/md5sum %(file)s 2>/dev/null || /sbin/md5 -q %(file)s 2>/dev/null || /usr/bin/digest -a md5 -v %(file)s 2>/dev/null" % {"file": f}).read().split()[0]
+    return md5sum
+
 if len(sys.argv) == 1:
-    sys.exit(1)
+	exit_kv(rc=1, failed=1, msg="incorrect number of arguments given")
+
 argfile = sys.argv[1]
 if not os.path.exists(argfile):
-    sys.exit(1)
-items = shlex.split(open(argfile, 'r').read())
+	exit_kv(rc=1, failed=1, msg="file %s does not exist" % (argfile))
+
+args = open(argfile, 'r').read()
+items = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 
 params = {}
@@ -46,24 +65,35 @@ if src:
     src = os.path.expanduser(src)
 if dest:
     dest = os.path.expanduser(dest)
- 
+
+md5sum_src = None
 # raise an error if there is no src file
 if not os.path.exists(src):
-    print "failed=1 msg='Source %s failed to transfer'" % src
-    sys.exit(1)
+        exit_kv(rc=1, failed=1, msg="Source %s failed to transfer" % (src))
+if not os.access(src, os.R_OK):
+    exit_kv(rc=1, failed=1, msg="Source %s not readable" % (src))
+md5sum_src = md5_sum(src)
 
-md5sum = None
-changed = False
+md5sum_dest = None
+# check if there is no dest file
 if os.path.exists(dest):
-    md5sum = os.popen("md5sum %s" % dest).read().split()[0]
+    # raise an error if copy has no permission on dest
+    if not os.access(dest, os.W_OK):
+        exit_kv(rc=1, failed=1, msg="Destination %s not writable" % (dest))
+    if not os.access(dest, os.R_OK):
+        exit_kv(rc=1, failed=1, msg="Destination %s not readable" % (dest))
+    md5sum_dest = md5_sum(dest)
+else:
+    if not os.access(os.path.dirname(dest), os.W_OK):
+        exit_kv(rc=1, failed=1, msg="Destination %s not writable" % (dest))
 
-md5sum2 = os.popen("md5sum %s" % src).read().split()[0]
-
-if md5sum != md5sum2:
+if md5sum_src != md5sum_dest:
     os.system("cp %s %s" % (src, dest))
     changed = True
+else:
+    changed = False
 
 # mission accomplished
-print "md5sum=%s changed=%s" % (md5sum2, changed)
-
+#print "md5sum=%s changed=%s" % (md5sum_src, changed)
+exit_kv(dest=dest, src=src, changed="md5sum=%s changed=%s" % (md5sum_src, changed))
 

library/facter

@@ -22,4 +22,5 @@
 #    facter
 #    ruby-json
 
-/usr/bin/facter --json
+/usr/bin/logger -t ansible-facter Invoked as-is
+/usr/bin/facter --json 2>/dev/null

library/file

@@ -17,10 +17,6 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
 import os
 import sys
 import shlex
@@ -29,27 +25,25 @@ import shutil
 import stat
 import grp
 import pwd
+import syslog
 try:
     import selinux
     HAVE_SELINUX=True
 except ImportError:
     HAVE_SELINUX=False
 
-def debug(msg):
-    # ansible ignores stderr, so it's safe to use for debug
-    # print >>sys.stderr, msg
-    pass
+def dump_kv(vars):
+    return " ".join("%s='%s'" % (k,v) for (k,v) in vars.items())
 
-def exit_json(rc=0, **kwargs):
+def exit_kv(rc=0, **kwargs):
     if 'path' in kwargs:
-        debug("adding path info")
         add_path_info(kwargs)
-    print json.dumps(kwargs)
+    print dump_kv(kwargs)
     sys.exit(rc)
 
-def fail_json(**kwargs):
+def fail_kv(**kwargs):
     kwargs['failed'] = True
-    exit_json(rc=1, **kwargs)
+    exit_kv(rc=1, **kwargs)
 
 def add_path_info(kwargs):
     path = kwargs['path']
@@ -81,20 +75,16 @@ def selinux_mls_enabled():
     if not HAVE_SELINUX:
         return False
     if selinux.is_selinux_mls_enabled() == 1:
-        debug('selinux mls is enabled')
         return True
     else:
-        debug('selinux mls is disabled')
         return False
 
 def selinux_enabled():
     if not HAVE_SELINUX:
         return False
     if selinux.is_selinux_enabled() == 1:
-        debug('selinux is enabled')
         return True
     else:
-        debug('selinux is disabled')
         return False
 
 # Determine whether we need a placeholder for selevel/mls
@@ -112,13 +102,10 @@ def selinux_default_context(path, mode=0):
     try:
         ret = selinux.matchpathcon(path, mode)
     except OSError:
-        debug("no default context available")
         return context
     if ret[0] == -1:
-        debug("no default context available")
         return context
     context = ret[1].split(':')
-    debug("got default secontext=%s" % ret[1])
     return context
 
 def selinux_context(path):
@@ -128,11 +115,10 @@ def selinux_context(path):
     try:
         ret = selinux.lgetfilecon(path)
     except:
-        fail_json(path=path, msg='failed to retrieve selinux context')
+        fail_kv(path=path, msg='failed to retrieve selinux context')
     if ret[0] == -1:
         return context
     context = ret[1].split(':')
-    debug("got current secontext=%s" % ret[1])
     return context
 
 # ===========================================
@@ -140,9 +126,11 @@ def selinux_context(path):
 argfile = sys.argv[1]
 args    = open(argfile, 'r').read()
 items   = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
-    fail_json(msg='the module requires arguments -a')
+    fail_kv(msg='the module requires arguments -a')
     sys.exit(1)
 
 params = {}
@@ -170,41 +158,40 @@ seuser    = params.get('seuser', None)
 serole    = params.get('serole', None)
 setype    = params.get('setype', None)
 selevel   = params.get('serange', 's0')
-context   = params.get('context', None)
 secontext = [seuser, serole, setype]
 if selinux_mls_enabled():
     secontext.append(selevel)
 
-if context is not None:
-    if context != 'default':
-        fail_json(msg='invalid context: %s' % context)
-    if seuser is not None or serole is not None or setype is not None:
-        fail_json(msg='cannot define context=default and seuser, serole or setype')
-    secontext = selinux_default_context(path)
+default_secontext = selinux_default_context(path)
+for i in range(len(default_secontext)):
+    if i is not None and secontext[i] == '_default':
+        secontext[i] = default_secontext[i]
 
 if state not in [ 'file', 'directory', 'link', 'absent']:
-    fail_json(msg='invalid state: %s' % state)
+    fail_kv(msg='invalid state: %s' % state)
 
 if state == 'link' and (src is None or dest is None):
-    fail_json(msg='src and dest are required for "link" state')
+    fail_kv(msg='src and dest are required for "link" state')
 elif path is None:
-    fail_json(msg='path is required')
+    fail_kv(msg='path is required')
 
 changed = False
 
 # ===========================================
 # support functions
 
-def md5sum(filename):
-    return os.popen("/usr/bin/md5sum %s" % f).read()
-
 def user_and_group(filename):
     st = os.stat(filename)
     uid = st.st_uid
     gid = st.st_gid
-    user = pwd.getpwuid(uid)[0]
-    group = grp.getgrgid(gid)[0]
-    debug("got user=%s and group=%s" % (user, group))
+    try:
+        user = pwd.getpwuid(uid)[0]
+    except KeyError:
+        user = str(uid)
+    try:
+        group = grp.getgrgid(gid)[0]
+    except KeyError:
+        group = str(gid)
     return (user, group)
 
 def set_context_if_different(path, context, changed):
@@ -212,59 +199,52 @@ def set_context_if_different(path, context, changed):
         return changed
     cur_context = selinux_context(path)
     new_context = list(cur_context)
-    debug("current secontext is %s" % ':'.join(cur_context))
     # Iterate over the current context instead of the
     # argument context, which may have selevel.
     for i in range(len(cur_context)):
         if context[i] is not None and context[i] != cur_context[i]:
             new_context[i] = context[i]
-    debug("new secontext is %s" % ':'.join(new_context))
     if cur_context != new_context:
         try:
             rc = selinux.lsetfilecon(path, ':'.join(new_context))
         except OSError:
-            fail_json(path=path, msg='invalid selinux context')
+            fail_kv(path=path, msg='invalid selinux context')
         if rc != 0:
-            fail_json(path=path, msg='set selinux context failed')
+            fail_kv(path=path, msg='set selinux context failed')
         changed = True
     return changed
     
 def set_owner_if_different(path, owner, changed):
    if owner is None:
-       debug('not tweaking owner')
        return changed
    user, group = user_and_group(path)
    if owner != user:
-       debug('setting owner')
-       rc = os.system("/bin/chown -R %s %s" % (owner, path))
+       rc = os.system("/bin/chown -R %s %s 2>/dev/null" % (owner, path))
        if rc != 0:
-           fail_json(path=path, msg='chown failed')
+           fail_kv(path=path, msg='chown failed')
        return True
 
    return changed
     
 def set_group_if_different(path, group, changed):
    if group is None:
-       debug('not tweaking group')
        return changed
    old_user, old_group = user_and_group(path)
    if old_group != group:
-       debug('setting group')
        rc = os.system("/bin/chgrp -R %s %s" % (group, path))
        if rc != 0:
-           fail_json(path=path, msg='chgrp failed')
+           fail_kv(path=path, msg='chgrp failed')
        return True
    return changed
 
 def set_mode_if_different(path, mode, changed):
    if mode is None:
-       debug('not tweaking mode')
        return changed
    try:
        # FIXME: support English modes
        mode = int(mode, 8)
    except Exception, e:
-       fail_json(path=path, msg='mode needs to be something octalish', details=str(e))  
+       fail_kv(path=path, msg='mode needs to be something octalish', details=str(e))  
  
    st = os.stat(path)
    prev_mode = stat.S_IMODE(st[stat.ST_MODE])
@@ -273,10 +253,9 @@ def set_mode_if_different(path, mode, changed):
        # FIXME: comparison against string above will cause this to be executed
        # every time
        try:
-           debug('setting mode')
            os.chmod(path, mode)
        except Exception, e:
-           fail_json(path=path, msg='chmod failed', details=str(e))
+           fail_kv(path=path, msg='chmod failed', details=str(e))
 
        st = os.stat(path)
        new_mode = stat.S_IMODE(st[stat.ST_MODE])
@@ -287,13 +266,13 @@ def set_mode_if_different(path, mode, changed):
 
 
 def rmtree_error(func, path, exc_info):
-   fail_json(path=path, msg='failed to remove directory')
+   fail_kv(path=path, msg='failed to remove directory')
 
 # ===========================================
 # go...
 
 prev_state = 'absent'
-if os.path.exists(path):
+if os.path.lexists(path):
     if os.path.islink(path):
         prev_state = 'link'
     elif os.path.isfile(path):
@@ -302,7 +281,6 @@ if os.path.exists(path):
         prev_state = 'directory'
 
 if prev_state != 'absent' and state == 'absent':
-    debug('requesting absent')
     try:
         if prev_state == 'directory':
             if os.path.islink(path):
@@ -312,21 +290,20 @@ if prev_state != 'absent' and state == 'absent':
         else:
             os.unlink(path)
     except Exception, e:
-        fail_json(path=path, msg=str(e))
-    exit_json(path=path, changed=True)
+        fail_kv(path=path, msg=str(e))
+    exit_kv(path=path, changed=True)
     sys.exit(0)
 
 if prev_state != 'absent' and prev_state != state:
-    fail_json(path=path, msg='refusing to convert between %s and %s' % (prev_state, state))
+    fail_kv(path=path, msg='refusing to convert between %s and %s' % (prev_state, state))
 
 if prev_state == 'absent' and state == 'absent':
-    exit_json(path=path, changed=False)
+    exit_kv(path=path, changed=False)
 
 if state == 'file':
 
-    debug('requesting file')
     if prev_state == 'absent':
-        fail_json(path=path, msg='file does not exist, use copy or template module to create')
+        fail_kv(path=path, msg='file does not exist, use copy or template module to create')
 
     # set modes owners and context as needed
     changed = set_context_if_different(path, secontext, changed)
@@ -334,11 +311,10 @@ if state == 'file':
     changed = set_group_if_different(path, group, changed)
     changed = set_mode_if_different(path, mode, changed)
 
-    exit_json(path=path, changed=changed)
+    exit_kv(path=path, changed=changed)
 
 elif state == 'directory':
 
-    debug('requesting directory')
     if prev_state == 'absent':
         os.makedirs(path)
         changed = True
@@ -349,7 +325,7 @@ elif state == 'directory':
     changed = set_group_if_different(path, group, changed)
     changed = set_mode_if_different(path, mode, changed)
 
-    exit_json(path=path, changed=changed)
+    exit_kv(path=path, changed=changed)
 
 elif state == 'link':
     
@@ -358,7 +334,7 @@ elif state == 'link':
     else:
         abs_src = os.path.join(os.path.dirname(dest), src)
     if not os.path.exists(abs_src):
-        fail_json(dest=dest, src=src, msg='src file does not exist')
+        fail_kv(dest=dest, src=src, msg='src file does not exist')
    
     if prev_state == 'absent':
         os.symlink(src, dest)
@@ -371,7 +347,7 @@ elif state == 'link':
             os.unlink(dest)
             os.symlink(src, dest)
     else:
-        fail_json(dest=dest, src=src, msg='unexpected position reached')
+        fail_kv(dest=dest, src=src, msg='unexpected position reached')
 
     # set modes owners and context as needed
     changed = set_context_if_different(dest, secontext, changed)
@@ -379,9 +355,9 @@ elif state == 'link':
     changed = set_group_if_different(dest, group, changed)
     changed = set_mode_if_different(dest, mode, changed)
 
-    exit_json(dest=dest, src=src, changed=changed)
+    exit_kv(dest=dest, src=src, changed=changed)
 
 
-fail_json(path=path, msg='unexpected position reached')
+fail_kv(path=path, msg='unexpected position reached')
 sys.exit(0)
 

library/git

@@ -18,19 +18,31 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
 # I wanted to keep this simple at first, so for now this checks out
-# from the MASTER branch of a repo at a particular SHA or
+# from the given branch of a repo at a particular SHA or
 # tag.  Latest is not supported, you should not be doing
-# that.  Branch checkouts are not supported.  Contribs
-# welcome! -- MPD
+# that. Contribs welcome! -- MPD
 
 try:
     import json
 except ImportError:
     import simplejson as json
 import os
+import re
 import sys
 import shlex
 import subprocess
+import syslog
+
+# ===========================================
+# Basic support methods
+
+def exit_json(rc=0, **kwargs):
+   print json.dumps(kwargs)
+   sys.exit(rc)
+
+def fail_json(**kwargs):
+   kwargs['failed'] = True
+   exit_json(rc=1, **kwargs)
 
 # ===========================================
 # convert arguments of form a=b c=d
@@ -38,29 +50,19 @@ import subprocess
 # FIXME: make more idiomatic
 
 if len(sys.argv) == 1:
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "the command module requires arguments (-a)"
-    })
-    sys.exit(1)
+   fail_json(msg="the command module requires arguments (-a)")
 
 argfile = sys.argv[1]
 if not os.path.exists(argfile):
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "Argument file not found"
-    })
-    sys.exit(1)
+   fail_json(msg="Argument file not found")
 
 args = open(argfile, 'r').read()
 items = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "the command module requires arguments (-a)"
-    })
-    sys.exit(1)
+   fail_json(msg="the command module requires arguments (-a)")
 
 params = {}
 for x in items:
@@ -69,7 +71,8 @@ for x in items:
 
 dest = params['dest']
 repo = params['repo']
-version = params['version']
+branch  = params.get('branch', 'master')
+version = params.get('version', 'HEAD')
 
 # ===========================================
 
@@ -81,7 +84,7 @@ def get_version(dest):
    sha = sha[0].split()[1]
    return sha
 
-def clone(repo, dest):
+def clone(repo, dest, branch):
    ''' makes a new git repo if it does not already exist '''
    try:
        os.makedirs(dest)
@@ -89,12 +92,68 @@ def clone(repo, dest):
        pass
    cmd = "git clone %s %s" % (repo, dest)
    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+   (out, err) = cmd.communicate()
+   rc = cmd.returncode
+   
+   if branch is None or rc != 0: 
+     return (out, err)
+
+   os.chdir(dest)
+   cmd = "git checkout -b %s origin/%s" % (branch, branch) 
+   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    return cmd.communicate()
 
-def pull(repo, dest):
+def reset(dest):
+   '''
+   Resets the index and working tree to HEAD.
+   Discards any changes to tracked files in working
+   tree since that commit.
+   '''
+   os.chdir(dest)
+   cmd = "git reset --hard HEAD"
+   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+   (out, err) = cmd.communicate()
+   rc = cmd.returncode
+   return (rc, out, err)
+
+def switchLocalBranch( branch ):
+   cmd = "git checkout %s" % branch
+   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+   return cmd.communicate()
+
+def pull(repo, dest, branch):
    ''' updates repo from remote sources '''
    os.chdir(dest)
-   cmd = "git pull -u origin"
+   cmd = "git branch -a"
+   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+   (gbranch_out, gbranch_err) = cmd.communicate()
+
+   try:
+      m = re.search( '^\* (\S+|\(no branch\))$', gbranch_out, flags=re.M )
+      cur_branch = m.group(1)
+      m = re.search( '\s+remotes/origin/HEAD -> origin/(\S+)', gbranch_out, flags=re.M )
+      default_branch = m.group(1)
+   except:
+      fail_json(msg="could not determine branch data - received: %s" % gbranch_out)
+
+   if branch is None:
+      if cur_branch != default_branch:
+         (out, err) = switchLocalBranch( default_branch )
+
+      cmd = "git pull -u origin"
+
+   elif branch == cur_branch:
+      cmd = "git pull -u origin"
+
+   else:
+      m = re.search( '^\s+%s$' % branch, gbranch_out, flags=re.M ) #see if we've already checked it out
+      if m is None:
+         cmd = "git checkout -b %s origin/%s" % (branch, branch)
+
+      else:
+         (out, err) = switchLocalBranch( branch )
+         cmd = "git pull -u origin"
+
    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    return cmd.communicate()
 
@@ -120,33 +179,26 @@ out, err, status = (None, None, None)
 
 before = None
 if not os.path.exists(gitconfig):
-   (out, err) = clone(repo, dest)
+   (out, err) = clone(repo, dest, branch)
 else:
    # else do a pull   
    before = get_version(dest)
-   (out, err) = pull(repo, dest)
+   (rc, out, err) = reset(dest)
+   if rc != 0:
+      fail_json(out=out, err=err)
+   (out, err) = pull(repo, dest, branch)
 
 # handle errors from clone or pull
 
 if out.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+   fail_json(out=out, err=err)
 
 # switch to version specified regardless of whether
 # we cloned or pulled
 
 (out, err) = switchver(version, dest)
 if err.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+   fail_json(out=out, err=err)
 
 # determine if we changed anything
 
@@ -156,9 +208,4 @@ changed = False
 if before != after:
    changed = True
 
-print json.dumps({
-   "changed" : changed,
-   "before"  : before,
-   "after"   : after
-})
-
+exit_json(changed=changed, before=before, after=after)

library/group

@@ -26,6 +26,7 @@ import grp
 import shlex
 import subprocess
 import sys
+import syslog
 
 GROUPADD = "/usr/sbin/groupadd"
 GROUPDEL = "/usr/sbin/groupdel"
@@ -131,6 +132,8 @@ if len(sys.argv) == 2 and os.path.exists(sys.argv[1]):
 else:
     args = ' '.join(sys.argv[1:])
 items   = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
     fail_json(msg='the module requires arguments -a')

library/ohai

@@ -18,4 +18,5 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+/usr/bin/logger -t ansible-ohai Invoked as-is
 /usr/bin/ohai

library/ping

@@ -22,4 +22,10 @@ try:
 except ImportError:
     import simplejson as json
 
+import os
+import syslog
+
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked as-is')
+
 print json.dumps({ "ping" : "pong" })

library/raw

@@ -0,0 +1,23 @@
+#!/usr/bin/python
+
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# hey the Ansible raw module isn't really a remote transferred
+# module.  All the magic happens in Runner.py, see the web docs
+# for more details.
+

library/service

@@ -24,14 +24,99 @@ except ImportError:
 import sys
 import shlex
 import subprocess
+import os.path
+import syslog
 
 # TODO: switch to fail_json and other helper functions
 # like other modules are using
 
 # ===========================================
 
-SERVICE = '/sbin/service'
-CHKCONFIG = '/sbin/chkconfig'
+SERVICE = None
+CHKCONFIG = None
+
+def fail_json(d):
+    print json.dumps(d)
+    sys.exit(1)
+
+def _find_binaries():
+    # list of possible paths for service/chkconfig binaries
+    # with the most probable first
+    global CHKCONFIG
+    global SERVICE
+    global INITCTL
+    paths = ['/sbin', '/usr/sbin', '/bin', '/usr/bin']
+    binaries = [ 'service', 'chkconfig', 'update-rc.d', 'initctl']
+    location = dict()
+
+    for binary in binaries:
+        location[binary] = None
+
+    for binary in binaries:
+        for path in paths:
+            if os.path.exists(path + '/' + binary):
+                location[binary] = path + '/' + binary
+                break
+
+    if location.get('chkconfig', None):
+        CHKCONFIG = location['chkconfig']
+    elif location.get('update-rc.d', None):
+        CHKCONFIG = location['update-rc.d']
+    else:
+        fail_json(dict(failed=True, msg='unable to find chkconfig or update-rc.d binary'))
+    if location.get('service', None):
+        SERVICE = location['service']
+    else:
+        fail_json(dict(failed=True, msg='unable to find service binary'))
+    if location.get('initctl', None):
+        INITCTL = location['initctl']
+    else:
+        INITCTL = None
+
+def _get_service_status(name):
+    rc, status_stdout, status_stderr = _run("%s %s status" % (SERVICE, name))
+    status = status_stdout + status_stderr
+
+    # set the running state to None because we don't know it yet
+    running = None
+
+    # Check if we got upstart on the system and then the job state
+    if INITCTL != None:
+        # check the job status by upstart response
+        initctl_rc, initctl_status_stdout, initctl_status_stderr = _run("%s status %s" % (INITCTL, name))
+        if initctl_status_stdout.find("stop/waiting") != -1:
+            running = False
+        elif initctl_status_stdout.find("start/running") != -1:
+            running = True
+
+    # if the job status is still not known check it by response code
+    if running == None:
+        if rc == 3:
+            running = False
+        elif rc == 0:
+            running = True
+
+    # if the job status is still not known check it by status output keywords
+    if running == None:
+        # first tranform the status output that could irritate keyword matching
+        cleaned_status_stdout = status_stdout.lower().replace(name.lower(),'')
+        if cleaned_status_stdout.find("stop") != -1:
+            running = False
+        elif cleaned_status_stdout.find("run") != -1 and cleaned_status_stdout.find("not") != -1:
+            running = False
+        elif cleaned_status_stdout.find("run") != -1 and cleaned_status_stdout.find("not") == -1:
+            running = True
+        elif cleaned_status_stdout.find("start") != -1 and cleaned_status_stdout.find("not") == -1:
+            running = True
+
+    # if the job status is still not known check it by special conditions
+    if running == None:
+        if name == 'iptables' and status_stdout.find("ACCEPT") != -1:
+            # iptables status command output is lame
+            # TODO: lookup if we can use a return code for this instead?
+            running = True
+    
+    return running
 
 def _run(cmd):
     # returns (rc, stdout, stderr) from shell command
@@ -41,67 +126,66 @@ def _run(cmd):
 
 
 def _do_enable(name, enable):
-    if enable.lower() in ['on', 'true', 'yes']:
-        rc, stdout, stderr = _run("%s %s on" % (CHKCONFIG, name))
-    elif enable.lower() in ['off', 'false', 'no']:
-        rc, stdout, stderr = _run("%s %s off" % (CHKCONFIG, name))
+    # we change argument depending on real binary used
+    # update-rc.d wants enable/disable while
+    # chkconfig wants on/off
+    valid_argument = dict({'on' : 'on', 'off' : 'off'})
+
+    if CHKCONFIG.endswith("update-rc.d"):
+        valid_argument['on'] = "enable"
+        valid_argument['off'] = "disable"
+
+    if enable.lower() in ['on', 'true', 'yes', 'enable']:
+        rc, stdout, stderr = _run("%s %s %s" % (CHKCONFIG, name, valid_argument['on']))
+    elif enable.lower() in ['off', 'false', 'no', 'disable']:
+        rc, stdout, stderr = _run("%s %s %s" % (CHKCONFIG, name, valid_argument['off']))
     
     return rc, stdout, stderr
     
 argfile = sys.argv[1]
 args = open(argfile, 'r').read()
 items = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
-    print json.dumps(dict(failed=True, msg='this module requires arguments (-a)'))
-    sys.exit(1)
+    fail_json(dict(failed=True, msg='this module requires arguments (-a)'))
 
 params = {}
 for arg in items:
     if "=" not in arg:
-        print json.dumps(dict(failed=True, msg='expected key=value format arguments'))
-        sys.exit(1)
+        fail_json(dict(failed=True, msg='expected key=value format arguments'))
+
     (name, value) = arg.split("=")
     params[name] = value
 
 name = params.get('name', None)
 
 if name is None:
-    print json.dumps(dict(failed=True, msg='missing name'))
-    sys.exit(1)
+    fail_json(dict(failed=True, msg='missing name'))
 
 state = params.get('state', None)
 list_items = params.get('list', None)
-enable = params.get('enable', None)
+enable = params.get('enabled', params.get('enable', None))
 
 # running and started are the same
-if state and state not in [ 'running', 'started', 'stopped', 'restarted' ]:
-    print json.dumps(dict(failed=True, msg='invalid value for state'))
-    sys.exit(1)
-if list_items and list_items not in [ 'status' ]:
-    print json.dumps(dict(failed=True, msg='invalid value for list'))
-    sys.exit(1)
-if enable and enable.lower() not in [ 'on', 'off', 'true', 'false', 'yes', 'no' ]:
-    print json.dumps(dict(failed=True, msg='invalid value for enable'))
-    sys.exit(1)
+if state and state.lower() not in [ 'running', 'started', 'stopped', 'restarted','reloaded' ]:
+    fail_json(dict(failed=True, msg='invalid value for state'))
+if list_items and list_items.lower() not in [ 'status' ]:
+    fail_json(dict(failed=True, msg='invalid value for list'))
+if enable and enable.lower() not in [ 'on', 'off', 'true', 'false', 'yes', 'no', 'enable', 'disable' ]:
+    fail_json(dict(failed=True, msg='invalid value for enable'))
+    
+
+# ===========================================
+# find binaries locations on minion
+_find_binaries()
 
 
 # ===========================================
 # get service status
+running = _get_service_status(name)
 
-rc, status_stdout, status_stderr = _run("%s %s status" % (SERVICE, name))
-status = status_stdout + status_stderr
-
-
-running = False
-if status_stdout.find("not running") != -1:
-    running = False
-elif status_stdout.find("running") != -1:
-    running = True
-elif name == 'iptables' and status_stdout.find("ACCEPT") != -1:
-    # iptables status command output is lame
-    # TODO: lookup if we can use a return code for this instead?
-    running = True
 
 if state or enable:
     rc = 0
@@ -115,15 +199,21 @@ if state or enable:
         out += out_enable
         err += err_enable
 
-    if state:
+    if state and running == None:
+        print json.dumps({
+            "failed" : True,
+            "msg"    : "failed determining the current service state => state stays unchanged",
+        })
+        print >> sys.stderr, out + err
+    elif state:
         # a state change command has been requested
 
         # ===========================================
         # determine if we are going to change anything
 
-        if not running and state == "started":
+        if not running and state in ("started", "running"):
             changed = True
-        elif running and state == "stopped":
+        elif running and state in ("stopped","reloaded"):
             changed = True
         elif state == "restarted":
             changed = True
@@ -136,17 +226,19 @@ if state or enable:
                 rc_state, stdout, stderr = _run("%s %s start" % (SERVICE, name))
             elif state == 'stopped':
                 rc_state, stdout, stderr = _run("%s %s stop" % (SERVICE, name))
+            elif state == 'reloaded':
+                rc_state, stdout, stderr = _run("%s %s reload" % (SERVICE, name))
             elif state == 'restarted':
                 rc1, stdout1, stderr1 = _run("%s %s stop" % (SERVICE, name))
                 rc2, stdout2, stderr2 = _run("%s %s start" % (SERVICE, name))
-                rc_state = rc and rc1 and rc2
+                rc_state = rc + rc1 + rc2
                 stdout = stdout1 + stdout2
                 stderr = stderr1 + stderr2
-            
+
             out += stdout
             err += stderr
-            rc = rc and rc_state
-        
+            rc = rc + rc_state
+
     if rc != 0:
 
         print json.dumps({
@@ -180,7 +272,7 @@ elif list_items is not None:
 
 else:
  
-    print json.dumps(dict(failed=True, msg="expected state or list parameters"))    
+    print json.dumps(dict(failed=True, msg="expected state or list parameters"))
 
 
 sys.exit(0)

library/setup

@@ -31,6 +31,13 @@ import socket
 import struct
 import subprocess
 import traceback
+import syslog
+
+try:
+    import selinux
+    HAVE_SELINUX=True
+except ImportError:
+    HAVE_SELINUX=False
 
 try:
     import json
@@ -47,7 +54,9 @@ DMI_DICT = { 'form_factor':  '/sys/devices/virtual/dmi/id/chassis_type',
              'product_serial': '/sys/devices/virtual/dmi/id/product_serial',
              'product_uuid': '/sys/devices/virtual/dmi/id/product_uuid',
              'product_version': '/sys/devices/virtual/dmi/id/product_version',
-             'system_vendor': '/sys/devices/virtual/dmi/id/sys_vendor' }
+             'system_vendor': '/sys/devices/virtual/dmi/id/sys_vendor',
+             'bios_date': '/sys/devices/virtual/dmi/id/bios_date',
+             'bios_version': '/sys/devices/virtual/dmi/id/bios_version' }
 # From smolt and DMI spec
 FORM_FACTOR = [ "Unknown", "Other", "Unknown", "Desktop",
                 "Low Profile Desktop", "Pizza Box", "Mini Tower", "Tower",
@@ -61,6 +70,7 @@ FORM_FACTOR = [ "Unknown", "Other", "Unknown", "Desktop",
 # This is the fallback to handle unknowns or exceptions
 OSDIST_DICT = { '/etc/redhat-release': 'RedHat',
                 '/etc/vmware-release': 'VMwareESX' }
+SELINUX_MODE_DICT = { 1: 'enforcing', 0: 'permissive', -1: 'disabled' }
 
 def get_file_content(path):
     if os.path.exists(path) and os.access(path, os.R_OK):
@@ -185,7 +195,7 @@ def get_linux_virtual_facts(facts):
         facts['virtualization_type'] = 'VMware'
         facts['virtualization_role'] = 'host'
     # You can spawn a dmidecode process and parse that or infer from devices
-    for dev_model in glob.glob('/proc/ide/hd*/model'):
+    for dev_model in glob.glob('/sys/block/?da/device/vendor'):
         info = open(dev_model).read()
         if 'VMware' in info:
             facts['virtualization_type'] = 'VMware'
@@ -201,7 +211,9 @@ def get_virtual_facts(facts):
         facts = get_linux_virtual_facts(facts)
 
 # get list of interfaces that are up
-def get_interfaces():
+def get_interfaces(facts):
+    if facts['system'] == 'SunOS':
+        return []
     length = 4096
     offset = 32
     step = 32
@@ -226,7 +238,7 @@ def get_iface_hwaddr(iface):
 def get_network_facts(facts):
     facts['fqdn'] = socket.getfqdn()
     facts['hostname'] = facts['fqdn'].split('.')[0]
-    facts['interfaces'] = get_interfaces()
+    facts['interfaces'] = get_interfaces(facts)
     for iface in facts['interfaces']:
         facts[iface] = { 'macaddress': get_iface_hwaddr(iface) }
         # This is lame, but there doesn't appear to be a good way
@@ -241,6 +253,9 @@ def get_network_facts(facts):
                     facts[iface]['ipv4'] = {}
                 facts[iface]['ipv4'] = { 'address': data[1].split(':')[1],
                                          'netmask': data[-1].split(':')[1] }
+                ip = struct.unpack("!L", socket.inet_aton(facts[iface]['ipv4']['address']))[0]
+                mask = struct.unpack("!L", socket.inet_aton(facts[iface]['ipv4']['netmask']))[0]
+                facts[iface]['ipv4']['network'] = socket.inet_ntoa(struct.pack("!L", ip & mask))
             if 'inet6 addr' in line:
                 (ip, prefix) = data[2].split('/')
                 scope = data[3].split(':')[1].lower()
@@ -263,8 +278,29 @@ def get_public_ssh_host_keys(facts):
     else:
         facts['ssh_host_key_rsa_public'] = rsa.split()[1]
 
+def get_selinux_facts(facts):
+    if not HAVE_SELINUX:
+        facts['selinux'] = False
+        return
+    facts['selinux'] = {}
+    if not selinux.is_selinux_enabled():
+        facts['selinux']['status'] = 'disabled'
+    else:
+        facts['selinux']['status'] = 'enabled'
+        facts['selinux']['policyvers'] = selinux.security_policyvers()
+        (rc, configmode) = selinux.selinux_getenforcemode()
+        if rc == 0 and SELINUX_MODE_DICT.has_key(configmode):
+            facts['selinux']['config_mode'] = SELINUX_MODE_DICT[configmode]
+        mode = selinux.security_getenforce()
+        if SELINUX_MODE_DICT.has_key(mode):
+            facts['selinux']['mode'] = SELINUX_MODE_DICT[mode]
+        (rc, policytype) = selinux.selinux_getpolicytype()
+        if rc == 0:
+            facts['selinux']['type'] = policytype
+
 def get_service_facts(facts):
     get_public_ssh_host_keys(facts)
+    get_selinux_facts(facts)
 
 def ansible_facts():
     facts = {}
@@ -275,6 +311,10 @@ def ansible_facts():
     get_service_facts(facts)
     return facts
 
+def md5_sum(f):
+    md5sum = os.popen("/usr/bin/md5sum %(file)s 2>/dev/null || /sbin/md5 -q %(file)s 2>/dev/null || /usr/bin/digest -a md5 -v %(file)s 2>/dev/null" % {"file": f}).read().split()[0]
+    return md5sum
+
 # load config & template variables
 
 if len(sys.argv) == 1:
@@ -295,6 +335,9 @@ except:
        (k,v) = opt.split("=")
        setup_options[k]=v   
 
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % setup_options)
+
 ansible_file = os.path.expandvars(setup_options.get('metadata', DEFAULT_ANSIBLE_SETUP))
 ansible_dir = os.path.dirname(ansible_file)
 
@@ -308,7 +351,7 @@ md5sum = None
 if not os.path.exists(ansible_file):
     changed = True
 else:
-    md5sum = os.popen("md5sum %s" % ansible_file).read().split()[0]
+    md5sum = md5_sum(ansible_file)
 
 # Get some basic facts in case facter or ohai are not installed
 for (k, v) in ansible_facts().items():
@@ -357,7 +400,7 @@ reformat = json.dumps(setup_options, sort_keys=True, indent=4)
 f.write(reformat)
 f.close()
 
-md5sum2 = os.popen("md5sum %s" % ansible_file).read().split()[0]
+md5sum2 = md5_sum(ansible_file)
 
 if md5sum != md5sum2:
    changed = True
@@ -368,5 +411,8 @@ setup_result['changed'] = changed
 setup_result['md5sum']  = md5sum2 
 setup_result['ansible_facts'] = setup_options
 
+# hack to keep --verbose from showing all the setup module results
+setup_result['verbose_override'] = True
+
 print json.dumps(setup_result)
 

library/slurp

@@ -21,6 +21,7 @@ import sys
 import os
 import shlex
 import base64
+import syslog
 
 try:
    import json
@@ -36,7 +37,11 @@ if len(sys.argv) == 1:
 argfile = sys.argv[1]
 if not os.path.exists(argfile):
     sys.exit(1)
-items = shlex.split(open(argfile, 'r').read())
+
+args = open(argfile, 'r').read()
+items = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 params = {}
 for x in items:

library/user

@@ -25,9 +25,14 @@ import os
 import pwd
 import grp
 import shlex
-import spwd
 import subprocess
 import sys
+import syslog
+try:
+    import spwd
+    HAVE_SPWD=True
+except:
+    HAVE_SPWD=False
 
 USERADD = "/usr/sbin/useradd"
 USERMOD = "/usr/sbin/usermod"
@@ -119,6 +124,10 @@ def user_add(user, **kwargs):
     else:
         return False
 
+"""
+Without spwd, we would have to resort to reading /etc/shadow
+to get the encrypted string.  For now, punt on idempotent password changes.
+"""
 def user_mod(user, **kwargs):
     cmd = [USERMOD]
     info = user_info(user)
@@ -135,14 +144,27 @@ def user_mod(user, **kwargs):
                 cmd.append('-g')
                 cmd.append(kwargs[key])
         elif key == 'groups' and kwargs[key] is not None:
-            defined_groups = kwargs[key].split(',')
-            for g in defined_groups:
+            current_groups = user_group_membership(user)
+            groups = kwargs[key].split(',')
+            for g in groups:
                 if not group_exists(g):
                     fail_json(msg="Group %s does not exist" % (g))
-            existing_groups = user_group_membership(user)
-            if sorted(defined_groups) != sorted(existing_groups):
+            group_diff = set(sorted(current_groups)).symmetric_difference(set(sorted(groups)))
+            groups_need_mod = False
+
+            if group_diff:
+                if kwargs['append'] is not None and kwargs['append'] == 'yes':
+                    for g in groups:
+                        if g in group_diff:
+                            cmd.append('-a')
+                            groups_need_mod = True
+                else:
+                   groups_need_mod = True
+
+            if groups_need_mod:
                 cmd.append('-G')
-                cmd.append(kwargs[key])
+                cmd.append(','.join(groups))
+
         elif key == 'comment':
             if kwargs[key] is not None and info[4] != kwargs[key]:
                 cmd.append('-c')
@@ -159,10 +181,6 @@ def user_mod(user, **kwargs):
             if kwargs[key] is not None and info[1] != kwargs[key]:
                 cmd.append('-p')
                 cmd.append(kwargs[key])
-        elif key == 'append':
-            if kwargs[key] is not None and kwargs[key] == 'yes':
-                if 'groups' in kwargs and kwargs['groups'] is not None:
-                    cmd.append('-a')
     # skip if no changes to be made
     if len(cmd) == 1:
         return False
@@ -217,10 +235,12 @@ def user_info(user):
         return False
     try:
         info = get_pwd_info(user)
-        sinfo = spwd.getspnam(user)
+        if HAVE_SPWD:
+            sinfo = spwd.getspnam(user)
     except KeyError:
         return False
-    info[1] = sinfo[1]
+    if HAVE_SPWD:
+        info[1] = sinfo[1]
     return info
 
 # ===========================================
@@ -244,6 +264,8 @@ if not os.path.exists(USERDEL):
 argfile = sys.argv[1]
 args    = open(argfile, 'r').read()
 items   = shlex.split(args)
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
 if not len(items):
     fail_json(msg='the module requires arguments -a')

library/virt

@@ -27,6 +27,7 @@ except ImportError:
 import os
 import sys
 import subprocess
+import syslog
 try:
     import libvirt
 except ImportError:
@@ -123,7 +124,6 @@ class LibvirtConnection(object):
 
     def get_status2(self, vm):
         state = vm.info()[0]
-        # print "DEBUG: state: %s" % state
         return VIRT_STATE_NAME_MAP.get(state,"unknown")
 
     def get_status(self, vmid):
@@ -366,6 +366,8 @@ def main():
 
     args = open(argfile, 'r').read()
     items = shlex.split(args)
+    syslog.openlog('ansible-%s' % os.path.basename(__file__))
+    syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
     if not len(items):
         return VIRT_FAILED, msg

library/yum

@@ -27,6 +27,7 @@ import datetime
 import shlex
 import re
 import traceback
+import syslog
 
 
 try:
@@ -299,6 +300,8 @@ def main():
 
     args = open(argfile, 'r').read()
     items = shlex.split(args)
+    syslog.openlog('ansible-%s' % os.path.basename(__file__))
+    syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % args)
 
     if not len(items):
         msg = "the yum module requires arguments (-a)"

packaging/debian/changelog

@@ -1,3 +1,15 @@
+ansible (0.5) debian; urgency=low
+
+  * 0.5 update
+
+ -- Michael DeHaan <michael.dehaan@gmail.com> Wed, 04 Jul 2012 13:40:00 -0400
+
+ansible (0.4) debian; urgency=low
+  
+  * 0.4 update
+ 
+ -- Michael DeHaan <michael.dehaan@gmail.com> Wed, 23 May 2012 19:40:00 -0400
+
 ansible (0.3) debian; urgency=low
 
   * 0.3 update

packaging/rpm/ansible.spec

@@ -6,7 +6,7 @@ BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
 Name: ansible
 Release: 1%{?dist}
 Summary: Minimal SSH command and control
-Version: 0.3
+Version: 0.5
 
 Group: Development/Libraries
 License: GPLv3
@@ -58,7 +58,13 @@ rm -rf $RPM_BUILD_ROOT
 
 
 %changelog
-* Mon Apr 23 2012 Michael DeHaan <mdehaan@redhat.com> - 0.3-1
+* Wed Jul 4 2012 Michael DeHaan <michael.dehaan@gmail.com> - 0.5-0
+- Not released yet
+
+* Wed May 23 2012 Michael DeHaan <michael.dehaan@gmail.com> - 0.4-0
+- Release of 0.4
+
+* Mon Apr 23 2012 Michael DeHaan <michael.dehaan@gmail.com> - 0.3-1
 - Release of 0.3
 
 * Tue Apr  3 2012 John Eckersberg <jeckersb@redhat.com> - 0.0.2-1

setup.py

@@ -22,6 +22,10 @@ setup(name='ansible',
       package_dir={ 'ansible': 'lib/ansible' },
       packages=[
          'ansible',
+         'ansible.inventory',
+         'ansible.playbook',
+         'ansible.runner',
+         'ansible.runner.connection',
       ],
       scripts=[
          'bin/ansible',

test/TestInventory.py

@@ -3,6 +3,7 @@ import unittest
 
 from ansible.inventory import Inventory
 from ansible.runner import Runner
+# from nose.plugins.skip import SkipTest
 
 class TestInventory(unittest.TestCase):
 
@@ -10,60 +11,73 @@ class TestInventory(unittest.TestCase):
         self.cwd = os.getcwd()
         self.test_dir = os.path.join(self.cwd, 'test')
 
-        self.inventory_file = os.path.join(self.test_dir, 'simple_hosts')
-        self.inventory_script = os.path.join(self.test_dir, 'inventory_api.py')
-        self.inventory_yaml = os.path.join(self.test_dir, 'yaml_hosts')
+        self.inventory_file         = os.path.join(self.test_dir, 'simple_hosts')
+        self.complex_inventory_file = os.path.join(self.test_dir, 'complex_hosts')
+        self.inventory_script       = os.path.join(self.test_dir, 'inventory_api.py')
+        self.inventory_yaml         = os.path.join(self.test_dir, 'yaml_hosts')
 
         os.chmod(self.inventory_script, 0755)
 
     def tearDown(self):
         os.chmod(self.inventory_script, 0644)
 
-    ### Simple inventory format tests
+    def compare(self, left, right):
+        left = sorted(left)
+        right = sorted(right)
+        print left
+        print right
+        assert left == right
+
 
     def simple_inventory(self):
-        return Inventory( self.inventory_file )
+        return Inventory(self.inventory_file)
 
     def script_inventory(self):
-        return Inventory( self.inventory_script )
+        return Inventory(self.inventory_script)
 
     def yaml_inventory(self):
-        return Inventory( self.inventory_yaml )
+        return Inventory(self.inventory_yaml)
+
+    def complex_inventory(self):
+        return Inventory(self.complex_inventory_file)
+
+    #####################################
+    ### Simple inventory format tests
 
     def test_simple(self):
         inventory = self.simple_inventory()
         hosts = inventory.list_hosts()
 
         expected_hosts=['jupiter', 'saturn', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_all(self):
         inventory = self.simple_inventory()
         hosts = inventory.list_hosts('all')
 
         expected_hosts=['jupiter', 'saturn', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_norse(self):
         inventory = self.simple_inventory()
         hosts = inventory.list_hosts("norse")
 
         expected_hosts=['thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_ungrouped(self):
         inventory = self.simple_inventory()
         hosts = inventory.list_hosts("ungrouped")
 
         expected_hosts=['jupiter', 'saturn']
-        assert hosts == expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_combined(self):
         inventory = self.simple_inventory()
         hosts = inventory.list_hosts("norse:greek")
 
         expected_hosts=['zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_restrict(self):
         inventory = self.simple_inventory()
@@ -74,17 +88,33 @@ class TestInventory(unittest.TestCase):
         inventory.restrict_to(restricted_hosts)
         hosts = inventory.list_hosts("norse:greek")
 
-        assert hosts == restricted_hosts
+        print "Hosts=%s" % hosts
+        print "Restricted=%s" % restricted_hosts
+        assert sorted(hosts) == sorted(restricted_hosts)
 
         inventory.lift_restriction()
         hosts = inventory.list_hosts("norse:greek")
 
-        assert hosts == expected_hosts
+        print hosts
+        print expected_hosts
+        assert sorted(hosts) == sorted(expected_hosts)
+
+    def test_simple_exclude(self):
+        inventory = self.simple_inventory()
+
+        hosts = inventory.list_hosts("all:!greek")
+        expected_hosts=['jupiter', 'saturn', 'thor', 'odin', 'loki']
+        assert sorted(hosts) == sorted(expected_hosts)
+
+        hosts = inventory.list_hosts("all:!norse:!greek")
+        expected_hosts=['jupiter', 'saturn']
+        assert sorted(hosts) == sorted(expected_hosts)
 
     def test_simple_vars(self):
         inventory = self.simple_inventory()
         vars = inventory.get_variables('thor')
 
+        print vars
         assert vars == {'group_names': ['norse'],
                         'inventory_hostname': 'thor'}
 
@@ -92,10 +122,41 @@ class TestInventory(unittest.TestCase):
         inventory = self.simple_inventory()
         vars = inventory.get_variables('hera')
 
-        assert vars == {'ansible_ssh_port': 3000,
+        print vars
+        expected = {'ansible_ssh_port': 3000,
                         'group_names': ['greek'],
                         'inventory_hostname': 'hera'}
+        print expected
+        assert vars == expected
 
+    ###################################################
+    ### INI file advanced tests
+
+    def test_complex_vars(self):
+        inventory = self.complex_inventory()
+
+        vars = inventory.get_variables('rtp_a')
+        print vars
+
+        expected = dict(
+            a='1', b='2', c='3', d='100002', rga='1', rgb='2', rgc='3', 
+            inventory_hostname='rtp_a', 
+            group_names=[ 'eastcoast', 'nc', 'redundantgroup', 'redundantgroup2', 'redundantgroup3', 'rtp', 'us' ]
+        )
+        print vars
+        print expected
+        assert vars == expected
+
+    def test_complex_exclude(self):
+        inventory = self.complex_inventory()
+
+        hosts = inventory.list_hosts("nc:!triangle:florida:!orlando")
+        expected_hosts=['rtp_a', 'rtp_b', 'rtb_c', 'miami']
+        assert sorted(hosts) == sorted(expected_hosts)
+
+
+
+    ###################################################
     ### Inventory API tests
 
     def test_script(self):
@@ -149,6 +210,8 @@ class TestInventory(unittest.TestCase):
         inventory = self.script_inventory()
         vars = inventory.get_variables('thor')
 
+        print "VARS=%s" % vars
+
         assert vars == {'hammer':True,
                         'group_names': ['norse'],
                         'inventory_hostname': 'thor'}
@@ -159,36 +222,36 @@ class TestInventory(unittest.TestCase):
         inventory = self.yaml_inventory()
         hosts = inventory.list_hosts()
         print hosts
-        expected_hosts=['jupiter', 'saturn', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        expected_hosts=['jupiter', 'saturn', 'mars', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
+        self.compare(hosts, expected_hosts)
 
     def test_yaml_all(self):
         inventory = self.yaml_inventory()
         hosts = inventory.list_hosts('all')
 
-        expected_hosts=['jupiter', 'saturn', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        expected_hosts=['jupiter', 'saturn', 'mars', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
+        self.compare(hosts, expected_hosts)
 
     def test_yaml_norse(self):
         inventory = self.yaml_inventory()
         hosts = inventory.list_hosts("norse")
 
         expected_hosts=['thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        self.compare(hosts, expected_hosts)
 
-    def test_simple_ungrouped(self):
+    def test_yaml_ungrouped(self):
         inventory = self.yaml_inventory()
         hosts = inventory.list_hosts("ungrouped")
 
-        expected_hosts=['jupiter']
-        assert hosts == expected_hosts
+        expected_hosts=['jupiter', 'mars']
+        self.compare(hosts, expected_hosts)
 
     def test_yaml_combined(self):
         inventory = self.yaml_inventory()
         hosts = inventory.list_hosts("norse:greek")
 
         expected_hosts=['zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
+        self.compare(hosts, expected_hosts)
 
     def test_yaml_restrict(self):
         inventory = self.yaml_inventory()
@@ -199,21 +262,28 @@ class TestInventory(unittest.TestCase):
         inventory.restrict_to(restricted_hosts)
         hosts = inventory.list_hosts("norse:greek")
 
-        assert hosts == restricted_hosts
+        self.compare(hosts, restricted_hosts)
 
         inventory.lift_restriction()
         hosts = inventory.list_hosts("norse:greek")
 
-        assert hosts == expected_hosts
+        self.compare(hosts, expected_hosts)
 
     def test_yaml_vars(self):
         inventory = self.yaml_inventory()
         vars = inventory.get_variables('thor')
-        print vars
         assert vars == {'group_names': ['norse'],
                         'hammer':True,
                         'inventory_hostname': 'thor'}
 
+    def test_yaml_list_vars(self):
+        inventory = self.yaml_inventory()
+        vars = inventory.get_variables('zeus')
+        assert vars == {'ansible_ssh_port': 3001,
+                        'group_names': ['greek', 'ruler'],
+                        'inventory_hostname': 'zeus',
+                        'ntp_server': 'olympus.example.com'}
+
     def test_yaml_change_vars(self):
         inventory = self.yaml_inventory()
         vars = inventory.get_variables('thor')
@@ -221,6 +291,7 @@ class TestInventory(unittest.TestCase):
         vars["hammer"] = False
 
         vars = inventory.get_variables('thor')
+        print vars
         assert vars == {'hammer':True,
                         'inventory_hostname': 'thor',
                         'group_names': ['norse']}
@@ -229,6 +300,7 @@ class TestInventory(unittest.TestCase):
         inventory = self.yaml_inventory()
         vars = inventory.get_variables('saturn')
 
+        print vars
         assert vars == {'inventory_hostname': 'saturn',
                         'moon': 'titan',
                         'moon2': 'enceladus',
@@ -238,6 +310,7 @@ class TestInventory(unittest.TestCase):
         inventory = self.yaml_inventory()
         vars = inventory.get_variables('hera')
 
+        print vars
         assert vars == {'ansible_ssh_port': 3000,
                         'inventory_hostname': 'hera',
                         'ntp_server': 'olympus.example.com',
@@ -250,25 +323,3 @@ class TestInventory(unittest.TestCase):
         assert 'group_names' in vars
         assert sorted(vars['group_names']) == [ 'norse', 'ruler' ]
 
-    ### Test Runner class method
-
-    def test_class_method(self):
-        hosts, groups = Runner.parse_hosts(self.inventory_file)
-
-        expected_hosts = ['jupiter', 'saturn', 'zeus', 'hera', 'poseidon', 'thor', 'odin', 'loki']
-        assert hosts == expected_hosts
-
-        expected_groups= {
-            'ungrouped': ['jupiter', 'saturn'],
-            'greek': ['zeus', 'hera', 'poseidon'],
-            'norse': ['thor', 'odin', 'loki']
-        }
-        assert groups == expected_groups
-
-    def test_class_override(self):
-        override_hosts = ['thor', 'odin']
-        hosts, groups = Runner.parse_hosts(self.inventory_file, override_hosts)
-
-        assert hosts == override_hosts
-
-        assert groups == { 'ungrouped': override_hosts }

test/TestPlayBook.py

@@ -63,7 +63,7 @@ class TestCallbacks(object):
     def on_ok(self, host, result):
         # delete certain info from host_result to make test comparisons easier
         host_result = result.copy()
-        for k in [ 'ansible_job_id', 'results_file', 'invocation', 'md5sum', 'delta', 'start', 'end' ]:
+        for k in [ 'ansible_job_id', 'results_file', 'md5sum', 'delta', 'start', 'end' ]:
             if k in host_result:
                 del host_result[k]
         for k in host_result.keys():
@@ -172,3 +172,16 @@ class TestPlaybook(unittest.TestCase):
        print data
        assert data.find("ears") != -1, "template success"
 
+   def test_yaml_hosts_list(self):
+       # Make sure playbooks support hosts: [host1, host2]
+       # TODO: Actually run the play on more than one host
+       test_callbacks = TestCallbacks()
+       playbook = ansible.playbook.PlayBook(
+           playbook=os.path.join(self.test_dir, 'hosts_list.yml'),
+           host_list='test/ansible_hosts',
+           stats=ans_callbacks.AggregateStats(),
+           callbacks=test_callbacks,
+           runner_callbacks=test_callbacks
+       )
+       play = ansible.playbook.Play(playbook, playbook.playbook[0])
+       assert play.hosts == ';'.join(('host1', 'host2', 'host3'))

test/TestRunner.py

@@ -10,9 +10,9 @@ import os
 import shutil
 import time
 try:
-   import json
+    import json
 except:
-   import simplejson as json
+    import simplejson as json
 
 from nose.plugins.skip import SkipTest
 
@@ -25,202 +25,234 @@ def get_binary(name):
 
 class TestRunner(unittest.TestCase):
 
-   def setUp(self):
-       self.user = getpass.getuser()
-       self.runner = ansible.runner.Runner(
-           module_name='ping',
-           module_path='library/',
-           module_args='',
-           remote_user=self.user,
-           remote_pass=None,
-           host_list='test/ansible_hosts',
-           timeout=5,
-           forks=1,
-           background=0,
-           pattern='all',
-       )
-       self.cwd = os.getcwd()
-       self.test_dir = os.path.join(self.cwd, 'test')
-       self.stage_dir = self._prepare_stage_dir()
+    def setUp(self):
+        self.user = getpass.getuser()
+        self.runner = ansible.runner.Runner(
+            module_name='ping',
+            module_path='library/',
+            module_args='',
+            remote_user=self.user,
+            remote_pass=None,
+            host_list='test/ansible_hosts',
+            timeout=5,
+            forks=1,
+            background=0,
+            pattern='all',
+        )
+        self.cwd = os.getcwd()
+        self.test_dir = os.path.join(self.cwd, 'test')
+        self.stage_dir = self._prepare_stage_dir()
 
-   def _prepare_stage_dir(self):
-       stage_path = os.path.join(self.test_dir, 'test_data')
-       if os.path.exists(stage_path):
-           shutil.rmtree(stage_path, ignore_errors=False)
-           assert not os.path.exists(stage_path)
-       os.makedirs(stage_path)
-       assert os.path.exists(stage_path)
-       return stage_path
+    def _prepare_stage_dir(self):
+        stage_path = os.path.join(self.test_dir, 'test_data')
+        if os.path.exists(stage_path):
+            shutil.rmtree(stage_path, ignore_errors=False)
+            assert not os.path.exists(stage_path)
+        os.makedirs(stage_path)
+        assert os.path.exists(stage_path)
+        return stage_path
 
-   def _get_test_file(self, filename):
-       # get a file inside the test input directory
-       filename = os.path.join(self.test_dir, filename)
-       assert os.path.exists(filename)
-       return filename
- 
-   def _get_stage_file(self, filename):
-       # get a file inside the test output directory
-       filename = os.path.join(self.stage_dir, filename)
-       return filename
+    def _get_test_file(self, filename):
+        # get a file inside the test input directory
+        filename = os.path.join(self.test_dir, filename)
+        assert os.path.exists(filename)
+        return filename
 
-   def _run(self, module_name, module_args, background=0):
-       ''' run a module and get the localhost results '''
-       self.runner.module_name = module_name
-       args = ' '.join(module_args)
-       print "DEBUG: using args=%s" % args
-       self.runner.module_args = args
-       self.runner.background  = background
-       results = self.runner.run()
-       # when using nosetests this will only show up on failure
-       # which is pretty useful
-       print "RESULTS=%s" % results
-       assert "127.0.0.2" in results['contacted']
-       return results['contacted']['127.0.0.2'] 
+    def _get_stage_file(self, filename):
+        # get a file inside the test output directory
+        filename = os.path.join(self.stage_dir, filename)
+        return filename
 
-   def test_ping(self):
-       result = self._run('ping',[])
-       assert "ping" in result
+    def _run(self, module_name, module_args, background=0):
+        ''' run a module and get the localhost results '''
+        self.runner.module_name = module_name
+        args = ' '.join(module_args)
+        print "DEBUG: using args=%s" % args
+        self.runner.module_args = args
+        self.runner.background = background
+        results = self.runner.run()
+        # when using nosetests this will only show up on failure
+        # which is pretty useful
+        print "RESULTS=%s" % results
+        assert "127.0.0.2" in results['contacted']
+        return results['contacted']['127.0.0.2']
 
-   def test_facter(self):
-       if not get_binary("facter"):
-           raise SkipTest
-       result = self._run('facter',[])
-       assert "hostname" in result
+    def test_ping(self):
+        result = self._run('ping', [])
+        assert "ping" in result
 
-   # temporarily disbabled since it occasionally hangs
-   # ohai's fault, setup module doesn't actually run this
-   # to get ohai's "facts" anyway
-   #
-   #def test_ohai(self):
-   #    if not get_binary("facter"):
-   #            raise SkipTest
-   #    result = self._run('ohai',[])
-   #    assert "hostname" in result
+    def test_facter(self):
+        if not get_binary("facter"):
+            raise SkipTest
+        result = self._run('facter', [])
+        assert "hostname" in result
 
-   def test_copy(self):
-       # test copy module, change trigger, etc
-       pass
+    # temporarily disbabled since it occasionally hangs
+    # ohai's fault, setup module doesn't actually run this
+    # to get ohai's "facts" anyway
+    #
+    #def test_ohai(self):
+    #    if not get_binary("facter"):
+    #            raise SkipTest
+    #    result = self._run('ohai',[])
+    #    assert "hostname" in result
 
-   def test_copy(self):
-       input = self._get_test_file('sample.j2')
-       output = self._get_stage_file('sample.out')
-       assert not os.path.exists(output)
-       result = self._run('copy', [
-           "src=%s" % input,
-           "dest=%s" % output,
-       ])
-       assert os.path.exists(output)
-       data_in = file(input).read()
-       data_out = file(output).read()
-       assert data_in == data_out
-       assert 'failed' not in result
-       assert result['changed'] == True
-       assert 'md5sum' in result
-       result = self._run('copy', [
-           "src=%s" % input,
-           "dest=%s" % output,
-       ])
-       assert result['changed'] == False
+    def test_copy(self):
+        # test copy module, change trigger, etc
+        input_ = self._get_test_file('sample.j2')
+        output = self._get_stage_file('sample.out')
+        assert not os.path.exists(output)
+        result = self._run('copy', [
+            "src=%s" % input_,
+            "dest=%s" % output,
+        ])
+        assert os.path.exists(output)
+        data_in = file(input_).read()
+        data_out = file(output).read()
+        assert data_in == data_out
+        assert 'failed' not in result
+        assert result['changed'] == True
+        assert 'md5sum' in result
+        result = self._run('copy', [
+            "src=%s" % input_,
+            "dest=%s" % output,
+        ])
+        assert result['changed'] == False
 
-   def test_template(self):
-       input = self._get_test_file('sample.j2')
-       metadata = self._get_test_file('metadata.json')
-       output = self._get_stage_file('sample.out')
-       result = self._run('template', [
-           "src=%s" % input,
-           "dest=%s" % output,
-           "metadata=%s" % metadata
-       ])
-       assert os.path.exists(output)
-       out = file(output).read()
-       assert out.find("duck") != -1
-       assert result['changed'] == True
-       assert 'md5sum' in result
-       assert 'failed' not in result
-       result = self._run('template', [
-           "src=%s" % input,
-           "dest=%s" % output,
-           "metadata=%s" % metadata
-       ])
-       assert result['changed'] == False
+    def test_template(self):
+        input_ = self._get_test_file('sample.j2')
+        metadata = self._get_test_file('metadata.json')
+        output = self._get_stage_file('sample.out')
+        result = self._run('template', [
+            "src=%s" % input_,
+            "dest=%s" % output,
+            "metadata=%s" % metadata
+        ])
+        assert os.path.exists(output)
+        out = file(output).read()
+        assert out.find("duck") != -1
+        assert result['changed'] == True
+        assert 'md5sum' in result
+        assert 'failed' not in result
+        result = self._run('template', [
+            "src=%s" % input_,
+            "dest=%s" % output,
+            "metadata=%s" % metadata
+        ])
+        assert result['changed'] == False
 
-   def test_command(self):
-   
-       # test command module, change trigger, etc
-       result = self._run('command', [ "/bin/echo", "hi" ])
-       assert "failed" not in result
-       assert "msg" not in result
-       assert result['rc'] == 0
-       assert result['stdout'] == 'hi'
-       assert result['stderr'] == ''
-   
-       result = self._run('command', [ "/bin/false" ])
-       assert result['rc'] == 1
-       assert 'failed' not in result
-   
-       result = self._run('command', [ "/usr/bin/this_does_not_exist", "splat" ]) 
-       assert 'msg' in result
-       assert 'failed' in result
-       assert 'rc' not in result
+    def test_command(self):
+        # test command module, change trigger, etc
+        result = self._run('command', [ "/bin/echo", "hi" ])
+        assert "failed" not in result
+        assert "msg" not in result
+        assert result['rc'] == 0
+        assert result['stdout'] == 'hi'
+        assert result['stderr'] == ''
 
-       result = self._run('shell', [ "/bin/echo", "$HOME" ])
-       assert 'failed' not in result
-       assert result['rc'] == 0 
-   
+        result = self._run('command', [ "/bin/false" ])
+        assert result['rc'] == 1
+        assert 'failed' not in result
 
-   def test_setup(self):
-       output = self._get_stage_file('output.json')
-       result = self._run('setup', [ "metadata=%s" % output, "a=2", "b=3", "c=4" ])
-       assert 'failed' not in result
-       assert 'md5sum' in result
-       assert result['changed'] == True
-       outds = json.loads(file(output).read())
-       assert outds['c'] == '4'
-       # not bothering to test change hooks here since ohai/facter results change
-       # almost every time so changed is always true, this just tests that
-       # rewriting the file is ok
-       result = self._run('setup', [ "metadata=%s" % output, "a=2", "b=3", "c=4" ])
-       print "RAW RESULT=%s" % result
-       assert 'md5sum' in result
+        result = self._run('command', [ "/usr/bin/this_does_not_exist", "splat" ])
+        assert 'msg' in result
+        assert 'failed' in result
+        assert 'rc' not in result
 
-   def test_async(self):
-       # test async launch and job status
-       # of any particular module
-       result = self._run('command', [ get_binary("sleep"), "3" ], background=20)
-       assert 'ansible_job_id' in result
-       assert 'started' in result
-       jid = result['ansible_job_id']
-       # no real chance of this op taking a while, but whatever
-       time.sleep(5)
-       # CLI will abstract this (when polling), but this is how it works internally
-       result = self._run('async_status', [ "jid=%s" % jid ])
-       # TODO: would be nice to have tests for supervisory process
-       # killing job after X seconds
-       assert 'finished' in result
-       assert 'failed' not in result
-       assert 'rc' in result
-       assert 'stdout' in result
-       assert result['ansible_job_id'] == jid
+        result = self._run('shell', [ "/bin/echo", "$HOME" ])
+        assert 'failed' not in result
+        assert result['rc'] == 0
 
-   def test_fetch(self):
-       input = self._get_test_file('sample.j2')
-       output = os.path.join(self.stage_dir, '127.0.0.2', input)
-       result = self._run('fetch', [ "src=%s" % input, "dest=%s" % self.stage_dir ])
-       assert os.path.exists(output)
-       assert open(input).read() == open(output).read()
+    def test_large_output(self):
+        large_path = "/usr/share/dict/words"
+        if not os.path.exists(large_path):
+            large_path = "/usr/share/dict/cracklib-small"
+            if not os.path.exists(large_path):
+                raise SkipTest
+        # Ensure reading a large amount of output from a command doesn't hang.
+        result = self._run('command', [ "/bin/cat", large_path ])
+        assert "failed" not in result
+        assert "msg" not in result
+        assert result['rc'] == 0
+        assert len(result['stdout']) > 100000
+        assert result['stderr'] == ''
 
-   def test_yum(self):
-       if not get_binary("yum"):
-           raise SkipTest
-       result = self._run('yum', [ "list=repos" ])
-       assert 'failed' not in result
+    def test_setup(self):
+        output = self._get_stage_file('output.json')
+        result = self._run('setup', [ "metadata=%s" % output, "a=2", "b=3", "c=4" ])
+        assert 'failed' not in result
+        assert 'md5sum' in result
+        assert result['changed'] == True
+        outds = json.loads(file(output).read())
+        assert outds['c'] == '4'
+        # not bothering to test change hooks here since ohai/facter results change
+        # almost every time so changed is always true, this just tests that
+        # rewriting the file is ok
+        result = self._run('setup', [ "metadata=%s" % output, "a=2", "b=3", "c=4" ])
+        print "RAW RESULT=%s" % result
+        assert 'md5sum' in result
 
-   def test_git(self):
-       # TODO: tests for the git module
-       pass
+    def test_async(self):
+        # test async launch and job status
+        # of any particular module
+        result = self._run('command', [ get_binary("sleep"), "3" ], background=20)
+        assert 'ansible_job_id' in result
+        assert 'started' in result
+        jid = result['ansible_job_id']
+        # no real chance of this op taking a while, but whatever
+        time.sleep(5)
+        # CLI will abstract this (when polling), but this is how it works internally
+        result = self._run('async_status', [ "jid=%s" % jid ])
+        # TODO: would be nice to have tests for supervisory process
+        # killing job after X seconds
+        assert 'finished' in result
+        assert 'failed' not in result
+        assert 'rc' in result
+        assert 'stdout' in result
+        assert result['ansible_job_id'] == jid
 
-   def test_service(self):
-       # TODO: tests for the service module
-       pass
+    def test_fetch(self):
+        input_ = self._get_test_file('sample.j2')
+        output = os.path.join(self.stage_dir, '127.0.0.2', input_)
+        result = self._run('fetch', [ "src=%s" % input_, "dest=%s" % self.stage_dir ])
+        assert os.path.exists(output)
+        assert open(input_).read() == open(output).read()
+
+    def test_yum(self):
+        if not get_binary("yum"):
+            raise SkipTest
+        result = self._run('yum', [ "list=repos" ])
+        assert 'failed' not in result
+
+    def test_git(self):
+        # TODO: tests for the git module
+        pass
+
+    def test_service(self):
+        # TODO: tests for the service module
+        pass
+    def test_assemble(self):
+        input = self._get_test_file('assemble.d')
+        metadata = self._get_test_file('metadata.json')
+        output = self._get_stage_file('sample.out')
+        result = self._run('assemble', [
+            "src=%s" % input,
+            "dest=%s" % output,
+            "metadata=%s" % metadata
+        ])
+        assert os.path.exists(output)
+        out = file(output).read()
+        assert out.find("first") != -1
+        assert out.find("second") != -1
+        assert out.find("third") != -1
+        assert result['changed'] == True
+        assert 'md5sum' in result
+        assert 'failed' not in result
+        result = self._run('assemble', [
+            "src=%s" % input,
+            "dest=%s" % output,
+            "metadata=%s" % metadata
+        ])
+        assert result['changed'] == False
 
 

test/TestUtils.py

@@ -0,0 +1,252 @@
+# -*- coding: utf-8 -*-
+
+import os
+import unittest
+
+import ansible.utils
+
+class TestUtils(unittest.TestCase):
+
+    #####################################
+    ### varLookup function tests
+
+    def test_varLookup_list(self):
+        vars = {
+            'data': {
+                'who': ['joe', 'jack', 'jeff']
+            }
+        }
+
+        res = ansible.utils._varLookup('data.who', vars)
+
+        assert sorted(res) == sorted(vars['data']['who'])
+
+    #####################################
+    ### varReplace function tests
+
+    def test_varReplace_simple(self):
+        template = 'hello $who'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_varReplace_multiple(self):
+        template = '$what $who'
+        vars = {
+            'what': 'hello',
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_varReplace_caps(self):
+        template = 'hello $whoVar'
+        vars = {
+            'whoVar': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+        print res
+        assert res == 'hello world'
+
+    def test_varReplace_middle(self):
+        template = 'hello $who!'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world!'
+
+    def test_varReplace_alternative(self):
+        template = 'hello ${who}'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_varReplace_almost_alternative(self):
+        template = 'hello $who}'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world}'
+
+    def test_varReplace_almost_alternative2(self):
+        template = 'hello ${who'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == template
+
+    def test_varReplace_alternative_greed(self):
+        template = 'hello ${who} }'
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world }'
+
+    def test_varReplace_notcomplex(self):
+        template = 'hello $mydata.who'
+        vars = {
+            'data': {
+                'who': 'world',
+            },
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        print res
+        assert res == template
+
+    def test_varReplace_nested(self):
+        template = 'hello ${data.who}'
+        vars = {
+            'data': {
+                'who': 'world'
+            },
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_varReplace_nested_int(self):
+        template = '$what ${data.who}'
+        vars = {
+            'data': {
+                'who': 2
+            },
+            'what': 'hello',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello 2'
+
+    def test_varReplace_unicode(self):
+        template = 'hello $who'
+        vars = {
+            'who': u'wórld',
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == u'hello wórld'
+
+    def test_varReplace_list(self):
+        template = 'hello ${data[1]}'
+        vars = {
+            'data': [ 'no-one', 'world' ]
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_varReplace_invalid_list(self):
+        template = 'hello ${data[1}'
+        vars = {
+            'data': [ 'no-one', 'world' ]
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == template
+
+    def test_varReplace_list_oob(self):
+        template = 'hello ${data[2]}'
+        vars = {
+            'data': [ 'no-one', 'world' ]
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == template
+
+    def test_varReplace_list_nolist(self):
+        template = 'hello ${data[1]}'
+        vars = {
+            'data': { 'no-one': 0, 'world': 1 }
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == template
+
+    def test_varReplace_nested_list(self):
+        template = 'hello ${data[1].msg[0]}'
+        vars = {
+            'data': [ 'no-one', {'msg': [ 'world'] } ]
+        }
+
+        res = ansible.utils.varReplace(template, vars)
+
+        assert res == 'hello world'
+
+    def test_template_varReplace_iterated(self):
+        template = 'hello $who'
+        vars = {
+            'who': 'oh great $person',
+            'person': 'one',
+        }
+
+        res = ansible.utils.template(template, vars)
+
+        assert res == u'hello oh great one'
+
+    #####################################
+    ### Template function tests
+
+    def test_template_basic(self):
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.template_from_file("test", "template-basic", vars, {})
+
+        assert res == 'hello world'
+
+    def test_template_whitespace(self):
+        vars = {
+            'who': 'world',
+        }
+
+        res = ansible.utils.template_from_file("test", "template-whitespace", vars, {})
+
+        assert res == 'hello world\n'
+
+    def test_template_unicode(self):
+        vars = {
+            'who': u'wórld',
+        }
+
+        res = ansible.utils.template_from_file("test", "template-basic", vars, {})
+
+        assert res == u'hello wórld'
+
+    #####################################
+    ### key-value parsing
+
+    def test_parse_kv_basic(self):
+        assert (ansible.utils.parse_kv('a=simple b="with space" c="this=that"') ==
+                {'a': 'simple', 'b': 'with space', 'c': 'this=that'})

test/assemble.d/00-source.txt

@@ -0,0 +1,2 @@
+# This is the first comment.
+[somegroup]

test/assemble.d/01-source.txt

@@ -0,0 +1,2 @@
+# This is the second comment.
+127.0.0.2

test/assemble.d/02.source.txt

@@ -0,0 +1,4 @@
+# This is the third comment.
+[somegroup:vars]
+foo=bar
+# 無 可 無 非 常 無

test/common_vars.yml

@@ -1,3 +1,4 @@
 ---
 duck: quack
 cow: moo
+extguard: " '$favcolor' == 'blue' "

test/complex_hosts

@@ -0,0 +1,76 @@
+# order of groups, children, and vars is not signficant
+# so this example mixes them up for maximum testing
+
+[nc:children]
+rtp
+triangle
+
+[eastcoast:children]
+nc
+florida
+
+[us:children]
+eastcoast
+
+[redundantgroup]
+rtp_a
+
+[redundantgroup2]
+rtp_a
+
+[redundantgroup3:children]
+rtp
+
+[redundantgroup:vars]
+rga=1
+
+[redundantgroup2:vars]
+rgb=2
+
+[redundantgroup3:vars]
+rgc=3
+
+[nc:vars]
+b=10000
+c=10001
+d=10002
+
+[rtp]
+rtp_a
+rtp_b
+rtb_c
+
+[rtp:vars]
+a=1
+b=2
+c=3
+
+[triangle]
+tri_a
+tri_b
+tri_c
+
+[triangle:vars]
+a=11
+b=12
+c=13
+
+[florida]
+orlando
+miami
+
+[florida:vars]
+a=100
+b=101
+c=102
+
+
+[eastcoast:vars]
+b=100000
+c=100001
+d=100002
+
+[us:vars]
+c=1000000
+
+

test/hosts_list.yml

@@ -0,0 +1,5 @@
+# Test that playbooks support YAML lists of hosts.
+---
+- hosts: [host1, host2, host3]
+  tasks:
+  - action: command true

test/playbook2.yml

@@ -1,71 +0,0 @@
----
-# this is an annotated example of some features available in playbooks
-# it shows how to make sure packages are updated, how to make sure
-# services are running, and how to template files.  It also demos
-# change handlers that can restart things (or trigger other actions)
-# when resources change.  For more advanced examples, see example2.yml
-
-# on all hosts, run as the user root...
-
-- hosts: all
-  user: root
-
-  # make these variables available inside of templates
-  # for when we use the 'template' action/module later on...
-
-  vars:
-    http_port: 80
-    max_clients: 200
-
-  # define the tasks that are part of this play...
-
-  tasks:
-
-  # task #1 is to run an arbitrary command
-  # we'll simulate a long running task, wait for up to 45 seconds, poll every 5
-  # obviously this does nothing useful but you get the idea
-
-  - name: longrunner
-    action: command /bin/sleep 15
-    async: 45
-    poll: 5
-
-  # let's demo file operations.
-  #
-  # We can 'copy' files or 'template' them instead, using jinja2
-  # as the templating engine.  This is done using the variables 
-  # from the vars section above mixed in with variables bubbled up 
-  # automatically from tools like facter and ohai.  'copy' 
-  # works just like 'template' but does not do variable subsitution.
-  #
-  # If and only if the file changes, restart apache at the very
-  # end of the playbook run
-
-  - name: write some_random_foo configuration
-    action: template src=templates/foo.j2 dest=/etc/some_random_foo.conf
-    notify:
-    - restart apache
-
-  # make sure httpd is installed at the latest version
-
-  - name: install httpd
-    action: yum pkg=httpd state=latest
-
-  # make sure httpd is running
-
-  - name: httpd start
-    action: service name=httpd state=running
-
-  # handlers are only run when things change, at the very end of each
-  # play.  Let's define some.  The names are significant and must
-  # match the 'notify' sections above
-
-  handlers:
-
-    # this particular handler is run when some_random_foo.conf
-    # is changed, and only then
-
-    - name: restart apache
-      action: service name=httpd state=restarted
-
-

test/playbook3.yml

@@ -1,71 +0,0 @@
----
-# this is an annotated example of some features available in playbooks
-# it shows how to make sure packages are updated, how to make sure
-# services are running, and how to template files.  It also demos
-# change handlers that can restart things (or trigger other actions)
-# when resources change.  For more advanced examples, see example2.yml
-
-# on all hosts, run as the user root...
-
-- hosts: all
-  user: root
-
-  # make these variables available inside of templates
-  # for when we use the 'template' action/module later on...
-
-  vars:
-    http_port: 80
-    max_clients: 200
-
-  # define the tasks that are part of this play...
-
-  tasks:
-
-  # task #1 is to run an arbitrary command
-  # we'll simulate a long running task, wait for up to 45 seconds, poll every 5
-  # obviously this does nothing useful but you get the idea
-
-  - name: longrunner
-    action: command /bin/sleep 15
-    async: 45
-    poll: 5
-
-  # let's demo file operations.
-  #
-  # We can 'copy' files or 'template' them instead, using jinja2
-  # as the templating engine.  This is done using the variables 
-  # from the vars section above mixed in with variables bubbled up 
-  # automatically from tools like facter and ohai.  'copy' 
-  # works just like 'template' but does not do variable subsitution.
-  #
-  # If and only if the file changes, restart apache at the very
-  # end of the playbook run
-
-  - name: write some_random_foo configuration
-    action: template src=templates/foo.j2 dest=/etc/some_random_foo.conf
-    notify:
-    - restart apache
-
-  # make sure httpd is installed at the latest version
-
-  - name: install httpd
-    action: yum pkg=httpd state=latest
-
-  # make sure httpd is running
-
-  - name: httpd start
-    action: service name=httpd state=running
-
-  # handlers are only run when things change, at the very end of each
-  # play.  Let's define some.  The names are significant and must
-  # match the 'notify' sections above
-
-  handlers:
-
-    # this particular handler is run when some_random_foo.conf
-    # is changed, and only then
-
-    - name: restart apache
-      action: service name=httpd state=restarted
-
-

test/template-basic

@@ -0,0 +1 @@
+hello {{ who }}

test/template-whitespace

@@ -0,0 +1 @@
+hello {{ who }}

test/yaml_hosts

@@ -6,7 +6,11 @@
     moon: titan
     moon2: enceladus
 
-- zeus
+- host: mars
+
+- host: zeus
+  vars:
+  - ansible_ssh_port: 3001
 
 - group: greek
   hosts: