Don't load plugins starting with "_".

Fixed a parsing HEAD problem, when ansible is checked out as a submodule

CHANGELOG.md

@@ -1,20 +1,212 @@
 Ansible Changes By Release
 ==========================
 
-0.4 "Unchained" ------- in progress, ETA May 2012
+0.7 "Panama" -- release pending -- I can barely see the roadmap from the heat coming off of it.
 
-* custom facts provided by the setup module mean no dependency on Ruby, facter, or ohai
+Module changes:
+
+* login_unix_socket option for mysql user and database modules (see PR #781 for doc notes)
+* new modules -- pip, easy_install, apt_repository, supervisorctl
+* error handling for setup module when SELinux is in a weird state
+* misc yum module fixes
+* better changed=True/False detection in user module on older Linux distros
+* nicer errors from modules when arguments are not key=value
+* backup option on copy (backup=yes), as well as template, assemble, and lineinfile
+* file module will not recurse on directory properties
+* yum module now workable without having repoquery installed, but doesn't support comparisons or list= if so
+* setup module now detects interfaces with aliases
+* better handling of VM guest type detection in setup module
+* new module boilerplate code to check for mutually required arguments, arguments required together, exclusive args
+* add pattern= as a paramter to the service module (for init scripts that don't do status, or do poor status)
+* various fixes to mysql & postresql modules
+* added a thirsty= option (boolean, default no) to the get_url module to decide to download the file every time or not
+* added a wait_for module to poll for ports being open
+* added a nagios module for controlling outage windows and alert statuses
+* added a seboolean module for getsebool/setsebool type operations
+* added a selinux module for controlling overall SELinux policy
+* added a subversion module
+* added lineinfile for adding and removing lines from basic files
+* added facts for ARM-based CPUs
+* support for systemd in the service module
+* git moduleforce reset behavior is now controllable
+* file module can now operate on special files (block devices, etc)
+
+Core changes:
+
+* ansible --version will now give branch/SHA information if running from git
+* better sudo permissions when encountering different umasks
+* when using paramiko and SFTP is not accessible, do not traceback, but return a nice human readable msg
+* use -vvv for extreme debug levels. -v gives more playbook output as before
+* -vv shows module arguments to all module calls (and maybe some other things later)
+* don not pass "--" to sudo to work on older EL5
+* make remote_md5 internal function work with non-bash shells
+* allow user to be passed in via --extra-vars (regression)
+* add --limit option, which can be used to further confine the pattern given in ansible-playbooks
+* adds ranged patterns like dbservers[0-49] for usage with patterns or --limit
+* -u and user: defaults to current user, rather than root, override as before
+* /etc/ansible/ansible.cfg and ~/ansible.cfg now available to set default values and other things
+* (developers) ANSIBLE_KEEP_REMOTE_FILES=1 can be used in debugging (envrionment variable)
+* (developers) connection types are now plugins
+* (developers) callbacks can now be extended via plugins
+* added FreeBSD ports packaging scripts
+* check for terminal properties prior to engaging color modes
+* explicitly disable password auth with -c ssh, as it is not used anyway
+
+Playbooks:
+
+* YAML syntax errors detected and show where the problem is
+* if you ctrl+c a playbook it will not traceback (usually)
+* vars_prompt now has encryption options (see examples/playbooks/prompts.yml)
+* allow variables in parameterized task include parameters (regression)
+* add ability to store the result of any command in a register (see examples/playbooks/register_logic.yml)
+* --list-hosts to show what hosts are included in each play of a playbook
+* fix a variable ordering issue that could affect vars_files with selective file source lists
+* adds 'delegate_to' for a task, which can be used to signal outage windows and load balancers on behalf of hosts
+* adds 'serial' to playbook, allowing you to specify how many hosts can be processing a playbook at one time (default 0=all)
+* adds 'local_action: <action parameters>' as an alias to 'delegate_to: 127.0.0.1'
+
+0.6 "Cabo" -- August 6, 2012
+
+playbooks:
+
+* support to tag tasks and includes and use --tags in playbook CLI
+* playbooks can now include other playbooks (example/playbooks/nested_playbooks.yml)
+* vars_files now usable with with_items, provided file paths don't contain host specific facts
+* error reporting if with_items value is unbound
+* with_items no longer creates lots of tasks, creates one task that makes multiple calls
+* can use host_specific facts inside with_items (see above)
+* at the top level of a playbook, set 'gather_facts: False' to skip fact gathering
+* first_available_file and with_items used together will now raise an error
+* to catch typos, like 'var' for 'vars', playbooks and tasks now yell on invalid parameters
+* automatically load (directory_of_inventory_file)/group_vars/groupname and /host_vars/hostname in vars_files
+* playbook is now colorized, set ANSIBLE_NOCOLOR=1 if you do not like this, does not colorize if not a TTY
+* hostvars now preserved between plays (regression in 0.5 from 0.4), useful for sharing vars in multinode configs
+* ignore_errors: True on a task can be used to allow a task to fail and not stop the play
+* with_items with the apt/yum module will install/remove/update everything in a single command
+
+inventory:
+
+* groups variable available as a hash to return the hosts in each group name
+* in YAML inventory, hosts can list their groups in inverted order now also (see tests/yaml_hosts)
+* YAML inventory is deprecated and will be removed in 0.7
+* ec2 inventory script
+* support ranges of hosts in the host file, like www[001-100].example.com (supports leading zeros and also not)
+
+modules:
+
+* fetch module now does not fail a system when requesting file paths (ex: logs) that don't exist
+* apt module now takes an optional install-recommends=yes|no (default yes)
+* fixes to the return codes of the copy module
+* copy module takes a remote md5sum to avoid large file transfer
+* various user and group module fixes (error handling, etc)
+* apt module now takes an optional force parameter
+* slightly better psychic service status handling for the service module
+* fetch module fixes for SSH connection type
+* modules now consistently all take yes/no for boolean parameters (and DWIM on true/false/1/0/y/n/etc)
+* setup module no longer saves to disk, template module now only used in playbooks
+* setup module no longer needs to run twice per playbook
+* apt module now passes DEBIAN_FRONTEND=noninteractive
+* mount module (manages active mounts + fstab)
+* setup module fixes if no ipv6 support
+* internals: template in common module boilerplate, also causes less SSH operations when used
+* git module fixes
+* setup module overhaul, more modular
+* minor caching logic added to inventory to reduce hammering of inventory scripts.
+* MySQL and PostgreSQL modules for user and db management
+* vars_prompt now supports private password entry (see examples/playbooks/prompts.yml)
+* yum module modified to be more tolerant of plugins spewing random console messages (ex: RHN)
+
+internals:
+
+* when sudoing to root, still use /etc/ansible/setup as the metadata path, as if root
+* paramiko is now only imported if needed when running from source checkout
+* cowsay support on Ubuntu
+* various ssh connection fixes for old Ubuntu clients
+* ./hacking/test-module now supports options like ansible takes and has a debugger mode
+* sudoing to a user other than root now works more seamlessly (uses /tmp, avoids umask issues)
+
+0.5 "Amsterdam" ------- July 04, 2012
+
+* Service module gets more accurate service states when running with upstart
+* Jinja2 usage in playbooks (not templates), reinstated, supports %include directive
+* support for --connection ssh (supports Kerberos, bastion hosts, etc), requires ControlMaster
+* misc tracebacks replaced with error messages
+* various API/internals refactoring
+* vars can be built from other variables
+* support for exclusion of hosts/groups with "!groupname"
+* various changes to support md5 tool differences for FreeBSD nodes & OS X clients
+* "unparseable" command output shows in command output for easier debugging
+* mktemp is no longer required on remotes (not available on BSD)
+* support for older versions of python-apt in the apt module
+* a new "assemble" module, for constructing files from pieces of files (inspired by Puppet "fragments" idiom)
+* ability to override most default values with ANSIBLE_FOO environment variables
+* --module-path parameter can support multiple directories seperated with the OS path seperator
+* with_items can take a variable of type list
+* ansible_python_interpreter variable available for systems with more than one Python
+* BIOS and VMware "fact" upgrades
+* cowsay is used by ansible-playbook if installed to improve output legibility (try installing it)
+* authorized_key module
+* SELinux facts now sourced from the python selinux library
+* removed module debug option -D
+* added --verbose, which shows output from successful playbook operations
+* print the output of the raw command inside /usr/bin/ansible as with command/shell
+* basic setup module support for Solaris
+* ./library relative to the playbook is always in path so modules can be included in tarballs with playbooks
+
+0.4 "Unchained" ------- May 23, 2012
+
+Internals/Core
+* internal inventory API now more object oriented, parsers decoupled
+* async handling improvements
+* misc fixes for running ansible on OS X (overlord only)
 * sudo improvements, now works much more smoothly
-* some support for reading some default settings from the ssh config file (key pairs, ports)
-* OS X support in progress for ansible on the host and clients (modules still TBA)
+* sudo to a particular user with -U/--sudo-user, or using 'sudo_user: foo' in a playbook
+* --private-key CLI option to work with pem files
+
+Inventory
+* can use -i host1,host2,host3:port to specify hosts not in inventory (replaces --override-hosts)
+* ansible INI style format can do groups of groups [groupname:children] and group vars [groupname:vars]
+* groups and users module takes an optional system=yes|no on creation (default no)
 * list of hosts in playbooks can be expressed as a YAML list in addition to ; delimited
-* tweaks to SELinux implementation for file module
-* first_available_file feature, see selective_file_sources.yml in examples/playbooks for info
-* fixes for yum module corner cases on EL5
-* --extra-vars="a=2 b=3" etc, now available to inject parameters into playbooks from CLI
-* file module now correctly returns the mode in octal
-* modules can no longer include stderr output (paramiko limitation from sudo)
+
+Playbooks
+* variables can be replaced like ${foo.nested_hash_key.nested_subkey[array_index]}
+* unicode now ok in templates (assumes utf8)
+* able to pass host specifier or group name in to "hosts:" with --extra-vars
+* ansible-pull script and example playbook (extreme scaling, remediation)
+* inventory_hostname variable available that contains the value of the host as ansible knows it
+* variables in the 'all' section can be used to define other variables based on those values
 * 'group_names' is now a variable made available to templates
+* first_available_file feature, see selective_file_sources.yml in examples/playbooks for info
+* --extra-vars="a=2 b=3" etc, now available to inject parameters into playbooks from CLI
+
+Incompatible Changes
+* jinja2 is only usable in templates, not playbooks, use $foo instead
+* --override-hosts removed, can use -i with comma notation (-i "ahost,bhost")
+* modules can no longer include stderr output (paramiko limitation from sudo)
+
+Module Changes
+* tweaks to SELinux implementation for file module
+* fixes for yum module corner cases on EL5
+* file module now correctly returns the mode in octal
+* fix for symlink handling in the file module
+* service takes an enable=yes|no which works with chkconfig or updates-rc.d as appropriate
+* service module works better on Ubuntu
+* git module now does resets and such to work more smoothly on updates
+* modules all now log to syslog
+* enabled=yes|no on a service can be used to toggle chkconfig & updates-rc.d states
+* git module supports branch=
+* service fixes to better detect status using return codes of the service script
+* custom facts provided by the setup module mean no dependency on Ruby, facter, or ohai
+* service now has a state=reloaded
+* raw module for bootstrapping and talking to routers w/o Python, etc
+
+Misc Bugfixes
+* fixes for variable parsing in only_if lines
+* misc fixes to key=value parsing
+* variables with mixed case now legal
+* fix to internals of hacking/test-module development script
+
 
 0.3 "Baluchitherium" -- April 23, 2012
 
@@ -44,8 +236,7 @@ in the playbook file
 in kickstarts
 * better module debugging with -D
 * fetch module for pulling in files from remote hosts
-* command task supports creates=foo for idempotent semantics, won't
-run if file foo already exists 
+* command task supports creates=foo for idempotent semantics, won't run if file foo already exists
 
 0.0.2 and 0.0.1
 

MANIFEST.in

@@ -1,5 +1,7 @@
 include README.md packaging/rpm/ansible.spec COPYING
 include examples/hosts
+include examples/ansible.cfg
+graft examples/playbooks
 include packaging/distutils/setup.py
 recursive-include docs *
 recursive-include library *

Makefile

@@ -1,12 +1,12 @@
 #!/usr/bin/make
-
+# WARN: gmake syntax
 ########################################################
 # Makefile for Ansible
 #
 # useful targets:
 #   make sdist ---------------- produce a tarball
 #   make rpm  ----------------- produce RPMs
-#   make debian --------------- produce a dpkg (FIXME?)
+#   make deb ------------------ produce a DEB
 #   make docs ----------------- rebuild the manpages (results are checked in)
 #   make tests ---------------- run the tests
 #   make pyflakes, make pep8 -- source code checks  
@@ -15,26 +15,40 @@
 # variable section
 
 NAME = "ansible"
+OS = $(shell uname -s)
 
 # Manpages are currently built with asciidoc -- would like to move to markdown
 # This doesn't evaluate until it's called. The -D argument is the
 # directory of the target file ($@), kinda like `dirname`.
 ASCII2MAN = a2x -D $(dir $@) -d manpage -f manpage $<
 ASCII2HTMLMAN = a2x -D docs/html/man/ -d manpage -f xhtml
-MANPAGES := docs/man/man1/ansible.1 docs/man/man1/ansible-playbook.1
+MANPAGES := docs/man/man1/ansible.1 docs/man/man1/ansible-playbook.1 docs/man/man1/ansible-pull.1
 
 SITELIB = $(shell python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")
 
 # VERSION file provides one place to update the software version
 VERSION := $(shell cat VERSION)
 
+### Get the branch information from git
+ifneq ($(shell which git),)
+GIT_DATE := $(shell git log -n 1 --format="%ai")
+endif
+
+ifeq ($(OS), FreeBSD)
+DATE := $(shell date -j -f "%Y-%m-%d %H:%M:%s"  "$(GIT_DATE)" +%Y%m%d%H%M)
+else
+DATE := $(shell date --date="$(GIT_DATE)" +%Y%m%d%H%M)
+endif
+
 # RPM build parameters
 RPMSPECDIR= packaging/rpm
 RPMSPEC = $(RPMSPECDIR)/ansible.spec
-RPMVERSION := $(shell awk '/Version/{print $$2; exit}' < $(RPMSPEC) | cut -d "%" -f1)
-RPMRELEASE := $(shell awk '/Release/{print $$2; exit}' < $(RPMSPEC) | cut -d "%" -f1)
 RPMDIST = $(shell rpm --eval '%dist')
-RPMNVR = "$(NAME)-$(RPMVERSION)-$(RPMRELEASE)$(RPMDIST)"
+RPMRELEASE = 1
+ifeq ($(OFFICIAL),)
+    RPMRELEASE = 0.git$(DATE)
+endif
+RPMNVR = "$(NAME)-$(VERSION)-$(RPMRELEASE)$(RPMDIST)"
 
 ########################################################
 
@@ -63,7 +77,8 @@ pep8:
 	@echo "#############################################"
 	@echo "# Running PEP8 Compliance Tests"
 	@echo "#############################################"
-	pep8 -r --ignore=E501,E221,W291,W391,E302,E251,E203,W293,E231,E303,E201,E225,E261 lib/ bin/
+	-pep8 -r --ignore=E501,E221,W291,W391,E302,E251,E203,W293,E231,E303,E201,E225,E261,E241 lib/ bin/
+	-pep8 -r --ignore=E501,E221,W291,W391,E302,E251,E203,W293,E231,E303,E201,E225,E261,E241 --filename "*" library/
 
 pyflakes:
 	pyflakes lib/ansible/*.py bin/*
@@ -92,8 +107,6 @@ python:
 	python setup.py build
 
 install:
-	mkdir -p /usr/share/ansible
-	cp ./library/* /usr/share/ansible/
 	python setup.py install
 
 sdist: clean
@@ -102,6 +115,7 @@ sdist: clean
 rpmcommon: sdist
 	@mkdir -p rpm-build
 	@cp dist/*.gz rpm-build/
+	@sed -e 's#^Version:.*#Version: $(VERSION)#' -e 's#^Release:.*#Release: $(RPMRELEASE)%{?dist}#' $(RPMSPEC) >rpm-build/$(NAME).spec
 
 srpm: rpmcommon
 	@rpmbuild --define "_topdir %(pwd)/rpm-build" \
@@ -110,7 +124,8 @@ srpm: rpmcommon
 	--define "_srcrpmdir %{_topdir}" \
 	--define "_specdir $(RPMSPECDIR)" \
 	--define "_sourcedir %{_topdir}" \
-	-bs $(RPMSPEC)
+	-bs rpm-build/$(NAME).spec
+	@rm -f rpm-build/$(NAME).spec
 	@echo "#############################################"
 	@echo "Ansible SRPM is built:"
 	@echo "    rpm-build/$(RPMNVR).src.rpm"
@@ -123,10 +138,12 @@ rpm: rpmcommon
 	--define "_srcrpmdir %{_topdir}" \
 	--define "_specdir $(RPMSPECDIR)" \
 	--define "_sourcedir %{_topdir}" \
-	-ba $(RPMSPEC)
+	--define "_rpmfilename $(RPMNVR).%%{ARCH}.rpm" \
+	-ba rpm-build/$(NAME).spec
+	@rm -f rpm-build/$(NAME).spec
 	@echo "#############################################"
 	@echo "Ansible RPM is built:"
-	@echo "    rpm-build/noarch/$(RPMNVR).noarch.rpm"
+	@echo "    rpm-build/$(RPMNVR).noarch.rpm"
 	@echo "#############################################"
 
 debian: sdist

README.md

@@ -1,10 +1,10 @@
 Ansible
 =======
 
-Ansible is a radically simple configuration-management, deployment, task-execution, and 
-multinode orchestration framework.   
+Ansible is a radically simple configuration-management, deployment, task-execution, and
+multinode orchestration framework.
 
-Read all about at it at (http://ansible.github.com)
+Read the documentation at http://ansible.github.com
 
 Design Principles
 =================
@@ -27,10 +27,11 @@ Get Involved
 Branch Info
 ===========
 
-   * The master branch corresponds to release 0.3 "Baluchitherium".
-   * Bug fixes WILL be periodically made to the master branch.
-   * The devel branch corresponds to release 0.4 "Unchained".
-   * Note: sudo support works much better in 0.4/devel, consider trying it.
+   * Releases are named after Van Halen songs.
+   * The devel branch corresponds to release 0.7, "Panama".
+   * Various release-X.Y branches exist for previous releases
+   * All feature work happens on the development branch.
+   * Major bug fixes will be made to the last release branch only
    * See CHANGELOG.md for release notes to track each release.
 
 Patch Instructions
@@ -40,18 +41,20 @@ Contributions to the core and modules are greatly welcome.
 
    * Required Process:
        * Submit github pull requests to the "ansible/devel" branch for features
-       * Fixes for bugs may be submitted to "ansible/master"
+       * Fixes for bugs may also be submitted to "ansible/release-X.Y" for the last release
        * Make sure "make tests" passes before submitting any requests.
    * Bonus points:
        * Joining the mailing list
+       * Fixing bugs instead of sending bug reports.
        * Using squash merges
        * Updating the "rst/*" files in the docs project and "docs/" manpage content
-       * Adding more unit tests 
+       * Adding more unit tests
    * Avoid:
        * Sending patches to the mailing list directly.
-       * Sending feature pull requests to the 'master' branch instead of the devel branch
+       * Sending feature pull requests to the 'release' branch instead of the devel branch
        * Sending pull requests to mpdehaan's personal ansible fork.
 
+
 Author
 ======
 

RELEASES.txt

@@ -0,0 +1,12 @@
+Ansible Releases at a Glance
+============================
+
+0.7 "Panama" ---------- X-XX-2012
+0.6 "Cabo" ------------ 8-06-2012
+0.5 "Amsterdam" ------- 7-04-2012
+0.4 "Unchained" ------- 5-23-2012
+0.3 "Baluchitherium" -- 4-23-2012
+0.0.2 Untitled
+0.0.1 Untitled
+
+

VERSION

@@ -1 +1 @@
-0.4
+0.7

bin/ansible

@@ -22,7 +22,7 @@
 import sys
 import getpass
 
-import ansible.runner
+from ansible.runner import Runner
 import ansible.constants as C
 from ansible import utils
 from ansible import errors
@@ -33,36 +33,35 @@ from ansible import inventory
 
 class Cli(object):
     ''' code behind bin/ansible '''
-    
-    # ----------------------------------------------     
+
+    # ----------------------------------------------
 
     def __init__(self):
         self.stats = callbacks.AggregateStats()
         self.callbacks = callbacks.CliRunnerCallbacks()
 
-    # ----------------------------------------------     
+    # ----------------------------------------------
 
     def parse(self):
         ''' create an options parser for bin/ansible '''
-        
-        parser = utils.base_parser(constants=C, runas_opts=True, async_opts=True,
+
+        parser = utils.base_parser(constants=C, runas_opts=True, subset_opts=True, async_opts=True,
             output_opts=True, connect_opts=True, usage='%prog <host-pattern> [options]')
         parser.add_option('-a', '--args', dest='module_args',
             help="module arguments", default=C.DEFAULT_MODULE_ARGS)
         parser.add_option('-m', '--module-name', dest='module_name',
-            help="module name to execute (default=%s)" % C.DEFAULT_MODULE_NAME, 
+            help="module name to execute (default=%s)" % C.DEFAULT_MODULE_NAME,
             default=C.DEFAULT_MODULE_NAME)
-
         options, args = parser.parse_args()
-        self.callbacks.options = options        
+        self.callbacks.options = options
 
         if len(args) == 0 or len(args) > 1:
             parser.print_help()
             sys.exit(1)
-        return (options, args)   
-    
-    # ----------------------------------------------     
- 
+        return (options, args)
+
+    # ----------------------------------------------
+
     def run(self, options, args):
         ''' use Runner lib to do SSH things '''
 
@@ -71,41 +70,45 @@ class Cli(object):
         inventory_manager = inventory.Inventory(options.inventory)
         hosts = inventory_manager.list_hosts(pattern)
         if len(hosts) == 0:
-             print >>sys.stderr, "No hosts matched"
-             sys.exit(1) 
-        
+            print >>sys.stderr, "No hosts matched"
+            sys.exit(1)
+
         sshpass = None
         sudopass = None
         if options.ask_pass:
             sshpass = getpass.getpass(prompt="SSH password: ")
         if options.ask_sudo_pass:
             sudopass = getpass.getpass(prompt="sudo password: ")
-
+            options.sudo = True
+        if options.sudo_user:
+            options.sudo = True
+        options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
         if options.tree:
             utils.prepare_writeable_dir(options.tree)
 
-        runner = ansible.runner.Runner(
+        runner = Runner(
             module_name=options.module_name, module_path=options.module_path,
             module_args=options.module_args,
             remote_user=options.remote_user, remote_pass=sshpass,
-            inventory=inventory_manager, timeout=options.timeout, 
-            forks=options.forks, 
-            pattern=pattern, 
-            callbacks=self.callbacks, sudo=options.sudo, 
-            sudo_pass=sudopass,
-            transport=options.connection, debug=options.debug
+            inventory=inventory_manager, timeout=options.timeout,
+            private_key_file=options.private_key_file,
+            forks=options.forks,
+            pattern=pattern,
+            callbacks=self.callbacks, sudo=options.sudo,
+            sudo_pass=sudopass,sudo_user=options.sudo_user,
+            transport=options.connection, subset=options.subset
         )
 
         if options.seconds:
             print "background launch...\n\n"
-            results, poller = runner.runAsync(options.seconds)
+            results, poller = runner.run_async(options.seconds)
             results = self.poll_while_needed(poller, options)
         else:
             results = runner.run()
 
         return (runner, results)
 
-    # ----------------------------------------------     
+    # ----------------------------------------------
 
     def poll_while_needed(self, poller, options):
         ''' summarize results from Runner '''

bin/ansible-playbook

@@ -18,6 +18,7 @@
 
 #######################################################
 
+import os
 import sys
 import getpass
 
@@ -26,17 +27,38 @@ import ansible.constants as C
 from ansible import errors
 from ansible import callbacks
 from ansible import utils
+from ansible.color import ANSIBLE_COLOR, stringc
+
+def colorize(lead, num, color):
+    """ Print 'lead' = 'num' in 'color' """
+    if num != 0 and ANSIBLE_COLOR:
+        return "%s%s%-15s" % (stringc(lead, color), stringc("=", color), stringc(str(num), color))
+    else:
+        return "%s=%-4s" % (lead, str(num))
+
+def hostcolor(host, stats):
+    if ANSIBLE_COLOR:
+        if stats['failures'] != 0 or stats['unreachable'] != 0:
+            return "%-41s" % stringc(host, 'red')
+        elif stats['changed'] != 0:
+            return "%-41s" % stringc(host, 'yellow')
+        else:
+            return "%-41s" % stringc(host, 'green')
+    return "%-30s" % host
+
 
 def main(args):
     ''' run ansible-playbook operations '''
 
     # create parser for CLI options
     usage = "%prog playbook.yml"
-    parser = utils.base_parser(constants=C, usage=usage, connect_opts=True, runas_opts=True)
-    parser.add_option('-O', '--override-hosts', dest="override_hosts", default=None,
-        help="run playbook against these hosts regardless of inventory settings")
+    parser = utils.base_parser(constants=C, usage=usage, connect_opts=True, runas_opts=True, subset_opts=True)
     parser.add_option('-e', '--extra-vars', dest="extra_vars", default=None,
         help="set additional key=value variables from the CLI")
+    parser.add_option('-t', '--tags', dest='tags', default='all',
+        help="only run plays and tasks tagged with these values")
+    parser.add_option('--list-hosts', dest='listhosts', action='store_true',
+        help="dump out a list of hosts, each play will run against, does not run playbook!")
 
     options, args = parser.parse_args(args)
 
@@ -46,50 +68,75 @@ def main(args):
 
     sshpass = None
     sudopass = None
-    if options.ask_pass:
-        sshpass = getpass.getpass(prompt="SSH password: ")
-    if options.ask_sudo_pass:
-        sudopass = getpass.getpass(prompt="sudo password: ")
-    override_hosts = None
-    if options.override_hosts:
-        override_hosts = options.override_hosts.split(",")
+    if not options.listhosts:
+        if options.ask_pass:
+            sshpass = getpass.getpass(prompt="SSH password: ")
+        if options.ask_sudo_pass:
+            sudopass = getpass.getpass(prompt="sudo password: ")
+            options.sudo = True
+        if options.sudo_user:
+            options.sudo = True
+        options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
     extra_vars = utils.parse_kv(options.extra_vars)
+    only_tags = options.tags.split(",")
 
     # run all playbooks specified on the command line
     for playbook in args:
 
         stats = callbacks.AggregateStats()
-        playbook_cb = callbacks.PlaybookCallbacks()
-        runner_cb = callbacks.PlaybookRunnerCallbacks(stats)
+        playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
+        runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)
 
         pb = ansible.playbook.PlayBook(
             playbook=playbook,
             module_path=options.module_path,
-            host_list=options.inventory, 
-            override_hosts=override_hosts,
-            forks=options.forks, 
-            debug=options.debug, 
+            host_list=options.inventory,
+            forks=options.forks,
             remote_user=options.remote_user,
-            remote_pass=sshpass, 
-            callbacks=playbook_cb, 
-            runner_callbacks=runner_cb, 
+            remote_pass=sshpass,
+            callbacks=playbook_cb,
+            runner_callbacks=runner_cb,
             stats=stats,
-            timeout=options.timeout, 
+            timeout=options.timeout,
             transport=options.connection,
             sudo=options.sudo,
+            sudo_user=options.sudo_user,
             sudo_pass=sudopass,
-            extra_vars=extra_vars
+            extra_vars=extra_vars,
+            private_key_file=options.private_key_file,
+            only_tags=only_tags,
+            subset=options.subset,
         )
+        
+        if options.listhosts:
+            playnum = 0
+            for play in pb.playbook:
+                playnum += 1
+                if 'hosts' in play:
+                    label = 'unnamed'
+                    if 'name' in play:
+                        label = play['name']
+                    print 'hosts in play %s: %s' % (playnum, label)
+                    for host in pb.inventory.list_hosts(play['hosts']):
+                        print '  %s' % host
+                    print '\n'
+            return 0
+            
         try:
 
             pb.run()
             hosts = sorted(pb.stats.processed.keys())
-            print "\n\nPLAY RECAP **********************\n\n"
+            print callbacks.banner("PLAY RECAP")
+            playbook_cb.on_stats(pb.stats)
             for h in hosts:
                 t = pb.stats.summarize(h)
-                print "%-30s : ok=%4s changed=%4s unreachable=%4s failed=%4s " % (h, 
-                   t['ok'], t['changed'], t['unreachable'], t['failures']
-                )
+                print "%-30s : %s %s %s %s " % (
+                    hostcolor(h, t),
+                    colorize('ok', t['ok'], 'green'),
+                    colorize('changed', t['changed'], 'yellow'),
+                    colorize('unreachable', t['unreachable'], 'red'),
+                    colorize('failed', t['failures'], 'red'))
+
             print "\n"
 
         except errors.AnsibleError, e:

bin/ansible-pull

@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+
+# ansible-pull is a script that runs ansible in local mode
+# after checking out a playbooks directory from git.  There is an
+# example playbook to bootstrap this script in the examples/ dir which
+# installs ansible and sets it up to run on cron.
+#
+# usage:
+#   ansible-pull -d /var/ansible/local -U http://wherever/content.git -C production
+#
+# the git repo must contain a playbook named 'local.yml'
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import subprocess
+import sys
+from optparse import OptionParser
+
+DEFAULT_PLAYBOOK = 'local.yml'
+
+def _run(cmd):
+    cmd = subprocess.Popen(cmd, shell=True,
+                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    print out
+    if cmd.returncode != 0:
+        print err
+    return cmd.returncode
+
+def main(args):
+    """ Set up and run a local playbook """
+    usage = "%prog [options]"
+    parser = OptionParser()
+    parser.add_option('-d', '--directory', dest='dest', default=None,
+                      help='Directory to checkout git repository')
+    parser.add_option('-U', '--url', dest='url',
+                      default=None,
+                      help='URL of git repository')
+    parser.add_option('-C', '--checkout', dest='checkout',
+                      default="HEAD",
+                      help='Branch/Tag/Commit to checkout.  Defaults to HEAD.')
+    options, args = parser.parse_args(args)
+
+    git_opts = "repo=%s dest=%s version=%s" % (options.url, options.dest, options.checkout)
+    cmd = 'ansible all -c local -m git -a "%s"' % git_opts
+    print "cmd=%s" % cmd
+    rc = _run(cmd)
+    if rc != 0:
+        return rc
+
+    os.chdir(options.dest)
+    cmd = 'ansible-playbook -c local %s' % DEFAULT_PLAYBOOK
+    rc = _run(cmd)
+    return rc
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except KeyboardInterrupt, e:
+        print >>sys.stderr, "Exit on user request.\n"
+        sys.exit(1)

docs/man/man1/ansible-playbook.1

@@ -2,12 +2,12 @@
 .\"     Title: ansible-playbook
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: 04/26/2012
+.\"      Date: 09/06/2012
 .\"    Manual: System administration commands
-.\"    Source: Ansible 0.4
+.\"    Source: Ansible 0.7
 .\"  Language: English
 .\"
-.TH "ANSIBLE\-PLAYBOOK" "1" "04/26/2012" "Ansible 0\&.4" "System administration commands"
+.TH "ANSIBLE\-PLAYBOOK" "1" "09/06/2012" "Ansible 0\&.7" "System administration commands"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -33,10 +33,11 @@ ansible\-playbook <filename\&.yml> \&... [options]
 The names of one or more YAML format files to run as ansible playbooks\&.
 .RE
 .SH "OPTIONS"
-.sp
-\fB\-D\fR, \fB\-\-debug\fR
-.sp
-Print any messages the remote module sends to standard error to the console
+.PP
+\fB\-v\fR, \fB\-\-verbose\fR
+.RS 4
+Verbose mode, more output from successful actions will be shown\&. Give up to three times for more output\&.
+.RE
 .PP
 \fB\-i\fR \fIPATH\fR, \fB\-\-inventory=\fR\fIPATH\fR
 .RS 4
@@ -73,7 +74,12 @@ Prompt for the SSH password instead of assuming key\-based authentication with s
 .PP
 \fB\-K\fR, \fB\-\-ask\-sudo\-pass\fR
 .RS 4
-Prompt for the password to use for playbook plays that request sudo access, if any
+Prompt for the password to use for playbook plays that request sudo access, if any\&.
+.RE
+.PP
+\fB\-U\fR, \fISUDO_USER\fR, \fB\-\-sudo\-user=\fR\fISUDO_USER\fR
+.RS 4
+Desired sudo user (default=root)\&.
 .RE
 .PP
 \fB\-T\fR \fISECONDS\fR, \fB\-\-timeout=\fR\fISECONDS\fR
@@ -82,13 +88,6 @@ Connection timeout to use when trying to talk to hosts, in
 \fISECONDS\fR\&.
 .RE
 .PP
-\fB\-O\fR \fIOVERRIDE_HOSTS\fR, \fB\-\-override\-hosts=\fR\fIOVERRIDE_HOSTS\fR
-.RS 4
-Ignore the inventory file and run the playbook against only these hosts\&. "hosts:" line in playbook should be set to
-\fIall\fR
-when using this option\&.
-.RE
-.PP
 \fB\-s\fR, \fB\-\-sudo\fR
 .RS 4
 Force all plays to use sudo, even if not marked as such\&.
@@ -98,6 +97,22 @@ Force all plays to use sudo, even if not marked as such\&.
 .RS 4
 Use this remote user name on playbook steps that do not indicate a user name to run as\&.
 .RE
+.PP
+\fB\-c\fR \fICONNECTION\fR, \fB\-\-connection=\fR\fICONNECTION\fR
+.RS 4
+Connection type to use\&. Possible options are
+\fIparamiko\fR
+(SSH),
+\fIssh\fR, and
+\fIlocal\fR\&.
+\fIlocal\fR
+is mostly useful for crontab or kickstarts\&.
+.RE
+.PP
+\fB\-l\fR \fISUBSET\fR, \fB\-\-limit=\fR\fISUBSET\fR
+.RS 4
+Further limits the selected host/group patterns\&.
+.RE
 .SH "ENVIRONMENT"
 .sp
 The following environment variables may specified\&.
@@ -115,6 +130,6 @@ Copyright \(co 2012, Michael DeHaan
 Ansible is released under the terms of the GPLv3 License\&.
 .SH "SEE ALSO"
 .sp
-\fBansible\fR(1)
+\fBansible\fR(1), \fBansible\-pull\fR(1)
 .sp
 Extensive documentation as well as IRC and mailing list info is available on the ansible home page: https://ansible\&.github\&.com/

docs/man/man1/ansible-playbook.1.asciidoc.in

@@ -34,13 +34,15 @@ The names of one or more YAML format files to run as ansible playbooks.
 OPTIONS
 -------
 
-*-D*, *--debug*          
+*-v*, *--verbose*::
 
-Debug mode
+Verbose mode, more output from successful actions will be shown. Give
+up to three times for more output.
 
 *-i* 'PATH', *--inventory=*'PATH'::
 
-The 'PATH' to the inventory hosts file, which defaults to '/etc/ansible/hosts'.
+The 'PATH' to the inventory hosts file, which defaults to
+'/etc/ansible/hosts'.
 
 
 *-M* 'DIRECTORY', *--module-path=*'DIRECTORY'::
@@ -58,25 +60,24 @@ Level of parallelism. 'NUM' is specified as an integer, the default is 5.
 
 *-k*, *--ask-pass*::
 
-Prompt for the SSH password instead of assuming key-based authentication with ssh-agent.
+Prompt for the SSH password instead of assuming key-based
+authentication with ssh-agent.
 
 
 *-K*, *--ask-sudo-pass*::
 
-Prompt for the password to use for playbook plays that request sudo access, if any
+Prompt for the password to use for playbook plays that request sudo
+access, if any.
 
+*-U*, 'SUDO_USER', *--sudo-user=*'SUDO_USER'::
+
+Desired sudo user (default=root).
 
 *-T* 'SECONDS', *--timeout=*'SECONDS'::
 
 Connection timeout to use when trying to talk to hosts, in 'SECONDS'.
 
 
-*-O* 'OVERRIDE_HOSTS', *--override-hosts=*'OVERRIDE_HOSTS'::
-
-Ignore the inventory file and run the playbook against only these hosts.  "hosts:" line
-in playbook should be set to 'all' when using this option.
-
-
 *-s*, *--sudo*::
 
 Force all plays to use sudo, even if not marked as such.
@@ -84,7 +85,17 @@ Force all plays to use sudo, even if not marked as such.
 
 *-u* 'USERNAME', *--remote-user=*'USERNAME'::
 
-Use this remote user name on playbook steps that do not indicate a user name to run as.
+Use this remote user name on playbook steps that do not indicate a
+user name to run as.
+
+*-c* 'CONNECTION', *--connection=*'CONNECTION'::
+
+Connection type to use.  Possible options are 'paramiko' (SSH), 'ssh',
+and 'local'.  'local' is mostly useful for crontab or kickstarts.
+
+*-l* 'SUBSET', *--limit=*'SUBSET'::
+
+Further limits the selected host/group patterns.
 
 
 ENVIRONMENT
@@ -115,10 +126,7 @@ Ansible is released under the terms of the GPLv3 License.
 SEE ALSO
 --------
 
-*ansible*(1)
+*ansible*(1), *ansible-pull*(1)
 
 Extensive documentation as well as IRC and mailing list info
 is available on the ansible home page: <https://ansible.github.com/>
-
-
-

docs/man/man1/ansible-pull.1

@@ -0,0 +1,73 @@
+'\" t
+.\"     Title: ansible
+.\"    Author: [see the "AUTHOR" section]
+.\" Generator: DocBook XSL Stylesheets v1.76.1 <http://docbook.sf.net/>
+.\"      Date: 08/14/2012
+.\"    Manual: System administration commands
+.\"    Source: Ansible 0.7
+.\"  Language: English
+.\"
+.TH "ANSIBLE" "1" "08/14/2012" "Ansible 0\&.7" "System administration commands"
+.\" -----------------------------------------------------------------
+.\" * Define some portability stuff
+.\" -----------------------------------------------------------------
+.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.\" http://bugs.debian.org/507673
+.\" http://lists.gnu.org/archive/html/groff/2009-02/msg00013.html
+.\" ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.ie \n(.g .ds Aq \(aq
+.el       .ds Aq '
+.\" -----------------------------------------------------------------
+.\" * set default formatting
+.\" -----------------------------------------------------------------
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.\" -----------------------------------------------------------------
+.\" * MAIN CONTENT STARTS HERE *
+.\" -----------------------------------------------------------------
+.SH "NAME"
+ansible-pull \- set up a remote copy of ansible on each managed node
+.SH "SYNOPSIS"
+.sp
+ansible \-d DEST \-U URL [ \-C CHECKOUT ]
+.SH "DESCRIPTION"
+.sp
+\fBAnsible\fR is an extra\-simple tool/framework/API for doing \*(Aqremote things\*(Aq over SSH\&.
+.sp
+Use ansible\-pull to set up a remote copy of ansible on each managed node, each set to run via cron and update playbook source via git\&. This inverts the default \fBpush\fR architecture of ansible into a \fBpull\fR architecture, which has near\-limitless scaling potential\&.
+.sp
+The setup playbook can be tuned to change the cron frequency, logging locations, and parameters to ansible\-pull\&.
+.sp
+This is useful both for extreme scale\-out as well as periodic remediation\&. Usage of the \fIfetch\fR module to retrieve logs from ansible\-pull runs would be an excellent way to gather and analyze remote logs from ansible\-pull\&.
+.SH "OPTIONS"
+.PP
+\fB\-d\fR \fIDEST\fR, \fB\-\-directory=\fR\fIDEST\fR
+.RS 4
+Directory to checkout git repository into\&.
+.RE
+.PP
+\fB\-U\fR \fIURL\fR, \fB\-\-url=\fR\fIURL\fR
+.RS 4
+URL of git repository to clone\&.
+.RE
+.PP
+\fB\-C\fR \fICHECKOUT\fR, \fB\-\-checkout=\fR\fICHECKOUT\fR
+.RS 4
+Branch/Tag/Commit to checkout\&. Defaults to
+\fIHEAD\fR\&.
+.RE
+.SH "AUTHOR"
+.sp
+Ansible was originally written by Michael DeHaan\&. See the AUTHORS file for a complete list of contributors\&.
+.SH "COPYRIGHT"
+.sp
+Copyright \(co 2012, Michael DeHaan
+.sp
+Ansible is released under the terms of the GPLv3 License\&.
+.SH "SEE ALSO"
+.sp
+\fBansible\fR(1), \fBansible\-playbook\fR(1)
+.sp
+Extensive documentation as well as IRC and mailing list info is available on the ansible home page: https://ansible\&.github\&.com/

docs/man/man1/ansible-pull.1.asciidoc.in

@@ -0,0 +1,76 @@
+ansible(1)
+=========
+:doctype:manpage
+:man source:   Ansible
+:man version:  %VERSION%
+:man manual:   System administration commands
+
+NAME
+----
+ansible-pull - set up a remote copy of ansible on each managed node
+
+
+SYNOPSIS
+--------
+ansible -d DEST -U URL [ -C CHECKOUT ]
+
+
+DESCRIPTION
+-----------
+
+*Ansible* is an extra-simple tool/framework/API for doing \'remote things' over
+SSH.
+
+Use ansible-pull to set up a remote copy of ansible on each managed
+node, each set to run via cron and update playbook source via
+git. This inverts the default *push* architecture of ansible into a
+*pull* architecture, which has near-limitless scaling potential.
+
+The setup playbook can be tuned to change the cron frequency, logging
+locations, and parameters to ansible-pull.
+
+This is useful both for extreme scale-out as well as periodic
+remediation. Usage of the 'fetch' module to retrieve logs from
+ansible-pull runs would be an excellent way to gather and analyze
+remote logs from ansible-pull.
+
+
+OPTIONS
+-------
+
+*-d* 'DEST', *--directory=*'DEST'::
+
+Directory to checkout git repository into.
+
+*-U* 'URL', *--url=*'URL'::
+
+URL of git repository to clone.
+
+*-C* 'CHECKOUT', *--checkout=*'CHECKOUT'::
+
+Branch/Tag/Commit to checkout.  Defaults to 'HEAD'.
+
+
+
+AUTHOR
+------
+
+Ansible was originally written by Michael DeHaan. See the AUTHORS file
+for a complete list of contributors.
+
+
+COPYRIGHT
+---------
+
+Copyright © 2012, Michael DeHaan
+
+Ansible is released under the terms of the GPLv3 License.
+
+
+SEE ALSO
+--------
+
+*ansible*(1), *ansible-playbook*(1)
+
+Extensive documentation as well as IRC and mailing list info
+is available on the ansible home page: <https://ansible.github.com/>

docs/man/man1/ansible.1

@@ -2,12 +2,12 @@
 .\"     Title: ansible
 .\"    Author: [see the "AUTHOR" section]
 .\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
-.\"      Date: 04/26/2012
+.\"      Date: 09/06/2012
 .\"    Manual: System administration commands
-.\"    Source: Ansible 0.4
+.\"    Source: Ansible 0.7
 .\"  Language: English
 .\"
-.TH "ANSIBLE" "1" "04/26/2012" "Ansible 0\&.4" "System administration commands"
+.TH "ANSIBLE" "1" "09/06/2012" "Ansible 0\&.7" "System administration commands"
 .\" -----------------------------------------------------------------
 .\" * set default formatting
 .\" -----------------------------------------------------------------
@@ -34,6 +34,11 @@ A name of a group in the inventory file, a shell\-like glob selecting hosts in i
 .RE
 .SH "OPTIONS"
 .PP
+\fB\-v\fR, \fB\-\-verbose\fR
+.RS 4
+Verbose mode, more output from successful actions will be shown\&. Give up to three times for more output\&.
+.RE
+.PP
 \fB\-i\fR \fIPATH\fR, \fB\-\-inventory=\fR\fIPATH\fR
 .RS 4
 The
@@ -49,6 +54,11 @@ Level of parallelism\&.
 is specified as an integer, the default is 5\&.
 .RE
 .PP
+\fB\-\-private\-key=\fR\fIPRIVATE_KEY_FILE\fR
+.RS 4
+Use this file to authenticate the connection\&.
+.RE
+.PP
 \fB\-m\fR \fINAME\fR, \fB\-\-module\-name=\fR\fINAME\fR
 .RS 4
 Execute the module called
@@ -70,11 +80,6 @@ The
 to pass to the module\&.
 .RE
 .PP
-\fB\-D\fR, \fB\-\-debug\fR
-.RS 4
-Print any messages the remote module sends to standard error to the console
-.RE
-.PP
 \fB\-k\fR, \fB\-\-ask\-pass\fR
 .RS 4
 Prompt for the SSH password instead of assuming key\-based authentication with ssh\-agent\&.
@@ -129,16 +134,38 @@ Use this remote
 instead of root\&.
 .RE
 .PP
+\fB\-U\fR \fISUDO_USERNAME\fR, \fB\-\-sudo\-user=\fR\fISUDO_USERNAME\fR
+.RS 4
+Sudo to
+\fISUDO_USERNAME\fR
+instead of root\&. Implies \-\-sudo\&.
+.RE
+.PP
+\fB\-\-private\-key=\fR\fIPRIVATE_KEY_FILE\fR
+.RS 4
+Use this file to authenticate the connection\&.
+.RE
+.PP
 \fB\-c\fR \fICONNECTION\fR, \fB\-\-connection=\fR\fICONNECTION\fR
 .RS 4
 Connection type to use\&. Possible options are
 \fIparamiko\fR
-(SSH) and
-\fIlocal\fR\&. Local is mostly useful for crontab or kickstarts\&.
+(SSH),
+\fIssh\fR, and
+\fIlocal\fR\&.
+\fIlocal\fR
+is mostly useful for crontab or kickstarts\&.
+.RE
+.PP
+\fB\-l\fR \fISUBSET\fR, \fB\-\-limit=\fR\fISUBSET\fR
+.RS 4
+Further limits the selected host/group patterns\&.
 .RE
 .SH "INVENTORY"
 .sp
-Ansible stores the hosts it can potentially operate on in an inventory file\&. The syntax is one host per line\&. Groups headers are allowed and are included on their own line, enclosed in square brackets\&.
+Ansible stores the hosts it can potentially operate on in an inventory file\&. The syntax is one host per line\&. Groups headers are allowed and are included on their own line, enclosed in square brackets that start the line\&.
+.sp
+Ranges of hosts are also supported\&. For more information and additional options, see the documentation on http://ansible\&.github\&.com/\&.
 .SH "FILES"
 .sp
 /etc/ansible/hosts \(em Default inventory file
@@ -161,6 +188,6 @@ Copyright \(co 2012, Michael DeHaan
 Ansible is released under the terms of the GPLv3 License\&.
 .SH "SEE ALSO"
 .sp
-\fBansible\-playbook\fR(1)
+\fBansible\-playbook\fR(1), \fBansible\-pull\fR(1)
 .sp
 Extensive documentation as well as IRC and mailing list info is available on the ansible home page: https://ansible\&.github\&.com/

docs/man/man1/ansible.1.asciidoc.in

@@ -31,10 +31,13 @@ A name of a group in the inventory file, a shell-like glob selecting
 hosts in inventory file, or any combination of the two separated by
 semicolons.
 
-
 OPTIONS
 -------
 
+*-v*, *--verbose*::
+
+Verbose mode, more output from successful actions will be shown. Give
+up to three times for more output.
 
 *-i* 'PATH', *--inventory=*'PATH'::
 
@@ -45,6 +48,10 @@ The 'PATH' to the inventory hosts file, which defaults to '/etc/ansible/hosts'.
 
 Level of parallelism. 'NUM' is specified as an integer, the default is 5.
 
+*--private-key=*'PRIVATE_KEY_FILE'::
+
+Use this file to authenticate the connection.
+
 
 *-m* 'NAME', *--module-name=*'NAME'::
 
@@ -60,10 +67,6 @@ The 'DIRECTORY' to load modules from. The default is '/usr/share/ansible'.
 
 The 'ARGUMENTS' to pass to the module.
 
-*-D*, *--debug*::
-
-Debug mode
-
 *-k*, *--ask-pass*::
 
 Prompt for the SSH password instead of assuming key-based authentication with ssh-agent.
@@ -101,18 +104,32 @@ Poll a background job every 'NUM' seconds. Requires *-B*.
 
 Use this remote 'USERNAME' instead of root.
 
+*-U* 'SUDO_USERNAME', *--sudo-user=*'SUDO_USERNAME'::
+
+Sudo to 'SUDO_USERNAME' instead of root.  Implies --sudo.
+
+*--private-key=*'PRIVATE_KEY_FILE'::
+Use this file to authenticate the connection.
+
 *-c* 'CONNECTION', *--connection=*'CONNECTION'::
 
-Connection type to use.  Possible options are 'paramiko' (SSH) and 'local'.
-Local is mostly useful for crontab or kickstarts.
+Connection type to use.  Possible options are 'paramiko' (SSH), 'ssh',
+and 'local'.  'local' is mostly useful for crontab or kickstarts.
 
+*-l* 'SUBSET', *--limit=*'SUBSET'::
+
+Further limits the selected host/group patterns.
 
 INVENTORY
 ---------
 
 Ansible stores the hosts it can potentially operate on in an inventory
 file. The syntax is one host per line.  Groups headers are allowed and
-are included on their own line, enclosed in square brackets.
+are included on their own line, enclosed in square brackets that
+start the line.
+
+Ranges of hosts are also supported.  For more information and
+additional options, see the documentation on http://ansible.github.com/.
 
 FILES
 -----
@@ -150,7 +167,7 @@ Ansible is released under the terms of the GPLv3 License.
 SEE ALSO
 --------
 
-*ansible-playbook*(1)
+*ansible-playbook*(1), *ansible-pull*(1)
 
 Extensive documentation as well as IRC and mailing list info
 is available on the ansible home page: <https://ansible.github.com/>

examples/ansible.cfg

@@ -0,0 +1,81 @@
+# config file for ansible -- http://ansible.github.com
+# nearly all parameters can be overridden in ansible-playbook or with command line flags
+# ansible will read ~/.ansible.cfg or /etc/ansible/ansible.cfg, whichever it finds first
+
+[defaults]
+
+# location of inventory file, eliminates need to specify -i
+
+hostfile = /etc/ansible/hosts
+
+# location of ansible library, eliminates need to specify --module-path
+
+library = /usr/share/ansible
+
+# default module name used in /usr/bin/ansible when -m is not specified
+
+module_name = command
+
+# home directory where temp files are stored on remote systems.  Should
+# almost always contain $HOME or be a directory writeable by all users
+
+remote_tmp = $HOME/.ansible/tmp
+
+# the default pattern for ansible-playbooks ("hosts:") 
+
+pattern = *
+
+# the default number of forks (parallelism) to be used.  Usually you
+# can crank this up.
+
+forks=5
+
+# the timeout used by various connection types.  Usually this corresponds
+# to an SSH timeout
+
+timeout=10
+
+# when using --poll or "poll:" in an ansible playbook, and not specifying
+# an explicit poll interval, use this interval
+
+poll_interval=15
+
+# when specifying --sudo to /usr/bin/ansible or "sudo:" in a playbook,
+# and not specifying "--sudo-user" or "sudo_user" respectively, sudo
+# to this user account
+
+sudo_user=root
+
+# connection to use when -c <connection_type> is not specified
+
+transport=paramiko
+
+# remote SSH port to be used when --port or "port:" or an equivalent inventory
+# variable is not specified.
+
+remote_port=22
+
+# if set, always run /usr/bin/ansible commands as this user, and assume this value
+# if "user:" is not set in a playbook.  If not set, use the current Unix user
+# as the default
+
+#remote_user=root
+
+# if set, always use this private key file for authentication, same as if passing
+# --private-key-file to ansible or ansible-playbook
+
+#private_key_file=/path/to/file
+
+[paramiko_connection]
+
+# nothing to configure yet
+
+[ssh_connection]
+
+# if uncommented, sets the ansible ssh arguments to the following.  Leaving off ControlPersist
+# will result in poor performance, so use transport=paramiko on older platforms rather than
+# removing it
+
+ssh_args=-o PasswordAuthentication=no -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r
+
+

examples/hosts

@@ -7,31 +7,38 @@
 #   - Groups of hosts are delimited by [header] elements
 #   - You can enter hostnames or ip addresses
 #   - A hostname/ip can be a member of multiple groups
-#
+
 # Ex 1: Ungrouped hosts, specify before any group headers.
-green.bikeshed.org
-blue.bikeshed.org
-red.bikeshed.org
-bikeshed.org
-bastion.secure.bikeshed.org
+
+green.example.com
+blue.example.com
 192.168.100.1
 192.168.100.10
 
 # Ex 2: A collection of hosts belonging to the 'webservers' group
+
 [webservers]
-www01.bikeshed.org
-www02.bikeshed.org
-wheel.colors.com
+alpha.example.org
+beta.example.org
 192.168.1.100
 192.168.1.110
-# Your personal website also runs a webserver:
-myserver.com
+
+# If you have multiple hosts following a pattern you can specify
+# them like this:
+
+www[001:006].example.com
 
 # Ex 3: A collection of database servers in the 'dbservers' group
+
 [dbservers]
+
 db01.intranet.mydomain.net
-10.25.1.56
 db02.intranet.mydomain.net
+10.25.1.56
 10.25.1.57
-# Perhaps you serve a db off your personal server too:
-myserver.com
+
+# Here's another example of host ranges, this time there are no
+# leading 0s:
+
+db-[99:101]-node.example.com
+

examples/playbooks/ansible_pull.yml

@@ -0,0 +1,51 @@
+# ansibple-pull setup
+#
+# on remote hosts, set up ansible to run periodically using the latest code
+# from a particular checkout, in pull based fashion, inverting Ansible's
+# usual push-based operating mode.
+#
+# This particular pull based mode is ideal for:
+#
+# (A) massive scale out
+# (B) continual system remediation
+#
+# DO NOT RUN THIS AGAINST YOUR HOSTS WITHOUT CHANGING THE repo_url
+# TO SOMETHING YOU HAVE PERSONALLY VERIFIED
+#
+#
+---
+
+- hosts: pull_mode_hosts
+  user: root
+
+  vars:
+
+    # schdule is fed directly to cron
+    schedule: '*/15 * * * *'
+
+    # User to run ansible-pull as from cron
+    cron_user: root
+
+    # Directory to where repository will be cloned
+    workdir: /var/lib/ansible/local
+
+    # Repository to check out -- YOU MUST CHANGE THIS
+    # repo must contain a local.yml file at top level
+    #repo_url: git://github.com/sfromm/ansible-playbooks.git
+    repo_url: SUPPLY_YOUR_OWN_GIT_URL_HERE
+
+  tasks:
+
+    - name: Install ansible
+      action: yum pkg=ansible state=installed
+
+    - name: Create local directory to work from
+      action: file path=$workdir state=directory owner=root group=root mode=0751
+
+    - name: Copy ansible inventory file to client
+      action: copy src=/etc/ansible/hosts dest=/etc/ansible/hosts
+              owner=root group=root mode=0644
+
+    - name: Create crontab entry to clone/pull git repository
+      action: template src=templates/ansible-pull.j2 dest=/etc/cron.d/ansible-pull owner=root group=root mode=0644
+

examples/playbooks/batch_size_control.yml

@@ -0,0 +1,19 @@
+# ordinarily, without the 'serial' keyword set, ansible will control all of your machines in a play at once, in parallel.
+# if you want to perform a rolling update, so that each play completes all the way through on a certain number of hosts
+# before moving on to the remaining hosts, use the 'serial' keyword like so:
+
+---
+- hosts: all
+  serial: 3
+
+# now each of the tasks below will complete on 3 hosts before moving on to the next 3, regardless of how many
+# hosts are selected by the "hosts:" line
+
+  tasks:
+
+  - name: ping
+    action: ping
+  - name: ping2
+    action: ping
+
+

examples/playbooks/conditionals_part1.yml

@@ -37,7 +37,7 @@
 # and this is just a regular task line from a playbook, as we're used to.
 # but with variables in it that come from above.  Note that the variables
 # from above are *also* available in templates
-     
+
   tasks:
 
   - name: ensure apache is latest
@@ -47,5 +47,5 @@
   - name: fail
     action: command /bin/false
 
- 
+
 

examples/playbooks/conditionals_part2.yml

@@ -16,9 +16,9 @@
 # extraordinarily clean, and not littered with programming language
 # constructs -- so it's easily skimmed by humans.
 #
-# Remember to quote any variables if they are not numbers!  
+# Remember to quote any variables if they are not numbers!
 #
-# Interesting fact: aside from the $variables, these expressions are actually 
+# Interesting fact: aside from the $variables, these expressions are actually
 # tiny bits of Python.  They are evaluated in the context of each host, so different
 # steps can be skipped on different hosts!  They should evaluate to either True
 # or False
@@ -40,7 +40,7 @@
      - name: "do this if my favcolor is blue"
        action: shell /bin/false
        only_if: '$is_favcolor_blue'
-     
+
      - name: "do this if my favcolor is not blue"
        action: shell /bin/true
        only_if: 'not ($is_favcolor_blue)'

examples/playbooks/delegation.yml

@@ -0,0 +1,39 @@
+---
+
+# this is an example of how we can perform actions on a given host on behalf of all the hosts
+# in a play.  
+#
+# The two main uses of this would be signalling an outage window for hosts that
+# we are going to start upgrading, or to take a machine out of rotation by talking to a load
+# balancer.   
+#
+# This example cheats by replacing the load balancer script with the 'echo' command,
+# leaving actual communication with the load balancer as an exercise to the reader.  In reality,
+# you could call anything you want, the main thing is that it should do something with 
+# $inventory_hostname
+
+# NOTE: see batch_size_control.yml for an example of the 'serial' keyword, which you almost certainly
+# want to use in this kind of example.  Here we have a mocked up example that does something to
+# 5 hosts at a time
+
+- hosts: all
+  serial: 5
+
+  tasks:
+  
+  - name: take the machine out of rotation
+    action: command echo taking out of rotation $inventory_hostname
+    delegate_to: 127.0.0.1
+
+# here's an alternate notation if you are delegating to 127.0.0.1, you can use 'local_action' 
+# instead of 'action' and leave off the 'delegate_to' part.
+#
+# - local_action: command echo taking out of rotation $inventory_hostname
+
+  - name: do several things on the actual host
+    action: command echo hi mom $inventory_hostname
+  
+  - name: put machine back into rotation
+    action: command echo inserting into rotation $inventory_hostname
+    delegate_to: 127.0.0.1
+

examples/playbooks/get_url.yml

@@ -0,0 +1,16 @@
+---
+- hosts: webservers
+  vars:
+  - jquery_directory: /var/www/html/javascript
+  - person: 'Susie%20Smith'
+  tasks:
+  - name: Create directory for jQuery
+    action: file dest=${jquery_directory} state=directory mode=0755
+  - name: Grab a bunch of jQuery stuff
+    action: get_url url=http://code.jquery.com/$item  dest=${jquery_directory} mode=0444
+    with_items:
+    - jquery.min.js
+    - mobile/latest/jquery.mobile.min.js
+    - ui/jquery-ui-git.css
+  #- name: Pass urlencoded name to CGI
+  #  action: get_url url=http://example.com/name.cgi?name='${person}' dest=/tmp/test

examples/playbooks/intermediate_example.yml

@@ -1,10 +1,10 @@
 ---
 # see examples.yml first!
-# This file explains some more advanced features of playbooks.  
+# This file explains some more advanced features of playbooks.
 # because of the comments it's less concise than it normally is.  But feel
 # free to comment your playbooks if you like.
 
-- hosts: dbservers
+- hosts: all
 
   # we can define variables the normal way...
 
@@ -34,12 +34,12 @@
 
   # we could also have done something like:
   # - include: wordpress.yml user=timmy
-  # and had access to the template variable {{ user }} in the 
+  # and had access to the template variable $user in the
   # included file, if we wanted to.  Variables from vars
   # and vars_files are also available inside include files
 
   handlers:
- 
+
     # handlers can also be included from files, to promote reuse
     # and simpler recipes, you may wish to only have one
     # handler file for all your plays and playbooks
@@ -70,7 +70,7 @@
   # vars must be specified again for the next play in the playbook
   # but can be reused by including from vars_files if you want
   # you can use vars, vars_files, or both.  vars_files overrides
-  # those set in vars. 
+  # those set in vars.
 
   vars:
      release: 2.0

examples/playbooks/intro_example.yml

@@ -38,9 +38,9 @@
   # let's demo file operations.
   #
   # We can 'copy' files or 'template' them instead, using jinja2
-  # as the templating engine.  This is done using the variables 
-  # from the vars section above mixed in with variables bubbled up 
-  # automatically from tools like facter and ohai.  'copy' 
+  # as the templating engine.  This is done using the variables
+  # from the vars section above mixed in with variables bubbled up
+  # automatically from tools like facter and ohai.  'copy'
   # works just like 'template' but does not do variable subsitution.
   #
   # If and only if the file changes, restart apache at the very

examples/playbooks/mysql.yml

@@ -0,0 +1,18 @@
+##
+# Example Ansible playbook that uses the MySQL module.
+#
+
+---
+- hosts: all
+  user: root
+
+  tasks:
+
+    - name: Create database user
+      action: mysql_user user=bob password=12345 priv=*.*:ALL state=present
+
+    - name: Create database
+      action: mysql_db db=bobdata state=present
+
+    - name: Ensure no user named 'sally' exists and delete if found.
+      action: mysql_user user=sally state=absent

examples/playbooks/nested_playbooks.yml

@@ -0,0 +1,26 @@
+---
+# it is possible to have top level playbook files import other playbook
+# files.  For example, a playbook called could include three
+# different playbooks, such as webservers, workers, dbservers, etc.
+#
+# Running the site playbook would run all playbooks, while individual
+# playbooks could still be run directly.  This is somewhat like
+# the tag feature and can be used in conjunction for very fine grained
+# control over what you want to target when running ansible.
+
+- name: this is a play at the top level of a file
+  hosts: all
+  user: root
+  tasks:
+  - name: say hi
+    tags: foo
+    action: shell echo "hi..."
+
+# and this is how we include another playbook, be careful and
+# don't recurse infinitely or anything.  Note you can't use
+# any variables in the include path here.
+
+- include: intro_example.yml
+
+# and if we wanted, we can continue with more includes here,
+# or more plays inline in this file

examples/playbooks/postgresql.yml

@@ -0,0 +1,38 @@
+##
+# Example Ansible playbook that uses the PostgreSQL module.
+#
+# This installs PostgreSQL on an Ubuntu system, creates a database called
+# "myapp" and a user called "django" with password "mysupersecretpassword"
+# with access to the "myapp" database.
+#
+---
+- hosts: webservers
+  sudo: True
+  gather_facts: False
+
+  tasks:
+  - name: ensure apt cache is up to date
+    action: apt update_cache=yes
+  - name: ensure packages are installed
+    action: apt pkg=$item
+    with_items:
+        - postgresql
+        - libpq-dev
+        - python-psycopg2
+
+- hosts: webservers
+  sudo: True
+  sudo_user: postgres
+  gather_facts: False
+
+  vars:
+    dbname: myapp
+    dbuser: django
+    dbpassword: mysupersecreetpassword
+
+  tasks:
+  - name: ensure database is created
+    action: postgresql_db db=$dbname
+
+  - name: ensure user has access to database
+    action: postgresql_user db=$dbname user=$dbuser password=$dbpassword priv=ALL

examples/playbooks/prompts.yml

@@ -12,22 +12,47 @@
      this_is_a_regular_var: 'moo'
      so_is_this: 'quack'
 
-# prompted variables are a list of variable names and a description
-# that will be presented to the user.
-#
 # alternatively, they can ALSO be passed in from the outside:
 #    ansible-playbook foo.yml --extra-vars="foo=100 bar=101"
 # or through external inventory scripts (see online API docs)
 
-  vars_prompt:
-     release_version: "product release version"
+# here's basic mode prompting.  Specify a hash of variable names and a prompt for
+# each.
+#
+# vars_prompt:
+#   release_version: "product release version"
 
-  tasks:
+# prompts can also be specified like this, allowing for hiding the prompt as
+# entered.  In the future, this may also be used to support crypted variables
+
+  vars_prompt:
+    - name: "some_password"
+      prompt: "Enter password"
+      private: True
+    - name: "release_version"
+      prompt: "Product release version"
+      private: False
+   
+    - name: "my_password2"
+      prompt: "Enter password2"
+      private: True
+      encrypt: "md5_crypt" 
+      confirm: True
+      salt_size: 7
+      salt: "foo" 
 
 # this is just a simple example to show that vars_prompt works, but
 # you might ask for a tag to use with the git module or perhaps
 # a package version to use with the yum module.
 
+  tasks:
+
   - name: imagine this did something interesting with $release_version
     action: shell echo foo >> /tmp/$release_version-$alpha
 
+  - name: look we crypted a password
+    action: shell echo my password is $my_password2
+
+
+
+

examples/playbooks/register_logic.yml

@@ -0,0 +1,29 @@
+# here's a cool advanced topic about how to perform conditional logic in ansible without resorting
+# to writing your own module that defines facts.  You can do that too, and it's easy to do, but
+# often you just want to run a command and then decide whether to run some steps or not.  That's
+# easy to do, and here we'll show you how.
+
+- name: test playbook
+  user: root
+  hosts: all
+
+  tasks:
+
+      # it is possible to save the result of any command in a named register.  This variable will be made
+      # available to tasks and templates made further down in the execution flow.  Here we save the result
+      # of a simple 'cat' command in a variable called 'motd_contents'
+
+      - action: shell cat /etc/motd
+        register: motd_contents
+
+      # and here we access the register.  Note that motd_contents as a variable is structured data because
+      # it is a return from the command module.   The shell module makes available variables such as
+      # as 'stdout', 'stderr', and 'rc'.  Here's a rather trivial example that runs an arbitrary step
+      # if and only if the motd file contained the word 'hi'.   Remember that only_if statements are
+      # Python expressions.  This is as complicated as Ansible syntax is going to get, and the only
+      # time python really seeps into ansible's language.
+ 
+      - action: shell echo "motd contains the word hi"
+        only_if: "'${motd_contents.stdout}'.find('hi') != -1"
+
+

examples/playbooks/selective_file_sources.yml

@@ -2,7 +2,7 @@
 # this is an example of how to template a file over using some variables derived
 # from the system.  For instance, if you wanted to have different configuration
 # templates by OS version, this is a neat way to do it.  Any Ansible facts, facter facts,
-# or ohai facts could be used to do this.  
+# or ohai facts could be used to do this.
 
 - hosts: all
 
@@ -13,13 +13,13 @@
     first_available_file:
 
        # first see if we have a file for this specific host
-       - /srv/whatever/{{ ansible_hostname }}.conf
+       - /srv/whatever/${ansible_hostname}.conf
 
        # next try to load something like CentOS6.2.conf
-       - /srv/whatever/{{ ansible_distribution }}{{ ansible_distribution_version }}.conf
+       - /srv/whatever/${ansible_distribution}${ansible_distribution_version}.conf
 
        # next see if there's a CentOS.conf
-       - /srv/whatever/{{ ansible_distribution }}.conf
+       - /srv/whatever/${ansible_distribution}.conf
 
        # finally give up and just use something generic
        - /srv/whatever/default

examples/playbooks/tags.yml

@@ -0,0 +1,44 @@
+---
+# tags allow us to run all of a playbook or part of it.
+#
+# assume: ansible-playbook tags.yml --tags foo
+#
+# try this with:
+#    --tags foo
+#    --tags bar
+#    --tags extra
+#
+# the value of a 'tags:' element can be a string or list
+# of tag names.  Variables are not usable in tag names.
+
+- name: example play one
+  hosts: all
+  user: root
+
+  # any tags applied to the play are shorthand to applying
+  # the tag to all tasks in it.  Here, each task is given
+  # the tag extra
+
+  tags:
+     - extra
+
+  tasks:
+
+  # this task will run if you don't specify any tags,
+  # if you specify 'foo' or if you specify 'extra'
+
+  - name: hi
+    tags: foo
+    action: shell echo "first task ran"
+
+- name: example play two
+  hosts: all
+  user: root
+  tasks:
+  - name: hi
+    tags:
+       - bar
+    action: shell echo "second task ran"
+  - include: tasks/base.yml tags=base
+
+

examples/playbooks/tasks/base.yml

@@ -2,7 +2,7 @@
 
 # this is the example of an included tasks file.  It contains a flat list of tasks
 # they can notify other tasks, and have full access to variables from 'vars'
-# or 'vars_files' directives.  Further, if ohai or facter were installed on 
+# or 'vars_files' directives.  Further, if ohai or facter were installed on
 # the remote machines, variables from those tools can be accessed on the 'action'
 # line or in templates.  Just prefix with 'facter_' or 'ohai_' before the particular
 # variable.

examples/playbooks/templates/ansible-pull.j2

@@ -0,0 +1,2 @@
+# Cron job to git clone/pull a repo and then run locally
+{{ schedule }} {{ cron_user }} ansible-pull -d {{ workdir }} -U {{ repo_url }} >/var/log/ansible-pull.log 2>&1

examples/playbooks/templates/foo.j2

@@ -1,4 +1,10 @@
 # This is a very simple Jinja2 template representing an imaginary configuration file
 # for an imaginary app.
 
+# this is an example of loading a fact from the setup module
+system={{ ansible_system }}
+
+# here is a variable that could be set in a playbook or inventory file
 http_port={{ http_port }}
+
+

examples/playbooks/templates/hostvars.j2

@@ -0,0 +1,7 @@
+# example of how to get the ipaddress of every machine in the webservers group
+# for use in a template
+
+{% for host in groups['webservers'] %}
+  HOST: {{ host }} IP: {{ hostvars[host]['ansible_all_ipv4_addresses'][0] }}
+{% endfor %}
+

examples/playbooks/vars/defaults.yml

@@ -1,3 +1,3 @@
 ---
-packager: aptitude
+packager: apt
 apache: apache

examples/scripts/cobbler_external_inventory.py

@@ -1,127 +0,0 @@
-#!/usr/bin/python
-
-"""
-Cobbler external inventory script
-=================================
-
-Ansible has a feature where instead of reading from /etc/ansible/hosts
-as a text file, it can query external programs to obtain the list
-of hosts, groups the hosts are in, and even variables to assign to each host.  
-
-To use this, copy this file over /etc/ansible/hosts and chmod +x the file.
-This, more or less, allows you to keep one central database containing
-info about all of your managed instances.
-
-This script is an example of sourcing that data from Cobbler
-(http://cobbler.github.com).  With cobbler each --mgmt-class in cobbler 
-will correspond to a group in Ansible, and --ks-meta variables will be 
-passed down for use in templates or even in argument lines.
-
-NOTE: The cobbler system names will not be used.  Make sure a
-cobbler --dns-name is set for each cobbler system.   If a system
-appears with two DNS names we do not add it twice because we don't want
-ansible talking to it twice.  The first one found will be used. If no
---dns-name is set the system will NOT be visible to ansible.  We do
-not add cobbler system names because there is no requirement in cobbler
-that those correspond to addresses.  
-
-See http://ansible.github.com/api.html for more info
-
-Tested with Cobbler 2.0.11. 
-"""
-
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible,
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-######################################################################
-
-
-import sys
-import xmlrpclib
-import shlex
-
-try:
-    import json
-except:
-    import simplejson as json
-
-# NOTE -- this file assumes Ansible is being accessed FROM the cobbler
-# server, so it does not attempt to login with a username and password.
-# this will be addressed in a future version of this script.
-
-conn = xmlrpclib.Server("http://127.0.0.1/cobbler_api", allow_none=True)
-
-###################################################
-# executed with no parameters, return the list of
-# all groups and hosts
-
-if len(sys.argv) == 1:
-
-    systems = conn.get_item_names('system')
-    groups = { 'ungrouped' : [] }
-
-    for system in systems:
-
-        data = conn.get_blended_data(None, system)
-
-        dns_name = None
-        interfaces = data['interfaces']
-        for (iname, ivalue) in interfaces.iteritems():
-            this_dns_name = ivalue.get('dns_name', None)
-            if this_dns_name is not None:
-                dns_name = this_dns_name       
- 
-        if dns_name is None:
-            continue
-
-        classes  = data['mgmt_classes']
-        for cls in classes:
-            if cls not in groups:
-                groups[cls] = []
-            # hostname is not really what we want to insert, really insert the
-            # first DNS name but no further DNS names
-            groups[cls].append(dns_name)
- 
-    print json.dumps(groups)
-    sys.exit(0)
-
-#####################################################
-# executed with a hostname as a parameter, return the
-# variables for that host
-
-if len(sys.argv) == 2:
-
-    # look up the system record for the given DNS name
-    result = conn.find_system_by_dns_name(sys.argv[1])
-    system = result.get('name', None)
-    data = {}
-    if system is None:
-        print json.dumps({})
-        sys.exit(1)
-    data = conn.get_system_for_koan(system)
-
-    # return the ksmeta data for that system
-    metadata = data['ks_meta']
-    tokens = shlex.split(metadata)
-    results = {}
-    for t in tokens:
-        if t.find("=") != -1:
-           (k,v) = t.split("=",1)
-           results[k]=v
-    print json.dumps(results)
-    sys.exit(0)
-

examples/scripts/uptime.py

@@ -8,7 +8,7 @@ import sys
 # construct the ansible runner and execute on all hosts
 results = ansible.runner.Runner(
     pattern='*', forks=10,
-    module_name='command', module_args=['/usr/bin/uptime'],
+    module_name='command', module_args='/usr/bin/uptime',
 ).run()
 
 if results is None:

examples/scripts/yaml_to_ini.py

@@ -0,0 +1,206 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+import os
+import yaml
+import sys
+
+class InventoryParserYaml(object):
+    ''' Host inventory parser for ansible '''
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        sys.stderr.write("WARNING: YAML inventory files are deprecated in 0.6 and will be removed in 0.7, to migrate" +
+            " download and run https://github.com/ansible/ansible/blob/devel/examples/scripts/yaml_to_ini.py\n")
+
+        fh = open(filename)
+        data = fh.read()
+        fh.close()
+        self._hosts = {}
+        self._parse(data)
+
+    def _make_host(self, hostname):
+
+        if hostname in self._hosts:
+            return self._hosts[hostname]
+        else:
+            host = Host(hostname)
+            self._hosts[hostname] = host
+            return host
+
+    # see file 'test/yaml_hosts' for syntax
+
+    def _parse(self, data):
+        # FIXME: refactor into subfunctions
+
+        all = Group('all')
+
+        ungrouped = Group('ungrouped')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        grouped_hosts = []
+
+        yaml = utils.parse_yaml(data)
+
+        # first add all groups
+        for item in yaml:
+            if type(item) == dict and 'group' in item:
+                group = Group(item['group'])
+
+                for subresult in item.get('hosts',[]):
+
+                    if type(subresult) in [ str, unicode ]:
+                        host = self._make_host(subresult)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+                    elif type(subresult) == dict:
+                        host = self._make_host(subresult['host'])
+                        vars = subresult.get('vars',{})
+                        if type(vars) == list:
+                            for subitem in vars:
+                                for (k,v) in subitem.items():
+                                    host.set_variable(k,v)
+                        elif type(vars) == dict:
+                            for (k,v) in subresult.get('vars',{}).items():
+                                host.set_variable(k,v)
+                        else:
+                            raise errors.AnsibleError("unexpected type for variable")
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                vars = item.get('vars',{})
+                if type(vars) == dict:
+                    for (k,v) in item.get('vars',{}).items():
+                        group.set_variable(k,v)
+                elif type(vars) == list:
+                    for subitem in vars:
+                        if type(subitem) != dict:
+                            raise errors.AnsibleError("expected a dictionary")
+                        for (k,v) in subitem.items():
+                            group.set_variable(k,v)
+
+                self.groups[group.name] = group
+                all.add_child_group(group)
+
+        # add host definitions
+        for item in yaml:
+            if type(item) in [ str, unicode ]:
+                host = self._make_host(item)
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+            elif type(item) == dict and 'host' in item:
+                host = self._make_host(item['host'])
+
+                vars = item.get('vars', {})
+                if type(vars)==list:
+                    varlist, vars = vars, {}
+                    for subitem in varlist:
+                        vars.update(subitem)
+                for (k,v) in vars.items():
+                    host.set_variable(k,v)
+
+                groups = item.get('groups', {})
+                if type(groups) in [ str, unicode ]:
+                    groups = [ groups ]
+                if type(groups)==list:
+                    for subitem in groups:
+                        if subitem in self.groups:
+                            group = self.groups[subitem]
+                        else:
+                            group = Group(subitem)
+                            self.groups[group.name] = group
+                            all.add_child_group(group)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+        # make sure ungrouped.hosts is the complement of grouped_hosts
+        ungrouped_hosts = [host for host in ungrouped.hosts if host not in grouped_hosts]
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print "usage: yaml_to_ini.py /path/to/ansible/hosts"
+        sys.exit(1)
+
+    result = ""
+
+    original = sys.argv[1]
+    yamlp = InventoryParserYaml(filename=sys.argv[1])
+    dirname = os.path.dirname(original)
+
+    group_names = [ g.name for g in yamlp.groups.values() ]
+
+    for group_name in sorted(group_names):
+
+        record = yamlp.groups[group_name]
+
+        if group_name == 'all':
+            continue
+
+        hosts = record.hosts
+        result = result + "[%s]\n" % record.name
+        for h in hosts:
+            result = result + "%s\n" % h.name
+        result = result + "\n"
+
+        groupfiledir = os.path.join(dirname, "group_vars")
+        if not os.path.exists(groupfiledir):
+            print "* creating: %s" % groupfiledir
+            os.makedirs(groupfiledir)
+        groupfile = os.path.join(groupfiledir, group_name)
+        print "* writing group variables for %s into %s" % (group_name, groupfile)
+        groupfh = open(groupfile, 'w')
+        groupfh.write(yaml.dump(record.get_variables()))
+        groupfh.close()
+
+    for (host_name, host_record) in yamlp._hosts.iteritems():
+        hostfiledir = os.path.join(dirname, "host_vars")
+        if not os.path.exists(hostfiledir):
+            print "* creating: %s" % hostfiledir
+            os.makedirs(hostfiledir)
+        hostfile = os.path.join(hostfiledir, host_record.name)
+        print "* writing host variables for %s into %s" % (host_record.name, hostfile)
+        hostfh = open(hostfile, 'w')
+        hostfh.write(yaml.dump(host_record.get_variables()))
+        hostfh.close()
+
+
+    # also need to keep a hash of variables per each host
+    # and variables per each group
+    # and write those to disk
+
+    newfilepath = os.path.join(dirname, "hosts.new")
+    fdh = open(newfilepath, 'w')
+    fdh.write(result)
+    fdh.close()
+
+    print "* COMPLETE: review your new inventory file and replace your original when ready"
+    print "*           new inventory file saved as %s" % newfilepath
+    print "*           edit group specific variables in %s/group_vars/" % dirname
+    print "*           edit host specific variables in %s/host_vars/" % dirname
+
+    # now need to write this to disk as (oldname).new
+    # and inform the user

hacking/env-setup

@@ -1,20 +1,35 @@
 #!/bin/bash
-# usage: source ./hacking/env-setup
+# usage: source ./hacking/env-setup [-q]
 #    modifies environment for running Ansible from checkout
 
-PREFIX_PYTHONPATH="$PWD/lib"
-PREFIX_PATH="$PWD/bin"
-PREFIX_MANPATH="$PWD/docs/man"
+# When run using source as directed, $0 gets set to bash, so we must use $BASH_SOURCE
+if [ -n "$BASH_SOURCE" ] ; then
+    HACKING_DIR=`dirname $BASH_SOURCE`
+else
+    HACKING_DIR="$PWD/hacking"
+fi
+# The below is an alternative to readlink -fn which doesn't exist on OS X
+# Source: http://stackoverflow.com/a/1678636
+FULL_PATH=`python -c "import os, sys; print os.path.realpath('$HACKING_DIR')"`
+ANSIBLE_HOME=`dirname $FULL_PATH`
+
+PREFIX_PYTHONPATH="$ANSIBLE_HOME/lib"
+PREFIX_PATH="$ANSIBLE_HOME/bin"
+PREFIX_MANPATH="$ANSIBLE_HOME/docs/man"
 
 export PYTHONPATH=$PREFIX_PYTHONPATH:$PYTHONPATH
 export PATH=$PREFIX_PATH:$PATH
-export ANSIBLE_LIBRARY="$PWD/library"
+export ANSIBLE_LIBRARY="$ANSIBLE_HOME/library"
 export MANPATH=$PREFIX_MANPATH:$MANPATH
 
-echo "PATH=$PATH"
-echo "PYTHONPATH=$PYTHONPATH"
-echo "ANSIBLE_LIBRARY=$ANSIBLE_LIBRARY"
-echo "MANPATH=$MANPATH"
+# Print out values unless -q is set
 
-echo "Reminder: specify your host file with -i"
-echo "Done."
+if [ $# -eq 0 -o "$1" != "-q" ] ; then
+    echo "PATH=$PATH"
+    echo "PYTHONPATH=$PYTHONPATH"
+    echo "ANSIBLE_LIBRARY=$ANSIBLE_LIBRARY"
+    echo "MANPATH=$MANPATH"
+    
+    echo "Reminder: specify your host file with -i"
+    echo "Done."
+fi

hacking/test-module

@@ -23,69 +23,129 @@
 # modules
 #
 # example:
-#    test-module ../library/command /bin/sleep 3
-#    test-module ../library/service name=httpd ensure=restarted
+#    test-module -m ../library/command -a "/bin/sleep 3"
+#    test-module -m ../library/service -a "name=httpd ensure=restarted"
+#    test-module -m ../library/service -a "name=httpd ensure=restarted" --debugger /usr/bin/pdb
 
 import sys
+import base64
 import os
 import subprocess
 import traceback
-from ansible import utils
+import optparse
+import ansible.utils as utils
+import ansible.module_common as module_common
 
 try:
     import json
 except ImportError:
     import simplejson as json
 
-modfile = None
+def parse():
+    """parse command line
 
-if len(sys.argv) == 1:
-    print >>sys.stderr, "usage: test-module ./library/command [key=value ...]"
-    sys.exit(1)
+    :return : (options, args)"""
+    parser = optparse.OptionParser()
 
-modfile = sys.argv[1]
-if len(sys.argv) > 1:
-    args = " ".join(sys.argv[2:])
-else:
-    args = ""
+    parser.usage = "%prog -[options] (-h for help)"
 
-argspath = os.path.expanduser("/.ansible_test_module_arguments")
-argsfile = open(argspath, 'w')
-argsfile.write(args)
-argsfile.close()
+    parser.add_option('-m', '--module-path', dest='module_path',
+        help="REQUIRED: full path of module source to execute")
+    parser.add_option('-a', '--args', dest='module_args', default="",
+        help="module argument string")
+    parser.add_option('-D', '--debugger', dest='debugger', 
+        help="path to python debugger (e.g. /usr/bin/pdb)")
+    options, args = parser.parse_args()
+    if not options.module_path:
+        parser.print_help()
+        sys.exit(1)
+    else:
+        return options, args
 
-os.system("chmod +x %s" % modfile)
-cmd = subprocess.Popen("%s %s" % (modfile, argspath), 
-    shell=True,
-    stdout=subprocess.PIPE, 
-    stderr=subprocess.PIPE)
-(out, err) = cmd.communicate()
+def write_argsfile(argstring):
+    """ Write args to a file for old-style module's use. """
+    argspath = os.path.expanduser("~/.ansible_test_module_arguments")
+    argsfile = open(argspath, 'w')
+    argsfile.write(argstring)
+    argsfile.close()
+    return argspath
+
+def boilerplate_module(modfile, args):
+    """ simulate what ansible does with new style modules """
+
+    module_fh = open(modfile)
+    module_data = module_fh.read()
+    included_boilerplate = module_data.find(module_common.REPLACER) != -1
+    module_fh.close()
+
+    if included_boilerplate:
+
+        module_data = module_data.replace(module_common.REPLACER, module_common.MODULE_COMMON)
+        encoded_args = "\"\"\"%s\"\"\"" % args.replace("\"","\\\"")
+        module_data = module_data.replace(module_common.REPLACER_ARGS, encoded_args)
+
+        modfile2_path = os.path.expanduser("~/.ansible_module_generated")
+        print "* including generated source, if any, saving to: %s" % modfile2_path
+        print "* this will offset any line numbers in tracebacks/debuggers!"
+        modfile2 = open(modfile2_path, 'w')
+        modfile2.write(module_data)
+        modfile2.close()
+        modfile = modfile2_path
+        return (modfile2_path, included_boilerplate)
+    else:
+        print "* module boilerplate substitution not requested in module, line numbers will be unaltered"
+        return (modfile, included_boilerplate)
+
+def runtest( modfile, argspath):
+    """Test run a module, piping it's output for reporting."""
+
+    os.system("chmod +x %s" % modfile)
+
+    invoke = "%s" % (modfile)
+    if argspath is not None:     
+        invoke = "%s %s" % (modfile, argspath)
+
+    cmd = subprocess.Popen(invoke, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+
+    try:
+        print "***********************************"
+        print "RAW OUTPUT"
+        print out
+        print err
+        results = utils.parse_json(out)
+    except:
+        print "***********************************"
+        print "INVALID OUTPUT FORMAT"
+        print out
+        traceback.print_exc()
+        sys.exit(1)
 
-if err and err != '':
     print "***********************************"
-    print "RECIEVED DATA ON STDERR, THIS WILL CRASH YOUR MODULE"
-    print err
-    sys.exit(1)
+    print "PARSED OUTPUT"
+    print utils.jsonify(results,format=True)
 
-try:
-    print "***********************************"
-    print "RAW OUTPUT"
-    print out
-    results = utils.parse_json(out)
-    
-except:
-    print "***********************************"
-    print "INVALID OUTPUT FORMAT"
-    print out
-    traceback.print_exc()
-    sys.exit(1)
+def rundebug(debugger, modfile, argspath):
+    """Run interactively with console debugger."""
 
-print "***********************************"
-print "PARSED OUTPUT"
+    if argspath is not None:
+        subprocess.call("%s %s %s" % (debugger, modfile, argspath), shell=True)
+    else:
+        subprocess.call("%s %s" % (debugger, modfile), shell=True)
 
-print utils.bigjson(results)
-
-sys.exit(0)
+def main(): 
 
+    options, args = parse()
+    (modfile, is_new_style) = boilerplate_module(options.module_path, options.module_args)
+    argspath=None
+    if not is_new_style:
+        argspath = write_argsfile(options.module_args)
+    if options.debugger: 
+        rundebug(options.debugger, modfile, argspath)
+    else:
+        runtest(modfile, argspath)
+	
+if __name__ == "__main__":
+    main()
 
 

lib/ansible/__init__.py

@@ -14,5 +14,5 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-__version__ = '0.4'
+__version__ = '0.7'
 __author__ = 'Michael DeHaan'

lib/ansible/callback_plugins/noop.py

@@ -0,0 +1,86 @@
+# (C) 2012, Michael DeHaan, <michael.dehaan@gmail.com>
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+
+class CallbackModule(object):
+
+    """
+    this is an example ansible callback file that does nothing.  You can drop
+    other classes in the same directory to define your own handlers.  Methods
+    you do not use can be omitted.
+
+    example uses include: logging, emailing, storing info, etc
+    """
+
+
+    def on_any(self, *args, **kwargs):
+        pass
+
+    def runner_on_failed(self, host, res, ignore_errors=False):
+        pass
+
+    def runner_on_ok(self, host, res):
+        pass
+
+    def runner_on_error(self, host, msg):
+        pass
+
+    def runner_on_skipped(self, host, item=None):
+        pass
+
+    def runner_on_unreachable(self, host, res):
+        pass
+
+    def runner_on_no_hosts(self):
+        pass
+
+    def runner_on_async_poll(self, host, res, jid, clock):
+        pass
+
+    def runner_on_async_ok(self, host, res, jid):
+        pass
+
+    def runner_on_async_failed(self, host, res, jid):
+        pass
+
+    def playbook_on_start(self):
+        pass
+
+    def playbook_on_notify(self, host, handler):
+        pass
+
+    def playbook_on_task_start(self, name, is_conditional):
+        pass
+
+    def playbook_on_vars_prompt(self, varname, private=True, prompt=None, encrypt=None, confirm=False, salt_size=None, salt=None):
+        pass
+
+    def playbook_on_setup(self):
+        pass
+
+    def playbook_on_import_for_host(self, host, imported_file):
+        pass
+
+    def playbook_on_not_import_for_host(self, host, missing_file):
+        pass
+
+    def playbook_on_play_start(self, pattern):
+        pass
+
+    def playbook_on_stats(self, stats):
+        pass
+

lib/ansible/callbacks.py

@@ -15,18 +15,53 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-#######################################################
-
 import utils
 import sys
 import getpass
+import os
+import subprocess
+import os.path
+from ansible.color import stringc
 
-#######################################################
+dirname = os.path.dirname(__file__)
+callbacks = utils.import_plugins(os.path.join(dirname, 'callback_plugins'))
+callbacks = [ c.CallbackModule() for c in callbacks.values() if c.__name__ != '__init__' ]
+
+cowsay = None
+if os.path.exists("/usr/bin/cowsay"):
+    cowsay = "/usr/bin/cowsay"
+elif os.path.exists("/usr/games/cowsay"):
+    cowsay = "/usr/games/cowsay"
+
+def call_callback_module(method_name, *args, **kwargs):
+   
+    for callback_plugin in callbacks:
+        methods = [ 
+            getattr(callback_plugin, method_name, None), 
+            getattr(callback_plugin, 'on_any', None)
+        ]
+        for method in methods:
+            if method is not None:
+                method(*args, **kwargs)
+
+def vv(msg, host=None):
+    return verbose(msg, host=host, caplevel=1)
+
+def vvv(msg, host=None):
+    return verbose(msg, host=host, caplevel=2)
+
+def verbose(msg, host=None, caplevel=2):
+    if utils.VERBOSITY > caplevel:
+        if host is None:
+            print stringc(msg, 'blue')
+        else:
+            print stringc("<%s> %s" % (host, msg), 'blue')
 
 class AggregateStats(object):
-    ''' holds stats about per-host activity during playbook runs '''   
- 
+    ''' holds stats about per-host activity during playbook runs '''
+
     def __init__(self):
+
         self.processed   = {}
         self.failures    = {}
         self.ok          = {}
@@ -41,11 +76,12 @@ class AggregateStats(object):
         prev = (getattr(self, what)).get(host, 0)
         getattr(self, what)[host] = prev+1
 
-    def compute(self, runner_results, setup=False, poll=False):
+    def compute(self, runner_results, setup=False, poll=False, ignore_errors=False):
         ''' walk through all results and increment stats '''
- 
+
         for (host, value) in runner_results.get('contacted', {}).iteritems():
-            if ('failed' in value and bool(value['failed'])) or ('rc' in value and value['rc'] != 0):
+            if not ignore_errors and (('failed' in value and bool(value['failed'])) or
+                ('rc' in value and value['rc'] != 0)):
                 self._increment('failures', host)
             elif 'skipped' in value and bool(value['skipped']):
                 self._increment('skipped', host)
@@ -59,7 +95,7 @@ class AggregateStats(object):
 
         for (host, value) in runner_results.get('dark', {}).iteritems():
             self._increment('dark', host)
-            
+
 
     def summarize(self, host):
         ''' return information about a particular host '''
@@ -74,38 +110,102 @@ class AggregateStats(object):
 
 ########################################################################
 
+def regular_generic_msg(hostname, result, oneline, caption):
+    ''' output on the result of a module run that is not command '''
+
+    if not oneline:
+        return "%s | %s >> %s\n" % (hostname, caption, utils.jsonify(result,format=True))
+    else:
+        return "%s | %s >> %s\n" % (hostname, caption, utils.jsonify(result))
+
+
+def banner(msg):
+
+    if cowsay != None:
+        cmd = subprocess.Popen([cowsay, "-W", "60", msg],
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (out, err) = cmd.communicate()
+        return "%s\n" % out
+    else:
+        return "\n%s ********************* " % msg
+
+def command_generic_msg(hostname, result, oneline, caption):
+    ''' output the result of a command run '''
+
+    rc     = result.get('rc', '0')
+    stdout = result.get('stdout','')
+    stderr = result.get('stderr', '')
+    msg    = result.get('msg', '')
+
+    hostname = hostname.encode('utf-8')
+    caption  = caption.encode('utf-8')
+
+    if not oneline:
+        buf = "%s | %s | rc=%s >>\n" % (hostname, caption, result.get('rc',0))
+        if stdout:
+            buf += stdout
+        if stderr:
+            buf += stderr
+        if msg:
+            buf += msg
+        return buf + "\n"
+    else:
+        if stderr:
+            return "%s | %s | rc=%s | (stdout) %s (stderr) %s\n" % (hostname, caption, rc, stdout, stderr)
+        else:
+            return "%s | %s | rc=%s | (stdout) %s\n" % (hostname, caption, rc, stdout)
+
+def host_report_msg(hostname, module_name, result, oneline):
+    ''' summarize the JSON results for a particular host '''
+
+    failed = utils.is_failed(result)
+    msg = ''
+    if module_name in [ 'command', 'shell', 'raw' ] and 'ansible_job_id' not in result and result.get('parsed',True) != False:
+        if not failed:
+            msg = command_generic_msg(hostname, result, oneline, 'success')
+        else:
+            msg = command_generic_msg(hostname, result, oneline, 'FAILED')
+    else:
+        if not failed:
+            msg = regular_generic_msg(hostname, result, oneline, 'success')
+        else:
+            msg = regular_generic_msg(hostname, result, oneline, 'FAILED')
+    return msg
+
+###############################################
+
 class DefaultRunnerCallbacks(object):
     ''' no-op callbacks for API usage of Runner() if no callbacks are specified '''
 
     def __init__(self):
         pass
 
-    def on_failed(self, host, res):
-        pass
+    def on_failed(self, host, res, ignore_errors=False):
+        call_callback_module('runner_on_failed', host, res, ignore_errors=ignore_errors)
 
     def on_ok(self, host, res):
-        pass
+        call_callback_module('runner_on_ok', host, res)
 
     def on_error(self, host, msg):
-        pass
+        call_callback_module('runner_on_error', host, msg)
 
-    def on_skipped(self, host):
-        pass
+    def on_skipped(self, host, item=None):
+        call_callback_module('runner_on_skipped', host, item=item)
 
     def on_unreachable(self, host, res):
-        pass
+        call_callback_module('runner_on_unreachable', host, res)
 
     def on_no_hosts(self):
-        pass
+        call_callback_module('runner_on_no_hosts')
 
     def on_async_poll(self, host, res, jid, clock):
-        pass
+        call_callback_module('runner_on_async_poll', host, res, jid, clock)
 
     def on_async_ok(self, host, res, jid):
-        pass
+        call_callback_module('runner_on_async_ok', host, res, jid)
 
     def on_async_failed(self, host, res, jid):
-        pass
+        call_callback_module('runner_on_async_failed', host, res, jid)
 
 ########################################################################
 
@@ -114,32 +214,38 @@ class CliRunnerCallbacks(DefaultRunnerCallbacks):
 
     def __init__(self):
         # set by /usr/bin/ansible later
-        self.options = None 
+        self.options = None
         self._async_notified = {}
 
-    def on_failed(self, host, res):
-        invocation = res.get('invocation','')
-        if not invocation.startswith('async_status'):
-            self._on_any(host,res)
+    def on_failed(self, host, res, ignore_errors=False):
+        self._on_any(host,res)
+        super(CliRunnerCallbacks, self).on_failed(host, res, ignore_errors=ignore_errors)
 
     def on_ok(self, host, res):
-        invocation = res.get('invocation','')
-        if not invocation.startswith('async_status'):
-            self._on_any(host,res)
- 
+        self._on_any(host,res)
+        super(CliRunnerCallbacks, self).on_ok(host, res)
+
     def on_unreachable(self, host, res):
+        if type(res) == dict:
+            res = res.get('msg','')
         print "%s | FAILED => %s" % (host, res)
         if self.options.tree:
-            utils.write_tree_file(self.options.tree, host, utils.bigjson(dict(failed=True, msg=res)))
- 
-    def on_skipped(self, host):
-        pass
+            utils.write_tree_file(
+                self.options.tree, host,
+                utils.jsonify(dict(failed=True, msg=res),format=True)
+            )
+        super(CliRunnerCallbacks, self).on_unreachable(host, res)
+
+    def on_skipped(self, host, item=None):
+        super(CliRunnerCallbacks, self).on_skipped(host, item)
 
     def on_error(self, host, err):
         print >>sys.stderr, "err: [%s] => %s\n" % (host, err)
-    
+        super(CliRunnerCallbacks, self).on_error(host, err)
+
     def on_no_hosts(self):
         print >>sys.stderr, "no hosts matched\n"
+        super(CliRunnerCallbacks, self).on_no_hosts()
 
     def on_async_poll(self, host, res, jid, clock):
         if jid not in self._async_notified:
@@ -147,102 +253,204 @@ class CliRunnerCallbacks(DefaultRunnerCallbacks):
         if self._async_notified[jid] > clock:
             self._async_notified[jid] = clock
             print "<job %s> polling, %ss remaining"%(jid, clock)
+        super(CliRunnerCallbacks, self).on_async_poll(host, res, jid, clock)
 
     def on_async_ok(self, host, res, jid):
-        print "<job %s> finished on %s => %s"%(jid, host, utils.bigjson(res))
+        print "<job %s> finished on %s => %s"%(jid, host, utils.jsonify(res,format=True))
+        super(CliRunnerCallbacks, self).on_async_ok(host, res, jid)
 
     def on_async_failed(self, host, res, jid):
-        print "<job %s> FAILED on %s => %s"%(jid, host, utils.bigjson(res))
+        print "<job %s> FAILED on %s => %s"%(jid, host, utils.jsonify(res,format=True))
+        super(CliRunnerCallbacks, self).on_async_failed(host,res,jid)
 
     def _on_any(self, host, result):
-        print utils.host_report_msg(host, self.options.module_name, result, self.options.one_line)
+        result2 = result.copy()
+        result2.pop('invocation', None)
+        print host_report_msg(host, self.options.module_name, result2, self.options.one_line)
         if self.options.tree:
-            utils.write_tree_file(self.options.tree, host, utils.bigjson(result))
+            utils.write_tree_file(self.options.tree, host, utils.jsonify(result2,format=True))
 
 ########################################################################
 
 class PlaybookRunnerCallbacks(DefaultRunnerCallbacks):
     ''' callbacks used for Runner() from /usr/bin/ansible-playbook '''
 
-    def __init__(self, stats):
+    def __init__(self, stats, verbose=utils.VERBOSITY):
+        self.verbose = verbose
         self.stats = stats
         self._async_notified = {}
 
     def on_unreachable(self, host, msg):
-        print "fatal: [%s] => %s" % (host, msg)
+        item = None
+        if type(msg) == dict:
+            item = msg.get('item', None)
+        if item:
+            print "fatal: [%s] => (item=%s) => %s" % (host, item, msg)
+        else:
+            print "fatal: [%s] => %s" % (host, msg)
+        super(PlaybookRunnerCallbacks, self).on_unreachable(host, msg)
 
-    def on_failed(self, host, results):
-        invocation = results.get('invocation',None)
-        if not invocation or invocation.startswith('setup ') or invocation.startswith('async_status '):
-            print "failed: [%s] => %s\n" % (host, utils.smjson(results))
-        else: 
-            print "failed: [%s] => %s => %s\n" % (host, invocation, utils.smjson(results))
+    def on_failed(self, host, results, ignore_errors=False):
+
+        results2 = results.copy()
+        results2.pop('invocation', None)
+
+        item = results2.get('item', None)
+        if item:
+            msg = "failed: [%s] => (item=%s) => %s" % (host, item, utils.jsonify(results2))
+        else:
+            msg = "failed: [%s] => %s" % (host, utils.jsonify(results2))
+        print stringc(msg, 'red')
+        if ignore_errors:
+            print stringc("...ignoring", 'yellow')
+        super(PlaybookRunnerCallbacks, self).on_failed(host, results, ignore_errors=ignore_errors)
 
     def on_ok(self, host, host_result):
-        invocation = host_result.get('invocation','')
-        if invocation.startswith('async_status'):
-            pass
-        elif not invocation or invocation.startswith('setup '):
-            print "ok: [%s]\n" % (host)
+        item = host_result.get('item', None)
+
+        host_result2 = host_result.copy()
+        host_result2.pop('invocation', None)
+
+        # show verbose output for non-setup module results if --verbose is used
+        msg = ''
+        if not self.verbose or host_result2.get("verbose_override",None) is not None:
+            if item:
+                msg = "ok: [%s] => (item=%s)" % (host,item)
+            else:
+                if 'ansible_job_id' not in host_result or 'finished' in host_result:
+                    msg = "ok: [%s]" % (host)
         else:
-            print "ok: [%s] => %s\n" % (host, invocation)
+            # verbose ...
+            if item:
+                msg = "ok: [%s] => (item=%s) => %s" % (host, item, utils.jsonify(host_result2))
+            else:
+                if 'ansible_job_id' not in host_result or 'finished' in host_result2:
+                    msg = "ok: [%s] => %s" % (host, utils.jsonify(host_result2))
+
+        if msg != '':
+            if not 'changed' in host_result2 or not host_result['changed']:
+                print stringc(msg, 'green')
+            else:
+                print stringc(msg, 'yellow')
+        super(PlaybookRunnerCallbacks, self).on_ok(host, host_result)
 
     def on_error(self, host, err):
-        print >>sys.stderr, "err: [%s] => %s\n" % (host, err)
 
-    def on_skipped(self, host):
-        print "skipping: [%s]\n" % host
+        item = err.get('item', None)
+        msg = ''
+        if item:
+            msg = "err: [%s] => (item=%s) => %s" % (host, item, err)
+        else:
+            msg = "err: [%s] => %s" % (host, err)
+
+        msg = stringc(msg, 'red')
+        print >>sys.stderr, msg
+        super(PlaybookRunnerCallbacks, self).on_error(host, err)
+
+    def on_skipped(self, host, item=None):
+        msg = ''
+        if item:
+            msg = "skipping: [%s] => (item=%s)" % (host, item)
+        else:
+            msg = "skipping: [%s]" % host
+        print stringc(msg, 'yellow')
+        super(PlaybookRunnerCallbacks, self).on_skipped(host, item)
 
     def on_no_hosts(self):
-        print "no hosts matched or remaining\n"
+        print stringc("no hosts matched or remaining\n", 'red')
+        super(PlaybookRunnerCallbacks, self).on_no_hosts()
 
     def on_async_poll(self, host, res, jid, clock):
         if jid not in self._async_notified:
             self._async_notified[jid] = clock + 1
         if self._async_notified[jid] > clock:
             self._async_notified[jid] = clock
-            print "<job %s> polling, %ss remaining"%(jid, clock)
+            msg = "<job %s> polling, %ss remaining"%(jid, clock)
+            print stringc(msg, 'cyan')
+        super(PlaybookRunnerCallbacks, self).on_async_poll(host,res,jid,clock)
 
     def on_async_ok(self, host, res, jid):
-        print "<job %s> finished on %s"%(jid, host)
+        msg = "<job %s> finished on %s"%(jid, host)
+        print stringc(msg, 'cyan')
+        super(PlaybookRunnerCallbacks, self).on_async_ok(host, res, jid)
 
     def on_async_failed(self, host, res, jid):
-        print "<job %s> FAILED on %s"%(jid, host)
+        msg = "<job %s> FAILED on %s"%(jid, host)
+        print stringc(msg, 'red')
+        super(PlaybookRunnerCallbacks, self).on_async_failed(host,res,jid)
+
 
 ########################################################################
 
 class PlaybookCallbacks(object):
     ''' playbook.py callbacks used by /usr/bin/ansible-playbook '''
-  
-    def __init__(self):
-        pass
+
+    def __init__(self, verbose=False):
+
+        self.verbose = verbose
 
     def on_start(self):
-        print "\n"
+        call_callback_module('playbook_on_start')
 
     def on_notify(self, host, handler):
-        pass
+        call_callback_module('playbook_on_notify', host, handler)
 
     def on_task_start(self, name, is_conditional):
-        print utils.task_start_msg(name, is_conditional)
+        msg = "TASK: [%s]" % name
+        if is_conditional:
+            msg = "NOTIFIED: [%s]" % name
+        print banner(msg)
+        call_callback_module('playbook_on_task_start', name, is_conditional)
 
-    def on_vars_prompt(self, varname, private=True):
-        msg = 'input for %s: ' % varname
-        if private:
-            return getpass.getpass(msg)
-        return raw_input(msg)
-        
-    def on_setup_primary(self):
-        print "SETUP PHASE ****************************\n"
-    
-    def on_setup_secondary(self):
-        print "\nVARIABLE IMPORT PHASE ******************\n"
+    def on_vars_prompt(self, varname, private=True, prompt=None, encrypt=None, confirm=False, salt_size=None, salt=None):
+
+        if prompt:
+            msg = prompt
+        else:
+            msg = 'input for %s: ' % varname
+
+        def prompt(prompt, private):
+            if private:
+                return getpass.getpass(prompt)
+            return raw_input(prompt)
+
+
+        if confirm:
+            while True:
+                result = prompt(msg, private)
+                second = prompt("confirm " + msg, private)
+                if result == second: 
+                    break
+                print "***** VALUES ENTERED DO NOT MATCH ****"
+        else:
+            result = prompt(msg, private)
+
+        if encrypt:
+            result = utils.do_encrypt(result,encrypt,salt_size,salt)
+
+        call_callback_module('playbook_on_vars_prompt', varname, private=private, prompt=prompt, encrypt=encrypt, confirm=confirm, salt_size=salt_size, salt=None)
+
+        return result
+
+    def on_setup(self):
+        print banner("GATHERING FACTS")
+        call_callback_module('playbook_on_setup')
 
     def on_import_for_host(self, host, imported_file):
-        print "%s: importing %s" % (host, imported_file)
+        msg = "%s: importing %s" % (host, imported_file)
+        print stringc(msg, 'cyan')
+        call_callback_module('playbook_on_import_for_host', host, imported_file)
 
     def on_not_import_for_host(self, host, missing_file):
-        print "%s: not importing file: %s" % (host, missing_file)
+        msg = "%s: not importing file: %s" % (host, missing_file)
+        print stringc(msg, 'cyan')
+        call_callback_module('playbook_on_not_import_for_host', host, missing_file)
 
     def on_play_start(self, pattern):
-        print "PLAY [%s] ****************************\n" % pattern
+        print banner("PLAY [%s]" % pattern)
+        call_callback_module('playbook_on_play_start', pattern)
+
+    def on_stats(self, stats):
+        call_callback_module('playbook_on_stats', stats)
+
+

lib/ansible/color.py

@@ -0,0 +1,71 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import sys
+
+ANSIBLE_COLOR=True
+if os.getenv("ANSIBLE_NOCOLOR") is not None:
+    ANSIBLE_COLOR=False
+elif not sys.stdout.isatty():
+    ANSIBLE_COLOR=False
+else:
+    try:
+        import curses
+        curses.setupterm()
+        if curses.tigetnum('colors') < 0:
+            ANSIBLE_COLOR=False
+    except ImportError:
+        # curses library was not found
+        pass
+    except curses.error:
+        # curses returns an error (e.g. could not find terminal)
+        ANSIBLE_COLOR=False
+
+# --- begin "pretty"
+#
+# pretty - A miniature library that provides a Python print and stdout
+# wrapper that makes colored terminal text easier to use (eg. without
+# having to mess around with ANSI escape sequences). This code is public
+# domain - there is no license except that you must leave this header.
+#
+# Copyright (C) 2008 Brian Nez <thedude at bri1 dot com>
+#
+# http://nezzen.net/2008/06/23/colored-text-in-python-using-ansi-escape-sequences/
+
+codeCodes = {
+    'black':     '0;30', 'bright gray':    '0;37',
+    'blue':      '0;34', 'white':          '1;37',
+    'green':     '0;32', 'bright blue':    '1;34',
+    'cyan':      '0;36', 'bright green':   '1;32',
+    'red':       '0;31', 'bright cyan':    '1;36',
+    'purple':    '0;35', 'bright red':     '1;31',
+    'yellow':    '0;33', 'bright purple':  '1;35',
+    'dark gray': '1;30', 'bright yellow':  '1;33',
+    'normal':    '0'
+}
+
+def stringc(text, color):
+    """String in color."""
+
+    if ANSIBLE_COLOR:
+        return "\033["+codeCodes[color]+"m"+text+"\033[0m"
+    else:
+        return text
+
+# --- end "pretty"
+

lib/ansible/connection.py

@@ -1,252 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-################################################
-
-import warnings
-import traceback
-import os
-import time
-import re
-import shutil
-import subprocess
-import pipes
-import socket
-import random
-
-from ansible import errors
-# prevent paramiko warning noise
-# see http://stackoverflow.com/questions/3920502/
-with warnings.catch_warnings():
-    warnings.simplefilter("ignore")
-    import paramiko
-
-
-################################################
-
-class Connection(object):
-    ''' Handles abstract connections to remote hosts '''
-
-    _LOCALHOSTRE = re.compile(r"^(127.0.0.1|localhost|%s)$" % os.uname()[1])
-
-    def __init__(self, runner, transport):
-        self.runner = runner
-        self.transport = transport
-
-    def connect(self, host, port=None):
-        conn = None
-        if self.transport == 'local' and self._LOCALHOSTRE.search(host):
-            conn = LocalConnection(self.runner, host)
-        elif self.transport == 'paramiko':
-            conn = ParamikoConnection(self.runner, host, port)
-        if conn is None:
-            raise Exception("unsupported connection type")
-        return conn.connect()
-
-################################################
-# want to implement another connection type?
-# follow duck-typing of ParamikoConnection
-# you may wish to read config files in __init__
-# if you have any.  Paramiko does not need any.
-
-class ParamikoConnection(object):
-    ''' SSH based connections with Paramiko '''
-
-    def __init__(self, runner, host, port=None):
-        self.ssh = None
-        self.runner = runner
-        self.host = host
-        self.port = port
-        if port is None:
-            self.port = self.runner.remote_port
-
-    def _get_conn(self):
-        credentials = {}
-        user = self.runner.remote_user
-        keypair = None
-
-        # Read file ~/.ssh/config, get data hostname, keyfile, port, etc
-        # This will *NOT* overrides the ansible username and hostname " , getting the port and keyfile only.
-	
-        try:
-            ssh_config = paramiko.SSHConfig()
-            config_file = ('~/.ssh/config')
-            if  os.path.exists(os.path.expanduser(config_file)):
-                ssh_config.parse(open(os.path.expanduser(config_file)))
-                credentials = ssh_config.lookup(self.host)
-
-        except IOError,e:
-                raise errors.AnsibleConnectionFailed(str(e))
-
-        if 'port' in credentials:
-            self.port = int(credentials['port'])
-        if 'identityfile' in credentials:
-            keypair = os.path.expanduser(credentials['identityfile'])
-
-        ssh = paramiko.SSHClient()
-        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
-
-        try:
-            ssh.connect(
-                self.host,
-                username=user,
-                allow_agent=True,
-                look_for_keys=True,
-                password=self.runner.remote_pass,
-                key_filename=keypair,
-                timeout=self.runner.timeout,
-                port=self.port
-            )
-        except Exception, e:
-            if str(e).find("PID check failed") != -1:
-                raise errors.AnsibleError("paramiko version issue, please upgrade paramiko on the machine running ansible")
-            else:
-                raise errors.AnsibleConnectionFailed(str(e))
-
-        return ssh
-
-    def connect(self):
-        ''' connect to the remote host '''
-
-        self.ssh = self._get_conn()
-        return self
-
-    def exec_command(self, cmd, tmp_path, sudoable=False):
-
-        ''' run a command on the remote host '''
-
-        bufsize = 4096
-        # reusing runner's TCP connect timeout as command progress timeout (correct?)
-        timeout_secs = self.runner.timeout 
-        chan = self.ssh.get_transport().open_session()
-        chan.settimeout(timeout_secs)
-        stdin = chan.makefile('wb', bufsize) 
-        stdout = chan.makefile('rb', bufsize)
-        stderr = chan.makefile_stderr('rb', bufsize) 
-        chan.get_pty() 
-        chan.set_combine_stderr(False)    
-
-        if not self.runner.sudo or not sudoable:
-            quoted_command = '"$SHELL" -c ' + pipes.quote(cmd) 
-            chan.exec_command(quoted_command)
-        else:
-            # Rather than detect if sudo wants a password this time, -k makes 
-            # sudo always ask for a password if one is required. The "--"
-            # tells sudo that this is the end of sudo options and the command
-            # follows.  Passing a quoted compound command to sudo (or sudo -s)
-            # directly doesn't work, so we shellquote it with pipes.quote() 
-            # and pass the quoted string to the user's shell.
-
-            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
-            prompt = '[sudo via ansible, key=%s] password: ' % randbits
-            sudocmd = 'sudo -k -p "%s" -- "$SHELL" -c %s' % (prompt, pipes.quote(cmd))
-            sudo_output = ''
-            try:
-                chan.exec_command(sudocmd)
-                if self.runner.sudo_pass:
-                    while not sudo_output.endswith(prompt):
-                        chunk = chan.recv(bufsize)
-                        if not chunk:
-                            raise errors.AnsibleError('ssh connection closed waiting for sudo password prompt')
-                        sudo_output += chunk
-                    chan.sendall(self.runner.sudo_pass + '\n')
-            except socket.timeout:
-                raise errors.AnsibleError('ssh timed out waiting for sudo.\n' + sudo_output)
-
-        stdin = chan.makefile('wb', bufsize)
-        stdout = chan.makefile('rb', bufsize)
-        stderr = chan.makefile_stderr('rb', bufsize)
-        return stdin, stdout, stderr
-
-    def put_file(self, in_path, out_path):
-        ''' transfer a file from local to remote '''
-        if not os.path.exists(in_path):
-            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
-        sftp = self.ssh.open_sftp()
-        try:
-            sftp.put(in_path, out_path)
-        except IOError:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
-        sftp.close()
-
-    def fetch_file(self, in_path, out_path):
-        sftp = self.ssh.open_sftp()
-        try:
-            sftp.get(in_path, out_path)
-        except IOError:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to transfer file from %s" % in_path)
-        sftp.close()
-
-    def close(self):
-        ''' terminate the connection '''
-
-        self.ssh.close()
-
-############################################
-# add other connection types here
-
-class LocalConnection(object):
-    ''' Local based connections '''
-
-    def __init__(self, runner, host):
-        self.runner = runner
-        self.host = host
-
-    def connect(self, port=None):
-        ''' connect to the local host; nothing to do here '''
-
-        return self
-
-    def exec_command(self, cmd, tmp_path, sudoable=False):
-        ''' run a command on the local host '''
-        if self.runner.sudo and sudoable:
-            cmd = "sudo -s %s" % cmd
-        if self.runner.sudo_pass:
-            # NOTE: if someone wants to add sudo w/ password to the local connection type, they are welcome
-            # to do so.  The primary usage of the local connection is for crontab and kickstart usage however
-            # so this doesn't seem to be a huge priority
-            raise errors.AnsibleError("sudo with password is presently only supported on the paramiko (SSH) connection type")
-
-        p = subprocess.Popen(cmd, shell=True, stdin=None,
-                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        stdout, stderr = p.communicate()
-        return ("", stdout, stderr)
-
-    def put_file(self, in_path, out_path):
-        ''' transfer a file from local to local '''
-        if not os.path.exists(in_path):
-            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
-        try:
-            shutil.copyfile(in_path, out_path)
-        except shutil.Error:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to copy: %s and %s are the same" % (in_path, out_path))
-        except IOError:
-            traceback.print_exc()
-            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
-
-    def fetch_file(self, in_path, out_path):
-        ''' fetch a file from local to local -- for copatibility '''
-        self.put_file(in_path, out_path)
-
-    def close(self):
-        ''' terminate the connection; nothing to do here '''
-
-        pass

lib/ansible/constants.py

@@ -14,26 +14,71 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
 
 import os
+import pwd
+import ConfigParser
+import traceback
 
+def get_config(p, section, key, env_var, default):
+    if p is not None:
+        try:
+            return p.get(section, key)
+        except:
+            if env_var is not None:
+                return os.environ.get(env_var, default)
+            return default
+    else:
+        if env_var is not None:
+            return os.environ.get(env_var, default)
+        return default
 
-DEFAULT_HOST_LIST      = os.environ.get('ANSIBLE_HOSTS',
-    '/etc/ansible/hosts')
-DEFAULT_MODULE_PATH    = os.environ.get('ANSIBLE_LIBRARY',
-    '/usr/share/ansible')
+def load_config_file():
+    p = ConfigParser.ConfigParser()
+    path1 = os.path.expanduser(
+        os.environ.get('ANSIBLE_CONFIG', "~/.ansible.cfg"))
+    path2 = "/etc/ansible/ansible.cfg"
+    if os.path.exists(path1):
+        p.read(path1)
+    elif os.path.exists(path2):
+        p.read(path2)
+    else:
+        return None
+    return p
 
-DEFAULT_MODULE_NAME    = 'command'
-DEFAULT_PATTERN        = '*'
-DEFAULT_FORKS          = 5
-DEFAULT_MODULE_ARGS    = ''
-DEFAULT_TIMEOUT        = 10
-DEFAULT_POLL_INTERVAL  = 15
-DEFAULT_REMOTE_USER    = 'root'
-DEFAULT_REMOTE_PASS    = None
-DEFAULT_SUDO_PASS      = None
-DEFAULT_REMOTE_PORT    = 22
-DEFAULT_TRANSPORT      = 'paramiko'
-DEFAULT_TRANSPORT_OPTS = ['local', 'paramiko']
+def shell_expand_path(path):
+    ''' shell_expand_path is needed as os.path.expanduser does not work 
+        when path is None, which is the default for ANSIBLE_PRIVATE_KEY_FILE '''
+    if path:
+        path = os.path.expanduser(path)
+    return path
 
+p = load_config_file()
+
+active_user   = pwd.getpwuid(os.geteuid())[0]
+
+# sections in config file
+DEFAULTS='defaults'
+
+# configurable things
+DEFAULT_HOST_LIST         = shell_expand_path(get_config(p, DEFAULTS, 'hostfile',         'ANSIBLE_HOSTS',            '/etc/ansible/hosts'))
+DEFAULT_MODULE_PATH       = shell_expand_path(get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          '/usr/share/ansible'))
+DEFAULT_REMOTE_TMP        = shell_expand_path(get_config(p, DEFAULTS, 'remote_tmp',       'ANSIBLE_REMOTE_TEMP',      '$HOME/.ansible/tmp'))
+DEFAULT_MODULE_NAME       = get_config(p, DEFAULTS, 'module_name',      None,                       'command')
+DEFAULT_PATTERN           = get_config(p, DEFAULTS, 'pattern',          None,                       '*')
+DEFAULT_FORKS             = get_config(p, DEFAULTS, 'forks',            'ANSIBLE_FORKS',            5)
+DEFAULT_MODULE_ARGS       = get_config(p, DEFAULTS, 'module_args',      'ANSIBLE_MODULE_ARGS',      '')
+DEFAULT_TIMEOUT           = get_config(p, DEFAULTS, 'timeout',          'ANSIBLE_TIMEOUT',          10)
+DEFAULT_POLL_INTERVAL     = get_config(p, DEFAULTS, 'poll_interval',    'ANSIBLE_POLL_INTERVAL',    15)
+DEFAULT_REMOTE_USER       = get_config(p, DEFAULTS, 'remote_user',      'ANSIBLE_REMOTE_USER',      active_user)
+DEFAULT_PRIVATE_KEY_FILE  = shell_expand_path(get_config(p, DEFAULTS, 'private_key_file', 'ANSIBLE_PRIVATE_KEY_FILE', None))
+DEFAULT_SUDO_USER         = get_config(p, DEFAULTS, 'sudo_user',        'ANSIBLE_SUDO_USER',        'root')
+DEFAULT_REMOTE_PORT       = int(get_config(p, DEFAULTS, 'remote_port',      'ANSIBLE_REMOTE_PORT',      22))
+DEFAULT_TRANSPORT         = get_config(p, DEFAULTS, 'transport',        'ANSIBLE_TRANSPORT',        'paramiko')
+
+# non-configurable things
+DEFAULT_REMOTE_PASS       = None
+DEFAULT_SUDO_PASS         = None
+DEFAULT_SUBSET            = None
+
+ANSIBLE_SSH_ARGS          = get_config(p, 'ssh_connection', 'ssh_args', 'ANSIBLE_SSH_ARGS', None)

lib/ansible/errors.py

@@ -15,23 +15,20 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-
 class AnsibleError(Exception):
-    """
-    The base Ansible exception from which all others should subclass.
-    """
+    ''' The base Ansible exception from which all others should subclass '''
 
     def __init__(self, msg):
         self.msg = msg
-    
+
     def __str__(self):
         return self.msg
 
-
 class AnsibleFileNotFound(AnsibleError):
     pass
 
 class AnsibleConnectionFailed(AnsibleError):
     pass
 
-
+class AnsibleYAMLValidationFailed(AnsibleError):
+    pass

lib/ansible/inventory.py

@@ -1,327 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-import fnmatch
-import os
-import subprocess
-
-import constants as C
-from ansible import errors
-from ansible import utils
-
-class Inventory(object):
-    """ Host inventory for ansible.
-
-    The inventory is either a simple text file with systems and [groups] of 
-    systems, or a script that will be called with --list or --host.
-    """
-
-    def __init__(self, host_list=C.DEFAULT_HOST_LIST, global_host_vars={}):
-
-        self._restriction = None
-        self._variables = {}
-        self._global_host_vars = global_host_vars # lets us pass in a global var list
-                                                 # that each host gets
-                                                 # after we have set up the inventory
-                                                 # to get all group variables
-        
-        if type(host_list) == list:
-            self.host_list = host_list
-            self.groups = dict(ungrouped=host_list)
-            self._is_script = False
-            return
-
-        inventory_file = os.path.expanduser(host_list)
-        if not os.path.exists(inventory_file):
-            raise errors.AnsibleFileNotFound("inventory file not found: %s" % host_list)
-
-        self.inventory_file = os.path.abspath(inventory_file)
-
-        if os.access(self.inventory_file, os.X_OK):
-            self.host_list, self.groups = self._parse_from_script()
-            self._is_script = True
-        else:
-            self.host_list, self.groups = self._parse_from_file()
-            self._is_script = False
-
-    # *****************************************************
-    # Public API
-
-    def list_hosts(self, pattern="all"):
-        """ Return a list of hosts [matching the pattern] """
-        if self._restriction is None:
-            host_list = self.host_list
-        else:
-            host_list = [ h for h in self.host_list if h in self._restriction ]
-        return [ h for h in host_list if self._matches(h, pattern) ]
-
-    def restrict_to(self, restriction):
-        """ Restrict list operations to the hosts given in restriction """
-        if type(restriction)!=list:
-            restriction = [ restriction ]
-
-        self._restriction = restriction
-
-    def lift_restriction(self):
-        """ Do not restrict list operations """
-        self._restriction = None
-
-    def get_variables(self, host):
-        """ Return the variables associated with this host. """
-
-        variables = {
-            'inventory_hostname': host,
-        }
-
-        if host in self._variables:
-            variables.update(self._variables[host].copy())
-
-        if self._is_script:
-            variables.update(self._get_variables_from_script(host))
-
-        variables['group_names'] = []
-        for name,hosts in self.groups.iteritems():
-            if host in hosts:
-                variables['group_names'].append(name)
-
-        return variables
-
-    def get_global_vars(self):
-        return self._global_host_vars
-        
-    # *****************************************************
-
-    def _parse_from_file(self):
-        ''' parse a textual host file '''
-
-        results = []
-        groups = dict(ungrouped=[])
-        lines = file(self.inventory_file).read().split("\n")
-        if "---" in lines:
-            return self._parse_yaml()
-        group_name = 'ungrouped'
-        for item in lines:
-            item = item.lstrip().rstrip()
-            if item.startswith("#"):
-                # ignore commented out lines
-                pass
-            elif item.startswith("["):
-                # looks like a group
-                group_name = item.replace("[","").replace("]","").lstrip().rstrip()
-                groups[group_name] = []
-            elif item != "":
-                # looks like a regular host
-                if ":" in item:
-                    # a port was specified
-                    item, port = item.split(":")
-                    try:
-                        port = int(port)
-                    except ValueError:
-                        raise errors.AnsibleError("SSH port for %s in inventory (%s) should be numerical."%(item, port))
-                    self._set_variable(item, "ansible_ssh_port", port)
-                groups[group_name].append(item)
-                if not item in results:
-                    results.append(item)
-        return (results, groups)
-
-    # *****************************************************
-
-    def _parse_from_script(self):
-        ''' evaluate a script that returns list of hosts by groups '''
-
-        results = []
-        groups = dict(ungrouped=[])
-
-        cmd = [self.inventory_file, '--list']
-
-        try:
-            cmd = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=False)
-            out, err = cmd.communicate()
-        except Exception, e:
-            raise errors.AnsibleError("Failure executing %s to produce host list:\n %s" % (self.inventory_file, str(e)))
-
-        rc = cmd.returncode
-        if rc:
-            raise errors.AnsibleError("%s: %s" % self.inventory_file, err)
-
-        try:
-            groups = utils.json_loads(out)
-        except:
-            raise errors.AnsibleError("invalid JSON response from script: %s" % self.inventory_file)
-
-        for (groupname, hostlist) in groups.iteritems():
-            for host in hostlist:
-                if host not in results:
-                    results.append(host)
-        return (results, groups)
-
-    # *****************************************************
-
-    def _parse_yaml(self):
-        """ Load the inventory from a yaml file.
-
-        returns hosts and groups"""
-        data = utils.parse_yaml_from_file(self.inventory_file)
-
-        if type(data) != list:
-            raise errors.AnsibleError("YAML inventory should be a list.")
-
-        hosts = []
-        groups = {}
-        ungrouped = []
-
-        
-        # go through once and grab up the global_host_vars from the 'all' group
-        for item in data:
-            if type(item) == dict:
-                if "group" in item and 'all' == item["group"]:
-                    if "vars" in item:
-                        variables = item['vars']
-                        if type(variables) == list:
-                            for variable in variables:
-                                if len(variable) != 1:
-                                    raise errors.AnsibleError("Only one item expected in %s"%(variable))
-                                k, v = variable.items()[0]
-                                self._global_host_vars[k] = utils.template(v, self._global_host_vars, {})
-                        elif type(variables) == dict:
-                            self._global_host_vars.update(variables)
-
-        for item in data:
-            if type(item) == dict:
-                if "group" in item:
-                    group_name = item["group"]
-
-                    group_vars = []
-                    if "vars" in item:
-                        group_vars.extend(item["vars"])
-
-                    group_hosts = []
-                    if "hosts" in item:
-                        for host in item["hosts"]:
-                            host_name = self._parse_yaml_host(host, group_vars)
-                            group_hosts.append(host_name)
-
-                    groups[group_name] = group_hosts
-                    hosts.extend(group_hosts)
-
-                elif "host" in item:
-                    host_name = self._parse_yaml_host(item)
-                    hosts.append(host_name)
-                    ungrouped.append(host_name)
-            else:
-                host_name = self._parse_yaml_host(item)
-                hosts.append(host_name)
-                ungrouped.append(host_name)
-
-        # filter duplicate hosts
-        output_hosts = []
-        for host in hosts:
-            if host not in output_hosts:
-                output_hosts.append(host)
-
-        if len(ungrouped) > 0 :
-            # hosts can be defined top-level, but also in a group
-            really_ungrouped = []
-            for host in ungrouped:
-                already_grouped = False
-                for name, group_hosts in groups.items():
-                    if host in group_hosts:
-                        already_grouped = True
-                if not already_grouped:
-                    really_ungrouped.append(host)
-            groups["ungrouped"] = really_ungrouped
-
-        return output_hosts, groups
-
-    def _parse_yaml_host(self, item, variables=[]):
-        def set_variables(host, variables):
-            if type(variables) == list:
-                for variable in variables:
-                    if len(variable) != 1:
-                        raise errors.AnsibleError("Only one item expected in %s"%(variable))
-                    k, v = variable.items()[0]
-                    self._set_variable(host, k, v)
-            elif type(variables) == dict:
-                for k, v in variables.iteritems():
-                    self._set_variable(host, k, v)
-
-
-        if type(item) in [str, unicode]:
-            set_variables(item, variables)
-            return item
-        elif type(item) == dict:
-            if "host" in item:
-                host_name = item["host"]
-                set_variables(host_name, variables)
-
-                if "vars" in item:
-                    set_variables(host_name, item["vars"])
-
-                return host_name
-        else:
-            raise errors.AnsibleError("Unknown item in inventory: %s"%(item))
-
-
-    def _get_variables_from_script(self, host):
-        ''' support per system variabes from external variable scripts, see web docs '''
-
-        cmd = [self.inventory_file, '--host', host]
-
-        cmd = subprocess.Popen(cmd,
-            stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE,
-            shell=False
-        )
-        out, err = cmd.communicate()
-
-        variables = {}
-        try:
-            variables = utils.json_loads(out)
-        except:
-            raise errors.AnsibleError("%s returned invalid result when called with hostname %s" % (
-                self.inventory_file,
-                host
-            ))
-        return variables
-
-    def _set_variable(self, host, key, value):
-        if not host in self._variables:
-            self._variables[host] = {}
-        self._variables[host][key] = value
-
-    def _matches(self, host_name, pattern):
-        ''' returns if a hostname is matched by the pattern '''
-
-        # a pattern is in fnmatch format but more than one pattern
-        # can be strung together with semicolons. ex:
-        #   atlanta-web*.example.com;dc-web*.example.com
-
-        if host_name == '':
-            return False
-        pattern = pattern.replace(";",":")
-        subpatterns = pattern.split(":")
-        for subpattern in subpatterns:
-            if subpattern == 'all':
-                return True
-            if fnmatch.fnmatch(host_name, subpattern):
-                return True
-            elif subpattern in self.groups:
-                if host_name in self.groups[subpattern]:
-                    return True
-        return False

lib/ansible/inventory/__init__.py

@@ -0,0 +1,340 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import fnmatch
+import os
+
+import subprocess
+import ansible.constants as C
+from ansible.inventory.ini import InventoryParser
+from ansible.inventory.script import InventoryScript
+from ansible.inventory.group import Group
+from ansible.inventory.host import Host
+from ansible import errors
+from ansible import utils
+
+class Inventory(object):
+    """
+    Host inventory for ansible.
+    """
+
+    __slots__ = [ 'host_list', 'groups', '_restriction', '_also_restriction', '_subset', '_is_script',
+                  'parser', '_vars_per_host', '_vars_per_group', '_hosts_cache' ]
+
+    def __init__(self, host_list=C.DEFAULT_HOST_LIST):
+
+        # the host file file, or script path, or list of hosts
+        # if a list, inventory data will NOT be loaded
+        self.host_list = host_list
+
+        # caching to avoid repeated calculations, particularly with
+        # external inventory scripts.
+
+        self._vars_per_host  = {}
+        self._vars_per_group = {}
+        self._hosts_cache    = {}
+
+        # the inventory object holds a list of groups
+        self.groups = []
+
+        # a list of host(names) to contain current inquiries to
+        self._restriction = None
+        self._also_restriction = None
+        self._subset = None
+
+        # whether the inventory file is a script
+        self._is_script = False
+
+        if type(host_list) in [ str, unicode ]:
+            if host_list.find(",") != -1:
+                host_list = host_list.split(",")
+                host_list = [ h for h in host_list if h and h.strip() ]
+
+        if type(host_list) == list:
+            all = Group('all')
+            self.groups = [ all ]
+            for x in host_list:
+                if x.find(":") != -1:
+                    tokens = x.split(":",1)
+                    all.add_host(Host(tokens[0], tokens[1]))
+                else:
+                    all.add_host(Host(x))
+        elif os.access(host_list, os.X_OK):
+            self._is_script = True
+            self.parser = InventoryScript(filename=host_list)
+            self.groups = self.parser.groups.values()
+        else:
+            data = file(host_list).read()
+            if not data.startswith("---"):
+                self.parser = InventoryParser(filename=host_list)
+                self.groups = self.parser.groups.values()
+            else:
+                raise errors.AnsibleError("YAML inventory support is deprecated in 0.6 and removed in 0.7, see the migration script in examples/scripts in the git checkout")
+
+    def _match(self, str, pattern_str):
+        return fnmatch.fnmatch(str, pattern_str)
+
+    def get_hosts(self, pattern="all"):
+        """ 
+        find all host names matching a pattern string, taking into account any inventory restrictions or
+        applied subsets.
+        """
+
+        # process patterns        
+        patterns = pattern.replace(";",":").split(":")
+        positive_patterns = [ p for p in patterns if not p.startswith("!") ]
+        negative_patterns = [ p for p in patterns if p.startswith("!") ]
+
+        # find hosts matching positive patterns
+        hosts = self._get_hosts(positive_patterns)
+
+        # exclude hosts mentioned in a negative pattern
+        if len(negative_patterns):
+            exclude_hosts = self._get_hosts(negative_patterns)
+            hosts = [ h for h in hosts if h not in exclude_hosts ]
+
+        # exclude hosts not in a subset, if defined
+        if self._subset:
+            positive_subsetp = [ p for p in self._subset if not p.startswith("!") ]
+            negative_subsetp = [ p for p in self._subset if p.startswith("!") ]
+            if len(positive_subsetp):
+                positive_subset = [ h.name for h in self._get_hosts(positive_subsetp) ]
+                hosts = [ h for h in hosts if (h.name in positive_subset) ]
+            if len(negative_subsetp):
+                negative_subset = [ h.name for h in self._get_hosts(negative_subsetp) ]
+                hosts = [ h for h in hosts if (h.name not in negative_subset)]
+
+        # exclude hosts mentioned in any restriction (ex: failed hosts)
+        if self._restriction is not None:
+            hosts = [ h for h in hosts if h.name in self._restriction ]
+        if self._also_restriction is not None:
+            hosts = [ h for h in hosts if h.name in self._also_restriction ]
+
+        return sorted(hosts, key=lambda x: x.name)
+
+    def _get_hosts(self, patterns):
+        """ 
+        finds hosts that postively match a particular list of patterns.  Does not
+        take into account negative matches.
+        """
+
+        by_pattern = {}
+        for p in patterns:
+            (name, enumeration_details) = self._enumeration_info(p)
+            hpat = self._hosts_in_unenumerated_pattern(name)
+            hpat = sorted(hpat, key=lambda x: x.name)
+            by_pattern[p] = hpat
+
+        ranged = {}
+        for (pat, hosts) in by_pattern.iteritems():
+            ranged[pat] = self._apply_ranges(pat, hosts)
+
+        results = []
+        for (pat, hosts) in ranged.iteritems():
+            results.extend(hosts)
+
+        return list(set(results))
+
+    def _enumeration_info(self, pattern):
+        """
+        returns (pattern, limits) taking a regular pattern and finding out
+        which parts of it correspond to start/stop offsets.  limits is
+        a tuple of (start, stop) or None
+        """
+
+        if not "[" in pattern:
+            return (pattern, None)
+        (first, rest) = pattern.split("[")
+        rest = rest.replace("]","")
+        if not "-" in rest:
+            raise errors.AnsibleError("invalid pattern: %s" % pattern)
+        (left, right) = rest.split("-",1)
+        return (first, (left, right))
+
+    def _apply_ranges(self, pat, hosts):
+        """
+        given a pattern like foo, that matches hosts, return all of hosts
+        given a pattern like foo[0:5], where foo matches hosts, return the first 6 hosts
+        """ 
+
+        (loose_pattern, limits) = self._enumeration_info(pat)
+        if not limits:
+            return hosts
+
+        (left, right) = limits
+        enumerated = enumerate(hosts)
+        if left == '':
+            left = 0
+        if right == '':
+            right = 0
+        left=int(left)
+        right=int(right)
+        enumerated = [ h for (i,h) in enumerated if i>=left and i<=right ]
+        return enumerated
+
+    # TODO: cache this logic so if called a second time the result is not recalculated
+    def _hosts_in_unenumerated_pattern(self, pattern):
+        """ Get all host names matching the pattern """
+
+        hosts = {}
+        # ignore any negative checks here, this is handled elsewhere
+        pattern = pattern.replace("!","")
+
+        groups = self.get_groups()
+        for group in groups:
+            for host in group.get_hosts():
+                if pattern == 'all' or self._match(group.name, pattern) or self._match(host.name, pattern):
+                    hosts[host.name] = host
+        return sorted(hosts.values(), key=lambda x: x.name)
+
+    def groups_for_host(self, host):
+        results = []
+        groups = self.get_groups()
+        for group in groups:
+            for hostn in group.get_hosts():
+                if host == hostn.name:
+                    results.append(group)
+                    continue
+        return results
+
+    def get_groups(self):
+        return self.groups
+
+    def get_host(self, hostname):
+        if hostname not in self._hosts_cache:
+            self._hosts_cache[hostname] = self._get_host(hostname)
+        return self._hosts_cache[hostname]
+
+    def _get_host(self, hostname):
+        for group in self.groups:
+            for host in group.get_hosts():
+                if hostname == host.name:
+                    return host
+        return None
+
+    def get_group(self, groupname):
+        for group in self.groups:
+            if group.name == groupname:
+                return group
+        return None
+
+    def get_group_variables(self, groupname):
+        if groupname not in self._vars_per_group:
+            self._vars_per_group[groupname] = self._get_group_variables(groupname)
+        return self._vars_per_group[groupname]
+
+    def _get_group_variables(self, groupname):
+        group = self.get_group(groupname)
+        if group is None:
+            raise Exception("group not found: %s" % groupname)
+        return group.get_variables()
+
+    def get_variables(self, hostname):
+        if hostname not in self._vars_per_host:
+            self._vars_per_host[hostname] = self._get_variables(hostname)
+        return self._vars_per_host[hostname]
+
+    def _get_variables(self, hostname):
+
+        if self._is_script:
+            host = self.get_host(hostname)
+            cmd = subprocess.Popen(
+                [self.host_list,"--host",hostname],
+                stdout=subprocess.PIPE,
+                stderr=subprocess.PIPE
+            )
+            (out, err) = cmd.communicate()
+            results = utils.parse_json(out)
+
+            # FIXME: this is a bit redundant with host.py and should share code
+            results['inventory_hostname'] = hostname
+            results['inventory_hostname_short'] = hostname.split('.')[0]
+            groups = [ g.name for g in host.get_groups() if g.name != 'all' ]
+            results['group_names'] = sorted(groups)
+
+            return results
+
+        host = self.get_host(hostname)
+        if host is None:
+            raise Exception("host not found: %s" % hostname)
+        return host.get_variables()
+
+    def add_group(self, group):
+        self.groups.append(group)
+
+    def list_hosts(self, pattern="all"):
+        return [ h.name for h in self.get_hosts(pattern) ]
+
+    def list_groups(self):
+        return sorted([ g.name for g in self.groups ], key=lambda x: x.name)
+
+    # TODO: remove this function
+    def get_restriction(self):
+        return self._restriction
+
+    def restrict_to(self, restriction):
+        """ 
+        Restrict list operations to the hosts given in restriction.  This is used
+        to exclude failed hosts in main playbook code, don't use this for other
+        reasons.
+        """
+        if type(restriction) != list:
+            restriction = [ restriction ]
+        self._restriction = restriction
+
+    def also_restrict_to(self, restriction):
+        """
+        Works like restict_to but offers an additional restriction.  Playbooks use this
+        to implement serial behavior.
+        """
+        if type(restriction) != list:
+            restriction = [ restriction ]
+        self._also_restriction = restriction
+    
+    def subset(self, subset_pattern):
+        """ 
+        Limits inventory results to a subset of inventory that matches a given
+        pattern, such as to select a given geographic of numeric slice amongst
+        a previous 'hosts' selection that only select roles, or vice versa.  
+        Corresponds to --limit parameter to ansible-playbook
+        """        
+        if subset_pattern is None:
+            self._subset = None
+        else:
+            self._subset = subset_pattern.replace(";",":").split(":")
+
+    def lift_restriction(self):
+        """ Do not restrict list operations """
+        self._restriction = None
+    
+    def lift_also_restriction(self):
+        """ Clears the also restriction """
+        self._also_restriction = None
+
+    def is_file(self):
+        """ did inventory come from a file? """
+        if not isinstance(self.host_list, basestring):
+            return False
+        return os.path.exists(self.host_list)
+
+    def basedir(self):
+        """ if inventory came from a file, what's the directory? """
+        if not self.is_file():
+            return None
+        return os.path.dirname(self.host_list)

lib/ansible/inventory/expand_hosts.py

@@ -0,0 +1,97 @@
+# (c) 2012, Zettar Inc.
+# Written by Chin Fang <fangchin@zettar.com>
+#
+# This file is part of Ansible
+#
+# This module is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This software is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this software.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+'''
+This module is for enhancing ansible's inventory parsing capability such
+that it can deal with hostnames specified using a simple pattern in the
+form of [beg:end], example: [1:5] where if beg is not specified, it
+defaults to 0.
+
+If beg is given and is left-zero-padded, e.g. '001', it is taken as a
+formatting hint when the range is expanded. e.g. [001:010] is to be
+expanded into 001, 002 ...009, 010.
+
+Note that when beg is specified with left zero padding, then the length of
+end must be the same as that of beg, else a exception is raised.
+'''
+
+from ansible import errors
+
+def detect_range(line = None):
+    '''
+    A helper function that checks a given host line to see if it contains
+    a range pattern descibed in the docstring above.
+
+    Returnes True if the given line contains a pattern, else False.
+    '''
+    if (not line.startswith("[") and
+        line.find("[") != -1 and
+        line.find(":") != -1 and
+        line.find("]") != -1 and
+        line.index("[") < line.index(":") < line.index("]")):
+        return True
+    else:
+        return False
+
+def expand_hostname_range(line = None):
+    '''
+    A helper function that expands a given line that contains a pattern
+    specified in top docstring, and returns a list that consists of the
+    expanded version.
+
+    The '[' and ']' characters are used to maintain the pseudo-code
+    appearance. They are replaced in this function with '|' to ease
+    string splitting.
+
+    References: http://ansible.github.com/patterns.html#hosts-and-groups
+    '''
+    all_hosts = []
+    if line:
+        # A hostname such as db[1:6]-node is considered to consists
+        # three parts:
+        # head: 'db'
+        # nrange: [1:6]; range() is a built-in. Can't use the name
+        # tail: '-node'
+
+        (head, nrange, tail) = line.replace('[','|').replace(']','|').split('|')
+        bounds = nrange.split(":")
+        if len(bounds) != 2:
+            raise errors.AnsibleError("host range incorrectly specified")
+        beg = bounds[0]
+        end = bounds[1]
+        if not beg:
+            beg = "0"
+        if not end:
+            raise errors.AnsibleError("host range end value missing")
+        if beg[0] == '0' and len(beg) > 1:
+            rlen = len(beg) # range length formatting hint
+        else:
+            rlen = None
+        if rlen > 1 and rlen != len(end):
+            raise errors.AnsibleError("host range format incorrectly specified!")
+
+        for _ in range(int(beg), int(end)+1):
+            if rlen:
+                rseq = str(_).zfill(rlen) # range sequence
+            else:
+                rseq = str(_)
+            hname = ''.join((head, rseq, tail))
+            all_hosts.append(hname)
+
+        return all_hosts

lib/ansible/inventory/group.py

@@ -0,0 +1,79 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+class Group(object):
+    ''' a group of ansible hosts '''
+
+    __slots__ = [ 'name', 'hosts', 'vars', 'child_groups', 'parent_groups', 'depth' ]
+
+    def __init__(self, name=None):
+
+        self.depth = 0
+        self.name = name
+        self.hosts = []
+        self.vars = {}
+        self.child_groups = []
+        self.parent_groups = []
+        if self.name is None:
+            raise Exception("group name is required")
+
+    def add_child_group(self, group):
+
+        if self == group:
+            raise Exception("can't add group to itself")
+        self.child_groups.append(group)
+        group.depth = group.depth + 1
+        group.parent_groups.append(self)
+
+    def add_host(self, host):
+
+        self.hosts.append(host)
+        host.add_group(self)
+
+    def set_variable(self, key, value):
+
+        self.vars[key] = value
+
+    def get_hosts(self):
+
+        hosts = []
+        for kid in self.child_groups:
+            hosts.extend(kid.get_hosts())
+        hosts.extend(self.hosts)
+        return hosts
+
+    def get_variables(self):
+
+        vars = {}
+        # FIXME: verify this variable override order is what we want
+        for ancestor in self.get_ancestors():
+            vars.update(ancestor.get_variables())
+        vars.update(self.vars)
+        return vars
+
+    def _get_ancestors(self):
+
+        results = {}
+        for g in self.parent_groups:
+            results[g.name] = g
+            results.update(g._get_ancestors())
+        return results
+
+    def get_ancestors(self):
+
+        return self._get_ancestors().values()
+

lib/ansible/inventory/host.py

@@ -0,0 +1,64 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.constants as C
+
+class Host(object):
+    ''' a single ansible host '''
+
+    __slots__ = [ 'name', 'vars', 'groups' ]
+
+    def __init__(self, name=None, port=None):
+
+        self.name = name
+        self.vars = {}
+        self.groups = []
+        if port and port != C.DEFAULT_REMOTE_PORT:
+            self.set_variable('ansible_ssh_port', int(port))
+
+        if self.name is None:
+            raise Exception("host name is required")
+
+    def add_group(self, group):
+
+        self.groups.append(group)
+
+    def set_variable(self, key, value):
+
+        self.vars[key]=value
+
+    def get_groups(self):
+
+        groups = {}
+        for g in self.groups:
+            groups[g.name] = g
+            ancestors = g.get_ancestors()
+            for a in ancestors:
+                groups[a.name] = a
+        return groups.values()
+
+    def get_variables(self):
+
+        results = {}
+        for group in self.groups:
+            results.update(group.get_variables())
+        results.update(self.vars)
+        results['inventory_hostname'] = self.name
+        results['inventory_hostname_short'] = self.name.split('.')[0]
+        groups = self.get_groups()
+        results['group_names'] = sorted([ g.name for g in groups if g.name != 'all'])
+        return results

lib/ansible/inventory/ini.py

@@ -0,0 +1,168 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible.inventory.expand_hosts import detect_range
+from ansible.inventory.expand_hosts import expand_hostname_range
+from ansible import errors
+import shlex
+
+class InventoryParser(object):
+    """
+    Host inventory for ansible.
+    """
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        fh = open(filename)
+        self.lines = fh.readlines()
+        self.groups = {}
+        self.hosts = {}
+        self._parse()
+
+    def _parse(self):
+
+        self._parse_base_groups()
+        self._parse_group_children()
+        self._parse_group_variables()
+        return self.groups
+
+
+    # [webservers]
+    # alpha
+    # beta:2345
+    # gamma sudo=True user=root
+    # delta asdf=jkl favcolor=red
+
+    def _parse_base_groups(self):
+        # FIXME: refactor
+
+        ungrouped = Group(name='ungrouped')
+        all = Group(name='all')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        active_group_name = 'ungrouped'
+
+        for line in self.lines:
+            if line.startswith("["):
+                active_group_name = line.replace("[","").replace("]","").strip()
+                if line.find(":vars") != -1 or line.find(":children") != -1:
+                    active_group_name = None
+                else:
+                    new_group = self.groups[active_group_name] = Group(name=active_group_name)
+                    all.add_child_group(new_group)
+            elif line.startswith("#") or line == '':
+                pass
+            elif active_group_name:
+                tokens = shlex.split(line)
+                if len(tokens) == 0:
+                    continue
+                hostname = tokens[0]
+                port = C.DEFAULT_REMOTE_PORT
+                # Two cases to check:
+                # 0. A hostname that contains a range pesudo-code and a port
+                # 1. A hostname that contains just a port
+                if (hostname.find("[") != -1 and
+                    hostname.find("]") != -1 and
+                    hostname.find(":") != -1 and
+                    (hostname.rindex("]") < hostname.rindex(":")) or
+                    (hostname.find("]") == -1 and hostname.find(":") != -1)):
+                        tokens2  = hostname.rsplit(":", 1)
+                        hostname = tokens2[0]
+                        port     = tokens2[1]
+
+                host = None
+                _all_hosts = []
+                if hostname in self.hosts:
+                    host = self.hosts[hostname]
+                    _all_hosts.append(host)
+                else:
+                    if detect_range(hostname):
+                        _hosts = expand_hostname_range(hostname)
+                        for _ in _hosts:
+                            host = Host(name=_, port=port)
+                            self.hosts[_] = host
+                            _all_hosts.append(host)
+                    else:
+                        host = Host(name=hostname, port=port)
+                        self.hosts[hostname] = host
+                        _all_hosts.append(host)
+                if len(tokens) > 1:
+                    for t in tokens[1:]:
+                        (k,v) = t.split("=")
+                        host.set_variable(k,v)
+                for _ in _all_hosts:
+                    self.groups[active_group_name].add_host(_)
+
+    # [southeast:children]
+    # atlanta
+    # raleigh
+
+    def _parse_group_children(self):
+        group = None
+
+        for line in self.lines:
+            line = line.strip()
+            if line is None or line == '':
+                continue
+            if line.startswith("[") and line.find(":children]") != -1:
+                line = line.replace("[","").replace(":children]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                    group = self.groups[line] = Group(name=line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif group:
+                kid_group = self.groups.get(line, None)
+                if kid_group is None:
+                    raise errors.AnsibleError("child group is not defined: (%s)" % line)
+                else:
+                    group.add_child_group(kid_group)
+
+
+    # [webservers:vars]
+    # http_port=1234
+    # maxRequestsPerChild=200
+
+    def _parse_group_variables(self):
+        group = None
+        for line in self.lines:
+            line = line.strip()
+            if line.startswith("[") and line.find(":vars]") != -1:
+                line = line.replace("[","").replace(":vars]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                    raise errors.AnsibleError("can't add vars to undefined group: %s" % line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif line == '':
+                pass
+            elif group:
+                if line.find("=") == -1:
+                    raise errors.AnsibleError("variables assigned to group must be in key=value form")
+                else:
+                    (k,v) = line.split("=",1)
+                    group.set_variable(k,v)

lib/ansible/inventory/script.py

@@ -0,0 +1,53 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import subprocess
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import utils
+
+class InventoryScript(object):
+    ''' Host inventory parser for ansible using external inventory scripts. '''
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        cmd = [ filename, "--list" ]
+        sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (stdout, stderr) = sp.communicate()
+        self.data = stdout
+        self.groups = self._parse()
+
+    def _parse(self):
+
+        groups = {}
+        self.raw = utils.parse_json(self.data)
+        all=Group('all')
+        self.groups = dict(all=all)
+        group = None
+        for (group_name, hosts) in self.raw.items():
+            group = groups[group_name] = Group(group_name)
+            host = None
+            for hostname in hosts:
+                host = Host(hostname)
+                group.add_host(host)
+                # FIXME: hack shouldn't be needed
+                all.add_host(host)
+            all.add_child_group(group)
+        return groups

lib/ansible/module_common.py

@@ -0,0 +1,294 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+REPLACER = "#<<INCLUDE_ANSIBLE_MODULE_COMMON>>"
+REPLACER_ARGS = "<<INCLUDE_ANSIBLE_MODULE_ARGS>>"
+
+MODULE_COMMON = """
+
+# == BEGIN DYNAMICALLY INSERTED CODE ==
+
+MODULE_ARGS = <<INCLUDE_ANSIBLE_MODULE_ARGS>>
+
+BOOLEANS_TRUE = ['yes', 'on', '1', 'true', 1]
+BOOLEANS_FALSE = ['no', 'off', '0', 'false', 0]
+BOOLEANS = BOOLEANS_TRUE + BOOLEANS_FALSE
+
+# ansible modules can be written in any language.  To simplify
+# development of Python modules, the functions available here
+# can be inserted in any module source automatically by including
+# #<<INCLUDE_ANSIBLE_MODULE_COMMON>> on a blank line by itself inside
+# of an ansible module. The source of this common code lives
+# in lib/ansible/module_common.py
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+import base64
+import os
+import re
+import shlex
+import subprocess
+import sys
+import syslog
+import types
+import time
+import shutil
+
+try:
+    from hashlib import md5 as _md5
+except ImportError:
+    from md5 import md5 as _md5
+
+class AnsibleModule(object):
+
+    def __init__(self, argument_spec, bypass_checks=False, no_log=False, 
+        check_invalid_arguments=True, mutually_exclusive=None, required_together=None,
+        required_one_of=None):
+
+        '''
+        common code for quickly building an ansible module in Python
+        (although you can write modules in anything that can return JSON)
+        see library/* for examples
+        '''
+
+        self.argument_spec = argument_spec
+        (self.params, self.args) = self._load_params()
+
+        self._legal_inputs = []
+        self._handle_aliases()
+
+        # this may be disabled where modules are going to daisy chain into others
+        if check_invalid_arguments:
+            self._check_invalid_arguments()
+
+        self._set_defaults(pre=True)
+
+        if not bypass_checks:
+            self._check_required_arguments()
+            self._check_argument_types()
+            self._check_mutually_exclusive(mutually_exclusive)
+            self._check_required_together(required_together)
+            self._check_required_one_of(required_one_of)
+
+        self._set_defaults(pre=False)
+        if not no_log:
+            self._log_invocation()
+
+    def _handle_aliases(self):
+        for (k,v) in self.argument_spec.iteritems():
+            self._legal_inputs.append(k)
+            aliases = v.get('aliases', None)
+            default = v.get('default', None)
+            required = v.get('required', False)
+            if default is not None and required:
+                # not alias specific but this is a good place to check this
+                self.fail_json(msg="internal error: required and default are mutally exclusive for %s" % k)
+            if aliases is None:
+                continue
+            if type(aliases) != list:
+                self.fail_json(msg='internal error: aliases must be a list')
+            for alias in aliases:
+                self._legal_inputs.append(alias)
+                if alias in self.params:
+                    self.params[k] = self.params[alias]
+
+    def _check_invalid_arguments(self):
+        for (k,v) in self.params.iteritems():
+            if k not in self._legal_inputs:
+                self.fail_json(msg="unsupported parameter for module: %s" % k)
+
+    def _count_terms(self, check):
+        count = 0
+        for term in check:
+           if term in self.params:
+               count += 1
+        return count
+
+    def _check_mutually_exclusive(self, spec):
+        if spec is None:
+            return
+        for check in spec:
+            count = self._count_terms(check)
+            if count > 1:
+                self.fail_json(msg="parameters are mutually exclusive: %s" % check)
+
+    def _check_required_one_of(self, spec):
+        if spec is None:
+            return
+        for check in spec:
+            count = self._count_terms(check)
+            if count == 0:
+                self.fail_json(msg="one of the following is required: %s" % check)
+
+    def _check_required_together(self, spec):
+        if spec is None:
+            return
+        for check in spec:
+            counts = [ self.count_terms([field]) for field in check ]
+            non_zero = [ c for c in counts if c > 0 ]
+            if len(non_zero) > 0:
+                if 0 in counts:
+                    self.fail_json(msg="parameters are required together: %s" % check) 
+
+    def _check_required_arguments(self):
+        ''' ensure all required arguments are present '''
+        missing = []
+        for (k,v) in self.argument_spec.iteritems():
+            required = v.get('required', False)
+            if required and k not in self.params:
+                missing.append(k)
+        if len(missing) > 0:
+            self.fail_json(msg="missing required arguments: %s" % ",".join(missing))
+
+    def _check_argument_types(self):
+        ''' ensure all arguments have the requested values, and there are no stray arguments '''
+        for (k,v) in self.argument_spec.iteritems():
+            choices = v.get('choices',None)
+            if choices is None:
+                continue
+            if type(choices) == list:
+                if k in self.params:
+                    if self.params[k] not in choices:
+                        choices_str=",".join(choices)
+                        msg="value of %s must be one of: %s, got: %s" % (k, choices_str, self.params[k])
+                        self.fail_json(msg=msg)
+            else:
+                self.fail_json(msg="internal error: do not know how to interpret argument_spec")
+
+    def _set_defaults(self, pre=True):
+         for (k,v) in self.argument_spec.iteritems():
+             default = v.get('default', None)
+             if pre == True:
+                 # this prevents setting defaults on required items
+                 if default and k not in self.params:
+                     self.params[k] = default
+             else:
+                 # make sure things without a default still get set None
+                 if k not in self.params:
+                     self.params[k] = default
+
+    def _load_params(self):
+        ''' read the input and return a dictionary and the arguments string '''
+        args = MODULE_ARGS
+        items   = shlex.split(args)
+        params = {}
+        for x in items:
+            try:
+                (k, v) = x.split("=",1)
+            except:
+                self.fail_json(msg="this module requires key=value arguments")
+            params[k] = v
+        return (params, args)
+
+    def _log_invocation(self):
+        ''' log that ansible ran the module '''
+        syslog.openlog('ansible-%s' % os.path.basename(__file__))
+        # Sanitize possible password argument when logging
+        log_args = re.sub(r'password=.+ (.*)', r"password=NOT_LOGGING_PASSWORD \1", self.args)
+        log_args = re.sub(r'login_password=.+ (.*)', r"login_password=NOT_LOGGING_PASSWORD \1", log_args)
+        syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % log_args)
+
+    def get_bin_path(self, arg, required=False, opt_dirs=[]):
+        '''
+        find system executable in PATH.
+        Optional arguments:
+           - required:  if executable is not found and required is true, fail_json
+           - opt_dirs:  optional list of directories to search in addition to PATH
+        if found return full path; otherwise return None
+        '''
+        sbin_paths = ['/sbin', '/usr/sbin', '/usr/local/sbin']
+        paths = []
+        for d in opt_dirs:
+            if d is not None and os.path.exists(d):
+                paths.append(d)
+        paths += os.environ.get('PATH').split(':')
+        bin_path = None
+        # mangle PATH to include /sbin dirs
+        for p in sbin_paths:
+            if p not in paths and os.path.exists(p):
+                paths.append(p)
+        for d in paths:
+            path = os.path.join(d, arg)
+            if os.path.exists(path) and os.access(path, os.X_OK):
+                bin_path = path
+                break
+        if required and bin_path is None:
+            self.fail_json(msg='Failed to find required executable %s' % arg)
+        return bin_path
+
+    def boolean(self, arg):
+        ''' return a bool for the arg '''
+        if arg is None or type(arg) == bool:
+            return arg
+        if type(arg) in types.StringTypes:
+            arg = arg.lower()
+        if arg in BOOLEANS_TRUE:
+            return True
+        elif arg in BOOLEANS_FALSE:
+            return False
+        else:
+            self.fail_json(msg='Boolean %s not in either boolean list' % arg)
+
+    def jsonify(self, data):
+        return json.dumps(data)
+
+    def exit_json(self, **kwargs):
+        ''' return from the module, without error '''
+        print self.jsonify(kwargs)
+        sys.exit(0)
+
+    def fail_json(self, **kwargs):
+        ''' return from the module, with an error message '''
+        assert 'msg' in kwargs, "implementation error -- msg to explain the error is required"
+        kwargs['failed'] = True
+        print self.jsonify(kwargs)
+        sys.exit(1)
+
+    def md5(self, filename):
+        ''' Return MD5 hex digest of local file, or None if file is not present. '''
+        if not os.path.exists(filename):
+            return None
+        if os.path.isdir(filename):
+            self.fail_json(msg="attempted to take md5sum of directory: %s" % filename)
+        digest = _md5()
+        blocksize = 64 * 1024
+        infile = open(filename, 'rb')
+        block = infile.read(blocksize)
+        while block:
+            digest.update(block)
+            block = infile.read(blocksize)
+        infile.close()
+        return digest.hexdigest()
+
+    def backup_local(self, fn):
+        '''make a date-marked backup of the specified file, return True or False on success or failure'''
+        # backups named basename-YYYY-MM-DD@HH:MM~
+        ext = time.strftime("%Y-%m-%d@%H:%M~", time.localtime(time.time()))
+        backupdest = '%s.%s' % (fn, ext)
+
+        try:
+            shutil.copy2(fn, backupdest)
+        except shutil.Error, e:
+            self.fail_json(msg='Could not make backup of %s to %s: %s' % (fn, backupdest, e))
+        return backupdest
+
+
+# == END DYNAMICALLY INSERTED CODE ===
+
+"""

lib/ansible/playbook.py

@@ -1,563 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-import ansible.inventory
-import ansible.runner
-import ansible.constants as C
-from ansible import utils
-from ansible import errors
-import os
-
-# used to transfer variables to Runner
-SETUP_CACHE={ }
-
-#############################################
-
-class PlayBook(object):
-
-    '''
-    runs an ansible playbook, given as a datastructure
-    or YAML filename.  a playbook is a deployment, config
-    management, or automation based set of commands to
-    run in series.
-
-    multiple plays/tasks do not execute simultaneously,
-    but tasks in each pattern do execute in parallel
-    (according to the number of forks requested) among
-    the hosts they address
-    '''
-
-    # *****************************************************
-
-    def __init__(self,
-        playbook         = None,
-        host_list        = C.DEFAULT_HOST_LIST,
-        module_path      = C.DEFAULT_MODULE_PATH,
-        forks            = C.DEFAULT_FORKS,
-        timeout          = C.DEFAULT_TIMEOUT,
-        remote_user      = C.DEFAULT_REMOTE_USER,
-        remote_pass      = C.DEFAULT_REMOTE_PASS,
-        sudo_pass        = C.DEFAULT_SUDO_PASS,
-        remote_port      = C.DEFAULT_REMOTE_PORT,
-        transport        = C.DEFAULT_TRANSPORT,
-        override_hosts   = None,
-        debug            = False,
-        callbacks        = None,
-        runner_callbacks = None,
-        stats            = None,
-        sudo             = False,
-        extra_vars       = None):
-
-        """
-        playbook:         path to a playbook file
-        host_list:        path to a file like /etc/ansible/hosts
-        module_path:      path to ansible modules, like /usr/share/ansible/
-        forks:            desired level of paralellism
-        timeout:          connection timeout
-        remote_user:      run as this user if not specified in a particular play
-        remote_pass:      use this remote password (for all plays) vs using SSH keys
-        sudo_pass:        if sudo==True, and a password is required, this is the sudo password
-        remote_port:      default remote port to use if not specified with the host or play
-        transport:        how to connect to hosts that don't specify a transport (local, paramiko, etc)
-        override_hosts:   skip the inventory file, just talk to these hosts
-        callbacks         output callbacks for the playbook
-        runner_callbacks: more callbacks, this time for the runner API
-        stats:            holds aggregrate data about events occuring to each host
-        sudo:             if not specified per play, requests all plays use sudo mode
-        """
-
-        if playbook is None or callbacks is None or runner_callbacks is None or stats is None:
-            raise Exception('missing required arguments')
-
-        if extra_vars is None:
-            extra_vars = {}
-       
-        self.module_path      = module_path
-        self.forks            = forks
-        self.timeout          = timeout
-        self.remote_user      = remote_user
-        self.remote_pass      = remote_pass
-        self.remote_port      = remote_port
-        self.transport        = transport
-        self.debug            = debug
-        self.callbacks        = callbacks
-        self.runner_callbacks = runner_callbacks
-        self.override_hosts   = override_hosts
-        self.stats            = stats
-        self.sudo             = sudo
-        self.sudo_pass        = sudo_pass
-        self.extra_vars       = extra_vars
-        self.global_vars      = {}
-
-        if override_hosts is not None:
-            if type(override_hosts) != list:
-                raise errors.AnsibleError("override hosts must be a list")
-            self.global_vars.update(ansible.inventory.Inventory(host_list).get_global_vars())
-            self.inventory = ansible.inventory.Inventory(override_hosts)
-
-        else:
-            self.inventory = ansible.inventory.Inventory(host_list)
-            self.global_vars.update(ansible.inventory.Inventory(host_list).get_global_vars())
-
-        self.basedir          = os.path.dirname(playbook)
-        self.playbook         = self._parse_playbook(playbook)
-
-    # *****************************************************
-
-    def _get_vars(self, play, dirname):
-        ''' load the vars section from a play '''
-
-        if play.get('vars') is None:
-            play['vars'] = {}
-        if type(play['vars']) not in [dict, list]:
-            raise errors.AnsibleError("'vars' section must contain only key/value pairs")
-        vars = self.global_vars
-        
-        # translate a list of vars into a dict
-        if type(play['vars']) == list:
-            for item in play['vars']:
-                k, v = item.items()[0]
-                vars[k] = v
-        else:
-            vars.update(play['vars'])
-
-        vars_prompt = play.get('vars_prompt', {})
-        if type(vars_prompt) != dict:
-            raise errors.AnsibleError("'vars_prompt' section must contain only key/value pairs")
-        for vname in vars_prompt:
-            # TODO: make this prompt one line and consider double entry
-            print vars_prompt[vname]
-            vars[vname] = self.callbacks.on_vars_prompt(vname)
-
-        results = self.extra_vars.copy()
-        results.update(vars)
-        return results
-
-    # *****************************************************
-
-    def _include_tasks(self, play, task, dirname, new_tasks):
-        ''' load tasks included from external files. '''
-
-        # include: some.yml a=2 b=3 c=4
-        play_vars = self._get_vars(play, dirname)
-        include_tokens = utils.template(task['include'], play_vars, SETUP_CACHE).split()
-        path = utils.path_dwim(dirname, include_tokens[0])
-        include_vars = {}
-        for i,x in enumerate(include_tokens):
-            if x.find("=") != -1:
-                (k,v) = x.split("=")
-                include_vars[k] = v
-        inject_vars = play_vars.copy()
-        inject_vars.update(include_vars)
-        included = utils.template_from_file(path, inject_vars, SETUP_CACHE)
-        included = utils.parse_yaml(included)
-        for x in included:
-            if len(include_vars):
-                x["vars"] = include_vars
-            new_tasks.append(x)
-
-    # *****************************************************
-
-    def _include_handlers(self, play, handler, dirname, new_handlers):
-        ''' load handlers from external files '''
-
-        path = utils.path_dwim(dirname, handler['include'])
-        inject_vars = self._get_vars(play, dirname)
-        included = utils.template_from_file(path, inject_vars, SETUP_CACHE)
-        included = utils.parse_yaml(included)
-        for x in included:
-            new_handlers.append(x)
-
-    # *****************************************************
-
-    def _parse_playbook(self, playbook):
-        ''' load YAML file, including handling for imported files '''
-        
-        dirname  = os.path.dirname(playbook)
-        playbook = utils.parse_yaml_from_file(playbook)
-
-        for play in playbook:
-            tasks = play.get('tasks',[])
-            handlers = play.get('handlers', [])
-
-            # process tasks in this file as well as imported tasks
-            new_tasks = []
-            for task in tasks:
-                if 'include' in task:
-                    self._include_tasks(play, task, dirname, new_tasks)
-                else:
-                    new_tasks.append(task)
-
-            # now new_tasks contains a list of tasks, but tasks may contain
-            # lists of with_items to loop over.  Do that.
-            # TODO: refactor into subfunction
-            new_tasks2 = []
-            for task in new_tasks:
-                if 'with_items' in task:
-                    for item in task['with_items']:
-                        produced_task = {}
-                        name    = task.get('name', task.get('action', 'unnamed task'))
-                        action  = task.get('action', None)
-                        only_if = task.get('only_if', None)
-                        if action is None:
-                            raise errors.AnsibleError('action is required')
-                        produced_task = task.copy()
-                        produced_task['action'] = utils.template(action, dict(item=item), SETUP_CACHE)
-                        produced_task['name'] = utils.template(name, dict(item=item), SETUP_CACHE)
-                        if only_if:
-                            produced_task['only_if'] = utils.template(only_if, dict(item=item), SETUP_CACHE)
-                        new_tasks2.append(produced_task)
-                else:
-                    new_tasks2.append(task)
-
-            play['tasks'] = new_tasks2
-
-            # process handlers as well as imported handlers
-            new_handlers = [] 
-            for handler in handlers:
-                if 'include' in handler:
-                    self._include_handlers(play, handler, dirname, new_handlers)
-                else:
-                    new_handlers.append(handler)
-            play['handlers'] = new_handlers
-
-        return playbook
-
-    # *****************************************************
-        
-    def run(self):
-        ''' run all patterns in the playbook '''
-
-        # loop through all patterns and run them
-        self.callbacks.on_start()
-        for pattern in self.playbook:
-            self._run_play(pattern)
-
-        # summarize the results
-        results = {}
-        for host in self.stats.processed.keys():
-            results[host] = self.stats.summarize(host)
-        return results
-
-    # *****************************************************
-
-    def _async_poll(self, poller, async_seconds, async_poll_interval):
-        ''' launch an async job, if poll_interval is set, wait for completion '''
-
-        results = poller.wait(async_seconds, async_poll_interval)
-
-        # mark any hosts that are still listed as started as failed
-        # since these likely got killed by async_wrapper
-        for host in poller.hosts_to_poll:
-            reason = { 'failed' : 1, 'rc' : None, 'msg' : 'timed out' }
-            self.runner_callbacks.on_failed(host, reason)
-            results['contacted'][host] = reason
-
-        return results
-
-    # *****************************************************
-
-    def _run_module(self, pattern, module, args, vars, remote_user, 
-        async_seconds, async_poll_interval, only_if, sudo, transport, port):
-        ''' run a particular module step in a playbook '''
-
-        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
-        self.inventory.restrict_to(hosts)
-
-        if port is None:
-            port=self.remote_port
-
-        runner = ansible.runner.Runner(
-            pattern=pattern, inventory=self.inventory, module_name=module,
-            module_args=args, forks=self.forks,
-            remote_pass=self.remote_pass, module_path=self.module_path,
-            timeout=self.timeout, remote_user=remote_user, 
-            remote_port=port, module_vars=vars,
-            setup_cache=SETUP_CACHE, basedir=self.basedir,
-            conditional=only_if, callbacks=self.runner_callbacks, 
-            debug=self.debug, sudo=sudo,
-            transport=transport, sudo_pass=self.sudo_pass, is_playbook=True
-        )
-
-        if async_seconds == 0:
-            results = runner.run()
-        else:
-            results, poller = runner.runAsync(async_seconds)
-            self.stats.compute(results)
-            if async_poll_interval > 0:
-                # if not polling, playbook requested fire and forget
-                # trust the user wanted that and return immediately
-                results = self._async_poll(poller, async_seconds, async_poll_interval)
-
-        self.inventory.lift_restriction()
-        return results
-
-    # *****************************************************
-
-    def _run_task(self, pattern=None, task=None, 
-        remote_user=None, handlers=None, conditional=False, sudo=False, transport=None, port=None):
-        ''' run a single task in the playbook and recursively run any subtasks.  '''
-
-        # load the module name and parameters from the task entry
-        name    = task.get('name', None) 
-        action  = task.get('action', None)
-        if action is None:
-            raise errors.AnsibleError("action is required for each item in tasks: offending task is %s" % name if name else "unknown")
-        if name is None:
-            name = action
-        only_if = task.get('only_if', 'True')
-        async_seconds = int(task.get('async', 0))  # not async by default
-        async_poll_interval = int(task.get('poll', 10))  # default poll = 10 seconds
-
-        tokens = action.split(None, 1)
-        module_name = tokens[0]
-        module_args = ''
-        if len(tokens) > 1:
-            module_args = tokens[1]
-
-        # include task specific vars
-        module_vars = task.get('vars', {})
-        if 'first_available_file' in task:
-            module_vars['first_available_file'] = task.get('first_available_file')
-        
-        # tasks can be direct (run on all nodes matching
-        # the pattern) or conditional, where they ran
-        # as the result of a change handler on a subset
-        # of all of the hosts
-
-        self.callbacks.on_task_start(name, conditional)
-
-        # load up an appropriate ansible runner to
-        # run the task in parallel
-        results = self._run_module(pattern, module_name, 
-            module_args, module_vars, remote_user, async_seconds, 
-            async_poll_interval, only_if, sudo, transport, port)
-
-        # add facts to the global setup cache
-        for host, result in results['contacted'].iteritems():
-            if "ansible_facts" in result:
-                for k,v in result['ansible_facts'].iteritems():
-                    SETUP_CACHE[host][k]=v
-
-        self.stats.compute(results)
-
-        # if no hosts are matched, carry on, unlike /bin/ansible
-        # which would warn you about this
-        if results is None:
-            results = {}
- 
-        # flag which notify handlers need to be run
-        # this will be on a SUBSET of the actual host list.  For instance
-        # a file might need to be written on only half of the nodes so
-        # we would only trigger restarting Apache on half of the nodes
-
-        subtasks = task.get('notify', [])
-        if len(subtasks) > 0:
-            for host, results in results.get('contacted',{}).iteritems():
-                if results.get('changed', False):
-                    for subtask in subtasks:
-                        self._flag_handler(handlers, subtask, host)
-
-    # *****************************************************
-
-    def _flag_handler(self, handlers, match_name, host):
-        ''' 
-        if a task has any notify elements, flag handlers for run
-        at end of execution cycle for hosts that have indicated
-        changes have been made
-        '''
-
-        # for all registered handlers in the ansible playbook
-        # for this particular pattern group
-
-        found = False
-        for x in handlers:
-            name = x.get('name', None)
-            if name is None:
-                raise errors.AnsibleError('handler is missing a name')
-            if match_name == name:
-                found = True
-                self.callbacks.on_notify(host, name)
-                # flag the handler with the list of hosts it needs to be run on, it will be run later
-                if not 'run' in x:
-                    x['run'] = []
-                x['run'].append(host)
-        if not found:
-            raise errors.AnsibleError("change handler (%s) is not defined" % match_name)
-
-    # *****************************************************
-
-    def _do_conditional_imports(self, vars_files):
-        ''' handle the vars_files section, which can contain variables '''
-        
-        # FIXME: save parsed variable results in memory to avoid excessive re-reading/parsing
-        # FIXME: currently parses imports for hosts not in the pattern, that is not wrong, but it's 
-        #        not super optimized yet either, because we wouldn't have hit them, ergo
-        #        it will raise false errors if there is no defaults variable file without any $vars
-        #        in it, which could happen on uncontacted hosts.
- 
-        if type(vars_files) != list:
-            raise errors.AnsibleError("vars_files must be a list")
-        for host in self.inventory.list_hosts():
-            cache_vars = SETUP_CACHE.get(host,{}) 
-            SETUP_CACHE[host] = cache_vars
-            for filename in vars_files:
-                if type(filename) == list:
-                    # loop over all filenames, loading the first one, and failing if # none found
-                    found = False
-                    sequence = []
-                    for real_filename in filename:
-                        filename2 = utils.path_dwim(self.basedir, utils.template(real_filename, cache_vars, SETUP_CACHE))
-                        sequence.append(filename2)
-                        if os.path.exists(filename2):
-                            found = True
-                            data = utils.parse_yaml_from_file(filename2)
-                            SETUP_CACHE[host].update(data)
-                            self.callbacks.on_import_for_host(host, filename2)
-                            break
-                        else:
-                            self.callbacks.on_not_import_for_host(host, filename2)
-                    if not found:
-                        raise errors.AnsibleError(
-                            "%s: FATAL, no files matched for vars_files import sequence: %s" % (host, sequence)
-                        )
-
-                else:
-                    filename2 = utils.path_dwim(self.basedir, utils.template(filename, cache_vars, SETUP_CACHE))
-                    if not os.path.exists(filename2):
-                        raise errors.AnsibleError("no file matched for vars_file import: %s" % filename2)
-                    data = utils.parse_yaml_from_file(filename2)
-                    SETUP_CACHE[host].update(data)
-                    self.callbacks.on_import_for_host(host, filename2)
-
-    # *****************************************************
-
-    def _do_setup_step(self, pattern, vars, user, port, sudo, transport, vars_files=None):
-        ''' push variables down to the systems and get variables+facts back up '''
-
-        # this enables conditional includes like $facter_os.yml and is only done
-        # after the original pass when we have that data.
-        #
-
-        if vars_files is not None:
-            self.callbacks.on_setup_secondary()
-            self._do_conditional_imports(vars_files)
-        else:
-            self.callbacks.on_setup_primary()
-
-        host_list = [ h for h in self.inventory.list_hosts(pattern) 
-                        if not (h in self.stats.failures or h in self.stats.dark) ]
-        self.inventory.restrict_to(host_list)
-
-        # push any variables down to the system
-        setup_results = ansible.runner.Runner(
-            pattern=pattern, module_name='setup',
-            module_args=vars, inventory=self.inventory,
-            forks=self.forks, module_path=self.module_path,
-            timeout=self.timeout, remote_user=user,
-            remote_pass=self.remote_pass, remote_port=port,
-            setup_cache=SETUP_CACHE,
-            callbacks=self.runner_callbacks, sudo=sudo, debug=self.debug,
-            transport=transport, sudo_pass=self.sudo_pass, is_playbook=True
-        ).run()
-        self.stats.compute(setup_results, setup=True)
-
-        self.inventory.lift_restriction()
-
-        # now for each result, load into the setup cache so we can
-        # let runner template out future commands
-        setup_ok = setup_results.get('contacted', {})
-        if vars_files is None:
-            # first pass only or we'll erase good work
-            for (host, result) in setup_ok.iteritems():
-                if 'ansible_facts' in result:
-                    SETUP_CACHE[host] = result['ansible_facts']
-
-    # *****************************************************
-
-    def _run_play(self, pg):
-        ''' run a list of tasks for a given pattern, in order '''
-
-        # get configuration information about the pattern
-        pattern = pg.get('hosts',None)
-        name = pg.get('name', pattern)
-        if isinstance(pattern, list):
-            pattern = ';'.join(pattern)
-        if self.override_hosts:
-            pattern = 'all'
-        if pattern is None:
-            raise errors.AnsibleError('hosts declaration is required')
-
-        vars       = self._get_vars(pg, self.basedir)
-        vars_files = pg.get('vars_files', {})
-        tasks      = pg.get('tasks', [])
-        handlers   = pg.get('handlers', [])
-        user       = pg.get('user', self.remote_user)
-        port       = pg.get('port', self.remote_port)
-        sudo       = pg.get('sudo', self.sudo)
-        transport  = pg.get('connection', self.transport)
-
-        self.callbacks.on_play_start(name)
-
-        # push any variables down to the system # and get facts/ohai/other data back up
-        self._do_setup_step(pattern, vars, user, port, sudo, transport, None)
-         
-        # now with that data, handle contentional variable file imports!
-        if len(vars_files) > 0:
-            self._do_setup_step(pattern, vars, user, port, sudo, transport, vars_files)
-
-        # run all the top level tasks, these get run on every node
-        for task in tasks:
-            self._run_task(
-                pattern=pattern,
-                task=task, 
-                handlers=handlers,
-                remote_user=user,
-                sudo=sudo,
-                transport=transport,
-                port=port
-            )
-
-        # handlers only run on certain nodes, they are flagged by _flag_handlers
-        # above.  They only run on nodes when things mark them as changed, and
-        # handlers only get run once.  For instance, the system is designed
-        # such that multiple config files if changed can ask for an Apache restart
-        # but Apache will only be restarted once (at the end).
-
-        for task in handlers:
-            triggered_by = task.get('run', None)
-            if type(triggered_by) == list:
-                self.inventory.restrict_to(triggered_by)
-                self._run_task(
-                   pattern=pattern, 
-                   task=task,
-                   handlers=[],
-                   conditional=True,
-                   remote_user=user,
-                   sudo=sudo,
-                   transport=transport,
-                   port=port
-                )
-                self.inventory.lift_restriction()
-
-        # end of execution for this particular pattern.  Multiple patterns
-        # can be in a single playbook file
-
- 
-

lib/ansible/playbook/__init__.py

@@ -0,0 +1,366 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.inventory
+import ansible.runner
+import ansible.constants as C
+from ansible import utils
+from ansible import errors
+import os
+import collections
+from play import Play
+
+SETUP_CACHE = collections.defaultdict(dict)
+
+class PlayBook(object):
+    '''
+    runs an ansible playbook, given as a datastructure or YAML filename.
+    A playbook is a deployment, config management, or automation based
+    set of commands to run in series.
+
+    multiple plays/tasks do not execute simultaneously, but tasks in each
+    pattern do execute in parallel (according to the number of forks
+    requested) among the hosts they address
+    '''
+
+    # *****************************************************
+
+    def __init__(self,
+        playbook         = None,
+        host_list        = C.DEFAULT_HOST_LIST,
+        module_path      = C.DEFAULT_MODULE_PATH,
+        forks            = C.DEFAULT_FORKS,
+        timeout          = C.DEFAULT_TIMEOUT,
+        remote_user      = C.DEFAULT_REMOTE_USER,
+        remote_pass      = C.DEFAULT_REMOTE_PASS,
+        sudo_pass        = C.DEFAULT_SUDO_PASS,
+        remote_port      = C.DEFAULT_REMOTE_PORT,
+        transport        = C.DEFAULT_TRANSPORT,
+        private_key_file = C.DEFAULT_PRIVATE_KEY_FILE,
+        callbacks        = None,
+        runner_callbacks = None,
+        stats            = None,
+        sudo             = False,
+        sudo_user        = C.DEFAULT_SUDO_USER,
+        extra_vars       = None,
+        only_tags        = None,
+        subset           = C.DEFAULT_SUBSET):
+
+        """
+        playbook:         path to a playbook file
+        host_list:        path to a file like /etc/ansible/hosts
+        module_path:      path to ansible modules, like /usr/share/ansible/
+        forks:            desired level of paralellism
+        timeout:          connection timeout
+        remote_user:      run as this user if not specified in a particular play
+        remote_pass:      use this remote password (for all plays) vs using SSH keys
+        sudo_pass:        if sudo==True, and a password is required, this is the sudo password
+        remote_port:      default remote port to use if not specified with the host or play
+        transport:        how to connect to hosts that don't specify a transport (local, paramiko, etc)
+        callbacks         output callbacks for the playbook
+        runner_callbacks: more callbacks, this time for the runner API
+        stats:            holds aggregrate data about events occuring to each host
+        sudo:             if not specified per play, requests all plays use sudo mode
+        """
+
+        self.SETUP_CACHE = SETUP_CACHE
+
+        if playbook is None or callbacks is None or runner_callbacks is None or stats is None:
+            raise Exception('missing required arguments')
+
+        if extra_vars is None:
+            extra_vars = {}
+        if only_tags is None:
+            only_tags = [ 'all' ]
+
+        self.module_path      = module_path
+        self.forks            = forks
+        self.timeout          = timeout
+        self.remote_user      = remote_user
+        self.remote_pass      = remote_pass
+        self.remote_port      = remote_port
+        self.transport        = transport
+        self.callbacks        = callbacks
+        self.runner_callbacks = runner_callbacks
+        self.stats            = stats
+        self.sudo             = sudo
+        self.sudo_pass        = sudo_pass
+        self.sudo_user        = sudo_user
+        self.extra_vars       = extra_vars
+        self.global_vars      = {}
+        self.private_key_file = private_key_file
+        self.only_tags        = only_tags
+
+        self.inventory        = ansible.inventory.Inventory(host_list)
+        self.inventory.subset(subset)
+
+        if not self.inventory._is_script:
+            self.global_vars.update(self.inventory.get_group_variables('all'))
+
+        self.basedir     = os.path.dirname(playbook)
+        self.playbook    = self._load_playbook_from_file(playbook)
+        self.module_path = self.module_path + os.pathsep + os.path.join(self.basedir, "library")
+
+    # *****************************************************
+
+    def _load_playbook_from_file(self, path):
+        '''
+        run top level error checking on playbooks and allow them to include other playbooks.
+        '''
+
+        playbook_data  = utils.parse_yaml_from_file(path)
+        accumulated_plays = []
+
+        if type(playbook_data) != list:
+            raise errors.AnsibleError("parse error: playbooks must be formatted as a YAML list")
+
+        for play in playbook_data:
+            if type(play) != dict:
+                raise errors.AnsibleError("parse error: each play in a playbook must a YAML dictionary (hash), recieved: %s" % play)
+            if 'include' in play:
+                if len(play.keys()) == 1:
+                    included_path = utils.path_dwim(self.basedir, play['include'])
+                    accumulated_plays.extend(self._load_playbook_from_file(included_path))
+                else:
+                    raise errors.AnsibleError("parse error: top level includes cannot be used with other directives: %s" % play)
+            else:
+                accumulated_plays.append(play)
+
+        return accumulated_plays
+
+    # *****************************************************
+
+    def run(self):
+        ''' run all patterns in the playbook '''
+        plays = []
+        matched_tags_all = set()
+        unmatched_tags_all = set()
+
+        # loop through all patterns and run them
+        self.callbacks.on_start()
+        for play_ds in self.playbook:
+            play = Play(self,play_ds)
+            matched_tags, unmatched_tags = play.compare_tags(self.only_tags)
+            matched_tags_all = matched_tags_all | matched_tags
+            unmatched_tags_all = unmatched_tags_all | unmatched_tags
+
+            # if we have matched_tags, the play must be run.
+            # if the play contains no tasks, assume we just want to gather facts
+            if (len(matched_tags) > 0 or len(play.tasks()) == 0):
+                plays.append(play)
+
+        # if the playbook is invoked with --tags that don't exist at all in the playbooks
+        # then we need to raise an error so that the user can correct the arguments.
+        unknown_tags = set(self.only_tags) - (matched_tags_all | unmatched_tags_all)
+        unknown_tags.discard('all')       
+ 
+        if len(unknown_tags) > 0:
+            unmatched_tags_all.discard('all')
+            msg = 'tag(s) not found in playbook: %s.  possible values: %s'
+            unknown = ','.join(sorted(unknown_tags))
+            unmatched = ','.join(sorted(unmatched_tags_all))
+            raise errors.AnsibleError(msg % (unknown, unmatched))
+
+        for play in plays:
+            self._run_play(play)
+
+        # summarize the results
+        results = {}
+        for host in self.stats.processed.keys():
+            results[host] = self.stats.summarize(host)
+        return results
+
+    # *****************************************************
+
+    def _async_poll(self, poller, async_seconds, async_poll_interval):
+        ''' launch an async job, if poll_interval is set, wait for completion '''
+
+        results = poller.wait(async_seconds, async_poll_interval)
+
+        # mark any hosts that are still listed as started as failed
+        # since these likely got killed by async_wrapper
+        for host in poller.hosts_to_poll:
+            reason = { 'failed' : 1, 'rc' : None, 'msg' : 'timed out' }
+            self.runner_callbacks.on_failed(host, reason)
+            results['contacted'][host] = reason
+
+        return results
+
+    # *****************************************************
+
+    def _run_task_internal(self, task):
+        ''' run a particular module step in a playbook '''
+
+        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
+        self.inventory.restrict_to(hosts)
+
+        runner = ansible.runner.Runner(
+            pattern=task.play.hosts, inventory=self.inventory, module_name=task.module_name,
+            module_args=task.module_args, forks=self.forks,
+            remote_pass=self.remote_pass, module_path=self.module_path,
+            timeout=self.timeout, remote_user=task.play.remote_user,
+            remote_port=task.play.remote_port, module_vars=task.module_vars,
+            private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, basedir=self.basedir,
+            conditional=task.only_if, callbacks=self.runner_callbacks,
+            sudo=task.play.sudo, sudo_user=task.play.sudo_user,
+            transport=task.play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        )
+
+        if task.async_seconds == 0:
+            results = runner.run()
+        else:
+            results, poller = runner.run_async(task.async_seconds)
+            self.stats.compute(results)
+            if task.async_poll_interval > 0:
+                # if not polling, playbook requested fire and forget, so don't poll
+                results = self._async_poll(poller, task.async_seconds, task.async_poll_interval)
+
+        self.inventory.lift_restriction()
+        return results
+
+    # *****************************************************
+
+    def _run_task(self, play, task, is_handler):
+        ''' run a single task in the playbook and recursively run any subtasks.  '''
+
+        self.callbacks.on_task_start(task.name, is_handler)
+
+        # load up an appropriate ansible runner to run the task in parallel
+        results = self._run_task_internal(task)
+        # if no hosts are matched, carry on
+        if results is None:
+            results = {}
+
+        self.stats.compute(results, ignore_errors=task.ignore_errors)
+
+        # add facts to the global setup cache
+        for host, result in results['contacted'].iteritems():
+            facts = result.get('ansible_facts', {})
+            self.SETUP_CACHE[host].update(facts)
+            if task.register:
+                self.SETUP_CACHE[host][task.register] = result
+
+        # flag which notify handlers need to be run
+        if len(task.notify) > 0:
+            for host, results in results.get('contacted',{}).iteritems():
+                if results.get('changed', False):
+                    for handler_name in task.notify:
+                        self._flag_handler(play.handlers(), utils.template(handler_name, task.module_vars), host)
+
+    # *****************************************************
+
+    def _flag_handler(self, handlers, handler_name, host):
+        '''
+        if a task has any notify elements, flag handlers for run
+        at end of execution cycle for hosts that have indicated
+        changes have been made
+        '''
+
+        found = False
+        for x in handlers:
+            if handler_name == x.name:
+                found = True
+                self.callbacks.on_notify(host, x.name)
+                x.notified_by.append(host)
+        if not found:
+            raise errors.AnsibleError("change handler (%s) is not defined" % handler_name)
+
+    # *****************************************************
+
+    def _do_setup_step(self, play):
+        ''' get facts from the remote system '''
+
+        host_list = [ h for h in self.inventory.list_hosts(play.hosts)
+            if not (h in self.stats.failures or h in self.stats.dark) ]
+
+        if not play.gather_facts:
+            return {}
+
+        self.callbacks.on_setup()
+        self.inventory.restrict_to(host_list)
+
+        # push any variables down to the system
+        setup_results = ansible.runner.Runner(
+            pattern=play.hosts, module_name='setup', module_args={}, inventory=self.inventory,
+            forks=self.forks, module_path=self.module_path, timeout=self.timeout, remote_user=play.remote_user,
+            remote_pass=self.remote_pass, remote_port=play.remote_port, private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, callbacks=self.runner_callbacks, sudo=play.sudo, sudo_user=play.sudo_user,
+            transport=play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        ).run()
+        self.stats.compute(setup_results, setup=True)
+
+        self.inventory.lift_restriction()
+
+        # now for each result, load into the setup cache so we can
+        # let runner template out future commands
+        setup_ok = setup_results.get('contacted', {})
+        for (host, result) in setup_ok.iteritems():
+            self.SETUP_CACHE[host] = result.get('ansible_facts', {})
+        return setup_results
+
+    # *****************************************************
+
+    def _run_play(self, play):
+        ''' run a list of tasks for a given pattern, in order '''
+
+        self.callbacks.on_play_start(play.name)
+
+        # get facts from system
+        self._do_setup_step(play)
+
+        # now with that data, handle contentional variable file imports!
+
+        all_hosts = self.inventory.list_hosts(play.hosts)
+        play.update_vars_files(all_hosts)
+
+        serialized_batch = []
+        if play.serial <= 0:
+            serialized_batch = [all_hosts]
+        else:
+            # do N forks all the way through before moving to next
+            while len(all_hosts) > 0:
+                play_hosts = []
+                for x in range(play.serial):
+                    if len(all_hosts) > 0:
+                        play_hosts.append(all_hosts.pop())
+                serialized_batch.append(play_hosts)
+
+        for on_hosts in serialized_batch:
+
+            self.inventory.also_restrict_to(on_hosts)
+
+            for task in play.tasks():
+                # only run the task if the requested tags match
+                should_run = False
+                for x in self.only_tags:
+                    for y in task.tags:
+                        if (x==y):
+                            should_run = True
+                            break
+                if should_run:
+                    self._run_task(play, task, False)
+
+            # run notify actions
+            for handler in play.handlers():
+                if len(handler.notified_by) > 0:
+                    self.inventory.restrict_to(handler.notified_by)
+                    self._run_task(play, handler, True)
+                    self.inventory.lift_restriction()
+
+            self.inventory.lift_also_restriction()
+

lib/ansible/playbook/play.py

@@ -0,0 +1,313 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+from ansible import utils
+from ansible import errors
+from ansible.playbook.task import Task
+import shlex
+import os
+
+class Play(object):
+
+    __slots__ = [
+       'hosts', 'name', 'vars', 'vars_prompt', 'vars_files',
+       'handlers', 'remote_user', 'remote_port',
+       'sudo', 'sudo_user', 'transport', 'playbook',
+       'tags', 'gather_facts', 'serial', '_ds', '_handlers', '_tasks'
+    ]
+
+    # to catch typos and so forth -- these are userland names
+    # and don't line up 1:1 with how they are stored
+    VALID_KEYS = [
+       'hosts', 'name', 'vars', 'vars_prompt', 'vars_files',
+       'tasks', 'handlers', 'user', 'port', 'include',
+       'sudo', 'sudo_user', 'connection', 'tags', 'gather_facts', 'serial'
+    ]
+
+    # *************************************************
+
+    def __init__(self, playbook, ds):
+        ''' constructor loads from a play datastructure '''
+
+        for x in ds.keys():
+            if not x in Play.VALID_KEYS:
+                raise errors.AnsibleError("%s is not a legal parameter in an Ansible Playbook" % x)
+
+        # TODO: more error handling
+
+        hosts = ds.get('hosts')
+        if hosts is None:
+            raise errors.AnsibleError('hosts declaration is required')
+        elif isinstance(hosts, list):
+            hosts = ';'.join(hosts)
+        hosts = utils.template(hosts, playbook.extra_vars)
+
+        self._ds          = ds
+        self.playbook     = playbook
+        self.hosts        = hosts
+        self.name         = ds.get('name', self.hosts)
+        self.vars         = ds.get('vars', {})
+        self.vars_files   = ds.get('vars_files', [])
+        self.vars_prompt  = ds.get('vars_prompt', {})
+        self.vars         = self._get_vars(self.playbook.basedir)
+        self._tasks       = ds.get('tasks', [])
+        self._handlers    = ds.get('handlers', [])
+        self.remote_user  = utils.template(ds.get('user', self.playbook.remote_user), playbook.extra_vars)
+        self.remote_port  = ds.get('port', self.playbook.remote_port)
+        self.sudo         = ds.get('sudo', self.playbook.sudo)
+        self.sudo_user    = ds.get('sudo_user', self.playbook.sudo_user)
+        self.transport    = ds.get('connection', self.playbook.transport)
+        self.tags         = ds.get('tags', None)
+        self.gather_facts = ds.get('gather_facts', True)
+        self.serial       = ds.get('serial', 0)
+
+        self._update_vars_files_for_host(None)
+
+        self._tasks      = self._load_tasks(self._ds, 'tasks')
+        self._handlers   = self._load_tasks(self._ds, 'handlers')
+
+        if self.tags is None:
+            self.tags = []
+        elif type(self.tags) in [ str, unicode ]:
+            self.tags = [ self.tags ]
+        elif type(self.tags) != list:
+            self.tags = []
+
+        if self.sudo_user != 'root':
+            self.sudo = True
+
+    # *************************************************
+
+    def _load_tasks(self, ds, keyname):
+        ''' handle task and handler include statements '''
+
+        tasks = ds.get(keyname, [])
+        results = []
+        for x in tasks:
+            task_vars = self.vars.copy()
+            if 'include' in x:
+                tokens = shlex.split(x['include'])
+                for t in tokens[1:]:
+                    (k,v) = t.split("=", 1)
+                    task_vars[k] = utils.template(v, task_vars)
+                include_file = utils.template(tokens[0], task_vars)
+                data = utils.parse_yaml_from_file(utils.path_dwim(self.playbook.basedir, include_file))
+            elif type(x) == dict:
+                data = [x]
+            else:
+                raise Exception("unexpected task type")
+
+            for y in data:
+                mv = task_vars.copy()
+                results.append(Task(self,y,module_vars=mv))
+
+        for x in results:
+            if self.tags is not None:
+                x.tags.extend(self.tags)
+
+        return results
+
+    # *************************************************
+
+    def tasks(self):
+        ''' return task objects for this play '''
+        return self._tasks
+
+    def handlers(self):
+        ''' return handler objects for this play '''
+        return self._handlers
+
+    # *************************************************
+
+    def _get_vars(self, dirname):
+        ''' load the vars section from a play, accounting for all sorts of variable features
+        including loading from yaml files, prompting, and conditional includes of the first
+        file found in a list. '''
+
+        if self.vars is None:
+            self.vars = {}
+
+        if type(self.vars) not in [dict, list]:
+            raise errors.AnsibleError("'vars' section must contain only key/value pairs")
+
+        vars = self.playbook.global_vars
+
+        # translate a list of vars into a dict
+        if type(self.vars) == list:
+            for item in self.vars:
+                if getattr(item, 'items', None) is None:
+                    raise errors.AnsibleError("expecting a key-value pair in 'vars' section")
+                k, v = item.items()[0]
+                vars[k] = v
+        else:
+            vars.update(self.vars)
+
+        if type(self.vars_prompt) == list:
+            for var in self.vars_prompt:
+                if not 'name' in var:
+                    raise errors.AnsibleError("'vars_prompt' item is missing 'name:'")
+
+                vname = var['name']
+                prompt = "%s: " % var.get("prompt", vname)
+                private = var.get("private", True)
+
+                confirm = var.get("confirm", False)
+                encrypt = var.get("encrypt", None)
+                salt_size = var.get("salt_size", None)
+                salt = var.get("salt", None)
+
+                vars[vname] = self.playbook.callbacks.on_vars_prompt(vname, private, prompt,encrypt, confirm, salt_size, salt)
+
+        elif type(self.vars_prompt) == dict:
+            for (vname, prompt) in self.vars_prompt.iteritems():
+                prompt_msg = "%s: " % prompt
+                vars[vname] = self.playbook.callbacks.on_vars_prompt(varname=vname, private=False, prompt=prompt_msg)
+
+        else:
+            raise errors.AnsibleError("'vars_prompt' section is malformed, see docs")
+
+        results = self.playbook.extra_vars.copy()
+        results.update(vars)
+        return results
+
+    # *************************************************
+
+    def update_vars_files(self, hosts):
+        ''' calculate vars_files, which requires that setup runs first so ansible facts can be mixed in '''
+
+        # now loop through all the hosts...
+        for h in hosts:
+            self._update_vars_files_for_host(h)
+
+    # *************************************************
+
+    def compare_tags(self, tags):
+        ''' given a list of tags that the user has specified, return two lists:
+        matched_tags:   tags were found within the current play and match those given
+                        by the user
+        unmatched_tags: tags that were found within the current play but do not match
+                        any provided by the user '''
+
+        # gather all the tags in all the tasks into one list
+        all_tags = []
+        for task in self._tasks:
+            all_tags.extend(task.tags)
+
+        # compare the lists of tags using sets and return the matched and unmatched
+        all_tags_set = set(all_tags)
+        tags_set = set(tags)
+        matched_tags = all_tags_set & tags_set
+        unmatched_tags = all_tags_set - tags_set
+
+        return matched_tags, unmatched_tags
+
+    # *************************************************
+
+    def _has_vars_in(self, msg):
+        return ((msg.find("$") != -1) or (msg.find("{{") != -1))
+
+    # *************************************************
+
+    def _update_vars_files_for_host(self, host):
+
+        if type(self.vars_files) != list:
+            self.vars_files = [ self.vars_files ]
+
+        if (host is not None):
+            self.playbook.SETUP_CACHE[host].update(self.vars)
+
+            inventory = self.playbook.inventory
+            hostrec = inventory.get_host(host)
+            groupz = sorted(inventory.groups_for_host(host), key=lambda g: g.depth)
+            groups = [ g.name for g in groupz ]
+            basedir = inventory.basedir()
+            if basedir is not None:
+                for x in groups:
+                    path = os.path.join(basedir, "group_vars/%s" % x)
+                    if os.path.exists(path):
+                        data = utils.parse_yaml_from_file(path)
+                        if type(data) != dict:
+                            raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
+                        self.playbook.SETUP_CACHE[host].update(data)
+                path = os.path.join(basedir, "host_vars/%s" % hostrec.name)
+                if os.path.exists(path):
+                    data = utils.parse_yaml_from_file(path)
+                    if type(data) != dict:
+                        raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
+                    self.playbook.SETUP_CACHE[host].update(data)
+
+        for filename in self.vars_files:
+
+            if type(filename) == list:
+
+                # loop over all filenames, loading the first one, and failing if # none found
+                found = False
+                sequence = []
+                for real_filename in filename:
+                    filename2 = utils.template(real_filename, self.vars)
+                    filename3 = filename2
+                    if host is not None:
+                        filename3 = utils.template(filename2, self.playbook.SETUP_CACHE[host])
+                    filename4 = utils.path_dwim(self.playbook.basedir, filename3)
+                    sequence.append(filename4)
+                    if os.path.exists(filename4):
+                        found = True
+                        data = utils.parse_yaml_from_file(filename4)
+                        if type(data) != dict:
+                            raise errors.AnsibleError("%s must be stored as a dictionary/hash" % filename4)
+                        if host is not None:
+                            if self._has_vars_in(filename2) and not self._has_vars_in(filename3):
+                                # this filename has variables in it that were fact specific
+                                # so it needs to be loaded into the per host SETUP_CACHE
+                                self.playbook.SETUP_CACHE[host].update(data)
+                                self.playbook.callbacks.on_import_for_host(host, filename4)
+                        elif not self._has_vars_in(filename4):
+                            # found a non-host specific variable, load into vars and NOT
+                            # the setup cache
+                            self.vars.update(data)
+                    elif host is not None:
+                        self.playbook.callbacks.on_not_import_for_host(host, filename4)
+                    if found:
+                        break
+                if not found:
+                    raise errors.AnsibleError(
+                        "%s: FATAL, no files matched for vars_files import sequence: %s" % (host, sequence)
+                    )
+
+            else:
+                # just one filename supplied, load it!
+
+                filename2 = utils.template(filename, self.vars)
+                filename3 = filename2
+                if host is not None:
+                    filename3 = utils.template(filename2, self.playbook.SETUP_CACHE[host])
+                filename4 = utils.path_dwim(self.playbook.basedir, filename3)
+                if self._has_vars_in(filename4):
+                    return
+                new_vars = utils.parse_yaml_from_file(filename4)
+                if new_vars:
+                    if type(new_vars) != dict:
+                        raise errors.AnsibleError("%s must be stored as dictonary/hash: %s" % filename4)
+                    if host is not None and self._has_vars_in(filename2) and not self._has_vars_in(filename3):
+                        # running a host specific pass and has host specific variables
+                        # load into setup cache
+                        self.playbook.SETUP_CACHE[host].update(new_vars)
+                    elif host is None:
+                        # running a non-host specific pass and we can update the global vars instead
+                        self.vars.update(new_vars)

lib/ansible/playbook/task.py

@@ -0,0 +1,134 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+from ansible import errors
+from ansible import utils
+
+
+class Task(object):
+
+    __slots__ = [
+        'name', 'action', 'only_if', 'async_seconds', 'async_poll_interval',
+        'notify', 'module_name', 'module_args', 'module_vars',
+        'play', 'notified_by', 'tags', 'register', 'with_items', 
+        'delegate_to', 'first_available_file', 'ignore_errors',
+        'local_action'
+    ]
+
+    # to prevent typos and such
+    VALID_KEYS = [
+         'name', 'action', 'only_if', 'async', 'poll', 'notify', 'with_items', 
+         'first_available_file', 'include', 'tags', 'register', 'ignore_errors',
+         'delegate_to', 'local_action'
+    ]
+
+    def __init__(self, play, ds, module_vars=None):
+        ''' constructor loads from a task or handler datastructure '''
+
+        for x in ds.keys():
+            if not x in Task.VALID_KEYS:
+                raise errors.AnsibleError("%s is not a legal parameter in an Ansible task or handler" % x)
+
+        self.module_vars = module_vars
+        self.play        = play
+
+        # load various attributes
+        self.name         = ds.get('name', None)
+        self.tags         = [ 'all' ]
+        self.register     = ds.get('register', None)
+
+        # Both are defined
+        if ('action' in ds) and ('local_action' in ds):
+            raise errors.AnsibleError("the 'action' and 'local_action' attributes can not be used together")
+        # Both are NOT defined
+        elif (not 'action' in ds) and (not 'local_action' in ds):
+            raise errors.AnsibleError("task missing an 'action' attribute")
+        # Only one of them is defined
+        elif 'local_action' in ds:
+            self.action      = ds.get('local_action', '')
+            self.delegate_to = '127.0.0.1'
+        else:
+            self.action      = ds.get('action', '')
+            self.delegate_to = ds.get('delegate_to', None)
+
+        # notified by is used by Playbook code to flag which hosts
+        # need to run a notifier
+        self.notified_by = []
+
+        # if no name is specified, use the action line as the name
+        if self.name is None:
+            self.name = self.action
+
+        # load various attributes
+        self.only_if = ds.get('only_if', 'True')
+        self.async_seconds = int(ds.get('async', 0))  # not async by default
+        self.async_poll_interval = int(ds.get('poll', 10))  # default poll = 10 seconds
+        self.notify = ds.get('notify', [])
+        self.first_available_file = ds.get('first_available_file', None)
+        self.with_items = ds.get('with_items', None)
+
+        self.ignore_errors = ds.get('ignore_errors', False)
+
+        # notify can be a string or a list, store as a list
+        if isinstance(self.notify, basestring):
+            self.notify = [ self.notify ]
+
+        # split the action line into a module name + arguments
+        tokens = self.action.split(None, 1)
+        if len(tokens) < 1:
+            raise errors.AnsibleError("invalid/missing action in task. name: %s" % self.name)
+        self.module_name = tokens[0]
+        self.module_args = ''
+        if len(tokens) > 1:
+            self.module_args = tokens[1]
+
+        import_tags = self.module_vars.get('tags',[])
+        if type(import_tags) in [str,unicode]:
+            # allow the user to list comma delimited tags
+            import_tags = import_tags.split(",")
+
+        self.name = utils.template(self.name, self.module_vars)
+        self.action = utils.template(self.action, self.module_vars)
+
+        # handle mutually incompatible options
+        if self.with_items is not None and self.first_available_file is not None:
+            raise errors.AnsibleError("with_items and first_available_file are mutually incompatible in a single task")
+
+        # make first_available_file accessable to Runner code
+        if self.first_available_file:
+            self.module_vars['first_available_file'] = self.first_available_file
+
+        # process with_items so it can be used by Runner code
+        if self.with_items is None:
+            self.with_items = [ ]
+        self.module_vars['items'] = self.with_items
+
+        # allow runner to see delegate_to option
+        self.module_vars['delegate_to'] = self.delegate_to
+
+        # make ignore_errors accessable to Runner code
+        self.module_vars['ignore_errors'] = self.ignore_errors
+
+        # tags allow certain parts of a playbook to be run without running the whole playbook
+        apply_tags = ds.get('tags', None)
+        if apply_tags is not None:
+            if type(apply_tags) in [ str, unicode ]:
+                self.tags.append(apply_tags)
+            elif type(apply_tags) == list:
+                self.tags.extend(apply_tags)
+        self.tags.extend(import_tags)
+

lib/ansible/runner.py

@@ -1,848 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-################################################
-
-import multiprocessing
-import signal
-import os
-import pwd
-import Queue
-import random
-import traceback
-import tempfile
-import time
-import base64
-import getpass
-
-import ansible.constants as C 
-import ansible.connection
-import ansible.inventory
-from ansible import utils
-from ansible import errors
-from ansible import callbacks as ans_callbacks
-    
-HAS_ATFORK=True
-try:
-    from Crypto.Random import atfork
-except ImportError:
-    HAS_ATFORK=False
-
-################################################
-
-def _executor_hook(job_queue, result_queue):
-    ''' callback used by multiprocessing pool '''
-
-    # attempt workaround of https://github.com/newsapps/beeswithmachineguns/issues/17
-    # does not occur for everyone, some claim still occurs on newer paramiko
-    # this function not present in CentOS 6
-    if HAS_ATFORK:
-        atfork()
-
-    signal.signal(signal.SIGINT, signal.SIG_IGN)
-    while not job_queue.empty():
-        try:
-            job = job_queue.get(block=False)
-            runner, host = job
-            result_queue.put(runner._executor(host))
-        except Queue.Empty:
-            pass
-        except:
-            traceback.print_exc()
- 
-################################################
-
-class Runner(object):
-
-    def __init__(self, 
-        host_list=C.DEFAULT_HOST_LIST, module_path=C.DEFAULT_MODULE_PATH,
-        module_name=C.DEFAULT_MODULE_NAME, module_args=C.DEFAULT_MODULE_ARGS, 
-        forks=C.DEFAULT_FORKS, timeout=C.DEFAULT_TIMEOUT, 
-        pattern=C.DEFAULT_PATTERN, remote_user=C.DEFAULT_REMOTE_USER, 
-        remote_pass=C.DEFAULT_REMOTE_PASS, remote_port=C.DEFAULT_REMOTE_PORT, 
-        sudo_pass=C.DEFAULT_SUDO_PASS, background=0, basedir=None, 
-        setup_cache=None, transport=C.DEFAULT_TRANSPORT, conditional='True', 
-        callbacks=None, debug=False, sudo=False, module_vars=None, 
-        is_playbook=False, inventory=None):
-
-        """
-        host_list    : path to a host list file, like /etc/ansible/hosts
-        module_path  : path to modules, like /usr/share/ansible
-        module_name  : which module to run (string)
-        module_args  : args to pass to the module (string)
-        forks        : desired level of paralellism (hosts to run on at a time)
-        timeout      : connection timeout, such as a SSH timeout, in seconds
-        pattern      : pattern or groups to select from in inventory
-        remote_user  : connect as this remote username
-        remote_pass  : supply this password (if not using keys)
-        remote_port  : use this default remote port (if not set by the inventory system)
-        sudo_pass    : sudo password if using sudo and sudo requires a password
-        background   : run asynchronously with a cap of this many # of seconds (if not 0)
-        basedir      : paths used by modules if not absolute are relative to here
-        setup_cache  : this is a internalism that is going away
-        transport    : transport mode (paramiko, local)
-        conditional  : only execute if this string, evaluated, is True
-        callbacks    : output callback class
-        sudo         : log in as remote user and immediately sudo to root
-        module_vars  : provides additional variables to a template.  FIXME: just use module_args, remove
-        is_playbook  : indicates Runner is being used by a playbook.  affects behavior in various ways.
-        inventory    : inventory object, if host_list is not provided
-        """
-
-        if setup_cache is None:
-            setup_cache = {}
-        if basedir is None: 
-            basedir = os.getcwd()
-
-        if callbacks is None:
-            callbacks = ans_callbacks.DefaultRunnerCallbacks()
-        self.callbacks = callbacks
-
-        self.generated_jid = str(random.randint(0, 999999999999))
-
-        self.transport = transport
-        self.connector = ansible.connection.Connection(self, self.transport)
-
-        if inventory is None:
-            self.inventory = ansible.inventory.Inventory(host_list)
-        else:
-            self.inventory = inventory
-
-        if module_vars is None:
-            module_vars = {}
-
-        self.setup_cache = setup_cache
-        self.conditional = conditional
-        self.module_path = module_path
-        self.module_name = module_name
-        self.forks       = int(forks)
-        self.pattern     = pattern
-        self.module_args = module_args
-        self.module_vars = module_vars
-        self.timeout     = timeout
-        self.debug       = debug
-        self.remote_user = remote_user
-        self.remote_pass = remote_pass
-        self.remote_port = remote_port
-        self.background  = background
-        self.basedir     = basedir
-        self.sudo        = sudo
-        self.sudo_pass   = sudo_pass
-        self.is_playbook = is_playbook
-
-        euid = pwd.getpwuid(os.geteuid())[0]
-        if self.transport == 'local' and self.remote_user != euid:
-            raise Exception("User mismatch: expected %s, but is %s" % (self.remote_user, euid))
-        if type(self.module_args) not in [str, unicode, dict]:
-            raise Exception("module_args must be a string or dict: %s" % self.module_args)
-
-        self._tmp_paths  = {}
-        random.seed()
-
-    # *****************************************************
-
-    @classmethod
-    def parse_hosts(cls, host_list, override_hosts=None):
-        ''' parse the host inventory file, returns (hosts, groups) '''
-
-        if override_hosts is None:
-            inventory = ansible.inventory.Inventory(host_list)
-        else:
-            inventory = ansible.inventory.Inventory(override_hosts)
-
-        return inventory.host_list, inventory.groups
-
-    # *****************************************************
-
-    def _return_from_module(self, conn, host, result, err, executed=None):
-        ''' helper function to handle JSON parsing of results '''
-
-        try:
-            result = utils.parse_json(result) 
-            if executed is not None:
-                result['invocation'] = executed
-                if 'stderr' in result:
-                    err="%s%s"%(err,result['stderr'])
-            return [host, True, result, err]
-        except Exception, e:
-            return [host, False, "%s/%s/%s" % (str(e), result, executed), err]
-
-    # *****************************************************
-
-    def _delete_remote_files(self, conn, files):
-        ''' deletes one or more remote files '''
-
-        if type(files) == str:
-            files = [ files ]
-        for filename in files:
-            if filename.find('/tmp/') == -1:
-                raise Exception("not going to happen")
-            self._exec_command(conn, "rm -rf %s" % filename, None)
-
-    # *****************************************************
-
-    def _transfer_module(self, conn, tmp, module):
-        ''' transfers a module file to the remote side to execute it, but does not execute it yet '''
-
-        outpath = self._copy_module(conn, tmp, module)
-        self._exec_command(conn, "chmod +x %s" % outpath, tmp)
-        return outpath
-
-    # *****************************************************
-
-    def _transfer_str(self, conn, tmp, name, data):
-        ''' transfer string to remote file '''
-
-        if type(data) == dict:
-            data = utils.smjson(data)
-
-        afd, afile = tempfile.mkstemp()
-        afo = os.fdopen(afd, 'w')
-        afo.write(data)
-        afo.flush()
-        afo.close()
-
-        remote = os.path.join(tmp, name)
-        conn.put_file(afile, remote)
-        os.unlink(afile)
-        return remote
-
-    # *****************************************************
-
-    def _add_setup_vars(self, inject, args):
-        ''' setup module variables need special handling '''
-
-        is_dict = False
-        if type(args) == dict:
-            is_dict = True
-
-        # TODO: keep this as a dict through the whole path to simplify this code
-        for (k,v) in inject.iteritems():
-            if not k.startswith('facter_') and not k.startswith('ohai_') and not k.startswith('ansible_'):
-                if not is_dict:
-                    if str(v).find(" ") != -1:
-                        v = "\"%s\"" % v
-                    args += " %s=%s" % (k, str(v).replace(" ","~~~"))
-                else:
-                    args[k]=v
-        return args   
- 
-    # *****************************************************
-
-    def _add_setup_metadata(self, args):
-        ''' automatically determine where to store variables for the setup module '''
-        
-        is_dict = False
-        if type(args) == dict:
-            is_dict = True
-
-        # TODO: keep this as a dict through the whole path to simplify this code
-        if not is_dict:
-            if args.find("metadata=") == -1:
-                if self.remote_user == 'root':
-                    args = "%s metadata=/etc/ansible/setup" % args
-                else:
-                    args = "%s metadata=$HOME/.ansible/setup" % args
-        else:
-            if not 'metadata' in args:
-                if self.remote_user == 'root':
-                    args['metadata'] = '/etc/ansible/setup'
-                else:
-                    args['metadata'] = "$HOME/.ansible/setup"
-        return args   
- 
-    # *****************************************************
-
-    def _execute_module(self, conn, tmp, remote_module_path, args, 
-        async_jid=None, async_module=None, async_limit=None):
-        ''' runs a module that has already been transferred '''
-
-        inject = self.setup_cache.get(conn.host,{})
-        conditional = utils.double_template(self.conditional, inject, self.setup_cache)
-        if not eval(conditional):
-            return [ utils.smjson(dict(skipped=True)), None, 'skipped' ]
-
-        host_variables = self.inventory.get_variables(conn.host)
-        inject.update(host_variables)
-
-        if self.module_name == 'setup':
-            args = self._add_setup_vars(inject, args)
-            args = self._add_setup_metadata(args)
-
-        if type(args) == dict:
-            args = utils.bigjson(args)
-        args = utils.template(args, inject, self.setup_cache)
-
-        module_name_tail = remote_module_path.split("/")[-1]
-
-        argsfile = self._transfer_str(conn, tmp, 'arguments', args)
-        if async_jid is None:
-            cmd = "%s %s" % (remote_module_path, argsfile)
-        else:
-            cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module, argsfile]])
-
-        res, err = self._exec_command(conn, cmd, tmp, sudoable=True)
-        client_executed_str = "%s %s" % (module_name_tail, args.strip())
-        return ( res, err, client_executed_str )
-
-    # *****************************************************
-
-    def _save_setup_result_to_disk(self, conn, result):
-       ''' cache results of calling setup '''
-
-       dest = os.path.expanduser("~/.ansible_setup_data")
-       user = getpass.getuser()
-       if user == 'root':
-           dest = "/var/lib/ansible/setup_data"
-       if not os.path.exists(dest):
-           os.makedirs(dest)
-
-       fh = open(os.path.join(dest, conn.host), "w")
-       fh.write(result)
-       fh.close()
-
-       return result
-
-    # *****************************************************
-
-    def _add_result_to_setup_cache(self, conn, result):
-        ''' allows discovered variables to be used in templates and action statements '''
-
-        host = conn.host
-        if 'ansible_facts' in result:
-            var_result = result['ansible_facts']
-        else:
-            var_result = {}
-
-        # note: do not allow variables from playbook to be stomped on
-        # by variables coming up from facter/ohai/etc.  They
-        # should be prefixed anyway
-        if not host in self.setup_cache:
-            self.setup_cache[host] = {}
-        for (k, v) in var_result.iteritems():
-            if not k in self.setup_cache[host]:
-                self.setup_cache[host][k] = v
-
-    # *****************************************************
-
-    def _execute_normal_module(self, conn, host, tmp, module_name):
-        ''' transfer & execute a module that is not 'copy' or 'template' '''
-
-        # shell and command are the same module
-        if module_name == 'shell':
-            module_name = 'command'
-            self.module_args += " #USE_SHELL"
-
-        module = self._transfer_module(conn, tmp, module_name)
-        (result, err, executed) = self._execute_module(conn, tmp, module, self.module_args)
-
-        (host, ok, data, err) = self._return_from_module(conn, host, result, err, executed)
-
-        if ok:
-            self._add_result_to_setup_cache(conn, data)
-
-        return (host, ok, data, err)
-
-    # *****************************************************
-
-    def _execute_async_module(self, conn, host, tmp, module_name):
-        ''' transfer the given module name, plus the async module, then run it '''
-
-        # hack to make the 'shell' module keyword really be executed
-        # by the command module
-        module_args = self.module_args
-        if module_name == 'shell':
-            module_name = 'command'
-            module_args += " #USE_SHELL"
-
-        async  = self._transfer_module(conn, tmp, 'async_wrapper')
-        module = self._transfer_module(conn, tmp, module_name)
-        (result, err, executed) = self._execute_module(conn, tmp, async, module_args,
-           async_module=module, 
-           async_jid=self.generated_jid, 
-           async_limit=self.background
-        )
-
-        return self._return_from_module(conn, host, result, err, executed)
-
-    # *****************************************************
-
-    def _execute_copy(self, conn, host, tmp):
-        ''' handler for file transfer operations '''
-
-        # load up options
-        options = utils.parse_kv(self.module_args)
-        source = options.get('src', None)
-        dest   = options.get('dest', None)
-        if (source is None and not 'first_available_file' in self.module_vars) or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
-
-        # apply templating to source argument
-        inject = self.setup_cache.get(conn.host,{})
-        
-        # FIXME: break duplicate code up into subfunction
-        # if we have first_available_file in our vars
-        # look up the files and use the first one we find as src
-        if 'first_available_file' in self.module_vars:
-            found = False
-            for fn in self.module_vars.get('first_available_file'):
-                fn = utils.template(fn, inject, self.setup_cache)
-                if os.path.exists(fn):
-                    source = fn
-                    found = True
-                    break
-            if not found:
-                return (host, True, dict(failed=True, msg="could not find src in first_available_file list"), '')
-        
-        source = utils.template(source, inject, self.setup_cache)
-
-        # transfer the file to a remote tmp location
-        tmp_src = tmp + source.split('/')[-1]
-        conn.put_file(utils.path_dwim(self.basedir, source), tmp_src)
-
-        # install the copy  module
-        self.module_name = 'copy'
-        module = self._transfer_module(conn, tmp, 'copy')
-
-        # run the copy module
-        args = "src=%s dest=%s" % (tmp_src, dest)
-        (result1, err, executed) = self._execute_module(conn, tmp, module, args)
-        (host, ok, data, err) = self._return_from_module(conn, host, result1, err, executed)
-
-        if ok:
-            return self._chain_file_module(conn, tmp, data, err, options, executed)
-        else:
-            return (host, ok, data, err) 
-
-    # *****************************************************
-
-    def _execute_fetch(self, conn, host, tmp):
-        ''' handler for fetch operations '''
-
-        # load up options
-        options = utils.parse_kv(self.module_args)
-        source = options.get('src', None)
-        dest = options.get('dest', None)
-        if source is None or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
-
-        # files are saved in dest dir, with a subdir for each host, then the filename
-        dest   = "%s/%s/%s" % (utils.path_dwim(self.basedir, dest), host, source)
-        dest   = dest.replace("//","/")
-
-        # compare old and new md5 for support of change hooks
-        local_md5 = None
-        if os.path.exists(dest):
-            local_md5 = os.popen("md5sum %s" % dest).read().split()[0]
-        remote_md5 = self._exec_command(conn, "md5sum %s" % source, tmp, True)[0].split()[0]
-
-        if remote_md5 != local_md5:
-            # create the containing directories, if needed
-            os.makedirs(os.path.dirname(dest))
-            # fetch the file and check for changes
-            conn.fetch_file(source, dest)
-            new_md5 = os.popen("md5sum %s" % dest).read().split()[0]
-            if new_md5 != remote_md5:
-                return (host, True, dict(failed=True, msg="md5 mismatch", md5sum=new_md5), '')
-            return (host, True, dict(changed=True, md5sum=new_md5), '')
-        else:
-            return (host, True, dict(changed=False, md5sum=local_md5), '')
-        
-        
-    # *****************************************************
-
-    def _chain_file_module(self, conn, tmp, data, err, options, executed):
-        ''' handles changing file attribs after copy/template operations '''
-
-        old_changed = data.get('changed', False)
-        module = self._transfer_module(conn, tmp, 'file')
-        args = ' '.join([ "%s=%s" % (k,v) for (k,v) in options.items() ])
-        (result2, err2, executed2) = self._execute_module(conn, tmp, module, args)
-        results2 = self._return_from_module(conn, conn.host, result2, err2, executed)
-        (host, ok, data2, err2) = results2
-        if ok:
-            new_changed = data2.get('changed', False)
-            data.update(data2)
-        else:
-            new_changed = False
-        if old_changed or new_changed:
-            data['changed'] = True
-        return (host, ok, data, "%s%s"%(err,err2))
-
-    # *****************************************************
-
-    def _execute_template(self, conn, host, tmp):
-        ''' handler for template operations '''
-
-        # load up options
-        options  = utils.parse_kv(self.module_args)
-        source   = options.get('src', None)
-        dest     = options.get('dest', None)
-        metadata = options.get('metadata', None)
-        if (source is None and 'first_available_file' not in self.module_vars) or dest is None:
-            return (host, True, dict(failed=True, msg="src and dest are required"), '')
-
-        # apply templating to source argument so vars can be used in the path
-        inject = self.setup_cache.get(conn.host,{})
-
-        # if we have first_available_file in our vars
-        # look up the files and use the first one we find as src
-        if 'first_available_file' in self.module_vars:
-            found = False
-            for fn in self.module_vars.get('first_available_file'):
-                fn = utils.template(fn, inject, self.setup_cache)
-                if os.path.exists(fn):
-                    source = fn
-                    found = True
-                    break
-            if not found:
-                return (host, True, dict(failed=True, msg="could not find src in first_available_file list"), '')
-
-        source = utils.template(source, inject, self.setup_cache)
-
-        (host, ok, data, err) = (None, None, None, None)
-
-        if not self.is_playbook:
-
-            # not running from a playbook so we have to fetch the remote
-            # setup file contents before proceeding...
-            if metadata is None:
-                if self.remote_user == 'root':
-                    metadata = '/etc/ansible/setup'
-                else:
-                    # path is expanded on remote side
-                    metadata = "~/.ansible/setup"
-            
-            # install the template module
-            slurp_module = self._transfer_module(conn, tmp, 'slurp')
-
-            # run the slurp module to get the metadata file
-            args = "src=%s" % metadata
-            (result1, err, executed) = self._execute_module(conn, tmp, slurp_module, args)
-            result1 = utils.json_loads(result1)
-            if not 'content' in result1 or result1.get('encoding','base64') != 'base64':
-                result1['failed'] = True
-                return self._return_from_module(conn, host, result1, err, executed)
-            content = base64.b64decode(result1['content'])
-            inject = utils.json_loads(content)
-
-        # install the template module
-        copy_module = self._transfer_module(conn, tmp, 'copy')
-
-        # template the source data locally
-        source_data = file(utils.path_dwim(self.basedir, source)).read()
-        resultant = ''            
-        try:
-            resultant = utils.template(source_data, inject, self.setup_cache)
-        except Exception, e:
-            return (host, False, dict(failed=True, msg=str(e)), '')
-        xfered = self._transfer_str(conn, tmp, 'source', resultant)
-            
-        # run the COPY module
-        args = "src=%s dest=%s" % (xfered, dest)
-        (result1, err, executed) = self._execute_module(conn, tmp, copy_module, args)
-        (host, ok, data, err) = self._return_from_module(conn, host, result1, err, executed)
- 
-        # modify file attribs if needed
-        if ok:
-            executed = executed.replace("copy","template",1)
-            return self._chain_file_module(conn, tmp, data, err, options, executed)
-        else:
-            return (host, ok, data, err)
-
-    # *****************************************************
-
-    def _executor(self, host):
-        try:
-            (host, ok, data, err) = self._executor_internal(host)
-            if not ok:
-                self.callbacks.on_unreachable(host, data)
-            return (host, ok, data)
-        except errors.AnsibleError, ae:
-            msg = str(ae)
-            self.callbacks.on_unreachable(host, msg)
-            return [host, False, msg]
-        except Exception:
-            msg = traceback.format_exc()
-            self.callbacks.on_unreachable(host, msg)
-            return [host, False, msg]
-
-    def _executor_internal(self, host):
-        ''' callback executed in parallel for each host. returns (hostname, connected_ok, extra) '''
-
-        host_variables = self.inventory.get_variables(host)
-        port = host_variables.get('ansible_ssh_port', self.remote_port)
-
-        conn = None
-        try:
-            conn = self.connector.connect(host, port)
-        except errors.AnsibleConnectionFailed, e:
-            return [ host, False, "FAILED: %s" % str(e), None ]
-
-        cache = self.setup_cache.get(host, {})
-        module_name = utils.template(self.module_name, cache, self.setup_cache)
-
-        tmp = self._get_tmp_path(conn)
-        result = None
-
-        if self.module_name == 'copy':
-            result = self._execute_copy(conn, host, tmp)
-        elif self.module_name == 'fetch':
-            result = self._execute_fetch(conn, host, tmp)
-        elif self.module_name == 'template':
-            result = self._execute_template(conn, host, tmp)
-        else:
-            if self.background == 0:
-                result = self._execute_normal_module(conn, host, tmp, module_name)
-            else:
-                result = self._execute_async_module(conn, host, tmp, module_name)
-
-        self._delete_remote_files(conn, tmp)
-        conn.close()
-
-        (host, connect_ok, data, err) = result
-        if not connect_ok:
-            self.callbacks.on_unreachable(host, data)
-        else:
-            if 'failed' in data or 'rc' in data and str(data['rc']) != '0':
-                self.callbacks.on_failed(host, data)
-            elif 'skipped' in data:
-                self.callbacks.on_skipped(host)
-            else:
-                self.callbacks.on_ok(host, data)
-
-            if err:
-                if self.debug or data.get('parsed', True) == False:
-                    self.callbacks.on_error(host, err)
-
-        return result
-
-    # *****************************************************
-
-    def _exec_command(self, conn, cmd, tmp, sudoable=False):
-        ''' execute a command string over SSH, return the output '''
-
-        stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudoable=sudoable)
-        err=None
-        out=None
-        if type(stderr) != str:
-            err="\n".join(stderr.readlines())
-        else:
-            err=stderr
-        if type(stdout) != str:
-            out="\n".join(stdout.readlines())
-        else:
-            out=stdout
-        return (out,err) 
-
-
-    # *****************************************************
-
-    def _get_tmp_path(self, conn):
-        ''' gets a temporary path on a remote box '''
-
-        # The problem with this is that it's executed on the
-        # overlord, not on the target so we can't use tempdir and os.path
-        # Only support the *nix world for now by using the $HOME env var
-
-        basetmp = "/var/tmp"
-        if self.remote_user != 'root':
-            basetmp = "$HOME/.ansible/tmp"
-        cmd = "mktemp -d %s/ansible.XXXXXX" % basetmp
-        if self.remote_user != 'root':
-            cmd = "mkdir -p %s && %s" % (basetmp, cmd)
-
-        result, err = self._exec_command(conn, cmd, None, sudoable=False)
-        cleaned = result.split("\n")[0].strip() + '/'
-        return cleaned
-
-
-    # *****************************************************
-
-    def _copy_module(self, conn, tmp, module):
-        ''' transfer a module over SFTP, does not run it '''
-
-        if module.startswith("/"):
-            raise errors.AnsibleFileNotFound("%s is not a module" % module)
-        in_path = os.path.expanduser(os.path.join(self.module_path, module))
-        if not os.path.exists(in_path):
-            raise errors.AnsibleFileNotFound("module not found: %s" % in_path)
-
-        out_path = tmp + module
-        conn.put_file(in_path, out_path)
-        return out_path
-
-    # *****************************************************
-
-    def _parallel_exec(self, hosts):
-        ''' handles mulitprocessing when more than 1 fork is required '''
-
-        job_queue = multiprocessing.Manager().Queue()
-        [job_queue.put(i) for i in hosts]
-
-        result_queue = multiprocessing.Manager().Queue()
-
-        workers = []
-        for i in range(self.forks):
-            prc = multiprocessing.Process(target=_executor_hook,
-                args=(job_queue, result_queue))
-            prc.start()
-            workers.append(prc)
-
-        try:
-            for worker in workers:
-                worker.join()
-        except KeyboardInterrupt:
-            for worker in workers:
-                worker.terminate()
-                worker.join()
-
-        results = []
-        while not result_queue.empty():
-            results.append(result_queue.get(block=False))
-        return results
-
-    # *****************************************************
-
-    def _partition_results(self, results):
-        ''' seperate results by ones we contacted & ones we didn't '''
-
-        results2 = dict(contacted={}, dark={})
-
-        if results is None:
-            return None
-
-        for result in results:
-            (host, contacted_ok, result) = result
-            if contacted_ok:
-                results2["contacted"][host] = result
-            else:
-                results2["dark"][host] = result
-
-        # hosts which were contacted but never got a chance to return
-        for host in self.inventory.list_hosts(self.pattern):
-            if not (host in results2['dark'] or host in results2['contacted']):
-                results2["dark"][host] = {}
-
-        return results2
-
-    # *****************************************************
-
-    def run(self):
-        ''' xfer & run module on all matched hosts '''
-       
-        # find hosts that match the pattern
-        hosts = self.inventory.list_hosts(self.pattern)
-        if len(hosts) == 0:
-            self.callbacks.on_no_hosts()
-            return dict(contacted={}, dark={})
- 
-        hosts = [ (self,x) for x in hosts ]
-        results = None
-        if self.forks > 1:
-            results = self._parallel_exec(hosts)
-        else:
-            results = [ self._executor(h[1]) for h in hosts ]
-        return self._partition_results(results)
-
-    def runAsync(self, time_limit):
-        ''' Run this module asynchronously and return a poller. '''
-        self.background = time_limit
-        results = self.run()
-
-        return results, AsyncPoller(results, self)
-
-class AsyncPoller(object):
-    """ Manage asynchronous jobs. """
-
-    def __init__(self, results, runner):
-        self.runner = runner
-
-        self.results = { 'contacted': {}, 'dark': {}}
-        self.hosts_to_poll = []
-        self.completed = False
-
-        # Get job id and which hosts to poll again in the future
-        jid = None
-        for (host, res) in results['contacted'].iteritems():
-            if res.get('started', False):
-                self.hosts_to_poll.append(host)
-                jid = res.get('ansible_job_id', None)
-            else:
-                self.results['contacted'][host] = res
-        for (host, res) in results['dark'].iteritems():
-            self.results['dark'][host] = res
-
-        if jid is None:
-            raise errors.AnsibleError("unexpected error: unable to determine jid")
-        if len(self.hosts_to_poll)==0:
-            raise errors.AnsibleErrot("unexpected error: no hosts to poll")
-        self.jid = jid
-
-    def poll(self):
-        """ Poll the job status.
-
-            Returns the changes in this iteration."""
-        self.runner.module_name = 'async_status'
-        self.runner.module_args = "jid=%s" % self.jid
-        self.runner.pattern = "*"
-        self.runner.background = 0
-
-        self.runner.inventory.restrict_to(self.hosts_to_poll)
-        results = self.runner.run()
-        self.runner.inventory.lift_restriction()
-
-        hosts = []
-        poll_results = { 'contacted': {}, 'dark': {}, 'polled': {}}
-        for (host, res) in results['contacted'].iteritems():
-            if res.get('started',False):
-                hosts.append(host)
-                poll_results['polled'][host] = res
-            else:
-                self.results['contacted'][host] = res
-                poll_results['contacted'][host] = res
-                if 'failed' in res:
-                    self.runner.callbacks.on_async_failed(host, res, self.jid)
-                else:
-                    self.runner.callbacks.on_async_ok(host, res, self.jid)
-        for (host, res) in results['dark'].iteritems():
-            self.results['dark'][host] = res
-            poll_results['dark'][host] = res
-            self.runner.callbacks.on_async_failed(host, res, self.jid)
-
-        self.hosts_to_poll = hosts
-        if len(hosts)==0:
-            self.completed = True
-
-        return poll_results
-
-    def wait(self, seconds, poll_interval):
-        """ Wait a certain time for job completion, check status every poll_interval. """
-        clock = seconds - poll_interval
-        while (clock >= 0 and not self.completed):
-            time.sleep(poll_interval)
-
-            poll_results = self.poll()
-
-            for (host, res) in poll_results['polled'].iteritems():
-                if res.get('started'):
-                    self.runner.callbacks.on_async_poll(host, res, self.jid, clock)
-
-            clock = clock - poll_interval
-
-        return self.results

lib/ansible/runner/__init__.py

@@ -0,0 +1,803 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import multiprocessing
+import signal
+import os
+import pwd
+import Queue
+import random
+import traceback
+import tempfile
+import time
+import collections
+import socket
+
+import ansible.constants as C
+import ansible.inventory
+from ansible import utils
+from ansible import errors
+from ansible import module_common
+import poller
+import connection
+from return_data import ReturnData
+from ansible.callbacks import DefaultRunnerCallbacks, vv
+
+HAS_ATFORK=True
+try:
+    from Crypto.Random import atfork
+except ImportError:
+    HAS_ATFORK=False
+
+################################################
+
+def _executor_hook(job_queue, result_queue):
+    ''' callback used by multiprocessing pool '''
+
+    # attempt workaround of https://github.com/newsapps/beeswithmachineguns/issues/17
+    # this function also not present in CentOS 6
+    if HAS_ATFORK:
+        atfork()
+
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+    while not job_queue.empty():
+        try:
+            job = job_queue.get(block=False)
+            runner, host = job
+            result_queue.put(runner._executor(host))
+        except Queue.Empty:
+            pass
+        except:
+            traceback.print_exc()
+
+class Runner(object):
+    ''' core API interface to ansible '''
+
+    # see bin/ansible for how this is used...
+
+    def __init__(self,
+        host_list=C.DEFAULT_HOST_LIST,      # ex: /etc/ansible/hosts, legacy usage
+        module_path=C.DEFAULT_MODULE_PATH,  # ex: /usr/share/ansible
+        module_name=C.DEFAULT_MODULE_NAME,  # ex: copy
+        module_args=C.DEFAULT_MODULE_ARGS,  # ex: "src=/tmp/a dest=/tmp/b"
+        forks=C.DEFAULT_FORKS,              # parallelism level
+        timeout=C.DEFAULT_TIMEOUT,          # SSH timeout
+        pattern=C.DEFAULT_PATTERN,          # which hosts?  ex: 'all', 'acme.example.org'
+        remote_user=C.DEFAULT_REMOTE_USER,  # ex: 'username'
+        remote_pass=C.DEFAULT_REMOTE_PASS,  # ex: 'password123' or None if using key
+        remote_port=C.DEFAULT_REMOTE_PORT,  # if SSH on different ports
+        private_key_file=C.DEFAULT_PRIVATE_KEY_FILE, # if not using keys/passwords
+        sudo_pass=C.DEFAULT_SUDO_PASS,      # ex: 'password123' or None
+        background=0,                       # async poll every X seconds, else 0 for non-async
+        basedir=None,                       # directory of playbook, if applicable
+        setup_cache=None,                   # used to share fact data w/ other tasks
+        transport=C.DEFAULT_TRANSPORT,      # 'ssh', 'paramiko', 'local'
+        conditional='True',                 # run only if this fact expression evals to true
+        callbacks=None,                     # used for output
+        sudo=False,                         # whether to run sudo or not
+        sudo_user=C.DEFAULT_SUDO_USER,      # ex: 'root'
+        module_vars=None,                   # a playbooks internals thing
+        is_playbook=False,                  # running from playbook or not?
+        inventory=None,                     # reference to Inventory object
+        subset=None                         # subset pattern
+        ):
+
+        # storage & defaults
+        self.setup_cache      = utils.default(setup_cache, lambda: collections.defaultdict(dict))
+        self.basedir          = utils.default(basedir, lambda: os.getcwd())
+        self.callbacks        = utils.default(callbacks, lambda: DefaultRunnerCallbacks())
+        self.generated_jid    = str(random.randint(0, 999999999999))
+        self.transport        = transport
+        self.inventory        = utils.default(inventory, lambda: ansible.inventory.Inventory(host_list))
+
+        self.module_vars      = utils.default(module_vars, lambda: {})
+        self.sudo_user        = sudo_user
+        self.connector        = connection.Connection(self)
+        self.conditional      = conditional
+        self.module_path      = module_path
+        self.module_name      = module_name
+        self.forks            = int(forks)
+        self.pattern          = pattern
+        self.module_args      = module_args
+        self.timeout          = timeout
+        self.remote_user      = remote_user
+        self.remote_pass      = remote_pass
+        self.remote_port      = remote_port
+        self.private_key_file = private_key_file
+        self.background       = background
+        self.sudo             = sudo
+        self.sudo_pass        = sudo_pass
+        self.is_playbook      = is_playbook
+
+        # misc housekeeping
+        if subset and self.inventory._subset is None:
+            # don't override subset when passed from playbook
+            self.inventory.subset(subset)
+
+        if self.transport == 'ssh' and remote_pass:
+            raise errors.AnsibleError("SSH transport does not support passwords, only keys or agents")
+        if self.transport == 'local':
+            self.remote_user = pwd.getpwuid(os.geteuid())[0]
+
+        # ensure we are using unique tmp paths
+        random.seed()
+
+    # *****************************************************
+
+    def _delete_remote_files(self, conn, files):
+        ''' deletes one or more remote files '''
+
+        if os.getenv("ANSIBLE_KEEP_REMOTE_FILES","0") == "1":
+            # ability to turn off temp file deletion for debug purposes
+            return
+
+        if type(files) == str:
+            files = [ files ]
+        for filename in files:
+            if filename.find('/tmp/') == -1:
+                raise Exception("not going to happen")
+            self._low_level_exec_command(conn, "rm -rf %s" % filename, None)
+
+    # *****************************************************
+
+    def _transfer_str(self, conn, tmp, name, data):
+        ''' transfer string to remote file '''
+
+        if type(data) == dict:
+            data = utils.jsonify(data)
+
+        afd, afile = tempfile.mkstemp()
+        afo = os.fdopen(afd, 'w')
+        afo.write(data.encode('utf8'))
+        afo.flush()
+        afo.close()
+
+        remote = os.path.join(tmp, name)
+        try:
+            conn.put_file(afile, remote)
+        finally:
+            os.unlink(afile)
+        return remote
+
+    # *****************************************************
+
+    def _execute_module(self, conn, tmp, module_name, args,
+        async_jid=None, async_module=None, async_limit=None, inject=None):
+
+        ''' runs a module that has already been transferred '''
+
+        if type(args) == dict:
+            args = utils.jsonify(args,format=True)
+
+        (remote_module_path, is_new_style) = self._copy_module(conn, tmp, module_name, inject)
+        cmd = "chmod u+x %s" % remote_module_path
+        if self.sudo and self.sudo_user != 'root':
+            # deal with possible umask issues once sudo'ed to other user
+            cmd = "chmod a+rx %s" % remote_module_path
+        self._low_level_exec_command(conn, cmd, tmp)
+
+        cmd = ""
+        if not is_new_style:
+            args = utils.template(args, inject)
+            argsfile = self._transfer_str(conn, tmp, 'arguments', args)
+            if async_jid is None:
+                cmd = "%s %s" % (remote_module_path, argsfile)
+            else:
+                cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module, argsfile]])
+        else:
+            if async_jid is None:
+                cmd = "%s" % (remote_module_path)
+            else:
+                cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module]])
+
+        res = self._low_level_exec_command(conn, cmd, tmp, sudoable=True)
+        return ReturnData(conn=conn, result=res)
+
+    # *****************************************************
+
+    def _execute_raw(self, conn, tmp, inject=None):
+        ''' execute a non-module command for bootstrapping, or if there's no python on a device '''
+        return ReturnData(conn=conn, result=dict(
+            stdout=self._low_level_exec_command(conn, self.module_args.encode('utf-8'), tmp, sudoable = True)
+        ))
+
+    # ***************************************************
+
+    def _execute_normal_module(self, conn, tmp, module_name, inject=None):
+        ''' transfer & execute a module that is not 'copy' or 'template' '''
+
+        # shell and command are the same module
+        if module_name == 'shell':
+            module_name = 'command'
+            self.module_args += " #USE_SHELL"
+
+        vv("REMOTE_MODULE %s %s" % (module_name, self.module_args), host=conn.host)
+        exec_rc = self._execute_module(conn, tmp, module_name, self.module_args, inject=inject)
+        return exec_rc
+
+    # *****************************************************
+
+    def _execute_async_module(self, conn, tmp, module_name, inject=None):
+        ''' transfer the given module name, plus the async module, then run it '''
+
+        # shell and command module are the same
+        module_args = self.module_args
+        if module_name == 'shell':
+            module_name = 'command'
+            module_args += " #USE_SHELL"
+
+        (module_path, is_new_style) = self._copy_module(conn, tmp, module_name, inject)
+        self._low_level_exec_command(conn, "chmod a+rx %s" % module_path, tmp)
+
+        return self._execute_module(conn, tmp, 'async_wrapper', module_args,
+           async_module=module_path,
+           async_jid=self.generated_jid,
+           async_limit=self.background,
+           inject=inject
+        )
+
+    # *****************************************************
+
+    def _execute_copy(self, conn, tmp, inject=None):
+        ''' handler for file transfer operations '''
+
+        # load up options
+        options = utils.parse_kv(self.module_args)
+        source  = options.get('src', None)
+        dest    = options.get('dest', None)
+        if (source is None and not 'first_available_file' in inject) or dest is None:
+            result=dict(failed=True, msg="src and dest are required")
+            return ReturnData(conn=conn, result=result)
+
+        # if we have first_available_file in our vars
+        # look up the files and use the first one we find as src
+        if 'first_available_file' in inject:
+            found = False
+            for fn in inject.get('first_available_file'):
+                fn = utils.template(fn, inject)
+                if os.path.exists(fn):
+                    source = fn
+                    found = True
+                    break
+            if not found:
+                results=dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(conn=conn, results=results)
+
+        source = utils.template(source, inject)
+        source = utils.path_dwim(self.basedir, source)
+
+        local_md5 = utils.md5(source)
+        if local_md5 is None:
+            result=dict(failed=True, msg="could not find src=%s" % source)
+            return ReturnData(conn=conn, result=result)
+
+        remote_md5 = self._remote_md5(conn, tmp, dest)
+
+        exec_rc = None
+        if local_md5 != remote_md5:
+            # transfer the file to a remote tmp location
+            tmp_src = tmp + os.path.basename(source)
+            conn.put_file(source, tmp_src)
+            # fix file permissions when the copy is done as a different user
+            if self.sudo and self.sudo_user != 'root':
+                self._low_level_exec_command(conn, "chmod a+r %s" % tmp_src, tmp)
+
+            # run the copy module
+            self.module_args = "%s src=%s" % (self.module_args, tmp_src)
+            return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
+
+        else:
+            # no need to transfer the file, already correct md5
+            result = dict(changed=False, md5sum=remote_md5, transferred=False)
+            return ReturnData(conn=conn, result=result).daisychain('file')
+
+    # *****************************************************
+
+    def _execute_fetch(self, conn, tmp, inject=None):
+        ''' handler for fetch operations '''
+
+        # load up options
+        options = utils.parse_kv(self.module_args)
+        source = options.get('src', None)
+        dest = options.get('dest', None)
+        if source is None or dest is None:
+            results = dict(failed=True, msg="src and dest are required")
+            return ReturnData(conn=conn, result=results)
+
+        # apply templating to source argument
+        source = utils.template(source, inject)
+        # apply templating to dest argument
+        dest = utils.template(dest, inject)
+
+        # files are saved in dest dir, with a subdir for each host, then the filename
+        dest   = "%s/%s/%s" % (utils.path_dwim(self.basedir, dest), conn.host, source)
+        dest   = dest.replace("//","/")
+
+        # calculate md5 sum for the remote file
+        remote_md5 = self._remote_md5(conn, tmp, source)
+
+        # these don't fail because you may want to transfer a log file that possibly MAY exist
+        # but keep going to fetch other log files
+        if remote_md5 == '0':
+            result = dict(msg="unable to calculate the md5 sum of the remote file", file=source, changed=False)
+            return ReturnData(conn=conn, result=result)
+        if remote_md5 == '1':
+            result = dict(msg="the remote file does not exist, not transferring, ignored", file=source, changed=False)
+            return ReturnData(conn=conn, result=result)
+        if remote_md5 == '2':
+            result = dict(msg="no read permission on remote file, not transferring, ignored", file=source, changed=False)
+            return ReturnData(conn=conn, result=result)
+
+        # calculate md5 sum for the local file
+        local_md5 = utils.md5(dest)
+
+        if remote_md5 != local_md5:
+            # create the containing directories, if needed
+            if not os.path.isdir(os.path.dirname(dest)):
+                os.makedirs(os.path.dirname(dest))
+
+            # fetch the file and check for changes
+            conn.fetch_file(source, dest)
+            new_md5 = utils.md5(dest)
+            if new_md5 != remote_md5:
+                result = dict(failed=True, md5sum=new_md5, msg="md5 mismatch", file=source)
+                return ReturnData(conn=conn, result=result)
+            result = dict(changed=True, md5sum=new_md5)
+            return ReturnData(conn=conn, result=result)
+        else:
+            result = dict(changed=False, md5sum=local_md5, file=source)
+            return ReturnData(conn=conn, result=result)
+
+    # *****************************************************
+
+    def _execute_template(self, conn, tmp, inject=None):
+        ''' handler for template operations '''
+
+        if not self.is_playbook:
+            raise errors.AnsibleError("in current versions of ansible, templates are only usable in playbooks")
+
+        # load up options
+        options  = utils.parse_kv(self.module_args)
+        source   = options.get('src', None)
+        dest     = options.get('dest', None)
+        if (source is None and 'first_available_file' not in inject) or dest is None:
+            result = dict(failed=True, msg="src and dest are required")
+            return ReturnData(conn=conn, comm_ok=False, result=result)
+
+        # if we have first_available_file in our vars
+        # look up the files and use the first one we find as src
+        if 'first_available_file' in inject:
+            found = False
+            for fn in self.module_vars.get('first_available_file'):
+                fn = utils.template(fn, inject)
+                if os.path.exists(fn):
+                    source = fn
+                    found = True
+                    break
+            if not found:
+                result = dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(conn=conn, comm_ok=False, result=result)
+
+        source = utils.template(source, inject)
+
+        # template the source data locally & transfer
+        try:
+            resultant = utils.template_from_file(self.basedir, source, inject)
+        except Exception, e:
+            result = dict(failed=True, msg=str(e))
+            return ReturnData(conn=conn, comm_ok=False, result=result)
+
+        xfered = self._transfer_str(conn, tmp, 'source', resultant)
+
+        # run the copy module, queue the file module
+        self.module_args = "%s src=%s dest=%s" % (self.module_args, xfered, dest)
+        return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
+
+    # *****************************************************
+
+    def _execute_assemble(self, conn, tmp, inject=None):
+        ''' handler for assemble operations '''
+
+        # FIXME: once assemble is ported over to the use the new common logic, this method
+        # will be unneccessary as it can decide to daisychain via it's own module returns.
+        # and this function can be deleted.
+
+        return self._execute_module(conn, tmp, 'assemble', self.module_args, inject=inject).daisychain('file')
+
+    # *****************************************************
+
+    def _executor(self, host):
+        ''' handler for multiprocessing library '''
+
+        try:
+            exec_rc = self._executor_internal(host)
+            if type(exec_rc) != ReturnData:
+                raise Exception("unexpected return type: %s" % type(exec_rc))
+            # redundant, right?
+            if not exec_rc.comm_ok:
+                self.callbacks.on_unreachable(host, exec_rc.result)
+            return exec_rc
+        except errors.AnsibleError, ae:
+            msg = str(ae)
+            self.callbacks.on_unreachable(host, msg)
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
+        except Exception:
+            msg = traceback.format_exc()
+            self.callbacks.on_unreachable(host, msg)
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
+
+    # *****************************************************
+
+    def _executor_internal(self, host):
+        ''' executes any module one or more times '''
+
+        host_variables = self.inventory.get_variables(host)
+        port = host_variables.get('ansible_ssh_port', self.remote_port)
+
+        inject = {}
+        inject.update(host_variables)
+        inject.update(self.module_vars)
+        inject.update(self.setup_cache[host])
+        inject['hostvars'] = self.setup_cache
+
+        # allow with_items to work in playbooks...
+        # apt and yum are converted into a single call, others run in a loop
+
+        items = self.module_vars.get('items', [])
+        if isinstance(items, basestring) and items.startswith("$"):
+            items = items.replace("$","")
+            if items in inject:
+                items = inject[items]
+            else:
+                raise errors.AnsibleError("unbound variable in with_items: %s" % items)
+        if type(items) != list:
+            raise errors.AnsibleError("with_items only takes a list: %s" % items)
+
+        if len(items) and self.module_name in [ 'apt', 'yum' ]:
+            # hack for apt and soon yum, with_items maps back into a single module call
+            inject['item'] = ",".join(items)
+            items = []
+
+        # logic to decide how to run things depends on whether with_items is used
+
+        if len(items) == 0:
+            return self._executor_internal_inner(host, inject, port)
+        else:
+            # executing using with_items, so make multiple calls
+            # TODO: refactor
+            aggregrate = {}
+            all_comm_ok = True
+            all_changed = False
+            all_failed = False
+            results = []
+            # Save module name and args since daisy-chaining can overwrite them
+            module_name = self.module_name
+            module_args = self.module_args
+            for x in items:
+                self.module_name = module_name
+                self.module_args = module_args
+                inject['item'] = x
+                result = self._executor_internal_inner(host, inject, port)
+                results.append(result.result)
+                if result.comm_ok == False:
+                    all_comm_ok = False
+                    break
+                for x in results:
+                    if x.get('changed') == True:
+                        all_changed = True
+                    if (x.get('failed') == True) or (('rc' in x) and (x['rc'] != 0)):
+                        all_failed = True
+                        break
+            msg = 'All items succeeded'
+            if all_failed:
+                msg = "One or more items failed."
+            rd_result = dict(failed=all_failed, changed=all_changed, results=results, msg=msg)
+            if not all_failed:
+                del rd_result['failed']
+            return ReturnData(host=host, comm_ok=all_comm_ok, result=rd_result)
+
+    # *****************************************************
+
+    def _executor_internal_inner(self, host, inject, port, is_chained=False):
+        ''' decides how to invoke a module '''
+
+        # FIXME: temporary, need to refactor to pass as parameters versus reassigning
+        prev_module_name = self.module_name
+        prev_module_args = self.module_args
+
+        # special non-user/non-fact variables:
+        # 'groups' variable is a list of host name in each group
+        # 'hostvars' variable contains variables for each host name
+        #  ... and is set elsewhere
+        # 'inventory_hostname' is also set elsewhere
+        group_hosts = {}
+        for g in self.inventory.groups:
+            group_hosts[g.name] = [ h.name for h in g.hosts ]
+        inject['groups'] = group_hosts
+
+        # allow module args to work as a dictionary
+        # though it is usually a string
+        new_args = ""
+        if type(self.module_args) == dict:
+            for (k,v) in self.module_args.iteritems():
+                new_args = new_args + "%s='%s' " % (k,v)
+            self.module_args = new_args
+        self.module_args = utils.template(self.module_args, inject)
+
+        conditional = utils.template(self.conditional, inject)
+        if not eval(conditional):
+            result = utils.jsonify(dict(skipped=True))
+            self.callbacks.on_skipped(host, inject.get('item',None))
+            return ReturnData(host=host, result=result)
+
+        conn = None
+        actual_host = host
+        try:
+            delegate_to = inject.get('delegate_to', None)
+            if delegate_to is not None:
+                actual_host = delegate_to    
+            conn = self.connector.connect(actual_host, port)
+            if delegate_to is not None:
+                conn._delegate_for = host
+        except errors.AnsibleConnectionFailed, e:
+            result = dict(failed=True, msg="FAILED: %s" % str(e))
+            return ReturnData(host=host, comm_ok=False, result=result)
+
+        module_name = utils.template(self.module_name, inject)
+
+        tmp = ''
+        if self.module_name != 'raw':
+            tmp = self._make_tmp_path(conn)
+        result = None
+
+        handler = getattr(self, "_execute_%s" % self.module_name, None)
+        if handler:
+            result = handler(conn, tmp, inject=inject)
+        else:
+            if self.background == 0:
+                result = self._execute_normal_module(conn, tmp, module_name, inject=inject)
+            else:
+                result = self._execute_async_module(conn, tmp, module_name, inject=inject)
+
+        if result.is_successful() and 'daisychain' in result.result:
+            self.module_name = result.result['daisychain']
+            if 'daisychain_args' in result.result:
+                self.module_args = result.result['daisychain_args']
+            result2 = self._executor_internal_inner(host, inject, port, is_chained=True)
+
+            # FIXME: remove this hack
+            self.module_name = prev_module_name
+            self.module_args = prev_module_args
+
+            changed = False
+            if result.result.get('changed',False) or result2.result.get('changed',False):
+                changed = True
+            result.result.update(result2.result)
+            result.result['changed'] = changed
+
+            del result.result['daisychain']
+
+        if self.module_name != 'raw':
+            self._delete_remote_files(conn, tmp)
+        conn.close()
+
+        if not result.comm_ok:
+            # connection or parsing errors...
+            self.callbacks.on_unreachable(host, result.result)
+        else:
+            data = result.result
+            if 'item' in inject:
+                result.result['item'] = inject['item']
+
+            result.result['invocation'] = dict(
+                module_args=self.module_args,
+                module_name=self.module_name
+            )
+
+            if is_chained:
+                # no callbacks
+                return result
+            if 'skipped' in data:
+                self.callbacks.on_skipped(host)
+            elif not result.is_successful():
+                ignore_errors = self.module_vars.get('ignore_errors', False)
+                self.callbacks.on_failed(host, data, ignore_errors)
+            else:
+                self.callbacks.on_ok(host, data)
+        return result
+
+    # *****************************************************
+
+    def _low_level_exec_command(self, conn, cmd, tmp, sudoable=False):
+        ''' execute a command string over SSH, return the output '''
+
+        sudo_user = self.sudo_user
+        stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudo_user, sudoable=sudoable)
+
+        if type(stdout) != str:
+            out = "\n".join(stdout.readlines())
+        else:
+            out = stdout
+
+        if type(stderr) != str:
+            err = "\n".join(stderr.readlines())
+        else:
+            err = stderr
+
+        return out + err
+
+    # *****************************************************
+
+    def _remote_md5(self, conn, tmp, path):
+        ''' takes a remote md5sum without requiring python, and returns 0 if no file '''
+
+        test = "rc=0; [ -r \"%s\" ] || rc=2; [ -f \"%s\" ] || rc=1" % (path,path)
+        md5s = [
+            "(/usr/bin/md5sum %s 2>/dev/null)" % path,
+            "(/sbin/md5sum -q %s 2>/dev/null)" % path,
+            "(/usr/bin/digest -a md5 -v %s 2>/dev/null)" % path
+        ]
+
+        cmd = " || ".join(md5s)
+        cmd = "%s; %s || (echo \"${rc}  %s\")" % (test, cmd, path)
+        data = self._low_level_exec_command(conn, cmd, tmp, sudoable=False)
+        data = utils.last_non_blank_line(data)
+        return data.split()[0]
+
+    # *****************************************************
+
+    def _make_tmp_path(self, conn):
+        ''' make and return a temporary path on a remote box '''
+
+        basefile = 'ansible-%s-%s' % (time.time(), random.randint(0, 2**48))
+        basetmp = os.path.join(C.DEFAULT_REMOTE_TMP, basefile)
+        if self.remote_user == 'root':
+            basetmp = os.path.join('/var/tmp', basefile)
+        elif self.sudo and self.sudo_user != 'root':
+            basetmp = os.path.join('/tmp', basefile)
+
+        cmd = 'mkdir -p %s' % basetmp
+        if self.remote_user != 'root':
+            cmd += ' && chmod a+rx %s' % basetmp
+        cmd += ' && echo %s' % basetmp
+
+        result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
+        return utils.last_non_blank_line(result).strip() + '/'
+
+    # *****************************************************
+
+    def _copy_module(self, conn, tmp, module, inject):
+        ''' transfer a module over SFTP, does not run it '''
+
+        if module.startswith("/"):
+            raise errors.AnsibleFileNotFound("%s is not a module" % module)
+
+        # Search module path(s) for named module.
+        for module_path in self.module_path.split(os.pathsep):
+            in_path = os.path.expanduser(os.path.join(module_path, module))
+            if os.path.exists(in_path):
+                break
+        else:
+            raise errors.AnsibleFileNotFound("module %s not found in %s" % (module, self.module_path))
+
+        out_path = os.path.join(tmp, module)
+
+        module_data = ""
+        is_new_style=False
+
+        with open(in_path) as f:
+            module_data = f.read()
+            if module_common.REPLACER in module_data:
+                is_new_style=True
+            module_data = module_data.replace(module_common.REPLACER, module_common.MODULE_COMMON)
+            encoded_args = "\"\"\"%s\"\"\"" % utils.template(self.module_args, inject).replace("\"","\\\"")
+            module_data = module_data.replace(module_common.REPLACER_ARGS, encoded_args)
+
+        # use the correct python interpreter for the host
+        if 'ansible_python_interpreter' in inject:
+            interpreter = inject['ansible_python_interpreter']
+            module_lines = module_data.split('\n')
+            if '#!' and 'python' in module_lines[0]:
+                module_lines[0] = "#!%s" % interpreter
+            module_data = "\n".join(module_lines)
+
+        self._transfer_str(conn, tmp, module, module_data)
+        return (out_path, is_new_style)
+
+    # *****************************************************
+
+    def _parallel_exec(self, hosts):
+        ''' handles mulitprocessing when more than 1 fork is required '''
+
+        job_queue = multiprocessing.Manager().Queue()
+        [job_queue.put(i) for i in hosts]
+        result_queue = multiprocessing.Manager().Queue()
+
+        workers = []
+        for i in range(self.forks):
+            prc = multiprocessing.Process(target=_executor_hook,
+                args=(job_queue, result_queue))
+            prc.start()
+            workers.append(prc)
+
+        try:
+            for worker in workers:
+                worker.join()
+        except KeyboardInterrupt:
+            for worker in workers:
+                worker.terminate()
+                worker.join()
+
+        results = []
+        try:
+            while not result_queue.empty():
+                results.append(result_queue.get(block=False))
+        except socket.error:
+            raise errors.AnsibleError("<interrupted>")
+        return results
+
+    # *****************************************************
+
+    def _partition_results(self, results):
+        ''' seperate results by ones we contacted & ones we didn't '''
+
+        if results is None:
+            return None
+        results2 = dict(contacted={}, dark={})
+
+        for result in results:
+            host = result.host
+            if host is None:
+                raise Exception("internal error, host not set")
+            if result.communicated_ok():
+                results2["contacted"][host] = result.result
+            else:
+                results2["dark"][host] = result.result
+
+        # hosts which were contacted but never got a chance to return
+        for host in self.inventory.list_hosts(self.pattern):
+            if not (host in results2['dark'] or host in results2['contacted']):
+                results2["dark"][host] = {}
+        return results2
+
+    # *****************************************************
+
+    def run(self):
+        ''' xfer & run module on all matched hosts '''
+
+        # find hosts that match the pattern
+        hosts = self.inventory.list_hosts(self.pattern)
+        if len(hosts) == 0:
+            self.callbacks.on_no_hosts()
+            return dict(contacted={}, dark={})
+
+        hosts = [ (self,x) for x in hosts ]
+        results = None
+        if self.forks > 1:
+            results = self._parallel_exec(hosts)
+        else:
+            results = [ self._executor(h[1]) for h in hosts ]
+        return self._partition_results(results)
+
+    # *****************************************************
+
+    def run_async(self, time_limit):
+        ''' Run this module asynchronously and return a poller. '''
+
+        self.background = time_limit
+        results = self.run()
+        return results, poller.AsyncPoller(results, self)

lib/ansible/runner/connection.py

@@ -0,0 +1,46 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+################################################
+
+from ansible import utils
+from ansible.errors import AnsibleError
+
+import os.path
+dirname = os.path.dirname(__file__)
+modules = utils.import_plugins(os.path.join(dirname, 'connection_plugins'))
+
+# rename this module
+modules['paramiko'] = modules['paramiko_ssh']
+del modules['paramiko_ssh']
+
+class Connection(object):
+    ''' Handles abstract connections to remote hosts '''
+
+    def __init__(self, runner):
+        self.runner = runner
+
+    def connect(self, host, port=None):
+        conn = None
+        transport = self.runner.transport
+        module = modules.get(transport, None)
+        if module is None:
+            raise AnsibleError("unsupported connection type: %s" % transport)
+        conn = module.Connection(self.runner, host, port)
+        return conn.connect()
+

lib/ansible/runner/connection_plugins/local.py

@@ -0,0 +1,78 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import traceback
+import os
+import shutil
+import subprocess
+from ansible import errors
+from ansible.callbacks import vvv
+
+class Connection(object):
+    ''' Local based connections '''
+
+    def __init__(self, runner, host, port):
+        self.runner = runner
+        self.host = host
+        # port is unused, since this is local
+        self.port = port 
+
+    def connect(self, port=None):
+        ''' connect to the local host; nothing to do here '''
+
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user, sudoable=False):
+        ''' run a command on the local host '''
+
+        if self.runner.sudo and sudoable:
+            if self.runner.sudo_pass:
+                # NOTE: if someone wants to add sudo w/ password to the local connection type, they are welcome
+                # to do so.  The primary usage of the local connection is for crontab and kickstart usage however
+                # so this doesn't seem to be a huge priority
+                raise errors.AnsibleError("sudo with password is presently only supported on the 'paramiko' (SSH) and native 'ssh' connection types")
+            cmd = "sudo -u {0} -s {1}".format(sudo_user, cmd)
+
+        vvv("EXEC %s" % cmd, host=self.host)
+        p = subprocess.Popen(cmd, shell=True, stdin=None,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        return ("", stdout, stderr)
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to local '''
+
+        vvv("PUT %s TO %s" % (in_path, out_path), host=self.host)
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        try:
+            shutil.copyfile(in_path, out_path)
+        except shutil.Error:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to copy: %s and %s are the same" % (in_path, out_path))
+        except IOError:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
+
+    def fetch_file(self, in_path, out_path):
+        vvv("FETCH %s TO %s" % (in_path, out_path), host=self.host)
+        ''' fetch a file from local to local -- for copatibility '''
+        self.put_file(in_path, out_path)
+
+    def close(self):
+        ''' terminate the connection; nothing to do here '''
+        pass

lib/ansible/runner/connection_plugins/paramiko_ssh.py

@@ -0,0 +1,160 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import warnings
+import os
+import pipes
+import socket
+import random
+from ansible.callbacks import vvv
+from ansible import errors
+
+# prevent paramiko warning noise -- see http://stackoverflow.com/questions/3920502/
+HAVE_PARAMIKO=False
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore")
+    try:
+        import paramiko
+        HAVE_PARAMIKO=True
+    except ImportError:
+        pass
+
+class Connection(object):
+    ''' SSH based connections with Paramiko '''
+
+    def __init__(self, runner, host, port=None):
+
+        self.ssh = None
+        self.runner = runner
+        self.host = host
+        self.port = port
+        if port is None:
+            self.port = self.runner.remote_port
+
+    def connect(self):
+        ''' activates the connection object '''
+
+        if not HAVE_PARAMIKO:
+            raise errors.AnsibleError("paramiko is not installed")
+
+        user = self.runner.remote_user
+
+        vvv("ESTABLISH CONNECTION FOR USER: %s" % user, host=self.host)
+
+        ssh = paramiko.SSHClient()
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+        try:
+            ssh.connect(self.host, username=user, allow_agent=True, look_for_keys=True,
+                key_filename=self.runner.private_key_file, password=self.runner.remote_pass,
+                timeout=self.runner.timeout, port=self.port)
+        except Exception, e:
+            msg = str(e)
+            if "PID check failed" in msg:
+                raise errors.AnsibleError("paramiko version issue, please upgrade paramiko on the machine running ansible")
+            elif "Private key file is encrypted" in msg:
+                msg = 'ssh %s@%s:%s : %s\nTo connect as a different user, use -u <username>.' % (
+                    user, self.host, self.port, msg)
+                raise errors.AnsibleConnectionFailed(msg)
+            else:
+                raise errors.AnsibleConnectionFailed(msg)
+
+        self.ssh = ssh
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user, sudoable=False):
+        ''' run a command on the remote host '''
+
+        bufsize = 4096
+        try:
+            chan = self.ssh.get_transport().open_session()
+        except Exception, e:
+            msg = "Failed to open session"
+            if len(str(e)) > 0:
+                msg += ": %s" % str(e)
+            raise errors.AnsibleConnectionFailed(msg)
+        chan.get_pty()
+
+        if not self.runner.sudo or not sudoable:
+            quoted_command = '"$SHELL" -c ' + pipes.quote(cmd)
+            vvv("EXEC %s" % quoted_command, host=self.host)
+            chan.exec_command(quoted_command)
+        else:
+            # Rather than detect if sudo wants a password this time, -k makes
+            # sudo always ask for a password if one is required. 
+            # Passing a quoted compound command to sudo (or sudo -s)
+            # directly doesn't work, so we shellquote it with pipes.quote()
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
+            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
+            prompt = '[sudo via ansible, key=%s] password: ' % randbits
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
+            vvv("EXEC %s" % sudocmd, host=self.host)
+            sudo_output = ''
+            try:
+                chan.exec_command(sudocmd)
+                if self.runner.sudo_pass:
+                    while not sudo_output.endswith(prompt):
+                        chunk = chan.recv(bufsize)
+                        if not chunk:
+                            if 'unknown user' in sudo_output:
+                                raise errors.AnsibleError(
+                                    'user %s does not exist' % sudo_user)
+                            else:
+                                raise errors.AnsibleError('ssh connection ' +
+                                    'closed waiting for password prompt')
+                        sudo_output += chunk
+                    chan.sendall(self.runner.sudo_pass + '\n')
+            except socket.timeout:
+                raise errors.AnsibleError('ssh timed out waiting for sudo.\n' + sudo_output)
+
+        return (chan.makefile('wb', bufsize), chan.makefile('rb', bufsize), '')
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to remote '''
+        vvv("PUT %s TO %s" % (in_path, out_path), host=self.host)
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        try:
+            sftp = self.ssh.open_sftp()
+        except:
+            raise errors.AnsibleError("failed to open a SFTP connection")
+        try:
+            sftp.put(in_path, out_path)
+        except IOError:
+            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
+        sftp.close()
+
+    def fetch_file(self, in_path, out_path):
+        ''' save a remote file to the specified path '''
+        vvv("FETCH %s TO %s" % (in_path, out_path), host=self.host)
+        try:
+            sftp = self.ssh.open_sftp()
+        except:
+            raise errors.AnsibleError("failed to open a SFTP connection")
+        try:
+            sftp.get(in_path, out_path)
+        except IOError:
+            raise errors.AnsibleError("failed to transfer file from %s" % in_path)
+        sftp.close()
+
+    def close(self):
+        ''' terminate the connection '''
+        self.ssh.close()
+

lib/ansible/runner/connection_plugins/ssh.py

@@ -0,0 +1,141 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import subprocess
+import shlex
+import pipes
+import random
+import select
+import fcntl
+import ansible.constants as C
+from ansible.callbacks import vvv
+from ansible import errors
+
+class Connection(object):
+    ''' ssh based connections '''
+
+    def __init__(self, runner, host, port):
+        self.runner = runner
+        self.host = host
+        self.port = port
+
+    def connect(self):
+        ''' connect to the remote host '''
+
+        vvv("ESTABLISH CONNECTION FOR USER: %s" % self.runner.remote_user, host=self.host)
+
+        self.common_args = []
+        extra_args = C.ANSIBLE_SSH_ARGS
+        if extra_args is not None:
+            self.common_args += shlex.split(extra_args)
+        else:
+            self.common_args += ["-o", "ControlMaster=auto",
+                                 "-o", "ControlPersist=60s",
+                                 "-o", "ControlPath=/tmp/ansible-ssh-%h-%p-%r"]
+        self.common_args += ["-o", "StrictHostKeyChecking=no"]
+        if self.port is not None:
+            self.common_args += ["-o", "Port=%d" % (self.port)]
+        if self.runner.private_key_file is not None:
+            self.common_args += ["-o", "IdentityFile="+self.runner.private_key_file]
+        self.common_args += ["-o", "User="+self.runner.remote_user]
+
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user,sudoable=False):
+        ''' run a command on the remote host '''
+
+        ssh_cmd = ["ssh", "-tt", "-q"] + self.common_args + [self.host]
+        if self.runner.sudo and sudoable:
+            # Rather than detect if sudo wants a password this time, -k makes
+            # sudo always ask for a password if one is required.
+            # Passing a quoted compound command to sudo (or sudo -s)
+            # directly doesn't work, so we shellquote it with pipes.quote()
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
+            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
+            prompt = '[sudo via ansible, key=%s] password: ' % randbits
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
+            sudo_output = ''
+            ssh_cmd.append(sudocmd)
+            vvv("EXEC %s" % ssh_cmd, host=self.host)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            if self.runner.sudo_pass:
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL,
+                            fcntl.fcntl(p.stdout, fcntl.F_GETFL) | os.O_NONBLOCK)
+                while not sudo_output.endswith(prompt):
+                    rfd, wfd, efd = select.select([p.stdout], [],
+                                                  [p.stdout], self.runner.timeout)
+                    if p.stdout in rfd:
+                        chunk = p.stdout.read()
+                        if not chunk:
+                            raise errors.AnsibleError('ssh connection closed waiting for sudo password prompt')
+                        sudo_output += chunk
+                    else:
+                        stdout = p.communicate()
+                        raise errors.AnsibleError('ssh connection error waiting for sudo password prompt')
+                p.stdin.write(self.runner.sudo_pass + '\n')
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) & ~os.O_NONBLOCK)
+        else:
+            ssh_cmd.append(cmd)
+            vvv("EXEC %s" % ssh_cmd, host=self.host)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+        # We can't use p.communicate here because the ControlMaster may have stdout open as well
+        stdout = ''
+        while p.poll() is None:
+            rfd, wfd, efd = select.select([p.stdout], [], [p.stdout], 1)
+            if p.stdout in rfd:
+                stdout += os.read(p.stdout.fileno(), 1024)
+        p.stdin.close()  # close stdin after we read from stdout (see also issue #848)
+
+        if p.returncode != 0 and stdout.find('Bad configuration option: ControlPersist') != -1:
+            raise errors.AnsibleError('using -c ssh on certain older ssh versions may not support ControlPersist, set ANSIBLE_SSH_ARGS="" (or ssh_args in the config file) before running again')
+
+        return ('', stdout, '')
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to remote '''
+        vvv("PUT %s TO %s" % (in_path, out_path), host=self.host)
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("put %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file to %s:\n%s\n%s" % (out_path, stdout, stderr))
+
+    def fetch_file(self, in_path, out_path):
+        ''' fetch a file from remote to local '''
+        vvv("FETCH %s TO %s" % (in_path, out_path), host=self.host)
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("get %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file from %s:\n%s\n%s" % (in_path, stdout, stderr))
+
+    def close(self):
+        ''' not applicable since we're executing openssh binaries '''
+        pass
+

lib/ansible/runner/poller.py

@@ -0,0 +1,101 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import time
+
+from ansible import errors
+
+class AsyncPoller(object):
+    """ Manage asynchronous jobs. """
+
+    def __init__(self, results, runner):
+        self.runner = runner
+
+        self.results = { 'contacted': {}, 'dark': {}}
+        self.hosts_to_poll = []
+        self.completed = False
+
+        # Get job id and which hosts to poll again in the future
+        jid = None
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started', False):
+                self.hosts_to_poll.append(host)
+                jid = res.get('ansible_job_id', None)
+            else:
+                self.results['contacted'][host] = res
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+
+        if jid is None:
+            raise errors.AnsibleError("unexpected error: unable to determine jid")
+        if len(self.hosts_to_poll)==0:
+            raise errors.AnsibleErrot("unexpected error: no hosts to poll")
+        self.jid = jid
+
+    def poll(self):
+        """ Poll the job status.
+
+            Returns the changes in this iteration."""
+        self.runner.module_name = 'async_status'
+        self.runner.module_args = "jid=%s" % self.jid
+        self.runner.pattern = "*"
+        self.runner.background = 0
+
+        self.runner.inventory.restrict_to(self.hosts_to_poll)
+        results = self.runner.run()
+        self.runner.inventory.lift_restriction()
+
+        hosts = []
+        poll_results = { 'contacted': {}, 'dark': {}, 'polled': {}}
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started',False):
+                hosts.append(host)
+                poll_results['polled'][host] = res
+            else:
+                self.results['contacted'][host] = res
+                poll_results['contacted'][host] = res
+                if 'failed' in res:
+                    self.runner.callbacks.on_async_failed(host, res, self.jid)
+                else:
+                    self.runner.callbacks.on_async_ok(host, res, self.jid)
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+            poll_results['dark'][host] = res
+            self.runner.callbacks.on_async_failed(host, res, self.jid)
+
+        self.hosts_to_poll = hosts
+        if len(hosts)==0:
+            self.completed = True
+
+        return poll_results
+
+    def wait(self, seconds, poll_interval):
+        """ Wait a certain time for job completion, check status every poll_interval. """
+        clock = seconds - poll_interval
+        while (clock >= 0 and not self.completed):
+            time.sleep(poll_interval)
+
+            poll_results = self.poll()
+
+            for (host, res) in poll_results['polled'].iteritems():
+                if res.get('started'):
+                    self.runner.callbacks.on_async_poll(host, res, self.jid, clock)
+
+            clock = clock - poll_interval
+
+        return self.results

lib/ansible/runner/return_data.py

@@ -0,0 +1,59 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+from ansible import utils
+
+class ReturnData(object):
+    ''' internal return class for runner execute methods, not part of public API signature '''
+
+    __slots__ = [ 'result', 'comm_ok', 'host' ]
+
+    def __init__(self, conn=None, host=None, result=None, comm_ok=True):
+
+        # which host is this ReturnData about?
+        if conn is not None:
+            delegate_for = getattr(conn, '_delegate_for', None)
+            if delegate_for:
+                self.host = delegate_for
+            else:
+                self.host = conn.host
+        else:
+            self.host = host
+
+        self.result = result
+        self.comm_ok = comm_ok
+
+        if type(self.result) in [ str, unicode ]:
+            self.result = utils.parse_json(self.result)
+
+        if self.host is None:
+            raise Exception("host not set")
+        if type(self.result) != dict:
+            raise Exception("dictionary result expected")
+
+    def communicated_ok(self):
+        return self.comm_ok
+
+    def is_successful(self):
+        return self.comm_ok and ('failed' not in self.result) and (self.result.get('rc',0) == 0)
+
+    def daisychain(self, module_name):
+        ''' request a module call follow this one '''
+        if self.is_successful():
+            self.result['daisychain'] = module_name
+        return self
+

lib/ansible/utils.py

@@ -15,23 +15,43 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-###############################################################
-
 import sys
 import os
 import shlex
 import re
+import codecs
 import jinja2
 import yaml
 import optparse
-from operator import methodcaller
+import operator
+from ansible import errors
+from ansible import __version__
+import ansible.constants as C
+import time
+import StringIO
+import imp
+import glob
+
+VERBOSITY=0
+
 try:
     import json
 except ImportError:
     import simplejson as json
 
-from ansible import errors
-import ansible.constants as C
+try:
+    from hashlib import md5 as _md5
+except ImportError:
+    from md5 import md5 as _md5
+
+# vars_prompt_encrypt
+PASSLIB_AVAILABLE = False
+
+try:
+    import passlib.hash
+    PASSLIB_AVAILABLE = True
+except:
+    pass
 
 ###############################################################
 # UTILITY FUNCTIONS FOR COMMAND LINE TOOLS
@@ -39,82 +59,27 @@ import ansible.constants as C
 
 def err(msg):
     ''' print an error message to stderr '''
+
     print >> sys.stderr, msg
 
 def exit(msg, rc=1):
     ''' quit with an error to stdout and a failure code '''
+
     err(msg)
     sys.exit(rc)
 
-def bigjson(result):
-    ''' format JSON output (uncompressed) '''
-    # hide some internals magic from command line userland
+def jsonify(result, format=False):
+    ''' format JSON output (uncompressed or uncompressed) '''
+
     result2 = result.copy()
-    if 'invocation' in result2:
-        del result2['invocation']
-    return json.dumps(result2, sort_keys=True, indent=4)
-
-def smjson(result):
-    ''' format JSON output (compressed) '''
-    # hide some internals magic from command line userland
-    result2 = result.copy()
-    if 'invocation' in result2:
-        del result2['invocation']
-    return json.dumps(result2, sort_keys=True)
-
-def task_start_msg(name, conditional):
-    if conditional:
-        return "\nNOTIFIED: [%s] **********\n" % name
+    if format:
+        return json.dumps(result2, sort_keys=True, indent=4)
     else:
-        return "\nTASK: [%s] *********\n" % name
-
-def regular_generic_msg(hostname, result, oneline, caption):
-    ''' output on the result of a module run that is not command '''
-    if not oneline:
-        return "%s | %s >> %s\n" % (hostname, caption, bigjson(result))
-    else:
-        return "%s | %s >> %s\n" % (hostname, caption, smjson(result))
-
-def regular_success_msg(hostname, result, oneline):
-    ''' output the result of a successful module run '''
-    return regular_generic_msg(hostname, result, oneline, 'success')
-
-def regular_failure_msg(hostname, result, oneline):
-    ''' output the result of a failed module run '''
-    return regular_generic_msg(hostname, result, oneline, 'FAILED')
-
-def command_generic_msg(hostname, result, oneline, caption):
-    ''' output the result of a command run '''
-    rc     = result.get('rc', '0')
-    stdout = result.get('stdout','')
-    stderr = result.get('stderr', '')
-    msg    = result.get('msg', '')
-    if not oneline:
-        buf = "%s | %s | rc=%s >>\n" % (hostname, caption, result.get('rc',0))
-        if stdout:
-            buf += stdout
-        if stderr:
-            buf += stderr
-        if msg:
-            buf += msg
-        buf += "\n"
-        return buf
-    else:
-        if stderr:
-            return "%s | %s | rc=%s | (stdout) %s (stderr) %s\n" % (hostname, caption, rc, stdout, stderr)
-        else:
-            return "%s | %s | rc=%s | (stdout) %s\n" % (hostname, caption, rc, stdout)
-
-def command_success_msg(hostname, result, oneline):
-    ''' output from a successful command run '''
-    return command_generic_msg(hostname, result, oneline, 'success')
-
-def command_failure_msg(hostname, result, oneline):
-    ''' output from a failed command run '''
-    return command_generic_msg(hostname, result, oneline, 'FAILED')
+        return json.dumps(result2, sort_keys=True)
 
 def write_tree_file(tree, hostname, buf):
     ''' write something into treedir/hostname '''
+
     # TODO: might be nice to append playbook runs per host in a similar way
     # in which case, we'd want append mode.
     path = os.path.join(tree, hostname)
@@ -124,33 +89,12 @@ def write_tree_file(tree, hostname, buf):
 
 def is_failed(result):
     ''' is a given JSON result a failed result? '''
-    failed = False
-    rc = 0
-    if type(result) == dict:
-        failed = result.get('failed', 0)
-        rc     = result.get('rc', 0)
-    if rc != 0:
-        return True    
-    return failed
 
-def host_report_msg(hostname, module_name, result, oneline):
-    ''' summarize the JSON results for a particular host '''
-    buf = ''
-    failed = is_failed(result)
-    if module_name in [ 'command', 'shell' ] and 'ansible_job_id' not in result:
-        if not failed:
-            buf = command_success_msg(hostname, result, oneline)
-        else:
-            buf = command_failure_msg(hostname, result, oneline)
-    else:
-        if not failed:
-            buf = regular_success_msg(hostname, result, oneline)
-        else:
-            buf = regular_failure_msg(hostname, result, oneline)
-    return buf
+    return ((result.get('rc', 0) != 0) or (result.get('failed', False) in [ True, 'True', 'true']))
 
 def prepare_writeable_dir(tree):
     ''' make sure a directory exists and is writeable '''
+
     if tree != '/':
         tree = os.path.realpath(os.path.expanduser(tree))
     if not os.path.exists(tree):
@@ -163,6 +107,7 @@ def prepare_writeable_dir(tree):
 
 def path_dwim(basedir, given):
     ''' make relative paths work like folks expect '''
+
     if given.startswith("/"):
         return given
     elif given.startswith("~/"):
@@ -171,20 +116,31 @@ def path_dwim(basedir, given):
         return os.path.join(basedir, given)
 
 def json_loads(data):
+    ''' parse a JSON string and return a data structure '''
+
     return json.loads(data)
 
-def parse_json(data):
+def parse_json(raw_data):
     ''' this version for module return data only '''
+
+    # ignore stuff like tcgetattr spewage or other warnings
+    data = filter_leading_non_json_lines(raw_data)
+
     try:
         return json.loads(data)
     except:
         # not JSON, but try "Baby JSON" which allows many of our modules to not
         # require JSON and makes writing modules in bash much simpler
         results = {}
-        tokens = shlex.split(data)
+        try:
+            tokens = shlex.split(data)
+        except:
+            print "failed to parse json: "+ data
+            raise
+
         for t in tokens:
             if t.find("=") == -1:
-                raise errors.AnsibleError("failed to parse: %s" % data)
+                raise errors.AnsibleError("failed to parse: %s" % raw_data)
             (key,value) = t.split("=", 1)
             if key == 'changed' or 'failed':
                 if value.lower() in [ 'true', '1' ]:
@@ -192,25 +148,55 @@ def parse_json(data):
                 elif value.lower() in [ 'false', '0' ]:
                     value = False
             if key == 'rc':
-                value = int(value)     
+                value = int(value)
             results[key] = value
         if len(results.keys()) == 0:
-            return { "failed" : True, "parsed" : False, "msg" : data }
+            return { "failed" : True, "parsed" : False, "msg" : raw_data }
         return results
 
-_KEYCRE = re.compile(r"\$(\w+)")
+_LISTRE = re.compile(r"(\w+)\[(\d+)\]")
+
+class VarNotFoundException(Exception):
+    pass
+
+def _varLookup(name, vars):
+    ''' find the contents of a possibly complex variable in vars. '''
+
+    path = name.split('.')
+    space = vars
+    for part in path:
+        if part in space:
+            space = space[part]
+        elif "[" in part:
+            m = _LISTRE.search(part)
+            if not m:
+                raise VarNotFoundException()
+            try:
+                space = space[m.group(1)][int(m.group(2))]
+            except (KeyError, IndexError):
+                raise VarNotFoundException()
+        else:
+            raise VarNotFoundException()
+    return space
+
+_KEYCRE = re.compile(r"\$(?P<complex>\{){0,1}((?(complex)[\w\.\[\]]+|\w+))(?(complex)\})")
+
+def varLookup(varname, vars):
+    ''' helper function used by varReplace '''
+
+    m = _KEYCRE.search(varname)
+    if not m:
+        return None
+    try:
+        return _varLookup(m.group(2), vars)
+    except VarNotFoundException:
+        return None
 
 def varReplace(raw, vars):
-    '''Perform variable replacement of $vars
-
-    @param raw: String to perform substitution on.  
-    @param vars: Dictionary of variables to replace. Key is variable name
-        (without $ prefix). Value is replacement string.
-    @return: Input raw string with substituted values.
-    '''
+    ''' Perform variable replacement of $variables in string raw using vars dictionary '''
     # this code originally from yum
 
-    done = []                      # Completed chunks to return
+    done = [] # Completed chunks to return
 
     while raw:
         m = _KEYCRE.search(raw)
@@ -220,8 +206,12 @@ def varReplace(raw, vars):
 
         # Determine replacement value (if unknown variable then preserve
         # original)
-        varname = m.group(1).lower()
-        replacement = str(vars.get(varname, m.group()))
+        varname = m.group(2)
+
+        try:
+            replacement = unicode(_varLookup(varname, vars))
+        except VarNotFoundException:
+            replacement = m.group()
 
         start, end = m.span()
         done.append(raw[:start])    # Keep stuff leading up to token
@@ -230,69 +220,192 @@ def varReplace(raw, vars):
 
     return ''.join(done)
 
-def template(text, vars, setup_cache):
-    ''' run a text buffer through the templating engine '''
-    vars = vars.copy()
-    text = varReplace(str(text), vars)
-    vars['hostvars'] = setup_cache
-    template = jinja2.Template(text)
-    return template.render(vars)
+def template(text, vars):
+    ''' run a text buffer through the templating engine until it no longer changes '''
 
-def double_template(text, vars, setup_cache):
-    return template(template(text, vars, setup_cache), vars, setup_cache)
+    prev_text = ''
+    try:
+        text = text.decode('utf-8')
+    except UnicodeEncodeError:
+        pass # already unicode
+    depth = 0
+    while prev_text != text:
+        depth = depth + 1
+        if (depth > 20):
+            raise errors.AnsibleError("template recursion depth exceeded")
+        prev_text = text
+        text = varReplace(unicode(text), vars)
+    return text
 
-def template_from_file(path, vars, setup_cache):
+def template_from_file(basedir, path, vars):
     ''' run a file through the templating engine '''
-    data = file(path).read()
-    return template(data, vars, setup_cache)
+
+    environment = jinja2.Environment(loader=jinja2.FileSystemLoader(basedir), trim_blocks=False)
+    environment.filters['to_json'] = json.dumps
+    environment.filters['from_json'] = json.loads
+    data = codecs.open(path_dwim(basedir, path), encoding="utf8").read()
+    t = environment.from_string(data)
+    vars = vars.copy()
+    res = t.render(vars)
+    if data.endswith('\n') and not res.endswith('\n'):
+        res = res + '\n'
+    return template(res, vars)
 
 def parse_yaml(data):
+    ''' convert a yaml string to a data structure '''
+
     return yaml.load(data)
-  
+
 def parse_yaml_from_file(path):
+    ''' convert a yaml file to a data structure '''
+
     try:
         data = file(path).read()
+        return parse_yaml(data)
     except IOError:
         raise errors.AnsibleError("file not found: %s" % path)
-    return parse_yaml(data)
+    except yaml.YAMLError, exc:
+        if hasattr(exc, 'problem_mark'):
+            mark = exc.problem_mark
+            if mark.line -1 >= 0:
+                before_probline = data.split("\n")[mark.line-1]
+            else:
+                before_probline = ''
+            probline = data.split("\n")[mark.line]
+            arrow = " " * mark.column + "^"
+            msg = """Syntax Error while loading YAML script, %s
+Note: The error may actually appear before this position: line %s, column %s
+
+%s
+%s
+%s""" % (path, mark.line + 1, mark.column + 1, before_probline, probline, arrow)
+        else:
+            # No problem markers means we have to throw a generic
+            # "stuff messed up" type message. Sry bud.
+            msg = "Could not parse YAML. Check over %s again." % path
+        raise errors.AnsibleYAMLValidationFailed(msg)
 
 def parse_kv(args):
     ''' convert a string of key/value items to a dict '''
+
     options = {}
     if args is not None:
-        vargs = shlex.split(args, posix=True)
+        # attempting to split a unicode here does bad things
+        vargs = shlex.split(str(args), posix=True)
         for x in vargs:
             if x.find("=") != -1:
-                k, v = x.split("=")
+                k, v = x.split("=",1)
                 options[k]=v
     return options
 
+def md5(filename):
+    ''' Return MD5 hex digest of local file, or None if file is not present. '''
+
+    if not os.path.exists(filename):
+        return None
+    digest = _md5()
+    blocksize = 64 * 1024
+    infile = open(filename, 'rb')
+    block = infile.read(blocksize)
+    while block:
+        digest.update(block)
+        block = infile.read(blocksize)
+    infile.close()
+    return digest.hexdigest()
+
+def default(value, function):
+    ''' syntactic sugar around lazy evaluation of defaults '''
+    if value is None:
+        return function()
+    return value
+
+def _gitinfo():
+    ''' returns a string containing git branch, commit id and commit date '''
+    result = None
+    repo_path = os.path.join(os.path.dirname(__file__), '..', '..', '.git')
+    
+    if os.path.exists(repo_path):
+        # Check if the .git is a file. If it is a file, it means that we are in a submodule structure.
+        if os.path.isfile(repo_path):
+            try:
+                gitdir = yaml.load(open(repo_path)).get('gitdir')
+                # There is a posibility the .git file to have an absolute path.
+                if os.path.isabs(gitdir):
+                    repo_path = gitdir
+                else:
+                    repo_path = os.path.join(repo_path.split('.git')[0], gitdir)
+            except (IOError, AttributeError):
+                return ''
+        f = open(os.path.join(repo_path, "HEAD"))
+        branch = f.readline().split('/')[-1].rstrip("\n")
+        f.close()
+        branch_path = os.path.join(repo_path, "refs", "heads", branch)
+        if os.path.exists(branch_path):
+            f = open(branch_path)
+            commit = f.readline()[:10]
+            f.close()
+            date = time.localtime(os.stat(branch_path).st_mtime)
+            if time.daylight == 0:  
+                offset = time.timezone
+            else:
+                offset = time.altzone
+            result = "({0} {1}) last updated {2} (GMT {3:+04d})".format(branch, commit,
+                time.strftime("%Y/%m/%d %H:%M:%S", date), offset / -36)
+    else:
+        result = ''
+    return result
+
+def version(prog):
+    result = "{0} {1}".format(prog, __version__)
+    gitinfo = _gitinfo()
+    if gitinfo:
+        result = result + " {0}".format(gitinfo)
+    return result
+
+####################################################################
+# option handling code for /usr/bin/ansible and ansible-playbook
+# below this line
+
 class SortedOptParser(optparse.OptionParser):
     '''Optparser which sorts the options by opt before outputting --help'''
+
     def format_help(self, formatter=None):
-        self.option_list.sort(key=methodcaller('get_opt_string'))
+        self.option_list.sort(key=operator.methodcaller('get_opt_string'))
         return optparse.OptionParser.format_help(self, formatter=None)
 
-def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, async_opts=False, connect_opts=False):
+def increment_debug(option, opt, value, parser):
+    global VERBOSITY
+    VERBOSITY += 1
+
+def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, 
+    async_opts=False, connect_opts=False, subset_opts=False):
     ''' create an options parser for any ansible script '''
 
-    parser = SortedOptParser(usage)
-    parser.add_option('-D','--debug', default=False, action="store_true",
-        help='debug mode')
+    parser = SortedOptParser(usage, version=version("%prog"))
+    parser.add_option('-v','--verbose', default=False, action="callback",
+        callback=increment_debug, help="verbose mode (-vvv for more)")
+
     parser.add_option('-f','--forks', dest='forks', default=constants.DEFAULT_FORKS, type='int',
         help="specify number of parallel processes to use (default=%s)" % constants.DEFAULT_FORKS)
     parser.add_option('-i', '--inventory-file', dest='inventory',
-        help="specify inventory host file (default=%s)" % constants.DEFAULT_HOST_LIST, 
+        help="specify inventory host file (default=%s)" % constants.DEFAULT_HOST_LIST,
         default=constants.DEFAULT_HOST_LIST)
     parser.add_option('-k', '--ask-pass', default=False, dest='ask_pass', action='store_true',
         help='ask for SSH password')
+    parser.add_option('--private-key', default=None, dest='private_key_file',
+        help='use this file to authenticate the connection')
     parser.add_option('-K', '--ask-sudo-pass', default=False, dest='ask_sudo_pass', action='store_true',
         help='ask for sudo password')
     parser.add_option('-M', '--module-path', dest='module_path',
-        help="specify path to module library (default=%s)" % constants.DEFAULT_MODULE_PATH, 
+        help="specify path(s) to module library (default=%s)" % constants.DEFAULT_MODULE_PATH,
         default=constants.DEFAULT_MODULE_PATH)
+
+    if subset_opts:
+        parser.add_option('-l', '--limit', default=constants.DEFAULT_SUBSET, dest='subset',
+            help='further limit selected hosts to an additional pattern')
+
     parser.add_option('-T', '--timeout', default=constants.DEFAULT_TIMEOUT, type='int',
-        dest='timeout', 
+        dest='timeout',
         help="override the SSH timeout in seconds (default=%s)" % constants.DEFAULT_TIMEOUT)
 
     if output_opts:
@@ -304,23 +417,77 @@ def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, asyn
     if runas_opts:
         parser.add_option("-s", "--sudo", default=False, action="store_true",
             dest='sudo', help="run operations with sudo (nopasswd)")
+        parser.add_option('-U', '--sudo-user', dest='sudo_user', help='desired sudo user (default=root)',
+            default=None)   # Can't default to root because we need to detect when this option was given
         parser.add_option('-u', '--user', default=constants.DEFAULT_REMOTE_USER,
-            dest='remote_user', 
+            dest='remote_user',
             help='connect as this user (default=%s)' % constants.DEFAULT_REMOTE_USER)
-    
+
     if connect_opts:
         parser.add_option('-c', '--connection', dest='connection',
-                          choices=C.DEFAULT_TRANSPORT_OPTS,
                           default=C.DEFAULT_TRANSPORT,
                           help="connection type to use (default=%s)" % C.DEFAULT_TRANSPORT)
 
     if async_opts:
         parser.add_option('-P', '--poll', default=constants.DEFAULT_POLL_INTERVAL, type='int',
-            dest='poll_interval', 
+            dest='poll_interval',
             help="set the poll interval if using -B (default=%s)" % constants.DEFAULT_POLL_INTERVAL)
         parser.add_option('-B', '--background', dest='seconds', type='int', default=0,
             help='run asynchronously, failing after X seconds (default=N/A)')
 
     return parser
 
+def do_encrypt(result, encrypt, salt_size=None, salt=None):
+    if PASSLIB_AVAILABLE:
+        try:
+            crypt = getattr(passlib.hash, encrypt)
+        except:
+            raise errors.AnsibleError("passlib does not support '%s' algorithm" % encrypt)
+
+        if salt_size:
+            result = crypt.encrypt(result, salt_size=salt_size)
+        elif salt:
+            result = crypt.encrypt(result, salt=salt)
+        else:
+            result = crypt.encrypt(result)
+    else:
+        raise errors.AnsibleError("passlib must be installed to encrypt vars_prompt values")
+
+    return result
+
+def last_non_blank_line(buf):
+
+    all_lines = buf.splitlines()
+    all_lines.reverse()
+    for line in all_lines:
+        if (len(line) > 0):
+            return line
+    # shouldn't occur unless there's no output
+    return ""  
+
+def filter_leading_non_json_lines(buf):
+    ''' 
+    used to avoid random output from SSH at the top of JSON output, like messages from
+    tcagetattr, or where dropbear spews MOTD on every single command (which is nuts).
+    
+    need to filter anything which starts not with '{', '[', ', '=' or is an empty line.
+    filter only leading lines since multiline JSON is valid. 
+    '''
+
+    filtered_lines = StringIO.StringIO()
+    stop_filtering = False
+    for line in buf.splitlines():
+        if stop_filtering or "=" in line or line.startswith('{') or line.startswith('['):
+            stop_filtering = True
+            filtered_lines.write(line + '\n')
+    return filtered_lines.getvalue()
+
+def import_plugins(directory):
+    modules = {}
+    for path in glob.glob(os.path.join(directory, '*.py')): 
+        name, ext = os.path.splitext(os.path.basename(path))
+        if not name.startswith("_"):
+            modules[name] = imp.load_source(name, path)
+    return modules
+
 

library/apt

@@ -1,4 +1,6 @@
 #!/usr/bin/python -tt
+# -*- coding: utf-8 -*-
+
 # (c) 2012, Flowroute LLC
 # Written by Matthew Williams <matthew@flowroute.com>
 # Based on yum module written by Seth Vidal <skvidal at fedoraproject.org>
@@ -17,36 +19,18 @@
 # along with this software.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import os
-import sys
-import shlex
-import subprocess
 import traceback
+# added to stave off future warnings about apt api
+import warnings
+warnings.filterwarnings('ignore', "apt API not stable yet", FutureWarning)
 
+# APT related constants
 APT_PATH = "/usr/bin/apt-get"
-APT = "DEBIAN_PRIORITY=critical %s" % APT_PATH
-
-def exit_json(rc=0, **kwargs):
-    print json.dumps(kwargs)
-    sys.exit(rc)
-
-def fail_json(**kwargs):
-    kwargs['failed'] = True
-    exit_json(rc=1, **kwargs)
-
-try:
-    import apt, apt_pkg
-except ImportError:
-    fail_json(msg="could not import apt, please install the python-apt package on this host")
+APT = "DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical %s" % APT_PATH
 
 def run_apt(command):
     try:
-        cmd = subprocess.Popen(command, shell=True, 
-            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        cmd = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
         out, err = cmd.communicate()
     except (OSError, IOError), e:
         rc = 1
@@ -67,105 +51,134 @@ def package_split(pkgspec):
     else:
         return parts[0], None
 
-def package_status(pkgname, version, cache):
+def package_status(m, pkgname, version, cache):
     try:
         pkg = cache[pkgname]
     except KeyError:
-        fail_json(msg="No package matching '%s' is available" % pkgname)
+        m.fail_json(msg="No package matching '%s' is available" % pkgname)
     if version:
-        return pkg.is_installed and pkg.installed.version == version, False
+        try :
+            return pkg.is_installed and pkg.installed.version == version, False
+        except AttributeError:
+            #assume older version of python-apt is installed
+            return pkg.isInstalled and pkg.installedVersion == version, False
     else:
-        return pkg.is_installed, pkg.is_upgradable
+        try :
+            return pkg.is_installed, pkg.is_upgradable
+        except AttributeError:
+            #assume older version of python-apt is installed
+            return pkg.isInstalled, pkg.isUpgradable
 
-def install(pkgspec, cache, upgrade=False, default_release=None):
-    name, version = package_split(pkgspec)
-    installed, upgradable = package_status(name, version, cache)
-    if not installed or (upgrade and upgradable):
-        cmd = "%s -q -y install '%s'" % (APT, pkgspec)
+def install(m, pkgspec, cache, upgrade=False, default_release=None, install_recommends=True, force=False):
+    packages = ""
+    for package in pkgspec:
+        name, version = package_split(package)
+        installed, upgradable = package_status(m, name, version, cache)
+        if not installed or (upgrade and upgradable):
+            packages += "'%s' " % package
+
+    if len(packages) != 0:
+        if force:
+            force_yes = '--force-yes'
+        else:
+            force_yes = ''
+
+        cmd = "%s --option Dpkg::Options::=--force-confold -q -y %s install %s" % (APT, force_yes,packages)
         if default_release:
             cmd += " -t '%s'" % (default_release,)
+        if not install_recommends:
+            cmd += " --no-install-recommends"
+
         rc, out, err = run_apt(cmd)
         if rc:
-            fail_json(msg="'apt-get install %s' failed: %s" % (pkgspec, err))
-        return True
+            m.fail_json(msg="'apt-get install %s' failed: %s" % (packages, err))
+        else:
+            m.exit_json(changed=True)
     else:
-        return False
+        m.exit_json(changed=False)
 
-def remove(pkgspec, cache, purge=False):
-    name, version = package_split(pkgspec)
-    installed, upgradable = package_status(name, version, cache)
-    if not installed:
-        return False
+def remove(m, pkgspec, cache, purge=False):
+    packages = ""
+    for package in pkgspec:
+        name, version = package_split(package)
+        installed, upgradable = package_status(m, name, version, cache)
+        if installed:
+            packages += "'%s' " % package
+
+    if len(packages) == 0:
+        m.exit_json(changed=False)
     else:
         purge = '--purge' if purge else ''
-        cmd = "%s -q -y %s remove '%s'" % (APT, purge, name)
+        cmd = "%s -q -y %s remove %s" % (APT, purge,packages)
         rc, out, err = run_apt(cmd)
         if rc:
-            fail_json(msg="'apt-get remove %s' failed: %s" % (name, err))
-        return True
-    
-
-# ===========================================
-
-if not os.path.exists(APT_PATH):
-   fail_json(msg="Cannot find apt-get")
-
-argfile = sys.argv[1]
-args    = open(argfile, 'r').read()
-items   = shlex.split(args)
-
-if not len(items):
-    fail_json(msg='the module requires arguments -a')
-    sys.exit(1)
-
-params = {}
-for x in items:
-    (k, v) = x.split("=", 1)
-    params[k] = v
-
-state            = params.get('state', 'installed')
-package          = params.get('pkg', params.get('package', params.get('name', None)))
-update_cache     = params.get('update-cache', 'no')
-purge            = params.get('purge', 'no')
-default_release  = params.get('default-release', None)
-
-if state not in ['installed', 'latest', 'removed']:
-    fail_json(msg='invalid state')
-
-if update_cache not in ['yes', 'no']:
-    fail_json(msg='invalid value for update_cache (requires yes or no -- default is no')
-
-if purge not in ['yes', 'no']:
-    fail_json(msg='invalid value for purge (requires yes or no -- default is no)')
-
-if package is None and update_cache != 'yes':
-    fail_json(msg='pkg=name and/or update-cache=yes is required')
-
-cache = apt.Cache()
-if default_release:
-    apt_pkg.config['APT::Default-Release'] = default_release
-    # reopen cache w/ modified config
-    cache.open()
-
-if update_cache == 'yes':
-    cache.update()
-    cache.open()
-    if package == None:
-        exit_json(changed=False)
-
-if package.count('=') > 1:
-    fail_json(msg='invalid package spec')
-
-if state == 'latest':
-    if '=' in package:
-        fail_json(msg='version number inconsistent with state=latest')
-    changed = install(package, cache, upgrade=True,
-                      default_release=default_release)
-elif state == 'installed':
-    changed = install(package, cache, default_release=default_release)
-elif state == 'removed':
-    changed = remove(package, cache, purge == 'yes')
-
-exit_json(changed=changed)
+            m.fail_json(msg="'apt-get remove %s' failed: %s" % (packages, err))
+        m.exit_json(changed=True)
 
 
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            state = dict(default='installed', choices=['installed', 'latest', 'removed', 'absent', 'present']),
+            update_cache = dict(default='no', choices=['yes', 'no'], aliases=['update-cache']),
+            purge = dict(default='no', choices=['yes', 'no']),
+            package = dict(default=None, aliases=['pkg', 'name']),
+            default_release = dict(default=None, aliases=['default-release']),
+            install_recommends = dict(default='yes', aliases=['install-recommends'], choices=['yes', 'no']),
+            force = dict(default='no', choices=['yes', 'no'])
+        )
+    )
+
+    try:
+        import apt
+        import apt_pkg
+    except:
+        module.fail_json(msg="Could not import python modules: apt, apt_pkg. Please install python-apt package.")
+
+    if not os.path.exists(APT_PATH):
+        module.fail_json(msg="Cannot find apt-get")
+
+    p = module.params
+    if p['package'] is None and p['update_cache'] != 'yes':
+        module.fail_json(msg='pkg=name and/or update_cache=yes is required')
+
+    install_recommends = module.boolean(p['install_recommends'])
+
+    cache = apt.Cache()
+    if p['default_release']:
+        apt_pkg.config['APT::Default-Release'] = p['default_release']
+        # reopen cache w/ modified config
+        cache.open(progress=None)
+
+    if module.boolean(p['update_cache']):
+        cache.update()
+        cache.open(progress=None)
+        if p['package'] == None:
+            module.exit_json(changed=False)
+
+    force_yes = module.boolean(p['force'])
+
+    packages = p['package'].split(',')
+    latest = p['state'] == 'latest'
+    for package in packages:
+        if package.count('=') > 1:
+            module.fail_json(msg="invalid package spec: %s" % package)
+        if latest and '=' in package:
+            module.fail_json(msg='version number inconsistent with state=latest: %s' % package)
+
+    if p['state'] == 'latest':
+        install(module, packages, cache, upgrade=True,
+                default_release=p['default_release'],
+                install_recommends=install_recommends,
+                force=force_yes)
+    elif p['state'] in [ 'installed', 'present' ]:
+        install(module, packages, cache, default_release=p['default_release'],
+                  install_recommends=install_recommends,force=force_yes)
+    elif p['state'] in [ 'removed', 'absent' ]:
+        remove(module, packages, cache, purge = module.boolean(p['purge']))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+

library/apt_repository

@@ -0,0 +1,82 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Matt Wright <matt@nobien.net>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+# Example:
+# - name: add nginx repo
+#   action: apt_repository repo=ppa:nginx/stable state=present
+#
+
+import platform
+
+APT = "/usr/bin/apt-get"
+ADD_APT_REPO = 'add-apt-repository'
+
+def _run(cmd):
+    if platform.dist()[0] == 'debian' or float(platform.dist()[1]) >= 11.10:
+        cmd = cmd + ' -y'
+    # returns (rc, stdout, stderr) from shell command
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, shell=True)
+    stdout, stderr = process.communicate()
+    return (process.returncode, stdout, stderr)
+
+
+def main():
+    add_apt_repository = None
+
+    arg_spec = dict(
+        repo=dict(required=True),
+        state=dict(default='present', choices=['present', 'absent'])
+    )
+
+    module = AnsibleModule(argument_spec=arg_spec)
+
+    add_apt_repository = module.get_bin_path(ADD_APT_REPO, True)
+
+    repo = module.params['repo']
+    state = module.params['state']
+
+    rc, out, err = _run('%s "%s" --remove' % (add_apt_repository, repo))
+    existed = 'Error' not in out
+
+    if state == 'absent':
+        if not existed:
+            module.exit_json(changed=False, repo=repo, state=state)
+        else:
+            module.exit_json(changed=True, repo=repo, state=state)
+
+    cmd = '%s "%s"' % (add_apt_repository, repo)
+
+    rc, out, err = _run(cmd)
+
+    changed = rc == 0 and not existed
+
+    if rc != 0:
+        module.fail_json(msg=err)
+
+    if changed:
+        rc, out, err = _run('%s update' % APT)
+
+    module.exit_json(changed=changed, repo=repo, state=state)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()

library/assemble

@@ -0,0 +1,91 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import os.path
+import shutil
+import tempfile
+
+# ===========================================
+# Support methods
+
+def assemble_from_fragments(path):
+    ''' assemble a file from a directory of fragments '''
+    assembled = []
+    for f in sorted(os.listdir(path)):
+        fragment = "%s/%s" % (path, f)
+        if os.path.isfile(fragment):
+            assembled.append(file(fragment).read())
+    return "".join(assembled)
+
+def write_temp_file(data):
+    fd, path = tempfile.mkstemp()
+    os.write(fd, data)
+    os.close(fd)
+    return path
+
+# ==============================================================
+# main
+
+def main():
+
+    module = AnsibleModule(
+        argument_spec = dict(
+            src = dict(required=True),
+            dest = dict(required=True),
+            backup=dict(default=False, choices=BOOLEANS),
+        )
+    )
+
+    changed=False
+    pathmd5 = None
+    destmd5 = None
+    src = os.path.expanduser(module.params['src'])
+    dest = os.path.expanduser(module.params['dest'])
+    backup = module.boolean(module.params.get('backup', False))
+
+    if not os.path.exists(src):
+        module.fail_json(msg="Source (%s) does not exist" % src)
+
+    if not os.path.isdir(src):
+        module.fail_json(msg="Source (%s) is not a directory" % src)
+
+    path = write_temp_file(assemble_from_fragments(src))
+    pathmd5 = module.md5(path)
+
+    if os.path.exists(dest):
+        destmd5 = module.md5(dest)
+
+    if pathmd5 != destmd5:
+        if backup and destmd5 is not None:
+            module.backup_local(dest)
+        shutil.copy(path, dest)
+        changed = True
+
+
+    # Mission complete
+    module.exit_json(src=src, dest=dest, md5sum=destmd5,
+        changed=changed, msg="OK",
+        daisychain="file", daisychain_args=module.params)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()

library/async_status

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -18,91 +19,50 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import shlex
-import os
-import subprocess
-import sys
 import datetime
 import traceback
 
-# ===========================================
+def main():
 
-# FIXME: better error handling
+    module = AnsibleModule(argument_spec=dict(
+        jid=dict(required=True),
+        mode=dict(default='status', choices=['status','cleanup']),
+    ))
 
-argsfile = sys.argv[1]
-items = shlex.split(file(argsfile).read())
+    mode = module.params['mode']
+    jid  = module.params['jid']
 
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
+    # setup logging directory
+    logdir = os.path.expanduser("~/.ansible_async")
+    log_path = os.path.join(logdir, jid)
 
-mode = params.get('mode', 'status')
-jid  = params.get('jid', None)
+    if not os.path.exists(log_path):
+        module.fail_json(msg="could not find job", ansible_job_id=jid)
 
-# ===========================================
+    if mode == 'cleanup':
+        os.unlink(log_path)
+        module.exit_json(ansible_job_id=jid, erased=log_path)
 
-if jid is None:
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "jid=INTEGER is required"
-    })
-    sys.exit(1)
+    # NOT in cleanup mode, assume regular status mode
+    # no remote kill mode currently exists, but probably should
+    # consider log_path + ".pid" file and also unlink that above
 
+    data = file(log_path).read()
+    try:
+        data = json.loads(data)
+    except Exception, e:
+        if data == '':
+            # file not written yet?  That means it is running
+            module.exit_json(results_file=log_path, ansible_job_id=jid, started=1)
+        else:
+            module_fail_json(ansible_job_id=jid, results_file=log_path,
+                msg="Could not parse job output: %s" % data)
 
-# setup logging directory
-logdir = os.path.expanduser("~/.ansible_async")
-log_path = os.path.join(logdir, jid)
-
-if not os.path.exists(log_path):
-    print json.dumps({
-        "failed" : 1,
-        "msg" : "could not find job",
-        "ansible_job_id" : jid
-    })
-    sys.exit(1)
-
-if mode == 'cleanup':
-    os.unlink(log_path)
-    print json.dumps({
-        "ansible_job_id" : jid,
-        "erased" : log_path
-    })
-    sys.exit(0)
-
-# NOT in cleanup mode, assume regular status mode
-# no remote kill mode currently exists, but probably should
-# consider log_path + ".pid" file and also unlink that above
-
-data = file(log_path).read()
-try:
-   data = json.loads(data)
-except Exception, e:
-   if data == '':
-       # file not written yet?  That means it is running
-       print json.dumps({
-           "results_file" : log_path,
-           "ansible_job_id" : jid,
-           "started" : 1,
-       })
-   else:
-       print json.dumps({
-           "failed" : True,
-           "ansible_job_id" : jid,
-           "results_file" : log_path,
-           "msg" : "Could not parse job output: %s" % data,
-       })
-   sys.exit(0)
-
-if not data.has_key("started"):
-   data['finished'] = 1
-data['ansible_job_id'] = jid
-
-print json.dumps(data)
-sys.exit(0)
-
+    if not 'started' in data:
+        data['finished'] = 1
+        data['ansible_job_id'] = jid
+    module.exit_json(**data)
 
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/async_wrapper

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -30,6 +31,7 @@ import datetime
 import traceback
 import signal
 import time
+import syslog
 
 def daemonize_self():
     # daemonizing code: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
@@ -61,7 +63,7 @@ def daemonize_self():
     dev_null = file('/dev/null','rw')
     os.dup2(dev_null.fileno(), sys.stdin.fileno())
     os.dup2(dev_null.fileno(), sys.stdout.fileno())
-    os.dup2(dev_null.fileno(), sys.stderr.fileno()) 
+    os.dup2(dev_null.fileno(), sys.stderr.fileno())
 
 if len(sys.argv) < 3:
     print json.dumps({
@@ -76,6 +78,9 @@ wrapped_module = sys.argv[3]
 argsfile = sys.argv[4]
 cmd = "%s %s" % (wrapped_module, argsfile)
 
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % " ".join(sys.argv[1:]))
+
 # setup logging directory
 logdir = os.path.expanduser("~/.ansible_async")
 log_path = os.path.join(logdir, jid)
@@ -96,11 +101,11 @@ def _run_command(wrapped_cmd, jid, log_path):
     logfile.close()
     logfile = open(log_path, "w")
     result = {}
-  
+
     outdata = ''
     try:
         cmd = shlex.split(wrapped_cmd)
-        script = subprocess.Popen(cmd, shell=False, 
+        script = subprocess.Popen(cmd, shell=False,
             stdin=None, stdout=logfile, stderr=logfile)
         script.communicate()
         outdata = file(log_path).read()
@@ -120,7 +125,7 @@ def _run_command(wrapped_cmd, jid, log_path):
             "cmd" : wrapped_cmd,
             "data" : outdata, # temporary debug only
             "msg" : traceback.format_exc()
-        }   
+        }
         result['ansible_job_id'] = jid
         logfile.write(json.dumps(result))
     logfile.close()

library/authorized_key

@@ -0,0 +1,136 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Ansible module to add authorized_keys for ssh logins.
+(c) 2012, Brad Olson <brado@movedbylight.com>
+
+This file is part of Ansible
+
+Ansible is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+Ansible is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+# Makes sure the public key line is present or absent in the user's .ssh/authorized_keys.
+#
+# Arguments
+# =========
+#    user = username
+#    key = line to add to authorized_keys for user
+#    state = absent|present (default: present)
+#
+# see example in examples/playbooks
+
+import sys
+import os
+import pwd
+import os.path
+
+def keyfile(user, write=False):
+    """
+    Calculate name of authorized keys file, optionally creating the
+    directories and file, properly setting permissions.
+
+    :param str user: name of user in passwd file
+    :param bool write: if True, write changes to authorized_keys file (creating directories if needed)
+    :return: full path string to authorized_keys for user
+    """
+
+    user_entry = pwd.getpwnam(user)
+    homedir    = user_entry.pw_dir
+    sshdir     = os.path.join(homedir, ".ssh")
+    keysfile   = os.path.join(sshdir, "authorized_keys")
+
+    if not write:
+        return keysfile
+
+    uid = user_entry.pw_uid
+    gid = user_entry.pw_gid
+
+    if not os.path.exists(sshdir):
+        os.mkdir(sshdir, 0700)
+    os.chown(sshdir, uid, gid)
+    os.chmod(sshdir, 0700)
+
+    if not os.path.exists( keysfile):
+        try:
+            f = open(keysfile, "w") #touches file so we can set ownership and perms
+        finally:
+            f.close()
+
+    os.chown(keysfile, uid, gid)
+    os.chmod(keysfile, 0600)
+    return keysfile
+
+def readkeys(filename):
+
+    if not os.path.isfile(filename):
+        return []
+    f = open(filename)
+    keys = [line.rstrip() for line in f.readlines()]
+    f.close()
+    return keys
+
+def writekeys( filename, keys):
+
+    f = open(filename,"w")
+    f.writelines( (key + "\n" for key in keys) )
+    f.close()
+
+def enforce_state(module, params):
+    """
+    Add or remove key.
+    """
+
+    user  = params["user"]
+    key   = params["key"]
+    state = params.get("state", "present")
+
+    # check current state -- just get the filename, don't create file
+    params["keyfile"] = keyfile(user, write=False)
+    keys = readkeys(params["keyfile"])
+    present = key in keys
+
+    # handle idempotent state=present
+    if state=="present":
+        if present:
+            module.exit_json(changed=False)
+        keys.append(key)
+        writekeys(keyfile(user,write=True), keys)
+
+    elif state=="absent":
+        if not present:
+            module.exit_json(changed=False)
+        keys.remove(key)
+        writekeys(keyfile(user,write=True), keys)
+
+    params['changed'] = True
+    return params
+
+def main():
+
+    module = AnsibleModule(
+        argument_spec = dict(
+           user  = dict(required=True),
+           key   = dict(required=True),
+           state = dict(default='present', choices=['absent','present'])
+        )
+    )
+
+    params = module.params
+    results = enforce_state(module, module.params)
+    module.exit_json(**results)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/command

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -16,12 +17,6 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
 
 import subprocess
 import sys
@@ -30,75 +25,105 @@ import traceback
 import shlex
 import os
 
-argfile = sys.argv[1]
-args = open(argfile, 'r').read()
+def main():
 
-shell = False
+    # the command module is the one ansible module that does not take key=value args
+    # hence don't copy this one if you are looking to build others!
+    module = CommandModule(argument_spec=dict())
 
-if args.find("#USE_SHELL") != -1:
-   args = args.replace("#USE_SHELL", "")
-   shell = True
+    shell = module.params['shell']
+    chdir = module.params['chdir']
+    args  = module.params['args']
 
-check_args = shlex.split(args)
-for x in check_args:
-   if x.startswith("creates="):
-       # do not run the command if the line contains creates=filename
-       # and the filename already exists.  This allows idempotence 
-       # of command executions.
-       (k,v) = x.split("=",1)
-       if os.path.exists(v):
-           print json.dumps({
-               "cmd"     : args,
-               "stdout"  : "skipped, since %s exists" % v,
-               "skipped" : True,
-               "changed" : False,
-               "stderr"  : "",
-               "rc"      : 0,
-           })
-           sys.exit(0)
-       args = args.replace(x,'')
-       
+    if args.strip() == '':
+        module.fail_json(msg="no command given")
 
-if not shell:
-    args = shlex.split(args)
+    if chdir:
+        os.chdir(os.path.expanduser(chdir))
 
-startd = datetime.datetime.now()
+    if not shell:
+        args = shlex.split(args)
+    startd = datetime.datetime.now()
 
-try:
-    cmd = subprocess.Popen(args, shell=shell, 
-        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    out, err = cmd.communicate()
-except (OSError, IOError), e:
-    print json.dumps({
-        "cmd"    : args,
-        "failed" : 1,
-        "msg"    : str(e),
-        })
-    sys.exit(1)
-except:
-    print json.dumps({
-        "failed" : 1,
-        "msg" : traceback.format_exc()
-    })   
-    sys.exit(1)
+    try:
+        cmd = subprocess.Popen(args, shell=shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        out, err = cmd.communicate()
+    except (OSError, IOError), e:
+        module.fail_json(cmd=args, msg=str(e))
+    except:
+        module.fail_json(msg=traceback.format_exc())
 
-endd = datetime.datetime.now()
-delta = endd - startd
+    endd = datetime.datetime.now()
+    delta = endd - startd
 
-if out is None:
-   out = ''
-if err is None:
-   err = ''
+    if out is None:
+        out = ''
+    if err is None:
+        err = ''
 
-result = {
-   "cmd"     : args,
-   "stdout"  : out.strip(),
-   "stderr"  : err.strip(),
-   "rc"      : cmd.returncode,
-   "start"   : str(startd),
-   "end"     : str(endd),
-   "delta"   : str(delta),
-   "changed" : True
-}
+    module.exit_json(
+        cmd     = args,
+        stdout  = out.strip(),
+        stderr  = err.strip(),
+        rc      = cmd.returncode,
+        start   = str(startd),
+        end     = str(endd),
+        delta   = str(delta),
+        changed = True
+    )
 
-print json.dumps(result)
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+# only the command module should ever need to do this
+# everything else should be simple key=value
+
+class CommandModule(AnsibleModule):
+
+    def _handle_aliases(self):
+        pass
+
+    def _check_invalid_arguments(self):
+        pass
+
+    def _load_params(self):
+        ''' read the input and return a dictionary and the arguments string '''
+        args = MODULE_ARGS
+        items   = shlex.split(args)
+        params = {}
+        params['chdir'] = None
+        params['shell'] = False
+        if args.find("#USE_SHELL") != -1:
+            args = args.replace("#USE_SHELL", "")
+            params['shell'] = True
+
+        check_args = shlex.split(args)
+        for x in check_args:
+            if x.startswith("creates="):
+                # do not run the command if the line contains creates=filename
+                # and the filename already exists.  This allows idempotence
+                # of command executions.
+                (k,v) = x.split("=",1)
+                if os.path.exists(v):
+                    self.exit_json(
+                        cmd=args,
+                        stdout="skipped, since %s exists" % v,
+                        skipped=True,
+                        changed=False,
+                        stderr=False,
+                        rc=0
+                    )
+                args = args.replace(x,'')
+            elif x.startswith("chdir="):
+                (k,v) = x.split("=", 1)
+                v = os.path.expanduser(v)
+                if not (os.path.exists(v) and os.path.isdir(v)):
+                    self.fail_json(msg="cannot change to directory '%s': path does not exist" % v)
+                elif v[0] != '/':
+                    self.fail_json(msg="the path for 'chdir' argument must be fully qualified")
+                params['chdir'] = v
+                args = args.replace(x, '')
+        params['args'] = args
+        return (params, args)
+
+main()

library/copy

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -16,54 +17,73 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
 
-import sys
 import os
-import shlex
+import shutil
 
-# ===========================================
-# convert arguments of form a=b c=d
-# to a dictionary
-# FIXME: make more idiomatic
+def main():
 
-if len(sys.argv) == 1:
-    sys.exit(1)
-argfile = sys.argv[1]
-if not os.path.exists(argfile):
-    sys.exit(1)
-items = shlex.split(open(argfile, 'r').read())
+    module = AnsibleModule(
+        # not checking because of daisy chain to file module
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            src=dict(required=True),
+            dest=dict(required=True),
+            backup=dict(default=False, choices=BOOLEANS),
+        )
+    )
 
+    src  = os.path.expanduser(module.params['src'])
+    dest = os.path.expanduser(module.params['dest'])
+    backup = module.boolean(module.params.get('backup', False))
+    
+    if not os.path.exists(src):
+        module.fail_json(msg="Source %s failed to transfer" % (src))
+    if not os.access(src, os.R_OK):
+        module.fail_json(msg="Source %s not readable" % (src))
 
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
+    md5sum_src = module.md5(src)
+    md5sum_dest = None
 
-src  = params['src']
-dest = params['dest']
-if src:
-    src = os.path.expanduser(src)
-if dest:
-    dest = os.path.expanduser(dest)
- 
-# raise an error if there is no src file
-if not os.path.exists(src):
-    print "failed=1 msg='Source %s failed to transfer'" % src
-    sys.exit(1)
+    if os.path.exists(dest):
+        if not os.access(dest, os.W_OK):
+            module.fail_json(msg="Destination %s not writable" % (dest))
+        if not os.access(dest, os.R_OK):
+            module.fail_json(msg="Destination %s not readable" % (dest))
+        # Allow dest to be directory without compromising md5 check
+        if (os.path.isdir(dest)):
+            module.fail_json(msg="Destination %s cannot be a directory" % (dest))
+        md5sum_dest = module.md5(dest)
+    else:
+        if not os.access(os.path.dirname(dest), os.W_OK):
+            module.fail_json(msg="Destination %s not writable" % (os.path.dirname(dest)))
 
-md5sum = None
-changed = False
-if os.path.exists(dest):
-    md5sum = os.popen("md5sum %s" % dest).read().split()[0]
-
-md5sum2 = os.popen("md5sum %s" % src).read().split()[0]
-
-if md5sum != md5sum2:
-    os.system("cp %s %s" % (src, dest))
-    changed = True
-
-# mission accomplished
-print "md5sum=%s changed=%s" % (md5sum2, changed)
+    backup_file = None
+    if md5sum_src != md5sum_dest:
+        try:
+            if backup:
+                if os.path.exists(dest):
+                    backup_file = module.backup_local(dest)
+            shutil.copyfile(src, dest)
+        except shutil.Error:
+            module.fail_json(msg="failed to copy: %s and %s are the same" % (src, dest))
+        except IOError:
+            module.fail_json(msg="failed to copy: %s to %s" % (src, dest))
+        changed = True
+    else:
+        changed = False
 
+    res_args = {
+        'dest':dest,
+        'src':src,
+        'md5sum':md5sum_src,
+        'changed':changed
+    }
+    if backup_file:
+        res_args['backup_file'] = backup_file
+        
+    module.exit_json(**res_args)
 
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/easy_install

@@ -0,0 +1,93 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Matt Wright <matt@nobien.net>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+def _ensure_virtualenv(env, virtualenv):
+    if os.path.exists(os.path.join(env, 'bin', 'activate')):
+        return 0, '', ''
+    else:
+        return _run('%s %s' % (virtualenv, env))
+
+
+def _is_package_installed(name, easy_install):
+    cmd = '%s --dry-run %s' % (easy_install, name)
+    rc, status_stdout, status_stderr = _run(cmd)
+    return not ('Reading' in status_stdout or 'Downloading' in status_stdout)
+
+
+def _run(cmd):
+    # returns (rc, stdout, stderr) from shell command
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, shell=True)
+    stdout, stderr = process.communicate()
+    return (process.returncode, stdout, stderr)
+
+
+def main():
+    arg_spec = dict(
+        name=dict(required=True),
+        virtualenv=dict(default=None, required=False)
+    )
+
+    module = AnsibleModule(argument_spec=arg_spec)
+
+    name = module.params['name']
+    env = module.params['virtualenv']
+    easy_install = module.get_bin_path('easy_install', True, ['%s/bin' % env])
+
+    rc = 0
+    err = ''
+    out = ''
+
+    if env:
+        virtualenv = module.get_bin_path('virtualenv', True)
+        if virtualenv is None:
+            module.fail_json(msg='virtualenv is not installed')
+
+        rc_venv, out_venv, err_venv = _ensure_virtualenv(env, virtualenv)
+
+        rc += rc_venv
+        out += out_venv
+        err += err_venv
+
+    cmd = None
+    changed = False
+    installed = _is_package_installed(name, easy_install)
+
+    if not installed:
+        cmd = '%s %s' % (easy_install, name)
+        rc_pip, out_pip, err_pip = _run(cmd)
+
+        rc += rc_pip
+        out += out_pip
+        err += err_pip
+
+        changed = True
+
+    if rc != 0:
+        module.fail_json(msg=err, cmd=cmd)
+
+    module.exit_json(changed=changed, binary=easy_install,
+                     name=name, virtualenv=env)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()

library/facter

@@ -1,4 +1,5 @@
-#!/bin/bash
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -22,4 +23,28 @@
 #    facter
 #    ruby-json
 
-/usr/bin/facter --json
+import subprocess
+
+def get_facter_data():
+    p = subprocess.Popen(["/usr/bin/env", "facter", "--json"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return rc, out, err
+
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict()
+    )
+
+    rc, out, err = get_facter_data()
+    if rc != 0:
+        module.fail_json(msg=err)
+    else:
+        module.exit_json(**json.loads(out))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+

library/failtest

@@ -1,34 +0,0 @@
-#!/usr/bin/python
-
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-import sys
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
-
-print >>sys.stderr, "THIS IS A TEST FAILURE"
-
-print json.dumps({ 
-    "failed" : True, 
-    "msg"    : "this module always fails" 
-})
-
-

library/fetch

@@ -1,24 +1 @@
-#!/usr/bin/python
-
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-### THIS FILE IS FOR REFERENCE OR FUTURE USE ###
-
-# See lib/ansible/runner.py for implementation of the fetch functionality #
-
+# this is a virtual module that is entirely implemented server side

library/file

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -17,14 +18,6 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import os
-import sys
-import shlex
-import subprocess
 import shutil
 import stat
 import grp
@@ -35,42 +28,34 @@ try:
 except ImportError:
     HAVE_SELINUX=False
 
-def debug(msg):
-    # ansible ignores stderr, so it's safe to use for debug
-    # print >>sys.stderr, msg
-    pass
-
-def exit_json(rc=0, **kwargs):
-    if 'path' in kwargs:
-        debug("adding path info")
-        add_path_info(kwargs)
-    print json.dumps(kwargs)
-    sys.exit(rc)
-
-def fail_json(**kwargs):
-    kwargs['failed'] = True
-    exit_json(rc=1, **kwargs)
-
 def add_path_info(kwargs):
     path = kwargs['path']
     if os.path.exists(path):
         (user, group) = user_and_group(path)
-        kwargs['user']  = user
+        kwargs['owner']  = user
         kwargs['group'] = group
         st = os.stat(path)
         kwargs['mode']  = oct(stat.S_IMODE(st[stat.ST_MODE]))
         # secontext not yet supported
         if os.path.islink(path):
             kwargs['state'] = 'link'
-        elif os.path.isfile(path):
-            kwargs['state'] = 'file'
-        else:
+        elif os.path.isdir(path):
             kwargs['state'] = 'directory'
+        else:
+            kwargs['state'] = 'file'
         if HAVE_SELINUX and selinux_enabled():
             kwargs['secontext'] = ':'.join(selinux_context(path))
     else:
         kwargs['state'] = 'absent'
-    return kwargs 
+    return kwargs
+
+def module_exit_json(**kwargs):
+    add_path_info(kwargs)
+    module.exit_json(**kwargs)
+
+def module_fail_json(**kwargs):
+    add_path_info(kwargs)
+    module.fail_json(**kwargs)
 
 # Detect whether using selinux that is MLS-aware.
 # While this means you can set the level/range with
@@ -81,20 +66,16 @@ def selinux_mls_enabled():
     if not HAVE_SELINUX:
         return False
     if selinux.is_selinux_mls_enabled() == 1:
-        debug('selinux mls is enabled')
         return True
     else:
-        debug('selinux mls is disabled')
         return False
 
 def selinux_enabled():
     if not HAVE_SELINUX:
         return False
     if selinux.is_selinux_enabled() == 1:
-        debug('selinux is enabled')
         return True
     else:
-        debug('selinux is disabled')
         return False
 
 # Determine whether we need a placeholder for selevel/mls
@@ -112,13 +93,10 @@ def selinux_default_context(path, mode=0):
     try:
         ret = selinux.matchpathcon(path, mode)
     except OSError:
-        debug("no default context available")
         return context
     if ret[0] == -1:
-        debug("no default context available")
         return context
     context = ret[1].split(':')
-    debug("got default secontext=%s" % ret[1])
     return context
 
 def selinux_context(path):
@@ -128,83 +106,27 @@ def selinux_context(path):
     try:
         ret = selinux.lgetfilecon(path)
     except:
-        fail_json(path=path, msg='failed to retrieve selinux context')
+        module_fail_json(path=path, msg='failed to retrieve selinux context')
     if ret[0] == -1:
         return context
     context = ret[1].split(':')
-    debug("got current secontext=%s" % ret[1])
     return context
 
-# ===========================================
-
-argfile = sys.argv[1]
-args    = open(argfile, 'r').read()
-items   = shlex.split(args)
-
-if not len(items):
-    fail_json(msg='the module requires arguments -a')
-    sys.exit(1)
-
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
-
-state     = params.get('state','file')
-path      = params.get('path', params.get('dest', params.get('name', None)))
-if path:
-   path = os.path.expanduser(path)
-src       = params.get('src', None)
-if src:
-   src = os.path.expanduser(src)
-dest      = params.get('dest', None)
-mode      = params.get('mode', None)
-owner     = params.get('owner', None)
-group     = params.get('group', None)
-
-# presently unused, we always use -R (FIXME?)
-recurse   = params.get('recurse', 'false')
-
-# selinux related options
-seuser    = params.get('seuser', None)
-serole    = params.get('serole', None)
-setype    = params.get('setype', None)
-selevel   = params.get('serange', 's0')
-context   = params.get('context', None)
-secontext = [seuser, serole, setype]
-if selinux_mls_enabled():
-    secontext.append(selevel)
-
-if context is not None:
-    if context != 'default':
-        fail_json(msg='invalid context: %s' % context)
-    if seuser is not None or serole is not None or setype is not None:
-        fail_json(msg='cannot define context=default and seuser, serole or setype')
-    secontext = selinux_default_context(path)
-
-if state not in [ 'file', 'directory', 'link', 'absent']:
-    fail_json(msg='invalid state: %s' % state)
-
-if state == 'link' and (src is None or dest is None):
-    fail_json(msg='src and dest are required for "link" state')
-elif path is None:
-    fail_json(msg='path is required')
-
-changed = False
-
 # ===========================================
 # support functions
 
-def md5sum(filename):
-    return os.popen("/usr/bin/md5sum %s" % f).read()
-
 def user_and_group(filename):
     st = os.stat(filename)
     uid = st.st_uid
     gid = st.st_gid
-    user = pwd.getpwuid(uid)[0]
-    group = grp.getgrgid(gid)[0]
-    debug("got user=%s and group=%s" % (user, group))
+    try:
+        user = pwd.getpwuid(uid)[0]
+    except KeyError:
+        user = str(uid)
+    try:
+        group = grp.getgrgid(gid)[0]
+    except KeyError:
+        group = str(gid)
     return (user, group)
 
 def set_context_if_different(path, context, changed):
@@ -212,176 +134,226 @@ def set_context_if_different(path, context, changed):
         return changed
     cur_context = selinux_context(path)
     new_context = list(cur_context)
-    debug("current secontext is %s" % ':'.join(cur_context))
     # Iterate over the current context instead of the
     # argument context, which may have selevel.
     for i in range(len(cur_context)):
         if context[i] is not None and context[i] != cur_context[i]:
             new_context[i] = context[i]
-    debug("new secontext is %s" % ':'.join(new_context))
     if cur_context != new_context:
         try:
             rc = selinux.lsetfilecon(path, ':'.join(new_context))
         except OSError:
-            fail_json(path=path, msg='invalid selinux context')
+            module_fail_json(path=path, msg='invalid selinux context')
         if rc != 0:
-            fail_json(path=path, msg='set selinux context failed')
+            module_fail_json(path=path, msg='set selinux context failed')
         changed = True
     return changed
-    
-def set_owner_if_different(path, owner, changed):
-   if owner is None:
-       debug('not tweaking owner')
-       return changed
-   user, group = user_and_group(path)
-   if owner != user:
-       debug('setting owner')
-       rc = os.system("/bin/chown -R %s %s" % (owner, path))
-       if rc != 0:
-           fail_json(path=path, msg='chown failed')
-       return True
 
-   return changed
-    
+def set_owner_if_different(path, owner, changed):
+    if owner is None:
+        return changed
+    user, group = user_and_group(path)
+    if owner != user:
+        try:
+            uid = pwd.getpwnam(owner).pw_uid
+        except KeyError:
+            module_fail_json(path=path, msg='chown failed: failed to look up user %s' % owner)
+        try:
+            os.chown(path, uid, -1)
+        except OSError:
+            module_fail_json(path=path, msg='chown failed')
+        return True
+
+    return changed
+
 def set_group_if_different(path, group, changed):
-   if group is None:
-       debug('not tweaking group')
-       return changed
-   old_user, old_group = user_and_group(path)
-   if old_group != group:
-       debug('setting group')
-       rc = os.system("/bin/chgrp -R %s %s" % (group, path))
-       if rc != 0:
-           fail_json(path=path, msg='chgrp failed')
-       return True
-   return changed
+    if group is None:
+        return changed
+    old_user, old_group = user_and_group(path)
+    if old_group != group:
+        try:
+            gid = grp.getgrnam(group).gr_gid
+        except KeyError:
+            module_fail_json(path=path, msg='chgrp failed: failed to look up group %s' % group)
+        try:
+            os.chown(path, -1, gid)
+        except OSError:
+            module_fail_json(path=path, msg='chgrp failed')
+        return True
+    return changed
 
 def set_mode_if_different(path, mode, changed):
-   if mode is None:
-       debug('not tweaking mode')
-       return changed
-   try:
-       # FIXME: support English modes
-       mode = int(mode, 8)
-   except Exception, e:
-       fail_json(path=path, msg='mode needs to be something octalish', details=str(e))  
+    if mode is None:
+        return changed
+    try:
+        # FIXME: support English modes
+        mode = int(mode, 8)
+    except Exception, e:
+        module_fail_json(path=path, msg='mode needs to be something octalish', details=str(e))
+
+    st = os.stat(path)
+    prev_mode = stat.S_IMODE(st[stat.ST_MODE])
+
+    if prev_mode != mode:
+        # FIXME: comparison against string above will cause this to be executed
+        # every time
+        try:
+            os.chmod(path, mode)
+        except Exception, e:
+            module_fail_json(path=path, msg='chmod failed', details=str(e))
  
-   st = os.stat(path)
-   prev_mode = stat.S_IMODE(st[stat.ST_MODE])
+        st = os.stat(path)
+        new_mode = stat.S_IMODE(st[stat.ST_MODE])
 
-   if prev_mode != mode:
-       # FIXME: comparison against string above will cause this to be executed
-       # every time
-       try:
-           debug('setting mode')
-           os.chmod(path, mode)
-       except Exception, e:
-           fail_json(path=path, msg='chmod failed', details=str(e))
-
-       st = os.stat(path)
-       new_mode = stat.S_IMODE(st[stat.ST_MODE])
-
-       if new_mode != prev_mode:
-           return True
-   return changed
+        if new_mode != prev_mode:
+            return True
+    return changed
 
 
 def rmtree_error(func, path, exc_info):
-   fail_json(path=path, msg='failed to remove directory')
+    module_fail_json(path=path, msg='failed to remove directory')
 
-# ===========================================
-# go...
+def main():
 
-prev_state = 'absent'
-if os.path.exists(path):
-    if os.path.islink(path):
-        prev_state = 'link'
-    elif os.path.isfile(path):
-        prev_state = 'file'
-    else:
-        prev_state = 'directory'
+    # FIXME: pass this around, should not use global
+    global module
 
-if prev_state != 'absent' and state == 'absent':
-    debug('requesting absent')
-    try:
-        if prev_state == 'directory':
-            if os.path.islink(path):
-                os.unlink(path)
-            else:
-                shutil.rmtree(path, ignore_errors=False, onerror=rmtree_error)
+    module = AnsibleModule(
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            state = dict(choices=['file','directory','link','absent'], default='file'),
+            path  = dict(aliases=['dest', 'name'], required=True),
+            src   = dict(),
+            mode  = dict(),
+            owner = dict(),
+            group = dict(),
+            seuser = dict(),
+            serole = dict(),
+            selevel = dict(),
+            setype = dict(),
+        )
+    )
+
+    params = module.params
+    state  = params['state']
+    path   = os.path.expanduser(params['path'])
+    src    = params.get('src', None)
+    if src:
+        src = os.path.expanduser(src)
+
+    mode   = params.get('mode', None)
+    owner  = params.get('owner', None)
+    group  = params.get('group', None)
+
+    # selinux related options
+    seuser    = params.get('seuser', None)
+    serole    = params.get('serole', None)
+    setype    = params.get('setype', None)
+    selevel   = params.get('serange', 's0')
+    secontext = [seuser, serole, setype]
+    if selinux_mls_enabled():
+        secontext.append(selevel)
+
+    default_secontext = selinux_default_context(path)
+    for i in range(len(default_secontext)):
+        if i is not None and secontext[i] == '_default':
+            secontext[i] = default_secontext[i]
+
+    if state == 'link' and (src is None or path is None):
+        module_fail_json(msg='src and dest are required for "link" state')
+    elif path is None:
+        module_fail_json(msg='path is required')
+
+    changed = False
+
+    prev_state = 'absent'
+    if os.path.lexists(path):
+        if os.path.islink(path):
+            prev_state = 'link'
+        elif os.path.isdir(path):
+            prev_state = 'directory'
         else:
-            os.unlink(path)
-    except Exception, e:
-        fail_json(path=path, msg=str(e))
-    exit_json(path=path, changed=True)
-    sys.exit(0)
+            prev_state = 'file'
 
-if prev_state != 'absent' and prev_state != state:
-    fail_json(path=path, msg='refusing to convert between %s and %s' % (prev_state, state))
+    if prev_state != 'absent' and state == 'absent':
+        try:
+            if prev_state == 'directory':
+                if os.path.islink(path):
+                    os.unlink(path)
+                else:
+                    shutil.rmtree(path, ignore_errors=False, onerror=rmtree_error)
+            else:
+                os.unlink(path)
+        except Exception, e:
+            module_fail_json(path=path, msg=str(e))
+        module_exit_json(path=path, changed=True)
 
-if prev_state == 'absent' and state == 'absent':
-    exit_json(path=path, changed=False)
+    if prev_state != 'absent' and prev_state != state:
+        module_fail_json(path=path, msg='refusing to convert between %s and %s' % (prev_state, state))
 
-if state == 'file':
+    if prev_state == 'absent' and state == 'absent':
+        module_exit_json(path=path, changed=False)
 
-    debug('requesting file')
-    if prev_state == 'absent':
-        fail_json(path=path, msg='file does not exist, use copy or template module to create')
+    if state == 'file':
 
-    # set modes owners and context as needed
-    changed = set_context_if_different(path, secontext, changed)
-    changed = set_owner_if_different(path, owner, changed)
-    changed = set_group_if_different(path, group, changed)
-    changed = set_mode_if_different(path, mode, changed)
+        if prev_state == 'absent':
+            module_fail_json(path=path, msg='file does not exist, use copy or template module to create')
 
-    exit_json(path=path, changed=changed)
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
 
-elif state == 'directory':
+        module_exit_json(path=path, changed=changed)
 
-    debug('requesting directory')
-    if prev_state == 'absent':
-        os.makedirs(path)
-        changed = True
- 
-    # set modes owners and context as needed
-    changed = set_context_if_different(path, secontext, changed)
-    changed = set_owner_if_different(path, owner, changed)
-    changed = set_group_if_different(path, group, changed)
-    changed = set_mode_if_different(path, mode, changed)
+    elif state == 'directory':
 
-    exit_json(path=path, changed=changed)
+        if prev_state == 'absent':
+            os.makedirs(path)
+            changed = True
 
-elif state == 'link':
-    
-    if os.path.isabs(src):
-        abs_src = src
-    else:
-        abs_src = os.path.join(os.path.dirname(dest), src)
-    if not os.path.exists(abs_src):
-        fail_json(dest=dest, src=src, msg='src file does not exist')
-   
-    if prev_state == 'absent':
-        os.symlink(src, dest)
-        changed = True
-    elif prev_state == 'link':
-        old_src = os.readlink(dest)
-        if not os.path.isabs(old_src):
-            old_src = os.path.join(os.path.dirname(dest), old_src)
-        if old_src != src:
-            os.unlink(dest)
-            os.symlink(src, dest)
-    else:
-        fail_json(dest=dest, src=src, msg='unexpected position reached')
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
 
-    # set modes owners and context as needed
-    changed = set_context_if_different(dest, secontext, changed)
-    changed = set_owner_if_different(dest, owner, changed)
-    changed = set_group_if_different(dest, group, changed)
-    changed = set_mode_if_different(dest, mode, changed)
+        module_exit_json(path=path, changed=changed)
 
-    exit_json(dest=dest, src=src, changed=changed)
+    elif state == 'link':
 
+        if os.path.isabs(src):
+            abs_src = src
+        else:
+            module.fail_json(msg="absolute paths are required")
+        if not os.path.exists(abs_src):
+            module_fail_json(path=path, src=src, msg='src file does not exist')
 
-fail_json(path=path, msg='unexpected position reached')
-sys.exit(0)
+        if prev_state == 'absent':
+            os.symlink(src, path)
+            changed = True
+        elif prev_state == 'link':
+            old_src = os.readlink(path)
+            if not os.path.isabs(old_src):
+                old_src = os.path.join(os.path.dirname(path), old_src)
+            if old_src != src:
+                os.unlink(path)
+                os.symlink(src, path)
+        else:
+            module_fail_json(dest=path, src=src, msg='unexpected position reached')
+
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
+
+        module.exit_json(dest=path, src=src, changed=changed)
+
+    module_fail_json(path=path, msg='unexpected position reached')
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()
 

library/get_url

@@ -0,0 +1,203 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Jan-Piet Mens <jpmens () gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+# see examples/playbooks/get_url.yml
+
+import shutil
+import datetime
+import tempfile
+
+HAS_URLLIB2=True
+try:
+    import urllib2
+except ImportError:
+    HAS_URLLIB2=False
+HAS_URLPARSE=True
+
+try:
+    import urlparse
+    import socket
+except ImportError:
+    HAS_URLPARSE=False
+
+# ==============================================================
+# url handling
+
+def url_filename(url):
+    fn = os.path.basename(urlparse.urlsplit(url)[2])
+    if fn == '':
+        return 'index.html'
+    return fn
+
+def url_do_get(module, url, dest):
+    """
+    Get url and return request and info
+    Credits: http://stackoverflow.com/questions/7006574/how-to-download-file-from-ftp
+    """
+
+    USERAGENT = 'ansible-httpget'
+    info = dict(url=url)
+    r = None
+    actualdest = None
+
+    if os.path.isdir(dest):
+        urlfilename = url_filename(url)
+        actualdest = "%s/%s" % (dest, urlfilename)
+        module.params['path'] = actualdest
+    else:
+        actualdest = dest
+    info['daisychain_args'] = module.params
+    info['daisychain_args']['state'] = 'file'
+    info['daisychain_args']['dest'] = actualdest
+    info['actualdest'] = actualdest
+
+    request = urllib2.Request(url)
+    request.add_header('User-agent', USERAGENT)
+
+    if os.path.exists(actualdest):
+        t = datetime.datetime.utcfromtimestamp(os.path.getmtime(actualdest))
+        tstamp = t.strftime('%a, %d %b %Y %H:%M:%S +0000')
+        request.add_header('If-Modified-Since', tstamp)
+
+    try:
+        r = urllib2.urlopen(request)
+        info.update(r.info())
+        info.update(dict(msg="OK (%s bytes)" % r.headers.get('Content-Length', 'unknown'), status=200))
+    except urllib2.HTTPError, e:
+        # Must not fail_json() here so caller can handle HTTP 304 unmodified
+        info.update(dict(msg=str(e), status=e.code))
+        return r, info
+    except urllib2.URLError, e:
+        code = getattr(e, 'code', -1)
+        module.fail_json(msg="Request failed: %s" % str(e), status_code=code)
+
+    return r, info
+
+def url_get(module, url, dest):
+    """
+    Download url and store at dest.
+    If dest is a directory, determine filename from url.
+    Return (tempfile, info about the request)
+    """
+
+    req, info = url_do_get(module, url, dest)
+
+    # TODO: should really handle 304, but how? src file could exist (and be newer) but empty
+    if info['status'] == 304:
+        module.exit_json(url=url, dest=info.get('actualdest', dest), changed=False, msg=info.get('msg', ''))
+
+    # create a temporary file and copy content to do md5-based replacement
+    if info['status'] != 200:
+        module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'], url=url)
+    actualdest = info['actualdest']
+
+    fd, tempname = tempfile.mkstemp()
+    f = os.fdopen(fd, 'wb')
+    try:
+        shutil.copyfileobj(req, f)
+    except Exception, err:
+        os.remove(tempname)
+        module.fail_json(msg="failed to create temporary content file: %s" % str(err))
+    f.close()
+    req.close()
+    return tempname, info
+
+# ==============================================================
+# main
+
+def main():
+
+    # does this really happen on non-ancient python?
+    if not HAS_URLLIB2:
+        module.fail_json(msg="urllib2 is not installed")
+    if not HAS_URLPARSE:
+        module.fail_json(msg="urlparse is not installed")
+
+    module = AnsibleModule(
+        # not checking because of daisy chain to file module
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            url = dict(required=True),
+            dest = dict(required=True),
+            thirsty = dict(default='no', choices=BOOLEANS)
+        )
+    )
+
+    url  = module.params['url']
+    dest = os.path.expanduser(module.params['dest'])
+    thirsty = module.boolean(module.params['thirsty'])
+
+    if not thirsty:
+        if os.path.exists(dest):
+            module.exit_json(msg="file already exists", changed=False)
+        if os.path.isdir(dest):
+            module.fail_json(msg="non-thirsty mode needs a filename for a destination, not a directory")
+
+
+    # download to tmpsrc
+    tmpsrc, info = url_get(module, url, dest)
+    md5sum_src   = None
+    md5sum_dest  = None
+    dest         = info['actualdest']
+
+    # raise an error if there is no tmpsrc file
+    if not os.path.exists(tmpsrc):
+        os.remove(tmpsrc)
+        module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'])
+    if not os.access(tmpsrc, os.R_OK):
+        os.remove(tmpsrc)
+        module.fail_json( msg="Source %s not readable" % (tmpsrc))
+    md5sum_src = module.md5(tmpsrc)
+
+    # check if there is no dest file
+    if os.path.exists(dest):
+        # raise an error if copy has no permission on dest
+        if not os.access(dest, os.W_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not writable" % (dest))
+        if not os.access(dest, os.R_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not readable" % (dest))
+        md5sum_dest = module.md5(dest)
+    else:
+        if not os.access(os.path.dirname(dest), os.W_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not writable" % (os.path.dirname(dest)))
+
+    if md5sum_src != md5sum_dest:
+        try:
+            shutil.copyfile(tmpsrc, dest)
+        except Exception, err:
+            os.remove(tmpsrc)
+            module.fail_json(msg="failed to copy %s to %s: %s" % (tmpsrc, dest, str(err)))
+        changed = True
+    else:
+        changed = False
+
+    os.remove(tmpsrc)
+
+    # Mission complete
+    module.exit_json(url=url, dest=dest, src=tmpsrc, md5sum=md5sum_src,
+        changed=changed, msg=info.get('msg',''),
+        daisychain="file", daisychain_args=info.get('daisychain_args',''))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/git

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -17,148 +18,185 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-# I wanted to keep this simple at first, so for now this checks out
-# from the MASTER branch of a repo at a particular SHA or
-# tag.  Latest is not supported, you should not be doing
-# that.  Branch checkouts are not supported.  Contribs
-# welcome! -- MPD
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import os
-import sys
-import shlex
-import subprocess
-
-# ===========================================
-# convert arguments of form a=b c=d
-# to a dictionary
-# FIXME: make more idiomatic
-
-if len(sys.argv) == 1:
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "the command module requires arguments (-a)"
-    })
-    sys.exit(1)
-
-argfile = sys.argv[1]
-if not os.path.exists(argfile):
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "Argument file not found"
-    })
-    sys.exit(1)
-
-args = open(argfile, 'r').read()
-items = shlex.split(args)
-
-if not len(items):
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "the command module requires arguments (-a)"
-    })
-    sys.exit(1)
-
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
-
-dest = params['dest']
-repo = params['repo']
-version = params['version']
-
-# ===========================================
+import re
+import tempfile
 
 def get_version(dest):
-   ''' samples the version of the git repo '''
-   os.chdir(dest)
-   cmd = "git show --abbrev-commit"
-   sha = os.popen(cmd).read().split("\n")
-   sha = sha[0].split()[1]
-   return sha
+    ''' samples the version of the git repo '''
+    os.chdir(dest)
+    cmd = "git show --abbrev-commit"
+    sha = os.popen(cmd).read().split("\n")
+    sha = sha[0].split()[1]
+    return sha
 
 def clone(repo, dest):
-   ''' makes a new git repo if it does not already exist '''
-   try:
-       os.makedirs(dest)
-   except:
-       pass
-   cmd = "git clone %s %s" % (repo, dest)
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   return cmd.communicate()
+    ''' makes a new git repo if it does not already exist '''
+    try:
+        os.makedirs(os.path.dirname(dest))
+    except:
+        pass
+    cmd = "git clone %s %s" % (repo, dest)
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=tempfile.gettempdir())
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-def pull(repo, dest):
-   ''' updates repo from remote sources '''
-   os.chdir(dest)
-   cmd = "git pull -u origin"
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   return cmd.communicate()
 
-def switchver(version, dest):
-   ''' once pulled, switch to a particular SHA or tag '''
-   os.chdir(dest)
-   if version != 'HEAD':
-      cmd = "git checkout %s --force" % version
-   else:
-      # is there a better way to do this?
-      cmd = "git rebase origin"
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   (out, err) = cmd.communicate()
-   return (out, err)
- 
+def has_local_mods(dest):
+    os.chdir(dest)
+    cmd = "git status -s"
+    lines = os.popen(cmd).read().splitlines()
+    lines = filter(lambda c: re.search('^\\?\\?.*$',c) == None,lines)
+    return len(lines) > 0
 
-gitconfig = os.path.join(dest, '.git', 'config')
+def reset(module,dest,force):
+    '''
+    Resets the index and working tree to HEAD.
+    Discards any changes to tracked files in working
+    tree since that commit.
+    '''
+    os.chdir(dest)
+    if not force and has_local_mods(dest):
+        module.fail_json(msg="Local modifications exist in repository (force=no).")
+    cmd = "git reset --hard HEAD"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-out, err, status = (None, None, None)
+def get_branches(module, dest):
+    os.chdir(dest)
+    branches = []
+    cmd = "git branch -a"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = cmd.communicate()
+    if cmd.returncode != 0:
+        module.fail_json(msg="Could not determine branch data - received %s" % out)
+    for line in out.split('\n'):
+        branches.append(line.strip())
+    return branches
 
-# if there is no git configuration, do a clone operation
-# else pull and switch the version
+def is_remote_branch(module, dest, remote, branch):
+    branches = get_branches(module, dest)
+    rbranch = 'remotes/%s/%s' % (remote, branch)
+    if rbranch in branches:
+        return True
+    else:
+        return False
 
-before = None
-if not os.path.exists(gitconfig):
-   (out, err) = clone(repo, dest)
-else:
-   # else do a pull   
-   before = get_version(dest)
-   (out, err) = pull(repo, dest)
+def is_local_branch(module, dest, branch):
+    branches = get_branches(module, dest)
+    lbranch = '%s' % branch
+    if lbranch in branches:
+        return True
+    elif '* %s' % branch in branches:
+        return True
+    else:
+        return False
 
-# handle errors from clone or pull
+def is_current_branch(module, dest, branch):
+    branches = get_branches(module, dest)
+    for b in branches:
+        if b.startswith('* '):
+            cur_branch = b
+    if branch == cur_branch or '* %s' % branch == cur_branch:
+        return True
+    else:
+        return True
 
-if out.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+def pull(module, repo, dest, version):
+    ''' updates repo from remote sources '''
+    os.chdir(dest)
+    branches = get_branches(module, dest)
+    cur_branch = ''
+    for b in branches:
+        if b.startswith('* '):
+            cur_branch = b
+    if is_local_branch(module, dest, version) and not is_current_branch(module, dest, version):
+        (out, err) = switch_version(module, dest, remote, version)
 
-# switch to version specified regardless of whether
-# we cloned or pulled
+    cmd = "git pull -u origin"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-(out, err) = switchver(version, dest)
-if err.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+def switch_version(module, dest, remote, version):
+    ''' once pulled, switch to a particular SHA, tag, or branch '''
+    os.chdir(dest)
+    cmd = ''
+    if version != 'HEAD':
+        if not is_local_branch(module, dest, version) and is_remote_branch(module, dest, remote, version):
+            cmd = "git checkout --track -b %s %s/%s" % (version, remote, version)
+        else:
+            cmd = "git checkout --force %s" % version
+    else:
+        # is there a better way to do this?
+        cmd = "git rebase origin"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-# determine if we changed anything
+# ===========================================
 
-after = get_version(dest)
-changed = False
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            dest=dict(required=True),
+            repo=dict(required=True, aliases=['name']),
+            version=dict(default='HEAD'),
+            remote=dict(default='origin'),
+            force=dict(default='yes', choices=['yes', 'no'], aliases=['force'])
+        )
+    )
 
-if before != after:
-   changed = True
+    dest    = module.params['dest']
+    repo    = module.params['repo']
+    version = module.params['version']
+    remote  = module.params['remote']
+    force   = module.boolean(module.params['force'])
 
-print json.dumps({
-   "changed" : changed,
-   "before"  : before,
-   "after"   : after
-})
+    gitconfig = os.path.join(dest, '.git', 'config')
 
+    rc, out, err, status = (0, None, None, None)
+
+    # if there is no git configuration, do a clone operation
+    # else pull and switch the version
+    before = None
+    local_mods = False
+    if not os.path.exists(gitconfig):
+        (rc, out, err) = clone(repo, dest)
+        if rc != 0:
+            module.fail_json(msg=err)
+    else:
+        # else do a pull
+        local_mods = has_local_mods(dest)
+        before = get_version(dest)
+        (rc, out, err) = reset(module,dest,force)
+        if rc != 0:
+            module.fail_json(msg=err)
+        (rc, out, err) = pull(module, repo, dest, version)
+
+    # handle errors from clone or pull
+    if out.find('error') != -1 or err.find('ERROR') != -1:
+        module.fail_json(msg=err)
+
+    # switch to version specified regardless of whether
+    # we cloned or pulled
+    (rc, out, err) = switch_version(module, dest, remote, version)
+    if rc != 0:
+        module.fail_json(msg=err)
+
+    # determine if we changed anything
+    after = get_version(dest)
+    changed = False
+
+    if before != after or local_mods:
+        changed = True
+
+    module.exit_json(changed=changed, before=before, after=after)
+
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/group

@@ -1,4 +1,5 @@
 #!/usr/bin/env python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Stephen Fromm <sfromm@gmail.com>
 #
@@ -17,50 +18,17 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import os
 import grp
-import shlex
-import subprocess
-import sys
 
-GROUPADD = "/usr/sbin/groupadd"
-GROUPDEL = "/usr/sbin/groupdel"
-GROUPMOD = "/usr/sbin/groupmod"
+def group_del(module, group):
+    cmd = [module.get_bin_path('groupdel', True), group]
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
 
-def exit_json(rc=0, **kwargs):
-    if 'name' in kwargs:
-        add_group_info(kwargs)
-    print json.dumps(kwargs)
-    sys.exit(rc)
-
-def fail_json(**kwargs):
-    kwargs['failed'] = True
-    exit_json(rc=1, **kwargs)
-
-def add_group_info(kwargs):
-    name = kwargs['name']
-    if group_exists(name):
-        kwargs['state'] = 'present'
-        info = group_info(name)
-        kwargs['gid'] = info[2]
-    else:
-        kwargs['state'] = 'absent'
-    return kwargs
-
-def group_del(group):
-    cmd = [GROUPDEL, group]
-    rc = subprocess.call(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    if rc == 0:
-        return True
-    else:
-        return False
-
-def group_add(group, **kwargs):
-    cmd = [GROUPADD]
+def group_add(module, group, **kwargs):
+    cmd = [module.get_bin_path('groupadd', True)]
     for key in kwargs:
         if key == 'gid' and kwargs[key] is not None:
             cmd.append('-g')
@@ -68,14 +36,13 @@ def group_add(group, **kwargs):
         elif key == 'system' and kwargs[key] == 'yes':
             cmd.append('-r')
     cmd.append(group)
-    rc = subprocess.call(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    if rc == 0:
-        return True
-    else:
-        return False
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
 
-def group_mod(group, **kwargs):
-    cmd = [GROUPMOD]
+def group_mod(module, group, **kwargs):
+    cmd = [module.get_bin_path('groupmod', True)]
     info = group_info(group)
     for key in kwargs:
         if key == 'gid':
@@ -83,13 +50,12 @@ def group_mod(group, **kwargs):
                 cmd.append('-g')
                 cmd.append(kwargs[key])
     if len(cmd) == 1:
-        return False
+        return (None, '', '')
     cmd.append(group)
-    rc = subprocess.call(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    if rc == 0:
-        return True
-    else:
-        return False
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
 
 def group_exists(group):
     try:
@@ -106,65 +72,59 @@ def group_info(group):
     except KeyError:
         return False
     return info
-    
+
 # ===========================================
 
-if not os.path.exists(GROUPADD):
-    if os.path.exists("/sbin/groupadd"):
-        GROUPADD = "/sbin/groupadd"
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            state=dict(default='present', choices=['present', 'absent']),
+            name=dict(required=True),
+            gid=dict(default=None),
+            system=dict(default='no', choices=['yes', 'no']),
+        )
+    )
+
+    state = module.params['state']
+    name = module.params['name']
+    gid = module.params['gid']
+    system = module.params['system']
+    rc = None
+    out = ''
+    err = ''
+    result = {}
+    result['name'] = name
+    result['state'] = state
+    if state == 'absent':
+        if group_exists(name):
+            (rc, out, err) = group_del(module, name)
+            if rc != 0:
+                module.fail_json(name=name, msg=err)
+    elif state == 'present':
+        if not group_exists(name):
+            (rc, out, err) = group_add(module, name, gid=gid, system=system)
+        else:
+            (rc, out, err) = group_mod(module, name, gid=gid)
+
+        if rc is not None and rc != 0:
+            module.fail_json(name=name, msg=err)
+
+    if rc is None:
+        result['changed'] = False
     else:
-        fail_json(msg="Cannot find groupadd")
-if not os.path.exists(GROUPDEL):
-    if os.path.exists("/sbin/groupdel"):
-        GROUPDEL = "/sbin/groupdel"
-    else:
-        fail_json(msg="Cannot find groupdel")
-if not os.path.exists(GROUPMOD):
-    if os.path.exists("/sbin/groupmod"):
-        GROUPDEL = "/sbin/groupmod"
-    else:
-        fail_json(msg="Cannot find groupmod")
+        result['changed'] = True
+    if out:
+        result['stdout'] = out
+    if err:
+        result['stderr'] = err
 
-if len(sys.argv) == 2 and os.path.exists(sys.argv[1]):
-    argfile = sys.argv[1]
-    args    = open(argfile, 'r').read()
-else:
-    args = ' '.join(sys.argv[1:])
-items   = shlex.split(args)
-
-if not len(items):
-    fail_json(msg='the module requires arguments -a')
-    sys.exit(1)
-
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
-
-state       = params.get('state','present')
-name        = params.get('name', None)
-gid         = params.get('gid', None)
-system      = params.get('system', 'no')
-
-if state not in [ 'present', 'absent' ]:
-    fail_json(msg='invalid state')
-if system not in ['yes', 'no']:
-    fail_json(msg='invalid system')
-if name is None:
-    fail_json(msg='name is required')
-
-changed = False
-rc = 0
-if state == 'absent':
     if group_exists(name):
-        changed = group_del(name)
-    exit_json(name=name, changed=changed)
-elif state == 'present':
-    if not group_exists(name):
-        changed = group_add(name, gid=gid, system=system)
-    else:
-        changed = group_mod(name, gid=gid)
+        info = group_info(name)
+        result['system'] = system
+        result['gid'] = info[2]
 
-    exit_json(name=name, changed=changed)
+    module.exit_json(**result)
 
-fail_json(name=name, msg='Unexpected position reached')
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/lineinfile

@@ -0,0 +1,129 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Daniel Hokka Zakrisson <daniel@hozac.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+import os
+
+def present(module, name, regexp, line, insertafter, backup):
+    f = open(name, 'rb')
+    lines = f.readlines()
+    f.close()
+
+    mre = re.compile(regexp)
+    if not mre.match(line):
+        module.fail_json(msg="line= doesn't match regexp=")
+
+    if insertafter in ('BOF', 'EOF'):
+        iare = None
+    else:
+        iare = re.compile(insertafter)
+
+    index = [-1, -1]
+    for lineno in range(0, len(lines)):
+        if mre.match(lines[lineno]):
+            index[0] = lineno
+        elif iare is not None and iare.match(lines[lineno]):
+            # + 1 for the next line
+            index[1] = lineno + 1
+
+    # Regexp matched a line in the file
+    if index[0] != -1:
+        if lines[index[0]] == line + os.linesep:
+            msg = ''
+            changed = False
+        else:
+            lines[index[0]] = line + os.linesep
+            msg = 'line replaced'
+            changed = True
+    # Add it to the beginning of the file
+    elif insertafter == 'BOF':
+        lines.insert(0, line + os.linesep)
+        msg = 'line added'
+        changed = True
+    # Add it to the end of the file if requested or if insertafter= didn't match
+    elif insertafter == 'EOF' or index[1] == -1:
+        lines.append(line + os.linesep)
+        msg = 'line added'
+        changed = True
+    # insertafter= matched
+    else:
+        lines.insert(index[1], line + os.linesep)
+        msg = 'line added'
+        changed = True
+
+    if changed:
+        if backup:
+            module.backup_local(name)
+        f = open(name, 'wb')
+        f.writelines(lines)
+        f.close()
+
+    module.exit_json(changed=changed, msg=msg)
+
+def absent(module, name, regexp, backup):
+    f = open(name, 'rb')
+    lines = f.readlines()
+    f.close()
+    cre = re.compile(regexp)
+    found = []
+    def matcher(line):
+        if cre.match(line):
+            found.append(line)
+            return False
+        else:
+            return True
+    lines = filter(matcher, lines)
+    changed = len(found) > 0
+    if changed:
+        if backup:
+            module.backup_local(name)
+        f = open(name, 'wb')
+        f.writelines(lines)
+        f.close()
+    module.exit_json(changed=changed, found=len(found))
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            name=dict(required=True, aliases=['dest', 'destfile']),
+            state=dict(default='present', choices=['absent', 'present']),
+            regexp=dict(required=True),
+            line=dict(aliases=['value']),
+            insertafter=dict(default='EOF'),
+            backup=dict(default=False, choices=BOOLEANS),
+        ),
+    )
+
+    params = module.params
+    backup = module.boolean(module.params.get('backup', False))
+
+    if params['state'] == 'present':
+        if 'line' not in params:
+            module.fail_json(msg='line= is required with state=present')
+        present(module, params['name'], params['regexp'], params['line'],
+                params['insertafter'], backup)
+    else:
+        absent(module, params['name'], params['regexp'], backup)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+

library/mount

@@ -0,0 +1,265 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Red Hat, inc
+# Written by Seth Vidal
+# based on the mount modules from salt and puppet
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# mount module - mount fs and define in fstab
+# usage:
+#
+# mount name=mountpoint, src=device_to_be_mounted fstype=fstype
+#        opts=mount_opts, dump=0 passno=0 state=[present|absent|mounted|unmounted]
+#
+#    absent == remove from fstab and unmounted
+#    present == add to fstab, do not change mount state
+#    mounted == add to fstab if not there and make sure it is mounted
+#    unmounted == do not change fstab state, but unmount
+
+def write_fstab(lines, dest):
+
+    fs_w = open(dest, 'w')
+    for l in lines:
+        fs_w.write(l)
+
+    fs_w.flush()
+    fs_w.close()
+
+def set_mount(**kwargs):
+    """ set/change a mount point location in fstab """
+
+    # kwargs: name, src, fstype, opts, dump, passno, state, fstab=/etc/fstab
+    args = dict(
+        opts   = 'defaults',
+        dump   = '0',
+        passno = '0',
+        fstab  = '/etc/fstab'
+    )
+    args.update(kwargs)
+
+    new_line = '%(src)s %(name)s %(fstype)s %(opts)s %(dump)s %(passno)s\n'
+
+    to_write = []
+    exists = False
+    changed = False
+    for line in open(args['fstab'], 'r').readlines():
+        if not line.strip():
+            to_write.append(line)
+            continue
+        if line.strip().startswith('#'):
+            to_write.append(line)
+            continue
+        if len(line.split()) != 6:
+            # not sure what this is or why it is here
+            # but it is not our fault so leave it be
+            to_write.append(line)
+            continue
+
+        ld = {}
+        ld['src'], ld['name'], ld['fstype'], ld['opts'], ld['dump'], ld['passno']  = line.split()
+
+        if ld['name'] != args['name']:
+            to_write.append(line)
+            continue
+
+        # it exists - now see if what we have is different
+        exists = True
+        for t in ('src', 'fstype','opts', 'dump', 'passno'):
+            if ld[t] != args[t]:
+                changed = True
+                ld[t] = args[t]
+
+        if changed:
+            to_write.append(new_line % ld)
+        else:
+            to_write.append(line)
+
+    if not exists:
+        to_write.append(new_line % args)
+        changed = True
+
+    if changed:
+        write_fstab(to_write, args['fstab'])
+
+    return (args['name'], changed)
+
+
+def unset_mount(**kwargs):
+    """ remove a mount point from fstab """
+
+    # kwargs: name, src, fstype, opts, dump, passno, state, fstab=/etc/fstab
+    args = dict(
+        opts   = 'default',
+        dump   = '0',
+        passno = '0',
+        fstab  = '/etc/fstab'
+    )
+    args.update(kwargs)
+
+    to_write = []
+    changed = False
+    for line in open(args['fstab'], 'r').readlines():
+        if not line.strip():
+            to_write.append(line)
+            continue
+        if line.strip().startswith('#'):
+            to_write.append(line)
+            continue
+        if len(line.split()) != 6:
+            # not sure what this is or why it is here
+            # but it is not our fault so leave it be
+            to_write.append(line)
+            continue
+
+        ld = {}
+        ld['src'], ld['name'], ld['fstype'], ld['opts'], ld['dump'], ld['passno']  = line.split()
+
+        if ld['name'] != args['name']:
+            to_write.append(line)
+            continue
+
+        # if we got here we found a match - continue and mark changed
+        changed = True
+
+    if changed:
+        write_fstab(to_write, args['fstab'])
+
+    return (args['name'], changed)
+
+
+def mount(**kwargs):
+    """ mount up a path or remount if needed """
+
+    name = kwargs['name']
+    if os.path.ismount(name):
+        cmd = [ '/bin/mount', '-o', 'remount', name ]
+    else:
+        cmd = [ '/bin/mount', name ]
+
+    call = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = call.communicate()
+    if call.returncode == 0:
+        return 0, ''
+    else:
+        return call.returncode, out+err
+
+def umount(**kwargs):
+    """ unmount a path """
+
+    name = kwargs['name']
+    cmd = ['/bin/umount', name]
+
+    call = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = call.communicate()
+    if call.returncode == 0:
+        return 0, ''
+    else:
+        return call.returncode, out+err
+
+def main():
+
+    module = AnsibleModule(
+        argument_spec = dict(
+            state  = dict(required=True, choices=['present', 'absent', 'mounted', 'unmounted']),
+            name   = dict(required=True),
+            opts   = dict(default=None),
+            passno = dict(default=None),
+            dump   = dict(default=None),
+            src    = dict(required=True),
+            fstype = dict(required=True),
+            fstab  = dict(default=None)
+        )
+    )
+
+    changed = False
+    rc = 0
+    args = {
+        'name': module.params['name'],
+        'src': module.params['src'],
+        'fstype': module.params['fstype']
+    }
+    if module.params['passno'] is not None:
+        args['passno'] = module.params['passno']
+    if module.params['opts'] is not None:
+        args['opts'] = module.params['opts']
+    if module.params['dump'] is not None:
+        args['dump'] = module.params['dump']
+    if module.params['fstab'] is not None:
+        args['fstab'] = module.params['fstab']
+
+    # absent == remove from fstab and unmounted
+    # unmounted == do not change fstab state, but unmount
+    # present == add to fstab, do not change mount state
+    # mounted == add to fstab if not there and make sure it is mounted, if it has changed in fstab then remount it
+
+    state = module.params['state']
+    name  = module.params['name']
+    if state == 'absent':
+        name, changed = unset_mount(**args)
+        if changed:
+            if os.path.ismount(name):
+                res,msg  = umount(**args)
+                if res:
+                    module.fail_json(msg="Error unmounting %s: %s" % (name, msg))
+
+            if os.path.exists(name):
+                try:
+                    os.rmdir(name)
+                except (OSError, IOError), e:
+                    module.fail_json(msg="Error rmdir %s: %s" % (name, str(e)))
+
+        module.exit_json(changed=changed, **args)
+
+    if state == 'unmounted':
+        if os.path.ismount(name):
+            res,msg  = umount(**args)
+            if res:
+                module.fail_json(msg="Error unmounting %s: %s" % (name, msg))
+            changed = True
+
+        module.exit_json(changed=changed, **args)
+
+    if state in ['mounted', 'present']:
+        name, changed = set_mount(**args)
+        if state == 'mounted':
+            if not os.path.exists(name):
+                try:
+                    os.makedirs(name)
+                except (OSError, IOError), e:
+                    module.fail_json(msg="Error making dir %s: %s" % (name, str(e)))
+
+            res = 0
+            if os.path.ismount(name):
+                if changed:
+                    res,msg = mount(**args)
+            else:
+                changed = True
+                res,msg = mount(**args)
+
+            if res:
+                module.fail_json(msg="Error mounting %s: %s" % (name, msg))
+
+
+        module.exit_json(changed=changed, **args)
+
+    module.fail_json(msg='Unexpected position reached')
+    sys.exit(0)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/mysql_db

@@ -0,0 +1,127 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Mark Theunissen <mark.theunissen@gmail.com>
+# Sponsored by Four Kitchens http://fourkitchens.com.
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ConfigParser
+try:
+    import MySQLdb
+except ImportError:
+    mysqldb_found = False
+else:
+    mysqldb_found = True
+
+# ===========================================
+# MySQL module specific support methods.
+#
+
+def db_exists(cursor, db):
+    res = cursor.execute("SHOW DATABASES LIKE %s", (db,))
+    return bool(res)
+
+def db_delete(cursor, db):
+    query = "DROP DATABASE %s" % db
+    cursor.execute(query)
+    return True
+
+def db_create(cursor, db, encoding, collation):
+    if encoding:
+        encoding = " CHARACTER SET %s" % encoding
+    if collation:
+        collation = " COLLATE %s" % collation
+    query = "CREATE DATABASE %s%s%s" % (db, encoding, collation)
+    res = cursor.execute(query)
+    return True
+
+def load_mycnf():
+    config = ConfigParser.RawConfigParser()
+    mycnf = os.path.expanduser('~/.my.cnf')
+    if not os.path.exists(mycnf):
+        return False
+    try:
+        config.readfp(open(mycnf))
+        creds = dict(user=config.get('client', 'user'),passwd=config.get('client', 'pass'))
+    except (ConfigParser.NoOptionError, IOError):
+        return False
+    return creds
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            login_user=dict(default=None),
+            login_password=dict(default=None),
+            login_host=dict(default="localhost"),
+            login_unix_socket=dict(default=None),
+            db=dict(required=True, aliases=['name']),
+            encoding=dict(default=""),
+            collation=dict(default=""),
+            state=dict(default="present", choices=["absent", "present"]),
+        )
+    )
+
+    if not mysqldb_found:
+        module.fail_json(msg="the python mysqldb module is required")
+
+    db = module.params["db"]
+    encoding = module.params["encoding"]
+    collation = module.params["collation"]
+    state = module.params["state"]
+
+    # Either the caller passes both a username and password with which to connect to
+    # mysql, or they pass neither and allow this module to read the credentials from
+    # ~/.my.cnf.
+    login_password = module.params["login_password"]
+    login_user = module.params["login_user"]
+    if login_user is None and login_password is None:
+        mycnf_creds = load_mycnf()
+        if mycnf_creds is False:
+            login_user = "root"
+            login_password = ""
+        else:
+            login_user = mycnf_creds["user"]
+            login_password = mycnf_creds["passwd"]
+    elif login_password is None or login_user is None:
+        module.fail_json(msg="when supplying login arguments, both login_user and login_password must be provided")
+
+    try:
+        if module.params["login_unix_socket"] != None:
+            db_connection = MySQLdb.connect(host=module.params["login_host"], unix_socket=module.params["login_unix_socket"], user=login_user, passwd=login_password, db="mysql")
+        else:
+            db_connection = MySQLdb.connect(host=module.params["login_host"], user=login_user, passwd=login_password, db="mysql")
+        cursor = db_connection.cursor()
+    except Exception as e:
+        module.fail_json(msg="unable to connect, check login_user and login_password are correct, or alternatively check ~/.my.cnf contains credentials")
+
+    changed = False
+    if db_exists(cursor, db):
+        if state == "absent":
+            changed = db_delete(cursor, db)
+    else:
+        if state == "present":
+            changed = db_create(cursor, db, encoding, collation)
+
+    module.exit_json(changed=changed, db=db)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/mysql_user

@@ -0,0 +1,231 @@
+#!/usr/bin/python
+
+# (c) 2012, Mark Theunissen <mark.theunissen@gmail.com>
+# Sponsored by Four Kitchens http://fourkitchens.com.
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ConfigParser
+try:
+    import MySQLdb
+except ImportError:
+    mysqldb_found = False
+else:
+    mysqldb_found = True
+
+# ===========================================
+# MySQL module specific support methods.
+#
+
+def user_exists(cursor, user, host):
+    cursor.execute("SELECT count(*) FROM user WHERE user = %s AND host = %s", (user,host))
+    count = cursor.fetchone()
+    return count[0] > 0
+
+def user_add(cursor, user, host, password, new_priv):
+    cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (user,host,password))
+    if new_priv is not None:
+        for db_table, priv in new_priv.iteritems():
+            privileges_grant(cursor, user,host,db_table,priv)
+    return True
+
+def user_mod(cursor, user, host, password, new_priv):
+    changed = False
+
+    # Handle passwords.
+    if password is not None:
+        cursor.execute("SELECT password FROM user WHERE user = %s AND host = %s", (user,host))
+        current_pass_hash = cursor.fetchone()
+        cursor.execute("SELECT PASSWORD(%s)", (password,))
+        new_pass_hash = cursor.fetchone()
+        if current_pass_hash[0] != new_pass_hash[0]:
+            cursor.execute("SET PASSWORD FOR %s@%s = PASSWORD(%s)", (user,host,password))
+            changed = True
+
+    # Handle privileges.
+    if new_priv is not None:
+        curr_priv = privileges_get(cursor, user,host)
+
+        # If the user has privileges on a db.table that doesn't appear at all in
+        # the new specification, then revoke all privileges on it.
+        for db_table, priv in curr_priv.iteritems():
+            if db_table not in new_priv:
+                privileges_revoke(cursor, user,host,db_table)
+                changed = True
+
+        # If the user doesn't currently have any privileges on a db.table, then
+        # we can perform a straight grant operation.
+        for db_table, priv in new_priv.iteritems():
+            if db_table not in curr_priv:
+                privileges_grant(cursor, user,host,db_table,priv)
+                changed = True
+
+        # If the db.table specification exists in both the user's current privileges
+        # and in the new privileges, then we need to see if there's a difference.
+        db_table_intersect = set(new_priv.keys()) & set(curr_priv.keys())
+        for db_table in db_table_intersect:
+            priv_diff = set(new_priv[db_table]) ^ set(curr_priv[db_table])
+            if (len(priv_diff) > 0):
+                privileges_revoke(cursor, user,host,db_table)
+                privileges_grant(cursor, user,host,db_table,new_priv[db_table])
+                changed = True
+
+    return changed
+
+def user_delete(cursor, user, host):
+    cursor.execute("DROP USER %s@%s", (user,host))
+    return True
+
+def privileges_get(cursor, user,host):
+    """ MySQL doesn't have a better method of getting privileges aside from the
+    SHOW GRANTS query syntax, which requires us to then parse the returned string.
+    Here's an example of the string that is returned from MySQL:
+
+     GRANT USAGE ON *.* TO 'user'@'localhost' IDENTIFIED BY 'pass';
+
+    This function makes the query and returns a dictionary containing the results.
+    The dictionary format is the same as that returned by privileges_unpack() below.
+    """
+    output = {}
+    cursor.execute("SHOW GRANTS FOR %s@%s", (user,host))
+    grants = cursor.fetchall()
+    for grant in grants:
+        res = re.match("GRANT\ (.+)\ ON\ (.+)\ TO", grant[0])
+        if res is None:
+            module.fail_json(msg="unable to parse the MySQL grant string")
+        privileges = res.group(1).split(", ")
+        privileges = ['ALL' if x=='ALL PRIVILEGES' else x for x in privileges]
+        db = res.group(2).replace('`', '')
+        output[db] = privileges
+    return output
+
+def privileges_unpack(priv):
+    """ Take a privileges string, typically passed as a parameter, and unserialize
+    it into a dictionary, the same format as privileges_get() above. We have this
+    custom format to avoid using YAML/JSON strings inside YAML playbooks. Example
+    of a privileges string:
+
+     mydb.*:INSERT,UPDATE/anotherdb.*:SELECT/yetanother.*:ALL
+
+    The privilege USAGE stands for no privileges, so we add that in on *.* if it's
+    not specified in the string, as MySQL will always provide this by default.
+    """
+    output = {}
+    for item in priv.split('/'):
+        pieces = item.split(':')
+        output[pieces[0]] = pieces[1].upper().split(',')
+
+    if '*.*' not in output:
+        output['*.*'] = ['USAGE']
+
+    return output
+
+def privileges_revoke(cursor, user,host,db_table):
+    query = "REVOKE ALL PRIVILEGES ON %s FROM '%s'@'%s'" % (db_table,user,host)
+    cursor.execute(query)
+
+def privileges_grant(cursor, user,host,db_table,priv):
+    priv_string = ",".join(priv)
+    query = "GRANT %s ON %s TO '%s'@'%s'" % (priv_string,db_table,user,host)
+    cursor.execute(query)
+
+def load_mycnf():
+    config = ConfigParser.RawConfigParser()
+    mycnf = os.path.expanduser('~/.my.cnf')
+    if not os.path.exists(mycnf):
+        return False
+    try:
+        config.readfp(open(mycnf))
+        creds = dict(user=config.get('client', 'user'),password=config.get('client', 'pass'))
+    except (ConfigParser.NoOptionError, IOError):
+        return False
+    return creds
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            login_user=dict(default=None),
+            login_password=dict(default=None),
+            login_host=dict(default="localhost"),
+            login_unix_socket=dict(default=None),
+            user=dict(required=True, aliases=['name']),
+            password=dict(default=None),
+            host=dict(default="localhost"),
+            state=dict(default="present", choices=["absent", "present"]),
+            priv=dict(default=None),
+        )
+    )
+    user = module.params["user"]
+    password = module.params["password"]
+    host = module.params["host"]
+    state = module.params["state"]
+    priv = module.params["priv"]
+
+    if not mysqldb_found:
+        module.fail_json(msg="the python mysqldb module is required")
+
+    if priv is not None:
+        try:
+            priv = privileges_unpack(priv)
+        except:
+            module.fail_json(msg="invalid privileges string")
+
+    # Either the caller passes both a username and password with which to connect to
+    # mysql, or they pass neither and allow this module to read the credentials from
+    # ~/.my.cnf.
+    login_password = module.params["login_password"]
+    login_user = module.params["login_user"]
+    if login_user is None and login_password is None:
+        mycnf_creds = load_mycnf()
+        if mycnf_creds is False:
+            login_user = "root"
+            login_password = ""
+        else:
+            login_user = mycnf_creds["user"]
+            login_password = mycnf_creds["password"]
+    elif login_password is None or login_user is None:
+        module.fail_json(msg="when supplying login arguments, both login_user and login_password must be provided")
+
+    try:
+        if module.params["login_unix_socket"] != None:
+            db_connection = MySQLdb.connect(host=module.params["login_host"], unix_socket=module.params["login_unix_socket"], user=login_user, passwd=login_password, db="mysql")
+        else:
+            db_connection = MySQLdb.connect(host=module.params["login_host"], user=login_user, passwd=login_password, db="mysql")
+        cursor = db_connection.cursor()
+    except Exception as e:
+        module.fail_json(msg="unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials")
+
+    if state == "present":
+        if user_exists(cursor, user, host):
+            changed = user_mod(cursor, user, host, password, priv)
+        else:
+            if password is None:
+                module.fail_json(msg="password parameter required when adding a user")
+            changed = user_add(cursor, user, host, password, priv)
+    elif state == "absent":
+        if user_exists(cursor, user, host):
+            changed = user_delete(cursor, user, host)
+        else:
+            changed = False
+    module.exit_json(changed=changed, user=user)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/nagios

@@ -0,0 +1,716 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# This file is largely copied from the Nagios module included in the
+# Func project. Original copyright follows:
+#
+# func-nagios - Schedule downtime and enables/disable notifications
+# Copyright 2011, Red Hat, Inc.
+# Tim Bielawa <tbielawa@redhat.com>
+#
+# This software may be freely redistributed under the terms of the GNU
+# general public license version 2.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+"""
+Action Summaries (all must be delegate_to'd a nagios server):
+
+* Set 30 minutes of downtime for a single service:
+  action: nagios action=downtime minutes=30 service=httpd host=$inventory_hostname
+
+* Set 30 minutes of downtime for a few services:
+  action: nagios action=downtime minutes=30 services=httpd,git,nfs host=$inventory_hostname
+
+* (Un)Silence all alerts from a host/service:
+  action: nagios action=silence host=$inventory_hostname
+  action: nagios action=unsilence host=$inventory_hostname
+
+* Enable/Disable alerts for a host or service(s):
+  action: nagios action=enable_alerts service=host host=$inventory_hostname
+  action: nagios action=disable_alerts services=httpd,git,nfs host=$inventory_hostname
+
+Note: 'service' is an alias for 'services'. Separate multiple services
+with commas.
+
+Set the path to the command file explicitly with the 'cmdfile'
+parameter.
+"""
+
+
+import ConfigParser
+import types
+import time
+import os.path
+
+######################################################################
+
+
+def which_cmdfile():
+    locations = [
+        # rhel
+        '/etc/nagios/nagios.cfg',
+        # debian
+        '/etc/nagios3/nagios.cfg',
+        # older debian
+        '/etc/nagios2/nagios.cfg',
+        # bsd, solaris
+        '/usr/local/etc/nagios/nagios.cfg',
+        # groundwork it monitoring
+        '/usr/local/groundwork/nagios/etc/nagios.cfg',
+        # open monitoring distribution
+        '/omd/sites/oppy/tmp/nagios/nagios.cfg',
+        # ???
+        '/usr/local/nagios/etc/nagios.cfg',
+        '/usr/local/nagios/nagios.cfg',
+        '/opt/nagios/etc/nagios.cfg',
+        '/opt/nagios/nagios.cfg'
+        ]
+
+    for path in locations:
+        if os.path.exists(path):
+            for line in open(path):
+                if line.startswith('command_file'):
+                    return line.partition('=')[2].strip()
+
+    return None
+
+######################################################################
+
+
+def main():
+    ACTION_CHOICES = [
+        'downtime',
+        'silence',
+        'unsilence',
+        'enable_alerts',
+        'disable_alerts'
+        ]
+
+    module = AnsibleModule(
+        argument_spec=dict(
+            action=dict(required=True, default=None, choices=ACTION_CHOICES),
+            author=dict(default='Ansible'),
+            host=dict(required=True, default=None),
+            minutes=dict(default=30),
+            cmdfile=dict(default=which_cmdfile()),
+            services=dict(default=None, aliases=['service']),
+            )
+        )
+
+    action = module.params['action']
+    minutes = module.params['minutes']
+    services = module.params['services']
+    cmdfile = module.params['cmdfile']
+
+    ##################################################################
+    # Required args per action:
+    # downtime = (minutes, service, host)
+    # (un)silence = (host)
+    # (enable/disable)_alerts = (service, host)
+    #
+    # AnsibleModule will verify most stuff, we need to verify
+    # 'minutes' and 'service' manually.
+
+    ##################################################################
+    if action == 'downtime':
+        # Make sure there's an actual service selected
+        if not services:
+            module.fail_json(msg='no service selected to set downtime for')
+        # Make sure minutes is a number
+        try:
+            m = int(minutes)
+            if not isinstance(m, types.IntType):
+                module.fail_json(msg='minutes must be a number')
+        except:
+            module.fail_json(msg='invalid entry for minutes')
+
+    ##################################################################
+    if action in ['enable_alerts', 'disable_alerts']:
+        if not services:
+            module.fail_json(msg='a service is required when setting alerts')
+
+    ##################################################################
+    if not cmdfile:
+        module.fail_json('unable to locate nagios.cfg')
+
+    ##################################################################
+    ansible_nagios = Nagios(module, **module.params)
+    ansible_nagios.act()
+    ##################################################################
+
+
+######################################################################
+class Nagios(object):
+    """
+    Perform common tasks in Nagios related to downtime and
+    notifications.
+
+    The complete set of external commands Nagios handles is documented
+    on their website:
+
+    http://old.nagios.org/developerinfo/externalcommands/commandlist.php
+
+    Note that in the case of `schedule_svc_downtime`,
+    `enable_svc_notifications`, and `disable_svc_notifications`, the
+    service argument should be passed as a list.
+    """
+
+    def __init__(self, module, **kwargs):
+        self.module = module
+        self.action = kwargs['action']
+        self.author = kwargs['author']
+        self.host = kwargs['host']
+        self.minutes = int(kwargs['minutes'])
+        self.cmdfile = kwargs['cmdfile']
+
+        if (kwargs['services'] is None) or (kwargs['services'] == 'host'):
+            self.services = kwargs['services']
+        else:
+            self.services = kwargs['services'].split(',')
+
+        self.command_results = []
+
+    def _now(self):
+        """
+        The time in seconds since 12:00:00AM Jan 1, 1970
+        """
+
+        return int(time.time())
+
+    def _write_command(self, cmd):
+        """
+        Write the given command to the Nagios command file
+        """
+
+        try:
+            fp = open(self.cmdfile, 'w')
+            fp.write(cmd)
+            fp.flush()
+            fp.close()
+            self.command_results.append(cmd.strip())
+        except IOError:
+            self.module.fail_json(msg='unable to write to nagios command file',
+                                  cmdfile=self.cmdfile)
+
+    def _fmt_dt_str(self, cmd, host, duration, author=None,
+                    comment="Scheduling downtime", start=None,
+                    svc=None, fixed=1, trigger=0):
+        """
+        Format an external-command downtime string.
+
+        cmd - Nagios command ID
+        host - Host schedule downtime on
+        duration - Minutes to schedule downtime for
+        author - Name to file the downtime as
+        comment - Reason for running this command (upgrade, reboot, etc)
+        start - Start of downtime in seconds since 12:00AM Jan 1 1970
+          Default is to use the entry time (now)
+        svc - Service to schedule downtime for, omit when for host downtime
+        fixed - Start now if 1, start when a problem is detected if 0
+        trigger - Optional ID of event to start downtime from. Leave as 0 for
+          fixed downtime.
+
+        Syntax: [submitted] COMMAND;<host_name>;[<service_description>]
+        <start_time>;<end_time>;<fixed>;<trigger_id>;<duration>;<author>;
+        <comment>
+        """
+
+        entry_time = self._now()
+        if start is None:
+            start = entry_time
+
+        hdr = "[%s] %s;%s;" % (entry_time, cmd, host)
+        duration_s = (duration * 60)
+        end = start + duration_s
+
+        if not author:
+            author = self.author
+
+        if svc is not None:
+            dt_args = [svc, str(start), str(end), str(fixed), str(trigger),
+                       str(duration_s), author, comment]
+        else:
+            # Downtime for a host if no svc specified
+            dt_args = [str(start), str(end), str(fixed), str(trigger),
+                       str(duration_s), author, comment]
+
+        dt_arg_str = ";".join(dt_args)
+        dt_str = hdr + dt_arg_str + "\n"
+
+        return dt_str
+
+    def _fmt_notif_str(self, cmd, host, svc=None):
+        """
+        Format an external-command notification string.
+
+        cmd - Nagios command ID.
+        host - Host to en/disable notifications on..
+        svc - Service to schedule downtime for. A value is not required
+          for host downtime.
+
+        Syntax: [submitted] COMMAND;<host_name>[;<service_description>]
+        """
+
+        entry_time = self._now()
+        if svc is not None:
+            notif_str = "[%s] %s;%s;%s\n" % (entry_time, cmd, host, svc)
+        else:
+            # Downtime for a host if no svc specified
+            notif_str = "[%s] %s;%s\n" % (entry_time, cmd, host)
+
+        return notif_str
+
+    def schedule_svc_downtime(self, host, services=[], minutes=30):
+        """
+        This command is used to schedule downtime for a particular
+        service.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the service.
+
+        Syntax: SCHEDULE_SVC_DOWNTIME;<host_name>;<service_description>
+        <start_time>;<end_time>;<fixed>;<trigger_id>;<duration>;<author>;
+        <comment>
+        """
+
+        cmd = "SCHEDULE_SVC_DOWNTIME"
+        for service in services:
+            dt_cmd_str = self._fmt_dt_str(cmd, host, minutes, svc=service)
+            self._write_command(dt_cmd_str)
+
+    def schedule_host_downtime(self, host, minutes=30):
+        """
+        This command is used to schedule downtime for a particular
+        host.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the host.
+
+        Syntax: SCHEDULE_HOST_DOWNTIME;<host_name>;<start_time>;<end_time>;
+        <fixed>;<trigger_id>;<duration>;<author>;<comment>
+        """
+
+        cmd = "SCHEDULE_HOST_DOWNTIME"
+        dt_cmd_str = self._fmt_dt_str(cmd, host, minutes)
+        self._write_command(dt_cmd_str)
+
+    def schedule_hostgroup_host_downtime(self, hostgroup, minutes=30):
+        """
+        This command is used to schedule downtime for all hosts in a
+        particular hostgroup.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the hosts.
+
+        Syntax: SCHEDULE_HOSTGROUP_HOST_DOWNTIME;<hostgroup_name>;<start_time>;
+        <end_time>;<fixed>;<trigger_id>;<duration>;<author>;<comment>
+        """
+
+        cmd = "SCHEDULE_HOSTGROUP_HOST_DOWNTIME"
+        dt_cmd_str = self._fmt_dt_str(cmd, hostgroup, minutes)
+        self._write_command(dt_cmd_str)
+
+    def schedule_hostgroup_svc_downtime(self, hostgroup, minutes=30):
+        """
+        This command is used to schedule downtime for all services in
+        a particular hostgroup.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the services.
+
+        Note that scheduling downtime for services does not
+        automatically schedule downtime for the hosts those services
+        are associated with.
+
+        Syntax: SCHEDULE_HOSTGROUP_SVC_DOWNTIME;<hostgroup_name>;<start_time>;
+        <end_time>;<fixed>;<trigger_id>;<duration>;<author>;<comment>
+        """
+
+        cmd = "SCHEDULE_HOSTGROUP_SVC_DOWNTIME"
+        dt_cmd_str = self._fmt_dt_str(cmd, hostgroup, minutes)
+        self._write_command(dt_cmd_str)
+
+    def schedule_servicegroup_host_downtime(self, servicegroup, minutes=30):
+        """
+        This command is used to schedule downtime for all hosts in a
+        particular servicegroup.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the hosts.
+
+        Syntax: SCHEDULE_SERVICEGROUP_HOST_DOWNTIME;<servicegroup_name>;
+        <start_time>;<end_time>;<fixed>;<trigger_id>;<duration>;<author>;
+        <comment>
+        """
+
+        cmd = "SCHEDULE_SERVICEGROUP_HOST_DOWNTIME"
+        dt_cmd_str = self._fmt_dt_str(cmd, servicegroup, minutes)
+        self._write_command(dt_cmd_str)
+
+    def schedule_servicegroup_svc_downtime(self, servicegroup, minutes=30):
+        """
+        This command is used to schedule downtime for all services in
+        a particular servicegroup.
+
+        During the specified downtime, Nagios will not send
+        notifications out about the services.
+
+        Note that scheduling downtime for services does not
+        automatically schedule downtime for the hosts those services
+        are associated with.
+
+        Syntax: SCHEDULE_SERVICEGROUP_SVC_DOWNTIME;<servicegroup_name>;
+        <start_time>;<end_time>;<fixed>;<trigger_id>;<duration>;<author>;
+        <comment>
+        """
+
+        cmd = "SCHEDULE_SERVICEGROUP_SVC_DOWNTIME"
+        dt_cmd_str = self._fmt_dt_str(cmd, servicegroup, minutes)
+        self._write_command(dt_cmd_str)
+
+    def disable_host_svc_notifications(self, host):
+        """
+        This command is used to prevent notifications from being sent
+        out for all services on the specified host.
+
+        Note that this command does not disable notifications from
+        being sent out about the host.
+
+        Syntax: DISABLE_HOST_SVC_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = "DISABLE_HOST_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, host)
+        self._write_command(notif_str)
+
+    def disable_host_notifications(self, host):
+        """
+        This command is used to prevent notifications from being sent
+        out for the specified host.
+
+        Note that this command does not disable notifications for
+        services associated with this host.
+
+        Syntax: DISABLE_HOST_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = "DISABLE_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, host)
+        self._write_command(notif_str)
+
+    def disable_svc_notifications(self, host, services=[]):
+        """
+        This command is used to prevent notifications from being sent
+        out for the specified service.
+
+        Note that this command does not disable notifications from
+        being sent out about the host.
+
+        Syntax: DISABLE_SVC_NOTIFICATIONS;<host_name>;<service_description>
+        """
+
+        cmd = "DISABLE_SVC_NOTIFICATIONS"
+        for service in services:
+            notif_str = self._fmt_notif_str(cmd, host, svc=service)
+            self._write_command(notif_str)
+
+    def disable_servicegroup_host_notifications(self, servicegroup):
+        """
+        This command is used to prevent notifications from being sent
+        out for all hosts in the specified servicegroup.
+
+        Note that this command does not disable notifications for
+        services associated with hosts in this service group.
+
+        Syntax: DISABLE_SERVICEGROUP_HOST_NOTIFICATIONS;<servicegroup_name>
+        """
+
+        cmd = "DISABLE_SERVICEGROUP_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, servicegroup)
+        self._write_command(notif_str)
+
+    def disable_servicegroup_svc_notifications(self, servicegroup):
+        """
+        This command is used to prevent notifications from being sent
+        out for all services in the specified servicegroup.
+
+        Note that this does not prevent notifications from being sent
+        out about the hosts in this servicegroup.
+
+        Syntax: DISABLE_SERVICEGROUP_SVC_NOTIFICATIONS;<servicegroup_name>
+        """
+
+        cmd = "DISABLE_SERVICEGROUP_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, servicegroup)
+        self._write_command(notif_str)
+
+    def disable_hostgroup_host_notifications(self, hostgroup):
+        """
+        Disables notifications for all hosts in a particular
+        hostgroup.
+
+        Note that this does not disable notifications for the services
+        associated with the hosts in the hostgroup - see the
+        DISABLE_HOSTGROUP_SVC_NOTIFICATIONS command for that.
+
+        Syntax: DISABLE_HOSTGROUP_HOST_NOTIFICATIONS;<hostgroup_name>
+        """
+
+        cmd = "DISABLE_HOSTGROUP_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, hostgroup)
+        self._write_command(notif_str)
+
+    def disable_hostgroup_svc_notifications(self, hostgroup):
+        """
+        Disables notifications for all services associated with hosts
+        in a particular hostgroup.
+
+        Note that this does not disable notifications for the hosts in
+        the hostgroup - see the DISABLE_HOSTGROUP_HOST_NOTIFICATIONS
+        command for that.
+
+        Syntax: DISABLE_HOSTGROUP_SVC_NOTIFICATIONS;<hostgroup_name>
+        """
+
+        cmd = "DISABLE_HOSTGROUP_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, hostgroup)
+        self._write_command(notif_str)
+
+    def enable_host_notifications(self, host):
+        """
+        Enables notifications for a particular host.
+
+        Note that this command does not enable notifications for
+        services associated with this host.
+
+        Syntax: ENABLE_HOST_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = "ENABLE_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, host)
+        self._write_command(notif_str)
+
+    def enable_host_svc_notifications(self, host):
+        """
+        Enables notifications for all services on the specified host.
+
+        Note that this does not enable notifications for the host.
+
+        Syntax: ENABLE_HOST_SVC_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = "ENABLE_HOST_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, host)
+        nagios_return = self._write_command(notif_str)
+
+        if nagios_return:
+            return notif_str
+        else:
+            return "Fail: could not write to the command file"
+
+    def enable_svc_notifications(self, host, services=[]):
+        """
+        Enables notifications for a particular service.
+
+        Note that this does not enable notifications for the host.
+
+        Syntax: ENABLE_SVC_NOTIFICATIONS;<host_name>;<service_description>
+        """
+
+        cmd = "ENABLE_SVC_NOTIFICATIONS"
+        nagios_return = True
+        return_str_list = []
+        for service in services:
+            notif_str = self._fmt_notif_str(cmd, host, svc=service)
+            nagios_return = self._write_command(notif_str) and nagios_return
+            return_str_list.append(notif_str)
+
+        if nagios_return:
+            return return_str_list
+        else:
+            return "Fail: could not write to the command file"
+
+    def enable_hostgroup_host_notifications(self, hostgroup):
+        """
+        Enables notifications for all hosts in a particular hostgroup.
+
+        Note that this command does not enable notifications for
+        services associated with the hosts in this hostgroup.
+
+        Syntax: ENABLE_HOSTGROUP_HOST_NOTIFICATIONS;<hostgroup_name>
+        """
+
+        cmd = "ENABLE_HOSTGROUP_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, hostgroup)
+        nagios_return = self._write_command(notif_str)
+
+        if nagios_return:
+            return notif_str
+        else:
+            return "Fail: could not write to the command file"
+
+    def enable_hostgroup_svc_notifications(self, hostgroup):
+        """
+        Enables notifications for all services that are associated
+        with hosts in a particular hostgroup.
+
+        Note that this does not enable notifications for the hosts in
+        this hostgroup.
+
+        Syntax: ENABLE_HOSTGROUP_SVC_NOTIFICATIONS;<hostgroup_name>
+        """
+
+        cmd = "ENABLE_HOSTGROUP_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, hostgroup)
+        nagios_return = self._write_command(notif_str)
+
+        if nagios_return:
+            return notif_str
+        else:
+            return "Fail: could not write to the command file"
+
+    def enable_servicegroup_host_notifications(self, servicegroup):
+        """
+        Enables notifications for all hosts that have services that
+        are members of a particular servicegroup.
+
+        Note that this command does not enable notifications for
+        services associated with the hosts in this servicegroup.
+
+        Syntax: ENABLE_SERVICEGROUP_HOST_NOTIFICATIONS;<servicegroup_name>
+        """
+
+        cmd = "ENABLE_SERVICEGROUP_HOST_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, servicegroup)
+        nagios_return = self._write_command(notif_str)
+
+        if nagios_return:
+            return notif_str
+        else:
+            return "Fail: could not write to the command file"
+
+    def enable_servicegroup_svc_notifications(self, servicegroup):
+        """
+        Enables notifications for all services that are members of a
+        particular servicegroup.
+
+        Note that this does not enable notifications for the hosts in
+        this servicegroup.
+
+        Syntax: ENABLE_SERVICEGROUP_SVC_NOTIFICATIONS;<servicegroup_name>
+        """
+
+        cmd = "ENABLE_SERVICEGROUP_SVC_NOTIFICATIONS"
+        notif_str = self._fmt_notif_str(cmd, servicegroup)
+        nagios_return = self._write_command(notif_str)
+
+        if nagios_return:
+            return notif_str
+        else:
+            return "Fail: could not write to the command file"
+
+    def silence_host(self, host):
+        """
+        This command is used to prevent notifications from being sent
+        out for the host and all services on the specified host.
+
+        This is equivalent to calling disable_host_svc_notifications
+        and disable_host_notifications.
+
+        Syntax: DISABLE_HOST_SVC_NOTIFICATIONS;<host_name>
+        Syntax: DISABLE_HOST_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = [
+            "DISABLE_HOST_SVC_NOTIFICATIONS",
+            "DISABLE_HOST_NOTIFICATIONS"
+            ]
+        nagios_return = True
+        return_str_list = []
+        for c in cmd:
+            notif_str = self._fmt_notif_str(c, host)
+            nagios_return = self._write_command(notif_str) and nagios_return
+            return_str_list.append(notif_str)
+
+        if nagios_return:
+            return return_str_list
+        else:
+            return "Fail: could not write to the command file"
+
+    def unsilence_host(self, host):
+        """
+        This command is used to enable notifications for the host and
+        all services on the specified host.
+
+        This is equivalent to calling enable_host_svc_notifications
+        and enable_host_notifications.
+
+        Syntax: ENABLE_HOST_SVC_NOTIFICATIONS;<host_name>
+        Syntax: ENABLE_HOST_NOTIFICATIONS;<host_name>
+        """
+
+        cmd = [
+            "ENABLE_HOST_SVC_NOTIFICATIONS",
+            "ENABLE_HOST_NOTIFICATIONS"
+            ]
+        nagios_return = True
+        return_str_list = []
+        for c in cmd:
+            notif_str = self._fmt_notif_str(c, host)
+            nagios_return = self._write_command(notif_str) and nagios_return
+            return_str_list.append(notif_str)
+
+        if nagios_return:
+            return return_str_list
+        else:
+            return "Fail: could not write to the command file"
+
+    def act(self):
+        """
+        Figure out what you want to do from ansible, and then do the
+        needful (at the earliest).
+        """
+        # host or service downtime?
+        if self.action == 'downtime':
+            if self.services == 'host':
+                self.schedule_host_downtime(self.host, self.minutes)
+            else:
+                self.schedule_svc_downtime(self.host,
+                                           services=self.services,
+                                           minutes=self.minutes)
+
+        # toggle the host AND service alerts
+        elif self.action == 'silence':
+            self.silence_host(self.host)
+
+        elif self.action == 'unsilence':
+            self.unsilence_host(self.host)
+
+        # toggle host/svc alerts
+        elif self.action == 'enable_alerts':
+            if self.services == 'host':
+                self.enable_host_notifications(self.host)
+            else:
+                self.enable_svc_notifications(self.host,
+                                              services=self.services)
+
+        elif self.action == 'disable_alerts':
+            if self.services == 'host':
+                self.disable_host_notifications(self.host)
+            else:
+                self.disable_svc_notifications(self.host,
+                                               services=self.services)
+        # wtf?
+        else:
+            self.module.fail_json(msg="unknown action specified: '%s'" % \
+                                      self.action)
+
+        self.module.exit_json(nagios_commands=self.command_results,
+                              changed=True)
+
+######################################################################
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/ohai

@@ -1,4 +1,5 @@
-#!/bin/bash
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -18,4 +19,25 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-/usr/bin/ohai
+def get_ohai_data():
+    p = subprocess.Popen(["/usr/bin/env", "ohai"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return rc, out, err
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict()
+    )
+    rc, out, err = get_ohai_data()
+    if rc != 0:
+        module.fail_json(msg=err)
+    else:
+        module.exit_json(**json.loads(out))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+
+

library/ping

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -17,9 +18,13 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
+def main():
+    module = AnsibleModule(
+        argument_spec = dict()
+    )
+    module.exit_json(ping='pong')
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()
 
-print json.dumps({ "ping" : "pong" })

library/pip

@@ -0,0 +1,170 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Matt Wright <matt@nobien.net>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+def _get_full_name(name, version=None):
+    if version is None:
+        resp = name
+    else:
+        resp = name + '==' + version
+    return resp
+
+def _ensure_virtualenv(module, env, virtualenv):
+    if os.path.exists(os.path.join(env, 'bin', 'activate')):
+        return 0, '', ''
+    else:
+        return _run('%s %s' % (virtualenv, env))
+
+
+def _is_package_installed(name, pip, version=None):
+    rc, status_stdout, status_stderr = _run('%s freeze' % pip)
+    return _get_full_name(name, version).lower() in status_stdout.lower()
+
+
+def _did_install(out):
+    return 'Successfully installed' in out
+
+
+def _run(cmd):
+    # returns (rc, stdout, stderr) from shell command
+    process = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                               stderr=subprocess.PIPE, shell=True)
+    stdout, stderr = process.communicate()
+    return (process.returncode, stdout, stderr)
+
+
+def main():
+    pip = None
+    virtualenv = None
+    env = None
+
+    arg_spec = dict(
+        state=dict(default='present', choices=['absent', 'present', 'latest']),
+        name=dict(default=None, required=False),
+        version=dict(default=None, required=False),
+        requirements=dict(default=None, required=False),
+        virtualenv=dict(default=None, required=False)
+    )
+
+    module = AnsibleModule(
+        argument_spec=arg_spec,
+        required_one_of=[['name','requirements']],
+        mutually_exclusive=[['name','requirements']],
+    )
+
+    rc = 0
+    err = ''
+    out = ''
+
+    env = module.params['virtualenv']
+
+    if env:
+        virtualenv = module.get_bin_path('virtualenv', True)
+
+        rc_venv, out_venv, err_venv = _ensure_virtualenv(module, env, virtualenv)
+
+        rc += rc_venv
+        out += out_venv
+        err += err_venv
+
+    pip = module.get_bin_path('pip', True, ['%s/bin' % env])
+
+    state = module.params['state']
+    name = module.params['name']
+    version = module.params['version']
+    requirements = module.params['requirements']
+    command_map = dict(present='install', absent='uninstall', latest='install')
+
+    if state == 'latest' and version is not None:
+        module.fail_json(msg='version is incompatible with state=latest')
+
+    if state == 'latest' and requirements is not None:
+        module.fail_json(msg='requirements is incompatible with state=latest')
+
+    if name is not None and '=' in name:
+        module.fail_json(msg='versions must be specified in the version= parameter')
+
+    cmd = None
+    installed = None
+
+    if requirements:
+
+        cmd = '%s %s -r %s --use-mirrors' % (pip, command_map[state], requirements)
+        rc_pip, out_pip, err_pip = _run(cmd)
+
+        rc += rc_pip
+        out += out_pip
+        err += err_pip
+
+        changed = ((_did_install(out) and state == 'present') or
+                   (not _did_install(out) and state == 'absent'))
+
+    if name and state == 'latest':
+
+        cmd = '%s %s %s --upgrade' % (pip, command_map[state], name)
+        rc_pip, out_pip, err_pip = _run(cmd)
+
+        rc += rc_pip
+        out += out_pip
+        err += err_pip
+
+        changed = 'Successfully installed' in out_pip
+
+    elif name:
+
+        installed = _is_package_installed(name, pip, version)
+        changed = ((installed and state == 'absent') or
+                   (not installed and state == 'present'))
+
+        if changed:
+            if state == 'present':
+                full_name = _get_full_name(name, version)
+            else:
+                full_name = name
+
+            cmd = '%s %s %s' % (pip, command_map[state], full_name)
+
+            if state == 'absent':
+                cmd = cmd + ' -y'
+            else:
+                cmd = cmd + ' --use-mirrors'
+
+            rc_pip, out_pip, err_pip = _run(cmd)
+            rc  += rc_pip
+            out += out_pip
+            err += err_pip
+
+    if rc != 0:
+        if not out:
+            msg = err
+        elif not err:
+            msg = out
+        else:
+            msg = "stdout: %s\n:stderr: %s" % (out, err)
+        module.fail_json(msg=msg, cmd=cmd)
+
+    module.exit_json(changed=changed, cmd=cmd, name=name, version=version,
+                     state=state, requirements=requirements, virtualenv=env)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()

library/postgresql_db

@@ -0,0 +1,133 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+try:
+    import psycopg2
+except ImportError:
+    postgresqldb_found = False
+else:
+    postgresqldb_found = True
+
+# ===========================================
+# PostgreSQL module specific support methods.
+#
+
+def set_owner(cursor, db, owner):
+    query = "ALTER DATABASE %s OWNER TO %s" % (db, owner)
+    cursor.execute(query)
+    return True
+
+def db_owned_by(cursor, db, user):
+    query = """SELECT * FROM pg_database JOIN pg_user ON datdba = usesysid
+    WHERE usename = %(user)s and datname = %(db)s"""
+    cursor.execute(query, {'db':db, 'user':user})
+    return cursor.rowcount == 1
+
+def db_exists(cursor, db):
+    query = "SELECT * FROM pg_database WHERE datname=%(db)s"
+    cursor.execute(query, {'db': db})
+    return cursor.rowcount == 1
+
+def db_delete(cursor, db):
+    if db_exists(cursor, db):
+        query = "DROP DATABASE %s" % db
+        cursor.execute(query)
+        return True
+    else:
+        return False
+
+def db_create(cursor, db, owner, template, encoding):
+    if not db_exists(cursor, db):
+        if owner:
+            owner = " OWNER %s" % owner
+        if template:
+            template = " TEMPLATE %s" % template
+        if encoding:
+            encoding = " ENCODING '%s'" % encoding
+        query = "CREATE DATABASE %s%s%s%s" % (db, owner, template, encoding)
+        cursor.execute(query)
+        return True
+    elif owner and not db_owned_by(cursor, db, owner):
+        return set_owner(cursor, db, owner)
+    else:
+        return False
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec=dict(
+            login_user=dict(default="postgres"),
+            login_password=dict(default=""),
+            login_host=dict(default=""),
+            db=dict(required=True, aliases=['name']),
+            owner=dict(default=""),
+            template=dict(default=""),
+            encoding=dict(default=""),
+            state=dict(default="present", choices=["absent", "present"]),
+        )
+    )
+
+    if not postgresqldb_found:
+        module.fail_json(msg="the python psycopg2 module is required")
+
+    db = module.params["db"]
+    owner = module.params["owner"]
+    template = module.params["template"]
+    encoding = module.params["encoding"]
+    state = module.params["state"]
+    changed = False
+
+    # To use defaults values, keyword arguments must be absent, so 
+    # check which values are empty and don't include in the **kw
+    # dictionary
+    params_map =  {
+        "login_host":"host",
+        "login_user":"user",
+        "login_password":"password"
+    }
+    kw = dict( (params_map[k], v) for (k, v) in module.params.iteritems() 
+              if k in params_map and v != '' )
+    try:
+        db_connection = psycopg2.connect(database="template1", **kw)
+        # Enable autocommit so we can create databases
+        if psycopg2.__version__ >= '2.4.2':
+            db_connection.autocommit = True
+        else:
+            db_connection.set_isolation_level(psycopg2
+                                              .extensions
+                                              .ISOLATION_LEVEL_AUTOCOMMIT)
+        cursor = db_connection.cursor()
+    except Exception, e:
+        module.fail_json(msg="unable to connect to database: %s" % e)
+
+    try:
+        if state == "absent":
+            changed = db_delete(cursor, db)
+        elif state == "present":
+            changed = db_create(cursor, db, owner, template, encoding)
+    except Exception, e:
+        module.fail_json(msg="Database query failed: %s" % e)
+
+    module.exit_json(changed=changed, db=db)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/postgresql_user

@@ -0,0 +1,289 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import re
+
+try:
+    import psycopg2
+except ImportError:
+    postgresqldb_found = False
+else:
+    postgresqldb_found = True
+
+# ===========================================
+# PostgreSQL module specific support methods.
+#
+
+
+def user_exists(cursor, user):
+    query = "SELECT rolname FROM pg_roles WHERE rolname=%(user)s"
+    cursor.execute(query, {'user': user})
+    return cursor.rowcount > 0
+
+
+def user_add(cursor, user, password):
+    """Create a new user with write access to the database"""
+    query = "CREATE USER %(user)s with PASSWORD '%(password)s'"
+    cursor.execute(query % {"user": user, "password": password})
+    return True
+
+def user_chpass(cursor, user, password):
+    """Change user password"""
+    changed = False
+
+    # Handle passwords.
+    if password is not None:
+        select = "SELECT rolpassword FROM pg_authid where rolname=%(user)s"
+        cursor.execute(select, {"user": user})
+        current_pass_hash = cursor.fetchone()[0]
+        # Not sure how to hash the new password, so we just initiate the
+        # change and check if the hash changed
+        alter = "ALTER USER %(user)s WITH PASSWORD '%(password)s'"
+        cursor.execute(alter % {"user": user, "password": password})
+        cursor.execute(select, {"user": user})
+        new_pass_hash = cursor.fetchone()[0]
+        if current_pass_hash != new_pass_hash:
+            changed = True
+
+    return changed
+
+def user_delete(cursor, user):
+    """Try to remove a user. Returns True if successful otherwise False"""
+    cursor.execute("SAVEPOINT ansible_pgsql_user_delete")
+    try:
+        cursor.execute("DROP USER %s" % user)
+    except:
+        cursor.execute("ROLLBACK TO SAVEPOINT ansible_pgsql_user_delete")
+        cursor.execute("RELEASE SAVEPOINT ansible_pgsql_user_delete")
+        return False
+    
+    cursor.execute("RELEASE SAVEPOINT ansible_pgsql_user_delete")
+    return True
+
+def has_table_privilege(cursor, user, table, priv):
+    query = 'SELECT has_table_privilege(%s, %s, %s)'
+    cursor.execute(query, (user, table, priv))
+    return cursor.fetchone()[0]
+
+def get_table_privileges(cursor, user, table):
+    if '.' in table:
+        schema, table = table.split('.', 1)
+    else:
+        schema = 'public'
+    query = '''SELECT privilege_type FROM information_schema.role_table_grants
+    WHERE grantee=%s AND table_name=%s AND table_schema=%s'''
+    cursor.execute(query, (user, table, schema))
+    return set([x[0] for x in cursor.fetchall()])
+    
+
+def grant_table_privilege(cursor, user, table, priv):
+    prev_priv = get_table_privileges(cursor, user, table)
+    query = 'GRANT %s ON TABLE %s TO %s' % (priv, table, user)
+    cursor.execute(query)
+    curr_priv = get_table_privileges(cursor, user, table)
+    return len(curr_priv) > len(prev_priv)
+
+def revoke_table_privilege(cursor, user, table, priv):
+    prev_priv = get_table_privileges(cursor, user, table)
+    query = 'REVOKE %s ON TABLE %s FROM %s' % (priv, table, user)
+    cursor.execute(query)
+    curr_priv = get_table_privileges(cursor, user, table)
+    return len(curr_priv) < len(prev_priv)
+
+
+def get_database_privileges(cursor, user, db):
+    priv_map = {
+        'C':'CREATE',
+        'T':'TEMPORARY',
+        'c':'CONNECT',
+    }
+    query = 'SELECT datacl FROM pg_database WHERE datname = %s'
+    cursor.execute(query, (db,))
+    datacl = cursor.fetchone()[0]
+    r = re.search('%s=(C?T?c?)/[a-z]+\,?' % user, datacl)
+    if r is None:
+        return []
+    o = []
+    for v in r.group(1):
+        o.append(priv_map[v])
+    return o
+
+def has_database_privilege(cursor, user, db, priv):
+    query = 'SELECT has_database_privilege(%s, %s, %s)'
+    cursor.execute(query, (user, db, priv))
+    return cursor.fetchone()[0]
+
+def grant_database_privilege(cursor, user, db, priv):
+    prev_priv = get_database_privileges(cursor, user, db)
+    query = 'GRANT %s ON DATABASE %s TO %s' % (priv, db, user)
+    cursor.execute(query)
+    curr_priv = get_database_privileges(cursor, user, db)
+    return len(curr_priv) > len(prev_priv)
+
+def revoke_database_privilege(cursor, user, db, priv):
+    prev_priv = get_database_privileges(cursor, user, db)
+    query = 'REVOKE %s ON DATABASE %s FROM %s' % (priv, db, user)
+    cursor.execute(query)
+    curr_priv = get_database_privileges(cursor, user, db)
+    return len(curr_priv) < len(prev_priv)
+
+def revoke_privileges(cursor, user, privs):
+    if privs is None:
+        return False
+
+    changed = False
+    for type_ in privs:
+        revoke_func = {
+            'table':revoke_table_privilege,
+            'database':revoke_database_privilege
+        }[type_]
+        for name, privileges in privs[type_].iteritems():
+            for privilege in privileges:
+                changed = revoke_func(cursor, user, name, privilege)\
+                        or changed
+
+    return changed
+
+def grant_privileges(cursor, user, privs):
+    if privs is None:
+        return False
+
+    changed = False
+    for type_ in privs:
+        grant_func = {
+            'table':grant_table_privilege,
+            'database':grant_database_privilege
+        }[type_]
+        for name, privileges in privs[type_].iteritems():
+            for privilege in privileges:
+                changed = grant_func(cursor, user, name, privilege)\
+                        or changed
+
+    return changed
+
+def parse_privs(privs, db):
+    """
+    Parse privilege string to determine permissions for database db.
+    Format:
+
+        privileges[/privileges/...]
+
+    Where:
+
+        privileges := DATABASE_PRIVILEGES[,DATABASE_PRIVILEGES,...] |
+            TABLE_NAME:TABLE_PRIVILEGES[,TABLE_PRIVILEGES,...]
+    """
+    if privs is None:
+        return privs
+
+    o_privs = {
+        'database':{},
+        'table':{}
+    }
+    for token in privs.split('/'):
+        if ':' not in token:
+            type_ = 'database'
+            name = db
+            priv_set = set(x.strip() for x in token.split(','))
+        else:
+            type_ = 'table'
+            name, privileges = token.split(':', 1)
+            priv_set = set(x.strip() for x in privileges.split(','))
+        
+        o_privs[type_][name] = priv_set
+    
+    return o_privs
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec=dict(
+            login_user=dict(default="postgres"),
+            login_password=dict(default=""),
+            login_host=dict(default=""),
+            user=dict(required=True, aliases=['name']),
+            password=dict(default=None),
+            state=dict(default="present", choices=["absent", "present"]),
+            priv=dict(default=None),
+            db=dict(default=''),
+            fail_on_user=dict(default='yes')
+        )
+    )
+    user = module.params["user"]
+    password = module.params["password"]
+    state = module.params["state"]
+    fail_on_user = module.params["fail_on_user"] == 'yes'
+    db = module.params["db"]
+    if db == '' and module.params["priv"] is not None:
+        module.fail_json(msg="privileges require a database to be specified")
+    privs = parse_privs(module.params["priv"], db)
+
+    if not postgresqldb_found:
+        module.fail_json(msg="the python psycopg2 module is required")
+    
+    # To use defaults values, keyword arguments must be absent, so 
+    # check which values are empty and don't include in the **kw
+    # dictionary
+    params_map =  {
+        "login_host":"host",
+        "login_user":"user",
+        "login_password":"password",
+        "db":"database" 
+    }
+    kw = dict( (params_map[k], v) for (k, v) in module.params.iteritems() 
+              if k in params_map and v != "" )
+    try:
+        db_connection = psycopg2.connect(**kw)
+        cursor = db_connection.cursor()
+    except Exception, e:
+        module.fail_json(msg="unable to connect to database: %s" % e)
+    
+    kw = dict(user=user)
+    changed = False
+    user_removed = False
+    if state == "present":
+        if user_exists(cursor, user):
+            changed = user_chpass(cursor, user, password)
+        else:
+            if password is None:
+                msg = "password parameter required when adding a user"
+                module.fail_json(msg=msg)
+            changed = user_add(cursor, user, password)
+        changed = grant_privileges(cursor, user, privs) or changed
+    else:
+        if user_exists(cursor, user):
+            changed = revoke_privileges(cursor, user, privs)
+            user_removed = user_delete(cursor, user)
+            changed = changed or user_removed
+            if fail_on_user and not user_removed:
+                msg = "unable to remove user"
+                module.fail_json(msg=msg)
+            kw['user_removed'] = user_removed
+
+    if changed:
+        db_connection.commit()
+
+    kw['changed'] = changed
+    module.exit_json(**kw)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/raw

@@ -0,0 +1 @@
+# this is a virtual module that is entirely implemented server side

library/seboolean

@@ -0,0 +1,173 @@
+#!/usr/bin/python
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+try:
+    import selinux
+    HAVE_SELINUX=True
+except ImportError:
+    HAVE_SELINUX=False
+
+try:
+    import semanage
+    HAVE_SEMANAGE=True
+except ImportError:
+    HAVE_SEMANAGE=False
+
+def has_boolean_value(module, name):
+    bools = []
+    try:
+        rc, bools = selinux.security_get_boolean_names()
+    except OSError, e:
+        module.fail_json(msg="Failed to get list of boolean names")
+    if name in bools:
+        return True
+    else:
+        return False
+
+def get_boolean_value(module, name):
+    state = 0
+    try:
+        state = selinux.security_get_boolean_active(name)
+    except OSError, e:
+        module.fail_json(msg="Failed to determine current state for boolean %s" % name)
+    if state == 1:
+        return True
+    else:
+        return False
+
+# The following method implements what setsebool.c does to change
+# a boolean and make it persist after reboot..
+def semanage_boolean_value(module, name, state):
+    rc = 0
+    value = 0
+    if state:
+        value = 1
+    handle = semanage.semanage_handle_create()
+    if handle is None:
+        module.fail_json(msg="Failed to create semanage library handle")
+    try:
+        managed = semanage.semanage_is_managed(handle)
+        if managed < 0:
+            module.fail_json(msg="Failed to determine whether policy is manage")
+        if managed == 0:
+            if os.getuid() == 0:
+                module.fail_json(msg="Cannot set persistent booleans without managed policy")
+            else:
+                module.fail_json(msg="Cannot set persistent booleans; please try as root")
+        if semanage.semanage_connect(handle) < 0:
+            module.fail_json(msg="Failed to connect to semanage")
+
+        if semanage.semanage_begin_transaction(handle) < 0:
+            module.fail_json(msg="Failed to begin semanage transaction")
+
+        rc, sebool = semanage.semanage_bool_create(handle)
+        if rc < 0:
+            module.fail_json(msg="Failed to create seboolean with semanage")
+        if semanage.semanage_bool_set_name(handle, sebool, name) < 0:
+            module.fail_json(msg="Failed to set seboolean name with semanage")
+        semanage.semanage_bool_set_value(sebool, value)
+
+        rc, boolkey = semanage.semanage_bool_key_extract(handle, sebool)
+        if rc < 0:
+            module.fail_json(msg="Failed to extract boolean key with semanage")
+
+        if semanage.semanage_bool_modify_local(handle, boolkey, sebool) < 0:
+            module.fail_json(msg="Failed to modify boolean key with semanage")
+
+        if semanage.semanage_bool_set_active(handle, boolkey, sebool) < 0:
+            module.fail_json(msg="Failed to set boolean key active with semanage")
+
+        semanage.semanage_bool_key_free(boolkey)
+        semanage.semanage_bool_free(sebool)
+
+        semanage.semanage_set_reload(handle, 0)
+        if semanage.semanage_commit(handle) < 0:
+            module.fail_json(msg="Failed to commit changes to semanage")
+
+        semanage.semanage_disconnect(handle)
+        semanage.semanage_handle_destroy(handle)
+    except Exception, e:
+        module.fail_json(msg="Failed to manage policy for boolean %s: %s" % (name, str(e)))
+    return True
+
+def set_boolean_value(module, name, state):
+    rc = 0
+    value = 0
+    if state:
+        value = 1
+    try:
+        rc = selinux.security_set_boolean(name, value)
+    except OSError, e:
+        module.fail_json(msg="Failed to set boolean %s to %s" % (name, value))
+    if rc == 0:
+        return True
+    else:
+        return False
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            name=dict(required=True),
+            persistent=dict(default='no', choices=BOOLEANS),
+            state=dict(required=True, choices=BOOLEANS)
+        )
+    )
+    
+    if not HAVE_SELINUX:
+        module.fail_json(msg="SELinux not supported on this host.")
+
+    if not HAVE_SEMANAGE:
+        module.fail_json(msg="This module requires libsemanage-python support")
+
+    if not selinux.is_selinux_enabled():
+        module.fail_json(msg="SELinux is disabled on this host.")
+
+    name = module.params['name']
+    persistent = module.boolean(module.params['persistent'])
+    state = module.boolean(module.params['state'])
+    result = {}
+    result['name'] = name
+
+    if not has_boolean_value(module, name):
+        module.fail_json(msg="SELinux boolean %s does not exist." % name)
+
+    cur_value = get_boolean_value(module, name)
+
+    if cur_value == state:
+        result['state'] = cur_value
+        result['changed'] = False
+        module.exit_json(**result)
+
+    if persistent:
+        r = semanage_boolean_value(module, name, state)
+    else:
+        r = set_boolean_value(module, name, state)
+
+    result['changed'] = r
+    if not r:
+        module.fail_json(msg="Failed to set boolean %s to %s" % (name, value))
+    try:
+        selinux.security_commit_booleans()
+    except:
+        module.fail_json(msg="Failed to commit pending boolean %s value" % name)
+    module.exit_json(**result)
+
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()