changelog dates

operations in the correct order, so the file module results overlay the original module results, not the other way
around (which keeps any failure msg's intact)

.gitignore

@@ -3,3 +3,10 @@ build
 # Emacs backup files...
 *~
 .\#*
+# (s)rpm building stuff
+MANIFEST
+dist
+rpm-build
+# Eclipse/PyDev stuff...
+.project
+.pydevproject

AUTHORS.md

@@ -1,13 +0,0 @@
-Patches and Contributions 
-=========================
-
-   * Michael DeHaan - michael.dehaan AT gmail DOT com
-   * Jeremy Katz - katzj AT fedoraproject DOT org
-   * Seth Vidal - skvidal AT fedoraproject DOT org
-   * Tim Bielawa - tbielawa AT redhat DOT com
-   * Christopher Johnston - chjohnst AT gmail DOT com 
-
-Send in a github pull request to get your name here.
-
-Upstream: github.com/mpdehaan/ansible
-

CHANGELOG.md

@@ -0,0 +1,181 @@
+Ansible Changes By Release
+==========================
+
+0.6 "Cabo" ------------ pending
+
+playbooks:
+
+* support to tag tasks and includes and use --tags in playbook CLI 
+* playbooks can now include other playbooks (example/playbooks/nested_playbooks.yml)
+* vars_files now usable with with_items, provided file paths don't contain host specific facts
+* error reporting if with_items value is unbound
+* with_items no longer creates lots of tasks, creates one task that makes multiple calls
+* can use host_specific facts inside with_items (see above)
+* at the top level of a playbook, set 'gather_facts: False' to skip fact gathering
+* first_available_file and with_items used together will now raise an error
+* to catch typos, like 'var' for 'vars', playbooks and tasks now yell on invalid parameters
+* automatically load (directory_of_inventory_file)/group_vars/groupname and /host_vars/hostname in vars_files
+* playbook is now colorized, set ANSIBLE_NOCOLOR=1 if you do not like this, does not colorize if not a TTY
+* hostvars now preserved between plays (regression in 0.5 from 0.4), useful for sharing vars in multinode configs
+* ignore_errors: True on a task can be used to allow a task to fail and not stop the play
+* with_items with the apt/yum module will install/remove/update everything in a single command
+
+inventory:
+
+* groups variable available as a hash to return the hosts in each group name
+* in YAML inventory, hosts can list their groups in inverted order now also (see tests/yaml_hosts)
+* YAML inventory is deprecated and will be removed in 0.7
+* ec2 inventory script
+* support ranges of hosts in the host file, like www[001-100].example.com (supports leading zeros and also not)
+
+modules:
+
+* fetch module now does not fail a system when requesting file paths (ex: logs) that don't exist
+* apt module now takes an optional install-recommends=yes|no (default yes)
+* fixes to the return codes of the copy module
+* copy module takes a remote md5sum to avoid large file transfer
+* various user and group module fixes (error handling, etc)
+* apt module now takes an optional force parameter
+* slightly better psychic service status handling for the service module
+* fetch module fixes for SSH connection type
+* modules now consistently all take yes/no for boolean parameters (and DWIM on true/false/1/0/y/n/etc)
+* setup module no longer saves to disk, template module now only used in playbooks
+* setup module no longer needs to run twice per playbook
+* apt module now passes DEBIAN_FRONTEND=noninteractive
+* mount module (manages active mounts + fstab)
+* setup module fixes if no ipv6 support
+* internals: template in common module boilerplate, also causes less SSH operations when used
+* git module fixes
+* setup module overhaul, more modular 
+* minor caching logic added to inventory to reduce hammering of inventory scripts.
+* MySQL and PostgreSQL modules for user and db management
+* vars_prompt now supports private password entry (see examples/playbooks/prompts.yml)
+* yum module modified to be more tolerant of plugins spewing random console messages (ex: RHN)
+
+internals:
+
+* when sudoing to root, still use /etc/ansible/setup as the metadata path, as if root
+* paramiko is now only imported if needed when running from source checkout
+* cowsay support on Ubuntu
+* various ssh connection fixes for old Ubuntu clients
+* ./hacking/test-module now supports options like ansible takes and has a debugger mode
+* sudoing to a user other than root now works more seamlessly (uses /tmp, avoids umask issues)
+
+0.5 "Amsterdam" ------- July 04, 2012
+
+* Service module gets more accurate service states when running with upstart
+* Jinja2 usage in playbooks (not templates), reinstated, supports %include directive
+* support for --connection ssh (supports Kerberos, bastion hosts, etc), requires ControlMaster
+* misc tracebacks replaced with error messages
+* various API/internals refactoring
+* vars can be built from other variables
+* support for exclusion of hosts/groups with "!groupname"
+* various changes to support md5 tool differences for FreeBSD nodes & OS X clients
+* "unparseable" command output shows in command output for easier debugging
+* mktemp is no longer required on remotes (not available on BSD)
+* support for older versions of python-apt in the apt module
+* a new "assemble" module, for constructing files from pieces of files (inspired by Puppet "fragments" idiom)
+* ability to override most default values with ANSIBLE_FOO environment variables
+* --module-path parameter can support multiple directories seperated with the OS path seperator
+* with_items can take a variable of type list
+* ansible_python_interpreter variable available for systems with more than one Python
+* BIOS and VMware "fact" upgrades
+* cowsay is used by ansible-playbook if installed to improve output legibility (try installing it)
+* authorized_key module
+* SELinux facts now sourced from the python selinux library
+* removed module debug option -D
+* added --verbose, which shows output from successful playbook operations
+* print the output of the raw command inside /usr/bin/ansible as with command/shell
+* basic setup module support for Solaris
+* ./library relative to the playbook is always in path so modules can be included in tarballs with playbooks
+
+0.4 "Unchained" ------- May 23, 2012
+
+Internals/Core
+* internal inventory API now more object oriented, parsers decoupled
+* async handling improvements
+* misc fixes for running ansible on OS X (overlord only)
+* sudo improvements, now works much more smoothly
+* sudo to a particular user with -U/--sudo-user, or using 'sudo_user: foo' in a playbook
+* --private-key CLI option to work with pem files
+
+Inventory
+* can use -i host1,host2,host3:port to specify hosts not in inventory (replaces --override-hosts)
+* ansible INI style format can do groups of groups [groupname:children] and group vars [groupname:vars]
+* groups and users module takes an optional system=yes|no on creation (default no)
+* list of hosts in playbooks can be expressed as a YAML list in addition to ; delimited
+
+Playbooks
+* variables can be replaced like ${foo.nested_hash_key.nested_subkey[array_index]}
+* unicode now ok in templates (assumes utf8)
+* able to pass host specifier or group name in to "hosts:" with --extra-vars
+* ansible-pull script and example playbook (extreme scaling, remediation)
+* inventory_hostname variable available that contains the value of the host as ansible knows it
+* variables in the 'all' section can be used to define other variables based on those values
+* 'group_names' is now a variable made available to templates
+* first_available_file feature, see selective_file_sources.yml in examples/playbooks for info
+* --extra-vars="a=2 b=3" etc, now available to inject parameters into playbooks from CLI
+
+Incompatible Changes
+* jinja2 is only usable in templates, not playbooks, use $foo instead
+* --override-hosts removed, can use -i with comma notation (-i "ahost,bhost")
+* modules can no longer include stderr output (paramiko limitation from sudo)
+
+Module Changes
+* tweaks to SELinux implementation for file module
+* fixes for yum module corner cases on EL5
+* file module now correctly returns the mode in octal
+* fix for symlink handling in the file module
+* service takes an enable=yes|no which works with chkconfig or updates-rc.d as appropriate
+* service module works better on Ubuntu
+* git module now does resets and such to work more smoothly on updates
+* modules all now log to syslog
+* enabled=yes|no on a service can be used to toggle chkconfig & updates-rc.d states
+* git module supports branch=
+* service fixes to better detect status using return codes of the service script
+* custom facts provided by the setup module mean no dependency on Ruby, facter, or ohai
+* service now has a state=reloaded
+* raw module for bootstrapping and talking to routers w/o Python, etc
+
+Misc Bugfixes
+* fixes for variable parsing in only_if lines
+* misc fixes to key=value parsing
+* variables with mixed case now legal
+* fix to internals of hacking/test-module development script
+
+
+0.3 "Baluchitherium" -- April 23, 2012
+
+* Packaging for Debian, Gentoo, and Arch
+* Improvements to the apt and yum modules
+* A virt module
+* SELinux support for the file module
+* Ability to use facts from other systems in templates (aka exported
+resources like support)
+* Built in Ansible facts so you don't need ohai, facter, or Ruby
+* tempdir selections that work with noexec mounted /tmp
+* templates happen locally, not remotely, so no dependency on
+python-jinja2 for remote computers
+* advanced inventory format in YAML allows more control over variables
+per host and per group
+* variables in playbooks can be structured/nested versus just a flat namespace
+* manpage upgrades (docs)
+* various bugfixes
+* can specify a default --user for playbooks rather than specifying it
+in the playbook file
+* able to specify ansible port in ansible host file (see docs)
+* refactored Inventory API to make it easier to write scripts using Ansible
+* looping capability for playbooks (with_items)
+* support for using sudo with a password
+* module arguments can be unicode
+* A local connection type, --connection=local,  for use with cron or
+in kickstarts
+* better module debugging with -D
+* fetch module for pulling in files from remote hosts
+* command task supports creates=foo for idempotent semantics, won't
+run if file foo already exists 
+
+0.0.2 and 0.0.1
+
+* Initial stages of project
+

COPYING

@@ -0,0 +1,675 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <http://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<http://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<http://www.gnu.org/philosophy/why-not-lgpl.html>.
+

MANIFEST.in

@@ -1,3 +1,6 @@
-include README.md AUTHORS.md ansible.spec
+include README.md packaging/rpm/ansible.spec COPYING
+include examples/hosts
+include packaging/distutils/setup.py
 recursive-include docs *
+recursive-include library *
 include Makefile

Makefile

@@ -1,23 +1,67 @@
 #!/usr/bin/make
 
+########################################################
+# Makefile for Ansible
+#
+# useful targets:
+#   make sdist ---------------- produce a tarball
+#   make rpm  ----------------- produce RPMs
+#   make deb ------------------ produce a DEB
+#   make docs ----------------- rebuild the manpages (results are checked in)
+#   make tests ---------------- run the tests
+#   make pyflakes, make pep8 -- source code checks  
+
+########################################################
+# variable section
+
+NAME = "ansible"
+
+# Manpages are currently built with asciidoc -- would like to move to markdown
+# This doesn't evaluate until it's called. The -D argument is the
+# directory of the target file ($@), kinda like `dirname`.
 ASCII2MAN = a2x -D $(dir $@) -d manpage -f manpage $<
 ASCII2HTMLMAN = a2x -D docs/html/man/ -d manpage -f xhtml
-MANPAGES := docs/man/man1/ansible.1
+MANPAGES := docs/man/man1/ansible.1 docs/man/man1/ansible-playbook.1
+
 SITELIB = $(shell python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")
 
+# VERSION file provides one place to update the software version
+VERSION := $(shell cat VERSION)
+
+### Get the branch information from git
+ifneq ($(shell which git),)
+GIT_DATE := $(shell git log -n 1 --format="%ai")
+endif
+DATE := $(shell date --date="$(GIT_DATE)" +%Y%m%d%H%M)
+
+# RPM build parameters
+RPMSPECDIR= packaging/rpm
+RPMSPEC = $(RPMSPECDIR)/ansible.spec
+RPMDIST = $(shell rpm --eval '%dist')
+RPMRELEASE = 1
+ifeq ($(OFFICIAL),)
+    RPMRELEASE = 0.git$(DATE)
+endif
+RPMNVR = "$(NAME)-$(VERSION)-$(RPMRELEASE)$(RPMDIST)"
+
+########################################################
+
 all: clean python
 
 tests: 
-	PYTHONPATH=./lib nosetests
+	PYTHONPATH=./lib nosetests -v
 
-docs: manuals
+# To force a rebuild of the docs run 'touch VERSION && make docs'
+docs: $(MANPAGES)
 
-manuals: $(MANPAGES)
+# Regenerate %.1.asciidoc if %.1.asciidoc.in has been modified more
+# recently than %.1.asciidoc.
+%.1.asciidoc: %.1.asciidoc.in
+	sed "s/%VERSION%/$(VERSION)/" $< > $@
 
-%.1: %.1.asciidoc
-	$(ASCII2MAN)
-
-%.5: %.5.asciidoc
+# Regenerate %.1 if %.1.asciidoc or VERSION has been modified more
+# recently than %.1. (Implicitly runs the %.1.asciidoc recipe)
+%.1: %.1.asciidoc VERSION
 	$(ASCII2MAN)
 
 loc:
@@ -27,38 +71,83 @@ pep8:
 	@echo "#############################################"
 	@echo "# Running PEP8 Compliance Tests"
 	@echo "#############################################"
-	pep8 -r --ignore=E501,E221,W291,W391,E302,E251,E203,W293,E231,E303,E201,E225 lib/ bin/
+	pep8 -r --ignore=E501,E221,W291,W391,E302,E251,E203,W293,E231,E303,E201,E225,E261 lib/ bin/
 
 pyflakes:
-	pyflakes lib/ansible/*.py
+	pyflakes lib/ansible/*.py bin/*
 
 clean:
 	@echo "Cleaning up distutils stuff"
-	-rm -rf build
-	-rm -rf dist
+	rm -rf build
+	rm -rf dist
 	@echo "Cleaning up byte compiled python stuff"
-	find . -regex ".*\.py[co]$$" -delete
+	find . -type f -regex ".*\.py[co]$$" -delete
 	@echo "Cleaning up editor backup files"
 	find . -type f \( -name "*~" -or -name "#*" \) -delete
 	find . -type f \( -name "*.swp" \) -delete
 	@echo "Cleaning up asciidoc to man transformations and results"
-	find ./docs/man -type f \( -name "*.xml" -or -regex ".*\.[0-9]$$" \) -delete
+	find ./docs/man -type f -name "*.xml" -delete
+	find ./docs/man -type f -name "*.asciidoc" -delete
 	@echo "Cleaning up output from test runs"
-	-rm -rf test/test_data
-	@echo "Cleaning up RPM stuff"
-	-rm MANIFEST
+	rm -rf test/test_data
+	@echo "Cleaning up RPM building stuff"
+	rm -rf MANIFEST rpm-build
+	@echo "Cleaning up Debian building stuff"
+	rm -rf debian
+	rm -rf deb-build
 
-python: docs
+python:
 	python setup.py build
 
-install: docs
+install:
+	mkdir -p /usr/share/ansible
+	cp ./library/* /usr/share/ansible/
 	python setup.py install
 
-rpm: 
-	python setup.py sdist
-	rpmbuild -ta dist/ansible-1.0.tar.gz
+sdist: clean
+	python setup.py sdist -t MANIFEST.in
 
-.PHONEY: docs manual clean pep8
-vpath %.asciidoc docs/man/man1
+rpmcommon: sdist
+	@mkdir -p rpm-build
+	@cp dist/*.gz rpm-build/
+	@sed -e 's#^Version:.*#Version: $(VERSION)#' -e 's#^Release:.*#Release: $(RPMRELEASE)%{?dist}#' $(RPMSPEC) >rpm-build/$(NAME).spec
 
+srpm: rpmcommon
+	@rpmbuild --define "_topdir %(pwd)/rpm-build" \
+	--define "_builddir %{_topdir}" \
+	--define "_rpmdir %{_topdir}" \
+	--define "_srcrpmdir %{_topdir}" \
+	--define "_specdir $(RPMSPECDIR)" \
+	--define "_sourcedir %{_topdir}" \
+	-bs rpm-build/$(NAME).spec
+	@rm -f rpm-build/$(NAME).spec
+	@echo "#############################################"
+	@echo "Ansible SRPM is built:"
+	@echo "    rpm-build/$(RPMNVR).src.rpm"
+	@echo "#############################################"
+
+rpm: rpmcommon
+	@rpmbuild --define "_topdir %(pwd)/rpm-build" \
+	--define "_builddir %{_topdir}" \
+	--define "_rpmdir %{_topdir}" \
+	--define "_srcrpmdir %{_topdir}" \
+	--define "_specdir $(RPMSPECDIR)" \
+	--define "_sourcedir %{_topdir}" \
+	--define "_rpmfilename $(RPMNVR).%%{ARCH}.rpm" \
+	-ba rpm-build/$(NAME).spec
+	@rm -f rpm-build/$(NAME).spec
+	@echo "#############################################"
+	@echo "Ansible RPM is built:"
+	@echo "    rpm-build/$(RPMNVR).noarch.rpm"
+	@echo "#############################################"
+
+debian: sdist
+deb: debian
+	cp -r packaging/debian ./
+	chmod 755 debian/rules
+	fakeroot debian/rules clean
+	fakeroot dh_install
+	fakeroot debian/rules binary
+
+# for arch or gentoo, read instructions in the appropriate 'packaging' subdirectory directory
 

README.md

@@ -4,7 +4,7 @@ Ansible
 Ansible is a radically simple configuration-management, deployment, task-execution, and 
 multinode orchestration framework.   
 
-Read all about at it at (http://ansible.github.com)
+Read the documentation at http://ansible.github.com
 
 Design Principles
 =================
@@ -24,6 +24,37 @@ Get Involved
    * [ansible-project mailing list](http://groups.google.com/group/ansible-project)
    * irc.freenode.net: #ansible
 
+Branch Info
+===========
+
+   * Releases are named after Van Halen songs.
+   * The master branch corresponds to release 0.5 "Amsterdam".
+   * The devel branch corresponds to release 0.6 "Cabo".
+   * All feature work happens on the development branch. 
+   * Major bug fixes will be made to the master branch, but not minor ones.
+   * See CHANGELOG.md for release notes to track each release.
+
+Patch Instructions
+==================
+
+Contributions to the core and modules are greatly welcome.
+
+   * Required Process:
+       * Submit github pull requests to the "ansible/devel" branch for features
+       * Fixes for bugs may be submitted to "ansible/master"
+       * Make sure "make tests" passes before submitting any requests.
+   * Bonus points:
+       * Joining the mailing list
+       * Fixing bugs instead of sending bug reports.
+       * Using squash merges
+       * Updating the "rst/*" files in the docs project and "docs/" manpage content
+       * Adding more unit tests 
+   * Avoid:
+       * Sending patches to the mailing list directly.
+       * Sending feature pull requests to the 'master' branch instead of the devel branch
+       * Sending pull requests to mpdehaan's personal ansible fork.
+
+
 Author
 ======
 

VERSION

@@ -0,0 +1 @@
+0.6

ansible.spec

@@ -1,50 +0,0 @@
-%{!?python_sitelib: %global python_sitelib %(%{__python} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
-
-Summary: Minimal SSH command and control
-Name: ansible
-Version: 1.0
-Release: 1
-Source0: ansible-%{version}.tar.gz
-License: GPLv3
-Group: Development/Libraries
-BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
-Prefix: %{_prefix}
-BuildArch: noarch
-Url: http://github.com/mpdehaan/ansible/
-BuildRequires: asciidoc
-Requires: python-paramiko
-Requires: python-jinja2
-
-%description
-Ansible is a extra-simple tool/API for doing 'parallel remote things' over SSH
-executing commands, running "modules", or executing larger 'playbooks' that
-can serve as a configuration management or deployment system.
-
-%prep
-%setup -n %{name}-%{version}
-
-%build
-python setup.py build
-make docs
-
-%install
-python setup.py install -O1 --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES
-mkdir -p $RPM_BUILD_ROOT/etc/ansible/
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
-%files
-%doc README.md AUTHORS.md PKG-INFO
-%defattr(-,root,root)
-%{_mandir}/man1/*.gz
-%{_mandir}/man5/*.gz
-%{python_sitelib}/*
-%{_bindir}/ansible*
-%{_datadir}/ansible/*
-%{_sysconfdir}/ansible/
-
-%changelog
-* Mon Mar  5 2012 Seth Vidal <skvidal at fedoraproject.org>
-- spec file
-

bin/ansible

@@ -1,4 +1,4 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -19,20 +19,15 @@
 
 ########################################################
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-from optparse import OptionParser
 import sys
-import os
 import getpass
-import shlex
-import ansible.runner
-import ansible.playbook
+
+from ansible.runner import Runner
 import ansible.constants as C
-from optparse import OptionParser
-from ansible.utils import *
+from ansible import utils
+from ansible import errors
+from ansible import callbacks
+from ansible import inventory
 
 ########################################################
 
@@ -42,38 +37,24 @@ class Cli(object):
     # ----------------------------------------------     
 
     def __init__(self):
-        pass
+        self.stats = callbacks.AggregateStats()
+        self.callbacks = callbacks.CliRunnerCallbacks()
 
     # ----------------------------------------------     
 
     def parse(self):
         ''' create an options parser for bin/ansible '''
-
-        parser = OptionParser(usage = 'ansible <host-pattern> [options]')
-        parser.add_option("-a", "--args", dest="module_args",
-            help="module arguments", default=C.DEFAULT_MODULE_ARGS)
-        parser.add_option("-B", "--background", dest="seconds", default=0,
-            help="run asynchronously, failing after X seconds")
-        parser.add_option('-f','--forks', dest='forks', default=C.DEFAULT_FORKS, type='int',
-            help='number of parallel processes to use')
-        parser.add_option("-i", "--inventory-file", dest="inventory",
-           help="inventory host file", default=C.DEFAULT_HOST_LIST)
-        parser.add_option("-k", "--ask-pass", default=False, action="store_true",
-            help="ask for SSH password")
-        parser.add_option("-M", "--module-path", dest="module_path",
-           help="path to module library", default=C.DEFAULT_MODULE_PATH)
-        parser.add_option("-m", "--module-name", dest="module_name",
-            help="module name to execute", default=C.DEFAULT_MODULE_NAME)
-        parser.add_option('-o', '--one-line', dest='one_line', action='store_true',
-            help="condense output")
-        parser.add_option('-t', '--tree', dest='tree', default=None,
-            help="log output to this directory")
-        parser.add_option('-T', '--timeout', default=C.DEFAULT_TIMEOUT, type='int',
-            dest='timeout', help="set the SSH timeout in seconds")
-        parser.add_option('-u', '--user', default=C.DEFAULT_REMOTE_USER,
-            dest='remote_user', help='connect as this user')
-        options, args = parser.parse_args()
         
+        parser = utils.base_parser(constants=C, runas_opts=True, async_opts=True,
+            output_opts=True, connect_opts=True, usage='%prog <host-pattern> [options]')
+        parser.add_option('-a', '--args', dest='module_args',
+            help="module arguments", default=C.DEFAULT_MODULE_ARGS)
+        parser.add_option('-m', '--module-name', dest='module_name',
+            help="module name to execute (default=%s)" % C.DEFAULT_MODULE_NAME, 
+            default=C.DEFAULT_MODULE_NAME)
+        options, args = parser.parse_args()
+        self.callbacks.options = options        
+
         if len(args) == 0 or len(args) > 1:
             parser.print_help()
             sys.exit(1)
@@ -85,57 +66,69 @@ class Cli(object):
         ''' use Runner lib to do SSH things '''
 
         pattern = args[0]
+
+        inventory_manager = inventory.Inventory(options.inventory)
+        hosts = inventory_manager.list_hosts(pattern)
+        if len(hosts) == 0:
+            print >>sys.stderr, "No hosts matched"
+            sys.exit(1)
+        
         sshpass = None
+        sudopass = None
         if options.ask_pass:
             sshpass = getpass.getpass(prompt="SSH password: ")
+        if options.ask_sudo_pass:
+            sudopass = getpass.getpass(prompt="sudo password: ")
+            options.sudo = True
+        if options.sudo_user:
+            options.sudo = True
+        options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
+        if options.tree:
+            utils.prepare_writeable_dir(options.tree)
 
-        return ansible.runner.Runner(
-            module_name=options.module_name,
-            module_path=options.module_path,
-            module_args=shlex.split(options.module_args),
-            remote_user=options.remote_user,
-            remote_pass=sshpass,
-            host_list=options.inventory,
-            timeout=options.timeout,
-            forks=options.forks,
-            background=options.seconds,
-            pattern=pattern,
-            verbose=True,
-        ).run()
+        runner = Runner(
+            module_name=options.module_name, module_path=options.module_path,
+            module_args=options.module_args,
+            remote_user=options.remote_user, remote_pass=sshpass,
+            inventory=inventory_manager, timeout=options.timeout, 
+            private_key_file=options.private_key_file,
+            forks=options.forks, 
+            pattern=pattern, 
+            callbacks=self.callbacks, sudo=options.sudo, 
+            sudo_pass=sudopass,sudo_user=options.sudo_user,
+            transport=options.connection, verbose=options.verbose
+        )
+
+        if options.seconds:
+            print "background launch...\n\n"
+            results, poller = runner.run_async(options.seconds)
+            results = self.poll_while_needed(poller, options)
+        else:
+            results = runner.run()
+
+        return (runner, results)
 
     # ----------------------------------------------     
 
-    def output(self, results, options, args):
+    def poll_while_needed(self, poller, options):
         ''' summarize results from Runner '''
 
-        if results is None:
-           exit("No hosts matched")
-        if options.tree:
-            prepare_writeable_dir(options.tree)
+        # BACKGROUND POLL LOGIC when -B and -P are specified
+        if options.seconds and options.poll_interval > 0:
+            poller.wait(options.seconds, options.poll_interval)
 
-        buf = '' 
-        for hostname in contacted_hosts(results):
-            msg = host_report_msg(
-                hostname, 
-                options.module_name, 
-                contacted_host_result(results, hostname),
-                options.one_line
-            )
-            if options.tree:
-                write_tree_file(options.tree, hostname, bigjson(results))
-            buf += msg
-            
-        if has_dark_hosts(results):
-            buf += dark_hosts_msg(results)
+        return poller.results
 
-        print buf
 
 ########################################################
 
 if __name__ == '__main__':
     cli = Cli()
     (options, args) = cli.parse()
-    results = cli.run(options, args)
-    cli.output(results, options, args)
-    
+    try:
+        (runner, results) = cli.run(options, args)
+    except errors.AnsibleError, e:
+        # Generic handler for ansible specific errors
+        print "ERROR: %s" % str(e)
+        sys.exit(1)
 

bin/ansible-playbook

@@ -1,4 +1,4 @@
-#!/usr/bin/python -tt
+#!/usr/bin/env python
 # (C) 2012, Michael DeHaan, <michael.dehaan@gmail.com>
 
 # This file is part of Ansible
@@ -18,87 +18,119 @@
 
 #######################################################
 
+import os
 import sys
+import getpass
+
 import ansible.playbook
 import ansible.constants as C
-from ansible.utils import *
-import getpass
-from optparse import OptionParser
+from ansible import errors
+from ansible import callbacks
+from ansible import utils
+from ansible.color import ANSIBLE_COLOR, stringc
 
-#######################################################
+def colorize(lead, num, color):
+    """ Print 'lead' = 'num' in 'color' """
+    if num != 0 and ANSIBLE_COLOR:
+        return "%s%s%-15s" % (stringc(lead, color), stringc("=", color), stringc(str(num), color))
+    else:
+        return "%s=%-4s" % (lead, str(num))
 
-class PlaybookCallbacks(object):
-  
-    def __init__(self):
-        pass
-
-    def set_playbook(self, playbook):
-        self.playbook = playbook
-
-    def on_start(self):
-        print "\n"
-
-    def on_task_start(self, name, is_conditional):
-        print task_start_msg(name, is_conditional)
-
-    def on_unreachable(self, host, msg):
-        print "unreachable: [%s] => %s" % (host, msg)
-
-    def on_failed(self, host, results):
-        print "failed: [%s] => %s\n" % (host, smjson(results))
- 
-    def on_ok(self, host):
-        print "ok: [%s]\n" % (host)
-
-    def on_play_start(self, pattern):
-        print "PLAY [%s] ****************************\n" % pattern
+def hostcolor(host, stats):
+    if ANSIBLE_COLOR:
+        if stats['failures'] != 0 or stats['unreachable'] != 0:
+            return "%-41s" % stringc(host, 'red')
+        elif stats['changed'] != 0:
+            return "%-41s" % stringc(host, 'yellow')
+        else:
+            return "%-41s" % stringc(host, 'green')
+    return "%-30s" % host
 
 
 def main(args):
     ''' run ansible-playbook operations '''
 
     # create parser for CLI options
-    parser = OptionParser()
-    parser.usage = "ans-playbook playbook.yml ..."
-    parser.add_option('-f','--forks', dest='forks', default=C.DEFAULT_FORKS, type='int',
-        help='set the number of forks to start up')
-    parser.add_option("-i", "--inventory-file", dest="inventory",
-        help="inventory host file", default=C.DEFAULT_HOST_LIST)
-    parser.add_option("-k", "--ask-pass", default=False, action="store_true",
-        help="ask for SSH password")
-    parser.add_option("-M", "--module-path", dest="module_path",
-        help="path to module library", default=C.DEFAULT_MODULE_PATH)
+    usage = "%prog playbook.yml"
+    parser = utils.base_parser(constants=C, usage=usage, connect_opts=True, runas_opts=True)
+    parser.add_option('-e', '--extra-vars', dest="extra_vars", default=None,
+        help="set additional key=value variables from the CLI")
+    parser.add_option('-t', '--tags', dest='tags', default='all',
+        help="only run plays and tasks tagged with these values")
 
     options, args = parser.parse_args(args)
 
     if len(args) == 0:
-        print >> sys.stderr, "playbook path is a required argument"
+        parser.print_help(file=sys.stderr)
         return 1
 
     sshpass = None
+    sudopass = None
     if options.ask_pass:
         sshpass = getpass.getpass(prompt="SSH password: ")
+    if options.ask_sudo_pass:
+        sudopass = getpass.getpass(prompt="sudo password: ")
+        options.sudo = True
+    if options.sudo_user:
+        options.sudo = True
+    options.sudo_user = options.sudo_user or C.DEFAULT_SUDO_USER
+    extra_vars = utils.parse_kv(options.extra_vars)
+    only_tags = options.tags.split(",")
 
     # run all playbooks specified on the command line
     for playbook in args:
+
+        stats = callbacks.AggregateStats()
+        playbook_cb = callbacks.PlaybookCallbacks(verbose=options.verbose)
+        runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=options.verbose)
+
         pb = ansible.playbook.PlayBook(
             playbook=playbook,
-            host_list=options.inventory,
             module_path=options.module_path,
-            forks=options.forks,
-            verbose=True,
-            remote_pass=sshpass,
-            callbacks=PlaybookCallbacks()
+            host_list=options.inventory, 
+            forks=options.forks, 
+            verbose=options.verbose, 
+            remote_user=options.remote_user,
+            remote_pass=sshpass, 
+            callbacks=playbook_cb, 
+            runner_callbacks=runner_cb, 
+            stats=stats,
+            timeout=options.timeout, 
+            transport=options.connection,
+            sudo=options.sudo,
+            sudo_user=options.sudo_user,
+            sudo_pass=sudopass,
+            extra_vars=extra_vars,
+            private_key_file=options.private_key_file,
+            only_tags=only_tags,
         )
-        pb.run()
+        try:
+
+            pb.run()
+            hosts = sorted(pb.stats.processed.keys())
+            print callbacks.banner("PLAY RECAP")
+            for h in hosts:
+                t = pb.stats.summarize(h)
+		print "%-30s : %s %s %s %s " % (
+			hostcolor(h, t),
+			colorize('ok', t['ok'], 'green'),
+			colorize('changed', t['changed'], 'yellow'),
+			colorize('unreachable', t['unreachable'], 'red'),
+			colorize('failed', t['failures'], 'red'))
+
+            print "\n"
+
+        except errors.AnsibleError, e:
+            print >>sys.stderr, "ERROR: %s" % e
+            return 1
 
     return 0
 
 
 if __name__ == "__main__":
-    sys.exit(main(sys.argv[1:]))
-
-
-
-
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except errors.AnsibleError, e:
+        print >>sys.stderr, "ERROR: %s" % e
+        sys.exit(1)
 

bin/ansible-pull

@@ -0,0 +1,75 @@
+#!/usr/bin/env python
+
+# ansible-pull is a script that runs ansible in local mode
+# after checking out a playbooks directory from git.  There is an
+# example playbook to bootstrap this script in the examples/ dir which
+# installs ansible and sets it up to run on cron.
+#
+# usage: 
+#   ansible-pull -d /var/ansible/local -U http://wherever/content.git -C production
+#
+# the git repo must contain a playbook named 'local.yml'
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import subprocess
+import sys
+from optparse import OptionParser
+
+DEFAULT_PLAYBOOK = 'local.yml'
+
+def _run(cmd):
+    cmd = subprocess.Popen(cmd, shell=True,
+                           stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    print out
+    if cmd.returncode != 0:
+        print err
+    return cmd.returncode
+
+def main(args):
+    """ Set up and run a local playbook """
+    usage = "%prog [options]"
+    parser = OptionParser()
+    parser.add_option('-d', '--directory', dest='dest', default=None,
+                      help='Directory to checkout git repository')
+    parser.add_option('-U', '--url', dest='url',
+                      default=None,
+                      help='URL of git repository')
+    parser.add_option('-C', '--checkout', dest='checkout',
+                      default="HEAD",
+                      help='Branch/Tag/Commit to checkout.  Defaults to HEAD.')
+    options, args = parser.parse_args(args)
+
+    git_opts = "repo=%s dest=%s version=%s" % (options.url, options.dest, options.checkout)
+    cmd = 'ansible all -c local -m git -a "%s"' % git_opts
+    print "cmd=%s" % cmd
+    rc = _run(cmd)
+    if rc != 0:
+        return rc
+
+    os.chdir(options.dest)
+    cmd = 'ansible-playbook -c local %s' % DEFAULT_PLAYBOOK
+    rc = _run(cmd)
+    return rc
+
+if __name__ == '__main__':
+    try:
+        sys.exit(main(sys.argv[1:]))
+    except KeyboardInterrupt, e:
+        print >>sys.stderr, "Exit on user request.\n"
+        sys.exit(1)

docs/comparisons.md

@@ -1,203 +0,0 @@
-Comparisons
-===========
-
-Confused about how Ansible fits in?  Here's a comparison with some common tools.
-
-Accuracy is important, so corrections are VERY welcome if we've got something wrong.
-For space reasons, we can't list everybody's favorite management tool.
-
-Ansible mainly tries to be different in it's goals of minimalism, while also cutting across
-the categories of configuration management, deployment, and ad-hoc task firing.
-
-It strives to be a super-easy to use app, requiring no daemons or databases, and having
-a super-simple configuration management language that takes minutes to learn.
-
-Generally speaking, Ansible is answer to the question "why is the world's systems
-management software so complicated?"  The code does not try to show off. It is an 
-experiment in radical simplicity and encourages anyone to be able to easily contribute to it.
-
-<table>
-   <tr>
-      <td></td>
-      <td><B><U>Ansible</U></B></font></td>
-      <td>Puppet</td>
-      <td>Chef</td>
-      <td>Func</td>
-      <td>Capistrano</td>
-      <td>mCollective</td>
-   </tr>
-   <tr>
-      <td>Purpose</td>
-      <td><B>Config, Deployment, Ad-Hoc</B></td>
-      <td>Config, Deployment</td>
-      <td>Config, Deployment</td>
-      <td>Ad-Hoc</td>
-      <td>Deployment</td> 
-      <td>Ad-Hoc</td>
-   </tr>
-   <tr>
-      <td>Inspirations</td>
-      <td>Cobbler, Puppet, Func, Taboot</B></td>
-      <td>cfengine</td>
-      <td>puppet</td>
-      <td>sysadmin SSH scripts</td>
-      <td>sysadmin SSH scripts</td> 
-      <td>Func, puppet</td>
-   </tr>
-
-   <tr>
-      <td>Config Language</td>
-      <td><B>Simple YAML format</B></td>
-      <td>Custom DSL</td>
-      <td>Ruby code</td>
-      <td>None</td>
-      <td>None</td>
-      <td>None</td>
-   </tr>
-   <tr>
-      <td>Config Language Style</td>
-      <td><B>Very simple</B></td>
-      <td>Very rich, complex</td>
-      <td>Very rich, complex</td>
-      <td>None</td>
-      <td>None</td>
-      <td>None</td>
-   </tr>
-   <tr>
-      <td>Config Ordering</td>
-      <td><B>Simply ordered, with notifiers</B></td>
-      <td>Strict dependency DAG</td>
-      <td>Simply ordered</td>
-      <td>None</td>
-      <td>Simply ordered</td>
-      <td>None</td>
-   </tr>
-   <tr>
-      <td>Communication</td>
-      <td>SSH push</td>
-      <td>SSL pull or push trigger</td>
-      <td>SSL pull or push trigger(?)</td>
-      <td>SSL push</td>
-      <td>SSH</td>
-      <td>AMQP bus</td>
-   </tr>
-   <tr>
-      <td>Daemons Required?</td>
-      <td><B>no</B></td>
-      <td>yes</td>
-      <td>yes</td>
-      <td>yes</td>
-      <td><B>no</B></td>
-      <td>yes</td>
-   </tr>
-   <tr>
-      <td>Database Needed?</td>
-      <td><B>no</B></td>
-      <td>some features</td>
-      <td>yes</td>
-      <td><B>no</B></td>
-      <td><B>no</B></td>
-      <td>some features</td>
-   </tr>
-   <tr>
-      <td>Message Bus Required?</td>
-      <td><B>no</B></td>
-      <td><B>no</B></td>
-      <td>yes</td>
-      <td><B>no</B></td>
-      <td><B>no</B></td>
-      <td>yes</td>
-   </tr>
-   <tr>
-      <td>Inventory Features</td>
-      <td>planned</td>
-      <td><B>yes</B></td>
-      <td><B>yes?</B></td>
-      <td>some</td>
-      <td>no</td>
-      <td>no</td>
-   </tr>
-   <tr>
-      <td>Implemented In</td>
-      <td>Python</td>
-      <td>Ruby</td>
-      <td>Ruby, Erlang</td>
-      <td>Python</td>
-      <td>Ruby</td>
-      <td>Ruby</td>
-   </tr>
-   <tr>
-      <td>Extensible In</td>
-      <td><B>Anything</B></td>
-      <td>Ruby</td>
-      <td>Ruby</td>
-      <td>Python</td>
-      <td>Ruby</td>
-      <td>Ruby</td>
-   </tr>
-   <tr>
-      <td>Codebase Size</td>
-      <td><B>Small</B></td>
-      <td>Large</td>
-      <td>Large</td>
-      <td>Medium</td>
-      <td>Medium</td>
-      <td>Medium</td>
-   </tr>
-   <tr>
-      <td>Module Support</td>
-      <td>Emerging</td>
-      <td><B>Wide/Established</B></td>
-      <td><B>Wide/Established</B></td>
-      <td>Medium/Established</td>
-      <td>Poor</td>
-      <td>Medium/Established</td>
-   </tr>
-   <tr>
-      <td>Users Targeted</td>
-      <td><B>Sysadmins, developers, QA, web admins</B></td>
-      <td>Sysadmins</td>
-      <td>Sysadmins, web admins</td>
-      <td>Sysadmins, developers</td>
-      <td>Web admins</td>
-      <td>Sysadmins</td>
-   </tr>
-   <tr>
-      <td>Learning & Setup Curve</td>
-      <td><B>Minimal<B></td>
-      <td>High</td>
-      <td>High</td> 
-      <td>Medium</td>
-      <td>Medium</td>
-      <td>Medium</td>
-   </tr>
-   <tr>
-      <td>Can Easily Build Applications On It</td>
-      <td><B>Yes</B></td>
-      <td>No</td>
-      <td>No</td>
-      <td><B>Yes</B></td>
-      <td>No</td>
-      <td>Yes?</td>
-   </tr>
-   <tr>
-      <td>Parallelism & Scaling Tech</td>
-      <td>Fork/Merge</td>
-      <td>Add Servers / Staged Commands / No Server</td>
-      <td>Adding More Erlang</td>
-      <td>Fork/Merge</td>
-      <td>No</td>
-      <td>Message Bus</td>
-   </tr>
-   <tr>
-      <td>Delegated Hierachies</td>
-      <td>No</td>
-      <td>No</td>
-      <td>No</td>
-      <td><B>Yes</B></td>
-      <td>No</td>
-      <td><B>Yes</B></td>
-   </tr>
-</td>
-
-

docs/man/.gitignore

@@ -1,2 +1,2 @@
 *.xml
-*.[0-9]
+*.asciidoc

docs/man/man1/ansible-playbook.1

@@ -0,0 +1,113 @@
+'\" t
+.\"     Title: ansible-playbook
+.\"    Author: [see the "AUTHOR" section]
+.\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
+.\"      Date: 07/24/2012
+.\"    Manual: System administration commands
+.\"    Source: Ansible 0.6
+.\"  Language: English
+.\"
+.TH "ANSIBLE\-PLAYBOOK" "1" "07/24/2012" "Ansible 0\&.6" "System administration commands"
+.\" -----------------------------------------------------------------
+.\" * set default formatting
+.\" -----------------------------------------------------------------
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.\" -----------------------------------------------------------------
+.\" * MAIN CONTENT STARTS HERE *
+.\" -----------------------------------------------------------------
+.SH "NAME"
+ansible-playbook \- run an ansible playbook
+.SH "SYNOPSIS"
+.sp
+ansible\-playbook <filename\&.yml> \&... [options]
+.SH "DESCRIPTION"
+.sp
+\fBAnsible playbooks\fR are a configuration and multinode deployment system\&. Ansible\-playbook is the tool used to run them\&. See the project home page (link below) for more information\&.
+.SH "ARGUMENTS"
+.PP
+\fBfilename\&.yml\fR
+.RS 4
+The names of one or more YAML format files to run as ansible playbooks\&.
+.RE
+.SH "OPTIONS"
+.sp
+\fB\-v\fR, \fB\-\-verbose\fR
+.sp
+Verbose mode, more output from successful actions will be shown
+.PP
+\fB\-i\fR \fIPATH\fR, \fB\-\-inventory=\fR\fIPATH\fR
+.RS 4
+The
+\fIPATH\fR
+to the inventory hosts file, which defaults to
+\fI/etc/ansible/hosts\fR\&.
+.RE
+.PP
+\fB\-M\fR \fIDIRECTORY\fR, \fB\-\-module\-path=\fR\fIDIRECTORY\fR
+.RS 4
+The
+\fIDIRECTORY\fR
+to load modules from\&. The default is
+\fI/usr/share/ansible\fR\&.
+.RE
+.PP
+\fB\-e\fR \fIVARS\fR, \fB\-\-extra\-vars=\fR\fIVARS\fR
+.RS 4
+Extra variables to inject into a playbook, in key=value key=value format\&.
+.RE
+.PP
+\fB\-f\fR \fINUM\fR, \fB\-\-forks=\fR\fINUM\fR
+.RS 4
+Level of parallelism\&.
+\fINUM\fR
+is specified as an integer, the default is 5\&.
+.RE
+.PP
+\fB\-k\fR, \fB\-\-ask\-pass\fR
+.RS 4
+Prompt for the SSH password instead of assuming key\-based authentication with ssh\-agent\&.
+.RE
+.PP
+\fB\-K\fR, \fB\-\-ask\-sudo\-pass\fR
+.RS 4
+Prompt for the password to use for playbook plays that request sudo access, if any
+.RE
+.PP
+\fB\-T\fR \fISECONDS\fR, \fB\-\-timeout=\fR\fISECONDS\fR
+.RS 4
+Connection timeout to use when trying to talk to hosts, in
+\fISECONDS\fR\&.
+.RE
+.PP
+\fB\-s\fR, \fB\-\-sudo\fR
+.RS 4
+Force all plays to use sudo, even if not marked as such\&.
+.RE
+.PP
+\fB\-u\fR \fIUSERNAME\fR, \fB\-\-remote\-user=\fR\fIUSERNAME\fR
+.RS 4
+Use this remote user name on playbook steps that do not indicate a user name to run as\&.
+.RE
+.SH "ENVIRONMENT"
+.sp
+The following environment variables may specified\&.
+.sp
+ANSIBLE_HOSTS  \(em Override the default ansible hosts file
+.sp
+ANSIBLE_LIBRARY \(em Override the default ansible module library path
+.SH "AUTHOR"
+.sp
+Ansible was originally written by Michael DeHaan\&. See the AUTHORS file for a complete list of contributors\&.
+.SH "COPYRIGHT"
+.sp
+Copyright \(co 2012, Michael DeHaan
+.sp
+Ansible is released under the terms of the GPLv3 License\&.
+.SH "SEE ALSO"
+.sp
+\fBansible\fR(1)
+.sp
+Extensive documentation as well as IRC and mailing list info is available on the ansible home page: https://ansible\&.github\&.com/

docs/man/man1/ansible-playbook.1.asciidoc.in

@@ -0,0 +1,118 @@
+ansible-playbook(1)
+===================
+:doctype:manpage
+:man source:   Ansible
+:man version:  %VERSION%
+:man manual:   System administration commands
+
+NAME
+----
+ansible-playbook - run an ansible playbook
+
+
+SYNOPSIS
+--------
+ansible-playbook <filename.yml> ... [options]
+
+
+DESCRIPTION
+-----------
+
+*Ansible playbooks* are a configuration and multinode deployment
+system.  Ansible-playbook is the tool used to run them.  See the
+project home page (link below) for more information.
+
+
+ARGUMENTS
+---------
+
+*filename.yml*::
+
+The names of one or more YAML format files to run as ansible playbooks.
+
+
+OPTIONS
+-------
+
+*-v*, *--verbose*          
+
+Verbose mode, more output from successful actions will be shown
+
+*-i* 'PATH', *--inventory=*'PATH'::
+
+The 'PATH' to the inventory hosts file, which defaults to '/etc/ansible/hosts'.
+
+
+*-M* 'DIRECTORY', *--module-path=*'DIRECTORY'::
+
+The 'DIRECTORY' to load modules from. The default is '/usr/share/ansible'.
+
+*-e* 'VARS', *--extra-vars=*'VARS'::
+
+Extra variables to inject into a playbook, in key=value key=value format.
+
+*-f* 'NUM', *--forks=*'NUM'::
+
+Level of parallelism. 'NUM' is specified as an integer, the default is 5.
+
+
+*-k*, *--ask-pass*::
+
+Prompt for the SSH password instead of assuming key-based authentication with ssh-agent.
+
+
+*-K*, *--ask-sudo-pass*::
+
+Prompt for the password to use for playbook plays that request sudo access, if any
+
+
+*-T* 'SECONDS', *--timeout=*'SECONDS'::
+
+Connection timeout to use when trying to talk to hosts, in 'SECONDS'.
+
+
+*-s*, *--sudo*::
+
+Force all plays to use sudo, even if not marked as such.
+
+
+*-u* 'USERNAME', *--remote-user=*'USERNAME'::
+
+Use this remote user name on playbook steps that do not indicate a user name to run as.
+
+
+ENVIRONMENT
+-----------
+
+The following environment variables may specified.
+
+ANSIBLE_HOSTS  -- Override the default ansible hosts file
+
+ANSIBLE_LIBRARY -- Override the default ansible module library path
+
+
+AUTHOR
+------
+
+Ansible was originally written by Michael DeHaan. See the AUTHORS file
+for a complete list of contributors.
+
+
+COPYRIGHT
+---------
+
+Copyright © 2012, Michael DeHaan
+
+Ansible is released under the terms of the GPLv3 License.
+
+
+SEE ALSO
+--------
+
+*ansible*(1)
+
+Extensive documentation as well as IRC and mailing list info
+is available on the ansible home page: <https://ansible.github.com/>
+
+
+

docs/man/man1/ansible.1

@@ -0,0 +1,170 @@
+'\" t
+.\"     Title: ansible
+.\"    Author: [see the "AUTHOR" section]
+.\" Generator: DocBook XSL Stylesheets v1.75.2 <http://docbook.sf.net/>
+.\"      Date: 07/24/2012
+.\"    Manual: System administration commands
+.\"    Source: Ansible 0.6
+.\"  Language: English
+.\"
+.TH "ANSIBLE" "1" "07/24/2012" "Ansible 0\&.6" "System administration commands"
+.\" -----------------------------------------------------------------
+.\" * set default formatting
+.\" -----------------------------------------------------------------
+.\" disable hyphenation
+.nh
+.\" disable justification (adjust text to left margin only)
+.ad l
+.\" -----------------------------------------------------------------
+.\" * MAIN CONTENT STARTS HERE *
+.\" -----------------------------------------------------------------
+.SH "NAME"
+ansible \- run a command somewhere else
+.SH "SYNOPSIS"
+.sp
+ansible <host\-pattern> [\-f forks] [\-m module_name] [\-a args]
+.SH "DESCRIPTION"
+.sp
+\fBAnsible\fR is an extra\-simple tool/framework/API for doing \'remote things\' over SSH\&.
+.SH "ARGUMENTS"
+.PP
+\fBhost\-pattern\fR
+.RS 4
+A name of a group in the inventory file, a shell\-like glob selecting hosts in inventory file, or any combination of the two separated by semicolons\&.
+.RE
+.SH "OPTIONS"
+.PP
+\fB\-i\fR \fIPATH\fR, \fB\-\-inventory=\fR\fIPATH\fR
+.RS 4
+The
+\fIPATH\fR
+to the inventory hosts file, which defaults to
+\fI/etc/ansible/hosts\fR\&.
+.RE
+.PP
+\fB\-f\fR \fINUM\fR, \fB\-\-forks=\fR\fINUM\fR
+.RS 4
+Level of parallelism\&.
+\fINUM\fR
+is specified as an integer, the default is 5\&.
+.RE
+.PP
+\fB\-m\fR \fINAME\fR, \fB\-\-module\-name=\fR\fINAME\fR
+.RS 4
+Execute the module called
+\fINAME\fR\&.
+.RE
+.PP
+\fB\-M\fR \fIDIRECTORY\fR, \fB\-\-module\-path=\fR\fIDIRECTORY\fR
+.RS 4
+The
+\fIDIRECTORY\fR
+to load modules from\&. The default is
+\fI/usr/share/ansible\fR\&.
+.RE
+.PP
+\fB\-a\fR \'\fIARGUMENTS\fR\', \fB\-\-args=\fR\'\fIARGUMENTS\fR\'
+.RS 4
+The
+\fIARGUMENTS\fR
+to pass to the module\&.
+.RE
+.PP
+\fB\-k\fR, \fB\-\-ask\-pass\fR
+.RS 4
+Prompt for the SSH password instead of assuming key\-based authentication with ssh\-agent\&.
+.RE
+.PP
+\fB\-K\fR, \fB\-\-ask\-sudo\-pass\fR
+.RS 4
+Prompt for the password to use with \-\-sudo, if any
+.RE
+.PP
+\fB\-o\fR, \fB\-\-one\-line\fR
+.RS 4
+Try to output everything on one line\&.
+.RE
+.PP
+\fB\-s\fR, \fB\-\-sudo\fR
+.RS 4
+Run the command as the user given by \-u and sudo to root\&.
+.RE
+.PP
+\fB\-t\fR \fIDIRECTORY\fR, \fB\-\-tree=\fR\fIDIRECTORY\fR
+.RS 4
+Save contents in this output
+\fIDIRECTORY\fR, with the results saved in a file named after each host\&.
+.RE
+.PP
+\fB\-T\fR \fISECONDS\fR, \fB\-\-timeout=\fR\fISECONDS\fR
+.RS 4
+Connection timeout to use when trying to talk to hosts, in
+\fISECONDS\fR\&.
+.RE
+.PP
+\fB\-B\fR \fINUM\fR, \fB\-\-background=\fR\fINUM\fR
+.RS 4
+Run commands in the background, killing the task after
+\fINUM\fR
+seconds\&.
+.RE
+.PP
+\fB\-P\fR \fINUM\fR, \fB\-\-poll=\fR\fINUM\fR
+.RS 4
+Poll a background job every
+\fINUM\fR
+seconds\&. Requires
+\fB\-B\fR\&.
+.RE
+.PP
+\fB\-u\fR \fIUSERNAME\fR, \fB\-\-remote\-user=\fR\fIUSERNAME\fR
+.RS 4
+Use this remote
+\fIUSERNAME\fR
+instead of root\&.
+.RE
+.PP
+\fB\-U\fR \fISUDO_USERNAME\fR, \fB\-\-sudo\-user=\fR\fISUDO_USERNAME\fR
+.RS 4
+Sudo to
+\fISUDO_USERNAME\fR
+instead of root\&. Implies \-\-sudo\&.
+.RE
+.PP
+\fB\-c\fR \fICONNECTION\fR, \fB\-\-connection=\fR\fICONNECTION\fR
+.RS 4
+Connection type to use\&. Possible options are
+\fIparamiko\fR
+(SSH) and
+\fIlocal\fR\&. Local is mostly useful for crontab or kickstarts\&.
+.RE
+.SH "INVENTORY"
+.sp
+Ansible stores the hosts it can potentially operate on in an inventory file\&. The syntax is one host per line\&. Groups headers are allowed and are included on their own line, enclosed in square brackets that start the line\&.
+.sp
+Ranges of hosts are also supported\&. For more information and additional options, see the documentation on http://ansible\&.github\&.com/\&.
+.SH "FILES"
+.sp
+/etc/ansible/hosts \(em Default inventory file
+.sp
+/usr/share/ansible/ \(em Default module library
+.SH "ENVIRONMENT"
+.sp
+The following environment variables may specified\&.
+.sp
+ANSIBLE_HOSTS  \(em Override the default ansible hosts file
+.sp
+ANSIBLE_LIBRARY \(em Override the default ansible module library path
+.SH "AUTHOR"
+.sp
+Ansible was originally written by Michael DeHaan\&. See the AUTHORS file for a complete list of contributors\&.
+.SH "COPYRIGHT"
+.sp
+Copyright \(co 2012, Michael DeHaan
+.sp
+Ansible is released under the terms of the GPLv3 License\&.
+.SH "SEE ALSO"
+.sp
+\fBansible\-playbook\fR(1)
+.sp
+Extensive documentation as well as IRC and mailing list info is available on the ansible home page: https://ansible\&.github\&.com/

docs/man/man1/ansible.1.asciidoc

@@ -1,108 +0,0 @@
-ansible(1)
-=========
-:doctype:manpage
-:man source:   Ansible
-:man version:  0.0.1
-:man manual:   System administration commands
-
-NAME
-----
-ansible - run a command somewhere else
-
-
-SYNOPSIS
---------
-ansible <host-pattern> [-f forks] [-m module_name] [-a args]
-
-
-DESCRIPTION
------------
-
-*Ansible* is an extra-simple tool/framework/API for doing \'remote things' over
-SSH.
-
-
-ARGUMENTS
----------
-
-*host-pattern*::
-
-A name of a group in the inventory file, a shell-like glob selecting hosts in inventory
-file, or any combination of the two seperated by semicolons.
-
-
-OPTIONS
--------
-
-
-*-i*, *--inventory*::
-
-Path to the inventory hosts file, which defaults to /etc/ansible/hosts.
-
-
-*-f*, *--forks*::
-
-Level of parallelism. Specify as an integer, the default is 5.
-
-
-*-m*, *--module-name*::
-
-Module name to execute.
-
-
-*-p*, *--pattern*::
-
-Hostname pattern. Accepts shell-like globs which can be seperated with ";"
-The default is "*" which matches all hosts in the ansible hosts file.  Group
-names from the ansible inventory file can also be used.
-
-
-See ansible --help for additional options.
-
-
-INVENTORY
----------
-
-Ansible stores the hosts it can potentially operate on in an inventory
-file. The syntax is one host per line.  Groups headers are allowed and
-are included on their own line, enclosed in square brackets.
-
-FILES
------
-
-/etc/ansible/hosts -- Default hosts file
-
-/usr/share/ansible -- Default module library
-
-
-ENVIRONMENT
------------
-
-The following environment variables may specified.
-
-ANSIBLE_HOSTS  -- Override the default ansible hosts file
-
-ANSIBLE_LIBRARY -- Override the default ansible module library path
-
-
-AUTHOR
-------
-
-Ansible was originally written by Michael DeHaan. See the AUTHORS file
-for a complete list of contributors.
-
-
-COPYRIGHT
----------
-
-Copyright © 2012, Michael DeHaan
-
-Ansible is released under the terms of the GPLv3 License.
-
-
-SEE ALSO
---------
-
-Ansible home page: <https://ansible.github.com/>
-
-

docs/man/man1/ansible.1.asciidoc.in

@@ -0,0 +1,159 @@
+ansible(1)
+=========
+:doctype:manpage
+:man source:   Ansible
+:man version:  %VERSION%
+:man manual:   System administration commands
+
+NAME
+----
+ansible - run a command somewhere else
+
+
+SYNOPSIS
+--------
+ansible <host-pattern> [-f forks] [-m module_name] [-a args]
+
+
+DESCRIPTION
+-----------
+
+*Ansible* is an extra-simple tool/framework/API for doing \'remote things' over
+SSH.
+
+
+ARGUMENTS
+---------
+
+*host-pattern*::
+
+A name of a group in the inventory file, a shell-like glob selecting
+hosts in inventory file, or any combination of the two separated by
+semicolons.
+
+OPTIONS
+-------
+
+
+*-i* 'PATH', *--inventory=*'PATH'::
+
+The 'PATH' to the inventory hosts file, which defaults to '/etc/ansible/hosts'.
+
+
+*-f* 'NUM', *--forks=*'NUM'::
+
+Level of parallelism. 'NUM' is specified as an integer, the default is 5.
+
+
+*-m* 'NAME', *--module-name=*'NAME'::
+
+Execute the module called 'NAME'.
+
+
+*-M* 'DIRECTORY', *--module-path=*'DIRECTORY'::
+
+The 'DIRECTORY' to load modules from. The default is '/usr/share/ansible'.
+
+
+*-a* \'_ARGUMENTS_', *--args=*\'_ARGUMENTS_'::
+
+The 'ARGUMENTS' to pass to the module.
+
+*-k*, *--ask-pass*::
+
+Prompt for the SSH password instead of assuming key-based authentication with ssh-agent.
+
+*-K*, *--ask-sudo-pass*::
+
+Prompt for the password to use with --sudo, if any
+
+*-o*, *--one-line*::
+
+Try to output everything on one line.
+
+*-s*, *--sudo*::
+
+Run the command as the user given by -u and sudo to root.
+
+*-t* 'DIRECTORY', *--tree=*'DIRECTORY'::
+
+Save contents in this output 'DIRECTORY', with the results saved in a
+file named after each host.
+
+*-T* 'SECONDS', *--timeout=*'SECONDS'::
+
+Connection timeout to use when trying to talk to hosts, in 'SECONDS'.
+
+*-B* 'NUM', *--background=*'NUM'::
+
+Run commands in the background, killing the task after 'NUM' seconds.
+
+*-P* 'NUM', *--poll=*'NUM'::
+
+Poll a background job every 'NUM' seconds. Requires *-B*.
+
+*-u* 'USERNAME', *--remote-user=*'USERNAME'::
+
+Use this remote 'USERNAME' instead of root.
+
+*-U* 'SUDO_USERNAME', *--sudo-user=*'SUDO_USERNAME'::
+
+Sudo to 'SUDO_USERNAME' instead of root.  Implies --sudo.
+
+*-c* 'CONNECTION', *--connection=*'CONNECTION'::
+
+Connection type to use.  Possible options are 'paramiko' (SSH) and 'local'.
+Local is mostly useful for crontab or kickstarts.
+
+
+INVENTORY
+---------
+
+Ansible stores the hosts it can potentially operate on in an inventory
+file. The syntax is one host per line.  Groups headers are allowed and
+are included on their own line, enclosed in square brackets that
+start the line.
+
+Ranges of hosts are also supported.  For more information and
+additional options, see the documentation on http://ansible.github.com/.
+
+FILES
+-----
+
+/etc/ansible/hosts -- Default inventory file
+
+/usr/share/ansible/ -- Default module library
+
+
+ENVIRONMENT
+-----------
+
+The following environment variables may specified.
+
+ANSIBLE_HOSTS  -- Override the default ansible hosts file
+
+ANSIBLE_LIBRARY -- Override the default ansible module library path
+
+
+AUTHOR
+------
+
+Ansible was originally written by Michael DeHaan. See the AUTHORS file
+for a complete list of contributors.
+
+
+COPYRIGHT
+---------
+
+Copyright © 2012, Michael DeHaan
+
+Ansible is released under the terms of the GPLv3 License.
+
+
+SEE ALSO
+--------
+
+*ansible-playbook*(1)
+
+Extensive documentation as well as IRC and mailing list info
+is available on the ansible home page: <https://ansible.github.com/>

docs/man/man5/ansible-modules.5.asciidoc

@@ -1,235 +0,0 @@
-ansible-modules(5)
-=================
-:doctype:manpage
-:man source: Ansible-modules
-:man version: 0.0.1
-:man manual: Ansible
-
-
-NAME
-----
-ansible-modules - stock modules shipped with ansible
-
-
-DESCRIPTION
------------
-
-Ansible ships with a number of modules that can be executed directly on remote hosts or through
-ansible playbooks.
-
-
-IDEMPOTENCE
------------
-
-Most modules other than command are idempotent, meaning they will seek to avoid changes
-unless a change needs to be made.  When using ansible playbooks, these modules can
-trigger change events, as described in *ansible-playbooks*(5).
-
-Unless otherwise noted, all modules support change hooks.
-
-
-command
--------
-
-The command module takes the command name followed by a list of arguments, space delimited.
-This is the only module that does not use key=value style parameters.
-
-Example usage::
-
-/sbin/shutdown -t now
-
-This module does not support change hooks.
-
-Returns the return code from the program as well as timing information.
-
-(Async command running and command execution time limits are in plan.)
-
-copy
-----
-
-The copy module moves a file on the local box to remote locations.
-
-*src=*::
-
-Local absolute path to a file to copy to the remote server
-
-
-*dest=*::
-
-Remote absolute path where the file should end up
-
-
-This module also returns md5sum information about the resultant file.
-
-
-facter
-------
-
-Runs the discovery program 'facter' on the remote system, returning
-JSON data that can be useful for inventory purposes.
-
-Requires that 'facter' and 'ruby-json' be installed on the remote end.
-
-This module is informative only - it takes no parameters & does not support change hooks,
-nor does it make any changes on the system.
-
-
-file
-----
-
-Ensures the ownership and permissions of files are as desired.
-
-Use copy or template first if you need to make sure a file is on the box.
-
-In plan.
-
-
-git
----
-
-Deploys software from git checkouts.
-
-*repo=*::
-
-git or http protocol address of the repo to checkout
-
-*dest=*::
-
-where to check it out, an absolute directory path
-
-*version=*::
-
-what version to check out -- either the git SHA, the literal string 'HEAD', or a tag name
-
-
-ohai
-----
-
-Similar to the facter module, this returns JSON inventory data.  Ohai
-data is a bit more verbose and nested than facter.
-
-Requires that 'ohai' be installed on the remote end.
-
-This module is information only - it takes no parameters & does not
-support change hooks, nor does it make any changes on the system.
-
-
-ping
-----
-
-A trivial test module, this module always returns the integer '1' on
-successful contact.
-
-This module does not support change hooks.
-
-This module is informative only - it takes no parameters & does not
-support change hooks, nor does it make any changes on the system.
-
-
-service
--------
-
-Controls services on remote machines.
-
-*state=*::
-
-Values are 'started', 'stopped', or 'restarted'.   Started/stopped
-are idempotent actions that will not run commands unless neccessary.
-'restarted' will always bounce the service
-
-
-*name=*::
-
-The name of the service
-
-
-setup
------
-
-Writes a JSON file containing key/value data, for use in templating.
-Call this once before using the template modules, usually as the very
-first step in your playbook.
-
-If facter or ohai are installed, variables from these programs will also
-be snapshotted into the JSON file for usage in templating. These variables
-are prefixed with 'facter_' and 'ohai_" so it's easy to tell their source.
-
-*metadata=*::
-
-Optionally overrides the default JSON file location of /etc/ansible/setup.
-If used, also supply the metadata parameter to 'template'.  Change if
-running as a non-root remote user who does not have permissions on /etc/ansible.
-
-*anything=*::
-
-any other parameters can be named basically anything, and set a key=value
-pair in the JSON file for use in templating.
-
-
-template
---------
-
-Templates a file out to a remote server.  Call the setup module prior to usage.
-
-*src=*::
-
-path of a Jinja2 formatted template on the local server
-
-
-*dest*::
-
-location to render the template on the remote server
-
-
-*metadata*::
-
-location of a JSON file to use to supply template data.  Default is /etc/ansible/setup
-which is the same as the default for the setup module.   Change if running as a non-root
-remote user who does not have permissions on /etc/ansible.
-
-
-This module also returns md5sum information about the resultant file.
-
-
-user
-----
-
-This module is in plan.
-
-
-yum
----
-
-This module is in plan.
-
-
-WRITING YOUR OWN MODULES
-------------------------
-
-To write your own modules, simply follow the convention of those already available in
-/usr/share/ansible.  Modules must return JSON but can be written in any language.
-To support change hooks, modules should return hashes, with a changed: True/False
-element at the top level.  Modules can also choose to indicate a failure scenario
-by returning a top level 'failure' element with a True value.
-
-
-ENVIRONMENT
------------
-
-ANSIBLE_LIBRARY -- Override the default ansible module library path
-
-
-AUTHOR
-------
-
-Ansible was originally written by Michael DeHaan. See the AUTHORS file
-for a complete list of contributors.
-
-SEE ALSO
---------
-
-*ansible*(1)
-
-*ansible-playbook*(5)
-
-Ansible home page: <https://github.com/mpdehaan/ansible/>

docs/man/man5/ansible-playbook.5.asciidoc

@@ -1,123 +0,0 @@
-ansible-modules(5)
-=================
-:doctype:manpage
-:man source: Ansible-playbook
-:man version: 0.0.1
-:man manual: Ansible
-
-
-NAME
-----
-ansible-playbook - format and function of an ansible playbook file
-
-
-DESCRIPTION
------------
-
-Ansible ships with 'ansible-playbook', a tool for running playbooks.
-Playbooks can represent frequent tasks, desired system configurations,
-or deployment processes.
-
-
-FORMAT
-------
-
-Playbooks are written in YAML.
-
-
-EXAMPLE
--------
-
-See:
-
-- https://github.com/mpdehaan/ansible/blob/master/examples/playbook.yml
-
-- https://github.com/mpdehaan/ansible/blob/master/examples/base.yml
-
-- https://github.com/mpdehaan/ansible/blob/master/examples/handlers.yml
-
-WHAT THE EXAMPLE MEANS
------------------------
-
-Here's what playbook.yml (above) will do.
-
-The first pattern will select all hosts.  The patterns are the same
-as supported by /usr/bin/ansible.
-
-First, it will run all the modules specified in base.yml.  Includes can
-be used to implement classes of things, and if you wanted, a playbook
-could consist of nothing but include files.  This is an example of an
-include.
-
-After processing base.yml, on each host we'll write for 
-a JSON file into /etc/ansible/setup on each remote system with the 
-values max_clients and http_port.
-
-Next, we'll use a Jinja2 template locally residing at
-/srv/templates/httpd.j2 to write the Apache config file on each host, 
-using the previous values in that setup file.
-
-Next, We'll ensure that apache is running if stopped.
-
-The template task set up a notifier, which means if the configuration
-file actually changed, we have a named handler, in this case, 'restart apache'
-to run.  In this case, all the notifiers come from handlers.yml, though it's
-also ok to express handlers directly in the main yaml file too.  Using
-the include promotes reuse.
-
-What does the handler say?  If and only if the config file changed, note that we need to restart
-apache at the end of the run, otherwise, don't bother because we
-already know it is running.
-
-
-HIGH LEVEL EXPLANATION
-----------------------
-
-Playbooks are executed top down and can contain multiple references to
-patterns.  For instance, a playbook could do something to all
-webservers, then do something to all database servers, then do
-something different to all webservers again.
-
-For each pattern, the tasks in the 'tasks' list are executed in order
-for all hosts in the host file matching the pattern.
-
-For each task, a name/action pair describes what the task is and what
-ansible module to use to accomplish the task, along with any
-arguments.   Additional fields like 'comment:' can be added and will
-be ignored, so feel free to take notes in the file.
-
-Most modules accept key=value format arguments.
-
-Handlers are like tasks, but are conditionally executed.  If a module
-reports a 'change', it can notify one or more handler by name.  If
-notified, it will run only for hosts that changed.
-
-
-ERROR HANDLING
---------------
-
-If a host has a failure, the host will be ignored for the remainder
-of the playbook execution.
-
-
-ENVIRONMENT
------------
-
-ANSIBLE_LIBRARY -- Override the default ansible module library path
-
-
-AUTHOR
-------
-
-Ansible was originally written by Michael DeHaan. See the AUTHORS file
-for a complete list of contributors.
-
-
-SEE ALSO
---------
-
-*ansible*(1)
-
-*ansible-modules*(5)
-
-Ansible home page: <https://github.com/mpdehaan/ansible/>

examples/ansible_hosts

@@ -1,12 +0,0 @@
-[webservers]
-alpha.example.org
-beta.example.org
-192.168.1.100
-192.168.1.110
-
-[dbservers]
-192.168.1.200
-192.168.1.201
-foo.example.org
-bar.example.org
-

examples/base.yml

@@ -1,7 +0,0 @@
----
-- name: no selinux
-  action: command /usr/sbin/setenforce 0
-- name: no iptables
-  action: service name=iptables state=stopped
-- name: this is just to show variables work here, favcolor={{ favcolor }}
-  action: command /bin/true

examples/handlers.yml

@@ -1,5 +0,0 @@
----
-- name: restart apache
-  action: service name=httpd state=restarted
-- name: restart memcached
-  action: service name=memcached state=restarted

examples/hosts

@@ -0,0 +1,44 @@
+# This is the default ansible 'hosts' file.
+#
+# It should live in /etc/ansible/hosts
+#
+#   - Comments begin with the '#' character
+#   - Blank lines are ignored
+#   - Groups of hosts are delimited by [header] elements
+#   - You can enter hostnames or ip addresses
+#   - A hostname/ip can be a member of multiple groups
+
+# Ex 1: Ungrouped hosts, specify before any group headers.
+
+green.example.com
+blue.example.com
+192.168.100.1
+192.168.100.10
+
+# Ex 2: A collection of hosts belonging to the 'webservers' group
+
+[webservers]
+alpha.example.org
+beta.example.org
+192.168.1.100
+192.168.1.110
+
+# If you have multiple hosts following a pattern you can specify
+# them like this:
+
+www[001:006].example.com
+
+# Ex 3: A collection of database servers in the 'dbservers' group
+
+[dbservers]
+
+db01.intranet.mydomain.net
+db02.intranet.mydomain.net
+10.25.1.56
+10.25.1.57
+
+# Here's another example of host ranges, this time there are no
+# leading 0s:
+
+db-[99:101]-node.example.com
+

examples/playbook.yml

@@ -1,18 +0,0 @@
----
-- hosts: all
-  user: root
-  vars:
-    http_port: 80
-    max_clients: 200
-  tasks:
-  - include: base.yml favcolor=blue
-  - name: write the foo config file using vars set above
-    action: template src=foo.j2 dest=/etc/some_random_foo.conf
-    notify:
-    - restart apache
-  - name: ensure apache is running
-    action: service name=httpd state=started
-  - name: pointless test action
-    action: command /bin/echo {{ http_port }}
-  handlers:
-    - include: handlers.yml

examples/playbook2.yml

@@ -1,24 +0,0 @@
----
-- hosts: '*'
-  vars:
-    a: 2
-    b: 3
-    c: 4
-  tasks:
-  - name: copy comand
-    action: copy src=/srv/a dest=/srv/b
-    notify: 
-    - restart apache
-  - name: template step
-    action: template src=/srv/template.j2 dest=/srv/file.out
-    notify:
-    - restart apache
-  - name: execute bin false
-    comment: call something that will fail just to demo failure counts and such
-    action: command /bin/false
-  - name: execute bin true
-    comment: this will never be executed because previous will fail
-    action: command /bin/true
-  handlers:
-    - name: restart apache
-      action: service name=httpd state=restarted

examples/playbooks/ansible_pull.yml

@@ -0,0 +1,51 @@
+# ansibple-pull setup 
+#
+# on remote hosts, set up ansible to run periodically using the latest code
+# from a particular checkout, in pull based fashion, inverting Ansible's
+# usual push-based operating mode.  
+#
+# This particular pull based mode is ideal for:
+#
+# (A) massive scale out
+# (B) continual system remediation
+#
+# DO NOT RUN THIS AGAINST YOUR HOSTS WITHOUT CHANGING THE repo_url
+# TO SOMETHING YOU HAVE PERSONALLY VERIFIED
+#
+# 
+---
+
+- hosts: pull_mode_hosts
+  user: root
+
+  vars:
+
+    # schdule is fed directly to cron
+    schedule: '*/15 * * * *'
+
+    # User to run ansible-pull as from cron
+    cron_user: root
+
+    # Directory to where repository will be cloned
+    workdir: /var/lib/ansible/local
+
+    # Repository to check out -- YOU MUST CHANGE THIS
+    # repo must contain a local.yml file at top level
+    #repo_url: git://github.com/sfromm/ansible-playbooks.git
+    repo_url: SUPPLY_YOUR_OWN_GIT_URL_HERE
+
+  tasks:
+
+    - name: Install ansible
+      action: yum pkg=ansible state=installed
+
+    - name: Create local directory to work from
+      action: file path=$workdir state=directory owner=root group=root mode=0751
+
+    - name: Copy ansible inventory file to client
+      action: copy src=/etc/ansible/hosts dest=/etc/ansible/hosts
+              owner=root group=root mode=0644
+
+    - name: Create crontab entry to clone/pull git repository
+      action: template src=templates/ansible-pull.j2 dest=/etc/cron.d/ansible-pull owner=root group=root mode=0644
+

examples/playbooks/conditionals_part1.yml

@@ -0,0 +1,51 @@
+---
+# this is a demo of conditional imports.  This is a powerful concept
+# and can be used to use the same recipe for different types of hosts,
+# based on variables that bubble up from the hosts from tools such
+# as ohai or facter.
+#
+# Here's an example use case:
+#
+# what to do if the service for apache is named 'httpd' on CentOS
+# but is named 'apache' on Debian?
+
+
+# there is only one play in this playbook, it runs on all hosts
+# as root
+
+- hosts: all
+  user: root
+
+# we have a common list of variables stored in /vars/external_vars.yml
+# that we will always import
+
+# next, we want to import files that are different per operating system
+# and if no per operating system file is found, load a defaults file.
+# for instance, if the OS was "CentOS", we'd try to load vars/CentOS.yml.
+# if that was found, we would immediately stop.  However if that wasn't
+# present, we'd try to load vars/defaults.yml.  If that in turn was not
+# found, we would fail immediately, because we had gotten to the end of
+# the list without importing anything.
+
+  vars_files:
+
+
+     - "vars/external_vars.yml"
+
+     - [ "vars/$facter_operatingsystem.yml", "vars/defaults.yml" ]
+
+# and this is just a regular task line from a playbook, as we're used to.
+# but with variables in it that come from above.  Note that the variables
+# from above are *also* available in templates
+     
+  tasks:
+
+  - name: ensure apache is latest
+    action: $packager pkg=$apache state=latest
+  - name: ensure apache is running
+    action: service name=$apache state=running
+  - name: fail
+    action: command /bin/false
+
+ 
+

examples/playbooks/conditionals_part2.yml

@@ -0,0 +1,47 @@
+---
+# this is a demo of conditional executions using 'only_if', which can skip
+# certain tasks on machines/platforms/etc where they do not apply.
+
+- hosts: all
+  user: root
+
+  vars:
+     favcolor: "red"
+     ssn: 8675309
+
+# Below we're going to define some expressions.
+#
+# Not only can we assign variables for reuse, but we can also assign conditional
+# expressions.  By keeping these in 'vars', the task section remains
+# extraordinarily clean, and not littered with programming language
+# constructs -- so it's easily skimmed by humans.
+#
+# Remember to quote any variables if they are not numbers!  
+#
+# Interesting fact: aside from the $variables, these expressions are actually 
+# tiny bits of Python.  They are evaluated in the context of each host, so different
+# steps can be skipped on different hosts!  They should evaluate to either True
+# or False
+
+     is_favcolor_blue: "'$favcolor' == 'blue'"
+     is_centos: "'$facter_operatingsystem' == 'CentOS'"
+
+# NOTE:
+#
+# facter and ohai variables can be used in only_if statements too
+# ex:  "'$facter_operatingsystem' == 'CentOS'", which bubble up automatically
+# from the managed machines
+#
+# this example won't do that though, as you might not have facter or ohai,
+# but you get the idea...
+
+  tasks:
+
+     - name: "do this if my favcolor is blue"
+       action: shell /bin/false
+       only_if: '$is_favcolor_blue'
+     
+     - name: "do this if my favcolor is not blue"
+       action: shell /bin/true
+       only_if: 'not ($is_favcolor_blue)'
+

examples/playbooks/file_secontext.yml

@@ -0,0 +1,18 @@
+---
+# This is a demo of how to manage the selinux context using the file module
+- hosts: test
+  user: root
+  tasks:
+    - name: Change setype of /etc/exports to non-default value
+      action: file path=/etc/exports setype=etc_t
+    - name: Change seuser of /etc/exports to non-default value
+      action: file path=/etc/exports seuser=unconfined_u
+    - name: Set selinux context back to default value
+      action: file path=/etc/exports context=default
+    - name: Create empty file
+      action: command /bin/touch /tmp/foo
+    - name: Change setype of /tmp/foo
+      action: file path=/tmp/foo setype=default_t
+    - name: Try to set secontext to default, but this will fail
+            because of the lack of a default in the policy
+      action: file path=/tmp/foo context=default

examples/playbooks/get_url.yml

@@ -0,0 +1,16 @@
+---
+- hosts: webservers
+  vars:
+  - jquery_directory: /var/www/html/javascript
+  - person: 'Susie%20Smith'
+  tasks:
+  - name: Create directory for jQuery
+    action: file dest=${jquery_directory} state=directory mode=0755
+  - name: Grab a bunch of jQuery stuff
+    action: get_url url=http://code.jquery.com/$item  dest=${jquery_directory} mode=0444
+    with_items:
+    - jquery.min.js
+    - mobile/latest/jquery.mobile.min.js
+    - ui/jquery-ui-git.css
+  #- name: Pass urlencoded name to CGI
+  #  action: get_url url=http://example.com/name.cgi?name='${person}' dest=/tmp/test

examples/playbooks/group_commands.yml

@@ -0,0 +1,18 @@
+---
+# This is a demo of how the group command works.
+
+- hosts: all
+  user: root
+  sudo: True
+
+  tasks:
+
+    # Walk through group creation, modification, and deletion
+    - name: create a group
+      action: group name=tset
+
+    # You can only modify the group's gid
+    - action: group name=tset gid=7777
+
+    # And finally remove the group
+    - action: group name=tset state=absent

examples/playbooks/handlers/handlers.yml

@@ -0,0 +1,10 @@
+---
+
+# this is an example to show that handlers can be included from yaml files,
+# to promote reuse between different plays or even playbooks.  They work
+# just like normal handlers.
+
+- name: restart apache
+  action: service name=httpd state=restarted
+- name: restart memcached
+  action: service name=memcached state=restarted

examples/playbooks/intermediate_example.yml

@@ -0,0 +1,89 @@
+---
+# see examples.yml first!
+# This file explains some more advanced features of playbooks.  
+# because of the comments it's less concise than it normally is.  But feel
+# free to comment your playbooks if you like.
+
+- hosts: all
+
+  # we can define variables the normal way...
+
+  vars:
+    release: 2.0
+
+  # but they can also come from other files.  This can be a relative
+  # or absolute path.  This is a good way to store 'secret' variable
+  # files but still keep the playbook in public source control
+
+  vars_files:
+    - vars/external_vars.yml
+
+  # as with before, every play has a list of tasks in it
+
+  tasks:
+
+  # tasks can be written the normal way...
+
+  - name: arbitrary command
+    action: command /bin/true
+
+  # or we can promote reuse and simplicity by including tasks
+  # from other files, for instance, to reuse common tasks
+
+  - include: tasks/base.yml
+
+  # we could also have done something like:
+  # - include: wordpress.yml user=timmy
+  # and had access to the template variable $user in the 
+  # included file, if we wanted to.  Variables from vars
+  # and vars_files are also available inside include files
+
+  handlers:
+ 
+    # handlers can also be included from files, to promote reuse
+    # and simpler recipes, you may wish to only have one
+    # handler file for all your plays and playbooks
+
+    - include: handlers/handlers.yml
+
+    # you can mix things that are directly in the file with things
+    # that are included.  Order is executed as written, but only
+    # handlers that have been notified get executed
+
+    - name: restart foo
+      action: service name=foo state=restarted
+
+# ===============================================================
+
+# Here's a second play in the same playbook.  This will be run
+# after the first playbook completes on all hosts.  You may want
+# a different play for each class of systems, or may want a different
+# play for each stage in a complex multi-node deployment push
+# process.  How you use them are up to you.
+
+# any play in a playbook can be executed by a user other than root
+# if you want.  sudo support is coming too.
+
+- hosts: webservers
+  user: mdehaan
+
+  # vars must be specified again for the next play in the playbook
+  # but can be reused by including from vars_files if you want
+  # you can use vars, vars_files, or both.  vars_files overrides
+  # those set in vars. 
+
+  vars:
+     release: 2.0
+  vars_files:
+     - vars/external_vars.yml
+
+
+  # these all runs as the user 'mdehaan'.  If there were any handlers
+  # they would as well.
+
+  tasks:
+
+     - name: some random command
+       action: command /bin/true
+
+

examples/playbooks/intro_example.yml

@@ -0,0 +1,76 @@
+---
+# this is an annotated example of some features available in playbooks
+# it shows how to make sure packages are updated, how to make sure
+# services are running, and how to template files.  It also demos
+# change handlers that can restart things (or trigger other actions)
+# when resources change.  For more advanced examples, see example2.yml
+
+# on all hosts, run as the user root...
+
+- name: example play
+  hosts: all
+  user: root
+
+# could have also have done:
+#  user: mdehaan
+#  sudo: True
+
+  # make these variables available inside of templates
+  # for when we use the 'template' action/module later on...
+
+  vars:
+    http_port: 80
+    max_clients: 200
+
+  # define the tasks that are part of this play...
+
+  tasks:
+
+  # task #1 is to run an arbitrary command
+  # we'll simulate a long running task, wait for up to 45 seconds, poll every 5
+  # obviously this does nothing useful but you get the idea
+
+  - name: longrunner
+    action: command /bin/sleep 15
+    async: 45
+    poll: 5
+
+  # let's demo file operations.
+  #
+  # We can 'copy' files or 'template' them instead, using jinja2
+  # as the templating engine.  This is done using the variables 
+  # from the vars section above mixed in with variables bubbled up 
+  # automatically from tools like facter and ohai.  'copy' 
+  # works just like 'template' but does not do variable subsitution.
+  #
+  # If and only if the file changes, restart apache at the very
+  # end of the playbook run
+
+  - name: write some_random_foo configuration
+    action: template src=templates/foo.j2 dest=/etc/some_random_foo.conf
+    notify:
+    - restart apache
+
+  # make sure httpd is installed at the latest version
+
+  - name: install httpd
+    action: yum pkg=httpd state=latest
+
+  # make sure httpd is running
+
+  - name: httpd start
+    action: service name=httpd state=running
+
+  # handlers are only run when things change, at the very end of each
+  # play.  Let's define some.  The names are significant and must
+  # match the 'notify' sections above
+
+  handlers:
+
+    # this particular handler is run when some_random_foo.conf
+    # is changed, and only then
+
+    - name: restart apache
+      action: service name=httpd state=restarted
+
+

examples/playbooks/loop_with_items.yml

@@ -0,0 +1,30 @@
+---
+# this is an example of how to run repeated task elements over lists
+# of items, for example, installing multiple packages or configuring
+# multiple users
+
+- hosts: all
+  user: root
+
+  tasks:
+
+  - name: install $item
+    action: yum pkg=$item state=installed
+    with_items:
+       - cobbler
+       - httpd
+
+  - name: configure user $item
+    action: user name=$item state=present groups=wheel
+    with_items:
+       - testuser1
+       - testuser2
+
+  - name: remove user $item
+    action: user name=$item state=absent
+    with_items:
+        - testuser1
+        - testuser2
+
+
+

examples/playbooks/mysql.yml

@@ -0,0 +1,21 @@
+##
+# Example Ansible playbook that uses the MySQL module.
+#
+
+---
+- hosts: all
+  user: root
+
+  vars:
+    mysql_root_password: 'password'
+
+  tasks:
+
+    - name: Create database user
+      action: mysql_user loginpass=$mysql_root_password user=bob passwd=12345 priv=*.*:ALL state=present
+
+    - name: Create database
+      action: mysql_db loginpass=$mysql_root_password db=bobdata state=present
+
+    - name: Ensure no user named 'sally' exists
+      action: mysql_user loginpass=$mysql_root_password user=sally state=absent

examples/playbooks/nested_playbooks.yml

@@ -0,0 +1,26 @@
+---
+# it is possible to have top level playbook files import other playbook
+# files.  For example, a playbook called could include three
+# different playbooks, such as webservers, workers, dbservers, etc.
+#
+# Running the site playbook would run all playbooks, while individual
+# playbooks could still be run directly.  This is somewhat like
+# the tag feature and can be used in conjunction for very fine grained
+# control over what you want to target when running ansible.
+
+- name: this is a play at the top level of a file  
+  hosts: all
+  user: root
+  tasks:
+  - name: say hi
+    tags: foo
+    action: shell echo "hi..."
+
+# and this is how we include another playbook, be careful and
+# don't recurse infinitely or anything.  Note you can't use
+# any variables here.
+
+- include: intro_example.yml
+
+# and if we wanted, we can continue with more includes here,
+# or more plays inline in this file

examples/playbooks/postgresql.yml

@@ -0,0 +1,38 @@
+##
+# Example Ansible playbook that uses the PostgreSQL module.
+#
+# This installs PostgreSQL on an Ubuntu system, creates a database called
+# "myapp" and a user called "django" with password "mysupersecretpassword"
+# with access to the "myapp" database.
+#
+---
+- hosts: webservers
+  sudo: True
+  gather_facts: False
+
+  tasks:
+  - name: ensure apt cache is up to date
+    action: apt update_cache=yes
+  - name: ensure packages are installed
+    action: apt pkg=$item
+    with_items:
+        - postgresql
+        - libpq-dev
+        - python-psycopg2
+
+- hosts: webservers
+  sudo: True
+  sudo_user: postgres
+  gather_facts: False
+
+  vars:
+    dbname: myapp
+    dbuser: django
+    dbpassword: mysupersecreetpassword
+
+  tasks:
+  - name: ensure database is created
+    action: postgresql_db db=$dbname
+
+  - name: ensure user has access to database
+    action: postgresql_user db=$dbname user=$dbuser password=$dbpassword

examples/playbooks/prompts.yml

@@ -0,0 +1,43 @@
+---
+
+# it is possible to ask for variables from the user at the start
+# of a playbook run, for example, as part of a release script.
+
+- hosts: all
+  user: root
+
+# regular variables are a dictionary of keys and values
+
+  vars:
+     this_is_a_regular_var: 'moo'
+     so_is_this: 'quack'
+
+# alternatively, they can ALSO be passed in from the outside:
+#    ansible-playbook foo.yml --extra-vars="foo=100 bar=101"
+# or through external inventory scripts (see online API docs)
+
+# here's basic mode prompting.  Specify a hash of variable names and a prompt for
+# each.
+#
+# vars_prompt:
+#   release_version: "product release version"
+
+# prompts can also be specified like this, allowing for hiding the prompt as 
+# entered.  In the future, this may also be used to support crypted variables
+
+  vars_prompt:
+    - name: "some_password"
+      prompt: "Enter password"
+      private: True
+    - name: "release_version"
+      prompt: "Product release version"
+      private: False
+
+# this is just a simple example to show that vars_prompt works, but
+# you might ask for a tag to use with the git module or perhaps
+# a package version to use with the yum module.
+
+  tasks:
+  - name: imagine this did something interesting with $release_version
+    action: shell echo foo >> /tmp/$release_version-$alpha
+

examples/playbooks/selective_file_sources.yml

@@ -0,0 +1,28 @@
+---
+# this is an example of how to template a file over using some variables derived
+# from the system.  For instance, if you wanted to have different configuration
+# templates by OS version, this is a neat way to do it.  Any Ansible facts, facter facts,
+# or ohai facts could be used to do this.  
+
+- hosts: all
+
+  tasks:
+
+  - name: template a config file
+    action: template dest=/etc/imaginary_file.conf
+    first_available_file:
+
+       # first see if we have a file for this specific host
+       - /srv/whatever/${ansible_hostname}.conf
+
+       # next try to load something like CentOS6.2.conf
+       - /srv/whatever/${ansible_distribution}${ansible_distribution_version}.conf
+
+       # next see if there's a CentOS.conf
+       - /srv/whatever/${ansible_distribution}.conf
+
+       # finally give up and just use something generic
+       - /srv/whatever/default
+
+
+

examples/playbooks/tags.yml

@@ -0,0 +1,44 @@
+---
+# tags allow us to run all of a playbook or part of it.
+#
+# assume: ansible-playbook tags.yml --tags foo
+#
+# try this with:
+#    --tags foo
+#    --tags bar
+#    --tags extra
+#
+# the value of a 'tags:' element can be a string or list
+# of tag names.  Variables are not usable in tag names.
+
+- name: example play one
+  hosts: all
+  user: root
+
+  # any tags applied to the play are shorthand to applying
+  # the tag to all tasks in it.  Here, each task is given
+  # the tag extra
+
+  tags: 
+     - extra
+
+  tasks:
+
+  # this task will run if you don't specify any tags,
+  # if you specify 'foo' or if you specify 'extra'
+
+  - name: hi
+    tags: foo
+    action: shell echo "first task ran"
+
+- name: example play two
+  hosts: all
+  user: root
+  tasks:
+  - name: hi
+    tags: 
+       - bar
+    action: shell echo "second task ran"
+  - include: tasks/base.yml tags=base
+
+

examples/playbooks/tasks/base.yml

@@ -0,0 +1,22 @@
+---
+
+# this is the example of an included tasks file.  It contains a flat list of tasks
+# they can notify other tasks, and have full access to variables from 'vars'
+# or 'vars_files' directives.  Further, if ohai or facter were installed on 
+# the remote machines, variables from those tools can be accessed on the 'action'
+# line or in templates.  Just prefix with 'facter_' or 'ohai_' before the particular
+# variable.
+
+# possible uses for a included yaml file might be to represent a 'class' of a system
+# like defining what makes up a webserver, or you might have a common 'base.yml'
+# (like this) that might be applied to all your systems as well.
+
+- name: no selinux
+  action: command /usr/sbin/setenforce 0
+
+- name: no iptables
+  action: service name=iptables state=stopped
+
+- name: made up task just to show variables work here
+  action: command /bin/echo release is $release
+

examples/playbooks/templates/ansible-pull.j2

@@ -0,0 +1,2 @@
+# Cron job to git clone/pull a repo and then run locally
+{{ schedule }} {{ cron_user }} ansible-pull -d {{ workdir }} -U {{ repo_url }} >/var/log/ansible-pull.log 2>&1

examples/playbooks/templates/foo.j2

@@ -0,0 +1,10 @@
+# This is a very simple Jinja2 template representing an imaginary configuration file
+# for an imaginary app.
+
+# this is an example of loading a fact from the setup module
+system={{ ansible_system }}
+
+# here is a variable that could be set in a playbook or inventory file
+http_port={{ http_port }}
+
+

examples/playbooks/user_commands.yml

@@ -0,0 +1,38 @@
+---
+# this is a demo of how the user commands work and how to reference salted passwords
+# in vars sections.  You could also use vars_files if you like (see other examples)
+
+- hosts: all
+  user: root
+  vars:
+    # created with:
+    # crypt.crypt('This is my Password', '$1$SomeSalt')
+    password: $1$SomeSalt$UqddPX3r4kH3UL5jq5/ZI.
+
+  tasks:
+
+    # Walk through account creation, modification, and deletion
+    - name: test basic user account creation
+      action: user name=tset comment=TsetUser group=users shell=/sbin/nologin createhome=no
+
+    # the following is just a simple example of how you don't have to include
+    # the 'name' element for each task
+
+    - action: user name=tset comment=NyetUser
+    - action: user name=tset password=$password
+
+    # The following will add the user to supplementary groups.
+
+    # Add the user to the groups dialout and uucp.
+    - action: user name=tset groups=dialout,uucp
+
+    # Add the user to the groups dialout and wheel,
+    # This will remove tset from the group uucp.
+    - action: user name=tset groups=dialout,wheel
+
+    # Add the user to the group uucp.  Because append=yes, the user
+    # will not be removed from the groups dialout and wheel.
+    - action: user name=tset groups=uucp append=yes
+
+    # Finally, remove the user.
+    - action: user name=tset state=absent

examples/playbooks/vars/CentOS.yml

@@ -0,0 +1,3 @@
+---
+apache: httpd
+packager: yum

examples/playbooks/vars/defaults.yml

@@ -0,0 +1,3 @@
+---
+packager: apt
+apache: apache

examples/playbooks/vars/external_vars.yml

@@ -0,0 +1,3 @@
+---
+alpha: one
+beta: two

examples/scripts/cobbler_external_inventory.py

@@ -0,0 +1,131 @@
+#!/usr/bin/python
+
+"""
+Cobbler external inventory script
+=================================
+
+Ansible has a feature where instead of reading from /etc/ansible/hosts
+as a text file, it can query external programs to obtain the list
+of hosts, groups the hosts are in, and even variables to assign to each host.  
+
+To use this, copy this file over /etc/ansible/hosts and chmod +x the file.
+This, more or less, allows you to keep one central database containing
+info about all of your managed instances.
+
+This script is an example of sourcing that data from Cobbler
+(http://cobbler.github.com).  With cobbler each --mgmt-class in cobbler 
+will correspond to a group in Ansible, and --ks-meta variables will be 
+passed down for use in templates or even in argument lines.
+
+NOTE: The cobbler system names will not be used.  Make sure a
+cobbler --dns-name is set for each cobbler system.   If a system
+appears with two DNS names we do not add it twice because we don't want
+ansible talking to it twice.  The first one found will be used. If no
+--dns-name is set the system will NOT be visible to ansible.  We do
+not add cobbler system names because there is no requirement in cobbler
+that those correspond to addresses.  
+
+See http://ansible.github.com/api.html for more info
+
+Tested with Cobbler 2.0.11. 
+"""
+
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible,
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+######################################################################
+
+
+import sys
+import xmlrpclib
+import shlex
+
+try:
+    import json
+except:
+    import simplejson as json
+
+# NOTE -- this file assumes Ansible is being accessed FROM the cobbler
+# server, so it does not attempt to login with a username and password.
+# this will be addressed in a future version of this script.
+
+conn = xmlrpclib.Server("http://127.0.0.1/cobbler_api", allow_none=True)
+
+###################################################
+# executed with no parameters, return the list of
+# all groups and hosts
+
+if len(sys.argv) == 2 and (sys.argv[1] == '--list'):
+
+    systems = conn.get_item_names('system')
+    groups = { 'ungrouped' : [] }
+
+    for system in systems:
+
+        data = conn.get_blended_data(None, system)
+
+        dns_name = None
+        interfaces = data['interfaces']
+        for (iname, ivalue) in interfaces.iteritems():
+            this_dns_name = ivalue.get('dns_name', None)
+            if this_dns_name is not None:
+                dns_name = this_dns_name       
+ 
+        if dns_name is None:
+            continue
+
+        classes  = data['mgmt_classes']
+        for cls in classes:
+            if cls not in groups:
+                groups[cls] = []
+            # hostname is not really what we want to insert, really insert the
+            # first DNS name but no further DNS names
+            groups[cls].append(dns_name)
+ 
+    print json.dumps(groups)
+    sys.exit(0)
+
+#####################################################
+# executed with a hostname as a parameter, return the
+# variables for that host
+
+elif len(sys.argv) == 3 and (sys.argv[1] == '--host'):
+
+    # look up the system record for the given DNS name
+    result = conn.find_system_by_dns_name(sys.argv[2])
+    system = result.get('name', None)
+    data = {}
+    if system is None:
+        print json.dumps({})
+        sys.exit(1)
+    data = conn.get_system_for_koan(system)
+
+    # return the ksmeta data for that system
+    metadata = data['ks_meta']
+    tokens = shlex.split(metadata)
+    results = {}
+    for t in tokens:
+        if t.find("=") != -1:
+           (k,v) = t.split("=",1)
+           results[k]=v
+    print json.dumps(results)
+    sys.exit(0)
+
+else:
+
+    print "usage: --list  ..OR.. --host <hostname>"
+    sys.exit(1)

examples/scripts/ec2.ini

@@ -0,0 +1,42 @@
+# Ansible EC2 external inventory script settings
+#
+
+[ec2]
+
+# AWS regions to make calls to. Set this to 'all' to make request to all regions
+# in AWS and merge the results together. Alternatively, set this to a comma
+# separated list of regions. E.g. 'us-east-1,us-west-1,us-west-2'
+regions = all
+
+# When generating inventory, Ansible needs to know how to address a server.
+# Each EC2 instance has a lot of variables associated with it. Here is the list:
+#   http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance
+# Below are 2 variables that are used as the address of a server:
+#   - destination_variable
+#   - vpc_destination_variable
+
+# This is the normal destination variable to use. If you are running Ansible
+# from outside EC2, then 'public_dns_name' makes the most sense. If you are
+# running Ansible from within EC2, then perhaps you want to use the internal
+# address, and should set this to 'private_dns_name'.
+destination_variable = public_dns_name
+
+# For server inside a VPC, using DNS names may not make sense. When an instance
+# has 'subnet_id' set, this variable is used. If the subnet is public, setting
+# this to 'ip_address' will return the public IP address. For instances in a
+# private subnet, this should be set to 'private_ip_address', and Ansible must
+# be run from with EC2.
+vpc_destination_variable = ip_address
+
+# API calls to EC2 are slow. For this reason, we cache the results of an API
+# call. Set this to the path you want cache files to be written to. Two files
+# will be written to this directory:
+#   - ansible-ec2.cache
+#   - ansible-ec2.index
+cache_path = /tmp
+
+# The number of seconds a cache file is considered valid. After this many
+# seconds, a new API call will be made, and the cache file will be updated.
+cache_max_age = 300
+
+

examples/scripts/ec2_external_inventory.py

@@ -0,0 +1,377 @@
+#!/usr/bin/python -tt
+
+'''
+EC2 external inventory script
+=================================
+
+Generates inventory that Ansible can understand by making API request to
+AWS EC2 using the Boto library. 
+
+NOTE: This script assumes Ansible is being executed where the environment
+variables needed for Boto have already been set:
+    export AWS_ACCESS_KEY_ID='AK123'
+    export AWS_SECRET_ACCESS_KEY='abc123'
+    
+For more details, see: http://docs.pythonboto.org/en/latest/boto_config_tut.html
+
+When run against a specific host, this script returns the following variables:
+ - ec2_ami_launch_index
+ - ec2_architecture
+ - ec2_association
+ - ec2_attachTime
+ - ec2_attachment
+ - ec2_attachmentId
+ - ec2_client_token
+ - ec2_deleteOnTermination
+ - ec2_description
+ - ec2_deviceIndex
+ - ec2_dns_name
+ - ec2_eventsSet
+ - ec2_group_name
+ - ec2_hypervisor
+ - ec2_id
+ - ec2_image_id
+ - ec2_instanceState
+ - ec2_instance_type
+ - ec2_ipOwnerId
+ - ec2_ip_address
+ - ec2_item
+ - ec2_kernel
+ - ec2_key_name
+ - ec2_launch_time
+ - ec2_monitored
+ - ec2_monitoring
+ - ec2_networkInterfaceId
+ - ec2_ownerId
+ - ec2_persistent
+ - ec2_placement
+ - ec2_platform
+ - ec2_previous_state
+ - ec2_private_dns_name
+ - ec2_private_ip_address
+ - ec2_publicIp
+ - ec2_public_dns_name
+ - ec2_ramdisk
+ - ec2_reason
+ - ec2_region
+ - ec2_requester_id
+ - ec2_root_device_name
+ - ec2_root_device_type
+ - ec2_security_group_ids
+ - ec2_security_group_names
+ - ec2_shutdown_state
+ - ec2_sourceDestCheck
+ - ec2_spot_instance_request_id
+ - ec2_state
+ - ec2_state_code
+ - ec2_state_reason
+ - ec2_status
+ - ec2_subnet_id
+ - ec2_tenancy
+ - ec2_virtualization_type
+ - ec2_vpc_id
+
+These variables are pulled out of a boto.ec2.instance object. There is a lack of
+consistency with variable spellings (camelCase and underscores) since this
+just loops through all variables the object exposes. It is preferred to use the
+ones with underscores when multiple exist.
+
+In addition, if an instance has AWS Tags associated with it, each tag is a new
+variable named:
+ - ec2_tag_[Key] = [Value]
+
+Security groups are comma-separated in 'ec2_security_group_ids' and 
+'ec2_security_group_names'.
+'''
+
+# (c) 2012, Peter Sankauskas
+#
+# This file is part of Ansible,
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+######################################################################
+
+import os
+import argparse
+import re
+from time import time
+from boto import ec2
+import ConfigParser
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+
+
+class Ec2Inventory(object):
+    def __init__(self):
+        ''' Main execution path '''
+        
+        # Inventory grouped by instance IDs, tags, security groups, regions,
+        # and availability zones
+        self.inventory = {}
+        
+        # Index of hostname (address) to instance ID
+        self.index = {}
+        
+        # Read settings and parse CLI arguments
+        self.read_settings()
+        self.parse_cli_args()
+        
+        # Cache
+        if self.args.refresh_cache:
+            self.do_api_calls_update_cache()  
+        elif not self.is_cache_valid():
+            self.do_api_calls_update_cache()
+        
+        # Data to print
+        if self.args.host:
+            data_to_print = self.get_host_info()
+        
+        elif self.args.list:
+            # Display list of instances for inventory
+            if len(self.inventory) == 0:
+                data_to_print = self.get_inventory_from_cache()
+            else:
+                data_to_print = self.json_format_dict(self.inventory, True)
+              
+        print data_to_print
+        
+        
+    def is_cache_valid(self):
+        ''' Determines if the cache files have expired, or if it is still valid '''
+
+        if os.path.isfile(self.cache_path_cache):
+            mod_time = os.path.getmtime(self.cache_path_cache)
+            current_time = time()
+            if (mod_time + self.cache_max_age) > current_time:
+                if os.path.isfile(self.cache_path_index):
+                    return True
+                
+        return False
+        
+        
+    def read_settings(self):
+        ''' Reads the settings from the ec2.ini file '''
+        
+        config = ConfigParser.SafeConfigParser()
+        config.read(os.path.dirname(os.path.realpath(__file__)) + '/ec2.ini')
+        
+        # Regions
+        self.regions = []
+        configRegions = config.get('ec2', 'regions')
+        if (configRegions == 'all'):
+            for regionInfo in ec2.regions():
+                self.regions.append(regionInfo.name)
+        else:
+            self.regions = configRegions.split(",")
+        
+        # Destination addresses
+        self.destination_variable = config.get('ec2', 'destination_variable')
+        self.vpc_destination_variable = config.get('ec2', 'vpc_destination_variable')
+
+        # Cache related
+        cache_path = config.get('ec2', 'cache_path')
+        self.cache_path_cache = cache_path + "/ansible-ec2.cache"
+        self.cache_path_index = cache_path + "/ansible-ec2.index"
+        self.cache_max_age = config.getint('ec2', 'cache_max_age')
+        
+        
+    def parse_cli_args(self):
+        ''' Command line argument processing '''
+        
+        parser = argparse.ArgumentParser(description='Produce an Ansible Inventory file based on EC2')
+        parser.add_argument('--list', action='store_true', default=True, 
+                           help='List instances (default: True)')
+        parser.add_argument('--host', action='store', 
+                           help='Get all the variables about a specific instance')
+        parser.add_argument('--refresh-cache', action='store_true', default=False, 
+                           help='Force refresh of cache by making API requests to EC2 (default: False - use cache files)')
+        self.args = parser.parse_args()
+        
+        
+    def do_api_calls_update_cache(self):
+        ''' Do API calls to each region, and save data in cache files '''
+        
+        for region in self.regions:
+            self.get_instances_by_region(region)
+        
+        self.write_to_cache(self.inventory, self.cache_path_cache)
+        self.write_to_cache(self.index, self.cache_path_index)
+
+
+    def get_instances_by_region(self, region):
+        ''' Makes an AWS EC2 API call to the list of instances in a particular
+        region '''
+        
+        conn = ec2.connect_to_region(region)
+        reservations = conn.get_all_instances()
+        for reservation in reservations:
+            for instance in reservation.instances:
+                self.add_instance(instance, region)
+
+
+    def get_instance(self, region, instance_id):
+        ''' Gets details about a specific instance '''
+        conn = ec2.connect_to_region(region)
+        reservations = conn.get_all_instances([instance_id])
+        for reservation in reservations:
+            for instance in reservation.instances:
+                return instance
+            
+        
+    def add_instance(self, instance, region):
+        ''' Adds an instance to the inventory and index, as long as it is
+        addressable '''
+        
+        # Only want running instances
+        if instance.state == 'terminated':
+            return
+        
+        # Select the best destination address
+        if instance.subnet_id:
+            dest = getattr(instance, self.vpc_destination_variable)
+        else:
+            dest =  getattr(instance, self.destination_variable)
+
+        if dest == None:
+            # Skip instances we cannot address (e.g. private VPC subnet)
+            return
+
+        # Add to index
+        self.index[dest] = [region, instance.id]
+
+        # Inventory: Group by instance ID (always a group of 1)
+        self.inventory[instance.id] = [dest]
+        
+        # Inventory: Group by region
+        self.push(self.inventory, region, dest)
+                        
+        # Inventory: Group by availability zone
+        self.push(self.inventory, instance.placement, dest)
+        
+        # Inventory: Group by security group
+        for group in instance.groups:
+            key = self.to_safe("security_group_" + group.name)
+            self.push(self.inventory, key, dest)
+                
+        # Inventory: Group by tag keys
+        for k, v in instance.tags.iteritems():
+            key = self.to_safe("tag_" + k + "=" + v)
+            self.push(self.inventory, key, dest)        
+    
+    
+    def get_host_info(self):
+        ''' Get variables about a specific host '''
+        
+        if len(self.index) == 0:
+            # Need to load index from cache
+            self.load_index_from_cache()
+        
+        (region, instance_id) = self.index[self.args.host]
+        instance = self.get_instance(region, instance_id)
+        instance_vars = {}
+        for key in vars(instance):
+            value = getattr(instance, key)
+            key = self.to_safe('ec2_' + key)
+            
+            # Handle complex types
+            if type(value) in [int, bool]:
+                instance_vars[key] = value
+            elif type(value) in [str, unicode]:
+                instance_vars[key] = value.strip()
+            elif type(value) == type(None):
+                instance_vars[key] = ''
+            elif key == 'ec2_region':
+                instance_vars[key] = value.name
+            elif key == 'ec2_tags':
+                for k, v in value.iteritems():
+                    key = self.to_safe('ec2_tag_' + k)
+                    instance_vars[key] = v
+            elif key == 'ec2_groups':
+                group_ids = []
+                group_names = []
+                for group in value:
+                    group_ids.append(group.id)
+                    group_names.append(group.name)
+                instance_vars["ec2_security_group_ids"] = ','.join(group_ids)
+                instance_vars["ec2_security_group_names"] = ','.join(group_names)
+            else:
+                pass
+                # TODO Product codes if someone finds them useful
+                #print key
+                #print type(value)
+                #print value
+                           
+        return self.json_format_dict(instance_vars, True)        
+    
+    
+    def push(self, my_dict, key, element):
+        ''' Pushed an element onto an array that may not have been defined in
+        the dict '''
+        
+        if key in my_dict:
+            my_dict[key].append(element);
+        else:
+            my_dict[key] = [element]        
+    
+    
+    def get_inventory_from_cache(self):
+        ''' Reads the inventory from the cache file and returns it as a JSON
+        object '''
+
+        cache = open(self.cache_path_cache, 'r')
+        json_inventory = cache.read()
+        return json_inventory
+
+
+    def load_index_from_cache(self):
+        ''' Reads the index from the cache file sets self.index '''
+
+        cache = open(self.cache_path_index, 'r')
+        json_index = cache.read()
+        self.index = json.loads(json_index)
+    
+    
+    def write_to_cache(self, data, filename):
+        ''' Writes data in JSON format to a file '''
+        
+        json_data = self.json_format_dict(data, True)
+        cache = open(filename, 'w')
+        cache.write(json_data)
+        cache.close()
+        
+        
+    def to_safe(self, word):
+        ''' Converts 'bad' characters in a string to underscores so they can be
+        used as Ansible groups '''
+        
+        return re.sub("[^A-Za-z0-9\-]", "_", word)
+        
+        
+    def json_format_dict(self, data, pretty=False):
+        ''' Converts a dict to a JSON object and dumps it as a formatted
+        string '''
+        
+        if pretty:
+            return json.dumps(data, sort_keys=True, indent=2)
+        else:
+            return json.dumps(data)
+        
+    
+# Run the script
+Ec2Inventory()
+

examples/scripts/yaml_to_ini.py

@@ -0,0 +1,206 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+import os
+import yaml
+import sys
+
+class InventoryParserYaml(object):
+    ''' Host inventory parser for ansible '''
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        sys.stderr.write("WARNING: YAML inventory files are deprecated in 0.6 and will be removed in 0.7, to migrate" +
+            " download and run https://github.com/ansible/ansible/blob/devel/examples/scripts/yaml_to_ini.py\n")
+
+        fh = open(filename)
+        data = fh.read()
+        fh.close()
+        self._hosts = {}
+        self._parse(data)
+
+    def _make_host(self, hostname):
+
+        if hostname in self._hosts:
+            return self._hosts[hostname]
+        else:
+            host = Host(hostname)
+            self._hosts[hostname] = host
+            return host 
+
+    # see file 'test/yaml_hosts' for syntax
+
+    def _parse(self, data):
+        # FIXME: refactor into subfunctions
+
+        all = Group('all')
+
+        ungrouped = Group('ungrouped')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        grouped_hosts = []
+
+        yaml = utils.parse_yaml(data)
+
+        # first add all groups
+        for item in yaml:
+            if type(item) == dict and 'group' in item:
+                group = Group(item['group'])
+
+                for subresult in item.get('hosts',[]):
+
+                    if type(subresult) in [ str, unicode ]:
+                        host = self._make_host(subresult)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+                    elif type(subresult) == dict:
+                        host = self._make_host(subresult['host'])
+                        vars = subresult.get('vars',{})
+                        if type(vars) == list:
+                            for subitem in vars:
+                                for (k,v) in subitem.items():
+                                    host.set_variable(k,v) 
+                        elif type(vars) == dict:
+                            for (k,v) in subresult.get('vars',{}).items():
+                                host.set_variable(k,v)
+                        else:
+                            raise errors.AnsibleError("unexpected type for variable")
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                vars = item.get('vars',{})
+                if type(vars) == dict:
+                    for (k,v) in item.get('vars',{}).items():
+                        group.set_variable(k,v) 
+                elif type(vars) == list:
+                    for subitem in vars:
+                        if type(subitem) != dict:
+                            raise errors.AnsibleError("expected a dictionary")
+                        for (k,v) in subitem.items():
+                            group.set_variable(k,v)
+
+                self.groups[group.name] = group
+                all.add_child_group(group)
+
+        # add host definitions
+        for item in yaml:
+            if type(item) in [ str, unicode ]:
+                host = self._make_host(item)
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+            elif type(item) == dict and 'host' in item:
+                host = self._make_host(item['host'])
+
+                vars = item.get('vars', {})
+                if type(vars)==list:
+                    varlist, vars = vars, {}
+                    for subitem in varlist:
+                        vars.update(subitem)
+                for (k,v) in vars.items():
+                    host.set_variable(k,v)
+
+                groups = item.get('groups', {})
+                if type(groups) in [ str, unicode ]:
+                    groups = [ groups ]
+                if type(groups)==list:
+                    for subitem in groups:
+                        if subitem in self.groups:
+                            group = self.groups[subitem]
+                        else:
+                            group = Group(subitem)
+                            self.groups[group.name] = group
+                            all.add_child_group(group)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+        # make sure ungrouped.hosts is the complement of grouped_hosts
+        ungrouped_hosts = [host for host in ungrouped.hosts if host not in grouped_hosts]
+
+if __name__ == "__main__":
+    if len(sys.argv) != 2:
+        print "usage: yaml_to_ini.py /path/to/ansible/hosts"
+        sys.exit(1)
+
+    result = ""
+
+    original = sys.argv[1]
+    yamlp = InventoryParserYaml(filename=sys.argv[1])
+    dirname = os.path.dirname(original)
+
+    group_names = [ g.name for g in yamlp.groups.values() ]
+
+    for group_name in sorted(group_names):
+
+        record = yamlp.groups[group_name]
+
+        if group_name == 'all':
+            continue
+ 
+        hosts = record.hosts
+        result = result + "[%s]\n" % record.name
+        for h in hosts:
+            result = result + "%s\n" % h.name
+        result = result + "\n"
+
+        groupfiledir = os.path.join(dirname, "group_vars")
+        if not os.path.exists(groupfiledir):
+            print "* creating: %s" % groupfiledir
+            os.makedirs(groupfiledir)
+        groupfile = os.path.join(groupfiledir, group_name)
+        print "* writing group variables for %s into %s" % (group_name, groupfile)
+        groupfh = open(groupfile, 'w')
+        groupfh.write(yaml.dump(record.get_variables()))
+        groupfh.close()
+
+    for (host_name, host_record) in yamlp._hosts.iteritems():
+        hostfiledir = os.path.join(dirname, "host_vars")
+        if not os.path.exists(hostfiledir):
+            print "* creating: %s" % hostfiledir
+            os.makedirs(hostfiledir)
+        hostfile = os.path.join(hostfiledir, host_record.name)
+        print "* writing host variables for %s into %s" % (host_record.name, hostfile)
+        hostfh = open(hostfile, 'w')
+        hostfh.write(yaml.dump(host_record.get_variables()))
+        hostfh.close()
+ 
+
+    # also need to keep a hash of variables per each host
+    # and variables per each group
+    # and write those to disk
+
+    newfilepath = os.path.join(dirname, "hosts.new")
+    fdh = open(newfilepath, 'w')
+    fdh.write(result)
+    fdh.close()
+
+    print "* COMPLETE: review your new inventory file and replace your original when ready"
+    print "*           new inventory file saved as %s" % newfilepath  
+    print "*           edit group specific variables in %s/group_vars/" % dirname
+    print "*           edit host specific variables in %s/host_vars/" % dirname
+
+    # now need to write this to disk as (oldname).new
+    # and inform the user

hacking/README

@@ -6,6 +6,3 @@ To use it from the root of a checkout:
     $ . ./hacking/env-setup
 
 Note the space between the '.' and the './'
-
-Man pages will not load until you run 'make docs' from the root of the
-checkout.

hacking/env-setup

@@ -1,18 +1,35 @@
-# -*- mode: shell-script -*-
-PREFIX_PYTHONPATH="`pwd`/lib"
-PREFIX_PATH="`pwd`/bin"
-PREFIX_MANPATH="`pwd`/docs/man"
-PREFIX_ANS_LIBRARY="`pwd`/library"
+#!/bin/bash
+# usage: source ./hacking/env-setup [-q]
+#    modifies environment for running Ansible from checkout
+
+# When run using source as directed, $0 gets set to bash, so we must use $BASH_SOURCE
+if [ -n "$BASH_SOURCE" ] ; then
+    HACKING_DIR=`dirname $BASH_SOURCE`
+else
+    HACKING_DIR="$PWD/hacking"
+fi
+# The below is an alternative to readlink -fn which doesn't exist on OS X
+# Source: http://stackoverflow.com/a/1678636
+FULL_PATH=`python -c "import os, sys; print os.path.realpath('$HACKING_DIR')"`
+ANSIBLE_HOME=`dirname $FULL_PATH`
+
+PREFIX_PYTHONPATH="$ANSIBLE_HOME/lib"
+PREFIX_PATH="$ANSIBLE_HOME/bin"
+PREFIX_MANPATH="$ANSIBLE_HOME/docs/man"
 
-echo "Prefixing PYTHONPATH with $PREFIX_PYTHONPATH"
 export PYTHONPATH=$PREFIX_PYTHONPATH:$PYTHONPATH
-
-echo "Prefixing PATH with $PREFIX_PATH"
 export PATH=$PREFIX_PATH:$PATH
-echo $PATH
-
-echo "Prefixing MANPATH with $PREFIX_MANPATH"
+export ANSIBLE_LIBRARY="$ANSIBLE_HOME/library"
 export MANPATH=$PREFIX_MANPATH:$MANPATH
 
-echo "Setting ANSIBLE_LIBRARY to ${PREFIX_ANS_LIBRARY}"
-export ANSIBLE_LIBRARY=${PREFIX_ANS_LIBRARY}
+# Print out values unless -q is set
+
+if [ $# -eq 0 -o "$1" != "-q" ] ; then
+    echo "PATH=$PATH"
+    echo "PYTHONPATH=$PYTHONPATH"
+    echo "ANSIBLE_LIBRARY=$ANSIBLE_LIBRARY"
+    echo "MANPATH=$MANPATH"
+    
+    echo "Reminder: specify your host file with -i"
+    echo "Done."
+fi

hacking/test-module

@@ -0,0 +1,151 @@
+#!/usr/bin/python
+
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# this script is for testing modules without running through the
+# entire guts of ansible, and is very helpful for when developing
+# modules
+#
+# example:
+#    test-module -m ../library/command -a "/bin/sleep 3"
+#    test-module -m ../library/service -a "name=httpd ensure=restarted"
+#    test-module -m ../library/service -a "name=httpd ensure=restarted" --debugger /usr/bin/pdb
+
+import sys
+import base64
+import os
+import subprocess
+import traceback
+import optparse
+import ansible.utils as utils
+import ansible.module_common as module_common
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+
+def parse():
+    """parse command line
+
+    :return : (options, args)"""
+    parser = optparse.OptionParser()
+
+    parser.usage = "%prog -[options] (-h for help)"
+
+    parser.add_option('-m', '--module-path', dest='module_path',
+        help="REQUIRED: full path of module source to execute")
+    parser.add_option('-a', '--args', dest='module_args', default="",
+        help="module argument string")
+    parser.add_option('-D', '--debugger', dest='debugger', 
+        help="path to python debugger (e.g. /usr/bin/pdb)")
+    options, args = parser.parse_args()
+    if not options.module_path:
+        parser.print_help()
+        sys.exit(1)
+    else:
+        return options, args
+
+def write_argsfile(argstring):
+    """ Write args to a file for old-style module's use. """
+    argspath = os.path.expanduser("~/.ansible_test_module_arguments")
+    argsfile = open(argspath, 'w')
+    argsfile.write(argstring)
+    argsfile.close()
+    return argspath
+
+def boilerplate_module(modfile, args):
+    """ simulate what ansible does with new style modules """
+
+    module_fh = open(modfile)
+    module_data = module_fh.read()
+    included_boilerplate = module_data.find(module_common.REPLACER) != -1
+    module_fh.close()
+
+    if included_boilerplate:
+
+        module_data = module_data.replace(module_common.REPLACER, module_common.MODULE_COMMON)
+        encoded_args = "\"\"\"%s\"\"\"" % args.replace("\"","\\\"")
+        module_data = module_data.replace(module_common.REPLACER_ARGS, encoded_args)
+
+        modfile2_path = os.path.expanduser("~/.ansible_module_generated")
+        print "* including generated source, if any, saving to: %s" % modfile2_path
+        print "* this will offset any line numbers in tracebacks/debuggers!"
+        modfile2 = open(modfile2_path, 'w')
+        modfile2.write(module_data)
+        modfile2.close()
+        modfile = modfile2_path
+        return (modfile2_path, included_boilerplate)
+    else:
+        print "* module boilerplate substitution not requested in module, line numbers will be unaltered"
+        return (modfile, included_boilerplate)
+
+def runtest( modfile, argspath):
+    """Test run a module, piping it's output for reporting."""
+
+    os.system("chmod +x %s" % modfile)
+
+    invoke = "%s" % (modfile)
+    if argspath is not None:     
+        invoke = "%s %s" % (modfile, argspath)
+
+    cmd = subprocess.Popen(invoke, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+
+    try:
+        print "***********************************"
+        print "RAW OUTPUT"
+        print out
+        print err
+        results = utils.parse_json(out)
+    except:
+        print "***********************************"
+        print "INVALID OUTPUT FORMAT"
+        print out
+        traceback.print_exc()
+        sys.exit(1)
+
+    print "***********************************"
+    print "PARSED OUTPUT"
+    print utils.jsonify(results,format=True)
+
+def rundebug(debugger, modfile, argspath):
+    """Run interactively with console debugger."""
+
+    if argspath is not None:
+        subprocess.call("%s %s %s" % (debugger, modfile, argspath), shell=True)
+    else:
+        subprocess.call("%s %s" % (debugger, modfile), shell=True)
+
+def main(): 
+
+    options, args = parse()
+    (modfile, is_new_style) = boilerplate_module(options.module_path, options.module_args)
+    argspath=None
+    if not is_new_style:
+        argspath = write_argsfile(options.module_args)
+    if options.debugger: 
+        rundebug(options.debugger, modfile, argspath)
+    else:
+        runtest(modfile, argspath)
+	
+if __name__ == "__main__":
+    main()
+
+

lib/ansible/__init__.py

@@ -14,5 +14,5 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-__version__ = '0.0.1'
+__version__ = '0.6'
 __author__ = 'Michael DeHaan'

lib/ansible/callbacks.py

@@ -0,0 +1,398 @@
+# (C) 2012, Michael DeHaan, <michael.dehaan@gmail.com>
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import utils
+import sys
+import getpass
+import os
+import subprocess
+from ansible.color import stringc
+
+cowsay = None
+if os.path.exists("/usr/bin/cowsay"):
+    cowsay = "/usr/bin/cowsay"
+elif os.path.exists("/usr/games/cowsay"):
+    cowsay = "/usr/games/cowsay"
+
+class AggregateStats(object):
+    ''' holds stats about per-host activity during playbook runs '''   
+ 
+    def __init__(self):
+
+        self.processed   = {}
+        self.failures    = {}
+        self.ok          = {}
+        self.dark        = {}
+        self.changed     = {}
+        self.skipped     = {}
+
+    def _increment(self, what, host):
+        ''' helper function to bump a statistic '''
+
+        self.processed[host] = 1
+        prev = (getattr(self, what)).get(host, 0)
+        getattr(self, what)[host] = prev+1
+
+    def compute(self, runner_results, setup=False, poll=False):
+        ''' walk through all results and increment stats '''
+ 
+        for (host, value) in runner_results.get('contacted', {}).iteritems():
+            if ('failed' in value and bool(value['failed'])) or ('rc' in value and value['rc'] != 0):
+                self._increment('failures', host)
+            elif 'skipped' in value and bool(value['skipped']):
+                self._increment('skipped', host)
+            elif 'changed' in value and bool(value['changed']):
+                if not setup and not poll:
+                    self._increment('changed', host)
+                self._increment('ok', host)
+            else:
+                if not poll or ('finished' in value and bool(value['finished'])):
+                    self._increment('ok', host)
+
+        for (host, value) in runner_results.get('dark', {}).iteritems():
+            self._increment('dark', host)
+            
+
+    def summarize(self, host):
+        ''' return information about a particular host '''
+
+        return dict(
+            ok          = self.ok.get(host, 0),
+            failures    = self.failures.get(host, 0),
+            unreachable = self.dark.get(host,0),
+            changed     = self.changed.get(host, 0),
+            skipped     = self.skipped.get(host, 0)
+        )
+
+########################################################################
+
+def regular_generic_msg(hostname, result, oneline, caption):
+    ''' output on the result of a module run that is not command '''
+
+    if not oneline:
+        return "%s | %s >> %s\n" % (hostname, caption, utils.jsonify(result,format=True))
+    else:
+        return "%s | %s >> %s\n" % (hostname, caption, utils.jsonify(result))
+
+
+def banner(msg):
+
+    if cowsay != None:
+        cmd = subprocess.Popen("%s -W 60 \"%s\"" % (cowsay, msg), 
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
+        (out, err) = cmd.communicate()
+        return "%s\n" % out 
+    else:
+        return "\n%s ********************* " % msg
+
+def command_generic_msg(hostname, result, oneline, caption):
+    ''' output the result of a command run '''
+
+    rc     = result.get('rc', '0')
+    stdout = result.get('stdout','')
+    stderr = result.get('stderr', '')
+    msg    = result.get('msg', '')
+
+    if not oneline:
+        buf = "%s | %s | rc=%s >>\n" % (hostname, caption, result.get('rc',0))
+        if stdout:
+            buf += stdout
+        if stderr:
+            buf += stderr
+        if msg:
+            buf += msg
+        return buf + "\n"
+    else:
+        if stderr:
+            return "%s | %s | rc=%s | (stdout) %s (stderr) %s\n" % (hostname, caption, rc, stdout, stderr)
+        else:
+            return "%s | %s | rc=%s | (stdout) %s\n" % (hostname, caption, rc, stdout)
+
+def host_report_msg(hostname, module_name, result, oneline):
+    ''' summarize the JSON results for a particular host '''
+
+    failed = utils.is_failed(result)
+    msg = ''
+    if module_name in [ 'command', 'shell', 'raw' ] and 'ansible_job_id' not in result:
+        if not failed:
+            msg = command_generic_msg(hostname, result, oneline, 'success')
+        else:
+            msg = command_generic_msg(hostname, result, oneline, 'FAILED')
+    else:
+        if not failed:
+            msg = regular_generic_msg(hostname, result, oneline, 'success')
+        else:
+            msg = regular_generic_msg(hostname, result, oneline, 'FAILED')
+    return msg
+
+###############################################
+
+class DefaultRunnerCallbacks(object):
+    ''' no-op callbacks for API usage of Runner() if no callbacks are specified '''
+
+    def __init__(self):
+        pass
+
+    def on_failed(self, host, res, ignore_errors=False):
+        pass
+
+    def on_ok(self, host, res):
+        pass
+
+    def on_error(self, host, msg):
+        pass
+
+    def on_skipped(self, host):
+        pass
+
+    def on_unreachable(self, host, res):
+        pass
+
+    def on_no_hosts(self):
+        pass
+
+    def on_async_poll(self, host, res, jid, clock):
+        pass
+
+    def on_async_ok(self, host, res, jid):
+        pass
+
+    def on_async_failed(self, host, res, jid):
+        pass
+
+########################################################################
+
+class CliRunnerCallbacks(DefaultRunnerCallbacks):
+    ''' callbacks for use by /usr/bin/ansible '''
+
+    def __init__(self):
+
+        # set by /usr/bin/ansible later
+        self.options = None 
+        self._async_notified = {}
+
+    def on_failed(self, host, res, ignore_errors=False):
+
+        self._on_any(host,res)
+
+    def on_ok(self, host, res):
+
+        self._on_any(host,res)
+ 
+    def on_unreachable(self, host, res):
+
+        if type(res) == dict:
+            res = res.get('msg','')
+        print "%s | FAILED => %s" % (host, res)
+        if self.options.tree:
+            utils.write_tree_file(
+                self.options.tree, host, 
+                utils.jsonify(dict(failed=True, msg=res),format=True)
+            )
+ 
+    def on_skipped(self, host):
+        pass
+
+    def on_error(self, host, err):
+
+        print >>sys.stderr, "err: [%s] => %s\n" % (host, err)
+    
+    def on_no_hosts(self):
+
+        print >>sys.stderr, "no hosts matched\n"
+
+    def on_async_poll(self, host, res, jid, clock):
+
+        if jid not in self._async_notified:
+            self._async_notified[jid] = clock + 1
+        if self._async_notified[jid] > clock:
+            self._async_notified[jid] = clock
+            print "<job %s> polling, %ss remaining"%(jid, clock)
+
+    def on_async_ok(self, host, res, jid):
+
+        print "<job %s> finished on %s => %s"%(jid, host, utils.jsonify(res,format=True))
+
+    def on_async_failed(self, host, res, jid):
+
+        print "<job %s> FAILED on %s => %s"%(jid, host, utils.jsonify(res,format=True))
+
+    def _on_any(self, host, result):
+
+        print host_report_msg(host, self.options.module_name, result, self.options.one_line)
+        if self.options.tree:
+            utils.write_tree_file(self.options.tree, host, utils.jsonify(result,format=True))
+
+########################################################################
+
+class PlaybookRunnerCallbacks(DefaultRunnerCallbacks):
+    ''' callbacks used for Runner() from /usr/bin/ansible-playbook '''
+
+    def __init__(self, stats, verbose=False):
+
+        self.stats = stats
+        self._async_notified = {}
+        self.verbose = verbose
+
+    def on_unreachable(self, host, msg):
+
+        item = None
+        if type(msg) == dict:
+            item = msg.get('item', None)
+
+        if item:
+            print "fatal: [%s] => (item=%s) => %s" % (host, item, msg)
+        else:
+            print "fatal: [%s] => %s" % (host, msg)
+
+    def on_failed(self, host, results, ignore_errors=False):
+
+        item = results.get('item', None)
+
+        if item:
+            msg = "failed: [%s] => (item=%s) => %s" % (host, item, utils.jsonify(results))
+        else:
+            msg = "failed: [%s] => %s" % (host, utils.jsonify(results))
+
+        print stringc(msg, 'red')
+        if ignore_errors:
+            print stringc("...ignoring", 'yellow')
+
+    def on_ok(self, host, host_result):
+
+        item = host_result.get('item', None)
+
+        # show verbose output for non-setup module results if --verbose is used
+        msg = '' 
+        if not self.verbose or host_result.get("verbose_override",None) is not None:
+            if item:
+                msg = "ok: [%s] => (item=%s)" % (host,item)
+            else:   
+                if 'ansible_job_id' not in host_result or 'finished' in host_result:
+                    msg = "ok: [%s]" % (host)
+        else:
+            # verbose ...
+            if item:
+                msg = "ok: [%s] => (item=%s) => %s" % (host, item, utils.jsonify(host_result))
+            else:
+                if 'ansible_job_id' not in host_result or 'finished' in host_result:
+                    msg = "ok: [%s] => %s" % (host, utils.jsonify(host_result))
+
+        if msg != '':
+            if not 'changed' in host_result or not host_result['changed']:
+                print stringc(msg, 'green')
+            else:
+                print stringc(msg, 'yellow')
+
+    def on_error(self, host, err):
+
+        item = err.get('item', None)
+        msg = ''
+        if item:
+            msg = "err: [%s] => (item=%s) => %s" % (host, item, err)
+        else:
+            msg = "err: [%s] => %s" % (host, err)
+
+        msg = stringc(msg, 'red')
+        print >>sys.stderr, msg
+
+    def on_skipped(self, host, item=None):
+
+        msg = ''
+        if item:
+            msg = "skipping: [%s] => (item=%s)" % (host, item)
+        else:
+            msg = "skipping: [%s]" % host
+        print stringc(msg, 'yellow')
+
+    def on_no_hosts(self):
+
+        print stringc("no hosts matched or remaining\n", 'red')
+
+    def on_async_poll(self, host, res, jid, clock):
+
+        if jid not in self._async_notified:
+            self._async_notified[jid] = clock + 1
+        if self._async_notified[jid] > clock:
+            self._async_notified[jid] = clock
+            msg = "<job %s> polling, %ss remaining"%(jid, clock)
+            print stringc(msg, 'cyan')
+
+    def on_async_ok(self, host, res, jid):
+
+        msg = "<job %s> finished on %s"%(jid, host)
+        print stringc(msg, 'cyan')
+
+    def on_async_failed(self, host, res, jid):
+
+        msg = "<job %s> FAILED on %s"%(jid, host)
+        print stringc(msg, 'red')
+
+
+########################################################################
+
+class PlaybookCallbacks(object):
+    ''' playbook.py callbacks used by /usr/bin/ansible-playbook '''
+  
+    def __init__(self, verbose=False):
+
+        self.verbose = verbose
+
+    def on_start(self):
+
+        pass
+
+    def on_notify(self, host, handler):
+
+        pass
+
+    def on_task_start(self, name, is_conditional):
+
+        msg = "TASK: [%s]" % name
+        if is_conditional:
+            msg = "NOTIFIED: [%s]" % name
+        print banner(msg)
+
+    def on_vars_prompt(self, varname, private=True, prompt=None):
+
+        if prompt:
+            msg = prompt
+        else:
+            msg = 'input for %s: ' % varname
+        if private:
+            return getpass.getpass(msg)
+        return raw_input(msg)
+        
+    def on_setup(self):
+
+        print banner("GATHERING FACTS")
+    
+    def on_import_for_host(self, host, imported_file):
+
+        msg = "%s: importing %s" % (host, imported_file)
+        print stringc(msg, 'cyan')
+
+    def on_not_import_for_host(self, host, missing_file):
+
+        msg = "%s: not importing file: %s" % (host, missing_file)
+        print stringc(msg, 'cyan')
+
+    def on_play_start(self, pattern):
+
+        print banner("PLAY [%s]" % pattern)
+
+

lib/ansible/color.py

@@ -0,0 +1,59 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import sys
+
+ANSIBLE_COLOR=True
+if os.getenv("ANSIBLE_NOCOLOR") is not None:
+    ANSIBLE_COLOR=False
+if not sys.stdout.isatty():
+    ANSIBLE_COLOR=False
+
+# --- begin "pretty"
+#
+# pretty - A miniature library that provides a Python print and stdout
+# wrapper that makes colored terminal text easier to use (eg. without
+# having to mess around with ANSI escape sequences). This code is public
+# domain - there is no license except that you must leave this header.
+#
+# Copyright (C) 2008 Brian Nez <thedude at bri1 dot com>
+#
+# http://nezzen.net/2008/06/23/colored-text-in-python-using-ansi-escape-sequences/
+
+codeCodes = {
+    'black':     '0;30',        'bright gray':    '0;37',
+    'blue':      '0;34',        'white':          '1;37',
+    'green':     '0;32',        'bright blue':    '1;34',
+    'cyan':      '0;36',        'bright green':   '1;32',
+    'red':       '0;31',        'bright cyan':    '1;36',
+    'purple':    '0;35',        'bright red':     '1;31',
+    'yellow':    '0;33',        'bright purple':  '1;35',
+    'dark gray': '1;30',        'bright yellow':  '1;33',
+    'normal':    '0'
+}
+
+def stringc(text, color):
+    """String in color."""
+
+    if ANSIBLE_COLOR:
+        return "\033["+codeCodes[color]+"m"+text+"\033[0m"
+    else:
+        return text
+
+# --- end "pretty"
+

lib/ansible/constants.py

@@ -14,20 +14,25 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
 
 import os
 
+DEFAULT_HOST_LIST      = os.environ.get('ANSIBLE_HOSTS', '/etc/ansible/hosts')
+DEFAULT_MODULE_PATH    = os.environ.get('ANSIBLE_LIBRARY', '/usr/share/ansible')
+DEFAULT_REMOTE_TMP     = os.environ.get('ANSIBLE_REMOTE_TMP', '$HOME/.ansible/tmp')
 
-# control side (aka 'overlord')
-DEFAULT_HOST_LIST      = os.environ.get('ANSIBLE_HOSTS',
-                                        '/etc/ansible/hosts')
-DEFAULT_MODULE_PATH    = os.environ.get('ANSIBLE_LIBRARY',
-                                        '/usr/share/ansible')
 DEFAULT_MODULE_NAME    = 'command'
 DEFAULT_PATTERN        = '*'
-DEFAULT_FORKS          = 5
-DEFAULT_MODULE_ARGS    = ''
-DEFAULT_TIMEOUT        = 10
-DEFAULT_REMOTE_USER    = 'root'
+DEFAULT_FORKS          = os.environ.get('ANSIBLE_FORKS',5)
+DEFAULT_MODULE_ARGS    = os.environ.get('ANSIBLE_MODULE_ARGS','')
+DEFAULT_TIMEOUT        = os.environ.get('ANSIBLE_TIMEOUT',10)
+DEFAULT_POLL_INTERVAL  = os.environ.get('ANSIBLE_POLL_INTERVAL',15)
+DEFAULT_REMOTE_USER    = os.environ.get('ANSIBLE_REMOTE_USER','root')
 DEFAULT_REMOTE_PASS    = None
+DEFAULT_PRIVATE_KEY_FILE    = os.environ.get('ANSIBLE_PRIVATE_KEY_FILE',None)
+DEFAULT_SUDO_PASS      = None
+DEFAULT_SUDO_USER      = os.environ.get('ANSIBLE_SUDO_USER','root')
+DEFAULT_REMOTE_PORT    = 22
+DEFAULT_TRANSPORT      = os.environ.get('ANSIBLE_TRANSPORT','paramiko')
+DEFAULT_TRANSPORT_OPTS = ['local', 'paramiko', 'ssh']
+

lib/ansible/errors.py

@@ -0,0 +1,32 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+class AnsibleError(Exception):
+    ''' The base Ansible exception from which all others should subclass '''
+
+    def __init__(self, msg):
+        self.msg = msg
+    
+    def __str__(self):
+        return self.msg
+
+class AnsibleFileNotFound(AnsibleError):
+    pass
+
+class AnsibleConnectionFailed(AnsibleError):
+    pass
+

lib/ansible/inventory/__init__.py

@@ -0,0 +1,213 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import fnmatch
+import os
+
+import subprocess
+import ansible.constants as C
+from ansible.inventory.ini import InventoryParser
+from ansible.inventory.yaml import InventoryParserYaml
+from ansible.inventory.script import InventoryScript
+from ansible.inventory.group import Group
+from ansible.inventory.host import Host
+from ansible import errors
+from ansible import utils
+
+class Inventory(object):
+    """ 
+    Host inventory for ansible.
+    """
+
+    __slots__ = [ 'host_list', 'groups', '_restriction', '_is_script',
+                  'parser', '_vars_per_host', '_vars_per_group', '_hosts_cache' ] 
+
+    def __init__(self, host_list=C.DEFAULT_HOST_LIST):
+
+        # the host file file, or script path, or list of hosts
+        # if a list, inventory data will NOT be loaded
+        self.host_list = host_list
+
+        # caching to avoid repeated calculations, particularly with
+        # external inventory scripts.
+ 
+        self._vars_per_host  = {}
+        self._vars_per_group = {}
+        self._hosts_cache    = {}
+
+        # the inventory object holds a list of groups
+        self.groups = []
+ 
+        # a list of host(names) to contain current inquiries to
+        self._restriction = None
+
+        # whether the inventory file is a script
+        self._is_script = False
+
+        if type(host_list) in [ str, unicode ]:
+            if host_list.find(",") != -1:
+                host_list = host_list.split(",")
+                host_list = [ h for h in host_list if h and h.strip() ]
+
+        if type(host_list) == list:
+            all = Group('all')
+            self.groups = [ all ]
+            for x in host_list:
+                if x.find(":") != -1:
+                    tokens = x.split(":",1)
+                    all.add_host(Host(tokens[0], tokens[1]))
+                else:
+                    all.add_host(Host(x))
+        elif os.access(host_list, os.X_OK):
+            self._is_script = True
+            self.parser = InventoryScript(filename=host_list)
+            self.groups = self.parser.groups.values()
+        else:
+            data = file(host_list).read()
+            if not data.startswith("---"):
+                self.parser = InventoryParser(filename=host_list)
+                self.groups = self.parser.groups.values()
+            else:         
+                self.parser = InventoryParserYaml(filename=host_list)
+                self.groups = self.parser.groups.values()
+      
+    def _match(self, str, pattern_str):
+        return fnmatch.fnmatch(str, pattern_str)
+
+    # TODO: cache this logic so if called a second time the result is not recalculated
+    def get_hosts(self, pattern="all"):
+        """ Get all host objects matching the pattern """
+        hosts = {}
+        patterns = pattern.replace(";",":").split(":")
+
+        groups = self.get_groups()
+        for pat in patterns:
+            if pat.startswith("!"):
+                pat = pat[1:]
+                inverted = True
+            else:
+                inverted = False
+            for group in groups:
+                for host in group.get_hosts():
+                    if self._match(group.name, pat) or pat == 'all' or self._match(host.name, pat):
+                        # must test explicitly for None because [] means no hosts allowed
+                        if self._restriction==None or host.name in self._restriction: 
+                            if inverted:
+                                if host.name in hosts:
+                                    del hosts[host.name]
+                            else:
+                                hosts[host.name] = host
+        return sorted(hosts.values(), key=lambda x: x.name)
+
+    def get_groups(self):
+        return self.groups
+
+    def get_host(self, hostname):
+        if hostname not in self._hosts_cache:
+            self._hosts_cache[hostname] = self._get_host(hostname)
+        return self._hosts_cache[hostname]
+
+    def _get_host(self, hostname):
+        for group in self.groups:
+            for host in group.get_hosts():
+                if hostname == host.name:
+                    return host
+        return None
+
+    def get_group(self, groupname):
+        for group in self.groups:
+            if group.name == groupname:
+                return group
+        return None
+   
+    def get_group_variables(self, groupname):
+        if groupname not in self._vars_per_group:
+            self._vars_per_group[groupname] = self._get_group_variables(groupname)
+        return self._vars_per_group[groupname]
+
+    def _get_group_variables(self, groupname):
+        group = self.get_group(groupname)
+        if group is None:
+            raise Exception("group not found: %s" % groupname)
+        return group.get_variables()
+
+    def get_variables(self, hostname):
+        if hostname not in self._vars_per_host:
+            self._vars_per_host[hostname] = self._get_variables(hostname)
+        return self._vars_per_host[hostname]
+
+    def _get_variables(self, hostname):
+
+        if self._is_script:
+            host = self.get_host(hostname)
+            cmd = subprocess.Popen(
+                [self.host_list,"--host",hostname], 
+                stdout=subprocess.PIPE, 
+                stderr=subprocess.PIPE
+            )
+            (out, err) = cmd.communicate()
+            results = utils.parse_json(out)
+
+            # FIXME: this is a bit redundant with host.py and should share code
+            results['inventory_hostname'] = hostname
+            results['inventory_hostname_short'] = hostname.split('.')[0]
+            groups = [ g.name for g in host.get_groups() if g.name != 'all' ]
+            results['group_names'] = sorted(groups)
+
+            return results
+
+        host = self.get_host(hostname)
+        if host is None:
+            raise Exception("host not found: %s" % hostname)
+        return host.get_variables()
+
+    def add_group(self, group):
+        self.groups.append(group)
+
+    def list_hosts(self, pattern="all"):
+        return [ h.name for h in self.get_hosts(pattern) ]
+
+    def list_groups(self):
+        return [ g.name for g in self.groups ] 
+
+    def get_restriction(self):
+        return self._restriction
+
+    def restrict_to(self, restriction, append_missing=False):
+        """ Restrict list operations to the hosts given in restriction """
+        if type(restriction) != list:
+            restriction = [ restriction ]
+        self._restriction = restriction
+
+    def lift_restriction(self):
+        """ Do not restrict list operations """
+
+        self._restriction = None
+
+    def is_file(self):
+        """ did inventory come from a file? """
+        if not isinstance(self.host_list, basestring):
+            return False
+        return os.path.exists(self.host_list)
+
+    def basedir(self):
+        """ if inventory came from a file, what's the directory? """
+        if not self.is_file():
+            return None
+        return os.path.dirname(self.host_list)

lib/ansible/inventory/expand_hosts.py

@@ -0,0 +1,97 @@
+# (c) 2012, Zettar Inc.
+# Written by Chin Fang <fangchin@zettar.com>
+#
+# This file is part of Ansible
+#
+# This module is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This software is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this software.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+'''
+This module is for enhancing ansible's inventory parsing capability such
+that it can deal with hostnames specified using a simple pattern in the
+form of [beg:end], example: [1:5] where if beg is not specified, it
+defaults to 0.
+
+If beg is given and is left-zero-padded, e.g. '001', it is taken as a
+formatting hint when the range is expanded. e.g. [001:010] is to be
+expanded into 001, 002 ...009, 010.
+
+Note that when beg is specified with left zero padding, then the length of
+end must be the same as that of beg, else a exception is raised.
+'''
+
+from ansible import errors
+
+def detect_range(line = None):
+    '''
+    A helper function that checks a given host line to see if it contains
+    a range pattern descibed in the docstring above.
+
+    Returnes True if the given line contains a pattern, else False.
+    '''
+    if (not line.startswith("[") and 
+        line.find("[") != -1 and 
+        line.find(":") != -1 and
+        line.find("]") != -1 and
+        line.index("[") < line.index(":") < line.index("]")):   
+        return True
+    else:
+        return False
+        
+def expand_hostname_range(line = None):
+    '''
+    A helper function that expands a given line that contains a pattern
+    specified in top docstring, and returns a list that consists of the
+    expanded version.
+
+    The '[' and ']' characters are used to maintain the pseudo-code
+    appearance. They are replaced in this function with '|' to ease
+    string splitting.
+
+    References: http://ansible.github.com/patterns.html#hosts-and-groups
+    '''
+    all_hosts = []
+    if line:
+        # A hostname such as db[1:6]-node is considered to consists
+        # three parts: 
+        # head: 'db'
+        # nrange: [1:6]; range() is a built-in. Can't use the name
+        # tail: '-node'
+        
+        (head, nrange, tail) = line.replace('[','|').replace(']','|').split('|')
+        bounds = nrange.split(":")
+        if len(bounds) != 2:
+            raise errors.AnsibleError("host range incorrectly specified")
+        beg = bounds[0]
+        end = bounds[1]
+        if not beg:
+            beg = "0"
+        if not end:
+            raise errors.AnsibleError("host range end value missing")
+        if beg[0] == '0' and len(beg) > 1:
+            rlen = len(beg) # range length formatting hint
+        else:
+            rlen = None
+        if rlen > 1 and rlen != len(end):
+            raise errors.AnsibleError("host range format incorrectly specified!")
+                
+        for _ in range(int(beg), int(end)+1):
+            if rlen:
+                rseq = str(_).zfill(rlen) # range sequence
+            else:
+                rseq = str(_)
+            hname = ''.join((head, rseq, tail))
+            all_hosts.append(hname)
+                   
+        return all_hosts

lib/ansible/inventory/group.py

@@ -0,0 +1,77 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+class Group(object):
+    ''' a group of ansible hosts '''
+
+    __slots__ = [ 'name', 'hosts', 'vars', 'child_groups', 'parent_groups' ]
+
+    def __init__(self, name=None):
+
+        self.name = name
+        self.hosts = []
+        self.vars = {}
+        self.child_groups = []
+        self.parent_groups = []
+        if self.name is None:
+            raise Exception("group name is required")
+
+    def add_child_group(self, group):
+
+        if self == group:
+            raise Exception("can't add group to itself")
+        self.child_groups.append(group)
+        group.parent_groups.append(self)
+
+    def add_host(self, host):
+
+        self.hosts.append(host)
+        host.add_group(self)
+
+    def set_variable(self, key, value):
+
+        self.vars[key] = value
+
+    def get_hosts(self):
+
+        hosts = []
+        for kid in self.child_groups:
+            hosts.extend(kid.get_hosts())
+        hosts.extend(self.hosts)
+        return hosts 
+
+    def get_variables(self):
+
+        vars = {}
+        # FIXME: verify this variable override order is what we want
+        for ancestor in self.get_ancestors():
+            vars.update(ancestor.get_variables())
+        vars.update(self.vars)
+        return vars
+
+    def _get_ancestors(self):
+
+        results = {}
+        for g in self.parent_groups:
+            results[g.name] = g
+            results.update(g._get_ancestors())
+        return results
+
+    def get_ancestors(self):
+
+        return self._get_ancestors().values()
+

lib/ansible/inventory/host.py

@@ -0,0 +1,66 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+from ansible import errors
+import ansible.constants as C
+
+class Host(object):
+    ''' a single ansible host '''
+
+    __slots__ = [ 'name', 'vars', 'groups' ]
+
+    def __init__(self, name=None, port=None):
+
+        self.name = name
+        self.vars = {}
+        self.groups = []
+        if port and port != C.DEFAULT_REMOTE_PORT:
+            self.set_variable('ansible_ssh_port', int(port))
+
+        if self.name is None:
+            raise Exception("host name is required")
+
+    def add_group(self, group):
+
+        self.groups.append(group)
+
+    def set_variable(self, key, value):
+
+        self.vars[key]=value
+
+    def get_groups(self):
+
+        groups = {}
+        for g in self.groups:
+            groups[g.name] = g
+            ancestors = g.get_ancestors()
+            for a in ancestors:
+                groups[a.name] = a                
+        return groups.values()
+
+    def get_variables(self):
+
+        results = {}
+        for group in self.groups:
+            results.update(group.get_variables())
+        results.update(self.vars)
+        results['inventory_hostname'] = self.name
+        results['inventory_hostname_short'] = self.name.split('.')[0]
+        groups = self.get_groups()
+        results['group_names'] = sorted([ g.name for g in groups if g.name != 'all'])
+        return results
+

lib/ansible/inventory/ini.py

@@ -0,0 +1,174 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import fnmatch
+import os
+import subprocess
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible.inventory.expand_hosts import detect_range
+from ansible.inventory.expand_hosts import expand_hostname_range
+from ansible import errors
+from ansible import utils
+
+class InventoryParser(object):
+    """ 
+    Host inventory for ansible.
+    """
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        fh = open(filename)
+        self.lines = fh.readlines()
+        self.groups = {}
+        self.hosts = {}
+        self._parse()
+ 
+    def _parse(self):
+
+        self._parse_base_groups()
+        self._parse_group_children()
+        self._parse_group_variables()
+        return self.groups
+ 
+
+    # [webservers]
+    # alpha
+    # beta:2345
+    # gamma sudo=True user=root
+    # delta asdf=jkl favcolor=red 
+
+    def _parse_base_groups(self):
+        # FIXME: refactor
+
+        ungrouped = Group(name='ungrouped')
+        all = Group(name='all')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        active_group_name = 'ungrouped'
+
+        for line in self.lines:
+            if line.startswith("["):
+                active_group_name = line.replace("[","").replace("]","").strip()
+                if line.find(":vars") != -1 or line.find(":children") != -1:
+                    active_group_name = None
+                else:
+                    new_group = self.groups[active_group_name] = Group(name=active_group_name)
+                    all.add_child_group(new_group)
+            elif line.startswith("#") or line == '':
+                pass
+            elif active_group_name:
+                tokens = line.split()
+                if len(tokens) == 0:
+                    continue
+                hostname = tokens[0]
+                port = C.DEFAULT_REMOTE_PORT
+                # Two cases to check:
+                # 0. A hostname that contains a range pesudo-code and a port
+                # 1. A hostname that contains just a port
+                if (hostname.find("[") != -1 and
+                    hostname.find("]") != -1 and
+                    hostname.find(":") != -1 and
+                    (hostname.rindex("]") < hostname.rindex(":")) or
+                    (hostname.find("]") == -1 and hostname.find(":") != -1)):
+                        tokens2  = hostname.rsplit(":", 1)
+                        hostname = tokens2[0]
+                        port     = tokens2[1]
+                
+                host = None
+                _all_hosts = []
+                if hostname in self.hosts:
+                    host = self.hosts[hostname]
+                    _all_hosts.append(host)
+                else:
+                    if detect_range(hostname):
+                        _hosts = expand_hostname_range(hostname)
+                        for _ in _hosts:
+                            host = Host(name=_, port=port)
+                            self.hosts[_] = host
+                            _all_hosts.append(host)
+                    else:
+                        host = Host(name=hostname, port=port)
+                        self.hosts[hostname] = host
+                        _all_hosts.append(host)
+                if len(tokens) > 1:
+                    for t in tokens[1:]:
+                        (k,v) = t.split("=")
+                        host.set_variable(k,v)
+                for _ in _all_hosts:
+                    self.groups[active_group_name].add_host(_)
+
+    # [southeast:children]
+    # atlanta
+    # raleigh
+
+    def _parse_group_children(self):
+        group = None
+
+        for line in self.lines:
+            line = line.strip()
+            if line is None or line == '':
+                continue
+            if line.startswith("[") and line.find(":children]") != -1:
+                line = line.replace("[","").replace(":children]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                    group = self.groups[line] = Group(name=line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif group:
+                kid_group = self.groups.get(line, None)
+                if kid_group is None:
+                    raise errors.AnsibleError("child group is not defined: (%s)" % line)
+                else:
+                    group.add_child_group(kid_group)
+
+
+    # [webservers:vars]
+    # http_port=1234
+    # maxRequestsPerChild=200
+
+    def _parse_group_variables(self):
+        group = None
+        for line in self.lines:
+            line = line.strip()
+            if line.startswith("[") and line.find(":vars]") != -1:
+                line = line.replace("[","").replace(":vars]","")
+                group = self.groups.get(line, None)
+                if group is None:
+                    raise errors.AnsibleError("can't add vars to undefined group: %s" % line)
+            elif line.startswith("#"):
+                pass
+            elif line.startswith("["):
+                group = None
+            elif line == '':
+                pass
+            elif group:
+                if line.find("=") == -1:
+                    raise errors.AnsibleError("variables assigned to group must be in key=value form")
+                else:
+                    (k,v) = line.split("=",1)
+                    group.set_variable(k,v)
+
+

lib/ansible/inventory/script.py

@@ -0,0 +1,56 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+import os
+import subprocess
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+
+class InventoryScript(object):
+    ''' Host inventory parser for ansible using external inventory scripts. '''
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        cmd = [ filename, "--list" ]
+        sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        (stdout, stderr) = sp.communicate()
+        self.data = stdout
+        self.groups = self._parse()
+
+    def _parse(self):
+
+        groups = {}
+        self.raw = utils.parse_json(self.data)
+        all=Group('all')
+        self.groups = dict(all=all)
+        group = None
+        for (group_name, hosts) in self.raw.items():
+            group = groups[group_name] = Group(group_name)
+            host = None
+            for hostname in hosts:
+                host = Host(hostname)
+                group.add_host(host)
+                # FIXME: hack shouldn't be needed
+                all.add_host(host)
+            all.add_child_group(group)
+        return groups  
+        

lib/ansible/inventory/yaml.py

@@ -0,0 +1,142 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.constants as C
+from ansible.inventory.host import Host
+from ansible.inventory.group import Group
+from ansible import errors
+from ansible import utils
+import sys
+
+class InventoryParserYaml(object):
+    ''' Host inventory parser for ansible '''
+
+    __slots__ = [ '_hosts', 'groups' ]
+
+    def __init__(self, filename=C.DEFAULT_HOST_LIST):
+
+        sys.stderr.write("WARNING: YAML inventory files are deprecated in 0.6 and will be removed in 0.7, to migrate" +
+            " download and run https://github.com/ansible/ansible/blob/devel/examples/scripts/yaml_to_ini.py\n")
+
+        fh = open(filename)
+        data = fh.read()
+        fh.close()
+        self._hosts = {}
+        self._parse(data)
+
+    def _make_host(self, hostname):
+
+        if hostname in self._hosts:
+            return self._hosts[hostname]
+        else:
+            host = Host(hostname)
+            self._hosts[hostname] = host
+            return host 
+
+    # see file 'test/yaml_hosts' for syntax
+
+    def _parse(self, data):
+        # FIXME: refactor into subfunctions
+
+        all = Group('all')
+        ungrouped = Group('ungrouped')
+        all.add_child_group(ungrouped)
+
+        self.groups = dict(all=all, ungrouped=ungrouped)
+        grouped_hosts = []
+
+        yaml = utils.parse_yaml(data)
+
+        # first add all groups
+        for item in yaml:
+            if type(item) == dict and 'group' in item:
+                group = Group(item['group'])
+
+                for subresult in item.get('hosts',[]):
+
+                    if type(subresult) in [ str, unicode ]:
+                        host = self._make_host(subresult)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+                    elif type(subresult) == dict:
+                        host = self._make_host(subresult['host'])
+                        vars = subresult.get('vars',{})
+                        if type(vars) == list:
+                            for subitem in vars:
+                                for (k,v) in subitem.items():
+                                    host.set_variable(k,v) 
+                        elif type(vars) == dict:
+                            for (k,v) in subresult.get('vars',{}).items():
+                                host.set_variable(k,v)
+                        else:
+                            raise errors.AnsibleError("unexpected type for variable")
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                vars = item.get('vars',{})
+                if type(vars) == dict:
+                    for (k,v) in item.get('vars',{}).items():
+                        group.set_variable(k,v) 
+                elif type(vars) == list:
+                    for subitem in vars:
+                        if type(subitem) != dict:
+                            raise errors.AnsibleError("expected a dictionary")
+                        for (k,v) in subitem.items():
+                            group.set_variable(k,v)
+
+                self.groups[group.name] = group
+                all.add_child_group(group)
+
+        # add host definitions
+        for item in yaml:
+            if type(item) in [ str, unicode ]:
+                host = self._make_host(item)
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+            elif type(item) == dict and 'host' in item:
+                host = self._make_host(item['host'])
+
+                vars = item.get('vars', {})
+                if type(vars)==list:
+                    varlist, vars = vars, {}
+                    for subitem in varlist:
+                        vars.update(subitem)
+                for (k,v) in vars.items():
+                    host.set_variable(k,v)
+
+                groups = item.get('groups', {})
+                if type(groups) in [ str, unicode ]:
+                    groups = [ groups ]
+                if type(groups)==list:
+                    for subitem in groups:
+                        if subitem in self.groups:
+                            group = self.groups[subitem]
+                        else:
+                            group = Group(subitem)
+                            self.groups[group.name] = group
+                            all.add_child_group(group)
+                        group.add_host(host)
+                        grouped_hosts.append(host)
+
+                if host not in grouped_hosts:
+                    ungrouped.add_host(host)
+
+        # make sure ungrouped.hosts is the complement of grouped_hosts
+        ungrouped_hosts = [host for host in ungrouped.hosts if host not in grouped_hosts]
+
+

lib/ansible/module_common.py

@@ -0,0 +1,211 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+REPLACER = "#<<INCLUDE_ANSIBLE_MODULE_COMMON>>"
+REPLACER_ARGS = "<<INCLUDE_ANSIBLE_MODULE_ARGS>>"
+
+MODULE_COMMON = """
+
+# == BEGIN DYNAMICALLY INSERTED CODE ==
+
+MODULE_ARGS = <<INCLUDE_ANSIBLE_MODULE_ARGS>>
+
+BOOLEANS_TRUE = ['yes', 'on', '1', 'true', 1]
+BOOLEANS_FALSE = ['no', 'off', '0', 'false', 0]
+BOOLEANS = BOOLEANS_TRUE + BOOLEANS_FALSE
+
+# ansible modules can be written in any language.  To simplify
+# development of Python modules, the functions available here
+# can be inserted in any module source automatically by including
+# #<<INCLUDE_ANSIBLE_MODULE_COMMON>> on a blank line by itself inside
+# of an ansible module. The source of this common code lives
+# in lib/ansible/module_common.py
+
+try:
+    import json
+except ImportError:
+    import simplejson as json
+import base64
+import os
+import re
+import shlex
+import subprocess
+import sys
+import syslog
+import types
+
+try:
+    from hashlib import md5 as _md5
+except ImportError:
+    from md5 import md5 as _md5
+
+class AnsibleModule(object):
+
+    def __init__(self, argument_spec, bypass_checks=False, no_log=False, check_invalid_arguments=True):
+        ''' 
+        common code for quickly building an ansible module in Python
+        (although you can write modules in anything that can return JSON)
+        see library/slurp and others for examples
+        '''
+
+        self.argument_spec = argument_spec
+        (self.params, self.args) = self._load_params()
+
+        self._legal_inputs = []
+        self._handle_aliases()
+    
+        # this may be disabled where modules are going to daisy chain into others
+        if check_invalid_arguments:
+            self._check_invalid_arguments()
+
+        self._set_defaults(pre=True)
+
+        if not bypass_checks:
+            self._check_required_arguments()
+            self._check_argument_types()
+
+        self._set_defaults(pre=False)
+        if not no_log:
+            self._log_invocation()
+
+
+    def _handle_aliases(self):
+        for (k,v) in self.argument_spec.iteritems():
+            self._legal_inputs.append(k)
+            aliases = v.get('aliases', None)
+            default = v.get('default', None)
+            required = v.get('required', False)
+            if default is not None and required:
+                # not alias specific but this is a good place to check this
+                self.fail_json(msg="internal error: required and default are mutally exclusive for %s" % k)
+            if aliases is None:
+                continue
+            if type(aliases) != list:
+                self.fail_json(msg='internal error: aliases must be a list')
+            for alias in aliases:
+                self._legal_inputs.append(alias)
+                if alias in self.params:
+                    self.params[k] = self.params[alias]
+
+    def _check_invalid_arguments(self):
+        for (k,v) in self.params.iteritems():
+            if k not in self._legal_inputs:
+                self.fail_json(msg="unsupported parameter for module: %s" % k)
+
+    def _check_required_arguments(self):
+        ''' ensure all required arguments are present '''
+        missing = []
+        for (k,v) in self.argument_spec.iteritems():
+            required = v.get('required', False)
+            if required and k not in self.params:
+                missing.append(k)
+        if len(missing) > 0:
+            self.fail_json(msg="missing required arguments: %s" % ",".join(missing))
+
+    def _check_argument_types(self):
+        ''' ensure all arguments have the requested values, and there are no stray arguments '''
+        for (k,v) in self.argument_spec.iteritems():
+            choices = v.get('choices',None)
+            if choices is None:
+                continue
+            if type(choices) == list:
+                if k in self.params:
+                    if self.params[k] not in choices:
+                        choices_str=",".join(choices)
+                        msg="value of %s must be one of: %s, got: %s" % (k, choices_str, self.params[k])
+                        self.fail_json(msg=msg)
+            else:
+                self.fail_json(msg="internal error: do not know how to interpret argument_spec")
+
+    def _set_defaults(self, pre=True):
+         for (k,v) in self.argument_spec.iteritems():
+             default = v.get('default', None)
+             if pre == True:
+                 # this prevents setting defaults on required items
+                 if default and k not in self.params:
+                     self.params[k] = default
+             else:
+                 # make sure things without a default still get set None
+                 if k not in self.params:
+                     self.params[k] = default
+
+    def _load_params(self):
+        ''' read the input and return a dictionary and the arguments string '''
+        args = MODULE_ARGS
+        items   = shlex.split(args)
+        params = {}
+        for x in items:
+            (k, v) = x.split("=",1)
+            params[k] = v
+        return (params, args)        
+
+    def _log_invocation(self):
+        ''' log that ansible ran the module '''
+        syslog.openlog('ansible-%s' % os.path.basename(__file__))
+        # Sanitize possible password argument when logging
+        log_args = re.sub(r'password=.+ (.*)', r"password=NOT_LOGGING_PASSWORD \1", self.args)
+        log_args = re.sub(r'login_password=.+ (.*)', r"login_password=NOT_LOGGING_PASSWORD \1", log_args)
+        syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % log_args)
+
+    def boolean(self, arg):
+        ''' return a bool for the arg '''
+        if arg is None or type(arg) == bool:
+            return arg
+        if type(arg) in types.StringTypes:
+            arg = arg.lower()
+        if arg in BOOLEANS_TRUE:
+            return True
+        elif arg in BOOLEANS_FALSE:
+            return False
+        else:
+            self.fail_json(msg='Boolean %s not in either boolean list' % arg)
+            
+    def jsonify(self, data):
+        return json.dumps(data)
+
+    def exit_json(self, **kwargs):
+        ''' return from the module, without error '''
+        print self.jsonify(kwargs)
+        sys.exit(0)
+
+    def fail_json(self, **kwargs):
+        ''' return from the module, with an error message '''
+        assert 'msg' in kwargs, "implementation error -- msg to explain the error is required"
+        kwargs['failed'] = True
+        print self.jsonify(kwargs)
+        sys.exit(1)
+
+    def md5(self, filename):
+        ''' Return MD5 hex digest of local file, or None if file is not present. '''
+        if not os.path.exists(filename):
+            return None
+        if os.path.isdir(filename):
+            self.fail_json(msg="attempted to take md5sum of directory: %s" % filename)
+        digest = _md5()
+        blocksize = 64 * 1024
+        infile = open(filename, 'rb')
+        block = infile.read(blocksize)
+        while block:
+            digest.update(block)
+            block = infile.read(blocksize)
+        infile.close()
+        return digest.hexdigest()
+
+
+# == END DYNAMICALLY INSERTED CODE ===
+
+"""

lib/ansible/playbook.py

@@ -1,368 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-import ansible.runner
-import ansible.constants as C
-from ansible.utils import *
-import yaml
-import shlex
-import os
-import jinja2
-
-# used to transfer variables to Runner
-SETUP_CACHE={ }
-
-#############################################
-
-class PlayBook(object):
-
-    '''
-    runs an ansible playbook, given as a datastructure
-    or YAML filename.  a playbook is a deployment, config
-    management, or automation based set of commands to
-    run in series.
-
-    multiple patterns do not execute simultaneously,
-    but tasks in each pattern do execute in parallel
-    according to the number of forks requested.
-    '''
-
-    def __init__(self,
-        playbook     =None,
-        host_list    =C.DEFAULT_HOST_LIST,
-        module_path  =C.DEFAULT_MODULE_PATH,
-        forks        =C.DEFAULT_FORKS,
-        timeout      =C.DEFAULT_TIMEOUT,
-        remote_user  =C.DEFAULT_REMOTE_USER,
-        remote_pass  =C.DEFAULT_REMOTE_PASS,
-        verbose=False,
-        callbacks=None):
-
-        # TODO, once ansible-playbook is it's own script this will
-        # have much LESS parameters to the constructor and will
-        # read most everything per pattern from the playbook
-        # and this will be greatly simplified
-
-        self.host_list   = host_list
-        self.module_path = module_path
-        self.forks       = forks
-        self.timeout     = timeout
-        self.remote_user = remote_user
-        self.remote_pass = remote_pass
-        self.verbose     = verbose
-        self.callbacks   = callbacks
-        self.callbacks.set_playbook(self)
-
-        # store the list of changes/invocations/failure counts
-        # as a dictionary of integers keyed off the hostname
-
-        self.processed    = {}
-        self.dark         = {}
-        self.changed      = {}
-        self.invocations  = {}
-        self.failures     = {}
-
-        # playbook file can be passed in as a path or
-        # as file contents (to support API usage)
-
-        self.basedir = os.path.dirname(playbook)
-        self.playbook = self._parse_playbook(playbook)
-
-    def _include_tasks(self, play, task, dirname, new_tasks):
-        # an include line looks like:
-        # include: some.yml a=2 b=3 c=4
-        include_tokens = task['include'].split()
-        path = path_dwim(dirname, include_tokens[0])
-        inject_vars = play.get('vars', {})
-        for i,x in enumerate(include_tokens):
-            if x.find("=") != -1:
-                (k,v) = x.split("=")
-                inject_vars[k] = v
-            included = file(path).read()
-            template = jinja2.Template(included)
-            included = template.render(inject_vars)
-            included = yaml.load(included)
-            for x in included:
-                new_tasks.append(x)
-
-    def _include_handlers(self, play, handler, dirname, new_handlers):
-        path = path_dwim(dirname, handler['include'])
-        included = file(path).read()
-        inject_vars = play.get('vars', {})
-        template = jinja2.Template(included)
-        included = template.render(inject_vars)
-        included = yaml.load(included)
-        for x in included:
-            new_handlers.append(x)
-
-    def _parse_playbook(self, playbook):
-        ''' load YAML file, including handling for imported files '''
-        
-        dirname  = os.path.dirname(playbook)
-        playbook = yaml.load(file(playbook).read())
-
-        for play in playbook:
-            tasks = play.get('tasks',[])
-            handlers = play.get('handlers', [])
-
-            # process tasks in this file as well as imported tasks
-            new_tasks = []
-            for task in tasks:
-               if 'include' in task:
-                   self._include_tasks(play, task, dirname, new_tasks)
-               else:
-                   new_tasks.append(task)
-            play['tasks'] = new_tasks
-
-            # process handlers as well as imported handlers
-            new_handlers = [] 
-            for handler in handlers:
-               if 'include' in handler:
-                   self._include_handlers(play, handler, dirname, new_handlers)
-               else:
-                   new_handlers.append(handler)
-            play['handlers'] = new_handlers
-
-        return playbook
-        
-    def run(self):
-        ''' run all patterns in the playbook '''
-
-        # loop through all patterns and run them
-        self.callbacks.on_start()
-        for pattern in self.playbook:
-            self._run_pattern(pattern)
-
-        # summarize the results
-        results = {}
-        for host in self.processed.keys():
-            results[host]  = {
-                'resources'   : self.invocations.get(host, 0),
-                'changed'     : self.changed.get(host, 0),
-                'dark'        : self.dark.get(host, 0),
-                'failed'      : self.failures.get(host, 0)
-            }
-        return results
-
-    def _prune_failed_hosts(self, host_list):
-        new_hosts = []
-        for x in host_list:
-            if not x in self.failures and not x in self.dark:
-                new_hosts.append(x)
-        return new_hosts
-
-    def _run_module(self, pattern, module, args, hosts, remote_user):
-        ''' run a particular module step in a playbook '''
-        return ansible.runner.Runner(
-            pattern=pattern,
-            module_name=module,
-            module_args=args,
-            host_list=hosts,
-            forks=self.forks,
-            remote_pass=self.remote_pass,
-            module_path=self.module_path,
-            timeout=self.timeout,
-            remote_user=remote_user,
-            setup_cache=SETUP_CACHE,
-            basedir=self.basedir
-        ).run()
-
-    def _run_task(self, pattern=None, task=None, host_list=None,
-        remote_user=None, handlers=None, conditional=False):
-        '''
-        run a single task in the playbook and
-        recursively run any subtasks.
-        '''
-
-        if host_list is None:
-            # pruned host lists occur when running triggered
-            # actions where not all hosts have changed
-            # though top-level tasks will pass in "None" here
-            host_list = self.host_list
-            (host_list, groups) = ansible.runner.Runner.parse_hosts(host_list)
-
-        # do not continue to run tasks on hosts that have had failures
-        host_list = self._prune_failed_hosts(host_list)
-
-        # load the module name and parameters from the task entry
-        name    = task['name']
-        action  = task['action']
-        # comment = task.get('comment', '')
-
-        tokens = shlex.split(action)
-        module_name = tokens[0]
-        module_args = tokens[1:]
-
-        # tasks can be direct (run on all nodes matching
-        # the pattern) or conditional, where they ran
-        # as the result of a change handler on a subset
-        # of all of the hosts
-
-        self.callbacks.on_task_start(name, conditional)
-
-        # load up an appropriate ansible runner to
-        # run the task in parallel
-        results = self._run_module(pattern, module_name, 
-            module_args, host_list, remote_user)
-
-        # if no hosts are matched, carry on, unlike /bin/ansible
-        # which would warn you about this
-        if results is None:
-            results = {}
- 
-        # walk through the results and build up
-        # summary information about successes and
-        # failures.  TODO: split into subfunction
-
-        dark      = results.get("dark", {})
-        contacted = results.get("contacted", {})
-
-        for host, msg in dark.items():
-            self.processed[host] = 1
-            self.callbacks.on_unreachable(host, msg)
-            if not host in self.dark:
-                self.dark[host] = 1
-            else:
-                self.dark[host] = self.dark[host] + 1
-
-        for host, results in contacted.items():
-            self.processed[host] = 1
-   
-            if is_failed(results):
-                self.callbacks.on_failed(host, results)
-                if not host in self.failures:
-                    self.failures[host] = 1
-                else:
-                    self.failures[host] = self.failures[host] + 1
-            else:
-                self.callbacks.on_ok(host)
-                if not host in self.invocations:
-                    self.invocations[host] = 1
-                else:
-                    self.invocations[host] = self.invocations[host] + 1
-                if results.get('changed', False):
-                    if not host in self.changed:
-                        self.changed[host] = 1
-                    else:
-                        self.changed[host] = self.changed[host] + 1
-
-        # flag which notify handlers need to be run
-        # this will be on a SUBSET of the actual host list.  For instance
-        # a file might need to be written on only half of the nodes so
-        # we would only trigger restarting Apache on half of the nodes
-
-        subtasks = task.get('notify', [])
-        if len(subtasks) > 0:
-            for host, results in contacted.items():
-                if results.get('changed', False):
-                    for subtask in subtasks:
-                        self._flag_handler(handlers, subtask, host)
-
-    def _flag_handler(self, handlers, match_name, host):
-        ''' 
-        if a task has any notify elements, flag handlers for run
-        at end of execution cycle for hosts that have indicated
-        changes have been made
-        '''
-
-        # for all registered handlers in the ansible playbook
-        # for this particular pattern group
-
-        for x in handlers:
-            name = x['name']
-            if match_name == name:
-                # flag the handler with the list of hosts
-                # it needs to be run on, it will be run later
-                if not 'run' in x:
-                    x['run'] = []
-                x['run'].append(host)
-
-    def _run_pattern(self, pg):
-        '''
-        run a list of tasks for a given pattern, in order
-        '''
-
-        # get configuration information about the pattern
-        pattern  = pg['hosts']
-        vars     = pg.get('vars', {})
-        tasks    = pg['tasks']
-        handlers = pg['handlers']
-        user     = pg.get('user', C.DEFAULT_REMOTE_USER)
-
-        self.host_list, groups = ansible.runner.Runner.parse_hosts(self.host_list)
-
-        self.callbacks.on_play_start(pattern)
-
-        # first run the setup task on every node, which gets the variables
-        # written to the JSON file and will also bubble facts back up via
-        # magic in Runner()
-        push_var_str=''
-        for (k,v) in vars.items():
-            push_var_str += "%s=%s " % (k,v)
-
-        # push any variables down to the system
-        setup_results = ansible.runner.Runner(
-            pattern=pattern,
-            module_name='setup',
-            module_args=push_var_str,
-            host_list=self.host_list,
-            forks=self.forks,
-            module_path=self.module_path,
-            timeout=self.timeout,
-            remote_user=user,
-            setup_cache=SETUP_CACHE
-        ).run()
-
-        # now for each result, load into the setup cache so we can
-        # let runner template out future commands
-        setup_ok = setup_results.get('contacted', {})
-        for (host, result) in setup_ok.items():
-            SETUP_CACHE[host] = result
-
-        # run all the top level tasks, these get run on every node
-        for task in tasks:
-            self._run_task(
-                pattern=pattern, 
-                task=task, 
-                handlers=handlers,
-                remote_user=user
-            )
-
-        # handlers only run on certain nodes, they are flagged by _flag_handlers
-        # above.  They only run on nodes when things mark them as changed, and
-        # handlers only get run once.  For instance, the system is designed
-        # such that multiple config files if changed can ask for an Apache restart
-        # but Apache will only be restarted once (at the end).
-
-        for task in handlers:
-            if type(task.get("run", None)) == list:
-                self._run_task(
-                   pattern=pattern, 
-                   task=task,
-                   handlers=handlers,
-                   host_list=task.get('run',[]),
-                   conditional=True,
-                   remote_user=user
-                )
-
-        # end of execution for this particular pattern.  Multiple patterns
-        # can be in a single playbook file
-
- 
-

lib/ansible/playbook/__init__.py

@@ -0,0 +1,324 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ansible.inventory
+import ansible.runner
+import ansible.constants as C
+from ansible import utils
+from ansible import errors
+import os
+import collections
+from play import Play
+
+SETUP_CACHE = collections.defaultdict(dict)
+
+class PlayBook(object):
+    '''
+    runs an ansible playbook, given as a datastructure or YAML filename.  
+    A playbook is a deployment, config management, or automation based 
+    set of commands to run in series.
+
+    multiple plays/tasks do not execute simultaneously, but tasks in each 
+    pattern do execute in parallel (according to the number of forks 
+    requested) among the hosts they address
+    '''
+
+    # *****************************************************
+
+    def __init__(self,
+        playbook         = None,
+        host_list        = C.DEFAULT_HOST_LIST,
+        module_path      = C.DEFAULT_MODULE_PATH,
+        forks            = C.DEFAULT_FORKS,
+        timeout          = C.DEFAULT_TIMEOUT,
+        remote_user      = C.DEFAULT_REMOTE_USER,
+        remote_pass      = C.DEFAULT_REMOTE_PASS,
+        sudo_pass        = C.DEFAULT_SUDO_PASS,
+        remote_port      = C.DEFAULT_REMOTE_PORT,
+        transport        = C.DEFAULT_TRANSPORT,
+        private_key_file = C.DEFAULT_PRIVATE_KEY_FILE,
+        verbose          = False,
+        callbacks        = None,
+        runner_callbacks = None,
+        stats            = None,
+        sudo             = False,
+        sudo_user        = C.DEFAULT_SUDO_USER,
+        extra_vars       = None,
+        only_tags        = None):
+
+        """
+        playbook:         path to a playbook file
+        host_list:        path to a file like /etc/ansible/hosts
+        module_path:      path to ansible modules, like /usr/share/ansible/
+        forks:            desired level of paralellism
+        timeout:          connection timeout
+        remote_user:      run as this user if not specified in a particular play
+        remote_pass:      use this remote password (for all plays) vs using SSH keys
+        sudo_pass:        if sudo==True, and a password is required, this is the sudo password
+        remote_port:      default remote port to use if not specified with the host or play
+        transport:        how to connect to hosts that don't specify a transport (local, paramiko, etc)
+        callbacks         output callbacks for the playbook
+        runner_callbacks: more callbacks, this time for the runner API
+        stats:            holds aggregrate data about events occuring to each host
+        sudo:             if not specified per play, requests all plays use sudo mode
+        """
+
+        self.SETUP_CACHE = SETUP_CACHE
+
+        if playbook is None or callbacks is None or runner_callbacks is None or stats is None:
+            raise Exception('missing required arguments')
+
+        if extra_vars is None:
+            extra_vars = {}
+        if only_tags is None:
+            only_tags = [ 'all' ]
+ 
+        self.module_path      = module_path
+        self.forks            = forks
+        self.timeout          = timeout
+        self.remote_user      = remote_user
+        self.remote_pass      = remote_pass
+        self.remote_port      = remote_port
+        self.transport        = transport
+        self.verbose          = verbose
+        self.callbacks        = callbacks
+        self.runner_callbacks = runner_callbacks
+        self.stats            = stats
+        self.sudo             = sudo
+        self.sudo_pass        = sudo_pass
+        self.sudo_user        = sudo_user
+        self.extra_vars       = extra_vars
+        self.global_vars      = {}
+        self.private_key_file = private_key_file
+        self.only_tags        = only_tags
+
+        self.inventory   = ansible.inventory.Inventory(host_list)
+        
+        if not self.inventory._is_script:
+            self.global_vars.update(self.inventory.get_group_variables('all'))
+
+        self.basedir     = os.path.dirname(playbook)
+        self.playbook    = self._load_playbook_from_file(playbook)
+        self.module_path = self.module_path + os.pathsep + os.path.join(self.basedir, "library")
+
+    # *****************************************************
+
+    def _load_playbook_from_file(self, path):
+        '''
+        run top level error checking on playbooks and allow them to include other playbooks.
+        '''
+
+        playbook_data  = utils.parse_yaml_from_file(path)
+        accumulated_plays = []
+
+        if type(playbook_data) != list:
+            raise errors.AnsibleError("parse error: playbooks must be formatted as a YAML list")
+
+        for play in playbook_data:
+            if type(play) != dict:
+                raise errors.AnsibleError("parse error: each play in a playbook must a YAML dictionary (hash), recieved: %s" % play)
+            if 'include' in play:
+                if len(play.keys()) == 1:
+                    included_path = utils.path_dwim(self.basedir, play['include'])
+                    accumulated_plays.extend(self._load_playbook_from_file(included_path))
+                else:
+                    raise errors.AnsibleError("parse error: top level includes cannot be used with other directives: %s" % play)
+            else:
+                accumulated_plays.append(play)
+
+        return accumulated_plays
+
+    # *****************************************************
+        
+    def run(self):
+        ''' run all patterns in the playbook '''
+
+        # loop through all patterns and run them
+        self.callbacks.on_start()
+        for play_ds in self.playbook:
+            self._run_play(Play(self,play_ds))
+
+        # summarize the results
+        results = {}
+        for host in self.stats.processed.keys():
+            results[host] = self.stats.summarize(host)
+        return results
+
+    # *****************************************************
+
+    def _async_poll(self, poller, async_seconds, async_poll_interval):
+        ''' launch an async job, if poll_interval is set, wait for completion '''
+
+        results = poller.wait(async_seconds, async_poll_interval)
+
+        # mark any hosts that are still listed as started as failed
+        # since these likely got killed by async_wrapper
+        for host in poller.hosts_to_poll:
+            reason = { 'failed' : 1, 'rc' : None, 'msg' : 'timed out' }
+            self.runner_callbacks.on_failed(host, reason)
+            results['contacted'][host] = reason
+
+        return results
+
+    # *****************************************************
+
+    def _run_task_internal(self, task):
+        ''' run a particular module step in a playbook '''
+
+        hosts = [ h for h in self.inventory.list_hosts() if (h not in self.stats.failures) and (h not in self.stats.dark)]
+        self.inventory.restrict_to(hosts)
+
+        runner = ansible.runner.Runner(
+            pattern=task.play.hosts, inventory=self.inventory, module_name=task.module_name,
+            module_args=task.module_args, forks=self.forks,
+            remote_pass=self.remote_pass, module_path=self.module_path,
+            timeout=self.timeout, remote_user=task.play.remote_user, 
+            remote_port=task.play.remote_port, module_vars=task.module_vars,
+            private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, basedir=self.basedir,
+            conditional=task.only_if, callbacks=self.runner_callbacks, 
+            verbose=self.verbose, sudo=task.play.sudo, sudo_user=task.play.sudo_user,
+            transport=task.play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        )
+
+        if task.async_seconds == 0:
+            results = runner.run()
+        else:
+            results, poller = runner.run_async(task.async_seconds)
+            self.stats.compute(results)
+            if task.async_poll_interval > 0:
+                # if not polling, playbook requested fire and forget, so don't poll
+                results = self._async_poll(poller, task.async_seconds, task.async_poll_interval)
+
+        self.inventory.lift_restriction()
+        return results
+
+    # *****************************************************
+
+    def _run_task(self, play, task, is_handler):
+        ''' run a single task in the playbook and recursively run any subtasks.  '''
+
+        self.callbacks.on_task_start(task.name, is_handler)
+
+        # load up an appropriate ansible runner to run the task in parallel
+        results = self._run_task_internal(task)
+        # if no hosts are matched, carry on
+        if results is None:
+            results = {}
+
+        self.stats.compute(results)
+
+        # add facts to the global setup cache
+        for host, result in results['contacted'].iteritems():
+            facts = result.get('ansible_facts', {})
+            self.SETUP_CACHE[host].update(facts)
+ 
+        # flag which notify handlers need to be run
+        if len(task.notify) > 0:
+            for host, results in results.get('contacted',{}).iteritems():
+                if results.get('changed', False):
+                    for handler_name in task.notify:
+                        self._flag_handler(play.handlers(), utils.template(handler_name, task.module_vars), host)
+
+    # *****************************************************
+
+    def _flag_handler(self, handlers, handler_name, host):
+        ''' 
+        if a task has any notify elements, flag handlers for run
+        at end of execution cycle for hosts that have indicated
+        changes have been made
+        '''
+
+        found = False
+        for x in handlers:
+            if handler_name == x.name:
+                found = True
+                self.callbacks.on_notify(host, x.name)
+                x.notified_by.append(host)
+        if not found:
+            raise errors.AnsibleError("change handler (%s) is not defined" % handler_name)
+
+    # *****************************************************
+
+    def _do_setup_step(self, play):
+        ''' get facts from the remote system '''
+        
+        host_list = [ h for h in self.inventory.list_hosts(play.hosts) 
+            if not (h in self.stats.failures or h in self.stats.dark) ]
+
+        if not play.gather_facts:
+            return {}
+
+        self.callbacks.on_setup()
+        self.inventory.restrict_to(host_list)
+
+        # push any variables down to the system
+        setup_results = ansible.runner.Runner(
+            pattern=play.hosts, module_name='setup', module_args={}, inventory=self.inventory,
+            forks=self.forks, module_path=self.module_path, timeout=self.timeout, remote_user=play.remote_user,
+            remote_pass=self.remote_pass, remote_port=play.remote_port, private_key_file=self.private_key_file,
+            setup_cache=self.SETUP_CACHE, callbacks=self.runner_callbacks, sudo=play.sudo, sudo_user=play.sudo_user, 
+            verbose=self.verbose, transport=play.transport, sudo_pass=self.sudo_pass, is_playbook=True
+        ).run()
+        self.stats.compute(setup_results, setup=True)
+
+        self.inventory.lift_restriction()
+
+        # now for each result, load into the setup cache so we can
+        # let runner template out future commands
+        setup_ok = setup_results.get('contacted', {})
+        for (host, result) in setup_ok.iteritems():
+            facts = result.get('ansible_facts', {})
+            self.SETUP_CACHE[host] = result.get('ansible_facts', {})
+        return setup_results
+
+    # *****************************************************
+
+    def _run_play(self, play):
+        ''' run a list of tasks for a given pattern, in order '''
+
+        if not play.should_run(self.only_tags):
+            return
+
+        self.callbacks.on_play_start(play.name)
+
+        # get facts from system
+        rc = self._do_setup_step(play) 
+
+        # now with that data, handle contentional variable file imports!
+        if play.vars_files and len(play.vars_files) > 0:
+            play.update_vars_files(self.inventory.list_hosts(play.hosts))
+
+        for task in play.tasks():
+            
+            # only run the task if the requested tags match
+            should_run = False
+            for x in self.only_tags:
+                for y in task.tags:
+                    if (x==y):
+                        should_run = True
+                        break
+            if should_run:
+                self._run_task(play, task, False)
+
+        # run notify actions
+        for handler in play.handlers():
+            if len(handler.notified_by) > 0:
+                self.inventory.restrict_to(handler.notified_by)
+                self._run_task(play, handler, True)
+                self.inventory.lift_restriction()
+

lib/ansible/playbook/play.py

@@ -0,0 +1,295 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+#############################################
+
+from ansible import utils
+from ansible import errors
+from ansible.playbook.task import Task
+import shlex
+import os
+
+class Play(object):
+
+    __slots__ = [ 
+       'hosts', 'name', 'vars', 'vars_prompt', 'vars_files', 
+       'handlers', 'remote_user', 'remote_port',
+       'sudo', 'sudo_user', 'transport', 'playbook', 
+       'tags', 'gather_facts', '_ds', '_handlers', '_tasks'
+    ]
+
+    # to catch typos and so forth -- these are userland names
+    # and don't line up 1:1 with how they are stored
+    VALID_KEYS = [
+       'hosts', 'name', 'vars', 'vars_prompt', 'vars_files',
+       'tasks', 'handlers', 'user', 'port', 'include',
+       'sudo', 'sudo_user', 'connection', 'tags', 'gather_facts'
+    ]
+
+    # *************************************************
+
+    def __init__(self, playbook, ds):
+        ''' constructor loads from a play datastructure '''
+
+        for x in ds.keys():
+             if not x in Play.VALID_KEYS:
+                 raise errors.AnsibleError("%s is not a legal parameter in an Ansible Playbook" % x)
+
+        # TODO: more error handling
+
+        hosts = ds.get('hosts')
+        if hosts is None:
+            raise errors.AnsibleError('hosts declaration is required')
+        elif isinstance(hosts, list):
+            hosts = ';'.join(hosts)
+        hosts = utils.template(hosts, playbook.extra_vars)
+
+        self._ds          = ds
+        self.playbook     = playbook
+        self.hosts        = hosts 
+        self.name         = ds.get('name', self.hosts)
+        self.vars         = ds.get('vars', {})
+        self.vars_files   = ds.get('vars_files', [])
+        self.vars_prompt  = ds.get('vars_prompt', {})
+        self.vars         = self._get_vars(self.playbook.basedir)
+        self._tasks       = ds.get('tasks', [])
+        self._handlers    = ds.get('handlers', [])
+        self.remote_user  = ds.get('user', self.playbook.remote_user)
+        self.remote_port  = ds.get('port', self.playbook.remote_port)
+        self.sudo         = ds.get('sudo', self.playbook.sudo)
+        self.sudo_user    = ds.get('sudo_user', self.playbook.sudo_user)
+        self.transport    = ds.get('connection', self.playbook.transport)
+        self.tags         = ds.get('tags', None)
+        self.gather_facts = ds.get('gather_facts', True)
+
+        self._update_vars_files_for_host(None)
+
+        self._tasks      = self._load_tasks(self._ds, 'tasks')
+        self._handlers   = self._load_tasks(self._ds, 'handlers')
+
+        if self.tags is None:
+            self.tags = []
+        elif type(self.tags) in [ str, unicode ]:
+            self.tags = [ self.tags ]
+        elif type(self.tags) != list:
+            self.tags = []
+
+        if self.sudo_user != 'root':
+            self.sudo = True
+
+    # *************************************************
+
+    def _load_tasks(self, ds, keyname):
+        ''' handle task and handler include statements '''
+
+        tasks = ds.get(keyname, [])
+        results = []
+        for x in tasks:
+            task_vars = self.vars.copy()
+            if 'include' in x:
+                tokens = shlex.split(x['include'])
+
+                for t in tokens[1:]:
+                    (k,v) = t.split("=", 1)
+                    task_vars[k]=v
+                include_file = utils.template(tokens[0], task_vars)
+                data = utils.parse_yaml_from_file(utils.path_dwim(self.playbook.basedir, include_file))
+            elif type(x) == dict:
+                data = [x]
+            else:
+                raise Exception("unexpected task type")
+            for y in data:
+                mv = task_vars.copy()
+                results.append(Task(self,y,module_vars=mv))
+
+        for x in results:
+            if self.tags is not None:
+                x.tags.extend(self.tags)
+
+        return results
+
+    # *************************************************
+
+    def tasks(self):
+        ''' return task objects for this play '''
+        return self._tasks      
+
+    def handlers(self):
+        ''' return handler objects for this play '''
+        return self._handlers
+
+    # *************************************************
+
+    def _get_vars(self, dirname):
+        ''' load the vars section from a play, accounting for all sorts of variable features
+        including loading from yaml files, prompting, and conditional includes of the first
+        file found in a list. '''
+
+        if self.vars is None:
+            self.vars = {}
+
+        if type(self.vars) not in [dict, list]:
+            raise errors.AnsibleError("'vars' section must contain only key/value pairs")
+
+        vars = self.playbook.global_vars
+    
+        # translate a list of vars into a dict
+        if type(self.vars) == list:
+            for item in self.vars:
+                k, v = item.items()[0]
+                vars[k] = v
+        else:
+            vars.update(self.vars)
+
+        if type(self.vars_prompt) == list:
+            for var in self.vars_prompt:
+                if not 'name' in var:
+                    raise errors.AnsibleError("'vars_prompt' item is missing 'name:'")
+                vname = var['name']
+                prompt = "%s: " % var.get("prompt", vname)
+                private = var.get("private", True)
+                vars[vname] = self.playbook.callbacks.on_vars_prompt(vname, private, prompt)
+        elif type(self.vars_prompt) == dict:
+            for (vname, prompt) in self.vars_prompt.iteritems():
+                prompt = "%s: " % prompt
+                vars[vname] = self.playbook.callbacks.on_vars_prompt(vname, False, prompt)
+        else:
+            raise errors.AnsibleError("'vars_prompt' section is malformed, see docs")
+
+        results = self.playbook.extra_vars.copy()
+        results.update(vars)
+        return results
+
+    # *************************************************
+
+    def update_vars_files(self, hosts):
+        ''' calculate vars_files, which requires that setup runs first so ansible facts can be mixed in '''
+         
+        # now loop through all the hosts...
+        for h in hosts:
+            self._update_vars_files_for_host(h)
+
+    # *************************************************
+
+    def should_run(self, tags):
+        ''' does the play match any of the tags? '''
+
+        tags_counted = 0
+        for task in self._tasks:
+            for task_tag in task.tags:
+                tags_counted = tags_counted + 1
+                if task_tag in tags:
+                    return True
+
+        if tags_counted > 0:
+            return False  
+   
+        # didn't tag the play, and the play contains no steps
+        # so assume we just want to gather facts
+        return True                  
+
+    # *************************************************
+
+    def _has_vars_in(self, msg):
+        return ((msg.find("$") != -1) or (msg.find("{{") != -1))
+
+    # *************************************************
+
+    def _update_vars_files_for_host(self, host):
+
+        if type(self.vars_files) != list:
+            self.vars_files = [ self.vars_files ]
+ 
+        if (host is not None):
+            inventory = self.playbook.inventory
+            hostrec = inventory.get_host(host)
+            groups = [ g.name for g in hostrec.groups ]
+            basedir = inventory.basedir()
+            if basedir is not None:
+                for x in groups:
+                    path = os.path.join(basedir, "group_vars/%s" % x)
+                    if os.path.exists(path):
+                        data = utils.parse_yaml_from_file(path)
+                        if type(data) != dict:
+                            raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
+                        self.playbook.SETUP_CACHE[host].update(data)
+                path = os.path.join(basedir, "host_vars/%s" % hostrec.name)
+                if os.path.exists(path):
+                    data = utils.parse_yaml_from_file(path)
+                    if type(data) != dict:
+                        raise errors.AnsibleError("%s must be stored as a dictionary/hash" % path)
+                    self.playbook.SETUP_CACHE[host].update(data)
+
+        for filename in self.vars_files:
+
+            if type(filename) == list:
+
+                # loop over all filenames, loading the first one, and failing if # none found
+                found = False
+                sequence = []
+                for real_filename in filename:
+                    filename2 = utils.template(real_filename, self.vars)
+                    filename3 = filename2
+                    if host is not None:
+                        filename3 = utils.template(filename2, self.playbook.SETUP_CACHE[host])
+                    filename4 = utils.path_dwim(self.playbook.basedir, filename3)
+                    sequence.append(filename4)
+                    if os.path.exists(filename4):
+                        found = True
+                        data = utils.parse_yaml_from_file(filename4)
+                        if type(data) != dict:
+                            raise errors.AnsibleError("%s must be stored as a dictionary/hash" % filename4)
+                        if host is not None:
+                            if self._has_vars_in(filename2) and not self._has_vars_in(filename3):
+                                # this filename has variables in it that were fact specific
+                                # so it needs to be loaded into the per host SETUP_CACHE
+                                self.playbook.SETUP_CACHE[host].update(data)
+                                self.playbook.callbacks.on_import_for_host(host, filename4)
+                        elif not self._has_vars_in(filename4):
+                            # found a non-host specific variable, load into vars and NOT
+                            # the setup cache
+                            self.vars.update(data)
+                    elif host is not None:
+                        self.playbook.callbacks.on_not_import_for_host(host, filename4)
+                    if found:
+                        break
+                if not found:
+                    raise errors.AnsibleError(
+                        "%s: FATAL, no files matched for vars_files import sequence: %s" % (host, sequence)
+                    )
+
+            else:
+                # just one filename supplied, load it!
+
+                filename2 = utils.template(filename, self.vars)
+                filename3 = filename2
+                if host is not None:
+                    filename3 = utils.template(filename2, self.playbook.SETUP_CACHE[host])
+                filename4 = utils.path_dwim(self.playbook.basedir, filename3)
+                if self._has_vars_in(filename4):
+                    return
+                new_vars = utils.parse_yaml_from_file(filename4)
+                if new_vars:
+                    if type(new_vars) != dict:
+                        raise errors.AnsibleError("%s must be stored as dictonary/hash: %s" % filename4)
+                    if host is not None and self._has_vars_in(filename2) and not self._has_vars_in(filename3):
+                        # running a host specific pass and has host specific variables
+                        # load into setup cache 
+                        self.playbook.SETUP_CACHE[host].update(new_vars)
+                    elif host is None:
+                        # running a non-host specific pass and we can update the global vars instead    
+                        self.vars.update(new_vars)

lib/ansible/playbook/task.py

@@ -0,0 +1,112 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+from ansible import errors
+from ansible import utils
+
+class Task(object):
+
+    __slots__ = [ 
+        'name', 'action', 'only_if', 'async_seconds', 'async_poll_interval',
+        'notify', 'module_name', 'module_args', 'module_vars', 
+        'play', 'notified_by', 'tags', 'with_items', 'first_available_file', 'ignore_errors'
+    ]
+
+    # to prevent typos and such
+    VALID_KEYS = [
+         'name', 'action', 'only_if', 'async', 'poll', 'notify', 'with_items', 'first_available_file',
+         'include', 'tags', 'ignore_errors'
+    ]
+
+    def __init__(self, play, ds, module_vars=None):
+        ''' constructor loads from a task or handler datastructure '''
+
+        for x in ds.keys():
+            if not x in Task.VALID_KEYS:
+                raise errors.AnsibleError("%s is not a legal parameter in an Ansible task or handler" % x)
+
+        self.module_vars = module_vars
+        self.play        = play
+
+        # load various attributes
+        self.name        = ds.get('name', None)
+        self.action      = ds.get('action', '')
+        self.tags        = [ 'all' ]
+
+        # notified by is used by Playbook code to flag which hosts
+        # need to run a notifier
+        self.notified_by = []
+
+        # if no name is specified, use the action line as the name
+        if self.name is None:
+            self.name = self.action
+
+        # load various attributes
+        self.only_if = ds.get('only_if', 'True')
+        self.async_seconds = int(ds.get('async', 0))  # not async by default
+        self.async_poll_interval = int(ds.get('poll', 10))  # default poll = 10 seconds
+        self.notify = ds.get('notify', [])
+        self.first_available_file = ds.get('first_available_file', None)
+        self.with_items = ds.get('with_items', None)
+        self.ignore_errors = ds.get('ignore_errors', False)
+        
+        # notify can be a string or a list, store as a list
+        if isinstance(self.notify, basestring):
+            self.notify = [ self.notify ]
+
+        # split the action line into a module name + arguments
+        tokens = self.action.split(None, 1)
+        if len(tokens) < 1:
+            raise errors.AnsibleError("invalid/missing action in task. name: %s" % self.name)
+        self.module_name = tokens[0]
+        self.module_args = ''
+        if len(tokens) > 1:
+            self.module_args = tokens[1]
+
+        import_tags = self.module_vars.get('tags',[])
+        if type(import_tags) in [str,unicode]:
+            # allow the user to list comma delimited tags
+            import_tags = import_tags.split(",")
+
+        self.name = utils.template(self.name, self.module_vars)
+        self.action = utils.template(self.action, self.module_vars)
+
+        # handle mutually incompatible options
+        if self.with_items is not None and self.first_available_file is not None:
+            raise errors.AnsibleError("with_items and first_available_file are mutually incompatible in a single task")
+
+        # make first_available_file accessable to Runner code
+        if self.first_available_file:
+            self.module_vars['first_available_file'] = self.first_available_file
+      
+        # process with_items so it can be used by Runner code 
+        if self.with_items is None:
+            self.with_items = [ ]
+        self.module_vars['items'] = self.with_items
+
+        # make ignore_errors accessable to Runner code
+        self.module_vars['ignore_errors'] = self.ignore_errors
+
+        # tags allow certain parts of a playbook to be run without running the whole playbook
+        apply_tags = ds.get('tags', None)
+        if apply_tags is not None:
+            if type(apply_tags) in [ str, unicode ]:
+                self.tags.append(apply_tags)
+            elif type(apply_tags) == list:
+                self.tags.extend(apply_tags)
+        self.tags.extend(import_tags)
+                

lib/ansible/runner.py

@@ -1,475 +0,0 @@
-# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-################################################
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
-
-import fnmatch
-import multiprocessing
-import signal
-import os
-import ansible.constants as C 
-import Queue
-import random
-import paramiko
-import jinja2
-from ansible.utils import *
-    
-################################################
-
-def noop(*args, **kwargs):
-    pass
-
-def _executor_hook(job_queue, result_queue):
-    ''' callback used by multiprocessing pool '''
-
-    signal.signal(signal.SIGINT, signal.SIG_IGN)
-    while not job_queue.empty():
-        try:
-            job = job_queue.get(block=False)
-            runner, host = job
-            result_queue.put(runner._executor(host))
-        except Queue.Empty:
-            pass
-
-class Runner(object):
-
-    def __init__(self, 
-        host_list=C.DEFAULT_HOST_LIST, 
-        module_path=C.DEFAULT_MODULE_PATH,
-        module_name=C.DEFAULT_MODULE_NAME, 
-        module_args=C.DEFAULT_MODULE_ARGS, 
-        forks=C.DEFAULT_FORKS, 
-        timeout=C.DEFAULT_TIMEOUT, 
-        pattern=C.DEFAULT_PATTERN,
-        remote_user=C.DEFAULT_REMOTE_USER,
-        remote_pass=C.DEFAULT_REMOTE_PASS,
-        background=0,
-        basedir=None,
-        setup_cache={},
-        verbose=False):
-    
-        ''' 
-        Constructor
-        host_list   -- file on disk listing hosts to manage, or an array of hostnames
-        pattern ------ a fnmatch pattern selecting some of the hosts in host_list
-        module_path -- location of ansible library on disk
-        module_name -- which module to run
-        module_args -- arguments to pass to module
-        forks -------- how parallel should we be? 1 is extra debuggable.
-        remote_user -- who to login as (default root)
-        remote_pass -- provide only if you don't want to use keys or ssh-agent
-        background --- if non 0, run async, failing after X seconds, -1 == infinite
-        setup_cache -- used only by playbook (complex explanation pending)
-        '''
-
-        self.setup_cache = setup_cache
-       
-        self.host_list, self.groups = self.parse_hosts(host_list)
-        self.module_path = module_path
-        self.module_name = module_name
-        self.forks       = int(forks)
-        self.pattern     = pattern
-        self.module_args = module_args
-        self.timeout     = timeout
-        self.verbose     = verbose
-        self.remote_user = remote_user
-        self.remote_pass = remote_pass
-        self.background  = background
-
-        if basedir is None: 
-            basedir = os.getcwd()
-        self.basedir = basedir
-
-        # hosts in each group name in the inventory file
-        self._tmp_paths  = {}
-
-        random.seed()
-        self.generated_jid = str(random.randint(0, 999999999999))
-
-    @classmethod
-    def parse_hosts(cls, host_list):
-        ''' 
-        parse the host inventory file, returns (hosts, groups) 
-        [groupname]
-        host1
-        host2
-        '''
-
-        if type(host_list) == list:
-            return (host_list, {})
-
-        host_list = os.path.expanduser(host_list)
-        lines = file(host_list).read().split("\n")
-        groups     = {}
-        groups['ungrouped'] = []
-        group_name = 'ungrouped'
-        results    = []
-        for item in lines:
-            if item.startswith("["):
-                group_name = item.replace("[","").replace("]","").lstrip().rstrip()
-                groups[group_name] = []
-            else:
-                groups[group_name].append(item)
-                results.append(item)
-
-        return (results, groups)
-
-
-    def _matches(self, host_name, pattern=None):
-        ''' returns if a hostname is matched by the pattern '''
-        # a pattern is in fnmatch format but more than one pattern
-        # can be strung together with semicolons. ex:
-        #   atlanta-web*.example.com;dc-web*.example.com
-
-        if host_name == '':
-            return False
-        pattern = pattern.replace(";",":")
-        subpatterns = pattern.split(":")
-        for subpattern in subpatterns:
-            # the pattern could be a real glob
-            if subpattern == 'all':
-                return True
-            if fnmatch.fnmatch(host_name, subpattern):
-                return True
-            # or it could be a literal group name instead
-            if subpattern in self.groups:
-                if host_name in self.groups[subpattern]:
-                    return True
-        return False
-
-    def _connect(self, host):
-        ''' 
-        obtains a paramiko connection to the host.
-        on success, returns (True, connection) 
-        on failure, returns (False, traceback str)
-        '''
-        ssh = paramiko.SSHClient()
-        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
-        try:
-            # try paramiko
-            ssh.connect(host, username=self.remote_user, allow_agent=True, 
-              look_for_keys=True, password=self.remote_pass, timeout=self.timeout)
-            return [ True, ssh ]
-        except Exception, e:
-            # it failed somehow, return the failure string
-            return [ False, "FAILED: %s" % str(e) ]
-
-    def _return_from_module(self, conn, host, result):
-        ''' helper function to handle JSON parsing of results '''
-        try:
-            # try to parse the JSON response
-            return [ host, True, json.loads(result) ]
-        except Exception, e:
-            # it failed, say so, but return the string anyway
-            return [ host, False, "%s/%s" % (str(e), result) ]
-
-    def _delete_remote_files(self, conn, files):
-        ''' deletes one or more remote files '''
-        if type(files) == str:
-            files = [ files ]
-        for filename in files:
-            if not filename.startswith('/tmp/'):
-                raise Exception("not going to happen")
-            self._exec_command(conn, "rm -rf %s" % filename)
-
-    def _transfer_file(self, conn, source, dest):
-        ''' transfers a remote file '''
-        self.remote_log(conn, 'COPY remote:%s local:%s' % (source, dest))
-        sftp = conn.open_sftp()
-        sftp.put(source, dest)
-        sftp.close()
-
-    def _transfer_module(self, conn, tmp, module):
-        ''' 
-        transfers a module file to the remote side to execute it,
-        but does not execute it yet
-        '''
-        outpath = self._copy_module(conn, tmp, module)
-        self._exec_command(conn, "chmod +x %s" % outpath)
-        return outpath
-
-    def _execute_module(self, conn, tmp, remote_module_path, module_args):
-        ''' 
-        runs a module that has already been transferred, but first
-        modifies the command using setup_cache variables (see playbook)
-        '''
-        args = module_args
-        if type(args) == list:
-            args = [ str(x) for x in module_args ]
-            args = " ".join(args)
-        inject_vars = self.setup_cache.get(conn._host,{})
-
-        # the metadata location for the setup module is transparently managed
-        # since it's an 'internals' module, kind of a black box. See playbook
-        # other modules are not allowed to have this kind of handling
-        if remote_module_path.endswith("/setup") and args.find("metadata=") == -1:
-            if self.remote_user == 'root':
-                args = "%s metadata=/etc/ansible/setup" % args
-            else:
-                args = "%s metadata=~/.ansible/setup" % args
-
-        template = jinja2.Template(args)
-        args = template.render(inject_vars)
-
-        cmd = "%s %s" % (remote_module_path, args)
-        result = self._exec_command(conn, cmd)
-        self._delete_remote_files(conn, [ tmp ])
-        return result
-
-    def _execute_normal_module(self, conn, host, tmp):
-        ''' 
-        transfer & execute a module that is not 'copy' or 'template'
-        because those require extra work.
-        '''
-        module = self._transfer_module(conn, tmp, self.module_name)
-
-        result = self._execute_module(conn, tmp, module, self.module_args)
-
-        # when running the setup module, which pushes vars to the host and ALSO
-        # returns them (+factoids), store the variables that were returned such that commands
-        # run AFTER setup use these variables for templating when executed
-        # from playbooks
-        if self.module_name == 'setup':
-            host = conn._host
-            try:
-                var_result = json.loads(result)
-            except:
-                var_result = {}
-
-        self._delete_remote_files(conn, tmp)
-        return self._return_from_module(conn, host, result)
-
-    def _execute_async_module(self, conn, host, tmp):
-        ''' 
-        transfer the given module name, plus the async module
-        and then run the async module wrapping the other module
-        '''
-        async  = self._transfer_module(conn, tmp, 'async_wrapper')
-        module = self._transfer_module(conn, tmp, self.module_name)
-        new_args = [] 
-        new_args = [ self.generated_jid, self.background, module ]
-        new_args.extend(self.module_args)
-        result = self._execute_module(conn, tmp, async, new_args)
-        return self._return_from_module(conn, host, result)
-
-    def _parse_kv(self, args):
-        ''' helper function to convert a string of key/value items to a dict '''
-        options = {}
-        for x in args:
-            if x.find("=") != -1:
-                k, v = x.split("=")
-                options[k]=v
-        return options
-
-    def _execute_copy(self, conn, host, tmp):
-        ''' handler for file transfer operations '''
-
-        # load up options
-        options = self._parse_kv(self.module_args)
-        source = options['src']
-        dest   = options['dest']
-        
-        # transfer the file to a remote tmp location
-        tmp_path = tmp
-        tmp_src = tmp_path + source.split('/')[-1]
-        self._transfer_file(conn, path_dwim(self.basedir, source), tmp_src)
-
-        # install the copy  module
-        self.module_name = 'copy'
-        module = self._transfer_module(conn, tmp, 'copy')
-
-        # run the copy module
-        args = [ "src=%s" % tmp_src, "dest=%s" % dest ]
-        result = self._execute_module(conn, tmp, module, args)
-        self._delete_remote_files(conn, tmp_path)
-        return self._return_from_module(conn, host, result)
-
-    def _execute_template(self, conn, host, tmp):
-        ''' handler for template operations '''
-
-        # load up options
-        options  = self._parse_kv(self.module_args)
-        source   = options['src']
-        dest     = options['dest']
-        metadata = options.get('metadata', None)
-
-        if metadata is None:
-            if self.remote_user == 'root':
-                metadata = '/etc/ansible/setup'
-            else:
-                metadata = '~/.ansible/setup'
-
-        # first copy the source template over
-        tpath = tmp
-        tempname = os.path.split(source)[-1]
-        temppath = tpath + tempname
-        self._transfer_file(conn, path_dwim(self.basedir, source), temppath)
-
-        # install the template module
-        template_module = self._transfer_module(conn, tmp, 'template')
-
-        # run the template module
-        args = [ "src=%s" % temppath, "dest=%s" % dest, "metadata=%s" % metadata ]
-        result = self._execute_module(conn, tmp, template_module, args)
-        self._delete_remote_files(conn, [ tpath ])
-        return self._return_from_module(conn, host, result)
-
-
-    def _executor(self, host):
-        ''' 
-        callback executed in parallel for each host.
-        returns (hostname, connected_ok, extra)
-        where extra is the result of a successful connect
-        or a traceback string
-        '''
-
-        # depending on whether it's a normal module,
-        # or a request to use the copy or template
-        # module, call the appropriate executor function
-
-        ok, conn = self._connect(host)
-        conn._host = host
-        if not ok:
-            return [ host, False, conn ]
-           
-        tmp = self._get_tmp_path(conn)
-        result = None
-        if self.module_name not in [ 'copy', 'template' ]:
-            if self.background == 0:
-                result = self._execute_normal_module(conn, host, tmp)
-            else:
-                result = self._execute_async_module(conn, host, tmp)
-        elif self.module_name == 'copy':
-            result = self._execute_copy(conn, host, tmp)
-        elif self.module_name == 'template':
-            result = self._execute_template(conn, host, tmp)
-        else:
-            # this would be a coding error in THIS module
-            # shouldn't occur
-            raise Exception("???")
-        self._delete_remote_files(conn, tmp)
-        conn.close()
-        
-        return result
-
-    def remote_log(self, conn, msg):
-        ''' this is the function we use to log things '''
-        stdin, stdout, stderr = conn.exec_command('/usr/bin/logger -t ansible -p auth.info %r' % msg)
-        # TODO: maybe make that optional
-
-    def _exec_command(self, conn, cmd):
-        ''' execute a command string over SSH, return the output '''
-        msg = '%s: %s' % (self.module_name, cmd)
-        self.remote_log(conn, msg)
-        stdin, stdout, stderr = conn.exec_command(cmd)
-        results = "\n".join(stdout.readlines())
-        return results
-
-    def _get_tmp_path(self, conn):
-        ''' gets a temporary path on a remote box '''
-        result = self._exec_command(conn, "mktemp -d /tmp/ansible.XXXXXX")
-        return result.split("\n")[0] + '/'
-
-    def _copy_module(self, conn, tmp, module):
-        ''' transfer a module over SFTP, does not run it '''
-        in_path = os.path.expanduser(
-            os.path.join(self.module_path, module)
-        )
-        out_path = tmp + module
-        sftp = conn.open_sftp()
-        sftp.put(in_path, out_path)
-        sftp.close()
-        return out_path
-
-    def match_hosts(self, pattern):
-        ''' return all matched hosts fitting a pattern '''
-        return [ h for h in self.host_list if self._matches(h, pattern) ]
-
-    def run(self):
-        ''' xfer & run module on all matched hosts '''
-       
-        # find hosts that match the pattern
-        hosts = self.match_hosts(self.pattern)
-        if len(hosts) == 0:
-            return None       
-
-        # attack pool of hosts in N forks
-        # _executor_hook does all of the work
-        hosts = [ (self,x) for x in hosts ]
-
-        if self.forks > 1:
-            job_queue = multiprocessing.Manager().Queue()
-            result_queue = multiprocessing.Manager().Queue()
- 
-            for i in hosts:
-                job_queue.put(i)
- 
-            workers = []
-            for i in range(self.forks):
-                tmp = multiprocessing.Process(target=_executor_hook,
-                                      args=(job_queue, result_queue))
-                tmp.start()
-                workers.append(tmp)
- 
-            try:
-                for worker in workers:
-                    worker.join()
-            except KeyboardInterrupt:
-                for worker in workers:
-                    worker.terminate()
-                    worker.join()
-            
-            results = []
-            while not result_queue.empty():
-                results.append(result_queue.get(block=False))
- 
-        else:
-            results = [ x._executor(h) for (x,h) in hosts ]
-
-        # sort hosts by ones we successfully contacted
-        # and ones we did not so that we can return a 
-        # dictionary containing results of everything
-
-        results2 = {
-          "contacted" : {},
-          "dark"      : {}
-        }
-        hosts_with_results = []
-        for x in results:
-            (host, is_ok, result) = x
-            hosts_with_results.append(host)
-            if not is_ok:
-                results2["dark"][host] = result
-            else:
-                results2["contacted"][host] = result
-        # hosts which were contacted but never got a chance
-        # to return a result before we exited/ctrl-c'd
-        # perhaps these shouldn't be 'dark' but I'm not sure if they fit
-        # anywhere else.
-        for host in self.match_hosts(self.pattern):
-            if host not in hosts_with_results:
-                results2["dark"][host] = {}
-                
-        return results2
-
-

lib/ansible/runner/__init__.py

@@ -0,0 +1,797 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import multiprocessing
+import signal
+import os
+import pwd
+import Queue
+import random
+import traceback
+import tempfile
+import time
+import base64
+import getpass
+import codecs
+import collections
+import re
+
+import ansible.constants as C 
+import ansible.inventory
+from ansible import utils
+from ansible import errors
+from ansible import module_common
+import poller
+import connection
+from ansible import callbacks as ans_callbacks
+    
+HAS_ATFORK=True
+try:
+    from Crypto.Random import atfork
+except ImportError:
+    HAS_ATFORK=False
+
+################################################
+
+def _executor_hook(job_queue, result_queue):
+    ''' callback used by multiprocessing pool '''
+
+    # attempt workaround of https://github.com/newsapps/beeswithmachineguns/issues/17
+    # this function also not present in CentOS 6
+    if HAS_ATFORK:
+        atfork()
+
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+    while not job_queue.empty():
+        try:
+            job = job_queue.get(block=False)
+            runner, host = job
+            result_queue.put(runner._executor(host))
+        except Queue.Empty:
+            pass
+        except:
+            traceback.print_exc()
+ 
+################################################
+
+class ReturnData(object):
+    ''' internal return class for execute methods, not part of API signature '''
+
+    __slots__ = [ 'result', 'comm_ok', 'host' ]
+
+    def __init__(self, host=None, result=None, comm_ok=True):
+        self.host = host
+        self.result = result
+        self.comm_ok = comm_ok
+
+        if type(self.result) in [ str, unicode ]:
+            self.result = utils.parse_json(self.result)
+
+        if host is None:
+            raise Exception("host not set")
+        if type(self.result) != dict:
+            raise Exception("dictionary result expected")
+
+    def communicated_ok(self):
+        return self.comm_ok
+
+    def is_successful(self):
+        return self.comm_ok and ('failed' not in self.result) and (self.result.get('rc',0) == 0)
+
+    def daisychain(self, module_name):
+        ''' request a module call follow this one '''
+        if self.is_successful():
+            self.result['daisychain'] = module_name
+        return self
+
+class Runner(object):
+    ''' core API interface to ansible '''
+
+    # see bin/ansible for how this is used...
+
+    def __init__(self, 
+        host_list=C.DEFAULT_HOST_LIST,      # ex: /etc/ansible/hosts, legacy usage
+        module_path=C.DEFAULT_MODULE_PATH,  # ex: /usr/share/ansible
+        module_name=C.DEFAULT_MODULE_NAME,  # ex: copy
+        module_args=C.DEFAULT_MODULE_ARGS,  # ex: "src=/tmp/a dest=/tmp/b"
+        forks=C.DEFAULT_FORKS,              # parallelism level
+        timeout=C.DEFAULT_TIMEOUT,          # SSH timeout
+        pattern=C.DEFAULT_PATTERN,          # which hosts?  ex: 'all', 'acme.example.org'
+        remote_user=C.DEFAULT_REMOTE_USER,  # ex: 'username'
+        remote_pass=C.DEFAULT_REMOTE_PASS,  # ex: 'password123' or None if using key
+        remote_port=C.DEFAULT_REMOTE_PORT,  # if SSH on different ports
+        private_key_file=C.DEFAULT_PRIVATE_KEY_FILE, # if not using keys/passwords 
+        sudo_pass=C.DEFAULT_SUDO_PASS,      # ex: 'password123' or None
+        background=0,                       # async poll every X seconds, else 0 for non-async
+        basedir=None,                       # directory of playbook, if applicable
+        setup_cache=None,                   # used to share fact data w/ other tasks
+        transport=C.DEFAULT_TRANSPORT,      # 'ssh', 'paramiko', 'local'
+        conditional='True',                 # run only if this fact expression evals to true
+        callbacks=None,                     # used for output
+        verbose=False,                      # whether to show more or less
+        sudo=False,                         # whether to run sudo or not
+        sudo_user=C.DEFAULT_SUDO_USER,      # ex: 'root'
+        module_vars=None,                   # a playbooks internals thing 
+        is_playbook=False,                  # running from playbook or not?
+        inventory=None                      # reference to Inventory object
+        ):
+
+        # storage & defaults
+        self.setup_cache      = utils.default(setup_cache, lambda: collections.defaultdict(dict))
+        self.basedir          = utils.default(basedir, lambda: os.getcwd())
+        self.callbacks        = utils.default(callbacks, lambda: ans_callbacks.DefaultRunnerCallbacks())
+        self.generated_jid    = str(random.randint(0, 999999999999))
+        self.transport        = transport
+        self.inventory        = utils.default(inventory, lambda: ansible.inventory.Inventory(host_list))
+        self.module_vars      = utils.default(module_vars, lambda: {})
+        self.sudo_user        = sudo_user
+        self.connector        = connection.Connection(self)
+        self.conditional      = conditional
+        self.module_path      = module_path
+        self.module_name      = module_name
+        self.forks            = int(forks)
+        self.pattern          = pattern
+        self.module_args      = module_args
+        self.timeout          = timeout
+        self.verbose          = verbose
+        self.remote_user      = remote_user
+        self.remote_pass      = remote_pass
+        self.remote_port      = remote_port
+        self.private_key_file = private_key_file
+        self.background       = background
+        self.sudo             = sudo
+        self.sudo_pass        = sudo_pass
+        self.is_playbook      = is_playbook
+
+        # misc housekeeping
+        if self.transport == 'ssh' and remote_pass:
+            raise errors.AnsibleError("SSH transport does not support passwords, only keys or agents")
+        if self.transport == 'local':
+            self.remote_user = pwd.getpwuid(os.geteuid())[0]
+ 
+        # ensure we are using unique tmp paths
+        random.seed()
+
+    # *****************************************************
+
+    def _delete_remote_files(self, conn, files):
+        ''' deletes one or more remote files '''
+
+        if type(files) == str:
+            files = [ files ]
+        for filename in files:
+            if filename.find('/tmp/') == -1:
+                raise Exception("not going to happen")
+            self._low_level_exec_command(conn, "rm -rf %s" % filename, None)
+
+    # *****************************************************
+
+    def _transfer_str(self, conn, tmp, name, data):
+        ''' transfer string to remote file '''
+
+        if type(data) == dict:
+            data = utils.jsonify(data)
+
+        afd, afile = tempfile.mkstemp()
+        afo = os.fdopen(afd, 'w')
+        afo.write(data.encode('utf8'))
+        afo.flush()
+        afo.close()
+
+        remote = os.path.join(tmp, name)
+        try:
+            conn.put_file(afile, remote)
+        finally:
+            os.unlink(afile)
+        return remote
+
+    # *****************************************************
+
+    def _execute_module(self, conn, tmp, module_name, args, 
+        async_jid=None, async_module=None, async_limit=None, inject=None):
+
+        ''' runs a module that has already been transferred '''
+
+        if type(args) == dict:
+            args = utils.jsonify(args,format=True)
+
+        (remote_module_path, is_new_style) = self._copy_module(conn, tmp, module_name, inject)
+        cmd = "chmod +x %s" % remote_module_path
+        if self.sudo and self.sudo_user != 'root':
+            # deal with possible umask issues once sudo'ed to other user
+            cmd = "chmod 555 %s" % remote_module_path
+        self._low_level_exec_command(conn, cmd, tmp)
+
+        cmd = ""
+        if not is_new_style:
+            args = utils.template(args, inject)
+            argsfile = self._transfer_str(conn, tmp, 'arguments', args)
+            if async_jid is None:
+                cmd = "%s %s" % (remote_module_path, argsfile)
+            else:
+                cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module, argsfile]])
+        else:
+            if async_jid is None:
+                cmd = "%s" % (remote_module_path)
+            else:
+                cmd = " ".join([str(x) for x in [remote_module_path, async_jid, async_limit, async_module]])
+
+        res = self._low_level_exec_command(conn, cmd, tmp, sudoable=True)
+        return ReturnData(host=conn.host, result=res)
+
+    # *****************************************************
+
+    def _execute_raw(self, conn, tmp, inject=None):
+        ''' execute a non-module command for bootstrapping, or if there's no python on a device '''
+        return ReturnData(host=conn.host, result=dict(
+            stdout=self._low_level_exec_command(conn, self.module_args, tmp, sudoable = True)
+        ))
+
+    # ***************************************************
+
+    def _execute_normal_module(self, conn, tmp, module_name, inject=None):
+        ''' transfer & execute a module that is not 'copy' or 'template' '''
+
+        # shell and command are the same module
+        if module_name == 'shell':
+            module_name = 'command'
+            self.module_args += " #USE_SHELL"
+
+        exec_rc = self._execute_module(conn, tmp, module_name, self.module_args, inject=inject)
+        return exec_rc
+
+    # *****************************************************
+
+    def _execute_async_module(self, conn, tmp, module_name, inject=None):
+        ''' transfer the given module name, plus the async module, then run it '''
+
+        # shell and command module are the same
+        module_args = self.module_args
+        if module_name == 'shell':
+            module_name = 'command'
+            module_args += " #USE_SHELL"
+
+        (module_path, is_new_style) = self._copy_module(conn, tmp, module_name, inject)
+        self._low_level_exec_command(conn, "chmod +x %s" % module_path, tmp)
+
+        return self._execute_module(conn, tmp, 'async_wrapper', module_args,
+           async_module=module_path,
+           async_jid=self.generated_jid, 
+           async_limit=self.background,
+           inject=inject
+        )
+
+    # *****************************************************
+
+    def _execute_copy(self, conn, tmp, inject=None):
+        ''' handler for file transfer operations '''
+
+        # load up options
+        options = utils.parse_kv(self.module_args)
+        source  = options.get('src', None)
+        dest    = options.get('dest', None)
+        if (source is None and not 'first_available_file' in inject) or dest is None:
+            result=dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, result=result)
+
+        # if we have first_available_file in our vars
+        # look up the files and use the first one we find as src
+        if 'first_available_file' in inject:
+            found = False
+            for fn in inject.get('first_available_file'):
+                fn = utils.template(fn, inject)
+                if os.path.exists(fn):
+                    source = fn
+                    found = True
+                    break
+            if not found:
+                results=dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(host=conn.host, results=results)
+       
+        source = utils.template(source, inject)
+        source = utils.path_dwim(self.basedir, source)
+
+        local_md5 = utils.md5(source)
+        if local_md5 is None:
+            result=dict(failed=True, msg="could not find src=%s" % source)
+            return ReturnData(host=conn.host, result=result)
+            
+        remote_md5 = self._remote_md5(conn, tmp, dest) 
+
+        exec_rc = None 
+        if local_md5 != remote_md5:
+            # transfer the file to a remote tmp location
+            tmp_src = tmp + source.split('/')[-1]
+            conn.put_file(source, tmp_src)
+
+            # run the copy module
+            self.module_args = "%s src=%s" % (self.module_args, tmp_src)
+            return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
+
+        else:
+            # no need to transfer the file, already correct md5
+            result = dict(changed=False, md5sum=remote_md5, transferred=False)
+            return ReturnData(host=conn.host, result=result).daisychain('file')
+
+    # *****************************************************
+
+    def _execute_fetch(self, conn, tmp, inject=None):
+        ''' handler for fetch operations '''
+
+        # load up options
+        options = utils.parse_kv(self.module_args)
+        source = options.get('src', None)
+        dest = options.get('dest', None)
+        if source is None or dest is None:
+            results = dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, result=results)
+
+        # apply templating to source argument
+        source = utils.template(source, inject)
+        # apply templating to dest argument
+        dest = utils.template(dest, inject)
+       
+        # files are saved in dest dir, with a subdir for each host, then the filename
+        dest   = "%s/%s/%s" % (utils.path_dwim(self.basedir, dest), conn.host, source)
+        dest   = dest.replace("//","/")
+
+        # calculate md5 sum for the remote file
+        remote_md5 = self._remote_md5(conn, tmp, source)
+
+        # these don't fail because you may want to transfer a log file that possibly MAY exist
+        # but keep going to fetch other log files
+        if remote_md5 == '0':
+            result = dict(msg="unable to calculate the md5 sum of the remote file", file=source, changed=False)
+            return ReturnData(host=conn.host, result=result)
+        if remote_md5 == '1':
+            result = dict(msg="the remote file does not exist, not transferring, ignored", file=source, changed=False)
+            return ReturnData(host=conn.host, result=result)
+        if remote_md5 == '2':
+            result = dict(msg="no read permission on remote file, not transferring, ignored", file=source, changed=False)
+            return ReturnData(host=conn.host, result=result)
+
+        # calculate md5 sum for the local file
+        local_md5 = utils.md5(dest)
+
+        if remote_md5 != local_md5:
+            # create the containing directories, if needed
+            if not os.path.isdir(os.path.dirname(dest)):
+                os.makedirs(os.path.dirname(dest))
+
+            # fetch the file and check for changes
+            conn.fetch_file(source, dest)
+            new_md5 = utils.md5(dest)
+            if new_md5 != remote_md5:
+                result = dict(failed=True, md5sum=new_md5, msg="md5 mismatch", file=source)
+                return ReturnData(host=conn.host, result=result)
+            result = dict(changed=True, md5sum=new_md5)
+            return ReturnData(host=conn.host, result=result)
+        else:
+            result = dict(changed=False, md5sum=local_md5, file=source)
+            return ReturnData(host=conn.host, result=result)
+        
+    # *****************************************************
+
+    def _execute_template(self, conn, tmp, inject=None):
+        ''' handler for template operations '''
+
+        if not self.is_playbook:
+            raise errors.AnsibleError("in current versions of ansible, templates are only usable in playbooks")
+
+        # load up options
+        options  = utils.parse_kv(self.module_args)
+        source   = options.get('src', None)
+        dest     = options.get('dest', None)
+        if (source is None and 'first_available_file' not in inject) or dest is None:
+            result = dict(failed=True, msg="src and dest are required")
+            return ReturnData(host=conn.host, comm_ok=False, result=result)
+
+        # if we have first_available_file in our vars
+        # look up the files and use the first one we find as src
+        if 'first_available_file' in inject:
+            found = False
+            for fn in self.module_vars.get('first_available_file'):
+                fn = utils.template(fn, inject)
+                if os.path.exists(fn):
+                    source = fn
+                    found = True
+                    break
+            if not found:
+                result = dict(failed=True, msg="could not find src in first_available_file list")
+                return ReturnData(host=conn.host, comm_ok=False, result=result)
+
+        source = utils.template(source, inject)
+
+        # template the source data locally & transfer
+        try:
+            resultant = utils.template_from_file(self.basedir, source, inject)
+        except Exception, e:
+            result = dict(failed=True, msg=str(e))
+            return ReturnData(host=conn.host, comm_ok=False, result=result)
+        xfered = self._transfer_str(conn, tmp, 'source', resultant)
+            
+        # run the copy module, queue the file module
+        self.module_args = "%s src=%s dest=%s" % (self.module_args, xfered, dest)
+        return self._execute_module(conn, tmp, 'copy', self.module_args, inject=inject).daisychain('file')
+
+    # *****************************************************
+
+    def _execute_assemble(self, conn, tmp, inject=None):
+        ''' handler for assemble operations '''
+
+        # FIXME: once assemble is ported over to the use the new common logic, this method
+        # will be unneccessary as it can decide to daisychain via it's own module returns.
+        # and this function can be deleted.  
+
+        return self._execute_module(conn, tmp, 'assemble', self.module_args, inject=inject).daisychain('file')
+
+    # *****************************************************
+
+    def _executor(self, host):
+        ''' handler for multiprocessing library '''
+
+        try:
+            exec_rc = self._executor_internal(host)
+            if type(exec_rc) != ReturnData:
+                raise Exception("unexpected return type: %s" % type(exec_rc))
+            # redundant, right?
+            if not exec_rc.comm_ok:
+                self.callbacks.on_unreachable(host, exec_rc.result)
+            return exec_rc
+        except errors.AnsibleError, ae:
+            msg = str(ae)
+            self.callbacks.on_unreachable(host, msg)
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
+        except Exception:
+            msg = traceback.format_exc()
+            self.callbacks.on_unreachable(host, msg)
+            return ReturnData(host=host, comm_ok=False, result=dict(failed=True, msg=msg))
+
+    # *****************************************************
+
+    def _executor_internal(self, host):
+        ''' executes any module one or more times '''
+
+        host_variables = self.inventory.get_variables(host)
+        port = host_variables.get('ansible_ssh_port', self.remote_port)
+
+        inject = self.setup_cache[host].copy()
+        inject.update(host_variables)
+        inject.update(self.module_vars)
+        inject['hostvars'] = self.setup_cache
+
+        items = self.module_vars.get('items', [])
+
+        if isinstance(items, basestring) and items.startswith("$"):
+            items = items.replace("$","")
+            if items in inject:
+                items = inject[items]
+            else:
+                raise errors.AnsibleError("unbound variable in with_items: %s" % items)
+        if type(items) != list:
+            raise errors.AnsibleError("with_items only takes a list: %s" % items)
+
+        if self.module_name in [ 'apt', 'yum' ]:
+            # hack for apt and soon yum, with_items maps back into a single module call
+            inject['item'] = ",".join(items)
+            items = []
+          
+        if len(items) == 0:
+            return self._executor_internal_inner(host, inject, port)
+        else:
+            # executing using with_items, so make multiple calls
+            # TODO: refactor
+            aggregrate = {}
+            all_comm_ok = True
+            all_changed = False
+            all_failed = False
+            results = []
+            # Save module name and args since daisy-chaining can overwrite them
+            module_name = self.module_name
+            module_args = self.module_args
+            for x in items:
+                self.module_name = module_name
+                self.module_args = module_args
+                inject['item'] = x
+                result = self._executor_internal_inner(host, inject, port)
+                results.append(result.result)
+                if result.comm_ok == False:
+                    all_comm_ok = False
+                    break
+                for x in results:
+                    if x.get('changed') == True:
+                        all_changed = True
+                    if (x.get('failed') == True) or (('rc' in x) and (x['rc'] != 0)):
+                        all_failed = True
+                        break
+            msg = 'All items succeeded'
+            if all_failed:
+                msg = "One or more items failed."
+            rd_result = dict(failed=all_failed, changed=all_changed, results=results, msg=msg)
+            if not all_failed:
+                del rd_result['failed']
+            return ReturnData(host=host, comm_ok=all_comm_ok, result=rd_result)
+
+    # *****************************************************
+
+    def _executor_internal_inner(self, host, inject, port, is_chained=False):
+        ''' decides how to invoke a module '''
+
+        # special non-user/non-fact variables:
+        # 'groups' variable is a list of host name in each group
+        # 'hostvars' variable contains variables for each host name
+        #  ... and is set elsewhere
+        # 'inventory_hostname' is also set elsewhere
+        group_hosts = {}
+        for g in self.inventory.groups:
+            group_hosts[g.name] = [ h.name for h in g.hosts ]
+        inject['groups'] = group_hosts
+
+        # allow module args to work as a dictionary
+        # though it is usually a string
+        new_args = ""
+        if type(self.module_args) == dict:
+            for (k,v) in self.module_args.iteritems():
+                new_args = new_args + "%s='%s' " % (k,v)
+            self.module_args = new_args
+        self.module_args = utils.template(self.module_args, inject)
+
+        conditional = utils.template(self.conditional, inject)
+        if not eval(conditional):
+            result = utils.jsonify(dict(skipped=True))
+            self.callbacks.on_skipped(host, inject.get('item',None))
+            return ReturnData(host=host, result=result)
+
+        conn = None
+        try:
+            conn = self.connector.connect(host, port)
+        except errors.AnsibleConnectionFailed, e:
+            result = dict(failed=True, msg="FAILED: %s" % str(e))
+            return ReturnData(host=host, comm_ok=False, result=result)
+
+        module_name = utils.template(self.module_name, inject)
+
+        tmp = self._make_tmp_path(conn)
+        result = None
+
+        handler = getattr(self, "_execute_%s" % self.module_name, None)
+        if handler:
+            result = handler(conn, tmp, inject=inject)
+        else:
+            if self.background == 0:
+                result = self._execute_normal_module(conn, tmp, module_name, inject=inject)
+            else:
+                result = self._execute_async_module(conn, tmp, module_name, inject=inject)
+
+        result.result['module'] = self.module_name
+        if result.is_successful() and 'daisychain' in result.result:
+            self.module_name = result.result['daisychain']
+            if 'daisychain_args' in result.result:
+                self.module_args = result.result['daisychain_args']
+            result2 = self._executor_internal_inner(host, inject, port, is_chained=True)
+            result2.result['module'] = self.module_name
+            changed = False
+            if result.result.get('changed',False) or result2.result.get('changed',False):
+                changed = True
+            result.result.update(result2.result)
+            result.result['changed'] = changed
+            
+            del result.result['daisychain']
+
+        self._delete_remote_files(conn, tmp)
+        conn.close()
+
+        if not result.comm_ok:
+            # connection or parsing errors...
+            self.callbacks.on_unreachable(host, result.result)
+        else:
+            data = result.result
+            if 'item' in inject:
+                result.result['item'] = inject['item']
+            if is_chained:
+                # no callbacks
+                return result
+            if 'skipped' in data:
+                self.callbacks.on_skipped(result.host)
+            elif not result.is_successful():
+                ignore_errors = self.module_vars.get('ignore_errors', False)
+                self.callbacks.on_failed(result.host, data, ignore_errors)
+                if ignore_errors:
+                    if 'failed' in result.result:
+                        result.result['failed'] = False
+                    if 'rc' in result.result:
+                        result.result['rc'] = 0
+            else:
+                self.callbacks.on_ok(result.host, data)
+        return result
+
+    # *****************************************************
+
+    def _low_level_exec_command(self, conn, cmd, tmp, sudoable=False):
+        ''' execute a command string over SSH, return the output '''
+
+        sudo_user = self.sudo_user
+        stdin, stdout, stderr = conn.exec_command(cmd, tmp, sudo_user, sudoable=sudoable)
+
+        if type(stdout) != str:
+            out = "\n".join(stdout.readlines())
+        else:
+            out = stdout
+        
+        if type(stderr) != str:
+            err = "\n".join(stderr.readlines())
+        else:
+            err = stderr
+
+        return out + err
+
+    # *****************************************************
+
+    def _remote_md5(self, conn, tmp, path):
+        ''' takes a remote md5sum without requiring python, and returns 0 if no file ''' 
+    
+        test = "rc=0; [[ -r \"%s\" ]] || rc=2; [[ -f \"%s\" ]] || rc=1" % (path,path)
+        md5s = [
+            "(/usr/bin/md5sum %s 2>/dev/null)" % path,
+            "(/sbin/md5sum -q %s 2>/dev/null)" % path,
+            "(/usr/bin/digest -a md5 -v %s 2>/dev/null)" % path
+        ]
+    
+        cmd = " || ".join(md5s)
+        cmd = "%s; %s || (echo \"${rc}  %s\")" % (test, cmd, path)
+        return self._low_level_exec_command(conn, cmd, tmp, sudoable=False).split()[0]
+
+    # *****************************************************
+
+    def _make_tmp_path(self, conn):
+        ''' make and return a temporary path on a remote box '''
+
+        basefile = 'ansible-%s-%s' % (time.time(), random.randint(0, 2**48))
+        basetmp = os.path.join(C.DEFAULT_REMOTE_TMP, basefile)
+        if self.remote_user == 'root':
+            basetmp = os.path.join('/var/tmp', basefile)
+        elif self.sudo and self.sudo_user != 'root':
+            basetmp = os.path.join('/tmp', basefile)
+
+        cmd = 'mkdir -p %s' % basetmp
+        if self.remote_user != 'root':
+            cmd += ' && chmod a+x %s' % basetmp
+        cmd += ' && echo %s' % basetmp
+
+        result = self._low_level_exec_command(conn, cmd, None, sudoable=False)
+        return result.split("\n")[0].strip() + '/'
+
+    # *****************************************************
+
+    def _copy_module(self, conn, tmp, module, inject):
+        ''' transfer a module over SFTP, does not run it '''
+
+        if module.startswith("/"):
+            raise errors.AnsibleFileNotFound("%s is not a module" % module)
+
+        # Search module path(s) for named module.
+        for module_path in self.module_path.split(os.pathsep):
+            in_path = os.path.expanduser(os.path.join(module_path, module))
+            if os.path.exists(in_path):
+                break
+        else:
+            raise errors.AnsibleFileNotFound("module %s not found in %s" % (module, self.module_path))
+
+        out_path = os.path.join(tmp, module)
+
+        module_data = ""
+        is_new_style=False
+        with open(in_path) as f:
+            module_data = f.read()
+            if module_common.REPLACER in module_data:
+                is_new_style=True
+            module_data = module_data.replace(module_common.REPLACER, module_common.MODULE_COMMON)
+            encoded_args = "\"\"\"%s\"\"\"" % utils.template(self.module_args, inject).replace("\"","\\\"")
+            module_data = module_data.replace(module_common.REPLACER_ARGS, encoded_args)
+ 
+        # use the correct python interpreter for the host
+        if 'ansible_python_interpreter' in inject:
+            interpreter = inject['ansible_python_interpreter']
+            module_lines = module_data.split('\n')
+            if '#!' and 'python' in module_lines[0]:
+                module_lines[0] = "#!%s" % interpreter
+            module_data = "\n".join(module_lines)
+
+        self._transfer_str(conn, tmp, module, module_data)
+        return (out_path, is_new_style)
+
+    # *****************************************************
+
+    def _parallel_exec(self, hosts):
+        ''' handles mulitprocessing when more than 1 fork is required '''
+
+        job_queue = multiprocessing.Manager().Queue()
+        [job_queue.put(i) for i in hosts]
+        result_queue = multiprocessing.Manager().Queue()
+
+        workers = []
+        for i in range(self.forks):
+            prc = multiprocessing.Process(target=_executor_hook,
+                args=(job_queue, result_queue))
+            prc.start()
+            workers.append(prc)
+
+        try:
+            for worker in workers:
+                worker.join()
+        except KeyboardInterrupt:
+            for worker in workers:
+                worker.terminate()
+                worker.join()
+
+        results = []
+        while not result_queue.empty():
+            results.append(result_queue.get(block=False))
+        return results
+
+    # *****************************************************
+
+    def _partition_results(self, results):
+        ''' seperate results by ones we contacted & ones we didn't '''
+
+        if results is None:
+            return None
+        results2 = dict(contacted={}, dark={})
+
+        for result in results:
+            host = result.host
+            if host is None:
+                raise Exception("internal error, host not set")
+            if result.communicated_ok():
+                results2["contacted"][host] = result.result
+            else:
+                results2["dark"][host] = result.result
+
+        # hosts which were contacted but never got a chance to return
+        for host in self.inventory.list_hosts(self.pattern):
+            if not (host in results2['dark'] or host in results2['contacted']):
+                results2["dark"][host] = {}
+        return results2
+
+    # *****************************************************
+
+    def run(self):
+        ''' xfer & run module on all matched hosts '''
+       
+        # find hosts that match the pattern
+        hosts = self.inventory.list_hosts(self.pattern)
+        if len(hosts) == 0:
+            self.callbacks.on_no_hosts()
+            return dict(contacted={}, dark={})
+ 
+        hosts = [ (self,x) for x in hosts ]
+        results = None
+        if self.forks > 1:
+            results = self._parallel_exec(hosts)
+        else:
+            results = [ self._executor(h[1]) for h in hosts ]
+        return self._partition_results(results)
+
+    # *****************************************************
+
+    def run_async(self, time_limit):
+        ''' Run this module asynchronously and return a poller. '''
+
+        self.background = time_limit
+        results = self.run()
+        return results, poller.AsyncPoller(results, self)
+

lib/ansible/runner/connection/__init__.py

@@ -0,0 +1,43 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+################################################
+
+import local
+import paramiko_ssh
+import ssh
+
+class Connection(object):
+    ''' Handles abstract connections to remote hosts '''
+
+    def __init__(self, runner):
+        self.runner = runner
+
+    def connect(self, host, port=None):
+        conn = None
+        transport = self.runner.transport
+        if transport == 'local':
+            conn = local.LocalConnection(self.runner, host)
+        elif transport == 'paramiko':
+            conn = paramiko_ssh.ParamikoConnection(self.runner, host, port)
+        elif transport == 'ssh':
+            conn = ssh.SSHConnection(self.runner, host, port)
+        if conn is None:
+            raise Exception("unsupported connection type")
+        return conn.connect()
+

lib/ansible/runner/connection/local.py

@@ -0,0 +1,72 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import traceback
+import os
+import shutil
+import subprocess
+from ansible import errors
+
+class LocalConnection(object):
+    ''' Local based connections '''
+
+    def __init__(self, runner, host):
+        self.runner = runner
+        self.host = host
+
+    def connect(self, port=None):
+        ''' connect to the local host; nothing to do here '''
+
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user, sudoable=False):
+        ''' run a command on the local host '''
+
+        if self.runner.sudo and sudoable:
+            if self.runner.sudo_pass:
+                # NOTE: if someone wants to add sudo w/ password to the local connection type, they are welcome
+                # to do so.  The primary usage of the local connection is for crontab and kickstart usage however
+                # so this doesn't seem to be a huge priority
+                raise errors.AnsibleError("sudo with password is presently only supported on the paramiko (SSH) connection type")
+            cmd = "sudo -u {0} -s {1}".format(sudo_user, cmd)
+
+        p = subprocess.Popen(cmd, shell=True, stdin=None,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate()
+        return ("", stdout, stderr)
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to local '''
+
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        try:
+            shutil.copyfile(in_path, out_path)
+        except shutil.Error:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to copy: %s and %s are the same" % (in_path, out_path))
+        except IOError:
+            traceback.print_exc()
+            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
+
+    def fetch_file(self, in_path, out_path):
+        ''' fetch a file from local to local -- for copatibility '''
+        self.put_file(in_path, out_path)
+
+    def close(self):
+        ''' terminate the connection; nothing to do here '''
+        pass

lib/ansible/runner/connection/paramiko_ssh.py

@@ -0,0 +1,144 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import warnings
+import traceback
+import os
+import re
+import shutil
+import subprocess
+import pipes
+import socket
+import random
+from ansible import errors
+
+# prevent paramiko warning noise -- see http://stackoverflow.com/questions/3920502/
+HAVE_PARAMIKO=False
+with warnings.catch_warnings():
+    warnings.simplefilter("ignore")
+    try:
+        import paramiko
+        HAVE_PARAMIKO=True
+    except ImportError:
+        pass
+
+class ParamikoConnection(object):
+    ''' SSH based connections with Paramiko '''
+
+    def __init__(self, runner, host, port=None):
+        self.ssh = None
+        self.runner = runner
+        self.host = host
+        self.port = port
+        if port is None:
+            self.port = self.runner.remote_port
+
+    def connect(self):
+        ''' activates the connection object '''
+
+        if not HAVE_PARAMIKO:
+            raise errors.AnsibleError("paramiko is not installed")
+
+        user = self.runner.remote_user
+        ssh = paramiko.SSHClient()
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+        try:
+            ssh.connect(self.host, username=user, allow_agent=True, look_for_keys=True,
+                key_filename=self.runner.private_key_file, password=self.runner.remote_pass,
+                timeout=self.runner.timeout, port=self.port)
+        except Exception, e:
+            msg = str(e)
+            if "PID check failed" in msg:
+                raise errors.AnsibleError("paramiko version issue, please upgrade paramiko on the machine running ansible")
+            elif "Private key file is encrypted" in msg:
+                msg = 'ssh %s@%s:%s : %s\nTo connect as a different user, use -u <username>.' % (
+                    user, self.host, self.port, msg)
+                raise errors.AnsibleConnectionFailed(msg)
+            else:
+                raise errors.AnsibleConnectionFailed(msg)
+
+        self.ssh = ssh
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user, sudoable=False):
+        ''' run a command on the remote host '''
+
+        bufsize = 4096
+        chan = self.ssh.get_transport().open_session()
+        chan.get_pty() 
+
+        if not self.runner.sudo or not sudoable:
+            quoted_command = '"$SHELL" -c ' + pipes.quote(cmd) 
+            chan.exec_command(quoted_command)
+        else:
+            # Rather than detect if sudo wants a password this time, -k makes 
+            # sudo always ask for a password if one is required. The "--"
+            # tells sudo that this is the end of sudo options and the command
+            # follows.  Passing a quoted compound command to sudo (or sudo -s)
+            # directly doesn't work, so we shellquote it with pipes.quote() 
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
+            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
+            prompt = '[sudo via ansible, key=%s] password: ' % randbits
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s -- "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
+            sudo_output = ''
+            try:
+                chan.exec_command(sudocmd)
+                if self.runner.sudo_pass:
+                    while not sudo_output.endswith(prompt):
+                        chunk = chan.recv(bufsize)
+                        if not chunk:
+                            if 'unknown user' in sudo_output:
+                                raise errors.AnsibleError(
+                                    'user %s does not exist' % sudo_user)
+                            else:
+                                raise errors.AnsibleError('ssh connection ' +
+                                    'closed waiting for password prompt')
+                        sudo_output += chunk
+                    chan.sendall(self.runner.sudo_pass + '\n')
+            except socket.timeout:
+                raise errors.AnsibleError('ssh timed out waiting for sudo.\n' + sudo_output)
+
+        return (chan.makefile('wb', bufsize), chan.makefile('rb', bufsize), '')
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to remote '''
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        sftp = self.ssh.open_sftp()
+        try:
+            sftp.put(in_path, out_path)
+        except IOError:
+            raise errors.AnsibleError("failed to transfer file to %s" % out_path)
+        sftp.close()
+
+    def fetch_file(self, in_path, out_path):
+        ''' save a remote file to the specified path '''
+        sftp = self.ssh.open_sftp()
+        try:
+            sftp.get(in_path, out_path)
+        except IOError:
+            raise errors.AnsibleError("failed to transfer file from %s" % in_path)
+        sftp.close()
+
+    def close(self):
+        ''' terminate the connection '''
+        self.ssh.close()
+

lib/ansible/runner/connection/ssh.py

@@ -0,0 +1,138 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import subprocess
+import shlex
+import pipes
+import random
+import select
+import fcntl
+from ansible import errors
+
+class SSHConnection(object):
+    ''' ssh based connections '''
+
+    def __init__(self, runner, host, port):
+        self.runner = runner
+        self.host = host
+        self.port = port
+
+    def connect(self):
+        ''' connect to the remote host '''
+
+        self.common_args = []
+        extra_args = os.getenv("ANSIBLE_SSH_ARGS", None)
+        if extra_args is not None:
+            self.common_args += shlex.split(extra_args)
+        else:
+            self.common_args += ["-o", "ControlMaster=auto",
+                                 "-o", "ControlPersist=60s",
+                                 "-o", "ControlPath=/tmp/ansible-ssh-%h-%p-%r"]
+        self.common_args += ["-o", "StrictHostKeyChecking=no"]
+        if self.port is not None:
+            self.common_args += ["-o", "Port=%d" % (self.port)]
+        if self.runner.private_key_file is not None:
+            self.common_args += ["-o", "IdentityFile="+self.runner.private_key_file]
+        self.common_args += ["-o", "User="+self.runner.remote_user]
+
+        return self
+
+    def exec_command(self, cmd, tmp_path, sudo_user,sudoable=False):
+        ''' run a command on the remote host '''
+
+        ssh_cmd = ["ssh", "-tt", "-q"] + self.common_args + [self.host]
+        if self.runner.sudo and sudoable:
+            # Rather than detect if sudo wants a password this time, -k makes 
+            # sudo always ask for a password if one is required. The "--"
+            # tells sudo that this is the end of sudo options and the command
+            # follows.  Passing a quoted compound command to sudo (or sudo -s)
+            # directly doesn't work, so we shellquote it with pipes.quote() 
+            # and pass the quoted string to the user's shell.  We loop reading
+            # output until we see the randomly-generated sudo prompt set with
+            # the -p option.
+            randbits = ''.join(chr(random.randint(ord('a'), ord('z'))) for x in xrange(32))
+            prompt = '[sudo via ansible, key=%s] password: ' % randbits
+            sudocmd = 'sudo -k && sudo -p "%s" -u %s -- "$SHELL" -c %s' % (
+                prompt, sudo_user, pipes.quote(cmd))
+            sudo_output = ''
+            ssh_cmd.append(sudocmd)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+            if self.runner.sudo_pass:
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL,
+                            fcntl.fcntl(p.stdout, fcntl.F_GETFL) | os.O_NONBLOCK)
+                while not sudo_output.endswith(prompt):
+                    rfd, wfd, efd = select.select([p.stdout], [],
+                                                  [p.stdout], self.runner.timeout)
+                    if p.stdout in rfd:
+                        chunk = p.stdout.read()
+                        if not chunk:
+                            raise errors.AnsibleError('ssh connection closed waiting for sudo password prompt')
+                        sudo_output += chunk
+                    else:
+                        stdout = p.communicate()
+                        raise errors.AnsibleError('ssh connection error waiting for sudo password prompt')
+                p.stdin.write(self.runner.sudo_pass + '\n')
+                fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) & ~os.O_NONBLOCK)
+        else:
+            ssh_cmd.append(cmd)
+            p = subprocess.Popen(ssh_cmd, stdin=subprocess.PIPE,
+                                 stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
+
+        # We can't use p.communicate here because the ControlMaster may have stdout open as well
+        p.stdin.close()
+        stdout = ''
+        while p.poll() is None:
+            rfd, wfd, efd = select.select([p.stdout], [], [p.stdout], 1)
+            if p.stdout in rfd:
+                stdout += os.read(p.stdout.fileno(), 1024)
+        # older versions of ssh generate this error which we ignore
+        stdout=stdout.replace("tcgetattr: Invalid argument\n", "")
+        # suppress Ubuntu 10.04/12.04 error on -tt option 
+        stdout=stdout.replace("tcgetattr: Inappropriate ioctl for device\n","")
+
+        if p.returncode != 0 and stdout.find('Bad configuration option: ControlPersist') != -1:
+            raise errors.AnsibleError('using -c ssh on certain older ssh versions may not support ControlPersist, set ANSIBLE_SSH_ARGS="" before running again')
+            
+        return ('', stdout, '')
+
+    def put_file(self, in_path, out_path):
+        ''' transfer a file from local to remote '''
+        if not os.path.exists(in_path):
+            raise errors.AnsibleFileNotFound("file or module does not exist: %s" % in_path)
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("put %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file to %s:\n%s\n%s" % (out_path, stdout, stderr))
+
+    def fetch_file(self, in_path, out_path):
+        ''' fetch a file from remote to local '''
+        sftp_cmd = ["sftp"] + self.common_args + [self.host]
+        p = subprocess.Popen(sftp_cmd, stdin=subprocess.PIPE,
+                             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        stdout, stderr = p.communicate("get %s %s\n" % (in_path, out_path))
+        if p.returncode != 0:
+            raise errors.AnsibleError("failed to transfer file from %s:\n%s\n%s" % (in_path, stdout, stderr))
+
+    def close(self):
+        ''' not applicable since we're executing openssh binaries '''
+        pass
+

lib/ansible/runner/poller.py

@@ -0,0 +1,101 @@
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import time
+
+from ansible import errors
+
+class AsyncPoller(object):
+    """ Manage asynchronous jobs. """
+
+    def __init__(self, results, runner):
+        self.runner = runner
+
+        self.results = { 'contacted': {}, 'dark': {}}
+        self.hosts_to_poll = []
+        self.completed = False
+
+        # Get job id and which hosts to poll again in the future
+        jid = None
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started', False):
+                self.hosts_to_poll.append(host)
+                jid = res.get('ansible_job_id', None)
+            else:
+                self.results['contacted'][host] = res
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+
+        if jid is None:
+            raise errors.AnsibleError("unexpected error: unable to determine jid")
+        if len(self.hosts_to_poll)==0:
+            raise errors.AnsibleErrot("unexpected error: no hosts to poll")
+        self.jid = jid
+
+    def poll(self):
+        """ Poll the job status.
+
+            Returns the changes in this iteration."""
+        self.runner.module_name = 'async_status'
+        self.runner.module_args = "jid=%s" % self.jid
+        self.runner.pattern = "*"
+        self.runner.background = 0
+
+        self.runner.inventory.restrict_to(self.hosts_to_poll)
+        results = self.runner.run()
+        self.runner.inventory.lift_restriction()
+
+        hosts = []
+        poll_results = { 'contacted': {}, 'dark': {}, 'polled': {}}
+        for (host, res) in results['contacted'].iteritems():
+            if res.get('started',False):
+                hosts.append(host)
+                poll_results['polled'][host] = res
+            else:
+                self.results['contacted'][host] = res
+                poll_results['contacted'][host] = res
+                if 'failed' in res:
+                    self.runner.callbacks.on_async_failed(host, res, self.jid)
+                else:
+                    self.runner.callbacks.on_async_ok(host, res, self.jid)
+        for (host, res) in results['dark'].iteritems():
+            self.results['dark'][host] = res
+            poll_results['dark'][host] = res
+            self.runner.callbacks.on_async_failed(host, res, self.jid)
+
+        self.hosts_to_poll = hosts
+        if len(hosts)==0:
+            self.completed = True
+
+        return poll_results
+
+    def wait(self, seconds, poll_interval):
+        """ Wait a certain time for job completion, check status every poll_interval. """
+        clock = seconds - poll_interval
+        while (clock >= 0 and not self.completed):
+            time.sleep(poll_interval)
+
+            poll_results = self.poll()
+
+            for (host, res) in poll_results['polled'].iteritems():
+                if res.get('started'):
+                    self.runner.callbacks.on_async_poll(host, res, self.jid, clock)
+
+            clock = clock - poll_interval
+
+        return self.results

lib/ansible/utils.py

@@ -15,88 +15,57 @@
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
-###############################################################
-
 import sys
 import os
+import shlex
+import re
+import codecs
+import jinja2
+import yaml
+import optparse
+import operator
+from ansible import errors
+from ansible import color
+from ansible import __version__
+import ansible.constants as C
+
 try:
     import json
 except ImportError:
     import simplejson as json
 
+try:
+    from hashlib import md5 as _md5
+except ImportError: 
+    from md5 import md5 as _md5
+
 ###############################################################
 # UTILITY FUNCTIONS FOR COMMAND LINE TOOLS
 ###############################################################
 
 def err(msg):
     ''' print an error message to stderr '''
+
     print >> sys.stderr, msg
 
 def exit(msg, rc=1):
     ''' quit with an error to stdout and a failure code '''
+
     err(msg)
     sys.exit(rc)
 
-def bigjson(result):
-    ''' format JSON output (uncompressed) '''
-    return json.dumps(result, sort_keys=True, indent=4)
+def jsonify(result, format=False):
+    ''' format JSON output (uncompressed or uncompressed) '''
 
-def smjson(result):
-    ''' format JSON output (compressed) '''
-    return json.dumps(result, sort_keys=True)
-
-def task_start_msg(name, conditional):
-    if conditional:
-        return "NOTIFIED: [%s] **********\n" % name
+    result2 = result.copy()
+    if format:
+        return json.dumps(result2, sort_keys=True, indent=4)
     else:
-        return "TASK: [%s] *********\n" % name
-
-def regular_generic_msg(hostname, result, oneline, caption):
-    ''' output on the result of a module run that is not command '''
-    if not oneline:
-        return "%s | %s >>\n%s" % (hostname, caption, bigjson(result))
-    else:
-        return "%s | %s >> %s" % (hostname, caption, smjson(result))
-
-def regular_success_msg(hostname, result, oneline):
-    ''' output the result of a successful module run '''
-    return regular_generic_msg(hostname, result, oneline, 'success')
-
-def regular_failure_msg(hostname, result, oneline):
-    ''' output the result of a failed module run '''
-    return regular_generic_msg(hostname, result, oneline, 'FAILED')
-
-def command_generic_msg(hostname, result, oneline, caption):
-    ''' output the result of a command run '''
-    rc     = result.get('rc', '0')
-    stdout = result.get('stdout','')
-    stderr = result.get('stderr', '')
-    msg    = result.get('msg', '')
-    if not oneline:
-        buf = "%s | %s | rc=%s >>\n" % (hostname, caption, result.get('rc',0))
-        if stdout:
-            buf += stdout
-        if stderr:
-            buf += stderr
-        if msg:
-            buf += msg
-        return buf
-    else:
-        if stderr:
-            return "%s | %s | rc=%s | (stdout) %s (stderr) %s" % (hostname, caption, rc, stdout, stderr)
-        else:
-            return "%s | %s | rc=%s | (stdout) %s" % (hostname, caption, rc, stdout)
-
-def command_success_msg(hostname, result, oneline):
-    ''' output from a successful command run '''
-    return command_generic_msg(hostname, result, oneline, 'success')
-
-def command_failure_msg(hostname, result, oneline):
-    ''' output from a failed command run '''
-    return command_generic_msg(hostname, result, oneline, 'FAILED')
+        return json.dumps(result2, sort_keys=True)
 
 def write_tree_file(tree, hostname, buf):
     ''' write something into treedir/hostname '''
+
     # TODO: might be nice to append playbook runs per host in a similar way
     # in which case, we'd want append mode.
     path = os.path.join(tree, hostname)
@@ -106,55 +75,12 @@ def write_tree_file(tree, hostname, buf):
 
 def is_failed(result):
     ''' is a given JSON result a failed result? '''
-    failed = False
-    rc = 0
-    if type(result) == dict:
-        failed = result.get('failed', 0)
-        rc     = result.get('rc', 0)
-    if rc != 0:
-        return True    
-    return failed
 
-def host_report_msg(hostname, module_name, result, oneline):
-    ''' summarize the JSON results for a particular host '''
-    buf = ''
-    failed = is_failed(result)
-    if module_name == 'command':
-        if not failed:
-            buf = command_success_msg(hostname, result, oneline)
-        else:
-            buf = command_failure_msg(hostname, result, oneline)
-    else:
-        if not failed:
-            buf = regular_success_msg(hostname, result, oneline)
-        else:
-            buf = regular_failure_msg(hostname, result, oneline)
-    return buf
-
-def dark_hosts_msg(results):
-    ''' summarize the results of all uncontactable hosts '''
-    buf = ''
-    if len(results['dark'].keys()) > 0:
-        buf += "*** Hosts which could not be contacted or did not respond: ***"
-        for hostname in results['dark'].keys():
-            buf += "%s:\n%s\n" % (hostname, results['dark'][hostname])
-    buf += "\n"
-    return buf
-
-def has_dark_hosts(results):
-    ''' are there any uncontactable hosts? '''
-    return len(results['dark'].keys()) > 0
-
-def contacted_hosts(results):
-    ''' what are the contactable hosts? '''
-    return sorted(results['contacted'])
-
-def contacted_host_result(results, hostname):
-    ''' what are the results for a given host? '''
-    return results['contacted'][hostname]
+    return ((result.get('rc', 0) != 0) or (result.get('failed', False) in [ True, 'True', 'true']))
 
 def prepare_writeable_dir(tree):
     ''' make sure a directory exists and is writeable '''
+
     if tree != '/':
         tree = os.path.realpath(os.path.expanduser(tree))
     if not os.path.exists(tree):
@@ -166,6 +92,8 @@ def prepare_writeable_dir(tree):
         exit("Cannot write to path %s" % tree)
 
 def path_dwim(basedir, given):
+    ''' make relative paths work like folks expect '''
+
     if given.startswith("/"):
         return given
     elif given.startswith("~/"):
@@ -173,4 +101,236 @@ def path_dwim(basedir, given):
     else:
         return os.path.join(basedir, given)
 
+def json_loads(data):
+    ''' parse a JSON string and return a data structure '''
+
+    return json.loads(data)
+
+def parse_json(data):
+    ''' this version for module return data only '''
+
+    try:
+        return json.loads(data)
+    except:
+        # not JSON, but try "Baby JSON" which allows many of our modules to not
+        # require JSON and makes writing modules in bash much simpler
+        results = {}
+        try:
+            tokens = shlex.split(data)
+        except: 
+            print "failed to parse json: "+ data
+            raise 
+            
+        for t in tokens:
+            if t.find("=") == -1:
+                raise errors.AnsibleError("failed to parse: %s" % data)
+            (key,value) = t.split("=", 1)
+            if key == 'changed' or 'failed':
+                if value.lower() in [ 'true', '1' ]:
+                    value = True
+                elif value.lower() in [ 'false', '0' ]:
+                    value = False
+            if key == 'rc':
+                value = int(value)     
+            results[key] = value
+        if len(results.keys()) == 0:
+            return { "failed" : True, "parsed" : False, "msg" : data }
+        return results
+
+_LISTRE = re.compile(r"(\w+)\[(\d+)\]")
+
+def _varLookup(name, vars):
+    ''' find the contents of a possibly complex variable in vars. '''
+
+    path = name.split('.')
+    space = vars
+    for part in path:
+        if part in space:
+            space = space[part]
+        elif "[" in part:
+            m = _LISTRE.search(part)
+            if not m:
+                return
+            try:
+                space = space[m.group(1)][int(m.group(2))]
+            except (KeyError, IndexError):
+                return
+        else:
+            return
+    return space
+
+_KEYCRE = re.compile(r"\$(?P<complex>\{){0,1}((?(complex)[\w\.\[\]]+|\w+))(?(complex)\})")
+
+def varLookup(varname, vars):
+    ''' helper function used by varReplace '''
+
+    m = _KEYCRE.search(varname)
+    if not m:
+        return None
+    return _varLookup(m.group(2), vars)
+
+def varReplace(raw, vars):
+    ''' Perform variable replacement of $variables in string raw using vars dictionary '''
+    # this code originally from yum
+
+    done = [] # Completed chunks to return
+
+    while raw:
+        m = _KEYCRE.search(raw)
+        if not m:
+            done.append(raw)
+            break
+
+        # Determine replacement value (if unknown variable then preserve
+        # original)
+        varname = m.group(2)
+
+        replacement = unicode(_varLookup(varname, vars) or m.group())
+
+        start, end = m.span()
+        done.append(raw[:start])    # Keep stuff leading up to token
+        done.append(replacement)    # Append replacement value
+        raw = raw[end:]             # Continue with remainder of string
+
+    return ''.join(done)
+
+def template(text, vars):
+    ''' run a text buffer through the templating engine until it no longer changes '''
+
+    prev_text = ''
+    depth = 0
+    while prev_text != text:
+        depth = depth + 1
+        if (depth > 20):
+            raise errors.AnsibleError("template recursion depth exceeded")
+        prev_text = text
+        text = varReplace(unicode(text), vars) 
+    return text
+
+def template_from_file(basedir, path, vars):
+    ''' run a file through the templating engine '''
+
+    environment = jinja2.Environment(loader=jinja2.FileSystemLoader(basedir), trim_blocks=False)
+    data = codecs.open(path_dwim(basedir, path), encoding="utf8").read()
+    t = environment.from_string(data)
+    vars = vars.copy()
+    res = t.render(vars)
+    if data.endswith('\n') and not res.endswith('\n'):
+        res = res + '\n'
+    return template(res, vars)
+
+def parse_yaml(data):
+    ''' convert a yaml string to a data structure '''
+
+    return yaml.load(data)
   
+def parse_yaml_from_file(path):
+    ''' convert a yaml file to a data structure '''
+
+    try:
+        data = file(path).read()
+    except IOError:
+        raise errors.AnsibleError("file not found: %s" % path)
+    return parse_yaml(data)
+
+def parse_kv(args):
+    ''' convert a string of key/value items to a dict '''
+
+    options = {}
+    if args is not None:
+        # attempting to split a unicode here does bad things
+        vargs = shlex.split(str(args), posix=True)
+        for x in vargs:
+            if x.find("=") != -1:
+                k, v = x.split("=",1)
+                options[k]=v
+    return options
+
+def md5(filename):
+    ''' Return MD5 hex digest of local file, or None if file is not present. '''
+
+    if not os.path.exists(filename):
+        return None
+    digest = _md5()
+    blocksize = 64 * 1024
+    infile = open(filename, 'rb')
+    block = infile.read(blocksize)
+    while block:
+        digest.update(block)
+        block = infile.read(blocksize)
+    infile.close()
+    return digest.hexdigest()
+
+def default(value, function):
+    ''' syntactic sugar around lazy evaluation of defaults '''
+    if value is None:
+        return function()
+    return value
+
+####################################################################
+# option handling code for /usr/bin/ansible and ansible-playbook 
+# below this line
+
+class SortedOptParser(optparse.OptionParser):
+    '''Optparser which sorts the options by opt before outputting --help'''
+
+    def format_help(self, formatter=None):
+        self.option_list.sort(key=operator.methodcaller('get_opt_string'))
+        return optparse.OptionParser.format_help(self, formatter=None)
+
+def base_parser(constants=C, usage="", output_opts=False, runas_opts=False, async_opts=False, connect_opts=False):
+    ''' create an options parser for any ansible script '''
+
+    parser = SortedOptParser(usage, version="%prog " + __version__)
+    parser.add_option('-v','--verbose', default=False, action="store_true",
+        help='verbose mode')
+    parser.add_option('-f','--forks', dest='forks', default=constants.DEFAULT_FORKS, type='int',
+        help="specify number of parallel processes to use (default=%s)" % constants.DEFAULT_FORKS)
+    parser.add_option('-i', '--inventory-file', dest='inventory',
+        help="specify inventory host file (default=%s)" % constants.DEFAULT_HOST_LIST, 
+        default=constants.DEFAULT_HOST_LIST)
+    parser.add_option('-k', '--ask-pass', default=False, dest='ask_pass', action='store_true',
+        help='ask for SSH password')
+    parser.add_option('--private-key', default=None, dest='private_key_file',
+        help='use this file to authenticate the connection')
+    parser.add_option('-K', '--ask-sudo-pass', default=False, dest='ask_sudo_pass', action='store_true',
+        help='ask for sudo password')
+    parser.add_option('-M', '--module-path', dest='module_path',
+        help="specify path(s) to module library (default=%s)" % constants.DEFAULT_MODULE_PATH,
+        default=constants.DEFAULT_MODULE_PATH)
+    parser.add_option('-T', '--timeout', default=constants.DEFAULT_TIMEOUT, type='int',
+        dest='timeout', 
+        help="override the SSH timeout in seconds (default=%s)" % constants.DEFAULT_TIMEOUT)
+
+    if output_opts:
+        parser.add_option('-o', '--one-line', dest='one_line', action='store_true',
+            help='condense output')
+        parser.add_option('-t', '--tree', dest='tree', default=None,
+            help='log output to this directory')
+
+    if runas_opts:
+        parser.add_option("-s", "--sudo", default=False, action="store_true",
+            dest='sudo', help="run operations with sudo (nopasswd)")
+        parser.add_option('-U', '--sudo-user', dest='sudo_user', help='desired sudo user (default=root)',
+            default=None)   # Can't default to root because we need to detect when this option was given
+        parser.add_option('-u', '--user', default=constants.DEFAULT_REMOTE_USER,
+            dest='remote_user', 
+            help='connect as this user (default=%s)' % constants.DEFAULT_REMOTE_USER)
+    
+    if connect_opts:
+        parser.add_option('-c', '--connection', dest='connection',
+                          choices=C.DEFAULT_TRANSPORT_OPTS,
+                          default=C.DEFAULT_TRANSPORT,
+                          help="connection type to use (default=%s)" % C.DEFAULT_TRANSPORT)
+
+    if async_opts:
+        parser.add_option('-P', '--poll', default=constants.DEFAULT_POLL_INTERVAL, type='int',
+            dest='poll_interval', 
+            help="set the poll interval if using -B (default=%s)" % constants.DEFAULT_POLL_INTERVAL)
+        parser.add_option('-B', '--background', dest='seconds', type='int', default=0,
+            help='run asynchronously, failing after X seconds (default=N/A)')
+
+    return parser
+
+
+

library/apt

@@ -0,0 +1,184 @@
+#!/usr/bin/python -tt
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Flowroute LLC
+# Written by Matthew Williams <matthew@flowroute.com>
+# Based on yum module written by Seth Vidal <skvidal at fedoraproject.org>
+#
+# This module is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This software is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this software.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import traceback
+# added to stave off future warnings about apt api
+import warnings;
+warnings.filterwarnings('ignore', "apt API not stable yet", FutureWarning)
+
+# APT related constants
+APT_PATH = "/usr/bin/apt-get"
+APT = "DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical %s" % APT_PATH
+
+def run_apt(command):
+    try:
+        cmd = subprocess.Popen(command, shell=True,
+            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        out, err = cmd.communicate()
+    except (OSError, IOError), e:
+        rc = 1
+        err = str(e)
+        out = ''
+    except:
+        rc = 1
+        err = traceback.format_exc()
+        out = ''
+    else:
+        rc = cmd.returncode
+        return rc, out, err
+
+def package_split(pkgspec):
+    parts = pkgspec.split('=')
+    if len(parts) > 1:
+        return parts[0], parts[1]
+    else:
+        return parts[0], None
+
+def package_status(m, pkgname, version, cache):
+    try:
+        pkg = cache[pkgname]
+    except KeyError:
+        m.fail_json(msg="No package matching '%s' is available" % pkgname)
+    if version:
+        try :
+            return pkg.is_installed and pkg.installed.version == version, False
+        except AttributeError:
+            #assume older version of python-apt is installed
+            return pkg.isInstalled and pkg.installedVersion == version, False
+    else:
+        try :
+            return pkg.is_installed, pkg.is_upgradable
+        except AttributeError:
+            #assume older version of python-apt is installed
+            return pkg.isInstalled, pkg.isUpgradable
+
+def install(m, pkgspec, cache, upgrade=False, default_release=None, install_recommends=True, force=False):
+    packages = ""
+    for package in pkgspec:
+        name, version = package_split(package)
+        installed, upgradable = package_status(m, name, version, cache)
+        if not installed or (upgrade and upgradable):
+            packages += "'%s' " % package
+    
+    if len(packages) != 0:
+        if force:
+            force_yes = '--force-yes'
+        else:
+            force_yes = ''
+
+        cmd = "%s --option Dpkg::Options::=--force-confold -q -y %s install %s" % (APT, force_yes,packages)
+        if default_release:
+            cmd += " -t '%s'" % (default_release,)
+        if not install_recommends:
+            cmd += " --no-install-recommends"
+
+        rc, out, err = run_apt(cmd)
+        if rc:
+            m.fail_json(msg="'apt-get install %s' failed: %s" % (packages, err))
+        else:
+            m.exit_json(changed=True)
+    else:
+        m.exit_json(changed=False)
+
+def remove(m, pkgspec, cache, purge=False):
+    packages = ""
+    for package in pkgspec:
+        name, version = package_split(package)
+        installed, upgradable = package_status(m, name, version, cache)
+        if installed:
+            packages += "'%s' " % package
+    
+    if len(packages) == 0:
+        m.exit_json(changed=False)
+    else:
+        purge = '--purge' if purge else ''
+        cmd = "%s -q -y %s remove %s" % (APT, purge,packages)
+        rc, out, err = run_apt(cmd)
+        if rc:
+            m.fail_json(msg="'apt-get remove %s' failed: %s" % (packages, err))
+        m.exit_json(changed=True)
+
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            state = dict(default='installed', choices=['installed', 'latest', 'removed']),
+            update_cache = dict(default='no', choices=['yes', 'no'], aliases=['update-cache']),
+            purge = dict(default='no', choices=['yes', 'no']),
+            package = dict(default=None, aliases=['pkg', 'name']),
+            default_release = dict(default=None, aliases=['default-release']),
+            install_recommends = dict(default='yes', aliases=['install-recommends'], choices=['yes', 'no']),
+            force = dict(default='no', choices=['yes', 'no'])
+        )
+    )
+
+    try:
+        import apt, apt_pkg
+    except:
+        module.fail_json("Could not import python modules: apt, apt_pkg. Please install python-apt package.")
+
+    if not os.path.exists(APT_PATH):
+        module.fail_json(msg="Cannot find apt-get")
+
+    p = module.params
+    if p['package'] is None and p['update_cache'] != 'yes':
+        module.fail_json(msg='pkg=name and/or update_cache=yes is required')
+
+    install_recommends = module.boolean(p['install_recommends'])
+
+    cache = apt.Cache()
+    if p['default_release']:
+        apt_pkg.config['APT::Default-Release'] = p['default_release']
+        # reopen cache w/ modified config
+        cache.open(progress=None)
+
+    if module.boolean(p['update_cache']):
+        cache.update()
+        cache.open(progress=None)
+        if p['package'] == None:
+            module.exit_json(changed=False)
+
+    force_yes = module.boolean(p['force'])
+    
+    packages = p['package'].split(',')
+    latest = p['state'] == 'latest' 
+    for package in packages:
+        if package.count('=') > 1:
+            module.fail_json(msg="invalid package spec: %s" % package)
+        if latest and '=' in package:
+            module.fail_json(msg='version number inconsistent with state=latest: %s' % package)
+
+    if p['state'] == 'latest':
+        install(module, packages, cache, upgrade=True,
+                default_release=p['default_release'],
+                install_recommends=install_recommends,
+                force=force_yes)
+    elif p['state'] == 'installed':
+        install(module, packages, cache, default_release=p['default_release'],
+                  install_recommends=install_recommends,force=force_yes)
+    elif p['state'] == 'removed':
+        remove(module, packages, cache, purge = module.boolean(p['purge']))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+

library/assemble

@@ -0,0 +1,87 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import os.path
+import shutil
+import tempfile
+
+# ===========================================
+# Support methods
+
+def assemble_from_fragments(path):
+    ''' assemble a file from a directory of fragments '''
+    assembled = []
+    for f in sorted(os.listdir(path)):
+        fragment = "%s/%s" % (path, f)
+        if os.path.isfile(fragment):
+            assembled.append(file(fragment).read())
+    return "".join(assembled)
+
+def write_temp_file(data):
+    fd, path = tempfile.mkstemp()
+    os.write(fd, data)
+    os.close(fd)
+    return path
+
+# ==============================================================
+# main
+
+def main():
+    
+    module = AnsibleModule(
+        argument_spec = dict(
+            src = dict(required=True),
+            dest = dict(required=True),
+        )
+    )
+    
+    changed=False
+    pathmd5 = None
+    destmd5 = None
+    src = os.path.expanduser(module.params['src'])
+    dest = os.path.expanduser(module.params['dest'])
+  
+    if not os.path.exists(src):
+        module.fail_json(msg="Source (%s) does not exist" % src)
+    
+    if not os.path.isdir(src):
+        module.fail_json(msg="Source (%s) is not a directory" % src)
+    
+    path = write_temp_file(assemble_from_fragments(src))
+    pathmd5 = module.md5(path)
+    
+    if os.path.exists(dest):
+        destmd5 = module.md5(dest)
+    
+    if pathmd5 != destmd5:
+        shutil.copy(path, dest)
+        changed = True
+    
+
+    # Mission complete
+    module.exit_json(src=src, dest=dest, md5sum=destmd5, 
+        changed=changed, msg="OK",
+        daisychain="file", daisychain_args=module.params)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()

library/async_status

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -18,81 +19,50 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import shlex
-import os
-import subprocess
-import sys
 import datetime
 import traceback
 
-# ===========================================
-# convert arguments of form a=b c=d
-# to a dictionary
-# FIXME: make more idiomatic
+def main():
 
-args = " ".join(sys.argv[1:])
-items = shlex.split(args)
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
+    module = AnsibleModule(argument_spec=dict(
+        jid=dict(required=True),
+        mode=dict(default='status', choices=['status','cleanup']),
+    ))
 
-mode = params.get('mode', 'status')
-jid  = params.get('jid', None)
+    mode = module.params['mode']
+    jid  = module.params['jid']
 
-# ===========================================
+    # setup logging directory
+    logdir = os.path.expanduser("~/.ansible_async")
+    log_path = os.path.join(logdir, jid)
 
-if jid is None:
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "jid=INTEGER is required"
-    })
-    sys.exit(1)
+    if not os.path.exists(log_path):
+        module.fail_json(msg="could not find job", ansible_job_id=jid)
 
+    if mode == 'cleanup':
+        os.unlink(log_path)
+        module.exit_json(ansible_job_id=jid, erased=log_path)
 
-# setup logging directory
-logdir = os.path.expanduser("~/.ansible_async")
-log_path = os.path.join(logdir, jid)
+    # NOT in cleanup mode, assume regular status mode
+    # no remote kill mode currently exists, but probably should
+    # consider log_path + ".pid" file and also unlink that above
 
-if not os.path.exists(log_path):
-    print json.dumps({
-        "failed" : 1,
-        "msg" : "could not find job",
-        "ansible_job_id" : jid
-    })
-    sys.exit(1)
-
-if mode == 'cleanup':
-    os.unlink(log_path)
-    print json.dumps({
-        "ansible_job_id" : jid,
-        "erased" : log_path
-    })
-    sys.exit(0)
-
-# NOT in cleanup mode, assume regular status mode
-# no remote kill mode currently exists, but probably should
-# consider log_path + ".pid" file and also unlink that above
-
-data = file(log_path).read()
-try:
-   data = json.loads(data)
-except:
-   print json.dumps({
-       "failed" : True,
-       "msg"    : "Could not parse job output"
-   })
-   sys.exit(1)
-
-if not data.has_key("started"):
-   data['finished'] = 1
-data['ansible_job_id'] = jid
-
-print json.dumps(data)
-sys.exit(0)
+    data = file(log_path).read()
+    try:
+        data = json.loads(data)
+    except Exception, e:
+        if data == '':
+            # file not written yet?  That means it is running
+            module.exit_json(results_file=log_path, ansible_job_id=jid, started=1)
+        else:
+            module_fail_json(ansible_job_id=jid, results_file=log_path, 
+                msg="Could not parse job output: %s" % data)
 
+    if not data.has_key("started"):
+        data['finished'] = 1
+        data['ansible_job_id'] = jid
+    module.exit_json(**data)
 
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/async_wrapper

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -30,20 +31,55 @@ import datetime
 import traceback
 import signal
 import time
+import syslog
+
+def daemonize_self():
+    # daemonizing code: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012
+    # logger.info("cobblerd started")
+    try:
+        pid = os.fork()
+        if pid > 0:
+            # exit first parent
+            sys.exit(0)
+    except OSError, e:
+        print >>sys.stderr, "fork #1 failed: %d (%s)" % (e.errno, e.strerror)
+        sys.exit(1)
+
+    # decouple from parent environment
+    os.chdir("/")
+    os.setsid()
+    os.umask(022)
+
+    # do second fork
+    try:
+        pid = os.fork()
+        if pid > 0:
+            # print "Daemon PID %d" % pid
+            sys.exit(0)
+    except OSError, e:
+        print >>sys.stderr, "fork #2 failed: %d (%s)" % (e.errno, e.strerror)
+        sys.exit(1)
+
+    dev_null = file('/dev/null','rw')
+    os.dup2(dev_null.fileno(), sys.stdin.fileno())
+    os.dup2(dev_null.fileno(), sys.stdout.fileno())
+    os.dup2(dev_null.fileno(), sys.stderr.fileno()) 
 
 if len(sys.argv) < 3:
     print json.dumps({
         "failed" : True,
-        "msg"    : "usage: async_wrapper <jid> <module_script> <time_limit> <args>.  Humans, do not call directly!"
+        "msg"    : "usage: async_wrapper <jid> <time_limit> <modulescript> <argsfile>.  Humans, do not call directly!"
     })
     sys.exit(1)
 
 jid = sys.argv[1]
 time_limit = sys.argv[2]
 wrapped_module = sys.argv[3]
-args = sys.argv[4:]
+argsfile = sys.argv[4]
+cmd = "%s %s" % (wrapped_module, argsfile)
 
-cmd = "%s %s" % (wrapped_module, " ".join(args))
+syslog.openlog('ansible-%s' % os.path.basename(__file__))
+syslog.syslog(syslog.LOG_NOTICE, 'Invoked with %s' % " ".join(sys.argv[1:]))
 
 # setup logging directory
 logdir = os.path.expanduser("~/.ansible_async")
@@ -60,65 +96,105 @@ if not os.path.exists(logdir):
 
 def _run_command(wrapped_cmd, jid, log_path):
 
-    logfile = open(log_path, "w+")
+    logfile = open(log_path, "w")
     logfile.write(json.dumps({ "started" : 1, "ansible_job_id" : jid }))
+    logfile.close()
+    logfile = open(log_path, "w")
     result = {}
-
+  
+    outdata = ''
     try:
         cmd = shlex.split(wrapped_cmd)
         script = subprocess.Popen(cmd, shell=False, 
-            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-        out, err = script.communicate()
-        result = json.loads(out)
-    
+            stdin=None, stdout=logfile, stderr=logfile)
+        script.communicate()
+        outdata = file(log_path).read()
+        result = json.loads(outdata)
+
     except (OSError, IOError), e:
         result = {
             "failed": 1,
             "cmd" : wrapped_cmd,
             "msg": str(e),
         }
+        result['ansible_job_id'] = jid
+        logfile.write(json.dumps(result))
     except:
         result = {
             "failed" : 1,
             "cmd" : wrapped_cmd,
+            "data" : outdata, # temporary debug only
             "msg" : traceback.format_exc()
         }   
-
-    result['ansible_job_id'] = jid
-    logfile = open(log_path, "w+")
-    logfile.write(json.dumps(result))
+        result['ansible_job_id'] = jid
+        logfile.write(json.dumps(result))
     logfile.close()
 
 # immediately exit this process, leaving an orphaned process
 # running which immediately forks a supervisory timing process
 
-pid = os.fork()
-if pid == 0:
-   # "RETURNING SUCCESS IN UNO"
-   print json.dumps({ "started" : 1, "ansible_job_id" : jid })
-   sys.exit(0)
-else:
-   # "DAEMONIZED DOS"
-   sub_pid = os.fork()
-   if sub_pid == 0:
-       # "RUNNING IN KID A"
-       _run_command(cmd, jid, log_path)
-       sys.exit(0)
-   else:
-       # "WATCHING IN KID B"
-       remaining = int(time_limit)
-       if os.path.exists("/proc/%s" % sub_pid):
-           # "STILL RUNNING"
-           time.sleep(1)
-           remaining = remaining - 1
-       else:
-           # "DONE IN KID B"
-           sys.exit(0)
-       if remaining == 0:
-           # "SLAYING IN KID B"
-           os.kill(sub_pid, signals.SIGKILL)
-           sys.exit(1)
+#import logging
+#import logging.handlers
 
-   sys.exit(0)
+#logger = logging.getLogger("ansible_async")
+#logger.setLevel(logging.WARNING)
+#logger.addHandler( logging.handlers.SysLogHandler("/dev/log") )
+def debug(msg):
+    #logger.warning(msg)
+    pass
 
+try:
+    pid = os.fork()
+    if pid:
+        # Notify the overlord that the async process started
 
+        # we need to not return immmediately such that the launched command has an attempt
+        # to initialize PRIOR to ansible trying to clean up the launch directory (and argsfile)
+        # this probably could be done with some IPC later.  Modules should always read
+        # the argsfile at the very first start of their execution anyway
+        time.sleep(1)
+        debug("Return async_wrapper task started.")
+        print json.dumps({ "started" : 1, "ansible_job_id" : jid, "results_file" : log_path })
+        sys.stdout.flush()
+        sys.exit(0)
+    else:
+        # The actual wrapper process
+
+        # Daemonize, so we keep on running
+        daemonize_self()
+
+        # we are now daemonized, create a supervisory process
+        debug("Starting module and watcher")
+
+        sub_pid = os.fork()
+        if sub_pid:
+            # the parent stops the process after the time limit
+            remaining = int(time_limit)
+
+            # set the child process group id to kill all children
+            os.setpgid(sub_pid, sub_pid)
+
+            debug("Start watching %s (%s)"%(sub_pid, remaining))
+            time.sleep(5)
+            while os.waitpid(sub_pid, os.WNOHANG) == (0, 0):
+                debug("%s still running (%s)"%(sub_pid, remaining))
+                time.sleep(5)
+                remaining = remaining - 5
+                if remaining == 0:
+                    debug("Now killing %s"%(sub_pid))
+                    os.killpg(sub_pid, signal.SIGKILL)
+                    debug("Sent kill to group %s"%sub_pid)
+                    time.sleep(1)
+                    sys.exit(0)
+            debug("Done in kid B.")
+            os._exit(0)
+        else:
+            # the child process runs the actual module
+            debug("Start module (%s)"%os.getpid())
+            _run_command(cmd, jid, log_path)
+            debug("Module complete (%s)"%os.getpid())
+            sys.exit(0)
+
+except Exception, err:
+    debug("error: %s"%(err))
+    raise err

library/authorized_key

@@ -0,0 +1,136 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Ansible module to add authorized_keys for ssh logins.
+(c) 2012, Brad Olson <brado@movedbylight.com>
+
+This file is part of Ansible
+
+Ansible is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+Ansible is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+"""
+
+# Makes sure the public key line is present or absent in the user's .ssh/authorized_keys.
+#
+# Arguments
+# =========
+#    user = username
+#    key = line to add to authorized_keys for user
+#    state = absent|present (default: present)
+#
+# see example in examples/playbooks
+
+import sys
+import os
+import pwd
+import os.path
+
+def keyfile(user, write=False):
+    """
+    Calculate name of authorized keys file, optionally creating the 
+    directories and file, properly setting permissions.
+
+    :param str user: name of user in passwd file
+    :param bool write: if True, write changes to authorized_keys file (creating directories if needed)
+    :return: full path string to authorized_keys for user
+    """
+
+    user_entry = pwd.getpwnam(user)
+    homedir    = user_entry.pw_dir
+    sshdir     = os.path.join(homedir, ".ssh")
+    keysfile   = os.path.join(sshdir, "authorized_keys")
+
+    if not write: 
+        return keysfile
+
+    uid = user_entry.pw_uid
+    gid = user_entry.pw_gid
+
+    if not os.path.exists(sshdir): 
+        os.mkdir(sshdir, 0700)
+    os.chown(sshdir, uid, gid)
+    os.chmod(sshdir, 0700)
+
+    if not os.path.exists( keysfile):
+        try:
+            f = open(keysfile, "w") #touches file so we can set ownership and perms
+        finally:
+            f.close()
+
+    os.chown(keysfile, uid, gid)
+    os.chmod(keysfile, 0600)
+    return keysfile
+
+def readkeys(filename):
+
+    if not os.path.isfile(filename): 
+        return []
+    f = open(filename)
+    keys = [line.rstrip() for line in f.readlines()]
+    f.close()
+    return keys
+
+def writekeys( filename, keys):
+
+    f = open(filename,"w")
+    f.writelines( (key + "\n" for key in keys) )
+    f.close()
+
+def enforce_state(module, params):
+    """
+    Add or remove key.
+    """
+
+    user  = params["user"]
+    key   = params["key"]
+    state = params.get("state", "present")
+
+    # check current state -- just get the filename, don't create file
+    params["keyfile"] = keyfile(user, write=False) 
+    keys = readkeys(params["keyfile"])
+    present = key in keys
+
+    # handle idempotent state=present
+    if state=="present":
+        if present: 
+            module.exit_json(changed=False)
+        keys.append(key)
+        writekeys(keyfile(user,write=True), keys)
+
+    elif state=="absent":
+        if not present: 
+            module.exit_json(changed=False)
+        keys.remove(key)
+        writekeys(keyfile(user,write=True), keys)
+
+    params['changed'] = True
+    return params
+
+def main():
+
+    module = AnsibleModule(
+        argument_spec = dict(
+           user  = dict(required=True),
+           key   = dict(required=True),
+           state = dict(default='present', choices=['absent','present'])
+        )
+    )
+
+    params = module.params
+    results = enforce_state(module, module.params)
+    module.exit_json(**results)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main() 

library/command

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>, and others
 #
@@ -16,60 +17,109 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-try:
-    import json
-except ImportError:
-    import simplejson as json
 
 import subprocess
 import sys
 import datetime
 import traceback
+import shlex
+import os
 
-if len(sys.argv) == 1:
-    print json.dumps({
-        "failed" : True,
-        "msg"    : "the command module requires arguments (-a)"
-    })
-    sys.exit(1)
+def main():
 
-args = sys.argv[1:]
-startd = datetime.datetime.now()
+    # the command module is the one ansible module that does not take key=value args
+    # hence don't copy this one if you are looking to build others!
+    module = CommandModule(argument_spec=dict())
 
-try:
-    cmd = subprocess.Popen(args, shell=False, 
-        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-    out, err = cmd.communicate()
-except (OSError, IOError), e:
-    print json.dumps({
-        "failed": 1,
-        "msg": str(e),
-        })
-    sys.exit(1)
-except:
-    print json.dumps({
-        "failed" : 1,
-        "msg" : traceback.format_exc()
-    })   
-    sys.exit(1)
+    shell = module.params['shell']
+    chdir = module.params['chdir']
+    args  = module.params['args']
 
-endd = datetime.datetime.now()
-delta = endd - startd
+    if chdir:
+        os.chdir(chdir)
 
-if out is None:
-   out = ''
-if err is None:
-   err = ''
+    if not shell:
+        args = shlex.split(args)
+    startd = datetime.datetime.now()
 
-result = {
-   "stdout" : out.strip(),
-   "stderr" : err.strip(),
-   "rc"     : cmd.returncode,
-   "start"  : str(startd),
-   "end"    : str(endd),
-   "delta"  : str(delta),
-}
+    try:
+        cmd = subprocess.Popen(args, shell=shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+        out, err = cmd.communicate()
+    except (OSError, IOError), e:
+        module.fail_json(cmd=args, msg=str(e))
+    except:
+        module.fail_json(msg=traceback.format_exc())
 
-print json.dumps(result)
+    endd = datetime.datetime.now()
+    delta = endd - startd
+
+    if out is None:
+       out = ''
+    if err is None:
+       err = ''
+
+    module.exit_json(
+        cmd     = args,
+        stdout  = out.strip(),
+        stderr  = err.strip(),
+        rc      = cmd.returncode,
+        start   = str(startd),
+        end     = str(endd),
+        delta   = str(delta),
+        changed = True
+    )
+
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+# only the command module should ever need to do this
+# everything else should be simple key=value
+
+class CommandModule(AnsibleModule):
+
+    def _handle_aliases(self):
+        pass
+
+    def _check_invalid_arguments(self):
+        pass
+
+    def _load_params(self):
+        ''' read the input and return a dictionary and the arguments string '''
+        args = MODULE_ARGS
+        items   = shlex.split(args)
+        params = {}
+        params['chdir'] = None
+        params['shell'] = False
+        if args.find("#USE_SHELL") != -1:
+             args = args.replace("#USE_SHELL", "")
+             params['shell'] = True
+
+        check_args = shlex.split(args)
+        for x in check_args:
+            if x.startswith("creates="):
+                # do not run the command if the line contains creates=filename
+                # and the filename already exists.  This allows idempotence
+                # of command executions.
+                (k,v) = x.split("=",1)
+                if os.path.exists(v):
+                    self.exit_json(
+                        cmd=args,
+                        stdout="skipped, since %s exists" % v,
+                        skipped=True,
+                        changed=False,
+                        stderr=False,
+                        rc=0
+                    )
+                args = args.replace(x,'')
+            elif x.startswith("chdir="):
+                (k,v) = x.split("=", 1)
+                if not (os.path.exists(v) and os.path.isdir(v)):
+                    self.fail_json(msg="cannot change to directory '%s': path does not exist" % v)
+                elif v[0] != '/':
+                    self.fail_json(msg="the path for 'chdir' argument must be fully qualified")
+                params['chdir'] = v
+                args = args.replace(x, '')
+        params['args'] = args
+        return (params, args)
+
+main()

library/copy

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -16,57 +17,58 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
 
-import sys
 import os
-import shlex
+import shutil
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
+def main():
 
-# ===========================================
-# convert arguments of form a=b c=d
-# to a dictionary
-# FIXME: make more idiomatic
+    module = AnsibleModule(
+        # not checking because of daisy chain to file module
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            src=dict(required=True),
+            dest=dict(required=True)
+        )
+    ) 
 
-args = " ".join(sys.argv[1:])
-items = shlex.split(args)
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
+    src  = os.path.expanduser(module.params['src'])
+    dest = os.path.expanduser(module.params['dest'])
 
-src  = params['src']
-dest = params['dest']
+    if not os.path.exists(src):
+        module.fail_json(msg="Source %s failed to transfer" % (src))
+    if not os.access(src, os.R_OK):
+        module.fail_json(msg="Source %s not readable" % (src))
 
- 
-# raise an error if there is no src file
-if not os.path.exists(src):
-    print json.dumps({
-        "failed" : 1,
-        "msg"    : "Source %s failed to transfer" % src
-    })
-    sys.exit(1)
+    md5sum_src = module.md5(src)
+    md5sum_dest = None
 
-md5sum = None
-changed = False
-if os.path.exists(dest):
-    md5sum = os.popen("md5sum %s" % dest).read().split()[0]
+    if os.path.exists(dest):
+        if not os.access(dest, os.W_OK):
+            module.fail_json(msg="Destination %s not writable" % (dest))
+        if not os.access(dest, os.R_OK):
+            module.fail_json(msg="Destination %s not readable" % (dest))
+        # Allow dest to be directory without compromising md5 check
+        if (os.path.isdir(dest)):
+            module.fail_json(msg="Destination %s cannot be a directory" % (dest))
+        md5sum_dest = module.md5(dest)
+    else:
+        if not os.access(os.path.dirname(dest), os.W_OK):
+            module.fail_json(msg="Destination %s not writable" % (os.path.dirname(dest)))
 
-os.system("cp %s %s" % (src, dest))
-
-md5sum2 = os.popen("md5sum %s" % dest).read().split()[0]
-
-if md5sum != md5sum2:
-    changed = True
-
-# mission accomplished
-print json.dumps({
-   "md5sum"   : md5sum2,
-   "changed"  : changed 
-})
+    if md5sum_src != md5sum_dest:
+        try:
+            shutil.copyfile(src, dest)
+        except shutil.Error:
+            module.fail_json(msg="failed to copy: %s and %s are the same" % (src, dest)) 
+        except IOError:
+            module.fail_json(msg="failed to copy: %s to %s" % (src, dest)) 
+        changed = True
+    else:
+        changed = False
 
+    module.exit_json(dest=dest, src=src, md5sum=md5sum_src, changed=changed)
 
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/facter

@@ -1,4 +1,5 @@
-#!/bin/bash
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -22,4 +23,28 @@
 #    facter
 #    ruby-json
 
-/usr/bin/facter --json
+import subprocess
+
+def get_facter_data():
+  p = subprocess.Popen(["/usr/bin/env", "facter", "--json"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+  (out, err) = p.communicate()
+  rc = p.returncode
+  return rc, out, err
+
+
+def main():
+  module = AnsibleModule(
+      argument_spec = dict()
+  )
+
+  rc, out, err = get_facter_data()
+  if rc != 0:
+    module.fail_json(msg=err)
+  else:
+    module.exit_json(**json.loads(out))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+

library/fetch

@@ -0,0 +1 @@
+# this is a virtual module that is entirely implemented server side

library/file

@@ -0,0 +1,354 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import shutil
+import stat
+import grp
+import pwd
+try:
+    import selinux
+    HAVE_SELINUX=True
+except ImportError:
+    HAVE_SELINUX=False
+
+def add_path_info(kwargs):
+    path = kwargs['path']
+    if os.path.exists(path):
+        (user, group) = user_and_group(path)
+        kwargs['owner']  = user
+        kwargs['group'] = group
+        st = os.stat(path)
+        kwargs['mode']  = oct(stat.S_IMODE(st[stat.ST_MODE]))
+        # secontext not yet supported
+        if os.path.islink(path):
+            kwargs['state'] = 'link'
+        elif os.path.isfile(path):
+            kwargs['state'] = 'file'
+        else:
+            kwargs['state'] = 'directory'
+        if HAVE_SELINUX and selinux_enabled():
+            kwargs['secontext'] = ':'.join(selinux_context(path))
+    else:
+        kwargs['state'] = 'absent'
+    return kwargs 
+
+def module_exit_json(**kwargs):
+    add_path_info(kwargs)
+    module.exit_json(**kwargs)
+
+def module_fail_json(**kwargs):
+    add_path_info(kwargs)
+    module.fail_json(**kwargs)
+
+# Detect whether using selinux that is MLS-aware.
+# While this means you can set the level/range with
+# selinux.lsetfilecon(), it may or may not mean that you
+# will get the selevel as part of the context returned
+# by selinux.lgetfilecon().
+def selinux_mls_enabled():
+    if not HAVE_SELINUX:
+        return False
+    if selinux.is_selinux_mls_enabled() == 1:
+        return True
+    else:
+        return False
+
+def selinux_enabled():
+    if not HAVE_SELINUX:
+        return False
+    if selinux.is_selinux_enabled() == 1:
+        return True
+    else:
+        return False
+
+# Determine whether we need a placeholder for selevel/mls
+def selinux_initial_context():
+    context = [None, None, None]
+    if selinux_mls_enabled():
+        context.append(None)
+    return context
+
+# If selinux fails to find a default, return an array of None
+def selinux_default_context(path, mode=0):
+    context = selinux_initial_context()
+    if not HAVE_SELINUX or not selinux_enabled():
+        return context
+    try:
+        ret = selinux.matchpathcon(path, mode)
+    except OSError:
+        return context
+    if ret[0] == -1:
+        return context
+    context = ret[1].split(':')
+    return context
+
+def selinux_context(path):
+    context = selinux_initial_context()
+    if not HAVE_SELINUX or not selinux_enabled():
+        return context
+    try:
+        ret = selinux.lgetfilecon(path)
+    except:
+        module_fail_json(path=path, msg='failed to retrieve selinux context')
+    if ret[0] == -1:
+        return context
+    context = ret[1].split(':')
+    return context
+
+# ===========================================
+
+
+
+
+
+
+# ===========================================
+# support functions
+
+def user_and_group(filename):
+    st = os.stat(filename)
+    uid = st.st_uid
+    gid = st.st_gid
+    try:
+        user = pwd.getpwuid(uid)[0]
+    except KeyError:
+        user = str(uid)
+    try:
+        group = grp.getgrgid(gid)[0]
+    except KeyError:
+        group = str(gid)
+    return (user, group)
+
+def set_context_if_different(path, context, changed):
+    if not HAVE_SELINUX or not selinux_enabled():
+        return changed
+    cur_context = selinux_context(path)
+    new_context = list(cur_context)
+    # Iterate over the current context instead of the
+    # argument context, which may have selevel.
+    for i in range(len(cur_context)):
+        if context[i] is not None and context[i] != cur_context[i]:
+            new_context[i] = context[i]
+    if cur_context != new_context:
+        try:
+            rc = selinux.lsetfilecon(path, ':'.join(new_context))
+        except OSError:
+            module_fail_json(path=path, msg='invalid selinux context')
+        if rc != 0:
+            module_fail_json(path=path, msg='set selinux context failed')
+        changed = True
+    return changed
+    
+def set_owner_if_different(path, owner, changed):
+   if owner is None:
+       return changed
+   user, group = user_and_group(path)
+   if owner != user:
+       rc = os.system("/bin/chown -R %s %s 2>/dev/null" % (owner, path))
+       if rc != 0:
+           module_fail_json(path=path, msg='chown failed')
+       return True
+
+   return changed
+    
+def set_group_if_different(path, group, changed):
+   if group is None:
+       return changed
+   old_user, old_group = user_and_group(path)
+   if old_group != group:
+       rc = os.system("/bin/chgrp -R %s %s" % (group, path))
+       if rc != 0:
+           module_fail_json(path=path, msg='chgrp failed')
+       return True
+   return changed
+
+def set_mode_if_different(path, mode, changed):
+   if mode is None:
+       return changed
+   try:
+       # FIXME: support English modes
+       mode = int(mode, 8)
+   except Exception, e:
+       module_fail_json(path=path, msg='mode needs to be something octalish', details=str(e))  
+ 
+   st = os.stat(path)
+   prev_mode = stat.S_IMODE(st[stat.ST_MODE])
+
+   if prev_mode != mode:
+       # FIXME: comparison against string above will cause this to be executed
+       # every time
+       try:
+           os.chmod(path, mode)
+       except Exception, e:
+           module_fail_json(path=path, msg='chmod failed', details=str(e))
+
+       st = os.stat(path)
+       new_mode = stat.S_IMODE(st[stat.ST_MODE])
+
+       if new_mode != prev_mode:
+           return True
+   return changed
+
+
+def rmtree_error(func, path, exc_info):
+    module_fail_json(path=path, msg='failed to remove directory')
+
+def main():
+
+    global module
+    module = AnsibleModule(
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            state = dict(choices=['file','directory','link','absent'], default='file'),
+            path  = dict(aliases=['dest', 'name'], required=True),
+            src   = dict(),
+            mode  = dict(),
+            owner = dict(),
+            group = dict(),
+            seuser = dict(),
+            serole = dict(),
+            selevel = dict(),
+            setype = dict(),
+        )
+    )
+
+    params = module.params
+    state  = params['state']
+    path   = os.path.expanduser(params['path'])
+    src    = params.get('src', None)
+    if src:
+        src = os.path.expanduser(src)
+
+    mode   = params.get('mode', None)
+    owner  = params.get('owner', None)
+    group  = params.get('group', None)
+
+    # selinux related options
+    seuser    = params.get('seuser', None)
+    serole    = params.get('serole', None)
+    setype    = params.get('setype', None)
+    selevel   = params.get('serange', 's0')
+    secontext = [seuser, serole, setype]
+    if selinux_mls_enabled():
+        secontext.append(selevel)
+
+    default_secontext = selinux_default_context(path)
+    for i in range(len(default_secontext)):
+        if i is not None and secontext[i] == '_default':
+            secontext[i] = default_secontext[i]
+
+    if state == 'link' and (src is None or path is None):
+        module_fail_json(msg='src and dest are required for "link" state')
+    elif path is None:
+        module_fail_json(msg='path is required')
+
+    changed = False
+
+    prev_state = 'absent'
+    if os.path.lexists(path):
+        if os.path.islink(path):
+            prev_state = 'link'
+        elif os.path.isfile(path):
+            prev_state = 'file'
+        else:
+            prev_state = 'directory'
+
+    if prev_state != 'absent' and state == 'absent':
+        try:
+            if prev_state == 'directory':
+                if os.path.islink(path):
+                    os.unlink(path)
+                else:
+                    shutil.rmtree(path, ignore_errors=False, onerror=rmtree_error)
+            else:
+                os.unlink(path)
+        except Exception, e:
+            module_fail_json(path=path, msg=str(e))
+        module_exit_json(path=path, changed=True)
+
+    if prev_state != 'absent' and prev_state != state:
+        module_fail_json(path=path, msg='refusing to convert between %s and %s' % (prev_state, state))
+
+    if prev_state == 'absent' and state == 'absent':
+        module_exit_json(path=path, changed=False)
+
+    if state == 'file':
+ 
+        if prev_state == 'absent':
+            module_fail_json(path=path, msg='file does not exist, use copy or template module to create')
+
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
+
+        module_exit_json(path=path, changed=changed)
+
+    elif state == 'directory':
+
+        if prev_state == 'absent':
+            os.makedirs(path)
+            changed = True
+ 
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
+
+        module_exit_json(path=path, changed=changed)
+
+    elif state == 'link':
+    
+        if os.path.isabs(src):
+            abs_src = src
+        else:
+            abs_src = os.path.join(os.path.dirname(dest), src)
+        if not os.path.exists(abs_src):
+            module_fail_json(path=path, src=src, msg='src file does not exist')
+   
+        if prev_state == 'absent':
+            os.symlink(src, path)
+            changed = True
+        elif prev_state == 'link':
+            old_src = os.readlink(path)
+            if not os.path.isabs(old_src):
+                old_src = os.path.join(os.path.dirname(path), old_src)
+            if old_src != src:
+                os.unlink(path)
+                os.symlink(src, path)
+        else:
+            module_fail_json(dest=path, src=src, msg='unexpected position reached')
+
+        # set modes owners and context as needed
+        changed = set_context_if_different(path, secontext, changed)
+        changed = set_owner_if_different(path, owner, changed)
+        changed = set_group_if_different(path, group, changed)
+        changed = set_mode_if_different(path, mode, changed)
+
+        module.exit_json(dest=path, src=src, changed=changed)
+
+    module_fail_json(path=path, msg='unexpected position reached')
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()
+

library/get_url

@@ -0,0 +1,194 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Jan-Piet Mens <jpmens () gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+# see examples/playbooks/get_url.yml
+
+import shutil
+import datetime
+import tempfile
+
+HAS_URLLIB2=True
+try:
+    import urllib2
+except ImportError: 
+    HAS_URLLIB2=False
+HAS_URLPARSE=True
+
+try:
+    import urlparse
+    import socket
+except ImportError: 
+    HAS_URLPARSE=False
+
+# ==============================================================
+# url handling
+
+def url_filename(url):
+    fn = os.path.basename(urlparse.urlsplit(url)[2])
+    if fn == '':
+        return 'index.html'
+    return fn
+
+def url_do_get(module, url, dest):
+    """
+    Get url and return request and info
+    Credits: http://stackoverflow.com/questions/7006574/how-to-download-file-from-ftp
+    """
+
+    USERAGENT = 'ansible-httpget'
+    info = dict(url=url)
+    r = None
+    actualdest = None
+
+    if os.path.isdir(dest):
+        urlfilename = url_filename(url)
+        actualdest = "%s/%s" % (dest, urlfilename)
+        module.params['path'] = actualdest
+    else:
+         actualdest = dest
+    info['daisychain_args'] = module.params
+    info['daisychain_args']['state'] = 'file'
+    info['daisychain_args']['dest'] = actualdest
+    info['actualdest'] = actualdest
+
+    request = urllib2.Request(url)
+    request.add_header('User-agent', USERAGENT)
+
+    if os.path.exists(actualdest):
+        t = datetime.datetime.utcfromtimestamp(os.path.getmtime(actualdest))
+        tstamp = t.strftime('%a, %d %b %Y %H:%M:%S +0000')
+        request.add_header('If-Modified-Since', tstamp)
+
+    try:
+        r = urllib2.urlopen(request)
+        info.update(r.info())
+        info.update(dict(msg="OK (%s bytes)" % r.headers.get('Content-Length', 'unknown'), status=200))
+    except urllib2.HTTPError, e:
+        # Must not fail_json() here so caller can handle HTTP 304 unmodified
+        info.update(dict(msg=str(e), status=e.code))
+        return r, info
+    except urllib2.URLError, e:
+        code = getattr(e, 'code', -1)
+        module.fail_json(msg="Request failed: %s" % str(e), status_code=code)
+
+    return r, info
+
+def url_get(module, url, dest):
+    """
+    Download url and store at dest. 
+    If dest is a directory, determine filename from url.
+    Return (tempfile, info about the request)
+    """
+
+    req, info = url_do_get(module, url, dest)
+
+    # TODO: should really handle 304, but how? src file could exist (and be newer) but empty
+    if info['status'] == 304:
+        module.exit_json(url=url, dest=info.get('actualdest', dest), changed=False, msg=info.get('msg', ''))
+
+    # create a temporary file and copy content to do md5-based replacement
+    if info['status'] != 200:
+        module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'], url=url)
+    actualdest = info['actualdest']
+
+    fd, tempname = tempfile.mkstemp()
+    f = os.fdopen(fd, 'wb')
+    try:
+        shutil.copyfileobj(req, f)
+    except Exception, err:
+        os.remove(tempname)
+        module.fail_json(msg="failed to create temporary content file: %s" % str(err))
+    f.close()
+    req.close()
+    return tempname, info
+
+# ==============================================================
+# main
+
+def main():
+    
+    # does this really happen on non-ancient python? 
+    if not HAS_URLLIB2:
+        module.fail_json(msg="urllib2 is not installed")
+    if not HAS_URLPARSE:
+        module.fail_json(msg="urlparse is not installed")
+
+    module = AnsibleModule(
+        # not checking because of daisy chain to file module
+        check_invalid_arguments = False,
+        argument_spec = dict(
+            url = dict(required=True),
+            dest = dict(required=True),
+        )
+    )
+    
+    url  = module.params['url']
+    dest = os.path.expanduser(module.params['dest'])
+    
+    # download to tmpsrc
+    tmpsrc, info = url_get(module, url, dest)
+    md5sum_src   = None
+    md5sum_dest  = None
+    dest         = info['actualdest']
+    
+    # raise an error if there is no tmpsrc file
+    if not os.path.exists(tmpsrc):
+        os.remove(tmpsrc)
+        module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'])
+    if not os.access(tmpsrc, os.R_OK):
+        os.remove(tmpsrc)
+        module.fail_json( msg="Source %s not readable" % (tmpsrc))
+    md5sum_src = module.md5(tmpsrc)
+     
+    # check if there is no dest file
+    if os.path.exists(dest):
+        # raise an error if copy has no permission on dest
+        if not os.access(dest, os.W_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not writable" % (dest))
+        if not os.access(dest, os.R_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not readable" % (dest))
+        md5sum_dest = module.md5(dest)
+    else:
+        if not os.access(os.path.dirname(dest), os.W_OK):
+            os.remove(tmpsrc)
+            module.fail_json( msg="Destination %s not writable" % (os.path.dirname(dest)))
+    
+    if md5sum_src != md5sum_dest:
+        try:
+            shutil.copyfile(tmpsrc, dest)
+        except Exception, err:
+            os.remove(tmpsrc)
+            module.fail_json(msg="failed to copy %s to %s: %s" % (tmpsrc, dest, str(err))) 
+        changed = True
+    else:
+        changed = False
+    
+    os.remove(tmpsrc)
+
+    # Mission complete
+    module.exit_json(url=url, dest=dest, src=tmpsrc, md5sum=md5sum_src, 
+        changed=changed, msg=info.get('msg',''), 
+        daisychain="file", daisychain_args=info.get('daisychain_args',''))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/git

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -18,124 +19,179 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 
 # I wanted to keep this simple at first, so for now this checks out
-# from the MASTER branch of a repo at a particular SHA or
+# from the given branch of a repo at a particular SHA or
 # tag.  Latest is not supported, you should not be doing
-# that.  Branch checkouts are not supported.  Contribs
-# welcome! -- MPD
+# that. Contribs welcome! -- MPD
 
-try:
-    import json
-except ImportError:
-    import simplejson as json
-import os
-import sys
-import shlex
-import subprocess
-
-# ===========================================
-# convert arguments of form a=b c=d
-# to a dictionary
-# FIXME: make more idiomatic
-
-args = " ".join(sys.argv[1:])
-items = shlex.split(args)
-params = {}
-for x in items:
-    (k, v) = x.split("=")
-    params[k] = v
-
-dest = params['dest']
-repo = params['repo']
-version = params['version']
-
-# ===========================================
+import re
 
 def get_version(dest):
-   ''' samples the version of the git repo '''
-   os.chdir(dest)
-   cmd = "git show --abbrev-commit"
-   sha = os.popen(cmd).read().split("\n")
-   sha = sha[0].split()[1]
-   return sha
+    ''' samples the version of the git repo '''
+    os.chdir(dest)
+    cmd = "git show --abbrev-commit"
+    sha = os.popen(cmd).read().split("\n")
+    sha = sha[0].split()[1]
+    return sha
 
 def clone(repo, dest):
-   ''' makes a new git repo if it does not already exist '''
-   try:
-       os.makedirs(dest)
-   except:
-       pass
-   cmd = "git clone %s %s" % (repo, dest)
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   return cmd.communicate()
+    ''' makes a new git repo if it does not already exist '''
+    try:
+        os.makedirs(os.path.dirname(dest))
+    except:
+        pass
+    cmd = "git clone %s %s" % (repo, dest)
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-def pull(repo, dest):
-   ''' updates repo from remote sources '''
-   os.chdir(dest)
-   cmd = "git pull -u origin"
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   return cmd.communicate()
+def reset(dest):
+    '''
+    Resets the index and working tree to HEAD.
+    Discards any changes to tracked files in working
+    tree since that commit.
+    '''
+    os.chdir(dest)
+    cmd = "git reset --hard HEAD"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-def switchver(version, dest):
-   ''' once pulled, switch to a particular SHA or tag '''
-   os.chdir(dest)
-   if version != 'HEAD':
-      cmd = "git checkout %s --force" % version
-   else:
-      # is there a better way to do this?
-      cmd = "git rebase origin"
-   cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-   (out, err) = cmd.communicate()
-   return (out, err)
- 
+def switchLocalBranch( branch ):
+    cmd = "git checkout %s" % branch
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    return cmd.communicate()
 
-gitconfig = os.path.join(dest, '.git', 'config')
+def get_branches(module, dest):
+    os.chdir(dest)
+    branches = []
+    cmd = "git branch -a"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = cmd.communicate()
+    if cmd.returncode != 0:
+        module.fail_json(msg="Could not determine branch data - received %s" % out)
+    for line in out.split('\n'):
+        branches.append(line.strip())
+    return branches
 
-out, err, status = (None, None, None)
+def is_remote_branch(module, dest, remote, branch):
+    branches = get_branches(module, dest)
+    rbranch = 'remotes/%s/%s' % (remote, branch)
+    if rbranch in branches:
+        return True
+    else:
+        return False
 
-# if there is no git configuration, do a clone operation
-# else pull and switch the version
+def is_local_branch(module, dest, branch):
+    branches = get_branches(module, dest)
+    lbranch = '%s' % branch
+    if lbranch in branches:
+        return True
+    elif '* %s' % branch in branches:
+        return True
+    else:
+        return False
 
-before = None
-if not os.path.exists(gitconfig):
-   (out, err) = clone(repo, dest)
-else:
-   # else do a pull   
-   before = get_version(dest)
-   (out, err) = pull(repo, dest)
+def is_current_branch(module, dest, branch):
+    branches = get_branches(module, dest)
+    for b in branches:
+        if b.startswith('* '):
+            cur_branch = b
+    if branch == cur_branch or '* %s' % branch == cur_branch:
+        return True
+    else:
+        return True
 
-# handle errors from clone or pull
+def pull(module, repo, dest, version):
+    ''' updates repo from remote sources '''
+    os.chdir(dest)
+    branches = get_branches(module, dest)
+    cur_branch = ''
+    for b in branches:
+        if b.startswith('* '):
+            cur_branch = b
+    if is_local_branch(module, dest, version) and not is_current_branch(module, dest, version):
+        (out, err) = switch_version(module, dest, remote, version)
 
-if out.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+    cmd = "git pull -u origin"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-# switch to version specified regardless of whether
-# we cloned or pulled
+def switch_version(module, dest, remote, version):
+    ''' once pulled, switch to a particular SHA, tag, or branch '''
+    os.chdir(dest)
+    cmd = ''
+    if version != 'HEAD':
+        if not is_local_branch(module, dest, version) and is_remote_branch(module, dest, remote, version):
+            cmd = "git checkout --track -b %s %s/%s" % (version, remote, version)
+        else:
+            cmd = "git checkout --force %s" % version
+    else:
+        # is there a better way to do this?
+        cmd = "git rebase origin"
+    cmd = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = cmd.communicate()
+    rc = cmd.returncode
+    return (rc, out, err)
 
-(out, err) = switchver(version, dest)
-if err.find('error') != -1:
-   print json.dumps({
-       "failed" : True,
-       "out"    : out,
-       "err"    : err
-   })
-   sys.exit(1)
+# ===========================================
 
-# determine if we changed anything
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            dest=dict(required=True),
+            repo=dict(required=True, aliases=['name']),
+            version=dict(default='HEAD'),
+            remote=dict(default='origin')
+        )
+    )
 
-after = get_version(dest)
-changed = False
+    dest    = module.params['dest']
+    repo    = module.params['repo']
+    version = module.params['version']
+    remote  = module.params['remote']
 
-if before != after:
-   changed = True
+    gitconfig = os.path.join(dest, '.git', 'config')
 
-print json.dumps({
-   "changed" : changed,
-   "before"  : before,
-   "after"   : after
-})
+    out, err, status = (None, None, None)
 
+    # if there is no git configuration, do a clone operation
+    # else pull and switch the version
+    before = None
+    if not os.path.exists(gitconfig):
+        (rc, out, err) = clone(repo, dest)
+        if rc != 0:
+            module.fail_json(msg=err)
+    else:
+        # else do a pull   
+        before = get_version(dest)
+        (rc, out, err) = reset(dest)
+        if rc != 0:
+            module.fail_json(msg=err)
+        (rc, out, err) = pull(module, repo, dest, version)
+
+    # handle errors from clone or pull
+    if out.find('error') != -1 or err.find('ERROR') != -1:
+        module.fail_json(msg=err)
+
+    # switch to version specified regardless of whether
+    # we cloned or pulled
+    (rc, out, err) = switch_version(module, dest, remote, version)
+    if err.find('error') != -1:
+        module.fail_json(msg=err)
+
+    # determine if we changed anything
+    after = get_version(dest)
+    changed = False
+
+    if before != after:
+        changed = True
+
+    module.exit_json(changed=changed, before=before, after=after)
+
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/group

@@ -0,0 +1,138 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Stephen Fromm <sfromm@gmail.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import grp
+
+def get_bin_path(module, arg):
+    if os.path.exists('/usr/sbin/%s' % arg):
+        return '/usr/sbin/%s' % arg
+    elif os.path.exists('/sbin/%s' % arg):
+        return '/sbin/%s' % arg
+    else:
+        module.fail_json(msg="Cannot find %s" % arg)
+
+def group_del(module, group):
+    cmd = [get_bin_path(module, 'groupdel'), group]
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
+
+def group_add(module, group, **kwargs):
+    cmd = [get_bin_path(module, 'groupadd')]
+    for key in kwargs:
+        if key == 'gid' and kwargs[key] is not None:
+            cmd.append('-g')
+            cmd.append(kwargs[key])
+        elif key == 'system' and kwargs[key] == 'yes':
+            cmd.append('-r')
+    cmd.append(group)
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
+
+def group_mod(module, group, **kwargs):
+    cmd = [get_bin_path(module, 'groupmod')]
+    info = group_info(group)
+    for key in kwargs:
+        if key == 'gid':
+            if kwargs[key] is not None and info[2] != int(kwargs[key]):
+                cmd.append('-g')
+                cmd.append(kwargs[key])
+    if len(cmd) == 1:
+        return (None, '', '')
+    cmd.append(group)
+    p = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return (rc, out, err)
+
+def group_exists(group):
+    try:
+        if grp.getgrnam(group):
+            return True
+    except KeyError:
+        return False
+
+def group_info(group):
+    if not group_exists(group):
+        return False
+    try:
+        info = list(grp.getgrnam(group))
+    except KeyError:
+        return False
+    return info
+    
+# ===========================================
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            state=dict(default='present', choices=['present', 'absent']),
+            name=dict(required=True),
+            gid=dict(default=None),
+            system=dict(default='no', choices=['yes', 'no']),
+        )
+    )
+
+    state = module.params['state']
+    name = module.params['name']
+    gid = module.params['gid']
+    system = module.params['system']
+    rc = None
+    out = ''
+    err = ''
+    result = {}
+    result['name'] = name
+    result['state'] = state
+    if state == 'absent':
+        if group_exists(name):
+            (rc, out, err) = group_del(module, name)
+            if rc != 0:
+                module.fail_json(name=name, msg=err)
+    elif state == 'present':
+        if not group_exists(name):
+            (rc, out, err) = group_add(module, name, gid=gid, system=system)
+        else:
+            (rc, out, err) = group_mod(module, name, gid=gid)
+
+        if rc is not None and rc != 0:
+            module.fail_json(name=name, msg=err)
+
+    if rc is None:
+        result['changed'] = False
+    else:
+        result['changed'] = True
+    if out:
+        result['stdout'] = out
+    if err:
+        result['stderr'] = err
+
+    if group_exists(name):
+        info = group_info(name)
+        result['system'] = system
+        result['gid'] = info[2]
+
+    module.exit_json(**result)
+
+# include magic from lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/mount

@@ -0,0 +1,265 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Red Hat, inc
+# Written by Seth Vidal
+# based on the mount modules from salt and puppet
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# mount module - mount fs and define in fstab
+# usage:
+#
+# mount name=mountpoint, src=device_to_be_mounted fstype=fstype 
+#        opts=mount_opts, dump=0 passno=0 state=[present|absent|mounted|unmounted] 
+#
+#    absent == remove from fstab and unmounted
+#    present == add to fstab, do not change mount state
+#    mounted == add to fstab if not there and make sure it is mounted
+#    unmounted == do not change fstab state, but unmount
+
+def write_fstab(lines, dest):
+
+    fs_w = open(dest, 'w')
+    for l in lines:
+        fs_w.write(l)
+
+    fs_w.flush()
+    fs_w.close()
+
+def set_mount(**kwargs):
+    """ set/change a mount point location in fstab """
+
+    # kwargs: name, src, fstype, opts, dump, passno, state, fstab=/etc/fstab
+    args = dict(
+        opts   = 'defaults',
+        dump   = '0',
+        passno = '0',
+        fstab  = '/etc/fstab'
+    )
+    args.update(kwargs)
+
+    new_line = '%(src)s %(name)s %(fstype)s %(opts)s %(dump)s %(passno)s\n' 
+
+    to_write = []
+    exists = False
+    changed = False
+    for line in open(args['fstab'], 'r').readlines():
+        if not line.strip():
+             to_write.append(line)
+             continue
+        if line.strip().startswith('#'):
+            to_write.append(line)
+            continue
+        if len(line.split()) != 6:
+            # not sure what this is or why it is here
+            # but it is not our fault so leave it be
+            to_write.append(line)
+            continue
+        
+        ld = {}
+        ld['src'], ld['name'], ld['fstype'], ld['opts'], ld['dump'], ld['passno']  = line.split()
+
+        if ld['name'] != args['name']:
+            to_write.append(line)
+            continue
+
+        # it exists - now see if what we have is different
+        exists = True
+        for t in ('src', 'fstype','opts', 'dump', 'passno'):
+            if ld[t] != args[t]:
+                changed = True
+                ld[t] = args[t]
+
+        if changed:
+             to_write.append(new_line % ld)
+        else:
+             to_write.append(line)
+         
+    if not exists:
+        to_write.append(new_line % args)
+        changed = True
+    
+    if changed:
+        write_fstab(to_write, args['fstab'])
+
+    return (args['name'], changed)
+            
+
+def unset_mount(**kwargs):
+    """ remove a mount point from fstab """
+
+    # kwargs: name, src, fstype, opts, dump, passno, state, fstab=/etc/fstab
+    args = dict(
+        opts   = 'default',
+        dump   = '0',
+        passno = '0',
+        fstab  = '/etc/fstab'
+    )
+    args.update(kwargs)
+
+    to_write = []
+    changed = False
+    for line in open(args['fstab'], 'r').readlines():
+        if not line.strip():
+             to_write.append(line)
+             continue
+        if line.strip().startswith('#'):
+            to_write.append(line)
+            continue
+        if len(line.split()) != 6:
+            # not sure what this is or why it is here
+            # but it is not our fault so leave it be
+            to_write.append(line)
+            continue
+        
+        ld = {}
+        ld['src'], ld['name'], ld['fstype'], ld['opts'], ld['dump'], ld['passno']  = line.split()
+
+        if ld['name'] != args['name']:
+            to_write.append(line)
+            continue
+
+        # if we got here we found a match - continue and mark changed
+        changed = True
+
+    if changed:
+        write_fstab(to_write, args['fstab'])
+
+    return (args['name'], changed)
+
+     
+def mount(**kwargs):
+    """ mount up a path or remount if needed """
+
+    name = kwargs['name']
+    if os.path.ismount(name):
+        cmd = ['/bin/mount', '-o', 'remount', name]
+    else:
+        cmd = ['/bin/mount', name ]
+
+    call = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = call.communicate()
+    if call.returncode == 0:
+        return 0, ''
+    else:
+        return call.returncode, out+err
+
+def umount(**kwargs):
+    """ unmount a path """
+
+    name = kwargs['name']
+    cmd = ['/bin/umount', name]
+
+    call = subprocess.Popen(cmd, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    out, err = call.communicate()
+    if call.returncode == 0:
+        return 0, ''
+    else:
+        return call.returncode, out+err
+
+def main():
+
+    module = AnsibleModule(
+        argument_spec = dict(
+            state  = dict(required=True, choices=['present', 'absent', 'mounted', 'unmounted']),
+            name   = dict(required=True),
+            opts   = dict(default=None),
+            passno = dict(default=None),
+            dump   = dict(default=None),
+            src    = dict(required=True),
+            fstype = dict(required=True),
+            fstab  = dict(default=None)
+        )
+    )
+    
+    changed = False
+    rc = 0
+    args = {
+        'name': module.params['name'],
+        'src': module.params['src'],
+        'fstype': module.params['fstype']
+    }
+    if module.params['passno'] is not None:
+        args['passno'] = module.params['passno']
+    if module.params['opts'] is not None:
+        args['opts'] = module.params['opts']
+    if module.params['dump'] is not None:
+        args['dump'] = module.params['dump']
+    if module.params['fstab'] is not None:
+        args['fstab'] = module.params['fstab']
+    
+    # absent == remove from fstab and unmounted
+    # unmounted == do not change fstab state, but unmount
+    # present == add to fstab, do not change mount state
+    # mounted == add to fstab if not there and make sure it is mounted, if it has changed in fstab then remount it
+    
+    state = module.params['state']
+    name  = module.params['name']
+    if state == 'absent':
+        name, changed = unset_mount(**args)
+        if changed:
+            if os.path.ismount(name):
+                res,msg  = umount(**args)
+                if res:
+                    fail_json(msg="Error unmounting %s: %s" % (name, msg))
+    
+            if os.path.exists(name):
+                try:
+                    os.rmdir(name)
+                except (OSError, IOError), e:
+                    fail_json(msg="Error rmdir %s: %s" % (name, str(e)))
+    
+        module.exit_json(changed=changed, **args)
+    
+    if state == 'unmounted':
+        if os.path.ismount(name):
+            res,msg  = umount(**args)
+            if res:
+                fail_json(msg="Error unmounting %s: %s" % (name, msg))
+            changed = True
+    
+        module.exit_json(changed=changed, **args)
+    
+    if state in ['mounted', 'present']:
+        name, changed = set_mount(**args)
+        if state == 'mounted':
+            if not os.path.exists(name):
+                try:
+                    os.makedirs(name)
+                except (OSError, IOError), e:
+                    fail_json(msg="Error making dir %s: %s" % (name, str(e)))
+    
+            res = 0
+            if os.path.ismount(name):
+                if changed:
+                    res,msg = mount(**args)
+            else:
+                changed = True
+                res,msg = mount(**args)
+    
+            if res:
+                fail_json(msg="Error mounting %s: %s" % (name, msg))
+    
+    
+        module.exit_json(changed=changed, **args)
+    
+    module.fail_json(msg='Unexpected position reached')
+    sys.exit(0)
+    
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/mysql_db

@@ -0,0 +1,123 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2012, Mark Theunissen <mark.theunissen@gmail.com>
+# Sponsored by Four Kitchens http://fourkitchens.com.
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ConfigParser
+try:
+    import MySQLdb
+except ImportError:
+    mysqldb_found = False
+else:
+    mysqldb_found = True
+
+# ===========================================
+# MySQL module specific support methods.
+#
+
+def db_exists(cursor, db):
+    res = cursor.execute("SHOW DATABASES LIKE %s", (db,))
+    return bool(res)
+
+def db_delete(cursor, db):
+    query = "DROP DATABASE %s" % db
+    cursor.execute(query)
+    return True
+
+def db_create(cursor, db, encoding, collation):
+    if encoding:
+        encoding = " CHARACTER SET %s" % encoding
+    if collation:
+        collation = " COLLATE %s" % collation
+    query = "CREATE DATABASE %s%s%s" % (db, encoding, collation)
+    res = cursor.execute(query)
+    return True
+
+def load_mycnf():
+    config = ConfigParser.RawConfigParser()
+    mycnf = os.path.expanduser('~/.my.cnf')
+    if not os.path.exists(mycnf):
+        return False
+    try:
+        config.readfp(open(mycnf))
+        creds = dict(user=config.get('client', 'user'),passwd=config.get('client', 'pass'))
+    except (ConfigParser.NoOptionError, IOError):
+        return False
+    return creds
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            login_user=dict(default=None),
+            login_password=dict(default=None),
+            login_host=dict(default="localhost"),
+            db=dict(required=True, aliases=['name']),
+            encoding=dict(default=""),
+            collation=dict(default=""),
+            state=dict(default="present", choices=["absent", "present"]),
+        )
+    )
+
+    if not mysqldb_found:
+        module.fail_json(msg="the python mysqldb module is required")
+
+    db = module.params["db"]
+    encoding = module.params["encoding"]
+    collation = module.params["collation"]
+    state = module.params["state"]
+
+    # Either the caller passes both a username and password with which to connect to
+    # mysql, or they pass neither and allow this module to read the credentials from
+    # ~/.my.cnf.
+    login_password = module.params["login_password"]
+    login_user = module.params["login_user"]
+    if login_user is None and login_password is None:
+        mycnf_creds = load_mycnf()
+        if mycnf_creds is False:
+            login_user = "root"
+            login_password = ""
+        else:
+            login_user = mycnf_creds["user"]
+            login_password = mycnf_creds["passwd"]
+    elif login_password is None or login_user is None:
+        module.fail_json(msg="when supplying login arguments, both login_user and login_password must be provided")
+
+    try:
+        db_connection = MySQLdb.connect(host=module.params["login_host"], user=login_user, passwd=login_password, db="mysql")
+        cursor = db_connection.cursor()
+    except Exception as e:
+        module.fail_json(msg="unable to connect, check login_user and login_password are correct, or alternatively check ~/.my.cnf contains credentials")
+
+    changed = False
+    if db_exists(cursor, db):
+        if state == "absent":
+            changed = db_delete(cursor, db)
+    else:
+        if state == "present":
+            changed = db_create(cursor, db, encoding, collation)
+
+    module.exit_json(changed=changed, db=db)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/mysql_user

@@ -0,0 +1,227 @@
+#!/usr/bin/python
+
+# (c) 2012, Mark Theunissen <mark.theunissen@gmail.com>
+# Sponsored by Four Kitchens http://fourkitchens.com.
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+import ConfigParser
+try:
+    import MySQLdb
+except ImportError:
+    mysqldb_found = False
+else:
+    mysqldb_found = True
+
+# ===========================================
+# MySQL module specific support methods.
+#
+
+def user_exists(cursor, user, host):
+    cursor.execute("SELECT count(*) FROM user WHERE user = %s AND host = %s", (user,host))
+    count = cursor.fetchone()
+    return count[0] > 0
+
+def user_add(cursor, user, host, password, new_priv):
+    cursor.execute("CREATE USER %s@%s IDENTIFIED BY %s", (user,host,password))
+    if new_priv is not None:
+        for db_table, priv in new_priv.iteritems():
+            privileges_grant(cursor, user,host,db_table,priv)
+    return True
+
+def user_mod(cursor, user, host, password, new_priv):
+    changed = False
+
+    # Handle passwords.
+    if password is not None:
+        cursor.execute("SELECT password FROM user WHERE user = %s AND host = %s", (user,host))
+        current_pass_hash = cursor.fetchone()
+        cursor.execute("SELECT PASSWORD(%s)", (password,))
+        new_pass_hash = cursor.fetchone()
+        if current_pass_hash[0] != new_pass_hash[0]:
+            cursor.execute("SET PASSWORD FOR %s@%s = PASSWORD(%s)", (user,host,password))
+            changed = True
+
+    # Handle privileges.
+    if new_priv is not None:
+        curr_priv = privileges_get(cursor, user,host)
+
+        # If the user has privileges on a db.table that doesn't appear at all in
+        # the new specification, then revoke all privileges on it.
+        for db_table, priv in curr_priv.iteritems():
+            if db_table not in new_priv:
+                privileges_revoke(cursor, user,host,db_table)
+                changed = True
+
+        # If the user doesn't currently have any privileges on a db.table, then
+        # we can perform a straight grant operation.
+        for db_table, priv in new_priv.iteritems():
+            if db_table not in curr_priv:
+                privileges_grant(cursor, user,host,db_table,priv)
+                changed = True
+
+        # If the db.table specification exists in both the user's current privileges
+        # and in the new privileges, then we need to see if there's a difference.
+        db_table_intersect = set(new_priv.keys()) & set(curr_priv.keys())
+        for db_table in db_table_intersect:
+            priv_diff = set(new_priv[db_table]) ^ set(curr_priv[db_table])
+            if (len(priv_diff) > 0):
+                privileges_revoke(cursor, user,host,db_table)
+                privileges_grant(cursor, user,host,db_table,new_priv[db_table])
+                changed = True
+
+    return changed
+
+def user_delete(cursor, user, host):
+    cursor.execute("DROP USER %s@%s", (user,host))
+    return True
+
+def privileges_get(cursor, user,host):
+    """ MySQL doesn't have a better method of getting privileges aside from the
+    SHOW GRANTS query syntax, which requires us to then parse the returned string.
+    Here's an example of the string that is returned from MySQL:
+
+     GRANT USAGE ON *.* TO 'user'@'localhost' IDENTIFIED BY 'pass';
+
+    This function makes the query and returns a dictionary containing the results.
+    The dictionary format is the same as that returned by privileges_unpack() below.
+    """
+    output = {}
+    cursor.execute("SHOW GRANTS FOR %s@%s", (user,host))
+    grants = cursor.fetchall()
+    for grant in grants:
+        res = re.match("GRANT\ (.+)\ ON\ (.+)\ TO", grant[0])
+        if res is None:
+            module.fail_json(msg="unable to parse the MySQL grant string")
+        privileges = res.group(1).split(", ")
+        privileges = ['ALL' if x=='ALL PRIVILEGES' else x for x in privileges]
+        db = res.group(2).replace('`', '')
+        output[db] = privileges
+    return output
+
+def privileges_unpack(priv):
+    """ Take a privileges string, typically passed as a parameter, and unserialize
+    it into a dictionary, the same format as privileges_get() above. We have this
+    custom format to avoid using YAML/JSON strings inside YAML playbooks. Example
+    of a privileges string:
+
+     mydb.*:INSERT,UPDATE/anotherdb.*:SELECT/yetanother.*:ALL
+
+    The privilege USAGE stands for no privileges, so we add that in on *.* if it's
+    not specified in the string, as MySQL will always provide this by default.
+    """
+    output = {}
+    for item in priv.split('/'):
+        pieces = item.split(':')
+        output[pieces[0]] = pieces[1].upper().split(',')
+
+    if '*.*' not in output:
+        output['*.*'] = ['USAGE']
+
+    return output
+
+def privileges_revoke(cursor, user,host,db_table):
+    query = "REVOKE ALL PRIVILEGES ON %s FROM '%s'@'%s'" % (db_table,user,host)
+    cursor.execute(query)
+
+def privileges_grant(cursor, user,host,db_table,priv):
+    priv_string = ",".join(priv)
+    query = "GRANT %s ON %s TO '%s'@'%s'" % (priv_string,db_table,user,host)
+    cursor.execute(query)
+
+def load_mycnf():
+    config = ConfigParser.RawConfigParser()
+    mycnf = os.path.expanduser('~/.my.cnf')
+    if not os.path.exists(mycnf):
+        return False
+    try:
+        config.readfp(open(mycnf))
+        creds = dict(user=config.get('client', 'user'),password=config.get('client', 'pass'))
+    except (ConfigParser.NoOptionError, IOError):
+        return False
+    return creds
+
+# ===========================================
+# Module execution.
+#
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            login_user=dict(default=None),
+            login_password=dict(default=None),
+            login_host=dict(default="localhost"),
+            user=dict(required=True, aliases=['name']),
+            password=dict(default=None),
+            host=dict(default="localhost"),
+            state=dict(default="present", choices=["absent", "present"]),
+            priv=dict(default=None),
+        )
+    )
+    user = module.params["user"]
+    password = module.params["password"]
+    host = module.params["host"]
+    state = module.params["state"]
+    priv = module.params["priv"]
+
+    if not mysqldb_found:
+        module.fail_json(msg="the python mysqldb module is required")
+
+    if priv is not None:
+        try:
+            priv = privileges_unpack(priv)
+        except:
+            module.fail_json(msg="invalid privileges string")
+
+    # Either the caller passes both a username and password with which to connect to
+    # mysql, or they pass neither and allow this module to read the credentials from
+    # ~/.my.cnf.
+    login_password = module.params["login_password"]
+    login_user = module.params["login_user"]
+    if login_user is None and login_password is None:
+        mycnf_creds = load_mycnf()
+        if mycnf_creds is False:
+            login_user = "root"
+            login_password = ""
+        else:
+            login_user = mycnf_creds["user"]
+            login_password = mycnf_creds["password"]
+    elif login_password is None or login_user is None:
+        module.fail_json(msg="when supplying login arguments, both login_user and login_password must be provided")
+
+    try:
+        db_connection = MySQLdb.connect(host=module.params["login_host"], user=login_user, passwd=login_password, db="mysql")
+        cursor = db_connection.cursor()
+    except Exception as e:
+        module.fail_json(msg="unable to connect to database, check login_user and login_password are correct or ~/.my.cnf has the credentials")
+
+    if state == "present":
+        if user_exists(cursor, user, host):
+            changed = user_mod(cursor, user, host, password, priv)
+        else:
+            if password is None:
+                module.fail_json(msg="password parameter required when adding a user")
+            changed = user_add(cursor, user, host, password, priv)
+    elif state == "absent":
+        if user_exists(cursor, user, host):
+            changed = user_delete(cursor, user, host)
+        else:
+            changed = False
+    module.exit_json(changed=changed, user=user)
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+main()

library/ohai

@@ -1,4 +1,5 @@
-#!/bin/bash
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
 
 # (c) 2012, Michael DeHaan <michael.dehaan@gmail.com>
 #
@@ -18,4 +19,25 @@
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-/usr/bin/ohai
+def get_ohai_data():
+    p = subprocess.Popen(["/usr/bin/env", "ohai"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    (out, err) = p.communicate()
+    rc = p.returncode
+    return rc, out, err
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict()
+    )
+    rc, out, err = get_ohai_data()
+    if rc != 0:
+       module.fail_json(msg=err)
+    else:
+       module.exit_json(**json.loads(out))
+
+# this is magic, see lib/ansible/module_common.py
+#<<INCLUDE_ANSIBLE_MODULE_COMMON>>
+
+main()
+
+