update submodule pointer

Fixes authentication error bug when user has rsa keys

.travis.yml

@@ -24,6 +24,7 @@ script:
 - ./test/code-smell/replace-urlopen.sh .
 - ./test/code-smell/use-compat-six.sh lib
 - ./test/code-smell/boilerplate.sh
+- ./test/code-smell/required-and-default-attributes.sh
 - if test x"$TOXENV" != x'py24' ; then tox ; fi
 - if test x"$TOXENV" = x'py24' ; then python2.4 -V && python2.4 -m compileall -fq -x 'module_utils/(a10|rax|openstack|ec2|gce).py' lib/ansible/module_utils ; fi
   #- make -C docsite all

CHANGELOG.md

@@ -1,7 +1,53 @@
 Ansible Changes By Release
 ==========================
 
-## 2.0 "Over the Hills and Far Away" - ACTIVE DEVELOPMENT
+## 2.1 TBD - ACTIVE DEVELOPMENT
+
+####New Modules:
+* cloudstack: cs_volume
+
+####New Filters:
+* extract
+
+## 2.0.1 "Over the Hills and Far Away"
+
+* Fixes a major compatibility break in the synchronize module shipped with
+  2.0.0.x.  That version of synchronize ran sudo on the controller prior to
+  running rsync.  In 1.9.x and previous, sudo was run on the host that rsync
+  connected to.  2.0.1 restores the 1.9.x behaviour.
+* Additionally, several other problems with where synchronize chose to run when
+  combined with delegate_to were fixed.  In particular, if a playbook targetted
+  localhost and then delegated_to a remote host the prior behavior (in 1.9.x
+  and 2.0.0.x) was to copy files between the src and destination directories on
+  the delegated host.  This has now been fixed to copy between localhost and
+  the delegated host.
+* Fix a regression where synchronize was unable to deal with unicode paths.
+* Fix a regression where synchronize deals with inventory hosts that use
+  localhost but with an alternate port.
+* Fixes a regression where the retry files feature was not implemented.
+* Fixes a regression where the any_errors_fatal option was implemented in 2.0
+  incorrectly, and also adds a feature where any_errors_fatal can be set at
+  the block level.
+* Fix tracebacks when playbooks or ansible itself were located in directories
+  with unicode characters.
+* Fix bug when sending unicode characters to an external pager for display.
+* Fix a bug with squashing loops for special modules (mostly package managers).
+  The optimization was squashing when the loop did not apply to the selection
+  of packages.  This has now been fixed.
+* Temp files created when using vault are now "shredded" using the unix shred
+  program which overwrites the file with random data.
+* Some fixes to cloudstack modules for case sensitivity
+* Fix non-newstyle modules (non-python modules and old-style modules) to
+  disabled pipelining.
+* Fix fetch module failing even if fail_on_missing is set to False
+* Fix for cornercase when local connections, sudo, and raw were used together.
+* Fix dnf module to remove dependent packages when state=absent is specified.
+  This was a feature of the 1.9.x version that was left out by mistake when the
+  module was rewritten for 2.0.
+* Fix bugs with non-english locales in yum, git, and apt modules
+* Fix a bug with the dnf module where state=latest could only upgrade, not install.
+
+## 2.0 "Over the Hills and Far Away" - Jan 12, 2016
 
 ###Major Changes:
 
@@ -24,10 +70,13 @@ Ansible Changes By Release
   by setting the `ANSIBLE_NULL_REPRESENTATION` environment variable.
 * Added `meta: refresh_inventory` to force rereading the inventory in a play.
   This re-executes inventory scripts, but does not force them to ignore any cache they might use.
-* Now when you delegate an action that returns ansible_facts, these facts will be applied to the delegated host, unlike before when they were applied to the current host.
+* New delegate_facts directive, a boolean that allows you to apply facts to the delegated host (true/yes) instead of the inventory_hostname (no/false) which is the default and previous behaviour.
+* local connections now work with 'su' as a privilege escalation method
+* Ansible 2.0 has deprecated the “ssh” from ansible_ssh_user, ansible_ssh_host, and ansible_ssh_port to become ansible_user, ansible_host, and ansible_port.
 * New ssh configuration variables(`ansible_ssh_common_args`, `ansible_ssh_extra_args`) can be used to configure a
   per-group or per-host ssh ProxyCommand or set any other ssh options.
   `ansible_ssh_extra_args` is used to set options that are accepted only by ssh (not sftp or scp, which have their own analogous settings).
+* ansible-pull can now verify the code it runs when using git as a source repository, using git's code signing and verification features.
 * Backslashes used when specifying parameters in jinja2 expressions in YAML dicts sometimes needed to be escaped twice.
   This has been fixed so that escaping once works. Here's an example of how playbooks need to be modified:
 
@@ -71,9 +120,43 @@ newline being stripped you can change your playbook like this:
     "msg": "Testing some things"
     ```
 
+* In 1.9.x, newlines in templates were converted to Unix EOL conventions.  If
+  someone wanted a templated file to end up with Windows or Mac EOL
+  conventions, this could cause problems for them.  In 2.x newlines now remain
+  as specified in the template file.
+
+* When specifying complex args as a variable, the variable must use the full jinja2
+variable syntax ('{{var_name}}') - bare variable names there are no longer accepted.
+In fact, even specifying args with variables has been deprecated, and will not be
+allowed in future versions:
+
+    ```
+    ---
+    - hosts: localhost
+      connection: local
+      gather_facts: false
+      vars:
+        my_dirs:
+          - { path: /tmp/3a, state: directory, mode: 0755 }
+          - { path: /tmp/3b, state: directory, mode: 0700 }
+      tasks:
+        - file:
+          args: "{{item}}"
+          with_items: my_dirs
+    ```
+
+* The bigip\* networking modules have a new parameter, validate_certs.  When
+  True (the default) the module will validate any hosts it connects to against
+  the TLS certificates it presents when run on new enough python versions.  If
+  the python version is too old to validate certificates or you used certificates
+  that cannot be validated against available CAs you will need to add
+  validate_certs=no to your playbook for those tasks.
+
 ###Plugins
 
 * Rewritten dnf module that should be faster and less prone to encountering bugs in cornercases
+* WinRM connection plugin passes all vars named `ansible_winrm_*` to the underlying pywinrm client. This allows, for instance, `ansible_winrm_server_cert_validation=ignore` to be used with newer versions of pywinrm to disable certificate validation on Python 2.7.9+.
+* WinRM connection plugin put_file is significantly faster and no longer has file size limitations. 
 
 ####Deprecated Modules (new ones in parens):
 
@@ -94,23 +177,31 @@ newline being stripped you can change your playbook like this:
 * amazon: ec2_eni
 * amazon: ec2_eni_facts
 * amazon: ec2_remote_facts
+* amazon: ec2_vpc_igw
 * amazon: ec2_vpc_net
+* amazon: ec2_vpc_net_facts
 * amazon: ec2_vpc_route_table
 * amazon: ec2_vpc_route_table_facts
 * amazon: ec2_vpc_subnet
+* amazon: ec2_vpc_subnet_facts
 * amazon: ec2_win_password
 * amazon: ecs_cluster
 * amazon: ecs_task
 * amazon: ecs_taskdefinition
-* amazon: elasticache_subnet_group
+* amazon: elasticache_subnet_group_facts
 * amazon: iam
+* amazon: iam_cert
 * amazon: iam_policy
-* amazon: route53_zone
+* amazon: route53_facts
 * amazon: route53_health_check
+* amazon: route53_zone
 * amazon: sts_assume_role
 * amazon: s3_bucket
 * amazon: s3_lifecycle
 * amazon: s3_logging
+* amazon: sqs_queue
+* amazon: sns_topic
+* amazon: sts_assume_role
 * apk
 * bigip_gtm_wide_ip
 * bundler
@@ -151,24 +242,29 @@ newline being stripped you can change your playbook like this:
 * cloudstack: cs_template
 * cloudstack: cs_user
 * cloudstack: cs_vmsnapshot
+* cronvar
 * datadog_monitor
 * deploy_helper
+* docker: docker_login
 * dpkg_selections
 * elasticsearch_plugin
 * expect
 * find
+* google: gce_tag
 * hall
 * ipify_facts
 * iptables
 * libvirt: virt_net
 * libvirt: virt_pool
 * maven_artifact
-* openstack: os_ironic
-* openstack: os_ironic_node
+* openstack: os_auth
 * openstack: os_client_config
-* openstack: os_floating_ip
 * openstack: os_image
 * openstack: os_image_facts
+* openstack: os_floating_ip
+* openstack: os_ironic
+* openstack: os_ironic_node
+* openstack: os_keypair
 * openstack: os_network
 * openstack: os_network_facts
 * openstack: os_nova_flavor
@@ -184,6 +280,7 @@ newline being stripped you can change your playbook like this:
 * openstack: os_server_volume
 * openstack: os_subnet
 * openstack: os_subnet_facts
+* openstack: os_user
 * openstack: os_user_group
 * openstack: os_volume
 * openvswitch_db.
@@ -194,14 +291,15 @@ newline being stripped you can change your playbook like this:
 * profitbricks: profitbricks
 * profitbricks: profitbricks_datacenter
 * profitbricks: profitbricks_nic
-* profitbricks: profitbricks_snapshot
 * profitbricks: profitbricks_volume
 * profitbricks: profitbricks_volume_attachments
-* proxmox
-* proxmox_template
+* profitbricks: profitbricks_snapshot
+* proxmox: proxmox
+* proxmox: proxmox_template
 * puppet
 * pushover
 * pushbullet
+* rax: rax_clb_ssl
 * rax: rax_mon_alarm
 * rax: rax_mon_check
 * rax: rax_mon_entity
@@ -211,6 +309,7 @@ newline being stripped you can change your playbook like this:
 * rabbitmq_exchange
 * rabbitmq_queue
 * selinux_permissive
+* sendgrid
 * sensu_check
 * sensu_subscription
 * seport
@@ -222,21 +321,24 @@ newline being stripped you can change your playbook like this:
 * vertica_role
 * vertica_schema
 * vertica_user
-* vmware: vmware_datacenter
+* vmware: vca_fw
+* vmware: vca_nat
 * vmware: vmware_cluster
+* vmware: vmware_datacenter
 * vmware: vmware_dns_config
 * vmware: vmware_dvs_host
 * vmware: vmware_dvs_portgroup
 * vmware: vmware_dvswitch
 * vmware: vmware_host
-* vmware: vmware_vmkernel_ip_config
+* vmware: vmware_migrate_vmk
 * vmware: vmware_portgroup
+* vmware: vmware_target_canonical_facts
 * vmware: vmware_vm_facts
+* vmware: vmware_vm_vss_dvs_migrate
 * vmware: vmware_vmkernel
+* vmware: vmware_vmkernel_ip_config
 * vmware: vmware_vsan_cluster
 * vmware: vmware_vswitch
-* vmware: vca_fw
-* vmware: vca_nat
 * vmware: vsphere_copy
 * webfaction_app
 * webfaction_db
@@ -244,17 +346,22 @@ newline being stripped you can change your playbook like this:
 * webfaction_mailbox
 * webfaction_site
 * win_acl
+* win_dotnet_ngen
 * win_environment
 * win_firewall_rule
-* win_package
-* win_scheduled_task
 * win_iis_virtualdirectory
 * win_iis_webapplication
 * win_iis_webapppool
 * win_iis_webbinding
 * win_iis_website
+* win_lineinfile
+* win_nssm
+* win_package
 * win_regedit
+* win_scheduled_task
 * win_unzip
+* win_updates
+* win_webpicmd
 * xenserver_facts
 * zabbix_host
 * zabbix_hostmacro
@@ -318,9 +425,21 @@ newline being stripped you can change your playbook like this:
 * Lookup, vars and action plugin pathing has been normalized, all now follow the same sequence to find relative files.
 * We do not ignore the explicitly set login user for ssh when it matches the 'current user' anymore, this allows overriding .ssh/config when it is set
   explicitly. Leaving it unset will still use the same user and respect .ssh/config. This also means ansible_ssh_user can now return a None value.
-* Handling of undefined variables has changed.  In most places they will now raise an error instead of silently injecting an empty string.  Use the default filter if you want to approximate the old behaviour::
+* environment variables passed to remote shells now default to 'controller' settings, with fallback to en_us.UTF8 which was the previous default.
+* add_hosts is much stricter about host name and will prevent invalid names from being added.
+* ansible-pull now defaults to doing shallow checkouts with git, use `--full` to return to previous behaviour.
+* random cows are more random
+* when: now gets the registered var after the first iteration, making it possible to break out of item loops
+* Handling of undefined variables has changed.  In most places they will now raise an error instead of silently injecting an empty string.  Use the default filter if you want to approximate the old behaviour:
 
+    ```
     - debug: msg="The error message was: {{error_code |default('') }}"
+    ```
+
+* The yum module's detection of installed packages has been made more robust by
+  using /usr/bin/rpm in cases where it woud have used repoquery before.
+* The pip module now properly reports changes when packages are coming from a VCS.
+* Fixes for retrieving files over https when a CONNECT-only proxy is in the middle.
 
 ## 1.9.4 "Dancing In the Street" - Oct 9, 2015
 

MANIFEST.in

@@ -4,12 +4,14 @@ prune ticket_stubs
 prune packaging
 prune test
 prune hacking
-include README.md packaging/rpm/ansible.spec COPYING
+include README.md COPYING
 include examples/hosts
 include examples/ansible.cfg
 include lib/ansible/module_utils/powershell.ps1
 recursive-include lib/ansible/modules *
+recursive-include lib/ansible/galaxy/data *
 recursive-include docs *
+recursive-include packaging *
 include Makefile
 include VERSION
 include MANIFEST.in

Makefile

@@ -44,7 +44,7 @@ GIT_HASH := $(shell git log -n 1 --format="%h")
 GIT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD | sed 's/[-_.\/]//g')
 GITINFO = .$(GIT_HASH).$(GIT_BRANCH)
 else
-GITINFO = ''
+GITINFO = ""
 endif
 
 ifeq ($(shell echo $(OS) | egrep -c 'Darwin|FreeBSD|OpenBSD'),1)
@@ -167,6 +167,9 @@ install:
 sdist: clean docs
 	$(PYTHON) setup.py sdist
 
+sdist_upload: clean docs
+	$(PYTHON) setup.py sdist upload 2>&1 |tee upload.log
+
 rpmcommon: $(MANPAGES) sdist
 	@mkdir -p rpm-build
 	@cp dist/*.gz rpm-build/

README.md

@@ -55,3 +55,4 @@ Ansible was created by [Michael DeHaan](https://github.com/mpdehaan) (michael.de
 
 Ansible is sponsored by [Ansible, Inc](http://ansible.com)
 
+

RELEASES.txt

@@ -4,11 +4,13 @@ Ansible Releases at a Glance
 Active Development
 ++++++++++++++++++
 
-2.0    "Over the Hills and Far Away" - in progress
+2.1    "TBD" - in progress
 
 Released
 ++++++++
 
+2.0.1  "Over the Hills and Far Away" TBD
+2.0.0  "Over the Hills and Far Away" 01-12-2015
 1.9.4  "Dancing In the Streets" 10-09-2015
 1.9.3  "Dancing In the Streets" 09-03-2015
 1.9.2  "Dancing In the Streets" 06-24-2015

VERSION

@@ -1 +1 @@
-2.0.0 0.5.beta3
+2.0.1.0 0.2.network

bin/ansible

@@ -40,6 +40,7 @@ from ansible.errors import AnsibleError, AnsibleOptionsError, AnsibleParserError
 from ansible.utils.display import Display
 from ansible.utils.unicode import to_unicode
 
+
 ########################################
 ### OUTPUT OF LAST RESORT ###
 class LastResort(object):
@@ -60,6 +61,7 @@ if __name__ == '__main__':
 
     try:
         display = Display()
+        display.debug("starting run")
 
         sub = None
         try:
@@ -107,7 +109,7 @@ if __name__ == '__main__':
         have_cli_options = cli is not None and cli.options is not None
         display.error("Unexpected Exception: %s" % to_unicode(e), wrap_text=False)
         if not have_cli_options or have_cli_options and cli.options.verbosity > 2:
-            display.display("the full traceback was:\n\n%s" % traceback.format_exc())
+            display.display(u"the full traceback was:\n\n%s" % to_unicode(traceback.format_exc()))
         else:
             display.display("to see the full traceback, use -vvv")
         sys.exit(250)

contrib/inventory/ec2.py

@@ -407,7 +407,7 @@ class Ec2Inventory(object):
                            help='Get all the variables about a specific instance')
         parser.add_argument('--refresh-cache', action='store_true', default=False,
                            help='Force refresh of cache by making API requests to EC2 (default: False - use cache files)')
-        parser.add_argument('--boto-profile', action='store',
+        parser.add_argument('--profile', '--boto-profile', action='store', dest='boto_profile',
                            help='Use boto profile for connections to EC2')
         self.args = parser.parse_args()
 

contrib/inventory/libvirt_lxc.py

@@ -27,11 +27,11 @@ result['all'] = {}
 pipe = Popen(['virsh', '-q', '-c', 'lxc:///', 'list', '--name', '--all'], stdout=PIPE, universal_newlines=True)
 result['all']['hosts'] = [x[:-1] for x in pipe.stdout.readlines()]
 result['all']['vars'] = {}
-result['all']['vars']['ansible_connection'] = 'lxc'
+result['all']['vars']['ansible_connection'] = 'libvirt_lxc'
 
 if len(sys.argv) == 2 and sys.argv[1] == '--list':
     print(json.dumps(result))
 elif len(sys.argv) == 3 and sys.argv[1] == '--host':
-    print(json.dumps({'ansible_connection': 'lxc'}))
+    print(json.dumps({'ansible_connection': 'libvirt_lxc'}))
 else:
     print("Need an argument, either --list or --host <host>")

contrib/inventory/openstack.py

@@ -32,6 +32,13 @@
 # all of them and present them as one contiguous inventory.
 #
 # See the adjacent openstack.yml file for an example config file
+# There are two ansible inventory specific options that can be set in
+# the inventory section.
+# expand_hostvars controls whether or not the inventory will make extra API
+#                 calls to fill out additional information about each server
+# use_hostnames changes the behavior from registering every host with its UUID
+#               and making a group of its hostname to only doing this if the
+#               hostname in question has more than one server
 
 import argparse
 import collections
@@ -51,7 +58,7 @@ import shade.inventory
 CONFIG_FILES = ['/etc/ansible/openstack.yaml']
 
 
-def get_groups_from_server(server_vars):
+def get_groups_from_server(server_vars, namegroup=True):
     groups = []
 
     region = server_vars['region']
@@ -76,7 +83,8 @@ def get_groups_from_server(server_vars):
             groups.append(extra_group)
 
     groups.append('instance-%s' % server_vars['id'])
-    groups.append(server_vars['name'])
+    if namegroup:
+        groups.append(server_vars['name'])
 
     for key in ('flavor', 'image'):
         if 'name' in server_vars[key]:
@@ -94,9 +102,9 @@ def get_groups_from_server(server_vars):
     return groups
 
 
-def get_host_groups(inventory):
+def get_host_groups(inventory, refresh=False):
     (cache_file, cache_expiration_time) = get_cache_settings()
-    if is_cache_stale(cache_file, cache_expiration_time):
+    if is_cache_stale(cache_file, cache_expiration_time, refresh=refresh):
         groups = to_json(get_host_groups_from_cloud(inventory))
         open(cache_file, 'w').write(groups)
     else:
@@ -106,23 +114,44 @@ def get_host_groups(inventory):
 
 def get_host_groups_from_cloud(inventory):
     groups = collections.defaultdict(list)
+    firstpass = collections.defaultdict(list)
     hostvars = {}
-    for server in inventory.list_hosts():
+    list_args = {}
+    if hasattr(inventory, 'extra_config'):
+        use_hostnames = inventory.extra_config['use_hostnames']
+        list_args['expand'] = inventory.extra_config['expand_hostvars']
+    else:
+        use_hostnames = False
+
+    for server in inventory.list_hosts(**list_args):
 
         if 'interface_ip' not in server:
             continue
-        for group in get_groups_from_server(server):
-            groups[group].append(server['id'])
-        hostvars[server['id']] = dict(
-            ansible_ssh_host=server['interface_ip'],
-            openstack=server,
-        )
+        firstpass[server['name']].append(server)
+    for name, servers in firstpass.items():
+        if len(servers) == 1 and use_hostnames:
+            server = servers[0]
+            hostvars[name] = dict(
+                ansible_ssh_host=server['interface_ip'],
+                openstack=server)
+            for group in get_groups_from_server(server, namegroup=False):
+                groups[group].append(server['name'])
+        else:
+            for server in servers:
+                server_id = server['id']
+                hostvars[server_id] = dict(
+                    ansible_ssh_host=server['interface_ip'],
+                    openstack=server)
+                for group in get_groups_from_server(server, namegroup=True):
+                    groups[group].append(server_id)
     groups['_meta'] = {'hostvars': hostvars}
     return groups
 
 
-def is_cache_stale(cache_file, cache_expiration_time):
+def is_cache_stale(cache_file, cache_expiration_time, refresh=False):
     ''' Determines if cache file has expired, or if it is still valid '''
+    if refresh:
+        return True
     if os.path.isfile(cache_file):
         mod_time = os.path.getmtime(cache_file)
         current_time = time.time()
@@ -169,14 +198,24 @@ def main():
     try:
         config_files = os_client_config.config.CONFIG_FILES + CONFIG_FILES
         shade.simple_logging(debug=args.debug)
-        inventory = shade.inventory.OpenStackInventory(
+        inventory_args = dict(
             refresh=args.refresh,
             config_files=config_files,
             private=args.private,
         )
+        if hasattr(shade.inventory.OpenStackInventory, 'extra_config'):
+            inventory_args.update(dict(
+                config_key='ansible',
+                config_defaults={
+                    'use_hostnames': False,
+                    'expand_hostvars': True,
+                }
+            ))
+
+        inventory = shade.inventory.OpenStackInventory(**inventory_args)
 
         if args.list:
-            output = get_host_groups(inventory)
+            output = get_host_groups(inventory, refresh=args.refresh)
         elif args.host:
             output = to_json(inventory.get_host(args.host))
         print(output)

contrib/inventory/openstack.yml

@@ -26,3 +26,6 @@ clouds:
       username: stack
       password: stack
       project_name: stack
+ansible:
+  use_hostnames: True
+  expand_hostvars: False

docs/man/man1/ansible-galaxy.1.asciidoc.in

@@ -12,7 +12,7 @@ ansible-galaxy - manage roles using galaxy.ansible.com
 
 SYNOPSIS
 --------
-ansible-galaxy [init|info|install|list|remove] [--help] [options] ...
+ansible-galaxy [delete|import|info|init|install|list|login|remove|search|setup] [--help] [options] ...
 
 
 DESCRIPTION
@@ -20,7 +20,7 @@ DESCRIPTION
 
 *Ansible Galaxy* is a shared repository for Ansible roles.
 The ansible-galaxy command can be used to manage these roles,
-or by creating a skeleton framework for roles you'd like to upload to Galaxy.
+or for creating a skeleton framework for roles you'd like to upload to Galaxy.
 
 COMMON OPTIONS
 --------------
@@ -29,7 +29,6 @@ COMMON OPTIONS
 
 Show a help message related to the given sub-command.
 
-
 INSTALL
 -------
 
@@ -145,6 +144,204 @@ The path to the directory containing your roles. The default is the *roles_path*
 configured in your *ansible.cfg* file (/etc/ansible/roles if not configured)
 
 
+SEARCH
+------
+
+The *search* sub-command returns a filtered list of roles found on the remote
+server.
+
+
+USAGE
+~~~~~
+
+$ ansible-galaxy search [options] [searchterm1 searchterm2]
+
+
+OPTIONS
+~~~~~~~
+*--galaxy-tags*::
+
+Provide a comma separated list of Galaxy Tags on which to filter.
+
+*--platforms*::
+
+Provide a comma separated list of Platforms on which to filter.
+
+*--author*::
+
+Specify the username of a Galaxy contributor on which to filter.
+
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+
+INFO
+----
+
+The *info* sub-command shows detailed information for a specific role.
+Details returned about the role included information from the local copy
+as well as information from galaxy.ansible.com.
+
+USAGE
+~~~~~
+
+$ ansible-galaxy info [options] role_name[, version]
+
+OPTIONS
+~~~~~~~
+
+*-p* 'ROLES_PATH', *--roles-path=*'ROLES_PATH'::
+
+The path to the directory containing your roles. The default is the *roles_path* 
+configured in your *ansible.cfg* file (/etc/ansible/roles if not configured)
+
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+
+LOGIN
+-----
+
+The *login* sub-command is used to authenticate with galaxy.ansible.com. 
+Authentication is required to use the import, delete and setup commands.
+It will authenticate the user, retrieve a token from Galaxy, and store it
+in the user's home directory.
+
+USAGE
+~~~~~
+
+$ ansible-galaxy login [options]
+
+The *login* sub-command prompts for a *GitHub* username and password. It does
+NOT send your password to Galaxy. It actually authenticates with GitHub and
+creates a personal access token. It then sends the personal access token to
+Galaxy, which in turn verifies that you are you and returns a Galaxy access
+token. After authentication completes the *GitHub* personal access token is
+destroyed. 
+
+If you do not wish to use your GitHub password, or if you have two-factor
+authentication enabled with GitHub, use the *--github-token* option to pass a
+personal access token that you create. Log into GitHub, go to Settings and
+click on Personal Access Token to create a token.
+
+OPTIONS
+~~~~~~~
+
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+*--github-token*::
+
+Authenticate using a *GitHub* personal access token rather than a password.
+
+
+IMPORT
+------
+
+Import a role from *GitHub* to galaxy.ansible.com. Requires the user first
+authenticate with galaxy.ansible.com using the *login* subcommand.
+
+USAGE
+~~~~~
+
+$ ansible-galaxy import [options] github_user github_repo
+
+OPTIONS
+~~~~~~~
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+*--branch*::
+
+Provide a specific branch to import. When a branch is not specified the
+branch found in meta/main.yml is used. If no branch is specified in
+meta/main.yml, the repo's default branch (usually master) is used.
+
+
+DELETE
+------
+
+The *delete* sub-command will delete a role from galaxy.ansible.com. Requires
+the user first authenticate with galaxy.ansible.com using the *login* subcommand.
+
+USAGE
+~~~~~
+
+$ ansible-galaxy delete [options] github_user github_repo
+
+OPTIONS
+~~~~~~~
+
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+
+SETUP
+-----
+
+The *setup* sub-command creates an integration point for *Travis CI*, enabling 
+galaxy.ansible.com to receive notifications from *Travis* on build completion.
+Requires the user first authenticate with galaxy.ansible.com using the *login*
+subcommand.
+
+USAGE
+~~~~~
+
+$ ansible-galaxy setup [options] source github_user github_repo secret
+
+* Use *travis* as the source value. In the future additional source values may
+  be added.
+
+* Provide your *Travis* user token as the secret. The token is not stored by
+  galaxy.ansible.com. A hash is created using github_user, github_repo
+  and your token. The hash value is what actually gets stored.
+
+OPTIONS
+~~~~~~~
+
+*-c*, *--ignore-certs*::
+
+Ignore TLS certificate errors. 
+
+*-s*, *--server*::
+
+Override the default server https://galaxy.ansible.com.
+
+--list::
+
+Show your configured integrations. Provids the ID of each integration
+which can be used with the remove option.
+
+--remove::
+
+Remove a specific integration. Provide the ID of the integration to 
+be removed.
+
 AUTHOR
 ------
 

docs/man/man1/ansible-playbook.1.asciidoc.in

@@ -96,7 +96,7 @@ Show help page and exit
 *-i* 'PATH', *--inventory=*'PATH'::
 
 The 'PATH' to the inventory, which defaults to '/etc/ansible/hosts'.
-Alternatively you can use a comma separated list of hosts or single host with traling comma 'host,'.
+Alternatively, you can use a comma-separated list of hosts or a single host with a trailing comma 'host,'.
 
 *-l* 'SUBSET', *--limit=*'SUBSET'::
 

docs/man/man1/ansible-pull.1.asciidoc.in

@@ -95,6 +95,10 @@ Force running of playbook even if unable to update playbook repository.  This
 can be useful, for example, to enforce run-time state when a network
 connection may not always be up or possible.
 
+*--full*::
+
+Do a full clone of the repository. By default ansible-pull will do a shallow clone based on the last revision.
+
 *-h*, *--help*::
 
 Show the help message and exit.

docsite/Makefile

@@ -20,6 +20,8 @@ viewdocs: clean staticmin
 htmldocs: staticmin
 	./build-site.py rst
 
+webdocs: htmldocs
+
 clean:
 	-rm -rf htmlout
 	-rm -f .buildinfo

docsite/build-site.py

@@ -15,6 +15,7 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+from __future__ import print_function
 
 __docformat__ = 'restructuredtext'
 
@@ -24,9 +25,9 @@ import traceback
 try:
     from sphinx.application import Sphinx
 except ImportError:
-    print "#################################"
-    print "Dependency missing: Python Sphinx"
-    print "#################################"
+    print("#################################")
+    print("Dependency missing: Python Sphinx")
+    print("#################################")
     sys.exit(1)
 import os
 
@@ -40,7 +41,7 @@ class SphinxBuilder(object):
         """
         Run the DocCommand.
         """
-        print "Creating html documentation ..."
+        print("Creating html documentation ...")
 
         try:
             buildername = 'html'
@@ -69,10 +70,10 @@ class SphinxBuilder(object):
 
             app.builder.build_all()
 
-        except ImportError, ie:
+        except ImportError:
             traceback.print_exc()
-        except Exception, ex:
-            print >> sys.stderr, "FAIL! exiting ... (%s)" % ex
+        except Exception as ex:
+            print("FAIL! exiting ... (%s)" % ex, file=sys.stderr)
 
     def build_docs(self):
         self.app.builder.build_all()
@@ -83,9 +84,9 @@ def build_rst_docs():
 
 if __name__ == '__main__':
     if '-h' in sys.argv or '--help' in sys.argv:
-        print "This script builds the html documentation from rst/asciidoc sources.\n"
-        print "    Run 'make docs' to build everything."
-        print "    Run 'make viewdocs' to build and then preview in a web browser."
+        print("This script builds the html documentation from rst/asciidoc sources.\n")
+        print("    Run 'make docs' to build everything.")
+        print("    Run 'make viewdocs' to build and then preview in a web browser.")
         sys.exit(0)
 
     build_rst_docs()
@@ -93,4 +94,4 @@ if __name__ == '__main__':
     if "view" in sys.argv:
         import webbrowser
         if not webbrowser.open('htmlout/index.html'):
-            print >> sys.stderr, "Could not open on your webbrowser."
+            print("Could not open on your webbrowser.", file=sys.stderr)

docsite/rst/become.rst

@@ -1,5 +1,5 @@
-Ansible Privilege Escalation
-++++++++++++++++++++++++++++
+Become (Privilege Escalation)
++++++++++++++++++++++++++++++
 
 Ansible can use existing privilege escalation systems to allow a user to execute tasks as another.
 
@@ -7,17 +7,17 @@ Ansible can use existing privilege escalation systems to allow a user to execute
 
 Become
 ``````
-Before 1.9 Ansible mostly allowed the use of sudo and a limited use of su to allow a login/remote user to become a different user
-and execute tasks, create resources with the 2nd user's permissions. As of 1.9 'become' supersedes the old sudo/su, while still
-being backwards compatible. This new system also makes it easier to add other privilege escalation tools like pbrun (Powerbroker),
-pfexec and others.
+Before 1.9 Ansible mostly allowed the use of `sudo` and a limited use of `su` to allow a login/remote user to become a different user
+and execute tasks, create resources with the 2nd user's permissions. As of 1.9 `become` supersedes the old sudo/su, while still
+being backwards compatible. This new system also makes it easier to add other privilege escalation tools like `pbrun` (Powerbroker),
+`pfexec` and others.
 
 
 New directives
 --------------
 
 become
-    equivalent to adding 'sudo:' or 'su:' to a play or task, set to 'true'/'yes' to activate privilege escalation
+    equivalent to adding `sudo:` or `su:` to a play or task, set to 'true'/'yes' to activate privilege escalation
 
 become_user
     equivalent to adding 'sudo_user:' or 'su_user:' to a play or task, set to user with desired privileges

docsite/rst/developing.rst

@@ -11,6 +11,7 @@ Learn how to build modules of your own in any language, and also how to extend A
    developing_modules
    developing_plugins
    developing_test_pr
+   developing_releases
 
 Developers will also likely be interested in the fully-discoverable in :doc:`tower`.  It's great for embedding Ansible in all manner of applications.
 

docsite/rst/developing_api.rst

@@ -17,11 +17,67 @@ This chapter discusses the Python API.
 
 .. _python_api:
 
-Python API
-----------
-
 The Python API is very powerful, and is how the ansible CLI and ansible-playbook
-are implemented.
+are implemented. In version 2.0 the core ansible got rewritten and the API was mostly rewritten.
+
+.. _python_api_20:
+
+Python API 2.0
+--------------
+
+In 2.0 things get a bit more complicated to start, but you end up with much more discrete and readable classes::
+
+
+    #!/usr/bin/python2
+
+    from collections import namedtuple
+    from ansible.parsing.dataloader import DataLoader
+    from ansible.vars import VariableManager
+    from ansible.inventory import Inventory
+    from ansible.playbook.play import Play
+    from ansible.executor.task_queue_manager import TaskQueueManager
+
+    Options = namedtuple('Options', ['connection','module_path', 'forks', 'remote_user', 'private_key_file', 'ssh_common_args', 'ssh_extra_args', 'sftp_extra_args', 'scp_extra_args', 'become', 'become_method', 'become_user', 'verbosity', 'check'])
+    # initialize needed objects
+    variable_manager = VariableManager()
+    loader = DataLoader()
+    options = Options(connection='local', module_path='/path/to/mymodules', forks=100, remote_user=None, private_key_file=None, ssh_common_args=None, ssh_extra_args=None, sftp_extra_args=None, scp_extra_args=None, become=None, become_method=None, become_user=None, verbosity=None, check=False)
+    passwords = dict(vault_pass='secret')
+
+    # create inventory and pass to var manager
+    inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list='localhost')
+    variable_manager.set_inventory(inventory)
+
+    # create play with tasks
+    play_source =  dict(
+            name = "Ansible Play",
+            hosts = 'localhost',
+            gather_facts = 'no',
+            tasks = [ dict(action=dict(module='debug', args=(msg='Hello Galaxy!'))) ]
+        )
+    play = Play().load(play_source, variable_manager=variable_manager, loader=loader)
+
+    # actually run it
+    tqm = None
+    try:
+        tqm = TaskQueueManager(
+                  inventory=inventory,
+                  variable_manager=variable_manager,
+                  loader=loader,
+                  options=options,
+                  passwords=passwords,
+                  stdout_callback='default',
+              )
+        result = tqm.run(play)
+    finally:
+        if tqm is not None:
+            tqm.cleanup()
+
+
+.. _python_api_old:
+
+Python API pre 2.0
+------------------
 
 It's pretty simple::
 
@@ -51,7 +107,7 @@ expressed in the :doc:`modules` documentation.::
 A module can return any type of JSON data it wants, so Ansible can
 be used as a framework to rapidly build powerful applications and scripts.
 
-.. _detailed_api_example:
+.. _detailed_api_old_example:
 
 Detailed API Example
 ````````````````````
@@ -87,9 +143,9 @@ The following script prints out the uptime information for all hosts::
     for (hostname, result) in results['dark'].items():
         print "%s >>> %s" % (hostname, result)
 
-Advanced programmers may also wish to read the source to ansible itself, for
-it uses the Runner() API (with all available options) to implement the
-command line tools ``ansible`` and ``ansible-playbook``.
+Advanced programmers may also wish to read the source to ansible itself,
+for it uses the API (with all available options) to implement the ``ansible``
+command line tools (``lib/ansible/cli/``).
 
 .. seealso::
 

docsite/rst/developing_modules.rst

@@ -219,7 +219,7 @@ this, just have the module return a `ansible_facts` key, like so, along with oth
     }
 
 These 'facts' will be available to all statements called after that module (but not before) in the playbook.
-A good idea might be make a module called 'site_facts' and always call it at the top of each playbook, though
+A good idea might be to make a module called 'site_facts' and always call it at the top of each playbook, though
 we're always open to improving the selection of core facts in Ansible as well.
 
 .. _common_module_boilerplate:

docsite/rst/galaxy.rst

@@ -1,55 +1,60 @@
 Ansible Galaxy
 ++++++++++++++
 
-"Ansible Galaxy" can either refer to a website for sharing and downloading Ansible roles, or a command line tool that helps work with roles.
+"Ansible Galaxy" can either refer to a website for sharing and downloading Ansible roles, or a command line tool for managing and creating roles.
 
 .. contents:: Topics
 
 The Website
 ```````````
 
-The website `Ansible Galaxy <https://galaxy.ansible.com>`_, is a free site for finding, downloading, rating, and reviewing all kinds of community developed Ansible roles and can be a great way to get a jumpstart on your automation projects.
+The website `Ansible Galaxy <https://galaxy.ansible.com>`_, is a free site for finding, downloading, and sharing community developed Ansible roles. Downloading roles from Galaxy is a great way to jumpstart your automation projects.
 
-You can sign up with social auth and use the download client 'ansible-galaxy' which is included in Ansible 1.4.2 and later.
+Access the Galaxy web site using GitHub OAuth, and to install roles use the 'ansible-galaxy' command line tool included in Ansible 1.4.2 and later.
 
 Read the "About" page on the Galaxy site for more information.
 
 The ansible-galaxy command line tool
 ````````````````````````````````````
 
-The command line ansible-galaxy has many different subcommands.
+The ansible-galaxy command has many different sub-commands for managing roles both locally and at `galaxy.ansible.com <https://galaxy.ansible.com>`_.
+
+.. note::
+
+    The search, login, import, delete, and setup commands in the Ansible 2.0 version of ansible-galaxy require access to the 
+    2.0 Beta release of the Galaxy web site available at `https://galaxy-qa.ansible.com <https://galaxy-qa.ansible.com>`_.
+
+    Use the ``--server`` option to access the beta site. For example::
+
+        $ ansible-galaxy search --server https://galaxy-qa.ansible.com mysql --author geerlingguy
+
+    Additionally, you can define a server in ansible.cfg::
+
+        [galaxy]
+        server=https://galaxy-qa.ansible.com
 
 Installing Roles
 ----------------
 
-The most obvious is downloading roles from the Ansible Galaxy website::
+The most obvious use of the ansible-galaxy command is downloading roles from `the Ansible Galaxy website <https://galaxy.ansible.com>`_::
 
-   ansible-galaxy install username.rolename
-
-.. _galaxy_cli_roles_path:
+   $ ansible-galaxy install username.rolename
 
 roles_path
-===============
+==========
 
 You can specify a particular directory where you want the downloaded roles to be placed::
 
-   ansible-galaxy install username.role -p ~/Code/ansible_roles/
+   $ ansible-galaxy install username.role -p ~/Code/ansible_roles/
    
 This can be useful if you have a master folder that contains ansible galaxy roles shared across several projects. The default is the roles_path configured in your ansible.cfg file (/etc/ansible/roles if not configured).
 
-Building out Role Scaffolding
------------------------------
-
-It can also be used to initialize the base structure of a new role, saving time on creating the various directories and main.yml files a role requires::
-
-   ansible-galaxy init rolename
-
 Installing Multiple Roles From A File
--------------------------------------
+=====================================
 
 To install multiple roles, the ansible-galaxy CLI can be fed a requirements file.  All versions of ansible allow the following syntax for installing roles from the Ansible Galaxy website::
 
-   ansible-galaxy install -r requirements.txt
+   $ ansible-galaxy install -r requirements.txt
 
 Where the requirements.txt looks like::
 
@@ -64,7 +69,7 @@ To request specific versions (tags) of a role, use this syntax in the roles file
 Available versions will be listed on the Ansible Galaxy webpage for that role.
 
 Advanced Control over Role Requirements Files
----------------------------------------------
+=============================================
 
 For more advanced control over where to download roles from, including support for remote repositories, Ansible 1.8 and later support a new YAML format for the role requirements file, which must end in a 'yml' extension.  It works like this::
 
@@ -77,14 +82,14 @@ And here's an example showing some specific version downloads from multiple sour
     # from galaxy
     - src: yatesr.timezone
 
-    # from github
+    # from GitHub
     - src: https://github.com/bennojoy/nginx
 
-    # from github installing to a relative path
+    # from GitHub installing to a relative path
     - src: https://github.com/bennojoy/nginx
       path: vagrant/roles/
 
-    # from github, overriding the name and specifying a specific tag
+    # from GitHub, overriding the name and specifying a specific tag
     - src: https://github.com/bennojoy/nginx
       version: master
       name: nginx_role
@@ -93,15 +98,15 @@ And here's an example showing some specific version downloads from multiple sour
     - src: https://some.webserver.example.com/files/master.tar.gz
       name: http-role
 
-    # from bitbucket, if bitbucket happens to be operational right now :)
+    # from Bitbucket
     - src: git+http://bitbucket.org/willthames/git-ansible-galaxy
       version: v1.4
 
-    # from bitbucket, alternative syntax and caveats
+    # from Bitbucket, alternative syntax and caveats
     - src: http://bitbucket.org/willthames/hg-ansible-galaxy
       scm: hg
    
-   # from gitlab or other git-based scm   
+   # from GitLab or other git-based scm   
     - src: git@gitlab.company.com:mygroup/ansible-base.git
       scm: git
       version: 0.1.0
@@ -121,3 +126,283 @@ Roles pulled from galaxy work as with other SCM sourced roles above. To download
    `irc.freenode.net <http://irc.freenode.net>`_
        #ansible IRC chat channel
 
+Building Role Scaffolding
+-------------------------
+
+Use the init command to initialize the base structure of a new role, saving time on creating the various directories and main.yml files a role requires::
+
+   $ ansible-galaxy init rolename
+
+The above will create the following directory structure in the current working directory:
+  
+::
+
+   README.md
+   .travis.yml
+   defaults/
+       main.yml
+   files/
+   handlers/
+       main.yml
+   meta/
+       main.yml
+   templates/
+   tests/
+       inventory
+       test.yml
+   vars/
+       main.yml
+
+.. note::
+
+    .travis.yml and tests/ are new in Ansible 2.0
+
+If a directory matching the name of the role already exists in the current working directory, the init command will result in an error. To ignore the error use the --force option. Force will create the above subdirectories and files, replacing anything that matches.
+
+Search for Roles
+----------------
+
+The search command provides for querying the Galaxy database, allowing for searching by tags, platforms, author and multiple keywords. For example:
+
+::
+
+    $ ansible-galaxy search elasticsearch --author geerlingguy
+
+The search command will return a list of the first 1000 results matching your search:
+
+::
+    
+    Found 2 roles matching your search:
+
+    Name                              Description
+    ----                              -----------
+    geerlingguy.elasticsearch         Elasticsearch for Linux.
+    geerlingguy.elasticsearch-curator Elasticsearch curator for Linux.
+
+.. note::
+
+   The format of results pictured here is new in Ansible 2.0.
+
+Get More Information About a Role
+---------------------------------
+
+Use the info command To view more detail about a specific role:
+
+::
+
+    $ ansible-galaxy info username.role_name
+
+This returns everything found in Galaxy for the role:
+
+::
+
+    Role: username.rolename
+        description: Installs and configures a thing, a distributed, highly available NoSQL thing.
+        active: True
+        commit: c01947b7bc89ebc0b8a2e298b87ab416aed9dd57
+        commit_message: Adding travis
+        commit_url: https://github.com/username/repo_name/commit/c01947b7bc89ebc0b8a2e298b87ab
+        company: My Company, Inc.
+        created: 2015-12-08T14:17:52.773Z
+        download_count: 1
+        forks_count: 0
+        github_branch:
+        github_repo: repo_name
+        github_user: username
+        id: 6381
+        is_valid: True
+        issue_tracker_url:
+        license: Apache
+        min_ansible_version: 1.4
+        modified: 2015-12-08T18:43:49.085Z
+        namespace: username
+        open_issues_count: 0
+        path: /Users/username/projects/roles
+        scm: None
+        src: username.repo_name
+        stargazers_count: 0
+        travis_status_url: https://travis-ci.org/username/repo_name.svg?branch=master
+        version:
+        watchers_count: 1
+
+
+List Installed Roles
+--------------------
+
+The list command shows the name and version of each role installed in roles_path.
+
+::
+
+    $ ansible-galaxy list
+
+    - chouseknecht.role-install_mongod, master
+    - chouseknecht.test-role-1, v1.0.2
+    - chrismeyersfsu.role-iptables, master
+    - chrismeyersfsu.role-required_vars, master
+
+Remove an Installed Role
+------------------------
+
+The remove command will delete a role from roles_path:
+
+::
+
+    $ ansible-galaxy remove username.rolename
+
+Authenticate with Galaxy
+------------------------
+
+To use the import, delete and setup commands authentication with Galaxy is required. The login command will authenticate the user,retrieve a token from Galaxy, and store it in the user's home directory.
+
+::
+
+    $ ansible-galaxy login
+
+    We need your Github login to identify you.
+    This information will not be sent to Galaxy, only to api.github.com.
+    The password will not be displayed.
+
+    Use --github-token if you do not want to enter your password.
+
+    Github Username: dsmith
+    Password for dsmith:
+    Succesfully logged into Galaxy as dsmith
+
+As depicted above, the login command prompts for a GitHub username and password. It does NOT send your password to Galaxy. It actually authenticates with GitHub and creates a personal access token. It then sends the personal access token to Galaxy, which in turn verifies that you are you and returns a Galaxy access token. After authentication completes the GitHub personal access token is destroyed. 
+
+If you do not wish to use your GitHub password, or if you have two-factor authentication enabled with GitHub, use the --github-token option to pass a personal access token that you create. Log into GitHub, go to Settings and click on Personal Access Token to create a token.
+
+.. note::
+
+    The login command in Ansible 2.0 requires using the Galaxy 2.0 Beta site. Use the ``--server`` option to access 
+    `https://galaxy-qa.ansible.com <https://galaxy-qa.ansible.com>`_. You can also add a *server* definition in the [galaxy] 
+    section of your ansible.cfg file.
+
+Import a Role
+-------------
+
+Roles can be imported using ansible-galaxy. The import command expects that the user previously authenticated with Galaxy using the login command.
+
+Import any GitHub repo you have access to:
+
+::
+
+    $ ansible-galaxy import github_user github_repo
+
+By default the command will wait for the role to be imported by Galaxy, displaying the results as the import progresses:
+
+::
+
+    Successfully submitted import request 41
+    Starting import 41: role_name=myrole repo=githubuser/ansible-role-repo ref=
+    Retrieving Github repo githubuser/ansible-role-repo
+    Accessing branch: master
+    Parsing and validating meta/main.yml
+    Parsing galaxy_tags
+    Parsing platforms
+    Adding dependencies
+    Parsing and validating README.md
+    Adding repo tags as role versions
+    Import completed
+    Status SUCCESS : warnings=0 errors=0
+
+Use the --branch option to import a specific branch. If not specified, the default branch for the repo will be used.
+
+If the --no-wait option is present, the command will not wait for results. Results of the most recent import for any of your roles is available on the Galaxy web site under My Imports.
+
+.. note::
+
+    The import command in Ansible 2.0 requires using the Galaxy 2.0 Beta site. Use the ``--server`` option to access 
+    `https://galaxy-qa.ansible.com <https://galaxy-qa.ansible.com>`_. You can also add a *server* definition in the [galaxy] 
+    section of your ansible.cfg file.
+
+Delete a Role
+-------------
+
+Remove a role from the Galaxy web site using the delete command.  You can delete any role that you have access to in GitHub. The delete command expects that the user previously authenticated with Galaxy using the login command.
+
+::
+
+    $ ansible-galaxy delete github_user github_repo
+
+This only removes the role from Galaxy. It does not impact the actual GitHub repo.
+
+.. note::
+
+    The delete command in Ansible 2.0 requires using the Galaxy 2.0 Beta site. Use the ``--server`` option to access 
+    `https://galaxy-qa.ansible.com <https://galaxy-qa.ansible.com>`_. You can also add a *server* definition in the [galaxy] 
+    section of your ansible.cfg file.
+
+Setup Travis Integerations
+--------------------------
+
+Using the setup command you can enable notifications from `travis <http://travis-ci.org>`_. The setup command expects that the user previously authenticated with Galaxy using the login command.
+
+::
+
+    $ ansible-galaxy setup travis github_user github_repo xxxtravistokenxxx
+
+    Added integration for travis github_user/github_repo 
+
+The setup command requires your Travis token. The Travis token is not stored in Galaxy. It is used along with the GitHub username and repo to create a hash as described in `the Travis documentation <https://docs.travis-ci.com/user/notifications/>`_. The calculated hash is stored in Galaxy and used to verify notifications received from Travis.
+
+The setup command enables Galaxy to respond to notifications. Follow the `Travis getting started guide <https://docs.travis-ci.com/user/getting-started/>`_ to enable the Travis build process for the role repository.
+
+When you create your .travis.yml file add the following to cause Travis to notify Galaxy when a build completes:
+
+::
+
+    notifications:
+        webhooks: https://galaxy.ansible.com/api/v1/notifications/
+
+.. note::
+
+    The setup command in Ansible 2.0 requires using the Galaxy 2.0 Beta site. Use the ``--server`` option to access 
+    `https://galaxy-qa.ansible.com <https://galaxy-qa.ansible.com>`_. You can also add a *server* definition in the [galaxy] 
+    section of your ansible.cfg file.
+
+
+List Travis Integrations
+========================
+
+Use the --list option to display your Travis integrations:
+
+::
+
+    $ ansible-galaxy setup --list
+
+
+    ID         Source     Repo
+    ---------- ---------- ----------
+    2          travis     github_user/github_repo
+    1          travis     github_user/github_repo
+
+
+Remove Travis Integrations
+==========================
+
+Use the --remove option to disable and remove a Travis integration:
+
+::
+
+    $ ansible-galaxy setup --remove ID
+
+Provide the ID of the integration you want disabled. Use the --list option to get the ID.
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+
+
+
+
+

docsite/rst/guide_cloudstack.rst

@@ -178,8 +178,8 @@ Now to the fun part. We create a playbook to create our infrastructure we call i
 
         - name: ensure firewall ports opened
           cs_firewall:
-            ip_address: {{ public_ip }}
-            port: {{ item.port }}
+            ip_address: "{{ public_ip }}"
+            port: "{{ item.port }}"
             cidr: "{{ item.cidr | default('0.0.0.0/0') }}"
           with_items: cs_firewall
           when: public_ip is defined

docsite/rst/guide_vagrant.rst

@@ -6,12 +6,13 @@ Using Vagrant and Ansible
 Introduction
 ````````````
 
-Vagrant is a tool to manage virtual machine environments, and allows you to
-configure and use reproducible work environments on top of various
-virtualization and cloud platforms. It also has integration with Ansible as a
-provisioner for these virtual machines, and the two tools work together well.
+`Vagrant <http://vagrantup.com/>`_ is a tool to manage virtual machine
+environments, and allows you to configure and use reproducible work
+environments on top of various virtualization and cloud platforms.
+It also has integration with Ansible as a provisioner for these virtual
+machines, and the two tools work together well.
 
-This guide will describe how to use Vagrant and Ansible together.
+This guide will describe how to use Vagrant 1.7+ and Ansible together.
 
 If you're not familiar with Vagrant, you should visit `the documentation
 <http://docs.vagrantup.com/v2/>`_.
@@ -27,54 +28,48 @@ Vagrant Setup
 
 The first step once you've installed Vagrant is to create a ``Vagrantfile``
 and customize it to suit your needs. This is covered in detail in the Vagrant
-documentation, but here is a quick example:
-
-.. code-block:: bash
-
-    $ mkdir vagrant-test
-    $ cd vagrant-test
-    $ vagrant init precise32 http://files.vagrantup.com/precise32.box
-
-This will create a file called Vagrantfile that you can edit to suit your
-needs. The default Vagrantfile has a lot of comments. Here is a simplified
-example that includes a section to use the Ansible provisioner:
+documentation, but here is a quick example that includes a section to use the
+Ansible provisioner to manage a single machine:
 
 .. code-block:: ruby
 
-    # Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
-    VAGRANTFILE_API_VERSION = "2"
-    
-    Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
-        config.vm.box = "precise32"
-        config.vm.box_url = "http://files.vagrantup.com/precise32.box"
-        
-        config.vm.network :public_network
+  # This guide is optimized for Vagrant 1.7 and above.
+  # Although versions 1.6.x should behave very similarly, it is recommended
+  # to upgrade instead of disabling the requirement below.
+  Vagrant.require_version ">= 1.7.0"
 
-        config.vm.provision "ansible" do |ansible|
-            ansible.playbook = "playbook.yml"
-        end
+  Vagrant.configure(2) do |config|
+
+    config.vm.box = "ubuntu/trusty64"
+
+    # Disable the new default behavior introduced in Vagrant 1.7, to
+    # ensure that all Vagrant machines will use the same SSH key pair.
+    # See https://github.com/mitchellh/vagrant/issues/5005
+    config.ssh.insert_key = false
+
+    config.vm.provision "ansible" do |ansible|
+      ansible.verbose = "v"
+      ansible.playbook = "playbook.yml"
     end
+  end
 
-The Vagrantfile has a lot of options, but these are the most important ones.
 Notice the ``config.vm.provision`` section that refers to an Ansible playbook
-called ``playbook.yml`` in the same directory as the Vagrantfile. Vagrant runs
-the provisioner once the virtual machine has booted and is ready for SSH
+called ``playbook.yml`` in the same directory as the ``Vagrantfile``. Vagrant
+runs the provisioner once the virtual machine has booted and is ready for SSH
 access.
 
+There are a lot of Ansible options you can configure in your ``Vagrantfile``.
+Visit the `Ansible Provisioner documentation
+<http://docs.vagrantup.com/v2/provisioning/ansible.html>`_ for more
+information.
+
 .. code-block:: bash
 
     $ vagrant up
 
-This will start the VM and run the provisioning playbook.
+This will start the VM, and run the provisioning playbook (on the first VM
+startup).
 
-There are a lot of Ansible options you can configure in your Vagrantfile. Some
-particularly useful options are ``ansible.extra_vars``, ``ansible.sudo`` and
-``ansible.sudo_user``, and ``ansible.host_key_checking`` which you can disable
-to avoid SSH connection problems to new virtual machines.
-
-Visit the `Ansible Provisioner documentation
-<http://docs.vagrantup.com/v2/provisioning/ansible.html>`_ for more
-information.
 
 To re-run a playbook on an existing VM, just run:
 
@@ -82,7 +77,19 @@ To re-run a playbook on an existing VM, just run:
 
     $ vagrant provision
 
-This will re-run the playbook.
+This will re-run the playbook against the existing VM.
+
+Note that having the ``ansible.verbose`` option enabled will instruct Vagrant
+to show the full ``ansible-playbook`` command used behind the scene, as
+illustrated by this example:
+
+.. code-block:: bash
+
+      $ PYTHONUNBUFFERED=1 ANSIBLE_FORCE_COLOR=true ANSIBLE_HOST_KEY_CHECKING=false ANSIBLE_SSH_ARGS='-o UserKnownHostsFile=/dev/null -o ControlMaster=auto -o ControlPersist=60s' ansible-playbook --private-key=/home/someone/.vagrant.d/insecure_private_key --user=vagrant --connection=ssh --limit='machine1' --inventory-file=/home/someone/coding-in-a-project/.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory playbook.yml
+
+This information can be quite useful to debug integration issues and can also
+be used to manually execute Ansible from a shell, as explained in the next
+section.
 
 .. _running_ansible:
 
@@ -90,44 +97,58 @@ Running Ansible Manually
 ````````````````````````
 
 Sometimes you may want to run Ansible manually against the machines. This is
-pretty easy to do.
+faster than kicking ``vagrant provision`` and pretty easy to do.
 
-Vagrant automatically creates an inventory file for each Vagrant machine in
-the same directory located under ``.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory``.
-It configures the inventory file according to the SSH tunnel that Vagrant
-automatically creates, and executes ``ansible-playbook`` with the correct
-username and SSH key options to allow access. A typical automatically-created
-inventory file may look something like this:
+With our ``Vagrantfile`` example, Vagrant automatically creates an Ansible
+inventory file in ``.vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory``.
+This inventory is configured according to the SSH tunnel that Vagrant
+automatically creates. A typical automatically-created inventory file for a
+single machine environment may look something like this:
 
 .. code-block:: none
 
     # Generated by Vagrant
 
-    machine ansible_host=127.0.0.1 ansible_port=2222
-
-.. include:: ansible_ssh_changes_note.rst
+    default ansible_ssh_host=127.0.0.1 ansible_ssh_port=2222
 
 If you want to run Ansible manually, you will want to make sure to pass
-``ansible`` or ``ansible-playbook`` commands the correct arguments for the
-username (usually ``vagrant``) and the SSH key (since Vagrant 1.7.0, this will be something like
-``.vagrant/machines/[machine name]/[provider]/private_key``), and the autogenerated inventory file.
+``ansible`` or ``ansible-playbook`` commands the correct arguments, at least
+for the *username*, the *SSH private key* and the *inventory*.
 
-Here is an example:
+Here is an example using the Vagrant global insecure key (``config.ssh.insert_key``
+must be set to ``false`` in your ``Vagrantfile``):
 
 .. code-block:: bash
-   
-    $ ansible-playbook -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory --private-key=.vagrant/machines/default/virtualbox/private_key -u vagrant playbook.yml
 
-Note: Vagrant versions prior to 1.7.0 will use the private key located at ``~/.vagrant.d/insecure_private_key.``
+    $ ansible-playbook --private-key=~/.vagrant.d/insecure_private_key -u vagrant -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory playbook.yml
+
+Here is a second example using the random private key that Vagrant 1.7+
+automatically configures for each new VM (each key is stored in a path like
+``.vagrant/machines/[machine name]/[provider]/private_key``):
+
+.. code-block:: bash
+
+    $ ansible-playbook --private-key=.vagrant/machines/default/virtualbox/private_key -u vagrant -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory playbook.yml
+
+Advanced Usages
+```````````````
+
+The "Tips and Tricks" chapter of the `Ansible Provisioner documentation
+<http://docs.vagrantup.com/v2/provisioning/ansible.html>`_ provides detailed information about more advanced Ansible features like:
+
+  - how to parallely execute a playbook in a multi-machine environment
+  - how to integrate a local ``ansible.cfg`` configuration file
 
 .. seealso::
 
-   `Vagrant Home <http://www.vagrantup.com/>`_
-       The Vagrant homepage with downloads
-   `Vagrant Documentation <http://docs.vagrantup.com/v2/>`_
-       Vagrant Documentation
-   `Ansible Provisioner <http://docs.vagrantup.com/v2/provisioning/ansible.html>`_
-       The Vagrant documentation for the Ansible provisioner
-   :doc:`playbooks`
-       An introduction to playbooks
+    `Vagrant Home <http://www.vagrantup.com/>`_
+        The Vagrant homepage with downloads
+    `Vagrant Documentation <http://docs.vagrantup.com/v2/>`_
+        Vagrant Documentation
+    `Ansible Provisioner <http://docs.vagrantup.com/v2/provisioning/ansible.html>`_
+        The Vagrant documentation for the Ansible provisioner
+    `Vagrant Issue Tracker <https://github.com/mitchellh/vagrant/issues?q=is%3Aopen+is%3Aissue+label%3Aprovisioners%2Fansible>`_
+        The open issues for the Ansible provisioner in the Vagrant project
+    :doc:`playbooks`
+        An introduction to playbooks
 

docsite/rst/intro_adhoc.rst

@@ -112,7 +112,7 @@ For example, using double rather than single quotes in the above example would
 evaluate the variable on the box you were on.
 
 So far we've been demoing simple command execution, but most Ansible modules usually do not work like
-simple scripts. They make the remote system look like you state, and run the commands necessary to
+simple scripts. They make the remote system look like a state, and run the commands necessary to
 get it there.  This is commonly referred to as 'idempotence', and is a core design goal of Ansible.
 However, we also recognize that running arbitrary commands is equally important, so Ansible easily supports both.
 

docsite/rst/intro_configuration.rst

@@ -897,3 +897,19 @@ The normal behaviour is for operations to copy the existing context or use the u
 The default list is: nfs,vboxsf,fuse,ramfs::
 
     special_context_filesystems = nfs,vboxsf,fuse,ramfs,myspecialfs
+
+Galaxy Settings
+---------------
+
+The following options can be set in the [galaxy] section of ansible.cfg:
+
+server
+======
+
+Override the default Galaxy server value of https://galaxy.ansible.com. Useful if you have a hosted version of the Galaxy web app or want to point to the testing site https://galaxy-qa.ansible.com. It does not work against private, hosted repos, which Galaxy can use for fetching and installing roles.
+
+ignore_certs
+============
+
+If set to *yes*, ansible-galaxy will not validate TLS certificates. Handy for testing against a server with a self-signed certificate
+.

docsite/rst/intro_dynamic_inventory.rst

@@ -111,9 +111,8 @@ If you use boto profiles to manage multiple AWS accounts, you can pass ``--profi
     aws_access_key_id = <prod access key>
     aws_secret_access_key = <prod secret key>
 
-You can then run ``ec2.py --profile prod`` to get the inventory for the prod account, or run playbooks with: ``ansible-playbook -i 'ec2.py --profile prod' myplaybook.yml``.
-
-Alternatively, use the ``AWS_PROFILE`` variable - e.g. ``AWS_PROFILE=prod ansible-playbook -i ec2.py myplaybook.yml``
+You can then run ``ec2.py --profile prod`` to get the inventory for the prod account, this option is not supported by ``anisble-playbook`` though.
+But you can use the ``AWS_PROFILE`` variable - e.g. ``AWS_PROFILE=prod ansible-playbook -i ec2.py myplaybook.yml``
 
 Since each region requires its own API call, if you are only using a small set of regions, feel free to edit ``ec2.ini`` and list only the regions you are interested in. There are other config options in ``ec2.ini`` including cache control, and destination variables.
 

docsite/rst/intro_installation.rst

@@ -27,12 +27,11 @@ What Version To Pick?
 `````````````````````
 
 Because it runs so easily from source and does not require any installation of software on remote
-machines, many users will actually track the development version.  
+machines, many users will actually track the development version.
 
-Ansible's release cycles are usually about two months long.  Due to this
-short release cycle, minor bugs will generally be fixed in the next release versus maintaining 
-backports on the stable branch.  Major bugs will still have maintenance releases when needed, though
-these are infrequent.
+Ansible's release cycles are usually about four months long. Due to this short release cycle,
+minor bugs will generally be fixed in the next release versus maintaining backports on the stable branch.
+Major bugs will still have maintenance releases when needed, though these are infrequent.
 
 If you are wishing to run the latest released version of Ansible and you are running Red Hat Enterprise Linux (TM), CentOS, Fedora, Debian, or Ubuntu, we recommend using the OS package manager.
 

docsite/rst/intro_windows.rst

@@ -26,12 +26,12 @@ Installing on the Control Machine
 
 On a Linux control machine::
 
-   pip install https://github.com/diyan/pywinrm/archive/master.zip#egg=pywinrm
+   pip install "pywinrm>=0.1.1"
 
 Active Directory Support
 ++++++++++++++++++++++++
 
-If you wish to connect to domain accounts published through Active Directory (as opposed to local accounts created on the remote host), you will need to install the "python-kerberos" module and the MIT krb5 libraries it depends on.
+If you wish to connect to domain accounts published through Active Directory (as opposed to local accounts created on the remote host), you will need to install the "python-kerberos" module on the Ansible control host (and the MIT krb5 libraries it depends on). The Ansible control host also requires a properly configured computer account in Active Directory.
 
 Installing python-kerberos dependencies
 ---------------------------------------
@@ -131,7 +131,9 @@ To test this, ping the windows host you want to control by name then use the ip
 
 If you get different hostnames back than the name you originally pinged, speak to your active directory administrator and get them to check that DNS Scavenging is enabled and that DNS and DHCP are updating each other.
 
-Check your ansible controller's clock is synchronised with your domain controller.  Kerberos is time sensitive and a little clock drift can cause tickets not be granted.
+Ensure that the Ansible controller has a properly configured computer account in the domain.
+
+Check your Ansible controller's clock is synchronised with your domain controller.  Kerberos is time sensitive and a little clock drift can cause tickets not be granted.
 
 Check you are using the real fully qualified domain name for the domain.  Sometimes domains are commonly known to users by aliases.  To check this run:
 
@@ -165,6 +167,8 @@ In group_vars/windows.yml, define the following inventory variables::
     ansible_password: SecretPasswordGoesHere
     ansible_port: 5986
     ansible_connection: winrm
+    # The following is necessary for Python 2.7.9+ when using default WinRM self-signed certificates:
+    ansible_winrm_server_cert_validation: ignore
 
 Although Ansible is mostly an SSH-oriented system, Windows management will not happen over SSH (`yet <http://blogs.msdn.com/b/powershell/archive/2015/06/03/looking-forward-microsoft-support-for-secure-shell-ssh.aspx>`).
 
@@ -189,6 +193,7 @@ Since 2.0, the following custom inventory variables are also supported for addit
 * ``ansible_winrm_path``: Specify an alternate path to the WinRM endpoint.  Ansible uses ``/wsman`` by default.
 * ``ansible_winrm_realm``: Specify the realm to use for Kerberos authentication.  If the username contains ``@``, Ansible will use the part of the username after ``@`` by default.
 * ``ansible_winrm_transport``: Specify one or more transports as a comma-separated list.  By default, Ansible will use ``kerberos,plaintext`` if the ``kerberos`` module is installed and a realm is defined, otherwise ``plaintext``.
+* ``ansible_winrm_server_cert_validation``: Specify the server certificate validation mode (``ignore`` or ``validate``). Ansible defaults to ``validate`` on Python 2.7.9 and higher, which will result in certificate validation errors against the Windows self-signed certificates. Unless verifiable certificates have been configured on the WinRM listeners, this should be set to ``ignore``
 * ``ansible_winrm_*``: Any additional keyword arguments supported by ``winrm.Protocol`` may be provided.
 
 .. _windows_system_prep:
@@ -221,7 +226,7 @@ Getting to PowerShell 3.0 or higher
 
 PowerShell 3.0 or higher is needed for most provided Ansible modules for Windows, and is also required to run the above setup script. Note that PowerShell 3.0 is only supported on Windows 7 SP1, Windows Server 2008 SP1, and later releases of Windows.
 
-Looking at an ansible checkout, copy the `examples/scripts/upgrade_to_ps3.ps1 <https://github.com/cchurch/ansible/blob/devel/examples/scripts/upgrade_to_ps3.ps1>`_ script onto the remote host and run a PowerShell console as an administrator.  You will now be running PowerShell 3 and can try connectivity again using the win_ping technique referenced above.
+Looking at an Ansible checkout, copy the `examples/scripts/upgrade_to_ps3.ps1 <https://github.com/cchurch/ansible/blob/devel/examples/scripts/upgrade_to_ps3.ps1>`_ script onto the remote host and run a PowerShell console as an administrator.  You will now be running PowerShell 3 and can try connectivity again using the win_ping technique referenced above.
 
 .. _what_windows_modules_are_available:
 
@@ -248,10 +253,10 @@ Note there are a few other Ansible modules that don't start with "win" that also
 Developers: Supported modules and how it works
 ``````````````````````````````````````````````
 
-Developing ansible modules are covered in a `later section of the documentation <http://docs.ansible.com/developing_modules.html>`_, with a focus on Linux/Unix.
-What if you want to write Windows modules for ansible though?
+Developing Ansible modules are covered in a `later section of the documentation <http://docs.ansible.com/developing_modules.html>`_, with a focus on Linux/Unix.
+What if you want to write Windows modules for Ansible though?
 
-For Windows, ansible modules are implemented in PowerShell.  Skim those Linux/Unix module development chapters before proceeding.
+For Windows, Ansible modules are implemented in PowerShell.  Skim those Linux/Unix module development chapters before proceeding.
 
 Windows modules live in a "windows/" subfolder in the Ansible "library/" subtree.  For example, if a module is named
 "library/windows/win_ping", there will be embedded documentation in the "win_ping" file, and the actual PowerShell code will live in a "win_ping.ps1" file.  Take a look at the sources and this will make more sense.
@@ -351,7 +356,7 @@ form of new modules, tweaks to existing modules, documentation, or something els
    :doc:`developing_modules`
        How to write modules
    :doc:`playbooks`
-       Learning ansible's configuration management language
+       Learning Ansible's configuration management language
    `List of Windows Modules <http://docs.ansible.com/list_of_windows_modules.html>`_
        Windows specific module list, all implemented in PowerShell
    `Mailing List <http://groups.google.com/group/ansible-project>`_

docsite/rst/modules_core.rst

@@ -8,6 +8,6 @@ The source of these modules is hosted on GitHub in the `ansible-modules-core <ht
 
 If you believe you have found a bug in a core module and are already running the latest stable or development version of Ansible, first look in the `issue tracker at github.com/ansible/ansible-modules-core <http://github.com/ansible/ansible-modules-core>`_ to see if a bug has already been filed.  If not, we would be grateful if you would file one.
 
-Should you have a question rather than a bug report, inquries are welcome on the `ansible-project google group <https://groups.google.com/forum/#!forum/ansible-project>`_ or on Ansible's "#ansible" channel, located on irc.freenode.net.   Development oriented topics should instead use the similar `ansible-devel google group <https://groups.google.com/forum/#!forum/ansible-devel>`_.
+Should you have a question rather than a bug report, inquiries are welcome on the `ansible-project google group <https://groups.google.com/forum/#!forum/ansible-project>`_ or on Ansible's "#ansible" channel, located on irc.freenode.net.   Development oriented topics should instead use the similar `ansible-devel google group <https://groups.google.com/forum/#!forum/ansible-devel>`_.
 
 Documentation updates for these modules can also be edited directly in the module itself and by submitting a pull request to the module source code, just look for the "DOCUMENTATION" block in the source tree.

docsite/rst/playbooks_delegation.rst

@@ -130,6 +130,29 @@ Here is an example::
 Note that you must have passphrase-less SSH keys or an ssh-agent configured for this to work, otherwise rsync
 will need to ask for a passphrase.
 
+.. _delegate_facts:
+
+Delegated facts
+```````````````
+
+.. versionadded:: 2.0
+
+By default, any fact gathered by a delegated task are assigned to the `inventory_hostname` (the current host) instead of the host which actually produced the facts (the delegated to host).
+In 2.0, the directive `delegate_facts` may be set to `True` to assign the task's gathered facts to the delegated host instead of the current one.::
+
+
+    - hosts: app_servers
+      tasks:
+        - name: gather facts from db servers
+          setup:
+          delegate_to: "{{item}}"
+          delegate_facts: True
+          with_items: "{{groups['dbservers'}}"
+
+The above will gather facts for the machines in the dbservers group and assign the facts to those machines and not to app_servers.
+This way you can lookup `hostvars['dbhost1']['default_ipv4_addresses'][0]` even though dbservers were not part of the play, or left out by using `--limit`.
+
+
 .. _run_once:
 
 Run Once
@@ -159,13 +182,18 @@ This can be optionally paired with "delegate_to" to specify an individual host t
           delegate_to: web01.example.org
 
 When "run_once" is not used with "delegate_to" it will execute on the first host, as defined by inventory,
-in the group(s) of hosts targeted by the play. e.g. webservers[0] if the play targeted "hosts: webservers".
+in the group(s) of hosts targeted by the play - e.g. webservers[0] if the play targeted "hosts: webservers".
 
-This approach is similar, although more concise and cleaner than applying a conditional to a task such as::
+This approach is similar to applying a conditional to a task such as::
 
         - command: /opt/application/upgrade_db.py
           when: inventory_hostname == webservers[0]
 
+.. note::
+     When used together with "serial", tasks marked as "run_once" will be ran on one host in *each* serial batch.
+     If it's crucial that the task is run only once regardless of "serial" mode, use
+     :code:`inventory_hostname == my_group_name[0]` construct.
+
 .. _local_playbooks:
 
 Local Playbooks

docsite/rst/playbooks_environment.rst

@@ -31,7 +31,7 @@ The environment can also be stored in a variable, and accessed like so::
       tasks:
 
         - apt: name=cobbler state=installed
-          environment: proxy_env
+          environment: "{{proxy_env}}"
 
 You can also use it at a playbook level::
 

docsite/rst/playbooks_filters.rst

@@ -514,20 +514,25 @@ To match strings against a regex, use the "match" or "search" filter::
 
 To replace text in a string with regex, use the "regex_replace" filter::
 
-    # convert "ansible" to "able"    
+    # convert "ansible" to "able"
     {{ 'ansible' | regex_replace('^a.*i(.*)$', 'a\\1') }}
 
     # convert "foobar" to "bar"
     {{ 'foobar' | regex_replace('^f.*o(.*)$', '\\1') }}
 
+    # convert "localhost:80" to "localhost, 80" using named groups
+    {{ 'localhost:80' | regex_replace('^(?P<host>.+):(?P<port>\\d+)$', '\\g<host>, \\g<port>') }}
+
 .. note:: Prior to ansible 2.0, if "regex_replace" filter was used with variables inside YAML arguments (as opposed to simpler 'key=value' arguments),
    then you needed to escape backreferences (e.g. ``\\1``) with 4 backslashes (``\\\\``) instead of 2 (``\\``).
 
+.. versionadded:: 2.0
+
 To escape special characters within a regex, use the "regex_escape" filter::
 
     # convert '^f.*o(.*)$' to '\^f\.\*o\(\.\*\)\$'
     {{ '^f.*o(.*)$' | regex_escape() }}
-    
+
 To make use of one attribute from each item in a list of complex variables, use the "map" filter (see the `Jinja2 map() docs`_ for more)::
 
     # get a comma-separated list of the mount points (e.g. "/,/mnt/stuff") on a host

docsite/rst/playbooks_intro.rst

@@ -386,6 +386,7 @@ won't need them for much else.
    * Handler names live in a global namespace.
    * If two handler tasks have the same name, only one will run.
      `* <https://github.com/ansible/ansible/issues/4943>`_
+   * You cannot notify a handler that is defined inside of an include
 
 Roles are described later on, but it's worthwhile to point out that:
 

docsite/rst/playbooks_lookups.rst

@@ -240,6 +240,112 @@ If you're not using 2.0 yet, you can do something similar with the credstash too
 
     debug: msg="Poor man's credstash lookup! {{ lookup('pipe', 'credstash -r us-west-1 get my-other-password') }}"
 
+.. _dns_lookup:
+
+The DNS Lookup (dig)
+````````````````````
+.. versionadded:: 1.9.0
+
+.. warning:: This lookup depends on the `dnspython <http://www.dnspython.org/>`_
+             library.
+
+The ``dig`` lookup runs queries against DNS servers to retrieve DNS records for
+a specific name (*FQDN* - fully qualified domain name). It is possible to lookup any DNS record in this manner.
+
+There is a couple of different syntaxes that can be used to specify what record
+should be retrieved, and for which name. It is also possible to explicitly
+specify the DNS server(s) to use for lookups.
+
+In its simplest form, the ``dig`` lookup plugin can be used to retrieve an IPv4
+address (DNS ``A`` record) associated with *FQDN*:
+
+.. note:: If you need to obtain the ``AAAA`` record (IPv6 address), you must
+          specify the record type explicitly. Syntax for specifying the record
+          type is described below.
+
+.. note:: The trailing dot in most of the examples listed is purely optional,
+          but is specified for completeness/correctness sake.
+
+::
+
+      - debug: msg="The IPv4 address for example.com. is {{ lookup('dig', 'example.com.')}}"
+
+In addition to (default) ``A`` record, it is also possible to specify a different
+record type that should be queried. This can be done by either passing-in
+additional parameter of format ``qtype=TYPE`` to the ``dig`` lookup, or by
+appending ``/TYPE`` to the *FQDN* being queried. For example::
+
+  - debug: msg="The TXT record for gmail.com. is {{ lookup('dig', 'gmail.com.', 'qtype=TXT') }}"
+  - debug: msg="The TXT record for gmail.com. is {{ lookup('dig', 'gmail.com./TXT') }}"
+
+If multiple values are associated with the requested record, the results will be
+returned as a comma-separated list. In such cases you may want to pass option
+``wantlist=True`` to the plugin, which will result in the record values being
+returned as a list over which you can iterate later on::
+
+  - debug: msg="One of the MX records for gmail.com. is {{ item }}"
+    with_items: "{{ lookup('dig', 'gmail.com./MX', wantlist=True) }}"
+
+In case of reverse DNS lookups (``PTR`` records), you can also use a convenience
+syntax of format ``IP_ADDRESS/PTR``. The following three lines would produce the
+same output::
+
+  - debug: msg="Reverse DNS for 8.8.8.8 is {{ lookup('dig', '8.8.8.8/PTR') }}"
+  - debug: msg="Reverse DNS for 8.8.8.8 is {{ lookup('dig', '8.8.8.8.in-addr.arpa./PTR') }}"
+  - debug: msg="Reverse DNS for 8.8.8.8 is {{ lookup('dig', '8.8.8.8.in-addr.arpa.', 'qtype=PTR') }}"
+
+By default, the lookup will rely on system-wide configured DNS servers for
+performing the query. It is also possible to explicitly specify DNS servers to
+query using the ``@DNS_SERVER_1,DNS_SERVER_2,...,DNS_SERVER_N`` notation. This
+needs to be passed-in as an additional parameter to the lookup. For example::
+
+  - debug: msg="Querying 8.8.8.8 for IPv4 address for example.com. produces {{ lookup('dig', 'example.com', '@8.8.8.8') }}"
+
+In some cases the DNS records may hold a more complex data structure, or it may
+be useful to obtain the results in a form of a dictionary for future
+processing. The ``dig`` lookup supports parsing of a number of such records,
+with the result being returned as a dictionary. This way it is possible to
+easily access such nested data. This return format can be requested by
+passing-in the ``flat=0`` option to the lookup. For example::
+
+  - debug: msg="XMPP service for gmail.com. is available at {{ item.target }} on port {{ item.port }}"
+    with_items: "{{ lookup('dig', '_xmpp-server._tcp.gmail.com./SRV', 'flat=0', wantlist=True) }}"
+
+Take note that due to the way Ansible lookups work, you must pass the
+``wantlist=True`` argument to the lookup, otherwise Ansible will report errors.
+
+Currently the dictionary results are supported for the following records:
+
+.. note:: *ALL* is not a record per-se, merely the listed fields are available
+          for any record results you retrieve in the form of a dictionary.
+
+==========   =============================================================================
+Record       Fields
+----------   -----------------------------------------------------------------------------
+*ALL*        owner, ttl, type
+A            address
+AAAA         address
+CNAME        target
+DNAME        target
+DLV          algorithm, digest_type, key_tag, digest
+DNSKEY       flags, algorithm, protocol, key
+DS           algorithm, digest_type, key_tag, digest
+HINFO        cpu, os
+LOC          latitude, longitude, altitude, size, horizontal_precision, vertical_precision
+MX           preference, exchange
+NAPTR        order, preference, flags, service, regexp, replacement
+NS           target
+NSEC3PARAM   algorithm, flags, iterations, salt
+PTR          target
+RP           mbox, txt
+SOA          mname, rname, serial, refresh, retry, expire, minimum
+SPF          strings
+SRV          priority, weight, port, target
+SSHFP        algorithm, fp_type, fingerprint
+TLSA         usage, selector, mtype, cert
+TXT          strings
+==========   =============================================================================
+
 .. _more_lookups:
 
 More Lookups

docsite/rst/playbooks_loops.rst

@@ -96,7 +96,7 @@ And you want to print every user's name and phone number.  You can loop through
 Looping over Files
 ``````````````````
 
-``with_file`` iterates over a list of files, setting `item` to the content of each file in sequence.  It can be used like this::
+``with_file`` iterates over the content of a list of files, `item` will be set to the content of each file in sequence.  It can be used like this::
 
     ---
     - hosts: all

docsite/rst/playbooks_roles.rst

@@ -132,7 +132,7 @@ Note that you cannot do variable substitution when including one playbook
 inside another.
 
 .. note::
-   You can not conditionally path the location to an include file,
+   You can not conditionally pass the location to an include file,
    like you can with 'vars_files'.  If you find yourself needing to do
    this, consider how you can restructure your playbook to be more
    class/role oriented.  This is to say you cannot use a 'fact' to
@@ -191,11 +191,8 @@ This designates the following behaviors, for each role 'x':
 - If roles/x/handlers/main.yml exists, handlers listed therein will be added to the play
 - If roles/x/vars/main.yml exists, variables listed therein will be added to the play
 - If roles/x/meta/main.yml exists, any role dependencies listed therein will be added to the list of roles (1.3 and later)
-- Any copy tasks can reference files in roles/x/files/ without having to path them relatively or absolutely
-- Any script tasks can reference scripts in roles/x/files/ without having to path them relatively or absolutely
-- Any template tasks can reference files in roles/x/templates/ without having to path them relatively or absolutely
-- Any include tasks can reference files in roles/x/tasks/ without having to path them relatively or absolutely
-   
+- Any copy, script, template or include tasks (in the role) can reference files in roles/x/{files,templates,tasks}/ (dir depends on task) without having to path them relatively or absolutely
+
 In Ansible 1.4 and later you can configure a roles_path to search for roles.  Use this to check all of your common roles out to one location, and share
 them easily between multiple playbook projects.  See :doc:`intro_configuration` for details about how to set this up in ansible.cfg.
 

docsite/rst/playbooks_variables.rst

@@ -793,10 +793,10 @@ Basically, anything that goes into "role defaults" (the defaults folder inside t
 
 .. rubric:: Footnotes
 
-.. [1] Tasks in each role will see their own role's defaults tasks outside of roles will the last role's defaults
-.. [2] Variables defined in inventory file or provided by dynamic inventory
+.. [1] Tasks in each role will see their own role's defaults. Tasks defined outside of a role will see the last role's defaults.
+.. [2] Variables defined in inventory file or provided by dynamic inventory.
 
-.. note:: Within a any section, redefining a var will overwrite the previous instance.
+.. note:: Within any section, redefining a var will overwrite the previous instance.
           If multiple groups have the same variable, the last one loaded wins.
           If you define a variable twice in a play's vars: section, the 2nd one wins.
 .. note:: the previous describes the default config `hash_behavior=replace`, switch to 'merge' to only partially overwrite.

docsite/rst/porting_guide_2.0.rst

@@ -0,0 +1,202 @@
+Porting Guide
+=============
+
+
+Playbook
+--------
+
+* backslash escapes When specifying parameters in jinja2 expressions in YAML
+  dicts, backslashes sometimes needed to be escaped twice. This has been fixed
+  in 2.0.x so that escaping once works. The following example shows how
+  playbooks must be modified::
+
+    # Syntax in 1.9.x
+    - debug:
+        msg: "{{ 'test1_junk 1\\\\3' | regex_replace('(.*)_junk (.*)', '\\\\1 \\\\2') }}"
+    # Syntax in 2.0.x
+    - debug:
+        msg: "{{ 'test1_junk 1\\3' | regex_replace('(.*)_junk (.*)', '\\1 \\2') }}"
+
+    # Output:
+    "msg": "test1 1\\3"
+
+To make an escaped string that will work on all versions you have two options::
+
+- debug: msg="{{ 'test1_junk 1\\3' | regex_replace('(.*)_junk (.*)', '\\1 \\2') }}"
+
+uses key=value escaping which has not changed.  The other option is to check for the ansible version::
+
+"{{ (ansible_version|version_compare('ge', '2.0'))|ternary( 'test1_junk 1\\3' | regex_replace('(.*)_junk (.*)', '\\1 \\2') , 'test1_junk 1\\\\3' | regex_replace('(.*)_junk (.*)', '\\\\1 \\\\2') ) }}"
+
+* trailing newline When a string with a trailing newline was specified in the
+  playbook via yaml dict format, the trailing newline was stripped. When
+  specified in key=value format, the trailing newlines were kept. In v2, both
+  methods of specifying the string will keep the trailing newlines. If you
+  relied on the trailing newline being stripped, you can change your playbook
+  using the following as an example::
+
+    # Syntax in 1.9.x
+    vars:
+      message: >
+        Testing
+        some things
+    tasks:
+    - debug:
+        msg: "{{ message }}"
+
+    # Syntax in 2.0.x
+    vars:
+      old_message: >
+        Testing
+        some things
+      message: "{{ old_messsage[:-1] }}"
+    - debug:
+        msg: "{{ message }}"
+    # Output
+    "msg": "Testing some things"
+
+* When specifying complex args as a variable, the variable must use the full jinja2
+  variable syntax ('{{var_name}}') - bare variable names there are no longer accepted.
+  In fact, even specifying args with variables has been deprecated, and will not be
+  allowed in future versions::
+
+    ---
+    - hosts: localhost
+      connection: local
+      gather_facts: false
+      vars:
+        my_dirs:
+          - { path: /tmp/3a, state: directory, mode: 0755 }
+          - { path: /tmp/3b, state: directory, mode: 0700 }
+      tasks:
+        - file:
+          args: "{{item}}" # <- args here uses the full variable syntax
+          with_items: my_dirs
+
+* porting task includes
+* More dynamic. Corner-case formats that were not supposed to work now do not, as expected.
+* variables defined in the yaml dict format https://github.com/ansible/ansible/issues/13324
+* templating (variables in playbooks and template lookups) has improved with regard to keeping the original instead of turning everything into a string.
+  If you need the old behavior, quote the value to pass it around as a string.
+* Empty variables and variables set to null in yaml are no longer converted to empty strings. They will retain the value of `None`.
+  You can override the `null_representation` setting to an empty string in your config file by setting the `ANSIBLE_NULL_REPRESENTATION` environment variable.
+* Extras callbacks must be whitelisted in ansible.cfg. Copying is no longer necessary but whitelisting in ansible.cfg must be completed.
+* dnf module has been rewritten. Some minor changes in behavior may be observed.
+* win_updates has been rewritten and works as expected now.
+
+Deprecated
+----------
+
+While all items listed here will show a deprecation warning message, they still work as they did in 1.9.x. Please note that they will be removed in 2.2 (Ansible always waits two major releases to remove a deprecated feature).
+
+* Bare variables in `with_` loops should instead use the “{{var}}” syntax, which helps eliminate ambiguity.
+* The ansible-galaxy text format requirements file. Users should use the YAML format for requirements instead.
+* Undefined variables within a `with_` loop’s list currently do not interrupt the loop, but they do issue a warning; in the future, they will issue an error.
+* Using variables for task parameters is unsafe and will be removed in a future version. For example::
+
+    - hosts: localhost
+      gather_facts: no
+      vars:
+        debug_params:
+          msg: "hello there"
+      tasks:
+        - debug: "{{debug_params}}"
+
+* Host patterns should use a comma (,) or colon (:) instead of a semicolon (;) to separate hosts/groups in the pattern.
+* Ranges specified in host patterns should use the [x:y] syntax, instead of [x-y].
+* Playbooks using privilege escalation should always use “become*” options rather than the old su*/sudo* options.
+* The “short form” for vars_prompt is no longer supported.
+  For example::
+
+    vars_prompt:
+        variable_name: "Prompt string"
+
+* Specifying variables at the top level of a task include statement is no longer supported. For example::
+
+    - include: foo.yml
+        a: 1
+
+Should now be::
+
+    - include: foo.yml
+      args:
+        a: 1
+
+* Setting any_errors_fatal on a task is no longer supported. This should be set at the play level only.
+* Bare variables in the `environment` dictionary (for plays/tasks/etc.) are no longer supported. Variables specified there should use the full variable syntax: ‘{{foo}}’.
+* Tags should no longer be specified with other parameters in a task include. Instead, they should be specified as an option on the task.
+  For example::
+
+    - include: foo.yml tags=a,b,c
+
+  Should be::
+
+    - include: foo.yml
+      tags: [a, b, c]
+
+* The first_available_file option on tasks has been deprecated. Users should use the with_first_found option or lookup (‘first_found’, …) plugin.
+
+
+Porting plugins
+===============
+
+In ansible-1.9.x, you would generally copy an existing plugin to create a new one. Simply implementing the methods and attributes that the caller of the plugin expected made it a plugin of that type. In ansible-2.0, most plugins are implemented by subclassing a base class for each plugin type. This way the custom plugin does not need to contain methods which are not customized.
+
+
+Lookup plugins
+--------------
+* lookup plugins ; import version
+
+
+Connection plugins
+------------------
+
+* connection plugins
+
+Action plugins
+--------------
+
+* action plugins
+
+Callback plugins
+----------------
+
+Although Ansible 2.0 provides a new callback API the old one continues to work
+for most callback plugins.  However, if your callback plugin makes use of
+:attr:`self.playbook`, :attr:`self.play`, or :attr:`self.task` then you will
+have to store the values for these yourself as ansible no longer automatically
+populates the callback with them.  Here's a short snippet that shows you how::
+
+    from ansible.plugins.callback import CallbackBase
+
+    class CallbackModule(CallbackBase):
+        def __init__(self):
+            self.playbook = None
+            self.play = None
+            self.task = None
+
+        def v2_playbook_on_start(self, playbook):
+            self.playbook = playbook
+
+        def v2_playbook_on_play_start(self, play):
+            self.play = play
+
+        def v2_playbook_on_task_start(self, task, is_conditional):
+            self.task = task
+
+        def v2_on_any(self, *args, **kwargs):
+            self._display.display('%s: %s: %s' % (self.playbook.name,
+            self.play.name, self.task))
+
+
+Connection plugins
+------------------
+
+* connection plugins
+
+
+Porting custom scripts
+======================
+
+Custom scripts that used the ``ansible.runner.Runner`` API in 1.x have to be ported in 2.x.  Please refer to:
+https://github.com/ansible/ansible/blob/devel/docsite/rst/developing_api.rst

examples/ansible.cfg

@@ -14,7 +14,6 @@
 #inventory      = /etc/ansible/hosts
 #library        = /usr/share/my_modules/
 #remote_tmp     = $HOME/.ansible/tmp
-#pattern        = *
 #forks          = 5
 #poll_interval  = 15
 #sudo_user      = root
@@ -182,7 +181,7 @@
 #no_log = False
 
 # prevents logging of tasks, but only on the targets, data is still logged on the master/controller
-#no_target_syslog = True
+#no_target_syslog = False
 
 # controls the compression level of variables sent to
 # worker processes. At the default of 0, no compression

examples/hosts

@@ -10,35 +10,35 @@
 
 # Ex 1: Ungrouped hosts, specify before any group headers.
 
-green.example.com
-blue.example.com
-192.168.100.1
-192.168.100.10
+## green.example.com
+## blue.example.com
+## 192.168.100.1
+## 192.168.100.10
 
 # Ex 2: A collection of hosts belonging to the 'webservers' group
 
-[webservers]
-alpha.example.org
-beta.example.org
-192.168.1.100
-192.168.1.110
+## [webservers]
+## alpha.example.org
+## beta.example.org
+## 192.168.1.100
+## 192.168.1.110
 
 # If you have multiple hosts following a pattern you can specify
 # them like this:
 
-www[001:006].example.com
+## www[001:006].example.com
 
 # Ex 3: A collection of database servers in the 'dbservers' group
 
-[dbservers]
-
-db01.intranet.mydomain.net
-db02.intranet.mydomain.net
-10.25.1.56
-10.25.1.57
+## [dbservers]
+## 
+## db01.intranet.mydomain.net
+## db02.intranet.mydomain.net
+## 10.25.1.56
+## 10.25.1.57
 
 # Here's another example of host ranges, this time there are no
 # leading 0s:
 
-db-[99:101]-node.example.com
+## db-[99:101]-node.example.com
 

hacking/module_formatter.py

@@ -140,7 +140,7 @@ def list_modules(module_dir, depth=0):
             if os.path.isdir(d):
 
                 res = list_modules(d, depth + 1)
-                for key in res.keys():
+                for key in list(res.keys()):
                     if key in categories:
                         categories[key] = merge_hash(categories[key], res[key])
                         res.pop(key, None)
@@ -451,7 +451,7 @@ def main():
 
     categories = list_modules(options.module_dir)
     last_category = None
-    category_names = categories.keys()
+    category_names = list(categories.keys())
     category_names.sort()
 
     category_list_path = os.path.join(options.output_dir, "modules_by_category.rst")

lib/ansible/__init__.py

@@ -19,5 +19,5 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-__version__ = '2.0.0'
+__version__ = '2.0.1.0'
 __author__  = 'Ansible, Inc.'

lib/ansible/cli/__init__.py

@@ -27,12 +27,13 @@ import time
 import yaml
 import re
 import getpass
+import signal
 import subprocess
 
 from ansible import __version__
 from ansible import constants as C
 from ansible.errors import AnsibleError, AnsibleOptionsError
-from ansible.utils.unicode import to_bytes
+from ansible.utils.unicode import to_bytes, to_unicode
 
 try:
     from __main__ import display
@@ -44,7 +45,7 @@ except ImportError:
 class SortedOptParser(optparse.OptionParser):
     '''Optparser which sorts the options by opt before outputting --help'''
 
-    # TODO: epilog parsing: OptionParser.format_epilog = lambda self, formatter: self.epilog
+    #FIXME: epilog parsing: OptionParser.format_epilog = lambda self, formatter: self.epilog
 
     def format_help(self, formatter=None, epilog=None):
         self.option_list.sort(key=operator.methodcaller('get_opt_string'))
@@ -66,7 +67,7 @@ class CLI(object):
     LESS_OPTS = 'FRSX'  # -F (quit-if-one-screen) -R (allow raw ansi control chars)
                         # -S (chop long lines) -X (disable termcap init and de-init)
 
-    def __init__(self, args):
+    def __init__(self, args, callback=None):
         """
         Base init method for all command line programs
         """
@@ -75,6 +76,21 @@ class CLI(object):
         self.options = None
         self.parser = None
         self.action = None
+        self.callback = callback
+
+    def _terminate(self, signum=None, framenum=None):
+        if signum == signal.SIGTERM:
+            if hasattr(os, 'getppid'):
+                display.debug("Termination requested in parent, shutting down gracefully")
+                signal.signal(signal.SIGTERM, signal.SIG_DFL)
+            else:
+                display.debug("Term signal in child, harakiri!")
+                signal.signal(signal.SIGTERM, signal.SIG_IGN)
+
+            raise SystemExit
+
+        #NOTE: if ever want to make this immediately kill children use on parent:
+        #os.killpg(os.getpgid(0), signal.SIGTERM)
 
     def set_action(self):
         """
@@ -104,9 +120,12 @@ class CLI(object):
 
         if self.options.verbosity > 0:
             if C.CONFIG_FILE:
-                display.display("Using %s as config file" % C.CONFIG_FILE)
+                display.display(u"Using %s as config file" % to_unicode(C.CONFIG_FILE))
             else:
-                display.display("No config file found; using defaults")
+                display.display(u"No config file found; using defaults")
+
+        # Manage user interruptions
+        #signal.signal(signal.SIGTERM, self._terminate)
 
     @staticmethod
     def ask_vault_passwords(ask_new_vault_pass=False, rekey=False):
@@ -191,12 +210,9 @@ class CLI(object):
 
         if runas_opts:
             # Check for privilege escalation conflicts
-            if (op.su or op.su_user or op.ask_su_pass) and \
-                        (op.sudo or op.sudo_user or op.ask_sudo_pass) or \
-                (op.su or op.su_user or op.ask_su_pass) and \
-                        (op.become or op.become_user or op.become_ask_pass) or \
-                (op.sudo or op.sudo_user or op.ask_sudo_pass) and \
-                        (op.become or op.become_user or op.become_ask_pass):
+            if (op.su or op.su_user) and (op.sudo or op.sudo_user) or \
+                (op.su or op.su_user) and (op.become or op.become_user) or \
+                (op.sudo or op.sudo_user) and (op.become or op.become_user):
 
                 self.parser.error("Sudo arguments ('--sudo', '--sudo-user', and '--ask-sudo-pass') "
                                   "and su arguments ('-su', '--su-user', and '--ask-su-pass') "
@@ -213,7 +229,7 @@ class CLI(object):
 
     @staticmethod
     def base_parser(usage="", output_opts=False, runas_opts=False, meta_opts=False, runtask_opts=False, vault_opts=False, module_opts=False,
-            async_opts=False, connect_opts=False, subset_opts=False, check_opts=False, inventory_opts=False, epilog=None, fork_opts=False):
+            async_opts=False, connect_opts=False, subset_opts=False, check_opts=False, inventory_opts=False, epilog=None, fork_opts=False, runas_prompt_opts=False):
         ''' create an options parser for most ansible scripts '''
 
         # TODO: implement epilog parsing
@@ -226,7 +242,7 @@ class CLI(object):
 
         if inventory_opts:
             parser.add_option('-i', '--inventory-file', dest='inventory',
-                help="specify inventory host path (default=%s) or comma separated host list" % C.DEFAULT_HOST_LIST,
+                help="specify inventory host path (default=%s) or comma separated host list." % C.DEFAULT_HOST_LIST,
                 default=C.DEFAULT_HOST_LIST, action="callback", callback=CLI.expand_tilde, type=str)
             parser.add_option('--list-hosts', dest='listhosts', action='store_true',
                 help='outputs a list of matching hosts; does not execute anything else')
@@ -246,14 +262,15 @@ class CLI(object):
                 help="specify number of parallel processes to use (default=%s)" % C.DEFAULT_FORKS)
 
         if vault_opts:
-            parser.add_option('--ask-vault-pass', default=False, dest='ask_vault_pass', action='store_true',
+            parser.add_option('--ask-vault-pass', default=C.DEFAULT_ASK_VAULT_PASS, dest='ask_vault_pass', action='store_true',
                 help='ask for vault password')
             parser.add_option('--vault-password-file', default=C.DEFAULT_VAULT_PASSWORD_FILE, dest='vault_password_file',
                 help="vault password file", action="callback", callback=CLI.expand_tilde, type=str)
             parser.add_option('--new-vault-password-file', dest='new_vault_password_file',
                 help="new vault password file for rekey", action="callback", callback=CLI.expand_tilde, type=str)
             parser.add_option('--output', default=None, dest='output_file',
-                help='output file name for encrypt or decrypt; use - for stdout')
+                help='output file name for encrypt or decrypt; use - for stdout',
+                action="callback", callback=CLI.expand_tilde, type=str)
 
         if subset_opts:
             parser.add_option('-t', '--tags', dest='tags', default='all',
@@ -269,10 +286,6 @@ class CLI(object):
 
         if runas_opts:
             # priv user defaults to root later on to enable detecting when this option was given here
-            parser.add_option('-K', '--ask-sudo-pass', default=C.DEFAULT_ASK_SUDO_PASS, dest='ask_sudo_pass', action='store_true',
-                help='ask for sudo password (deprecated, use become)')
-            parser.add_option('--ask-su-pass', default=C.DEFAULT_ASK_SU_PASS, dest='ask_su_pass', action='store_true',
-                help='ask for su password (deprecated, use become)')
             parser.add_option("-s", "--sudo", default=C.DEFAULT_SUDO, action="store_true", dest='sudo',
                 help="run operations with sudo (nopasswd) (deprecated, use become)")
             parser.add_option('-U', '--sudo-user', dest='sudo_user', default=None,
@@ -289,6 +302,12 @@ class CLI(object):
                 help="privilege escalation method to use (default=%s), valid choices: [ %s ]" % (C.DEFAULT_BECOME_METHOD, ' | '.join(C.BECOME_METHODS)))
             parser.add_option('--become-user', default=None, dest='become_user', type='string',
                 help='run operations as this user (default=%s)' % C.DEFAULT_BECOME_USER)
+
+        if runas_opts or runas_prompt_opts:
+            parser.add_option('-K', '--ask-sudo-pass', default=C.DEFAULT_ASK_SUDO_PASS, dest='ask_sudo_pass', action='store_true',
+                help='ask for sudo password (deprecated, use become)')
+            parser.add_option('--ask-su-pass', default=C.DEFAULT_ASK_SU_PASS, dest='ask_su_pass', action='store_true',
+                help='ask for su password (deprecated, use become)')
             parser.add_option('--ask-become-pass', default=False, dest='become_ask_pass', action='store_true',
                 help='ask for privilege escalation password')
 
@@ -458,7 +477,7 @@ class CLI(object):
             os.environ['LESS'] = CLI.LESS_OPTS
         try:
             cmd = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE, stdout=sys.stdout)
-            cmd.communicate(input=text.encode(sys.stdout.encoding))
+            cmd.communicate(input=to_bytes(text))
         except IOError:
             pass
         except KeyboardInterrupt:

lib/ansible/cli/adhoc.py

@@ -70,7 +70,7 @@ class AdHocCLI(CLI):
             help="module name to execute (default=%s)" % C.DEFAULT_MODULE_NAME,
             default=C.DEFAULT_MODULE_NAME)
 
-        self.options, self.args = self.parser.parse_args()
+        self.options, self.args = self.parser.parse_args(self.args[1:])
 
         if len(self.args) != 1:
             raise AnsibleOptionsError("Missing target hosts")
@@ -124,17 +124,13 @@ class AdHocCLI(CLI):
         inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list=self.options.inventory)
         variable_manager.set_inventory(inventory)
 
-        hosts = inventory.list_hosts(pattern)
-        no_hosts = False
-        if len(hosts) == 0:
-            display.warning("provided hosts list is empty, only localhost is available")
-            no_hosts = True
 
         if self.options.subset:
             inventory.subset(self.options.subset)
-            if len(inventory.list_hosts(pattern)) == 0 and not no_hosts:
-                # Invalid limit
-                raise AnsibleError("Specified --limit does not match any hosts")
+
+        hosts = inventory.list_hosts(pattern)
+        if len(hosts) == 0:
+            raise AnsibleError("Specified hosts options do not match any hosts")
 
         if self.options.listhosts:
             display.display('  hosts (%d):' % len(hosts))
@@ -158,14 +154,18 @@ class AdHocCLI(CLI):
         play_ds = self._play_ds(pattern, self.options.seconds, self.options.poll_interval)
         play = Play().load(play_ds, variable_manager=variable_manager, loader=loader)
 
-        if self.options.one_line:
+        if self.callback: 
+            cb = self.callback
+        elif self.options.one_line:
             cb = 'oneline'
         else:
             cb = 'minimal'
 
+        run_tree=False
         if self.options.tree:
             C.DEFAULT_CALLBACK_WHITELIST.append('tree')
             C.TREE_DIR = self.options.tree
+            run_tree=True
 
         # now create a task queue manager to execute the play
         self._tqm = None
@@ -177,7 +177,10 @@ class AdHocCLI(CLI):
                     options=self.options,
                     passwords=passwords,
                     stdout_callback=cb,
+                    run_additional_callbacks=C.DEFAULT_LOAD_CALLBACK_PLUGINS,
+                    run_tree=run_tree,
                 )
+
             result = self._tqm.run(play)
         finally:
             if self._tqm:

lib/ansible/cli/doc.py

@@ -62,7 +62,7 @@ class DocCLI(CLI):
         self.parser.add_option("-s", "--snippet", action="store_true", default=False, dest='show_snippet',
                 help='Show playbook snippet for specified module(s)')
 
-        self.options, self.args = self.parser.parse_args()
+        self.options, self.args = self.parser.parse_args(self.args[1:])
         display.verbosity = self.options.verbosity
 
     def run(self):
@@ -90,7 +90,8 @@ class DocCLI(CLI):
         for module in self.args:
 
             try:
-                filename = module_loader.find_plugin(module)
+                # if the module lives in a non-python file (eg, win_X.ps1), require the corresponding python file for docs
+                filename = module_loader.find_plugin(module, mod_type='.py')
                 if filename is None:
                     display.warning("module %s not found in %s\n" % (module, DocCLI.print_paths(module_loader)))
                     continue
@@ -167,7 +168,8 @@ class DocCLI(CLI):
             if module in module_docs.BLACKLIST_MODULES:
                 continue
 
-            filename = module_loader.find_plugin(module)
+            # if the module lives in a non-python file (eg, win_X.ps1), require the corresponding python file for docs
+            filename = module_loader.find_plugin(module, mod_type='.py')
 
             if filename is None:
                 continue

lib/ansible/cli/galaxy.py

@@ -22,10 +22,10 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import os
 import os.path
 import sys
 import yaml
+import time
 
 from collections import defaultdict
 from jinja2 import Environment
@@ -36,7 +36,10 @@ from ansible.errors import AnsibleError, AnsibleOptionsError
 from ansible.galaxy import Galaxy
 from ansible.galaxy.api import GalaxyAPI
 from ansible.galaxy.role import GalaxyRole
+from ansible.galaxy.login import GalaxyLogin
+from ansible.galaxy.token import GalaxyToken
 from ansible.playbook.role.requirement import RoleRequirement
+from ansible.utils.unicode import to_unicode
 
 try:
     from __main__ import display
@@ -44,14 +47,12 @@ except ImportError:
     from ansible.utils.display import Display
     display = Display()
 
-
 class GalaxyCLI(CLI):
 
-    VALID_ACTIONS = ("init", "info", "install", "list", "remove", "search")
     SKIP_INFO_KEYS = ("name", "description", "readme_html", "related", "summary_fields", "average_aw_composite", "average_aw_score", "url" )
-
+    VALID_ACTIONS = ("delete", "import", "info", "init", "install", "list", "login", "remove", "search", "setup")
+    
     def __init__(self, args):
-
         self.api = None
         self.galaxy = None
         super(GalaxyCLI, self).__init__(args)
@@ -67,7 +68,17 @@ class GalaxyCLI(CLI):
         self.set_action()
 
         # options specific to actions
-        if self.action == "info":
+        if self.action == "delete":
+            self.parser.set_usage("usage: %prog delete [options] github_user github_repo")
+        elif self.action == "import":
+            self.parser.set_usage("usage: %prog import [options] github_user github_repo")
+            self.parser.add_option('--no-wait', dest='wait', action='store_false', default=True,
+                help='Don\'t wait for import results.')
+            self.parser.add_option('--branch', dest='reference',
+                help='The name of a branch to import. Defaults to the repository\'s default branch (usually master)')
+            self.parser.add_option('--status', dest='check_status', action='store_true', default=False,
+                help='Check the status of the most recent import request for given github_user/github_repo.')
+        elif self.action == "info":
             self.parser.set_usage("usage: %prog info [options] role_name[,version]")
         elif self.action == "init":
             self.parser.set_usage("usage: %prog init [options] role_name")
@@ -88,31 +99,42 @@ class GalaxyCLI(CLI):
             self.parser.set_usage("usage: %prog remove role1 role2 ...")
         elif self.action == "list":
             self.parser.set_usage("usage: %prog list [role_name]")
+        elif self.action == "login":
+            self.parser.set_usage("usage: %prog login [options]")
+            self.parser.add_option('--github-token', dest='token', default=None,
+                help='Identify with github token rather than username and password.')
         elif self.action == "search":
             self.parser.add_option('--platforms', dest='platforms',
                 help='list of OS platforms to filter by')
             self.parser.add_option('--galaxy-tags', dest='tags',
                 help='list of galaxy tags to filter by')
-            self.parser.set_usage("usage: %prog search [<search_term>] [--galaxy-tags <galaxy_tag1,galaxy_tag2>] [--platforms platform]")
+            self.parser.add_option('--author', dest='author',
+                help='GitHub username')
+            self.parser.set_usage("usage: %prog search [searchterm1 searchterm2] [--galaxy-tags galaxy_tag1,galaxy_tag2] [--platforms platform1,platform2] [--author username]")
+        elif self.action == "setup":
+            self.parser.set_usage("usage: %prog setup [options] source github_user github_repo secret")
+            self.parser.add_option('--remove', dest='remove_id', default=None,
+                help='Remove the integration matching the provided ID value. Use --list to see ID values.')
+            self.parser.add_option('--list', dest="setup_list", action='store_true', default=False,
+                help='List all of your integrations.')
 
         # options that apply to more than one action
-        if self.action != "init":
+        if not self.action in ("delete","import","init","login","setup"):
             self.parser.add_option('-p', '--roles-path', dest='roles_path', default=C.DEFAULT_ROLES_PATH,
                 help='The path to the directory containing your roles. '
                      'The default is the roles_path configured in your '
                      'ansible.cfg file (/etc/ansible/roles if not configured)')
 
-        if self.action in ("info","init","install","search"):
-            self.parser.add_option('-s', '--server', dest='api_server', default="https://galaxy.ansible.com",
+        if self.action in ("import","info","init","install","login","search","setup","delete"):
+            self.parser.add_option('-s', '--server', dest='api_server', default=C.GALAXY_SERVER,
                 help='The API server destination')
-            self.parser.add_option('-c', '--ignore-certs', action='store_false', dest='validate_certs', default=True,
+            self.parser.add_option('-c', '--ignore-certs', action='store_true', dest='ignore_certs', default=False,
                 help='Ignore SSL certificate validation errors.')
 
         if self.action in ("init","install"):
             self.parser.add_option('-f', '--force', dest='force', action='store_true', default=False,
                 help='Force overwriting an existing role')
 
-        # get options, args and galaxy object
         self.options, self.args =self.parser.parse_args()
         display.verbosity = self.options.verbosity
         self.galaxy = Galaxy(self.options)
@@ -120,15 +142,13 @@ class GalaxyCLI(CLI):
         return True
 
     def run(self):
-
+        
         super(GalaxyCLI, self).run()
 
         # if not offline, get connect to galaxy api
-        if self.action in ("info","install", "search") or (self.action == 'init' and not self.options.offline):
-            api_server = self.options.api_server
-            self.api = GalaxyAPI(self.galaxy, api_server)
-            if not self.api:
-                raise AnsibleError("The API server (%s) is not responding, please try again later." % api_server)
+        if self.action in ("import","info","install","search","login","setup","delete") or \
+            (self.action == 'init' and not self.options.offline):
+            self.api = GalaxyAPI(self.galaxy)
 
         self.execute()
 
@@ -142,8 +162,8 @@ class GalaxyCLI(CLI):
 
     def _display_role_info(self, role_info):
 
-        text = "\nRole: %s \n" % role_info['name']
-        text += "\tdescription: %s \n" % role_info.get('description', '')
+        text = [u"", u"Role: %s" % to_unicode(role_info['name'])]
+        text.append(u"\tdescription: %s" % role_info.get('description', ''))
 
         for k in sorted(role_info.keys()):
 
@@ -152,14 +172,15 @@ class GalaxyCLI(CLI):
 
             if isinstance(role_info[k], dict):
                 text += "\t%s: \n" % (k)
+                text.append(u"\t%s:" % (k))
                 for key in sorted(role_info[k].keys()):
                     if key in self.SKIP_INFO_KEYS:
                         continue
-                    text += "\t\t%s: %s\n" % (key, role_info[k][key])
+                    text.append(u"\t\t%s: %s" % (key, role_info[k][key]))
             else:
-                text += "\t%s: %s\n" % (k, role_info[k])
+                text.append(u"\t%s: %s" % (k, role_info[k]))
 
-        return text
+        return u'\n'.join(text)
 
 ############################
 # execute actions
@@ -188,7 +209,7 @@ class GalaxyCLI(CLI):
                             "however it will reset any main.yml files that may have\n"
                             "been modified there already." % role_path)
 
-        # create the default README.md
+        # create default README.md
         if not os.path.exists(role_path):
             os.makedirs(role_path)
         readme_path = os.path.join(role_path, "README.md")
@@ -196,9 +217,16 @@ class GalaxyCLI(CLI):
         f.write(self.galaxy.default_readme)
         f.close()
 
+        # create default .travis.yml
+        travis = Environment().from_string(self.galaxy.default_travis).render()
+        f = open(os.path.join(role_path, '.travis.yml'), 'w')
+        f.write(travis)
+        f.close()
+
         for dir in GalaxyRole.ROLE_DIRS:
             dir_path = os.path.join(init_path, role_name, dir)
             main_yml_path = os.path.join(dir_path, 'main.yml')
+
             # create the directory if it doesn't exist already
             if not os.path.exists(dir_path):
                 os.makedirs(dir_path)
@@ -234,6 +262,20 @@ class GalaxyCLI(CLI):
                 f.write(rendered_meta)
                 f.close()
                 pass
+            elif dir == "tests":
+                # create tests/test.yml
+                inject = dict(
+                    role_name = role_name
+                )
+                playbook = Environment().from_string(self.galaxy.default_test).render(inject)
+                f = open(os.path.join(dir_path, 'test.yml'), 'w')
+                f.write(playbook)
+                f.close()
+
+                # create tests/inventory
+                f = open(os.path.join(dir_path, 'inventory'), 'w')
+                f.write('localhost')
+                f.close()
             elif dir not in ('files','templates'):
                 # just write a (mostly) empty YAML file for main.yml
                 f = open(main_yml_path, 'w')
@@ -282,9 +324,11 @@ class GalaxyCLI(CLI):
             if role_spec:
                 role_info.update(role_spec)
 
-            data += self._display_role_info(role_info)
+            data = self._display_role_info(role_info)
+            ### FIXME: This is broken in both 1.9 and 2.0 as
+            # _display_role_info() always returns something
             if not data:
-                data += "\n- the role %s was not found" % role
+                data = u"\n- the role %s was not found" % role
 
         self.pager(data)
 
@@ -325,7 +369,7 @@ class GalaxyCLI(CLI):
 
                     for role in required_roles:
                         role = RoleRequirement.role_yaml_parse(role)
-                        display.debug('found role %s in yaml file' % str(role))
+                        display.vvv('found role %s in yaml file' % str(role))
                         if 'name' not in role and 'scm' not in role:
                             raise AnsibleError("Must specify name or src for role")
                         roles_left.append(GalaxyRole(self.galaxy, **role))
@@ -345,10 +389,11 @@ class GalaxyCLI(CLI):
             # roles were specified directly, so we'll just go out grab them
             # (and their dependencies, unless the user doesn't want us to).
             for rname in self.args:
-                roles_left.append(GalaxyRole(self.galaxy, rname.strip()))
+                role = RoleRequirement.role_yaml_parse(rname.strip())
+                roles_left.append(GalaxyRole(self.galaxy, **role))
 
         for role in roles_left:
-            display.debug('Installing role %s ' % role.name)
+            display.vvv('Installing role %s ' % role.name)
             # query the galaxy API for the role data
 
             if role.install_info is not None and not force:
@@ -458,21 +503,188 @@ class GalaxyCLI(CLI):
         return 0
 
     def execute_search(self):
-
+        page_size = 1000
         search = None
-        if len(self.args) > 1:
-            raise AnsibleOptionsError("At most a single search term is allowed.")
-        elif len(self.args) == 1:
-            search = self.args.pop()
 
-        response = self.api.search_roles(search, self.options.platforms, self.options.tags)
+        if len(self.args):
+            terms = []
+            for i in range(len(self.args)):
+               terms.append(self.args.pop())
+            search = '+'.join(terms[::-1])
 
-        if 'count' in response:
-            display.display("Found %d roles matching your search:\n" % response['count'])
+        if not search and not self.options.platforms and not self.options.tags and not self.options.author:
+            raise AnsibleError("Invalid query. At least one search term, platform, galaxy tag or author must be provided.")
 
-        data = ''
-        if 'results' in response:
-            for role in response['results']:
-                data += self._display_role_info(role)
+        response = self.api.search_roles(search, platforms=self.options.platforms,
+            tags=self.options.tags, author=self.options.author, page_size=page_size)
 
+        if response['count'] == 0:
+            display.display("No roles match your search.", color="yellow")
+            return True
+
+        data = [u'']
+
+        if response['count'] > page_size:
+            data.append(u"Found %d roles matching your search. Showing first %s." % (response['count'], page_size))
+        else:
+            data.append(u"Found %d roles matching your search:" % response['count'])
+
+        max_len = []
+        for role in response['results']:
+            max_len.append(len(role['username'] + '.' + role['name']))
+        name_len = max(max_len)
+        format_str = u" %%-%ds %%s" % name_len
+        data.append(u'')
+        data.append(format_str % (u"Name", u"Description"))
+        data.append(format_str % (u"----", u"-----------"))
+        for role in response['results']:
+            data.append(format_str % (u'%s.%s' % (role['username'], role['name']), role['description']))
+
+        data = u'\n'.join(data)
         self.pager(data)
+
+        return True
+
+    def execute_login(self):
+        """
+        Verify user's identify via Github and retreive an auth token from Galaxy.
+        """
+        # Authenticate with github and retrieve a token
+        if self.options.token is None:
+            login = GalaxyLogin(self.galaxy)
+            github_token = login.create_github_token()
+        else:
+            github_token = self.options.token
+
+        galaxy_response = self.api.authenticate(github_token)
+
+        if self.options.token is None:
+            # Remove the token we created
+            login.remove_github_token()
+
+        # Store the Galaxy token
+        token = GalaxyToken()
+        token.set(galaxy_response['token'])
+
+        display.display("Succesfully logged into Galaxy as %s" % galaxy_response['username'])
+        return 0
+
+    def execute_import(self):
+        """
+        Import a role into Galaxy
+        """
+
+        colors = {
+            'INFO':    'normal',
+            'WARNING': 'yellow',
+            'ERROR':   'red',
+            'SUCCESS': 'green',
+            'FAILED':  'red'
+        }
+
+        if len(self.args) < 2:
+            raise AnsibleError("Expected a github_username and github_repository. Use --help.")
+
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+
+        if self.options.check_status:
+            task = self.api.get_import_task(github_user=github_user, github_repo=github_repo)
+        else:
+            # Submit an import request
+            task = self.api.create_import_task(github_user, github_repo, reference=self.options.reference)
+
+            if len(task) > 1:
+                # found multiple roles associated with github_user/github_repo
+                display.display("WARNING: More than one Galaxy role associated with Github repo %s/%s." % (github_user,github_repo),
+                    color='yellow')
+                display.display("The following Galaxy roles are being updated:" + u'\n', color='yellow')
+                for t in task:
+                    display.display('%s.%s' % (t['summary_fields']['role']['namespace'],t['summary_fields']['role']['name']), color='yellow')
+                display.display(u'\n' + "To properly namespace this role, remove each of the above and re-import %s/%s from scratch" % (github_user,github_repo),
+                    color='yellow')
+                return 0
+            # found a single role as expected
+            display.display("Successfully submitted import request %d" % task[0]['id'])
+            if not self.options.wait:
+                display.display("Role name: %s" % task[0]['summary_fields']['role']['name'])
+                display.display("Repo: %s/%s" % (task[0]['github_user'],task[0]['github_repo']))
+
+        if self.options.check_status or self.options.wait:
+            # Get the status of the import
+            msg_list = []
+            finished = False
+            while not finished:
+                task = self.api.get_import_task(task_id=task[0]['id'])
+                for msg in task[0]['summary_fields']['task_messages']:
+                    if msg['id'] not in msg_list:
+                        display.display(msg['message_text'], color=colors[msg['message_type']])
+                        msg_list.append(msg['id'])
+                if task[0]['state'] in ['SUCCESS', 'FAILED']:
+                    finished = True
+                else:
+                    time.sleep(10)
+
+        return 0
+
+    def execute_setup(self):
+        """
+        Setup an integration from Github or Travis
+        """
+
+        if self.options.setup_list:
+            # List existing integration secrets
+            secrets = self.api.list_secrets()
+            if len(secrets) == 0:
+                # None found
+                display.display("No integrations found.")
+                return 0
+            display.display(u'\n' + "ID         Source     Repo", color="green")
+            display.display("---------- ---------- ----------", color="green")
+            for secret in secrets:
+                display.display("%-10s %-10s %s/%s" % (secret['id'], secret['source'], secret['github_user'],
+                    secret['github_repo']),color="green")
+            return 0
+
+        if self.options.remove_id:
+            # Remove a secret
+            self.api.remove_secret(self.options.remove_id)
+            display.display("Secret removed. Integrations using this secret will not longer work.", color="green")
+            return 0
+
+        if len(self.args) < 4:
+            raise AnsibleError("Missing one or more arguments. Expecting: source github_user github_repo secret")
+            return 0
+
+        secret = self.args.pop()
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+        source = self.args.pop()
+
+        resp = self.api.add_secret(source, github_user, github_repo, secret)
+        display.display("Added integration for %s %s/%s" % (resp['source'], resp['github_user'], resp['github_repo']))
+
+        return 0
+
+    def execute_delete(self):
+        """
+        Delete a role from galaxy.ansible.com
+        """
+
+        if len(self.args) < 2:
+            raise AnsibleError("Missing one or more arguments. Expected: github_user github_repo")
+
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+        resp = self.api.delete_role(github_user, github_repo)
+
+        if len(resp['deleted_roles']) > 1:
+            display.display("Deleted the following roles:")
+            display.display("ID     User            Name")
+            display.display("------ --------------- ----------")
+            for role in resp['deleted_roles']:
+                display.display("%-8s %-15s %s" % (role.id,role.namespace,role.name))
+
+        display.display(resp['status'])
+
+        return True

lib/ansible/cli/playbook.py

@@ -30,6 +30,7 @@ from ansible.errors import AnsibleError, AnsibleOptionsError
 from ansible.executor.playbook_executor import PlaybookExecutor
 from ansible.inventory import Inventory
 from ansible.parsing.dataloader import DataLoader
+from ansible.playbook.play_context import PlayContext
 from ansible.utils.vars import load_extra_vars
 from ansible.vars import VariableManager
 
@@ -72,7 +73,7 @@ class PlaybookCLI(CLI):
         parser.add_option('--start-at-task', dest='start_at_task',
             help="start the playbook at the task matching this name")
 
-        self.options, self.args = parser.parse_args()
+        self.options, self.args = parser.parse_args(self.args[1:])
 
 
         self.parser = parser
@@ -152,18 +153,10 @@ class PlaybookCLI(CLI):
             for p in results:
 
                 display.display('\nplaybook: %s' % p['playbook'])
-                i = 1
-                for play in p['plays']:
-                    if play.name:
-                        playname = play.name
-                    else:
-                        playname = '#' + str(i)
-
-                    msg = "\n  PLAY: %s" % (playname)
-                    mytags = set()
-                    if self.options.listtags and play.tags:
-                        mytags = mytags.union(set(play.tags))
-                        msg += '    TAGS: [%s]' % (','.join(mytags))
+                for idx, play in enumerate(p['plays']):
+                    msg = "\n  play #%d (%s): %s" % (idx + 1, ','.join(play.hosts), play.name)
+                    mytags = set(play.tags)
+                    msg += '\tTAGS: [%s]' % (','.join(mytags))
 
                     if self.options.listhosts:
                         playhosts = set(inventory.get_hosts(play.hosts))
@@ -173,23 +166,40 @@ class PlaybookCLI(CLI):
 
                     display.display(msg)
 
+                    all_tags = set()
                     if self.options.listtags or self.options.listtasks:
-                        taskmsg = '    tasks:'
+                        taskmsg = ''
+                        if self.options.listtasks:
+                            taskmsg = '    tasks:\n'
 
+                        all_vars = variable_manager.get_vars(loader=loader, play=play)
+                        play_context = PlayContext(play=play, options=self.options)
                         for block in play.compile():
+                            block = block.filter_tagged_tasks(play_context, all_vars)
                             if not block.has_tasks():
                                 continue
 
-                            j = 1
                             for task in block.block:
-                                taskmsg += "\n      %s" % task
-                                if self.options.listtags and task.tags:
-                                    taskmsg += "    TAGS: [%s]" % ','.join(mytags.union(set(task.tags)))
-                                j = j + 1
+                                if task.action == 'meta':
+                                    continue
+
+                                all_tags.update(task.tags)
+                                if self.options.listtasks:
+                                    cur_tags = list(mytags.union(set(task.tags)))
+                                    cur_tags.sort()
+                                    if task.name:
+                                        taskmsg += "      %s" % task.get_name()
+                                    else:
+                                        taskmsg += "      %s" % task.action
+                                    taskmsg += "\tTAGS: [%s]\n" % ', '.join(cur_tags)
+
+                        if self.options.listtags:
+                            cur_tags = list(mytags.union(all_tags))
+                            cur_tags.sort()
+                            taskmsg += "      TASK TAGS: [%s]\n" % ', '.join(cur_tags)
 
                         display.display(taskmsg)
 
-                    i = i + 1
             return 0
         else:
             return results

lib/ansible/cli/pull.py

@@ -64,18 +64,24 @@ class PullCLI(CLI):
             subset_opts=True,
             inventory_opts=True,
             module_opts=True,
+            runas_prompt_opts=True,
         )
 
         # options unique to pull
-        self.parser.add_option('--purge', default=False, action='store_true', help='purge checkout after playbook run')
+        self.parser.add_option('--purge', default=False, action='store_true',
+            help='purge checkout after playbook run')
         self.parser.add_option('-o', '--only-if-changed', dest='ifchanged', default=False, action='store_true',
             help='only run the playbook if the repository has been updated')
         self.parser.add_option('-s', '--sleep', dest='sleep', default=None,
             help='sleep for random interval (between 0 and n number of seconds) before starting. This is a useful way to disperse git requests')
         self.parser.add_option('-f', '--force', dest='force', default=False, action='store_true',
             help='run the playbook even if the repository could not be updated')
-        self.parser.add_option('-d', '--directory', dest='dest', default='~/.ansible/pull', help='directory to checkout repository to')
-        self.parser.add_option('-U', '--url', dest='url', default=None, help='URL of the playbook repository')
+        self.parser.add_option('-d', '--directory', dest='dest', default=None,
+            help='directory to checkout repository to')
+        self.parser.add_option('-U', '--url', dest='url', default=None,
+            help='URL of the playbook repository')
+        self.parser.add_option('--full', dest='fullclone', action='store_true',
+            help='Do a full clone, instead of a shallow one.')
         self.parser.add_option('-C', '--checkout', dest='checkout',
             help='branch/tag/commit to checkout.  ' 'Defaults to behavior of repository module.')
         self.parser.add_option('--accept-host-key', default=False, dest='accept_host_key', action='store_true',
@@ -86,7 +92,16 @@ class PullCLI(CLI):
             help='verify GPG signature of checked out commit, if it fails abort running the playbook.'
                  ' This needs the corresponding VCS module to support such an operation')
 
-        self.options, self.args = self.parser.parse_args()
+        # for pull we don't wan't a default
+        self.parser.set_defaults(inventory=None)
+
+        self.options, self.args = self.parser.parse_args(self.args[1:])
+
+        if not self.options.dest:
+            hostname = socket.getfqdn()
+            # use a hostname dependent directory, in case of $HOME on nfs
+            self.options.dest = os.path.join('~/.ansible/pull', hostname)
+        self.options.dest = os.path.expandvars(os.path.expanduser(self.options.dest))
 
         if self.options.sleep:
             try:
@@ -119,18 +134,18 @@ class PullCLI(CLI):
         node = platform.node()
         host = socket.getfqdn()
         limit_opts = 'localhost,%s,127.0.0.1' % ','.join(set([host, node, host.split('.')[0], node.split('.')[0]]))
-        base_opts = '-c local "%s"' % limit_opts
+        base_opts = '-c local '
         if self.options.verbosity > 0:
             base_opts += ' -%s' % ''.join([ "v" for x in range(0, self.options.verbosity) ])
 
         # Attempt to use the inventory passed in as an argument
-        # It might not yet have been downloaded so use localhost if note
-        if not self.options.inventory or not os.path.exists(self.options.inventory):
+        # It might not yet have been downloaded so use localhost as default
+        if not self.options.inventory or ( ',' not in self.options.inventory and not os.path.exists(self.options.inventory)):
             inv_opts = 'localhost,'
         else:
             inv_opts = self.options.inventory
 
-        #TODO: enable more repo modules hg/svn?
+        #FIXME: enable more repo modules hg/svn?
         if self.options.module_name == 'git':
             repo_opts = "name=%s dest=%s" % (self.options.url, self.options.dest)
             if self.options.checkout:
@@ -145,14 +160,16 @@ class PullCLI(CLI):
             if self.options.verify:
                 repo_opts += ' verify_commit=yes'
 
+            if not self.options.fullclone:
+                repo_opts += ' depth=1'
+
         path = module_loader.find_plugin(self.options.module_name)
         if path is None:
             raise AnsibleOptionsError(("module '%s' not found.\n" % self.options.module_name))
 
         bin_path = os.path.dirname(os.path.abspath(sys.argv[0]))
-        cmd = '%s/ansible -i "%s" %s -m %s -a "%s"' % (
-            bin_path, inv_opts, base_opts, self.options.module_name, repo_opts
-        )
+        # hardcode local and inventory/host as this is just meant to fetch the repo
+        cmd = '%s/ansible -i "%s" %s -m %s -a "%s" all -l "%s"' % (bin_path, inv_opts, base_opts, self.options.module_name, repo_opts, limit_opts)
 
         for ev in self.options.extra_vars:
             cmd += ' -e "%s"' % ev
@@ -163,6 +180,8 @@ class PullCLI(CLI):
             time.sleep(self.options.sleep)
 
         # RUN the Checkout command
+        display.debug("running ansible with VCS module to checkout repo")
+        display.vvvv('EXEC: %s' % cmd)
         rc, out, err = run_cmd(cmd, live=True)
 
         if rc != 0:
@@ -174,8 +193,7 @@ class PullCLI(CLI):
             display.display("Repository has not changed, quitting.")
             return 0
 
-        playbook = self.select_playbook(path)
-
+        playbook = self.select_playbook(self.options.dest)
         if playbook is None:
             raise AnsibleOptionsError("Could not find a playbook to run.")
 
@@ -187,16 +205,20 @@ class PullCLI(CLI):
             cmd += ' -i "%s"' % self.options.inventory
         for ev in self.options.extra_vars:
             cmd += ' -e "%s"' % ev
-        if self.options.ask_sudo_pass:
-            cmd += ' -K'
+        if self.options.ask_sudo_pass or self.options.ask_su_pass or self.options.become_ask_pass:
+            cmd += ' --ask-become-pass'
         if self.options.tags:
             cmd += ' -t "%s"' % self.options.tags
-        if self.options.limit:
-            cmd += ' -l "%s"' % self.options.limit
+        if self.options.subset:
+            cmd += ' -l "%s"' % self.options.subset
+        else:
+            cmd += ' -l "%s"' % limit_opts
 
         os.chdir(self.options.dest)
 
         # RUN THE PLAYBOOK COMMAND
+        display.debug("running ansible-playbook to do actual work")
+        display.debug('EXEC: %s' % cmd)
         rc, out, err = run_cmd(cmd, live=True)
 
         if self.options.purge:

lib/ansible/cli/vault.py

@@ -26,6 +26,7 @@ from ansible.errors import AnsibleError, AnsibleOptionsError
 from ansible.parsing.dataloader import DataLoader
 from ansible.parsing.vault import VaultEditor
 from ansible.cli import CLI
+from ansible.utils.unicode import to_unicode
 
 try:
     from __main__ import display
@@ -69,7 +70,7 @@ class VaultCLI(CLI):
         elif self.action == "rekey":
             self.parser.set_usage("usage: %prog rekey [options] file_name")
 
-        self.options, self.args = self.parser.parse_args()
+        self.options, self.args = self.parser.parse_args(self.args[1:])
         display.verbosity = self.options.verbosity
 
         can_output = ['encrypt', 'decrypt']
@@ -157,7 +158,12 @@ class VaultCLI(CLI):
     def execute_view(self):
 
         for f in self.args:
-            self.pager(self.editor.plaintext(f))
+            # Note: vault should return byte strings because it could encrypt
+            # and decrypt binary files.  We are responsible for changing it to
+            # unicode here because we are displaying it and therefore can make
+            # the decision that the display doesn't have to be precisely what
+            # the input was (leave that to decrypt instead)
+            self.pager(to_unicode(self.editor.plaintext(f)))
 
     def execute_rekey(self):
         for f in self.args:

lib/ansible/constants.py

@@ -120,19 +120,23 @@ DEFAULT_COW_WHITELIST = ['bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'd
 # sections in config file
 DEFAULTS='defaults'
 
+# FIXME: add deprecation warning when these get set
+#### DEPRECATED VARS #### 
+# use more sanely named 'inventory'
 DEPRECATED_HOST_LIST  = get_config(p, DEFAULTS, 'hostfile', 'ANSIBLE_HOSTS', '/etc/ansible/hosts', ispath=True)
+# this is not used since 0.5 but people might still have in config
+DEFAULT_PATTERN           = get_config(p, DEFAULTS, 'pattern', None, None)
 
-# generally configurable things
+#### GENERALLY CONFIGURABLE THINGS ####
 DEFAULT_DEBUG             = get_config(p, DEFAULTS, 'debug',            'ANSIBLE_DEBUG',            False, boolean=True)
 DEFAULT_HOST_LIST         = get_config(p, DEFAULTS,'inventory', 'ANSIBLE_INVENTORY', DEPRECATED_HOST_LIST, ispath=True)
 DEFAULT_MODULE_PATH       = get_config(p, DEFAULTS, 'library',          'ANSIBLE_LIBRARY',          None, ispath=True)
 DEFAULT_ROLES_PATH        = get_config(p, DEFAULTS, 'roles_path',       'ANSIBLE_ROLES_PATH',       '/etc/ansible/roles', ispath=True)
 DEFAULT_REMOTE_TMP        = get_config(p, DEFAULTS, 'remote_tmp',       'ANSIBLE_REMOTE_TEMP',      '$HOME/.ansible/tmp')
 DEFAULT_MODULE_NAME       = get_config(p, DEFAULTS, 'module_name',      None,                       'command')
-DEFAULT_PATTERN           = get_config(p, DEFAULTS, 'pattern',          None,                       '*')
 DEFAULT_FORKS             = get_config(p, DEFAULTS, 'forks',            'ANSIBLE_FORKS',            5, integer=True)
 DEFAULT_MODULE_ARGS       = get_config(p, DEFAULTS, 'module_args',      'ANSIBLE_MODULE_ARGS',      '')
-DEFAULT_MODULE_LANG       = get_config(p, DEFAULTS, 'module_lang',      'ANSIBLE_MODULE_LANG',      'en_US.UTF-8')
+DEFAULT_MODULE_LANG       = get_config(p, DEFAULTS, 'module_lang',      'ANSIBLE_MODULE_LANG',      os.getenv('LANG', 'en_US.UTF-8'))
 DEFAULT_TIMEOUT           = get_config(p, DEFAULTS, 'timeout',          'ANSIBLE_TIMEOUT',          10, integer=True)
 DEFAULT_POLL_INTERVAL     = get_config(p, DEFAULTS, 'poll_interval',    'ANSIBLE_POLL_INTERVAL',    15, integer=True)
 DEFAULT_REMOTE_USER       = get_config(p, DEFAULTS, 'remote_user',      'ANSIBLE_REMOTE_USER',      None)
@@ -159,7 +163,7 @@ DEFAULT_VAR_COMPRESSION_LEVEL = get_config(p, DEFAULTS, 'var_compression_level',
 
 # disclosure
 DEFAULT_NO_LOG           = get_config(p, DEFAULTS, 'no_log', 'ANSIBLE_NO_LOG', False, boolean=True)
-DEFAULT_NO_TARGET_SYSLOG   = get_config(p, DEFAULTS, 'no_target_syslog', 'ANSIBLE_NO_TARGET_SYSLOG', True, boolean=True)
+DEFAULT_NO_TARGET_SYSLOG   = get_config(p, DEFAULTS, 'no_target_syslog', 'ANSIBLE_NO_TARGET_SYSLOG', False, boolean=True)
 
 # selinux
 DEFAULT_SELINUX_SPECIAL_FS = get_config(p, 'selinux', 'special_context_filesystems', None, 'fuse, nfs, vboxsf, ramfs', islist=True)
@@ -197,7 +201,7 @@ DEFAULT_BECOME_ASK_PASS   = get_config(p, 'privilege_escalation', 'become_ask_pa
 # the module takes both, bad things could happen.
 # In the future we should probably generalize this even further
 # (mapping of param: squash field)
-DEFAULT_SQUASH_ACTIONS         = get_config(p, DEFAULTS, 'squash_actions',     'ANSIBLE_SQUASH_ACTIONS', "apt, yum, pkgng, zypper, dnf", islist=True)
+DEFAULT_SQUASH_ACTIONS         = get_config(p, DEFAULTS, 'squash_actions',     'ANSIBLE_SQUASH_ACTIONS', "apt, dnf, package, pkgng, yum, zypper", islist=True)
 # paths
 DEFAULT_ACTION_PLUGIN_PATH     = get_config(p, DEFAULTS, 'action_plugins',     'ANSIBLE_ACTION_PLUGINS', '~/.ansible/plugins/action:/usr/share/ansible/plugins/action', ispath=True)
 DEFAULT_CACHE_PLUGIN_PATH      = get_config(p, DEFAULTS, 'cache_plugins',      'ANSIBLE_CACHE_PLUGINS', '~/.ansible/plugins/cache:/usr/share/ansible/plugins/cache', ispath=True)
@@ -231,7 +235,7 @@ COMMAND_WARNINGS               = get_config(p, DEFAULTS, 'command_warnings', 'AN
 DEFAULT_LOAD_CALLBACK_PLUGINS  = get_config(p, DEFAULTS, 'bin_ansible_callbacks', 'ANSIBLE_LOAD_CALLBACK_PLUGINS', False, boolean=True)
 DEFAULT_CALLBACK_WHITELIST     = get_config(p, DEFAULTS, 'callback_whitelist', 'ANSIBLE_CALLBACK_WHITELIST', [], islist=True)
 RETRY_FILES_ENABLED            = get_config(p, DEFAULTS, 'retry_files_enabled', 'ANSIBLE_RETRY_FILES_ENABLED', True, boolean=True)
-RETRY_FILES_SAVE_PATH          = get_config(p, DEFAULTS, 'retry_files_save_path', 'ANSIBLE_RETRY_FILES_SAVE_PATH', '~/', ispath=True)
+RETRY_FILES_SAVE_PATH          = get_config(p, DEFAULTS, 'retry_files_save_path', 'ANSIBLE_RETRY_FILES_SAVE_PATH', None, ispath=True)
 DEFAULT_NULL_REPRESENTATION    = get_config(p, DEFAULTS, 'null_representation', 'ANSIBLE_NULL_REPRESENTATION', None, isnone=True)
 
 # CONNECTION RELATED
@@ -255,12 +259,14 @@ ACCELERATE_MULTI_KEY           = get_config(p, 'accelerate', 'accelerate_multi_k
 PARAMIKO_PTY                   = get_config(p, 'paramiko_connection', 'pty', 'ANSIBLE_PARAMIKO_PTY', True, boolean=True)
 
 # galaxy related
-DEFAULT_GALAXY_URI             = get_config(p, 'galaxy', 'server_uri', 'ANSIBLE_GALAXY_SERVER_URI', 'https://galaxy.ansible.com')
+GALAXY_SERVER                  = get_config(p, 'galaxy', 'server', 'ANSIBLE_GALAXY_SERVER', 'https://galaxy.ansible.com')
+GALAXY_IGNORE_CERTS            = get_config(p, 'galaxy', 'ignore_certs', 'ANSIBLE_GALAXY_IGNORE', False, boolean=True)
 # this can be configured to blacklist SCMS but cannot add new ones unless the code is also updated
 GALAXY_SCMS                    = get_config(p, 'galaxy', 'scms', 'ANSIBLE_GALAXY_SCMS', 'git, hg', islist=True)
 
 # characters included in auto-generated passwords
 DEFAULT_PASSWORD_CHARS = ascii_letters + digits + ".,:-_"
+STRING_TYPE_FILTERS = get_config(p, 'jinja2', 'dont_type_filters', 'ANSIBLE_STRING_TYPE_FILTERS', ['string', 'to_json', 'to_nice_json', 'to_yaml', 'ppretty', 'json'], islist=True )
 
 # non-configurable things
 MODULE_REQUIRE_ARGS       = ['command', 'shell', 'raw', 'script']

lib/ansible/errors/__init__.py

@@ -44,7 +44,7 @@ class AnsibleError(Exception):
     which should be returned by the DataLoader() class.
     '''
 
-    def __init__(self, message, obj=None, show_content=True):
+    def __init__(self, message="", obj=None, show_content=True):
         # we import this here to prevent an import loop problem,
         # since the objects code also imports ansible.errors
         from ansible.parsing.yaml.objects import AnsibleBaseYAMLObject
@@ -54,9 +54,9 @@ class AnsibleError(Exception):
         if obj and isinstance(obj, AnsibleBaseYAMLObject):
             extended_error = self._get_extended_error()
             if extended_error:
-                self.message = 'ERROR! %s\n\n%s' % (message, to_str(extended_error))
+                self.message = '%s\n\n%s' % (to_str(message), to_str(extended_error))
         else:
-            self.message = 'ERROR! %s' % message
+            self.message = '%s' % to_str(message)
 
     def __str__(self):
         return self.message

lib/ansible/executor/module_common.py

@@ -97,7 +97,7 @@ def _find_snippet_imports(module_data, module_path, strip_comments):
             if " import *" not in line:
                 import_error = True
             if import_error:
-                raise AnsibleError("error importing module in %s, expecting format like 'from ansible.module_utils.basic import *'" % module_path)
+                raise AnsibleError("error importing module in %s, expecting format like 'from ansible.module_utils.<lib name> import *'" % module_path)
             snippet_name = tokens[2].split()[0]
             snippet_names.append(snippet_name)
             output.write(_slurp(os.path.join(_SNIPPET_PATH, snippet_name + ".py")))

lib/ansible/executor/play_iterator.py

@@ -49,6 +49,7 @@ class HostState:
         self.cur_rescue_task    = 0
         self.cur_always_task    = 0
         self.cur_role           = None
+        self.cur_dep_chain      = None
         self.run_state          = PlayIterator.ITERATING_SETUP
         self.fail_state         = PlayIterator.FAILED_NONE
         self.pending_setup      = False
@@ -57,14 +58,32 @@ class HostState:
         self.always_child_state = None
 
     def __repr__(self):
-        return "HOST STATE: block=%d, task=%d, rescue=%d, always=%d, role=%s, run_state=%d, fail_state=%d, pending_setup=%s, tasks child state? %s, rescue child state? %s, always child state? %s" % (
+        def _run_state_to_string(n):
+            states = ["ITERATING_SETUP", "ITERATING_TASKS", "ITERATING_RESCUE", "ITERATING_ALWAYS", "ITERATING_COMPLETE"]
+            try:
+                return states[n]
+            except IndexError:
+                return "UNKNOWN STATE"
+
+        def _failed_state_to_string(n):
+            states = {1:"FAILED_SETUP", 2:"FAILED_TASKS", 4:"FAILED_RESCUE", 8:"FAILED_ALWAYS"}
+            if n == 0:
+                return "FAILED_NONE"
+            else:
+                ret = []
+                for i in (1, 2, 4, 8):
+                    if n & i:
+                        ret.append(states[i])
+                return "|".join(ret)
+
+        return "HOST STATE: block=%d, task=%d, rescue=%d, always=%d, role=%s, run_state=%s, fail_state=%s, pending_setup=%s, tasks child state? %s, rescue child state? %s, always child state? %s" % (
             self.cur_block,
             self.cur_regular_task,
             self.cur_rescue_task,
             self.cur_always_task,
             self.cur_role,
-            self.run_state,
-            self.fail_state,
+            _run_state_to_string(self.run_state),
+            _failed_state_to_string(self.fail_state),
             self.pending_setup,
             self.tasks_child_state,
             self.rescue_child_state,
@@ -84,6 +103,8 @@ class HostState:
         new_state.run_state        = self.run_state
         new_state.fail_state       = self.fail_state
         new_state.pending_setup    = self.pending_setup
+        if self.cur_dep_chain is not None:
+            new_state.cur_dep_chain = self.cur_dep_chain[:]
         if self.tasks_child_state is not None:
             new_state.tasks_child_state = self.tasks_child_state.copy()
         if self.rescue_child_state is not None:
@@ -111,38 +132,54 @@ class PlayIterator:
 
     def __init__(self, inventory, play, play_context, variable_manager, all_vars, start_at_done=False):
         self._play = play
-
         self._blocks = []
+
+        setup_block = Block(play=self._play)
+        setup_task = Task(block=setup_block)
+        setup_task.action = 'setup'
+        setup_task.tags   = ['always']
+        setup_task.args   = {}
+        setup_task.set_loader(self._play._loader)
+        setup_block.block = [setup_task]
+
+        setup_block = setup_block.filter_tagged_tasks(play_context, all_vars)
+        self._blocks.append(setup_block)
+
         for block in self._play.compile():
             new_block = block.filter_tagged_tasks(play_context, all_vars)
             if new_block.has_tasks():
                 self._blocks.append(new_block)
 
         self._host_states = {}
+        start_at_matched = False
         for host in inventory.get_hosts(self._play.hosts):
-             self._host_states[host.name] = HostState(blocks=self._blocks)
-             # if the host's name is in the variable manager's fact cache, then set
-             # its _gathered_facts flag to true for smart gathering tests later
-             if host.name in variable_manager._fact_cache:
-                 host._gathered_facts = True
-             # if we're looking to start at a specific task, iterate through
-             # the tasks for this host until we find the specified task
-             if play_context.start_at_task is not None and not start_at_done:
-                 while True:
-                     (s, task) = self.get_next_task_for_host(host, peek=True)
-                     if s.run_state == self.ITERATING_COMPLETE:
-                         break
-                     if task.name == play_context.start_at_task or fnmatch.fnmatch(task.name, play_context.start_at_task) or \
-                        task.get_name() == play_context.start_at_task or fnmatch.fnmatch(task.get_name(), play_context.start_at_task):
-                         # we have our match, so clear the start_at_task field on the
-                         # play context to flag that we've started at a task (and future
-                         # plays won't try to advance)
-                         play_context.start_at_task = None
-                         break
-                     else:
-                         self.get_next_task_for_host(host)
-                 # finally, reset the host's state to ITERATING_SETUP
-                 self._host_states[host.name].run_state = self.ITERATING_SETUP
+            self._host_states[host.name] = HostState(blocks=self._blocks)
+            # if the host's name is in the variable manager's fact cache, then set
+            # its _gathered_facts flag to true for smart gathering tests later
+            if host.name in variable_manager._fact_cache:
+                host._gathered_facts = True
+            # if we're looking to start at a specific task, iterate through
+            # the tasks for this host until we find the specified task
+            if play_context.start_at_task is not None and not start_at_done:
+                while True:
+                    (s, task) = self.get_next_task_for_host(host, peek=True)
+                    if s.run_state == self.ITERATING_COMPLETE:
+                        break
+                    if task.name == play_context.start_at_task or fnmatch.fnmatch(task.name, play_context.start_at_task) or \
+                       task.get_name() == play_context.start_at_task or fnmatch.fnmatch(task.get_name(), play_context.start_at_task):
+                        start_at_matched = True
+                        break
+                    else:
+                        self.get_next_task_for_host(host)
+
+                # finally, reset the host's state to ITERATING_SETUP
+                self._host_states[host.name].run_state = self.ITERATING_SETUP
+
+        if start_at_matched:
+            # we have our match, so clear the start_at_task field on the
+            # play context to flag that we've started at a task (and future
+            # plays won't try to advance)
+            play_context.start_at_task = None
 
         # Extend the play handlers list to include the handlers defined in roles
         self._play.handlers.extend(play.compile_roles_handlers())
@@ -161,41 +198,23 @@ class PlayIterator:
         task = None
         if s.run_state == self.ITERATING_COMPLETE:
             display.debug("host %s is done iterating, returning" % host.name)
-            return (None, None)
-        elif s.run_state == self.ITERATING_SETUP:
-            s.run_state = self.ITERATING_TASKS
-            s.pending_setup = True
+            return (s, None)
 
-            # Gather facts if the default is 'smart' and we have not yet
-            # done it for this host; or if 'explicit' and the play sets
-            # gather_facts to True; or if 'implicit' and the play does
-            # NOT explicitly set gather_facts to False.
+        old_s = s
+        (s, task) = self._get_next_task_from_state(s, host=host, peek=peek)
 
-            gathering = C.DEFAULT_GATHERING
-            implied = self._play.gather_facts is None or boolean(self._play.gather_facts)
-
-            if (gathering == 'implicit' and implied) or \
-               (gathering == 'explicit' and boolean(self._play.gather_facts)) or \
-               (gathering == 'smart' and implied and not host._gathered_facts):
-                if not peek:
-                    # mark the host as having gathered facts
-                    host.set_gathered_facts(True)
-
-                task = Task()
-                task.action = 'setup'
-                task.args   = {}
-                task.set_loader(self._play._loader)
+        def _roles_are_different(ra, rb):
+            if ra != rb:
+                return True
             else:
-                s.pending_setup = False
-
-        if not task:
-            (s, task) = self._get_next_task_from_state(s, peek=peek)
+                return old_s.cur_dep_chain != task._block.get_dep_chain()
 
         if task and task._role:
             # if we had a current role, mark that role as completed
-            if s.cur_role and task._role != s.cur_role and host.name in s.cur_role._had_task_run and not peek:
+            if s.cur_role and _roles_are_different(task._role, s.cur_role) and host.name in s.cur_role._had_task_run and not peek:
                 s.cur_role._completed[host.name] = True
             s.cur_role = task._role
+            s.cur_dep_chain = task._block.get_dep_chain()
 
         if not peek:
             self._host_states[host.name] = s
@@ -206,7 +225,7 @@ class PlayIterator:
         return (s, task)
 
 
-    def _get_next_task_from_state(self, state, peek):
+    def _get_next_task_from_state(self, state, host, peek):
 
         task = None
 
@@ -221,7 +240,38 @@ class PlayIterator:
                 state.run_state = self.ITERATING_COMPLETE
                 return (state, None)
 
-            if state.run_state == self.ITERATING_TASKS:
+            if state.run_state == self.ITERATING_SETUP:
+                if not state.pending_setup:
+                    state.pending_setup = True
+
+                    # Gather facts if the default is 'smart' and we have not yet
+                    # done it for this host; or if 'explicit' and the play sets
+                    # gather_facts to True; or if 'implicit' and the play does
+                    # NOT explicitly set gather_facts to False.
+
+                    gathering = C.DEFAULT_GATHERING
+                    implied = self._play.gather_facts is None or boolean(self._play.gather_facts)
+
+                    if (gathering == 'implicit' and implied) or \
+                       (gathering == 'explicit' and boolean(self._play.gather_facts)) or \
+                       (gathering == 'smart' and implied and not host._gathered_facts):
+                        # mark the host as having gathered facts
+                        setup_block = self._blocks[0]
+                        if setup_block.has_tasks() and len(setup_block.block) > 0:
+                            task = setup_block.block[0]
+                        if not peek:
+                            host.set_gathered_facts(True)
+                else:
+                    state.pending_setup = False
+
+                    state.cur_block += 1
+                    state.cur_regular_task = 0
+                    state.cur_rescue_task  = 0
+                    state.cur_always_task  = 0
+                    state.run_state = self.ITERATING_TASKS
+                    state.child_state = None
+
+            elif state.run_state == self.ITERATING_TASKS:
                 # clear the pending setup flag, since we're past that and it didn't fail
                 if state.pending_setup:
                     state.pending_setup = False
@@ -238,7 +288,7 @@ class PlayIterator:
                             state.tasks_child_state = HostState(blocks=[task])
                             state.tasks_child_state.run_state = self.ITERATING_TASKS
                             state.tasks_child_state.cur_role = state.cur_role
-                        (state.tasks_child_state, task) = self._get_next_task_from_state(state.tasks_child_state, peek=peek)
+                        (state.tasks_child_state, task) = self._get_next_task_from_state(state.tasks_child_state, host=host, peek=peek)
                         if task is None:
                             # check to see if the child state was failed, if so we need to
                             # fail here too so we don't continue iterating tasks
@@ -264,7 +314,7 @@ class PlayIterator:
                             state.rescue_child_state = HostState(blocks=[task])
                             state.rescue_child_state.run_state = self.ITERATING_TASKS
                             state.rescue_child_state.cur_role = state.cur_role
-                        (state.rescue_child_state, task) = self._get_next_task_from_state(state.rescue_child_state, peek=peek)
+                        (state.rescue_child_state, task) = self._get_next_task_from_state(state.rescue_child_state, host=host, peek=peek)
                         if task is None:
                             # check to see if the child state was failed, if so we need to
                             # fail here too so we don't continue iterating rescue
@@ -294,7 +344,7 @@ class PlayIterator:
                             state.always_child_state = HostState(blocks=[task])
                             state.always_child_state.run_state = self.ITERATING_TASKS
                             state.always_child_state.cur_role = state.cur_role
-                        (state.always_child_state, task) = self._get_next_task_from_state(state.always_child_state, peek=peek)
+                        (state.always_child_state, task) = self._get_next_task_from_state(state.always_child_state, host=host, peek=peek)
                         if task is None:
                             # check to see if the child state was failed, if so we need to
                             # fail here too so we don't continue iterating always
@@ -324,13 +374,21 @@ class PlayIterator:
                 state.tasks_child_state = self._set_failed_state(state.tasks_child_state)
             else:
                 state.fail_state |= self.FAILED_TASKS
-                state.run_state = self.ITERATING_RESCUE
+                if state._blocks[state.cur_block].rescue:
+                    state.run_state = self.ITERATING_RESCUE
+                elif state._blocks[state.cur_block].always:
+                    state.run_state = self.ITERATING_ALWAYS
+                else:
+                    state.run_state = self.ITERATING_COMPLETE
         elif state.run_state == self.ITERATING_RESCUE:
             if state.rescue_child_state is not None:
                 state.rescue_child_state = self._set_failed_state(state.rescue_child_state)
             else:
                 state.fail_state |= self.FAILED_RESCUE
-                state.run_state = self.ITERATING_ALWAYS
+                if state._blocks[state.cur_block].always:
+                    state.run_state = self.ITERATING_ALWAYS
+                else:
+                    state.run_state = self.ITERATING_COMPLETE
         elif state.run_state == self.ITERATING_ALWAYS:
             if state.always_child_state is not None:
                 state.always_child_state = self._set_failed_state(state.always_child_state)
@@ -347,6 +405,32 @@ class PlayIterator:
     def get_failed_hosts(self):
         return dict((host, True) for (host, state) in iteritems(self._host_states) if state.run_state == self.ITERATING_COMPLETE and state.fail_state != self.FAILED_NONE)
 
+    def _check_failed_state(self, state):
+        if state is None:
+            return False
+        elif state.run_state == self.ITERATING_TASKS and self._check_failed_state(state.tasks_child_state):
+            cur_block = self._blocks[state.cur_block]
+            if len(cur_block.rescue) > 0 and state.fail_state & self.FAILED_RESCUE == 0:
+                return False
+            else:
+                return True
+        elif state.run_state == self.ITERATING_RESCUE and self._check_failed_state(state.rescue_child_state):
+            return True
+        elif state.run_state == self.ITERATING_ALWAYS and self._check_failed_state(state.always_child_state):
+            return True
+        elif state.run_state == self.ITERATING_COMPLETE and state.fail_state != self.FAILED_NONE:
+            if state.run_state == self.ITERATING_RESCUE and state.fail_state&self.FAILED_RESCUE == 0:
+                return False
+            elif state.run_state == self.ITERATING_ALWAYS and state.fail_state&self.FAILED_ALWAYS == 0:
+                return False
+            else:
+                return True
+        return False
+
+    def is_failed(self, host):
+        s = self.get_host_state(host)
+        return self._check_failed_state(s)
+
     def get_original_task(self, host, task):
         '''
         Finds the task in the task list which matches the UUID of the given task.
@@ -396,7 +480,8 @@ class PlayIterator:
         return None
 
     def _insert_tasks_into_state(self, state, task_list):
-        if state.fail_state != self.FAILED_NONE:
+        # if we've failed at all, or if the task list is empty, just return the current state
+        if state.fail_state != self.FAILED_NONE and state.run_state not in (self.ITERATING_RESCUE, self.ITERATING_ALWAYS) or not task_list:
             return state
 
         if state.run_state == self.ITERATING_TASKS:

lib/ansible/executor/playbook_executor.py

@@ -19,22 +19,15 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import getpass
-import locale
 import os
-import signal
-import sys
 
 from ansible.compat.six import string_types
 
+from ansible import constants as C
 from ansible.executor.task_queue_manager import TaskQueueManager
 from ansible.playbook import Playbook
 from ansible.template import Templar
 
-from ansible.utils.color import colorize, hostcolor
-from ansible.utils.encrypt import do_encrypt
-from ansible.utils.unicode import to_unicode
-
 try:
     from __main__ import display
 except ImportError:
@@ -70,8 +63,6 @@ class PlaybookExecutor:
         may limit the runs to serialized groups, etc.
         '''
 
-        signal.signal(signal.SIGINT, self._cleanup)
-
         result = 0
         entrylist = []
         entry = {}
@@ -83,6 +74,10 @@ class PlaybookExecutor:
                 if self._tqm is None: # we are doing a listing
                     entry = {'playbook': playbook_path}
                     entry['plays'] = []
+                else:
+                    # make sure the tqm has callbacks loaded
+                    self._tqm.load_callbacks()
+                    self._tqm.send_callback('v2_playbook_on_start', pb)
 
                 i = 1
                 plays = pb.get_plays()
@@ -108,10 +103,12 @@ class PlaybookExecutor:
                             salt_size = var.get("salt_size", None)
                             salt      = var.get("salt", None)
 
-                            if vname not in play.vars:
+                            if vname not in self._variable_manager.extra_vars:
                                 if self._tqm:
                                     self._tqm.send_callback('v2_playbook_on_vars_prompt', vname, private, prompt, encrypt, confirm, salt_size, salt, default)
-                                play.vars[vname] = self._do_var_prompt(vname, private, prompt, encrypt, confirm, salt_size, salt, default)
+                                    play.vars[vname] = display.do_var_prompt(vname, private, prompt, encrypt, confirm, salt_size, salt, default)
+                                else: # we are either in --list-<option> or syntax check
+                                    play.vars[vname] = default
 
                     # Create a temporary copy of the play here, so we can run post_validate
                     # on it without the templating changes affecting the original object.
@@ -128,8 +125,6 @@ class PlaybookExecutor:
                         entry['plays'].append(new_play)
 
                     else:
-                        # make sure the tqm has callbacks loaded
-                        self._tqm.load_callbacks()
                         self._tqm._unreachable_hosts.update(self._unreachable_hosts)
 
                         # we are actually running plays
@@ -149,9 +144,7 @@ class PlaybookExecutor:
                             # conditions are met, we break out, otherwise we only break out if the entire
                             # batch failed
                             failed_hosts_count = len(self._tqm._failed_hosts) + len(self._tqm._unreachable_hosts)
-                            if new_play.any_errors_fatal and failed_hosts_count > 0:
-                                break
-                            elif new_play.max_fail_percentage is not None and \
+                            if new_play.max_fail_percentage is not None and \
                                int((new_play.max_fail_percentage)/100.0 * len(batch)) > int((len(batch) - failed_hosts_count) / len(batch) * 100.0):
                                 break
                             elif len(batch) == failed_hosts_count:
@@ -171,6 +164,24 @@ class PlaybookExecutor:
                 if entry:
                     entrylist.append(entry) # per playbook
 
+                # send the stats callback for this playbook
+                if self._tqm is not None:
+                    if C.RETRY_FILES_ENABLED:
+                        retries = list(set(self._tqm._failed_hosts.keys() + self._tqm._unreachable_hosts.keys()))
+                        retries.sort()
+                        if len(retries) > 0:
+                            if C.RETRY_FILES_SAVE_PATH:
+                                basedir = C.shell_expand(C.RETRY_FILES_SAVE_PATH)
+                            else:
+                                basedir = os.path.dirname(playbook_path)
+
+                            (retry_name, _) = os.path.splitext(os.path.basename(playbook_path))
+                            filename = os.path.join(basedir, "%s.retry" % retry_name)
+                            if self._generate_retry_inventory(filename, retries):
+                                display.display("\tto retry, use: --limit @%s\n" % filename)
+
+                    self._tqm.send_callback('v2_playbook_on_stats', self._tqm._stats)
+
                 # if the last result wasn't zero, break out of the playbook file name loop
                 if result != 0:
                     break
@@ -180,46 +191,14 @@ class PlaybookExecutor:
 
         finally:
             if self._tqm is not None:
-                self._cleanup()
+                self._tqm.cleanup()
 
         if self._options.syntax:
             display.display("No issues encountered")
             return result
 
-        # TODO: this stat summary stuff should be cleaned up and moved
-        #        to a new method, if it even belongs here...
-        display.banner("PLAY RECAP")
-
-        hosts = sorted(self._tqm._stats.processed.keys())
-        for h in hosts:
-            t = self._tqm._stats.summarize(h)
-
-            display.display(u"%s : %s %s %s %s" % (
-                hostcolor(h, t),
-                colorize(u'ok', t['ok'], 'green'),
-                colorize(u'changed', t['changed'], 'yellow'),
-                colorize(u'unreachable', t['unreachable'], 'red'),
-                colorize(u'failed', t['failures'], 'red')),
-                screen_only=True
-            )
-
-            display.display(u"%s : %s %s %s %s" % (
-                hostcolor(h, t, False),
-                colorize(u'ok', t['ok'], None),
-                colorize(u'changed', t['changed'], None),
-                colorize(u'unreachable', t['unreachable'], None),
-                colorize(u'failed', t['failures'], None)),
-                log_only=True
-            )
-
-        display.display("", screen_only=True)
-        # END STATS STUFF
-
         return result
 
-    def _cleanup(self, signum=None, framenum=None):
-        return self._tqm.cleanup()
-
     def _get_serialized_batches(self, play):
         '''
         Returns a list of hosts, subdivided into batches based on
@@ -258,48 +237,19 @@ class PlaybookExecutor:
 
             return serialized_batches
 
-    def _do_var_prompt(self, varname, private=True, prompt=None, encrypt=None, confirm=False, salt_size=None, salt=None, default=None):
+    def _generate_retry_inventory(self, retry_path, replay_hosts):
+        '''
+        Called when a playbook run fails. It generates an inventory which allows
+        re-running on ONLY the failed hosts.  This may duplicate some variable
+        information in group_vars/host_vars but that is ok, and expected.
+        '''
 
-        if sys.__stdin__.isatty():
-            if prompt and default is not None:
-                msg = "%s [%s]: " % (prompt, default)
-            elif prompt:
-                msg = "%s: " % prompt
-            else:
-                msg = 'input for %s: ' % varname
+        try:
+            with open(retry_path, 'w') as fd:
+                for x in replay_hosts:
+                    fd.write("%s\n" % x)
+        except Exception as e:
+            display.error("Could not create retry file '%s'. The error was: %s" % (retry_path, e))
+            return False
 
-            def do_prompt(prompt, private):
-                if sys.stdout.encoding:
-                    msg = prompt.encode(sys.stdout.encoding)
-                else:
-                    # when piping the output, or at other times when stdout
-                    # may not be the standard file descriptor, the stdout
-                    # encoding may not be set, so default to something sane
-                    msg = prompt.encode(locale.getpreferredencoding())
-                if private:
-                    return getpass.getpass(msg)
-                return raw_input(msg)
-
-            if confirm:
-                while True:
-                    result = do_prompt(msg, private)
-                    second = do_prompt("confirm " + msg, private)
-                    if result == second:
-                        break
-                    display.display("***** VALUES ENTERED DO NOT MATCH ****")
-            else:
-                result = do_prompt(msg, private)
-        else:
-            result = None
-            display.warning("Not prompting as we are not in interactive mode")
-
-        # if result is false and default is not None
-        if not result and default is not None:
-            result = default
-
-        if encrypt:
-            result = do_encrypt(result, encrypt, salt_size, salt)
-
-        # handle utf-8 chars
-        result = to_unicode(result, errors='strict')
-        return result
+        return True

lib/ansible/executor/process/result.py

@@ -110,7 +110,7 @@ class ResultProcess(multiprocessing.Process):
 
                 # if this task is registering a result, do it now
                 if result._task.register:
-                    self._send_result(('register_host_var', result._host, result._task.register, clean_copy))
+                    self._send_result(('register_host_var', result._host, result._task, clean_copy))
 
                 # send callbacks, execute other options based on the result status
                 # TODO: this should all be cleaned up and probably moved to a sub-function.
@@ -163,7 +163,7 @@ class ResultProcess(multiprocessing.Process):
 
             except queue.Empty:
                 pass
-            except (KeyboardInterrupt, IOError, EOFError):
+            except (KeyboardInterrupt, SystemExit, IOError, EOFError):
                 break
             except:
                 # TODO: we should probably send a proper callback here instead of

lib/ansible/executor/process/worker.py

@@ -48,6 +48,7 @@ from ansible.playbook.task import Task
 from ansible.vars.unsafe_proxy import AnsibleJSONUnsafeDecoder
 
 from ansible.utils.debug import debug
+from ansible.utils.unicode import to_unicode
 
 __all__ = ['WorkerProcess']
 
@@ -59,14 +60,18 @@ class WorkerProcess(multiprocessing.Process):
     for reading later.
     '''
 
-    def __init__(self, tqm, main_q, rslt_q, hostvars_manager, loader):
+    def __init__(self, rslt_q, task_vars, host, task, play_context, loader, variable_manager, shared_loader_obj):
 
         super(WorkerProcess, self).__init__()
         # takes a task queue manager as the sole param:
-        self._main_q   = main_q
-        self._rslt_q   = rslt_q
-        self._hostvars = hostvars_manager
-        self._loader   = loader
+        self._rslt_q            = rslt_q
+        self._task_vars         = task_vars
+        self._host              = host
+        self._task              = task
+        self._play_context      = play_context
+        self._loader            = loader
+        self._variable_manager  = variable_manager
+        self._shared_loader_obj = shared_loader_obj
 
         # dupe stdin, if we have one
         self._new_stdin = sys.stdin
@@ -97,73 +102,45 @@ class WorkerProcess(multiprocessing.Process):
         if HAS_ATFORK:
             atfork()
 
-        while True:
-            task = None
-            try:
-                #debug("waiting for work")
-                (host, task, basedir, zip_vars, compressed_vars, play_context, shared_loader_obj) = self._main_q.get(block=False)
+        try:
+            # execute the task and build a TaskResult from the result
+            debug("running TaskExecutor() for %s/%s" % (self._host, self._task))
+            executor_result = TaskExecutor(
+                self._host,
+                self._task,
+                self._task_vars,
+                self._play_context,
+                self._new_stdin,
+                self._loader,
+                self._shared_loader_obj,
+            ).run()
 
-                if compressed_vars:
-                    job_vars = json.loads(zlib.decompress(zip_vars))
-                else:
-                    job_vars = zip_vars
+            debug("done running TaskExecutor() for %s/%s" % (self._host, self._task))
+            self._host.vars = dict()
+            self._host.groups = []
+            task_result = TaskResult(self._host, self._task, executor_result)
 
-                job_vars['hostvars'] = self._hostvars.hostvars()
+            # put the result on the result queue
+            debug("sending task result")
+            self._rslt_q.put(task_result)
+            debug("done sending task result")
 
-                debug("there's work to be done! got a task/handler to work on: %s" % task)
+        except AnsibleConnectionFailure:
+            self._host.vars = dict()
+            self._host.groups = []
+            task_result = TaskResult(self._host, self._task, dict(unreachable=True))
+            self._rslt_q.put(task_result, block=False)
 
-                # because the task queue manager starts workers (forks) before the
-                # playbook is loaded, set the basedir of the loader inherted by
-                # this fork now so that we can find files correctly
-                self._loader.set_basedir(basedir)
-
-                # Serializing/deserializing tasks does not preserve the loader attribute,
-                # since it is passed to the worker during the forking of the process and
-                # would be wasteful to serialize. So we set it here on the task now, and
-                # the task handles updating parent/child objects as needed.
-                task.set_loader(self._loader)
-
-                # execute the task and build a TaskResult from the result
-                debug("running TaskExecutor() for %s/%s" % (host, task))
-                executor_result = TaskExecutor(
-                    host,
-                    task,
-                    job_vars,
-                    play_context,
-                    self._new_stdin,
-                    self._loader,
-                    shared_loader_obj,
-                ).run()
-                debug("done running TaskExecutor() for %s/%s" % (host, task))
-                task_result = TaskResult(host, task, executor_result)
-
-                # put the result on the result queue
-                debug("sending task result")
-                self._rslt_q.put(task_result)
-                debug("done sending task result")
-
-            except queue.Empty:
-                time.sleep(0.0001)
-            except AnsibleConnectionFailure:
+        except Exception as e:
+            if not isinstance(e, (IOError, EOFError, KeyboardInterrupt, SystemExit)) or isinstance(e, TemplateNotFound):
                 try:
-                    if task:
-                        task_result = TaskResult(host, task, dict(unreachable=True))
-                        self._rslt_q.put(task_result, block=False)
+                    self._host.vars = dict()
+                    self._host.groups = []
+                    task_result = TaskResult(self._host, self._task, dict(failed=True, exception=to_unicode(traceback.format_exc()), stdout=''))
+                    self._rslt_q.put(task_result, block=False)
                 except:
-                    break
-            except Exception as e:
-                if isinstance(e, (IOError, EOFError, KeyboardInterrupt)) and not isinstance(e, TemplateNotFound):
-                    break
-                else:
-                    try:
-                        if task:
-                            task_result = TaskResult(host, task, dict(failed=True, exception=traceback.format_exc(), stdout=''))
-                            self._rslt_q.put(task_result, block=False)
-                    except:
-                        debug("WORKER EXCEPTION: %s" % e)
-                        debug("WORKER EXCEPTION: %s" % traceback.format_exc())
-                        break
+                    debug(u"WORKER EXCEPTION: %s" % to_unicode(e))
+                    debug(u"WORKER TRACEBACK: %s" % to_unicode(traceback.format_exc()))
 
         debug("WORKER PROCESS EXITING")
 
-

lib/ansible/executor/task_executor.py

@@ -24,6 +24,7 @@ import json
 import subprocess
 import sys
 import time
+import traceback
 
 from ansible.compat.six import iteritems, string_types
 
@@ -35,7 +36,7 @@ from ansible.template import Templar
 from ansible.utils.encrypt import key_for_hostname
 from ansible.utils.listify import listify_lookup_plugin_terms
 from ansible.utils.unicode import to_unicode
-from ansible.vars.unsafe_proxy import UnsafeProxy
+from ansible.vars.unsafe_proxy import UnsafeProxy, wrap_var
 
 try:
     from __main__ import display
@@ -67,6 +68,7 @@ class TaskExecutor:
         self._new_stdin         = new_stdin
         self._loader            = loader
         self._shared_loader_obj = shared_loader_obj
+        self._connection        = None
 
     def run(self):
         '''
@@ -139,13 +141,15 @@ class TaskExecutor:
             return res
         except AnsibleError as e:
             return dict(failed=True, msg=to_unicode(e, nonstring='simplerepr'))
+        except Exception as e:
+            return dict(failed=True, msg='Unexpected failure during module execution.', exception=to_unicode(traceback.format_exc()), stdout='')
         finally:
             try:
                 self._connection.close()
             except AttributeError:
                 pass
             except Exception as e:
-                display.debug("error closing connection: %s" % to_unicode(e))
+                display.debug(u"error closing connection: %s" % to_unicode(e))
 
     def _get_loop_items(self):
         '''
@@ -174,21 +178,15 @@ class TaskExecutor:
                     # first_found loops are special.  If the item is undefined
                     # then we want to fall through to the next value rather
                     # than failing.
-                    loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar,
-                            loader=self._loader, fail_on_undefined=False, convert_bare=True)
+                    loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar, loader=self._loader, fail_on_undefined=False, convert_bare=True)
                     loop_terms = [t for t in loop_terms if not templar._contains_vars(t)]
                 else:
                     try:
-                        loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar,
-                                loader=self._loader, fail_on_undefined=True, convert_bare=True)
+                        loop_terms = listify_lookup_plugin_terms(terms=self._task.loop_args, templar=templar, loader=self._loader, fail_on_undefined=True, convert_bare=True)
                     except AnsibleUndefinedVariable as e:
-                        if 'has no attribute' in str(e):
-                            loop_terms = []
-                            display.deprecated("Skipping task due to undefined attribute, in the future this will be a fatal error.")
-                        else:
-                            raise
-                items = self._shared_loader_obj.lookup_loader.get(self._task.loop, loader=self._loader,
-                        templar=templar).run(terms=loop_terms, variables=self._job_vars)
+                        loop_terms = []
+                        display.deprecated("Skipping task due to undefined Error, in the future this will be a fatal error.")
+                items = self._shared_loader_obj.lookup_loader.get(self._task.loop, loader=self._loader, templar=templar).run(terms=loop_terms, variables=self._job_vars)
             else:
                 raise AnsibleError("Unexpected failure in finding the lookup named '%s' in the available lookup plugins" % self._task.loop)
 
@@ -230,7 +228,7 @@ class TaskExecutor:
                 tmp_task = self._task.copy()
                 tmp_play_context = self._play_context.copy()
             except AnsibleParserError as e:
-                results.append(dict(failed=True, msg=str(e)))
+                results.append(dict(failed=True, msg=to_unicode(e)))
                 continue
 
             # now we swap the internal task and play context with their copies,
@@ -244,6 +242,7 @@ class TaskExecutor:
             # now update the result with the item info, and append the result
             # to the list of results
             res['item'] = item
+            #TODO: send item results to callback here, instead of all at the end
             results.append(res)
 
         return results
@@ -267,28 +266,39 @@ class TaskExecutor:
             if all(isinstance(o, string_types) for o in items):
                 final_items = []
                 name = self._task.args.pop('name', None) or self._task.args.pop('pkg', None)
-                # The user is doing an upgrade or some other operation
-                # that doesn't take name or pkg.
-                if name:
+
+                # This gets the information to check whether the name field
+                # contains a template that we can squash for
+                template_no_item = template_with_item = None
+                if templar._contains_vars(name):
+                    variables['item'] = '\0$'
+                    template_no_item = templar.template(name, variables, cache=False)
+                    variables['item'] = '\0@'
+                    template_with_item = templar.template(name, variables, cache=False)
+                    del variables['item']
+
+                # Check if the user is doing some operation that doesn't take
+                # name/pkg or the name/pkg field doesn't have any variables
+                # and thus the items can't be squashed
+                if name and (template_no_item != template_with_item):
                     for item in items:
                         variables['item'] = item
                         if self._task.evaluate_conditional(templar, variables):
-                            if templar._contains_vars(name):
-                                new_item = templar.template(name, cache=False)
-                                final_items.append(new_item)
-                            else:
-                                final_items.append(item)
+                            new_item = templar.template(name, cache=False)
+                            final_items.append(new_item)
                     self._task.args['name'] = final_items
+                    # Wrap this in a list so that the calling function loop
+                    # executes exactly once
                     return [final_items]
+                else:
+                    # Restore the name parameter
+                    self._task.args['name'] = name
             #elif:
                 # Right now we only optimize single entries.  In the future we
                 # could optimize more types:
                 # * lists can be squashed together
                 # * dicts could squash entries that match in all cases except the
                 #   name or pkg field.
-                # Note: we really should be checking that the name or pkg field
-                # contains a template that expands with our with_items values.
-                # If it doesn't then we may break things
         return items
 
     def _execute(self, variables=None):
@@ -314,6 +324,11 @@ class TaskExecutor:
             # do the same kind of post validation step on it here before we use it.
             self._play_context.post_validate(templar=templar)
 
+            # now that the play context is finalized, if the remote_addr is not set
+            # default to using the host's address field as the remote address
+            if not self._play_context.remote_addr:
+                self._play_context.remote_addr = self._host.address
+
             # We also add "magic" variables back into the variables dict to make sure
             # a certain subset of variables exist.
             self._play_context.update_vars(variables)
@@ -360,8 +375,13 @@ class TaskExecutor:
                 self._task.args = variable_params
 
         # get the connection and the handler for this execution
-        self._connection = self._get_connection(variables=variables, templar=templar)
-        self._connection.set_host_overrides(host=self._host)
+        if not self._connection or not getattr(self._connection, 'connected', False) or self._play_context.remote_addr != self._connection._play_context.remote_addr:
+            self._connection = self._get_connection(variables=variables, templar=templar)
+            self._connection.set_host_overrides(host=self._host)
+        else:
+            # if connection is reused, its _play_context is no longer valid and needs
+            # to be replaced with the one templated above, in case other data changed
+            self._connection._play_context = self._play_context
 
         self._handler = self._get_action_handler(connection=self._connection, templar=templar)
 
@@ -384,7 +404,6 @@ class TaskExecutor:
 
         # make a copy of the job vars here, in case we need to update them
         # with the registered variable value later on when testing conditions
-        #vars_copy = variables.copy()
         vars_copy = variables.copy()
 
         display.debug("starting attempt loop")
@@ -398,9 +417,14 @@ class TaskExecutor:
             try:
                 result = self._handler.run(task_vars=variables)
             except AnsibleConnectionFailure as e:
-                return dict(unreachable=True, msg=str(e))
+                return dict(unreachable=True, msg=to_unicode(e))
             display.debug("handler run complete")
 
+            # update the local copy of vars with the registered value, if specified,
+            # or any facts which may have been generated by the module execution
+            if self._task.register:
+                vars_copy[self._task.register] = wrap_var(result.copy())
+
             if self._task.async > 0:
                 # the async_wrapper module returns dumped JSON via its stdout
                 # response, so we parse it here and replace the result
@@ -409,7 +433,7 @@ class TaskExecutor:
                         return result
                     result = json.loads(result.get('stdout'))
                 except (TypeError, ValueError) as e:
-                    return dict(failed=True, msg="The async task did not return valid JSON: %s" % str(e))
+                    return dict(failed=True, msg=u"The async task did not return valid JSON: %s" % to_unicode(e))
 
                 if self._task.poll > 0:
                     result = self._poll_async_result(result=result, templar=templar)
@@ -430,11 +454,6 @@ class TaskExecutor:
                     return failed_when_result
                 return False
 
-            # update the local copy of vars with the registered value, if specified,
-            # or any facts which may have been generated by the module execution
-            if self._task.register:
-                vars_copy[self._task.register] = result
-
             if 'ansible_facts' in result:
                 vars_copy.update(result['ansible_facts'])
 
@@ -451,7 +470,7 @@ class TaskExecutor:
 
             if attempt < retries - 1:
                 cond = Conditional(loader=self._loader)
-                cond.when = self._task.until
+                cond.when = [ self._task.until ]
                 if cond.evaluate_conditional(templar, vars_copy):
                     break
 
@@ -464,7 +483,7 @@ class TaskExecutor:
         # do the final update of the local variables here, for both registered
         # values and any facts which may have been created
         if self._task.register:
-            variables[self._task.register] = result
+            variables[self._task.register] = wrap_var(result)
 
         if 'ansible_facts' in result:
             variables.update(result['ansible_facts'])
@@ -542,9 +561,6 @@ class TaskExecutor:
         correct connection object from the list of connection plugins
         '''
 
-        if not self._play_context.remote_addr:
-            self._play_context.remote_addr = self._host.address
-
         if self._task.delegate_to is not None:
             # since we're delegating, we don't want to use interpreter values
             # which would have been set for the original target host

lib/ansible/executor/task_queue_manager.py

@@ -19,7 +19,6 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-from multiprocessing.managers import SyncManager, DictProxy
 import multiprocessing
 import os
 import tempfile
@@ -27,13 +26,14 @@ import tempfile
 from ansible import constants as C
 from ansible.errors import AnsibleError
 from ansible.executor.play_iterator import PlayIterator
-from ansible.executor.process.worker import WorkerProcess
 from ansible.executor.process.result import ResultProcess
 from ansible.executor.stats import AggregateStats
 from ansible.playbook.play_context import PlayContext
 from ansible.plugins import callback_loader, strategy_loader, module_loader
 from ansible.template import Templar
 from ansible.vars.hostvars import HostVars
+from ansible.plugins.callback import CallbackBase
+from ansible.utils.unicode import to_unicode
 
 try:
     from __main__ import display
@@ -56,7 +56,7 @@ class TaskQueueManager:
     which dispatches the Play's tasks to hosts.
     '''
 
-    def __init__(self, inventory, variable_manager, loader, options, passwords, stdout_callback=None):
+    def __init__(self, inventory, variable_manager, loader, options, passwords, stdout_callback=None, run_additional_callbacks=True, run_tree=False):
 
         self._inventory        = inventory
         self._variable_manager = variable_manager
@@ -65,6 +65,8 @@ class TaskQueueManager:
         self._stats            = AggregateStats()
         self.passwords         = passwords
         self._stdout_callback  = stdout_callback
+        self._run_additional_callbacks = run_additional_callbacks
+        self._run_tree         = run_tree
 
         self._callbacks_loaded = False
         self._callback_plugins = []
@@ -96,14 +98,10 @@ class TaskQueueManager:
     def _initialize_processes(self, num):
         self._workers = []
 
-        for i in xrange(num):
+        for i in range(num):
             main_q = multiprocessing.Queue()
             rslt_q = multiprocessing.Queue()
-
-            prc = WorkerProcess(self, main_q, rslt_q, self._hostvars_manager, self._loader)
-            prc.start()
-
-            self._workers.append((prc, main_q, rslt_q))
+            self._workers.append([None, main_q, rslt_q])
 
         self._result_prc = ResultProcess(self._final_q, self._workers)
         self._result_prc.start()
@@ -144,8 +142,16 @@ class TaskQueueManager:
         if self._stdout_callback is None:
             self._stdout_callback = C.DEFAULT_STDOUT_CALLBACK
 
-        if self._stdout_callback not in callback_loader:
-            raise AnsibleError("Invalid callback for stdout specified: %s" % self._stdout_callback)
+        if isinstance(self._stdout_callback, CallbackBase):
+            stdout_callback_loaded = True
+        elif isinstance(self._stdout_callback, basestring):
+            if self._stdout_callback not in callback_loader:
+                raise AnsibleError("Invalid callback for stdout specified: %s" % self._stdout_callback)
+            else:
+                self._stdout_callback = callback_loader.get(self._stdout_callback)
+                stdout_callback_loaded = True
+        else:
+            raise AnsibleError("callback must be an instance of CallbackBase or the name of a callback plugin")
 
         for callback_plugin in callback_loader.all(class_only=True):
             if hasattr(callback_plugin, 'CALLBACK_VERSION') and callback_plugin.CALLBACK_VERSION >= 2.0:
@@ -159,7 +165,9 @@ class TaskQueueManager:
                     if callback_name != self._stdout_callback or stdout_callback_loaded:
                         continue
                     stdout_callback_loaded = True
-                elif callback_needs_whitelist and (C.DEFAULT_CALLBACK_WHITELIST is None or callback_name not in C.DEFAULT_CALLBACK_WHITELIST):
+                elif callback_name == 'tree' and self._run_tree:
+                    pass
+                elif not self._run_additional_callbacks or (callback_needs_whitelist and (C.DEFAULT_CALLBACK_WHITELIST is None or callback_name not in C.DEFAULT_CALLBACK_WHITELIST)):
                     continue
 
             self._callback_plugins.append(callback_plugin())
@@ -184,31 +192,12 @@ class TaskQueueManager:
         new_play = play.copy()
         new_play.post_validate(templar)
 
-        class HostVarsManager(SyncManager):
-            pass
-
-        hostvars = HostVars(
-            play=new_play,
+        self.hostvars = HostVars(
             inventory=self._inventory,
             variable_manager=self._variable_manager,
             loader=self._loader,
         )
 
-        HostVarsManager.register(
-            'hostvars',
-            callable=lambda: hostvars,
-            # FIXME: this is the list of exposed methods to the DictProxy object, plus our
-            #        special ones (set_variable_manager/set_inventory). There's probably a better way
-            #        to do this with a proper BaseProxy/DictProxy derivative
-            exposed=(
-                'set_variable_manager', 'set_inventory', '__contains__', '__delitem__',
-                '__getitem__', '__len__', '__setitem__', 'clear', 'copy', 'get', 'has_key',
-                'items', 'keys', 'pop', 'popitem', 'setdefault', 'update', 'values'
-            ),
-        )
-        self._hostvars_manager = HostVarsManager()
-        self._hostvars_manager.start()
-
         # Fork # of forks, # of hosts or serial, whichever is lowest
         contenders =  [self._options.forks, play.serial, len(self._inventory.get_hosts(new_play.hosts))]
         contenders =  [ v for v in contenders if v is not None and v > 0 ]
@@ -248,7 +237,6 @@ class TaskQueueManager:
         # and run the play using the strategy and cleanup on way out
         play_return = strategy.run(iterator, play_context)
         self._cleanup_processes()
-        self._hostvars_manager.shutdown()
         return play_return
 
     def cleanup(self):
@@ -264,7 +252,11 @@ class TaskQueueManager:
             for (worker_prc, main_q, rslt_q) in self._workers:
                 rslt_q.close()
                 main_q.close()
-                worker_prc.terminate()
+                if worker_prc and worker_prc.is_alive():
+                    try:
+                        worker_prc.terminate()
+                    except AttributeError:
+                        pass
 
     def clear_failed_hosts(self):
         self._failed_hosts = dict()
@@ -288,22 +280,36 @@ class TaskQueueManager:
         self._terminated = True
 
     def send_callback(self, method_name, *args, **kwargs):
-        for callback_plugin in self._callback_plugins:
+        for callback_plugin in [self._stdout_callback] + self._callback_plugins:
             # a plugin that set self.disabled to True will not be called
             # see osx_say.py example for such a plugin
             if getattr(callback_plugin, 'disabled', False):
                 continue
-            methods = [
-                getattr(callback_plugin, method_name, None),
-                getattr(callback_plugin, 'v2_on_any', None)
-            ]
+
+            # try to find v2 method, fallback to v1 method, ignore callback if no method found
+            methods = []
+            for possible in [method_name, 'v2_on_any']:
+                gotit =  getattr(callback_plugin, possible, None)
+                if gotit is None:
+                    gotit = getattr(callback_plugin, possible.replace('v2_',''), None)
+                if gotit is not None:
+                    methods.append(gotit)
+
             for method in methods:
-                if method is not None:
-                    try:
+                try:
+                    # temporary hack, required due to a change in the callback API, so
+                    # we don't break backwards compatibility with callbacks which were
+                    # designed to use the original API
+                    # FIXME: target for removal and revert to the original code here after a year (2017-01-14)
+                    if method_name == 'v2_playbook_on_start':
+                        import inspect
+                        (f_args, f_varargs, f_keywords, f_defaults) = inspect.getargspec(method)
+                        if 'playbook' in f_args:
+                            method(*args, **kwargs)
+                        else:
+                            method()
+                    else:
                         method(*args, **kwargs)
-                    except Exception as e:
-                        try:
-                            v1_method = method.replace('v2_','')
-                            v1_method(*args, **kwargs)
-                        except Exception:
-                            display.warning('Error when using %s: %s' % (method, str(e)))
+                except Exception as e:
+                    #TODO: add config toggle to make this fatal or not?
+                    display.warning(u"Failure when attempting to use callback plugin (%s): %s" % (to_unicode(callback_plugin), to_unicode(e)))

lib/ansible/galaxy/__init__.py

@@ -49,9 +49,34 @@ class Galaxy(object):
         this_dir, this_filename = os.path.split(__file__)
         self.DATA_PATH = os.path.join(this_dir, "data")
 
-        #TODO: move to getter for lazy loading
-        self.default_readme = self._str_from_data_file('readme')
-        self.default_meta = self._str_from_data_file('metadata_template.j2')
+        self._default_readme = None
+        self._default_meta = None
+        self._default_test = None
+        self._default_travis = None
+
+    @property
+    def default_readme(self):
+        if self._default_readme is None:
+            self._default_readme = self._str_from_data_file('readme')
+        return self._default_readme
+
+    @property
+    def default_meta(self):
+        if self._default_meta is None:
+            self._default_meta = self._str_from_data_file('metadata_template.j2')
+        return self._default_meta
+
+    @property
+    def default_test(self):
+        if self._default_test is None:
+            self._default_test = self._str_from_data_file('test_playbook.j2')
+        return self._default_test
+
+    @property
+    def default_travis(self):
+        if self._default_travis is None:
+            self._default_travis = self._str_from_data_file('travis.j2')
+        return self._default_travis
 
     def add_role(self, role):
         self.roles[role.name] = role

lib/ansible/galaxy/api.py

@@ -25,11 +25,15 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import json
+import urllib
+
 from urllib2 import quote as urlquote, HTTPError
 from urlparse import urlparse
 
+import ansible.constants as C
 from ansible.errors import AnsibleError
 from ansible.module_utils.urls import open_url
+from ansible.galaxy.token import GalaxyToken
 
 try:
     from __main__ import display
@@ -43,45 +47,111 @@ class GalaxyAPI(object):
 
     SUPPORTED_VERSIONS = ['v1']
 
-    def __init__(self, galaxy, api_server):
-
+    def __init__(self, galaxy):
         self.galaxy = galaxy
+        self.token = GalaxyToken()
+        self._api_server = C.GALAXY_SERVER
+        self._validate_certs = not C.GALAXY_IGNORE_CERTS
 
-        try:
-            urlparse(api_server, scheme='https')
-        except:
-            raise AnsibleError("Invalid server API url passed: %s" % api_server)
+        # set validate_certs
+        if galaxy.options.ignore_certs:
+            self._validate_certs = False
+        display.vvv('Validate TLS certificates: %s' % self._validate_certs)
 
-        server_version = self.get_server_api_version('%s/api/' % (api_server))
-        if not server_version:
-            raise AnsibleError("Could not retrieve server API version: %s" % api_server)
+        # set the API server
+        if galaxy.options.api_server != C.GALAXY_SERVER:
+            self._api_server = galaxy.options.api_server
+        display.vvv("Connecting to galaxy_server: %s" % self._api_server)
 
-        if server_version in self.SUPPORTED_VERSIONS:
-            self.baseurl = '%s/api/%s' % (api_server, server_version)
-            self.version = server_version # for future use
-            display.vvvvv("Base API: %s" % self.baseurl)
-        else:
+        server_version = self.get_server_api_version()
+        if not server_version in self.SUPPORTED_VERSIONS:
             raise AnsibleError("Unsupported Galaxy server API version: %s" % server_version)
 
-    def get_server_api_version(self, api_server):
+        self.baseurl = '%s/api/%s' % (self._api_server, server_version)
+        self.version = server_version # for future use
+        display.vvv("Base API: %s" % self.baseurl)           
+
+    def __auth_header(self):
+        token = self.token.get()
+        if token is None:
+            raise AnsibleError("No access token. You must first use login to authenticate and obtain an access token.")
+        return {'Authorization': 'Token ' + token}
+
+    def __call_galaxy(self, url, args=None, headers=None, method=None):
+        if args and not headers:
+            headers = self.__auth_header()
+        try:
+            display.vvv(url)
+            resp = open_url(url, data=args, validate_certs=self._validate_certs, headers=headers, method=method)
+            data = json.load(resp)
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['detail'])
+        return data
+
+    @property
+    def api_server(self):
+        return self._api_server    
+    
+    @property
+    def validate_certs(self):
+        return self._validate_certs
+
+    def get_server_api_version(self):
         """
         Fetches the Galaxy API current version to ensure
         the API server is up and reachable.
         """
-        #TODO: fix galaxy server which returns current_version path (/api/v1) vs actual version (v1)
-        # also should set baseurl using supported_versions which has path
-        return 'v1'
-
         try:
-            data = json.load(open_url(api_server, validate_certs=self.galaxy.options.validate_certs))
-            return data.get("current_version", 'v1')
-        except Exception:
-            # TODO: report error
-            return None
+            url = '%s/api/' % self._api_server
+            data = json.load(open_url(url, validate_certs=self._validate_certs))
+            return data['current_version']
+        except Exception as e:
+            raise AnsibleError("The API server (%s) is not responding, please try again later." % url)
+        
+    def authenticate(self, github_token):
+        """
+        Retrieve an authentication token
+        """
+        url = '%s/tokens/' % self.baseurl
+        args = urllib.urlencode({"github_token": github_token})
+        resp = open_url(url, data=args, validate_certs=self._validate_certs, method="POST")
+        data = json.load(resp)
+        return data
 
+    def create_import_task(self, github_user, github_repo, reference=None):
+        """
+        Post an import request
+        """
+        url = '%s/imports/' % self.baseurl
+        args = urllib.urlencode({
+            "github_user": github_user,
+            "github_repo": github_repo,
+            "github_reference": reference if reference else ""
+        })
+        data = self.__call_galaxy(url, args=args)
+        if data.get('results', None):
+            return data['results']
+        return data
+
+    def get_import_task(self, task_id=None, github_user=None, github_repo=None):
+        """
+        Check the status of an import task.
+        """
+        url = '%s/imports/' % self.baseurl
+        if not task_id is None:
+            url = "%s?id=%d" % (url,task_id)
+        elif not github_user is None and not github_repo is None:
+            url = "%s?github_user=%s&github_repo=%s" % (url,github_user,github_repo)
+        else:
+            raise AnsibleError("Expected task_id or github_user and github_repo")
+        
+        data = self.__call_galaxy(url)
+        return data['results']
+       
     def lookup_role_by_name(self, role_name, notify=True):
         """
-        Find a role by name
+        Find a role by name.
         """
         role_name = urlquote(role_name)
 
@@ -92,18 +162,12 @@ class GalaxyAPI(object):
             if notify:
                 display.display("- downloading role '%s', owned by %s" % (role_name, user_name))
         except:
-            raise AnsibleError("- invalid role name (%s). Specify role as format: username.rolename" % role_name)
+            raise AnsibleError("Invalid role name (%s). Specify role as format: username.rolename" % role_name)
 
         url = '%s/roles/?owner__username=%s&name=%s' % (self.baseurl, user_name, role_name)
-        display.vvvv("- %s" % (url))
-        try:
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
-            if len(data["results"]) != 0:
-                return data["results"][0]
-        except:
-            # TODO: report on connection/availability errors
-            pass
-
+        data = self.__call_galaxy(url)
+        if len(data["results"]) != 0:
+            return data["results"][0]
         return None
 
     def fetch_role_related(self, related, role_id):
@@ -114,13 +178,12 @@ class GalaxyAPI(object):
 
         try:
             url = '%s/roles/%d/%s/?page_size=50' % (self.baseurl, int(role_id), related)
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+            data = self.__call_galaxy(url)
             results = data['results']
             done = (data.get('next', None) is None)
             while not done:
                 url = '%s%s' % (self.baseurl, data['next'])
-                display.display(url)
-                data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+                data = self.__call_galaxy(url)
                 results += data['results']
                 done = (data.get('next', None) is None)
             return results
@@ -131,10 +194,9 @@ class GalaxyAPI(object):
         """
         Fetch the list of items specified.
         """
-
         try:
             url = '%s/%s/?page_size' % (self.baseurl, what)
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+            data = self.__call_galaxy(url)
             if "results" in data:
                 results = data['results']
             else:
@@ -144,41 +206,64 @@ class GalaxyAPI(object):
                 done = (data.get('next', None) is None)
             while not done:
                 url = '%s%s' % (self.baseurl, data['next'])
-                display.display(url)
-                data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+                data = self.__call_galaxy(url)
                 results += data['results']
                 done = (data.get('next', None) is None)
             return results
         except Exception as error:
             raise AnsibleError("Failed to download the %s list: %s" % (what, str(error)))
 
-    def search_roles(self, search, platforms=None, tags=None):
+    def search_roles(self, search, **kwargs):
 
-        search_url = self.baseurl + '/roles/?page=1'
+        search_url = self.baseurl + '/search/roles/?'
 
         if search:
-            search_url += '&search=' + urlquote(search)
+            search_url += '&autocomplete=' + urlquote(search)
 
-        if tags is None:
-            tags = []
-        elif isinstance(tags, basestring):
+        tags = kwargs.get('tags',None)
+        platforms = kwargs.get('platforms', None)
+        page_size = kwargs.get('page_size', None)
+        author = kwargs.get('author', None)
+
+        if tags and isinstance(tags, basestring):
             tags = tags.split(',')
-
-        for tag in tags:
-            search_url += '&chain__tags__name=' + urlquote(tag)
-
-        if platforms is None:
-            platforms = []
-        elif isinstance(platforms, basestring):
+            search_url += '&tags_autocomplete=' + '+'.join(tags)
+        
+        if platforms and isinstance(platforms, basestring):
             platforms = platforms.split(',')
+            search_url += '&platforms_autocomplete=' + '+'.join(platforms)
 
-        for plat in platforms:
-            search_url += '&chain__platforms__name=' + urlquote(plat)
-
-        display.debug("Executing query: %s" % search_url)
-        try:
-            data = json.load(open_url(search_url, validate_certs=self.galaxy.options.validate_certs))
-        except HTTPError as e:
-            raise AnsibleError("Unsuccessful request to server: %s" % str(e))
+        if page_size:
+            search_url += '&page_size=%s' % page_size
 
+        if author:
+            search_url += '&username_autocomplete=%s' % author
+   
+        data = self.__call_galaxy(search_url)
+        return data
+
+    def add_secret(self, source, github_user, github_repo, secret):
+        url = "%s/notification_secrets/" % self.baseurl
+        args = urllib.urlencode({
+            "source": source,
+            "github_user": github_user,
+            "github_repo": github_repo,
+            "secret": secret
+        })
+        data = self.__call_galaxy(url, args=args)
+        return data
+
+    def list_secrets(self):
+        url = "%s/notification_secrets" % self.baseurl
+        data = self.__call_galaxy(url, headers=self.__auth_header())
+        return data
+
+    def remove_secret(self, secret_id):
+        url = "%s/notification_secrets/%s/" % (self.baseurl, secret_id)
+        data = self.__call_galaxy(url, headers=self.__auth_header(), method='DELETE')
+        return data
+
+    def delete_role(self, github_user, github_repo):
+        url = "%s/removerole/?github_user=%s&github_repo=%s" % (self.baseurl,github_user,github_repo)
+        data = self.__call_galaxy(url, headers=self.__auth_header(), method='DELETE')
         return data

lib/ansible/galaxy/data/metadata_template.j2

@@ -2,9 +2,11 @@ galaxy_info:
   author: {{ author }}
   description: {{description}}
   company: {{ company }}
+  
   # If the issue tracker for your role is not on github, uncomment the
   # next line and provide a value
   # issue_tracker_url: {{ issue_tracker_url }}
+  
   # Some suggested licenses:
   # - BSD (default)
   # - MIT
@@ -13,7 +15,17 @@ galaxy_info:
   # - Apache
   # - CC-BY
   license: {{ license }}
+  
   min_ansible_version: {{ min_ansible_version }}
+
+  # Optionally specify the branch Galaxy will use when accessing the GitHub
+  # repo for this role. During role install, if no tags are available,
+  # Galaxy will use this branch. During import Galaxy will access files on
+  # this branch. If travis integration is cofigured, only notification for this
+  # branch will be accepted. Otherwise, in all cases, the repo's default branch
+  # (usually master) will be used.
+  #github_branch:
+  
   #
   # Below are all platforms currently available. Just uncomment
   # the ones that apply to your role. If you don't see your
@@ -28,6 +40,7 @@ galaxy_info:
   #  - {{ version }}
     {%- endfor %}
   {%- endfor %}
+  
   galaxy_tags: []
     # List tags for your role here, one per line. A tag is
     # a keyword that describes and categorizes the role.
@@ -36,6 +49,7 @@ galaxy_info:
     #
     # NOTE: A tag is limited to a single word comprised of
     # alphanumeric characters. Maximum 20 tags per role.
+
 dependencies: []
   # List your role dependencies here, one per line.
   # Be sure to remove the '[]' above if you add dependencies

lib/ansible/galaxy/data/test_playbook.j2

@@ -0,0 +1,5 @@
+---
+- hosts: localhost
+  remote_user: root
+  roles:
+    - {{ role_name }}

lib/ansible/galaxy/data/travis.j2

@@ -0,0 +1,29 @@
+---
+language: python
+python: "2.7"
+
+# Use the new container infrastructure
+sudo: false
+
+# Install ansible
+addons:
+  apt:
+    packages:
+    - python-pip
+
+install:
+  # Install ansible
+  - pip install ansible
+
+  # Check ansible version
+  - ansible --version
+
+  # Create ansible.cfg with correct roles_path
+  - printf '[defaults]\nroles_path=../' >ansible.cfg
+
+script:
+  # Basic role syntax check
+  - ansible-playbook tests/test.yml -i tests/inventory --syntax-check
+
+notifications:
+  webhooks: https://galaxy.ansible.com/api/v1/notifications/

lib/ansible/galaxy/login.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+
+########################################################################
+#
+# (C) 2015, Chris Houseknecht <chouse@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+########################################################################
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import getpass
+import json
+import urllib
+
+from urllib2 import quote as urlquote, HTTPError
+from urlparse import urlparse
+
+from ansible.errors import AnsibleError, AnsibleOptionsError
+from ansible.module_utils.urls import open_url
+from ansible.utils.color import stringc
+
+try:
+    from __main__ import display
+except ImportError:
+    from ansible.utils.display import Display
+    display = Display()
+
+class GalaxyLogin(object):
+    ''' Class to handle authenticating user with Galaxy API prior to performing CUD operations '''
+
+    GITHUB_AUTH = 'https://api.github.com/authorizations'
+    
+    def __init__(self, galaxy, github_token=None):
+        self.galaxy = galaxy
+        self.github_username = None
+        self.github_password = None
+        
+        if github_token == None:
+            self.get_credentials()
+
+    def get_credentials(self):
+        display.display(u'\n\n' + "We need your " + stringc("Github login",'bright cyan') + 
+            " to identify you.", screen_only=True)
+        display.display("This information will " + stringc("not be sent to Galaxy",'bright cyan') + 
+            ", only to " + stringc("api.github.com.","yellow"), screen_only=True)
+        display.display("The password will not be displayed." + u'\n\n', screen_only=True)
+        display.display("Use " + stringc("--github-token",'yellow') + 
+            " if you do not want to enter your password." + u'\n\n', screen_only=True)
+        
+        try:
+            self.github_username = raw_input("Github Username: ")
+        except:
+            pass
+
+        try:
+            self.github_password = getpass.getpass("Password for %s: " % self.github_username)
+        except:
+            pass
+
+        if not self.github_username or not self.github_password:
+            raise AnsibleError("Invalid Github credentials. Username and password are required.")
+
+    def remove_github_token(self):
+        '''
+        If for some reason an ansible-galaxy token was left from a prior login, remove it. We cannot
+        retrieve the token after creation, so we are forced to create a new one.
+        '''
+        try:
+            tokens = json.load(open_url(self.GITHUB_AUTH, url_username=self.github_username,
+                url_password=self.github_password, force_basic_auth=True,))
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['message'])
+
+        for token in tokens:
+            if token['note'] == 'ansible-galaxy login':
+                display.vvvvv('removing token: %s' % token['token_last_eight'])
+                try: 
+                    open_url('https://api.github.com/authorizations/%d' % token['id'], url_username=self.github_username,
+                        url_password=self.github_password, method='DELETE', force_basic_auth=True,)
+                except HTTPError as e:
+                    res = json.load(e)
+                    raise AnsibleError(res['message'])
+
+    def create_github_token(self):
+        '''
+        Create a personal authorization token with a note of 'ansible-galaxy login'
+        '''
+        self.remove_github_token()
+        args = json.dumps({"scopes":["public_repo"], "note":"ansible-galaxy login"})
+        try:
+            data = json.load(open_url(self.GITHUB_AUTH, url_username=self.github_username,
+                url_password=self.github_password, force_basic_auth=True, data=args))
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['message'])
+        return data['token']

lib/ansible/galaxy/role.py

@@ -46,7 +46,7 @@ class GalaxyRole(object):
     SUPPORTED_SCMS = set(['git', 'hg'])
     META_MAIN = os.path.join('meta', 'main.yml')
     META_INSTALL = os.path.join('meta', '.galaxy_install_info')
-    ROLE_DIRS = ('defaults','files','handlers','meta','tasks','templates','vars')
+    ROLE_DIRS = ('defaults','files','handlers','meta','tasks','templates','vars','tests')
 
 
     def __init__(self, galaxy, name, src=None, version=None, scm=None, path=None):
@@ -130,13 +130,11 @@ class GalaxyRole(object):
             install_date=datetime.datetime.utcnow().strftime("%c"),
         )
         info_path = os.path.join(self.path, self.META_INSTALL)
-        try:
-            f = open(info_path, 'w+')
-            self._install_info = yaml.safe_dump(info, f)
-        except:
-            return False
-        finally:
-            f.close()
+        with open(info_path, 'w+') as f:
+            try:
+                self._install_info = yaml.safe_dump(info, f)
+            except:
+                return False
 
         return True
 
@@ -198,10 +196,10 @@ class GalaxyRole(object):
                 role_data = self.src
                 tmp_file = self.fetch(role_data)
             else:
-                api = GalaxyAPI(self.galaxy, self.options.api_server)
+                api = GalaxyAPI(self.galaxy)
                 role_data = api.lookup_role_by_name(self.src)
                 if not role_data:
-                    raise AnsibleError("- sorry, %s was not found on %s." % (self.src, self.options.api_server))
+                    raise AnsibleError("- sorry, %s was not found on %s." % (self.src, api.api_server))
 
                 role_versions = api.fetch_role_related('versions', role_data['id'])
                 if not self.version:
@@ -213,8 +211,10 @@ class GalaxyRole(object):
                         loose_versions = [LooseVersion(a.get('name',None)) for a in role_versions]
                         loose_versions.sort()
                         self.version = str(loose_versions[-1])
+                    elif role_data.get('github_branch', None):
+                        self.version = role_data['github_branch']
                     else:
-                        self.version = 'master'
+                        self.version = 'master' 
                 elif self.version != 'master':
                     if role_versions and self.version not in [a.get('name', None) for a in role_versions]:
                         raise AnsibleError("- the specified version (%s) of %s was not found in the list of available versions (%s)." % (self.version, self.name, role_versions))

lib/ansible/galaxy/token.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+
+########################################################################
+#
+# (C) 2015, Chris Houseknecht <chouse@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+########################################################################
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import os
+import yaml
+from stat import *
+
+try:
+    from __main__ import display
+except ImportError:
+    from ansible.utils.display import Display
+    display = Display()
+
+
+class GalaxyToken(object):
+    ''' Class to storing and retrieving token in ~/.ansible_galaxy '''
+
+    def __init__(self):
+        self.file = os.path.expanduser("~") + '/.ansible_galaxy'
+        self.config = yaml.safe_load(self.__open_config_for_read())
+        if not self.config:
+            self.config = {}
+        
+    def __open_config_for_read(self):
+        if os.path.isfile(self.file):
+            display.vvv('Opened %s' % self.file)
+            return open(self.file, 'r')
+        # config.yml not found, create and chomd u+rw
+        f = open(self.file,'w')
+        f.close()
+        os.chmod(self.file,S_IRUSR|S_IWUSR) # owner has +rw
+        display.vvv('Created %s' % self.file) 
+        return open(self.file, 'r')
+
+    def set(self, token): 
+        self.config['token'] = token
+        self.save()
+    
+    def get(self):
+        return self.config.get('token', None)
+    
+    def save(self):
+        with open(self.file,'w') as f:
+            yaml.safe_dump(self.config,f,default_flow_style=False)
+        

lib/ansible/inventory/__init__.py

@@ -78,6 +78,10 @@ class Inventory(object):
         self._restriction = None
         self._subset = None
 
+        # clear the cache here, which is only useful if more than
+        # one Inventory objects are created when using the API directly
+        self.clear_pattern_cache()
+
         self.parse_inventory(host_list)
 
     def serialize(self):
@@ -109,7 +113,12 @@ class Inventory(object):
             pass
         elif isinstance(host_list, list):
             for h in host_list:
-                (host, port) = parse_address(h, allow_ranges=False)
+                try:
+                    (host, port) = parse_address(h, allow_ranges=False)
+                except AnsibleError as e:
+                    display.vvv("Unable to parse address from hostname, leaving unchanged: %s" % to_unicode(e))
+                    host = h
+                    port = None
                 all.add_host(Host(host, port))
         elif self._loader.path_exists(host_list):
             #TODO: switch this to a plugin loader and a 'condition' per plugin on which it should be tried, restoring 'inventory pllugins'
@@ -178,25 +187,26 @@ class Inventory(object):
             if self._restriction:
                 pattern_hash += u":%s" % to_unicode(self._restriction)
 
-        if pattern_hash in HOSTS_PATTERNS_CACHE:
-            return HOSTS_PATTERNS_CACHE[pattern_hash][:]
+        if pattern_hash not in HOSTS_PATTERNS_CACHE:
 
-        patterns = Inventory.split_host_pattern(pattern)
-        hosts = self._evaluate_patterns(patterns)
+            patterns = Inventory.split_host_pattern(pattern)
+            hosts = self._evaluate_patterns(patterns)
 
-        # mainly useful for hostvars[host] access
-        if not ignore_limits_and_restrictions:
-            # exclude hosts not in a subset, if defined
-            if self._subset:
-                subset = self._evaluate_patterns(self._subset)
-                hosts = [ h for h in hosts if h in subset ]
+            # mainly useful for hostvars[host] access
+            if not ignore_limits_and_restrictions:
+                # exclude hosts not in a subset, if defined
+                if self._subset:
+                    subset = self._evaluate_patterns(self._subset)
+                    hosts = [ h for h in hosts if h in subset ]
 
-            # exclude hosts mentioned in any restriction (ex: failed hosts)
-            if self._restriction is not None:
-                hosts = [ h for h in hosts if h in self._restriction ]
+                # exclude hosts mentioned in any restriction (ex: failed hosts)
+                if self._restriction is not None:
+                    hosts = [ h for h in hosts if h in self._restriction ]
 
-        HOSTS_PATTERNS_CACHE[pattern_hash] = hosts[:]
-        return hosts
+            seen = set()
+            HOSTS_PATTERNS_CACHE[pattern_hash] = [x for x in hosts if x not in seen and not seen.add(x)]
+
+        return HOSTS_PATTERNS_CACHE[pattern_hash][:]
 
     @classmethod
     def split_host_pattern(cls, pattern):
@@ -227,15 +237,13 @@ class Inventory(object):
         # If it doesn't, it could still be a single pattern. This accounts for
         # non-separator uses of colons: IPv6 addresses and [x:y] host ranges.
         else:
-            (base, port) = parse_address(pattern, allow_ranges=True)
-            if base:
+            try:
+                (base, port) = parse_address(pattern, allow_ranges=True)
                 patterns = [pattern]
-
-            # The only other case we accept is a ':'-separated list of patterns.
-            # This mishandles IPv6 addresses, and is retained only for backwards
-            # compatibility.
-
-            else:
+            except:
+                # The only other case we accept is a ':'-separated list of patterns.
+                # This mishandles IPv6 addresses, and is retained only for backwards
+                # compatibility.
                 patterns = re.findall(
                     r'''(?:             # We want to match something comprising:
                             [^\s:\[\]]  # (anything other than whitespace or ':[]'
@@ -388,7 +396,7 @@ class Inventory(object):
                     end = -1
                 subscript = (int(start), int(end))
                 if sep == '-':
-                    display.deprecated("Use [x:y] inclusive subscripts instead of [x-y]", version=2.0, removed=True)
+                    display.warning("Use [x:y] inclusive subscripts instead of [x-y] which has been removed")
 
         return (pattern, subscript)
 
@@ -731,12 +739,12 @@ class Inventory(object):
 
             if group and host is None:
                 # load vars in dir/group_vars/name_of_group
-                base_path = os.path.realpath(os.path.join(basedir, "group_vars/%s" % group.name))
-                results = self._variable_manager.add_group_vars_file(base_path, self._loader)
+                base_path = os.path.realpath(os.path.join(to_unicode(basedir, errors='strict'), "group_vars/%s" % group.name))
+                results = combine_vars(results, self._variable_manager.add_group_vars_file(base_path, self._loader))
             elif host and group is None:
                 # same for hostvars in dir/host_vars/name_of_host
-                base_path = os.path.realpath(os.path.join(basedir, "host_vars/%s" % host.name))
-                results = self._variable_manager.add_host_vars_file(base_path, self._loader)
+                base_path = os.path.realpath(os.path.join(to_unicode(basedir, errors='strict'), "host_vars/%s" % host.name))
+                results = combine_vars(results, self._variable_manager.add_host_vars_file(base_path, self._loader))
 
         # all done, results is a dictionary of variables for this particular host.
         return results

lib/ansible/inventory/dir.py

@@ -192,6 +192,8 @@ class InventoryDirectory(object):
         if group.name not in self.groups:
             # it's brand new, add him!
             self.groups[group.name] = group
+        # the Group class does not (yet) implement __eq__/__ne__,
+        # so unlike Host we do a regular comparison here
         if self.groups[group.name] != group:
             # different object, merge
             self._merge_groups(self.groups[group.name], group)
@@ -200,6 +202,9 @@ class InventoryDirectory(object):
         if host.name not in self.hosts:
             # Papa's got a brand new host
             self.hosts[host.name] = host
+        # because the __eq__/__ne__ methods in Host() compare the
+        # name fields rather than references, we use id() here to
+        # do the object comparison for merges
         if self.hosts[host.name] != host:
             # different object, merge
             self._merge_hosts(self.hosts[host.name], host)

lib/ansible/inventory/host.py

@@ -19,6 +19,8 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
+import uuid
+
 from ansible.inventory.group import Group
 from ansible.utils.vars import combine_vars
 
@@ -38,7 +40,7 @@ class Host:
     def __eq__(self, other):
         if not isinstance(other, Host):
             return False
-        return self.name == other.name
+        return self._uuid == other._uuid
 
     def __ne__(self, other):
         return not self.__eq__(other)
@@ -55,6 +57,7 @@ class Host:
             name=self.name,
             vars=self.vars.copy(),
             address=self.address,
+            uuid=self._uuid,
             gathered_facts=self._gathered_facts,
             groups=groups,
         )
@@ -65,6 +68,7 @@ class Host:
         self.name    = data.get('name')
         self.vars    = data.get('vars', dict())
         self.address = data.get('address', '')
+        self._uuid   = data.get('uuid', uuid.uuid4())
 
         groups = data.get('groups', [])
         for group_data in groups:
@@ -84,6 +88,7 @@ class Host:
             self.set_variable('ansible_port', int(port))
 
         self._gathered_facts = False
+        self._uuid = uuid.uuid4()
 
     def __repr__(self):
         return self.get_name()

lib/ansible/inventory/ini.py

@@ -124,6 +124,9 @@ class InventoryParser(object):
                     del pending_declarations[groupname]
 
                 continue
+            elif line.startswith('['):
+                self._raise_error("Invalid section entry: '%s'. Please make sure that there are no spaces" % line + \
+                                  "in the section entry, and that there are no other invalid characters")
 
             # It's not a section, so the current state tells us what kind of
             # definition it must be. The individual parsers will raise an
@@ -264,9 +267,12 @@ class InventoryParser(object):
         # Can the given hostpattern be parsed as a host with an optional port
         # specification?
 
-        (pattern, port) = parse_address(hostpattern, allow_ranges=True)
-        if not pattern:
-            self._raise_error("Can't parse '%s' as host[:port]" % hostpattern)
+        try:
+            (pattern, port) = parse_address(hostpattern, allow_ranges=True)
+        except:
+            # not a recognizable host pattern
+            pattern = hostpattern
+            port = None
 
         # Once we have separated the pattern, we expand it into list of one or
         # more hostnames, depending on whether it contains any [x:y] ranges.

lib/ansible/inventory/script.py

@@ -31,6 +31,7 @@ from ansible.errors import AnsibleError
 from ansible.inventory.host import Host
 from ansible.inventory.group import Group
 from ansible.module_utils.basic import json_dict_bytes_to_unicode
+from ansible.utils.unicode import to_str, to_unicode
 
 
 class InventoryScript:
@@ -57,12 +58,17 @@ class InventoryScript:
         if sp.returncode != 0:
             raise AnsibleError("Inventory script (%s) had an execution error: %s " % (filename,stderr))
 
-        self.data = stdout
+        # make sure script output is unicode so that json loader will output
+        # unicode strings itself
+        try:
+            self.data = to_unicode(stdout, errors="strict")
+        except Exception as e:
+            raise AnsibleError("inventory data from {0} contained characters that cannot be interpreted as UTF-8: {1}".format(to_str(self.filename), to_str(e)))
+
         # see comment about _meta below
         self.host_vars_from_top = None
         self._parse(stderr)
 
-
     def _parse(self, err):
 
         all_hosts = {}
@@ -72,13 +78,11 @@ class InventoryScript:
             self.raw = self._loader.load(self.data)
         except Exception as e:
             sys.stderr.write(err + "\n")
-            raise AnsibleError("failed to parse executable inventory script results from {0}: {1}".format(self.filename, str(e)))
+            raise AnsibleError("failed to parse executable inventory script results from {0}: {1}".format(to_str(self.filename), to_str(e)))
 
         if not isinstance(self.raw, Mapping):
             sys.stderr.write(err + "\n")
-            raise AnsibleError("failed to parse executable inventory script results from {0}: data needs to be formatted as a json dict".format(self.filename))
-
-        self.raw  = json_dict_bytes_to_unicode(self.raw)
+            raise AnsibleError("failed to parse executable inventory script results from {0}: data needs to be formatted as a json dict".format(to_str(self.filename)))
 
         group = None
         for (group_name, data) in self.raw.items():
@@ -103,7 +107,7 @@ class InventoryScript:
             if not isinstance(data, dict):
                 data = {'hosts': data}
             # is not those subkeys, then simplified syntax, host with vars
-            elif not any(k in data for k in ('hosts','vars')):
+            elif not any(k in data for k in ('hosts','vars','children')):
                 data = {'hosts': [group_name], 'vars': data}
 
             if 'hosts' in data:
@@ -112,7 +116,7 @@ class InventoryScript:
                         "data for the host list:\n %s" % (group_name, data))
 
                 for hostname in data['hosts']:
-                    if not hostname in all_hosts:
+                    if hostname not in all_hosts:
                         all_hosts[hostname] = Host(hostname)
                     host = all_hosts[hostname]
                     group.add_host(host)
@@ -145,10 +149,12 @@ class InventoryScript:
     def get_host_variables(self, host):
         """ Runs <script> --host <hostname> to determine additional host variables """
         if self.host_vars_from_top is not None:
-            got = self.host_vars_from_top.get(host.name, {})
+            try:
+                got = self.host_vars_from_top.get(host.name, {})
+            except AttributeError as e:
+                raise AnsibleError("Improperly formated host information for %s: %s" % (host.name,to_str(e)))
             return got
 
-
         cmd = [self.filename, "--host", host.name]
         try:
             sp = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@@ -161,4 +167,3 @@ class InventoryScript:
             return json_dict_bytes_to_unicode(self._loader.load(out))
         except ValueError:
             raise AnsibleError("could not parse post variable response: %s, %s" % (cmd, out))
-

lib/ansible/module_utils/basic.py

@@ -65,6 +65,7 @@ import grp
 import pwd
 import platform
 import errno
+import datetime
 from itertools import repeat, chain
 
 try:
@@ -213,7 +214,7 @@ except ImportError:
             elif isinstance(node, ast.List):
                 return list(map(_convert, node.nodes))
             elif isinstance(node, ast.Dict):
-                return dict((_convert(k), _convert(v)) for k, v in node.items)
+                return dict((_convert(k), _convert(v)) for k, v in node.items())
             elif isinstance(node, ast.Name):
                 if node.name in _safe_names:
                     return _safe_names[node.name]
@@ -369,7 +370,12 @@ def return_values(obj):
     sensitive values pre-jsonification."""
     if isinstance(obj, basestring):
         if obj:
-            yield obj
+            if isinstance(obj, bytes):
+                yield obj
+            else:
+                # Unicode objects should all convert to utf-8
+                # (still must deal with surrogateescape on python3)
+                yield obj.encode('utf-8')
         return
     elif isinstance(obj, Sequence):
         for element in obj:
@@ -391,10 +397,22 @@ def remove_values(value, no_log_strings):
     """ Remove strings in no_log_strings from value.  If value is a container
     type, then remove a lot more"""
     if isinstance(value, basestring):
-        if value in no_log_strings:
+        if isinstance(value, unicode):
+            # This should work everywhere on python2. Need to check
+            # surrogateescape on python3
+            bytes_value = value.encode('utf-8')
+            value_is_unicode = True
+        else:
+            bytes_value = value
+            value_is_unicode = False
+        if bytes_value in no_log_strings:
             return 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER'
         for omit_me in no_log_strings:
-            value = value.replace(omit_me, '*' * 8)
+            bytes_value = bytes_value.replace(omit_me, '*' * 8)
+        if value_is_unicode:
+            value = unicode(bytes_value, 'utf-8', errors='replace')
+        else:
+            value = bytes_value
     elif isinstance(value, Sequence):
         return [remove_values(elem, no_log_strings) for elem in value]
     elif isinstance(value, Mapping):
@@ -406,10 +424,13 @@ def remove_values(value, no_log_strings):
         for omit_me in no_log_strings:
             if omit_me in stringy_value:
                 return 'VALUE_SPECIFIED_IN_NO_LOG_PARAMETER'
+    elif isinstance(value, datetime.datetime):
+        value = value.isoformat()
     else:
         raise TypeError('Value of unknown type: %s, %s' % (type(value), value))
     return value
 
+
 def heuristic_log_sanitize(data, no_log_values=None):
     ''' Remove strings that look like passwords from log messages '''
     # Currently filters:
@@ -497,8 +518,12 @@ class AnsibleModule(object):
         self.no_log = no_log
         self.cleanup_files = []
         self._debug = False
+        # May be used to set modifications to the environment for any
+        # run_command invocation
+        self.run_command_environ_update = {}
 
         self.aliases = {}
+        self._legal_inputs = ['_ansible_check_mode', '_ansible_no_log', '_ansible_debug']
 
         if add_file_common_args:
             for k, v in FILE_COMMON_ARGUMENTS.items():
@@ -507,6 +532,15 @@ class AnsibleModule(object):
 
         self.params = self._load_params()
 
+        # append to legal_inputs and then possibly check against them
+        try:
+            self.aliases = self._handle_aliases()
+        except Exception:
+            e = get_exception()
+            # use exceptions here cause its not safe to call vail json until no_log is processed
+            print('{"failed": true, "msg": "Module alias error: %s"}' % str(e))
+            sys.exit(1)
+
         # Save parameter values that should never be logged
         self.no_log_values = set()
         # Use the argspec to determine which args are no_log
@@ -517,15 +551,10 @@ class AnsibleModule(object):
                 if no_log_object:
                     self.no_log_values.update(return_values(no_log_object))
 
-        # check the locale as set by the current environment, and
-        # reset to LANG=C if it's an invalid/unavailable locale
+        # check the locale as set by the current environment, and reset to
+        # a known valid (LANG=C) if it's an invalid/unavailable locale
         self._check_locale()
 
-        self._legal_inputs = ['_ansible_check_mode', '_ansible_no_log', '_ansible_debug']
-
-        # append to legal_inputs and then possibly check against them
-        self.aliases = self._handle_aliases()
-
         self._check_arguments(check_invalid_arguments)
 
         # check exclusive early
@@ -1034,7 +1063,6 @@ class AnsibleModule(object):
             # as it would be returned by locale.getdefaultlocale()
             locale.setlocale(locale.LC_ALL, '')
         except locale.Error:
-            e = get_exception()
             # fallback to the 'C' locale, which may cause unicode
             # issues but is preferable to simply failing because
             # of an unknown locale
@@ -1047,6 +1075,7 @@ class AnsibleModule(object):
             self.fail_json(msg="An unknown error was encountered while attempting to validate the locale: %s" % e)
 
     def _handle_aliases(self):
+        # this uses exceptions as it happens before we can safely call fail_json
         aliases_results = {} #alias:canon
         for (k,v) in self.argument_spec.items():
             self._legal_inputs.append(k)
@@ -1055,11 +1084,11 @@ class AnsibleModule(object):
             required = v.get('required', False)
             if default is not None and required:
                 # not alias specific but this is a good place to check this
-                self.fail_json(msg="internal error: required and default are mutually exclusive for %s" % k)
+                raise Exception("internal error: required and default are mutually exclusive for %s" % k)
             if aliases is None:
                 continue
             if type(aliases) != list:
-                self.fail_json(msg='internal error: aliases must be a list')
+                raise Exception('internal error: aliases must be a list')
             for alias in aliases:
                 self._legal_inputs.append(alias)
                 aliases_results[alias] = k
@@ -1257,7 +1286,7 @@ class AnsibleModule(object):
         if isinstance(value, bool):
             return value
 
-        if isinstance(value, basestring):
+        if isinstance(value, basestring) or isinstance(value, int):
             return self.boolean(value)
 
         raise TypeError('%s cannot be converted to a bool' % type(value))
@@ -1414,7 +1443,6 @@ class AnsibleModule(object):
         self.log(msg, log_args=log_args)
 
 
-
     def _set_cwd(self):
         try:
             cwd = os.getcwd()
@@ -1507,6 +1535,8 @@ class AnsibleModule(object):
         self.add_path_info(kwargs)
         if not 'changed' in kwargs:
             kwargs['changed'] = False
+        if 'invocation' not in kwargs:
+            kwargs['invocation'] = {'module_args': self.params}
         kwargs = remove_values(kwargs, self.no_log_values)
         self.do_cleanup_files()
         print(self.jsonify(kwargs))
@@ -1517,6 +1547,8 @@ class AnsibleModule(object):
         self.add_path_info(kwargs)
         assert 'msg' in kwargs, "implementation error -- msg to explain the error is required"
         kwargs['failed'] = True
+        if 'invocation' not in kwargs:
+            kwargs['invocation'] = {'module_args': self.params}
         kwargs = remove_values(kwargs, self.no_log_values)
         self.do_cleanup_files()
         print(self.jsonify(kwargs))
@@ -1687,25 +1719,29 @@ class AnsibleModule(object):
             # rename might not preserve context
             self.set_context_if_different(dest, context, False)
 
-    def run_command(self, args, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None):
+    def run_command(self, args, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None, environ_update=None):
         '''
         Execute a command, returns rc, stdout, and stderr.
-        args is the command to run
-        If args is a list, the command will be run with shell=False.
-        If args is a string and use_unsafe_shell=False it will split args to a list and run with shell=False
-        If args is a string and use_unsafe_shell=True it run with shell=True.
-        Other arguments:
-        - check_rc (boolean)    Whether to call fail_json in case of
-                                non zero RC.  Default is False.
-        - close_fds (boolean)   See documentation for subprocess.Popen().
-                                Default is True.
-        - executable (string)   See documentation for subprocess.Popen().
-                                Default is None.
-        - prompt_regex (string) A regex string (not a compiled regex) which
-                                can be used to detect prompts in the stdout
-                                which would otherwise cause the execution
-                                to hang (especially if no input data is
-                                specified)
+
+        :arg args: is the command to run
+            * If args is a list, the command will be run with shell=False.
+            * If args is a string and use_unsafe_shell=False it will split args to a list and run with shell=False
+            * If args is a string and use_unsafe_shell=True it runs with shell=True.
+        :kw check_rc: Whether to call fail_json in case of non zero RC.
+            Default False
+        :kw close_fds: See documentation for subprocess.Popen(). Default True
+        :kw executable: See documentation for subprocess.Popen(). Default None
+        :kw data: If given, information to write to the stdin of the command
+        :kw binary_data: If False, append a newline to the data.  Default False
+        :kw path_prefix: If given, additional path to find the command in.
+            This adds to the PATH environment vairable so helper commands in
+            the same directory can also be found
+        :kw cwd: iIf given, working directory to run the command inside
+        :kw use_unsafe_shell: See `args` parameter.  Default False
+        :kw prompt_regex: Regex string (not a compiled regex) which can be
+            used to detect prompts in the stdout which would otherwise cause
+            the execution to hang (especially if no input data is specified)
+        :kwarg environ_update: dictionary to *update* os.environ with
         '''
 
         shell = False
@@ -1716,7 +1752,9 @@ class AnsibleModule(object):
         elif isinstance(args, basestring) and use_unsafe_shell:
             shell = True
         elif isinstance(args, basestring):
-            args = shlex.split(args.encode('utf-8'))
+            if isinstance(args, unicode):
+                args = args.encode('utf-8')
+            args = shlex.split(args)
         else:
             msg = "Argument 'args' to run_command must be list or string"
             self.fail_json(rc=257, cmd=args, msg=msg)
@@ -1736,10 +1774,19 @@ class AnsibleModule(object):
         msg = None
         st_in = None
 
-        # Set a temporary env path if a prefix is passed
-        env=os.environ
+        # Manipulate the environ we'll send to the new process
+        old_env_vals = {}
+        # We can set this from both an attribute and per call
+        for key, val in self.run_command_environ_update.items():
+            old_env_vals[key] = os.environ.get(key, None)
+            os.environ[key] = val
+        if environ_update:
+            for key, val in environ_update.items():
+                old_env_vals[key] = os.environ.get(key, None)
+                os.environ[key] = val
         if path_prefix:
-            env['PATH']="%s:%s" % (path_prefix, env['PATH'])
+            old_env_vals['PATH'] = os.environ['PATH']
+            os.environ['PATH'] = "%s:%s" % (path_prefix, os.environ['PATH'])
 
         # create a printable version of the command for use
         # in reporting later, which strips out things like
@@ -1781,11 +1828,10 @@ class AnsibleModule(object):
             close_fds=close_fds,
             stdin=st_in,
             stdout=subprocess.PIPE,
-            stderr=subprocess.PIPE
+            stderr=subprocess.PIPE,
+            env=os.environ,
         )
 
-        if path_prefix:
-            kwargs['env'] = env
         if cwd and os.path.isdir(cwd):
             kwargs['cwd'] = cwd
 
@@ -1808,7 +1854,6 @@ class AnsibleModule(object):
                 else:
                     running = args
                 self.log('Executing: ' + running)
-
             cmd = subprocess.Popen(args, **kwargs)
 
             # the communication logic here is essentially taken from that
@@ -1864,6 +1909,13 @@ class AnsibleModule(object):
         except:
             self.fail_json(rc=257, msg=traceback.format_exc(), cmd=clean_args)
 
+        # Restore env settings
+        for key, val in old_env_vals.items():
+            if val is None:
+                del os.environ[key]
+            else:
+                os.environ[key] = val
+
         if rc != 0 and check_rc:
             msg = heuristic_log_sanitize(stderr.rstrip(), self.no_log_values)
             self.fail_json(cmd=clean_args, rc=rc, stdout=stdout, stderr=stderr, msg=msg)

lib/ansible/module_utils/cloudstack.py

@@ -78,6 +78,10 @@ class AnsibleCloudStack(object):
         self.returns = {}
         # these values will be casted to int
         self.returns_to_int = {}
+        # these keys will be compared case sensitive in self.has_changed()
+        self.case_sensitive_keys = [
+            'id',
+        ]
 
         self.module = module
         self._connect()
@@ -138,16 +142,14 @@ class AnsibleCloudStack(object):
                 continue
 
             if key in current_dict:
-
-                # API returns string for int in some cases, just to make sure
-                if isinstance(value, int):
-                    current_dict[key] = int(current_dict[key])
-                elif isinstance(value, str):
-                    current_dict[key] = str(current_dict[key])
-
-                # Only need to detect a singe change, not every item
-                if value != current_dict[key]:
+                if self.case_sensitive_keys and key in self.case_sensitive_keys:
+                    if str(value) != str(current_dict[key]):
+                        return True
+                # Test for diff in case insensitive way
+                elif str(value).lower() != str(current_dict[key]).lower():
                     return True
+            else:
+                return True
         return False
 
 
@@ -218,7 +220,7 @@ class AnsibleCloudStack(object):
         vms = self.cs.listVirtualMachines(**args)
         if vms:
             for v in vms['virtualmachine']:
-                if vm in [ v['name'], v['displayname'], v['id'] ]:
+                if vm.lower() in [ v['name'].lower(), v['displayname'].lower(), v['id'] ]:
                     self.vm = v
                     return self._get_by_key(key, self.vm)
         self.module.fail_json(msg="Virtual machine '%s' not found" % vm)
@@ -238,7 +240,7 @@ class AnsibleCloudStack(object):
 
         if zones:
             for z in zones['zone']:
-                if zone in [ z['name'], z['id'] ]:
+                if zone.lower() in [ z['name'].lower(), z['id'] ]:
                     self.zone = z
                     return self._get_by_key(key, self.zone)
         self.module.fail_json(msg="zone '%s' not found" % zone)

lib/ansible/module_utils/eapi.py

@@ -1,155 +0,0 @@
-#
-# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-"""
-This module adds shared support for Arista EOS devices using eAPI over
-HTTP/S transport.   It is built on module_utils/urls.py which is required
-for proper operation.
-
-In order to use this module, include it as part of a custom
-module as shown below.
-
-** Note: The order of the import statements does matter. **
-
-from ansible.module_utils.basic import *
-from ansible.module_utils.urls import *
-from ansible.module_utils.eapi import *
-
-The eapi module provides the following common argument spec:
-
-    * host (str) - [Required] The IPv4 address or FQDN of the network device
-
-    * port (str) - Overrides the default port to use for the HTTP/S
-        connection.  The default values are 80 for HTTP and
-        443 for HTTPS
-
-    * url_username (str) - [Required] The username to use to authenticate
-        the HTTP/S connection.  Aliases: username
-
-    * url_password (str) - [Required] The password to use to authenticate
-        the HTTP/S connection.  Aliases: password
-
-    * use_ssl (bool) - Specifies whether or not to use an encrypted (HTTPS)
-        connection or not.  The default value is False.
-
-    * enable_mode (bool) - Specifies whether or not to enter `enable` mode
-        prior to executing the command list.  The default value is True
-
-    * enable_password (str) - The password for entering `enable` mode
-        on the switch if configured.
-
-In order to communicate with Arista EOS devices, the eAPI feature
-must be enabled and configured on the device.
-
-"""
-def eapi_argument_spec(spec=None):
-    """Creates an argument spec for working with eAPI
-    """
-    arg_spec = url_argument_spec()
-    arg_spec.update(dict(
-        host=dict(required=True),
-        port=dict(),
-        url_username=dict(required=True, aliases=['username']),
-        url_password=dict(required=True, aliases=['password']),
-        use_ssl=dict(default=True, type='bool'),
-        enable_mode=dict(default=True, type='bool'),
-        enable_password=dict()
-    ))
-    if spec:
-        arg_spec.update(spec)
-    return arg_spec
-
-def eapi_url(module):
-    """Construct a valid Arist eAPI URL
-    """
-    if module.params['use_ssl']:
-        proto = 'https'
-    else:
-        proto = 'http'
-    host = module.params['host']
-    url = '{}://{}'.format(proto, host)
-    if module.params['port']:
-        url = '{}:{}'.format(url, module.params['port'])
-    return '{}/command-api'.format(url)
-
-def to_list(arg):
-    """Convert the argument to a list object
-    """
-    if isinstance(arg, (list, tuple)):
-        return list(arg)
-    elif arg is not None:
-        return [arg]
-    else:
-        return []
-
-def eapi_body(commands, encoding, reqid=None):
-    """Create a valid eAPI JSON-RPC request message
-    """
-    params = dict(version=1, cmds=to_list(commands), format=encoding)
-    return dict(jsonrpc='2.0', id=reqid, method='runCmds', params=params)
-
-def eapi_enable_mode(module):
-    """Build commands for entering `enable` mode on the switch
-    """
-    if module.params['enable_mode']:
-        passwd = module.params['enable_password']
-        if passwd:
-            return dict(cmd='enable', input=passwd)
-        else:
-            return 'enable'
-
-def eapi_command(module, commands, encoding='json'):
-    """Send an ordered list of commands to the device over eAPI
-    """
-    commands = to_list(commands)
-    url = eapi_url(module)
-
-    enable = eapi_enable_mode(module)
-    if enable:
-        commands.insert(0, enable)
-
-    data = eapi_body(commands, encoding)
-    data = module.jsonify(data)
-
-    headers = {'Content-Type': 'application/json-rpc'}
-
-    response, headers = fetch_url(module, url, data=data, headers=headers,
-                                  method='POST')
-
-    if headers['status'] != 200:
-        module.fail_json(**headers)
-
-    response = module.from_json(response.read())
-    if 'error' in response:
-        err = response['error']
-        module.fail_json(msg='json-rpc error', **err)
-
-    if enable:
-        response['result'].pop(0)
-
-    return response['result'], headers
-
-def eapi_configure(module, commands):
-    """Send configuration commands to the device over eAPI
-    """
-    commands.insert(0, 'configure')
-    response, headers = eapi_command(module, commands)
-    response.pop(0)
-    return response, headers
-
-

lib/ansible/module_utils/ec2.py

@@ -152,8 +152,7 @@ def get_aws_connection_info(module, boto3=False):
         boto_params = dict(aws_access_key_id=access_key,
                            aws_secret_access_key=secret_key,
                            aws_session_token=security_token)
-        if validate_certs:
-            boto_params['verify'] = validate_certs
+        boto_params['verify'] = validate_certs
 
         if profile_name:
             boto_params['profile_name'] = profile_name
@@ -170,7 +169,7 @@ def get_aws_connection_info(module, boto3=False):
                 module.fail_json("boto does not support profile_name before 2.24")
             boto_params['profile_name'] = profile_name
 
-        if validate_certs and HAS_LOOSE_VERSION and LooseVersion(boto.Version) >= LooseVersion("2.6.0"):
+        if HAS_LOOSE_VERSION and LooseVersion(boto.Version) >= LooseVersion("2.6.0"):
             boto_params['validate_certs'] = validate_certs
 
     return region, ec2_url, boto_params

lib/ansible/module_utils/eos.py

@@ -0,0 +1,236 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+NET_PASSWD_RE = re.compile(r"[\r\n]?password: $", re.I)
+
+NET_COMMON_ARGS = dict(
+    host=dict(required=True),
+    port=dict(type='int'),
+    username=dict(required=True),
+    password=dict(no_log=True),
+    authorize=dict(default=False, type='bool'),
+    auth_pass=dict(no_log=True),
+    transport=dict(choices=['cli', 'eapi']),
+    use_ssl=dict(default=True, type='bool'),
+    provider=dict()
+)
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+
+class Eapi(object):
+
+    def __init__(self, module):
+        self.module = module
+
+        # sets the module_utils/urls.py req parameters
+        self.module.params['url_username'] = module.params['username']
+        self.module.params['url_password'] = module.params['password']
+
+        self.url = None
+        self.enable = None
+
+    def _get_body(self, commands, encoding, reqid=None):
+        """Create a valid eAPI JSON-RPC request message
+        """
+        params = dict(version=1, cmds=commands, format=encoding)
+        return dict(jsonrpc='2.0', id=reqid, method='runCmds', params=params)
+
+    def connect(self):
+        host = self.module.params['host']
+        port = self.module.params['port']
+
+        if self.module.params['use_ssl']:
+            proto = 'https'
+            if not port:
+                port = 443
+        else:
+            proto = 'http'
+            if not port:
+                port = 80
+
+        self.url = '%s://%s:%s/command-api' % (proto, host, port)
+
+    def authorize(self):
+        if self.module.params['auth_pass']:
+            passwd = self.module.params['auth_pass']
+            self.enable = dict(cmd='enable', input=passwd)
+        else:
+            self.enable = 'enable'
+
+    def send(self, commands, encoding='json'):
+        """Send commands to the device.
+        """
+        clist = to_list(commands)
+
+        if self.enable is not None:
+            clist.insert(0, self.enable)
+
+        data = self._get_body(clist, encoding)
+        data = self.module.jsonify(data)
+
+        headers = {'Content-Type': 'application/json-rpc'}
+
+        response, headers = fetch_url(self.module, self.url, data=data,
+                headers=headers, method='POST')
+
+        if headers['status'] != 200:
+            self.module.fail_json(**headers)
+
+        response = self.module.from_json(response.read())
+        if 'error' in response:
+            err = response['error']
+            self.module.fail_json(msg='json-rpc error', **err)
+
+        if self.enable:
+            response['result'].pop(0)
+
+        return response['result']
+
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+
+        try:
+            self.shell.open(host, port=port, username=username, password=password)
+        except Exception, exc:
+            msg = 'failed to connecto to %s:%s - %s' % (host, port, str(exc))
+            self.module.fail_json(msg=msg)
+
+    def authorize(self):
+        passwd = self.module.params['auth_pass']
+        self.send(Command('enable', prompt=NET_PASSWD_RE, response=passwd))
+
+    def send(self, commands):
+        return self.shell.send(commands)
+
+
+class NetworkModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(NetworkModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def _load_params(self):
+        params = super(NetworkModule, self)._load_params()
+        provider = params.get('provider') or dict()
+        for key, value in provider.items():
+            if key in NET_COMMON_ARGS.keys():
+                if not params.get(key) and value is not None:
+                    params[key] = value
+        return params
+
+    def connect(self):
+        if self.params['transport'] == 'eapi':
+            self.connection = Eapi(self)
+        else:
+            self.connection = Cli(self)
+
+        try:
+            self.connection.connect()
+            self.execute('terminal length 0')
+
+            if self.params['authorize']:
+                self.connection.authorize()
+
+        except Exception, exc:
+            self.fail_json(msg=exc.message)
+
+    def configure(self, commands):
+        commands = to_list(commands)
+        commands.insert(0, 'configure terminal')
+        responses = self.execute(commands)
+        responses.pop(0)
+
+        return responses
+
+    def config_replace(self, commands):
+        if self.params['transport'] == 'cli':
+            self.fail_json(msg='config replace only supported over eapi')
+
+        cmd = 'configure replace terminal:'
+        commands = '\n'.join(to_list(commands))
+        command = dict(cmd=cmd, input=commands)
+        self.execute(command)
+
+    def execute(self, commands, **kwargs):
+        try:
+            return self.connection.send(commands, **kwargs)
+        except Exception, exc:
+            self.fail_json(msg=exc.message, commands=commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=3)
+
+    def get_config(self):
+        cmd = 'show running-config'
+        if self.params.get('include_defaults'):
+            cmd += ' all'
+        if self.params['transport'] == 'cli':
+            return self.execute(cmd)[0]
+        else:
+            resp = self.execute(cmd, encoding='text')
+            return resp[0]
+
+
+def get_module(**kwargs):
+    """Return instance of NetworkModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+
+    module = NetworkModule(**kwargs)
+
+    # HAS_PARAMIKO is set by module_utils/shell.py
+    if module.params['transport'] == 'cli' and not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    module.connect()
+
+    return module
+

lib/ansible/module_utils/f5.py

@@ -51,19 +51,35 @@ def f5_argument_spec():
 def f5_parse_arguments(module):
     if not bigsuds_found:
         module.fail_json(msg="the python bigsuds module is required")
-    if not module.params['validate_certs']:
-        disable_ssl_cert_validation()
+
+    if module.params['validate_certs']:
+        import ssl
+        if not hasattr(ssl, 'SSLContext'):
+            module.fail_json(msg='bigsuds does not support verifying certificates with python < 2.7.9.  Either update python or set validate_certs=False on the task')
+
     return (module.params['server'],module.params['user'],module.params['password'],module.params['state'],module.params['partition'],module.params['validate_certs'])
 
-def bigip_api(bigip, user, password):
-    api = bigsuds.BIGIP(hostname=bigip, username=user, password=password)
-    return api
+def bigip_api(bigip, user, password, validate_certs):
+    try:
+        # bigsuds >= 1.0.3
+        api = bigsuds.BIGIP(hostname=bigip, username=user, password=password, verify=validate_certs)
+    except TypeError:
+        # bigsuds < 1.0.3, no verify param
+        if validate_certs:
+            # Note: verified we have SSLContext when we parsed params
+            api = bigsuds.BIGIP(hostname=bigip, username=user, password=password)
+        else:
+            import ssl
+            if hasattr(ssl, 'SSLContext'):
+                # Really, you should never do this.  It disables certificate
+                # verification *globally*.  But since older bigip libraries
+                # don't give us a way to toggle verification we need to
+                # disable it at the global level.
+                # From https://www.python.org/dev/peps/pep-0476/#id29
+                ssl._create_default_https_context = ssl._create_unverified_context
+            api = bigsuds.BIGIP(hostname=bigip, username=user, password=password)
 
-def disable_ssl_cert_validation():
-    # You probably only want to do this for testing and never in production.
-    # From https://www.python.org/dev/peps/pep-0476/#id29
-    import ssl
-    ssl._create_default_https_context = ssl._create_unverified_context
+    return api
 
 # Fully Qualified name (with the partition)
 def fq_name(partition,name):

lib/ansible/module_utils/facts.py

@@ -119,6 +119,7 @@ class Facts(object):
                     ('/etc/gentoo-release', 'Gentoo'),
                     ('/etc/os-release', 'Debian'),
                     ('/etc/lsb-release', 'Mandriva'),
+                    ('/etc/altlinux-release', 'Altlinux'),
                     ('/etc/os-release', 'NA'),
                 )
     SELINUX_MODE_DICT = { 1: 'enforcing', 0: 'permissive', -1: 'disabled' }
@@ -270,7 +271,7 @@ class Facts(object):
             OracleLinux = 'RedHat', OVS = 'RedHat', OEL = 'RedHat', Amazon = 'RedHat',
             XenServer = 'RedHat', Ubuntu = 'Debian', Debian = 'Debian', Raspbian = 'Debian', Slackware = 'Slackware', SLES = 'Suse',
             SLED = 'Suse', openSUSE = 'Suse', SuSE = 'Suse', SLES_SAP = 'Suse', Gentoo = 'Gentoo', Funtoo = 'Gentoo',
-            Archlinux = 'Archlinux', Manjaro = 'Archlinux', Mandriva = 'Mandrake', Mandrake = 'Mandrake',
+            Archlinux = 'Archlinux', Manjaro = 'Archlinux', Mandriva = 'Mandrake', Mandrake = 'Mandrake', Altlinux = 'Altlinux',
             Solaris = 'Solaris', Nexenta = 'Solaris', OmniOS = 'Solaris', OpenIndiana = 'Solaris',
             SmartOS = 'Solaris', AIX = 'AIX', Alpine = 'Alpine', MacOSX = 'Darwin',
             FreeBSD = 'FreeBSD', HPUX = 'HP-UX'
@@ -323,7 +324,7 @@ class Facts(object):
             for (path, name) in Facts.OSDIST_LIST:
                 if os.path.exists(path):
                     if os.path.getsize(path) > 0:
-                        if self.facts['distribution'] in ('Fedora', ):
+                        if self.facts['distribution'] in ('Fedora', 'Altlinux', ):
                             # Once we determine the value is one of these distros
                             # we trust the values are always correct
                             break
@@ -356,6 +357,13 @@ class Facts(object):
                             else:
                                 self.facts['distribution'] = data.split()[0]
                             break
+                        elif name == 'Altlinux':
+                            data = get_file_content(path)
+                            if 'ALT Linux' in data:
+                                self.facts['distribution'] = name
+                            else:
+                                self.facts['distribution'] = data.split()[0]
+                            break
                         elif name == 'OtherLinux':
                             data = get_file_content(path)
                             if 'Amazon' in data:
@@ -430,9 +438,9 @@ class Facts(object):
                                             release = re.search("^PRETTY_NAME=[^(]+ \(?([^)]+?)\)", line)
                                             if release:
                                                 self.facts['distribution_release'] = release.groups()[0]
-                                        elif 'enterprise' in data.lower():
+                                        elif 'enterprise' in data.lower() and 'VERSION_ID' in line:
                                              release = re.search('^VERSION_ID="?[0-9]+\.?([0-9]*)"?', line) # SLES doesn't got funny release names
-                                             if release:
+                                             if release.group(1):
                                                  release = release.group(1)
                                              else:
                                                  release = "0" # no minor number, so it is the first release
@@ -524,7 +532,12 @@ class Facts(object):
         keytypes = ('dsa', 'rsa', 'ecdsa', 'ed25519')
 
         if self.facts['system'] == 'Darwin':
-            keydir = '/etc'
+            if self.facts['distribution'] == 'MacOSX' and LooseVersion(self.facts['distribution_version']) >= LooseVersion('10.11') :
+                keydir = '/etc/ssh'
+            else:
+                keydir = '/etc'
+        if self.facts['distribution'] == 'Altlinux':
+            keydir = '/etc/openssh'
         else:
             keydir = '/etc/ssh'
 
@@ -551,14 +564,16 @@ class Facts(object):
         proc_1 = get_file_content('/proc/1/comm')
         if proc_1 is None:
             rc, proc_1, err = module.run_command("ps -p 1 -o comm|tail -n 1", use_unsafe_shell=True)
+        else:
+            proc_1 = os.path.basename(proc_1)
 
-        if proc_1 in ['init', '/sbin/init']:
-            # many systems return init, so this cannot be trusted
+        if proc_1 in ['init', '/sbin/init', 'bash']:
+            # many systems return init, so this cannot be trusted, bash is from docker
             proc_1 = None
 
         # if not init/None it should be an identifiable or custom init, so we are done!
         if proc_1 is not None:
-            self.facts['service_mgr'] = proc_1
+            self.facts['service_mgr'] = proc_1.strip()
 
         # start with the easy ones
         elif  self.facts['distribution'] == 'MacOSX':
@@ -2971,14 +2986,19 @@ def get_file_content(path, default=None, strip=True):
     data = default
     if os.path.exists(path) and os.access(path, os.R_OK):
         try:
-            datafile = open(path)
-            data = datafile.read()
-            if strip:
-                data = data.strip()
-            if len(data) == 0:
-                data = default
-        finally:
-            datafile.close()
+            try:
+                datafile = open(path)
+                data = datafile.read()
+                if strip:
+                    data = data.strip()
+                if len(data) == 0:
+                    data = default
+            finally:
+                datafile.close()
+        except:
+            # ignore errors as some jails/containers might have readable permissions but not allow reads to proc
+            # done in 2 blocks for 2.4 compat
+            pass
     return data
 
 def get_file_lines(path):

lib/ansible/module_utils/ios.py

@@ -0,0 +1,146 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+NET_PASSWD_RE = re.compile(r"[\r\n]?password: $", re.I)
+
+NET_COMMON_ARGS = dict(
+    host=dict(required=True),
+    port=dict(default=22, type='int'),
+    username=dict(required=True),
+    password=dict(no_log=True),
+    authorize=dict(default=False, type='bool'),
+    auth_pass=dict(no_log=True),
+    provider=dict()
+)
+
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+
+        try:
+            self.shell.open(host, port=port, username=username, password=password)
+        except Exception, exc:
+            msg = 'failed to connecto to %s:%s - %s' % (host, port, str(exc))
+            self.module.fail_json(msg=msg)
+
+    def authorize(self):
+        passwd = self.module.params['auth_pass']
+        self.send(Command('enable', prompt=NET_PASSWD_RE, response=passwd))
+
+    def send(self, commands):
+        return self.shell.send(commands)
+
+
+class NetworkModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(NetworkModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def _load_params(self):
+        params = super(NetworkModule, self)._load_params()
+        provider = params.get('provider') or dict()
+        for key, value in provider.items():
+            if key in NET_COMMON_ARGS.keys():
+                params[key] = value
+        return params
+
+    def connect(self):
+        try:
+            self.connection = Cli(self)
+            self.connection.connect()
+            self.execute('terminal length 0')
+
+            if self.params['authorize']:
+                self.connection.authorize()
+
+        except Exception, exc:
+            self.fail_json(msg=exc.message)
+
+    def configure(self, commands):
+        commands = to_list(commands)
+        commands.insert(0, 'configure terminal')
+        responses = self.execute(commands)
+        responses.pop(0)
+        return responses
+
+    def execute(self, commands, **kwargs):
+        try:
+            return self.connection.send(commands, **kwargs)
+        except Exception, exc:
+            self.fail_json(msg=exc.message, commands=commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=1)
+
+    def get_config(self):
+        cmd = 'show running-config'
+        if self.params.get('include_defaults'):
+            cmd += ' all'
+        return self.execute(cmd)[0]
+
+
+def get_module(**kwargs):
+    """Return instance of NetworkModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+
+    module = NetworkModule(**kwargs)
+
+    # HAS_PARAMIKO is set by module_utils/shell.py
+    if not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    module.connect()
+    return module
+

lib/ansible/module_utils/iosxr.py

@@ -0,0 +1,127 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+NET_PASSWD_RE = re.compile(r"[\r\n]?password: $", re.I)
+
+NET_COMMON_ARGS = dict(
+    host=dict(required=True),
+    port=dict(default=22, type='int'),
+    username=dict(required=True),
+    password=dict(no_log=True),
+    provider=dict()
+)
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+
+        try:
+            self.shell.open(host, port=port, username=username, password=password)
+        except Exception, exc:
+            msg = 'failed to connecto to %s:%s - %s' % (host, port, str(exc))
+            self.module.fail_json(msg=msg)
+
+    def send(self, commands):
+        return self.shell.send(commands)
+
+class NetworkModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(NetworkModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def _load_params(self):
+        params = super(NetworkModule, self)._load_params()
+        provider = params.get('provider') or dict()
+        for key, value in provider.items():
+            if key in NET_COMMON_ARGS.keys():
+                params[key] = value
+        return params
+
+    def connect(self):
+        try:
+            self.connection = Cli(self)
+            self.connection.connect()
+            self.execute('terminal length 0')
+        except Exception, exc:
+            self.fail_json(msg=exc.message)
+
+    def configure(self, commands):
+        commands = to_list(commands)
+        commands.insert(0, 'configure terminal')
+        commands.append('commit')
+        responses = self.execute(commands)
+        responses.pop(0)
+        responses.pop()
+        return responses
+
+    def execute(self, commands, **kwargs):
+        return self.connection.send(commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=1)
+
+    def get_config(self):
+        return self.execute('show running-config')[0]
+
+def get_module(**kwargs):
+    """Return instance of NetworkModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+
+    module = NetworkModule(**kwargs)
+
+    if not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    module.connect()
+    return module
+

lib/ansible/module_utils/ismount.py

@@ -0,0 +1,90 @@
+# This code is part of Ansible, but is an independent component.
+# This particular file snippet, and this file snippet only, is based on
+# Lib/posixpath.py of cpython
+# It is licensed under the PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
+#
+# 1. This LICENSE AGREEMENT is between the Python Software Foundation
+# ("PSF"), and the Individual or Organization ("Licensee") accessing and
+# otherwise using this software ("Python") in source or binary form and
+# its associated documentation.
+#
+# 2. Subject to the terms and conditions of this License Agreement, PSF hereby
+# grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
+# analyze, test, perform and/or display publicly, prepare derivative works,
+# distribute, and otherwise use Python alone or in any derivative version,
+# provided, however, that PSF's License Agreement and PSF's notice of copyright,
+# i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
+# 2011, 2012, 2013, 2014, 2015 Python Software Foundation; All Rights Reserved"
+# are retained in Python alone or in any derivative version prepared by Licensee.
+#
+# 3. In the event Licensee prepares a derivative work that is based on
+# or incorporates Python or any part thereof, and wants to make
+# the derivative work available to others as provided herein, then
+# Licensee hereby agrees to include in any such work a brief summary of
+# the changes made to Python.
+#
+# 4. PSF is making Python available to Licensee on an "AS IS"
+# basis.  PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
+# IMPLIED.  BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
+# DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
+# FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
+# INFRINGE ANY THIRD PARTY RIGHTS.
+#
+# 5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
+# FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
+# A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
+# OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
+#
+# 6. This License Agreement will automatically terminate upon a material
+# breach of its terms and conditions.
+#
+# 7. Nothing in this License Agreement shall be deemed to create any
+# relationship of agency, partnership, or joint venture between PSF and
+# Licensee.  This License Agreement does not grant permission to use PSF
+# trademarks or trade name in a trademark sense to endorse or promote
+# products or services of Licensee, or any third party.
+#
+# 8. By copying, installing or otherwise using Python, Licensee
+# agrees to be bound by the terms and conditions of this License
+# Agreement.
+
+import os
+
+
+def ismount(path):
+    """Test whether a path is a mount point
+    clone of os.path.ismount (from cpython Lib/posixpath.py)
+    fixed to solve https://github.com/ansible/ansible-modules-core/issues/2186
+    and workaround non-fixed http://bugs.python.org/issue2466
+    this should be rewritten as soon as python issue 2466 is fixed
+    probably check for python version and use os.path.ismount if fixed
+
+    to remove replace in this file ismount( -> os.path.ismount( and remove this
+    function"""
+
+    try:
+        s1 = os.lstat(path)
+    except OSError:
+        # the OSError should be handled with more care
+        # it could be a "permission denied" but path is still a mount
+        return False
+    else:
+        # A symlink can never be a mount point
+        if os.path.stat.S_ISLNK(s1.st_mode):
+            return False
+
+    parent = os.path.join(path, os.path.pardir)
+    parent = os.path.realpath(parent)
+
+    try:
+        s2 = os.lstat(parent)
+    except OSError:
+        # one should handle the returned OSError with more care to figure
+        # out whether this is still a mount
+        return False
+
+    if s1.st_dev != s2.st_dev:
+        return True     # path/.. on a different device as path
+    if s1.st_ino == s2.st_ino:
+        return True     # path/.. is the same i-node as path, i.e. path=='/'
+    return False

lib/ansible/module_utils/junos.py

@@ -0,0 +1,128 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+NET_COMMON_ARGS = dict(
+    host=dict(required=True),
+    port=dict(default=22, type='int'),
+    username=dict(required=True),
+    password=dict(no_log=True),
+    provider=dict()
+)
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+
+        try:
+            self.shell.open(host, port=port, username=username, password=password)
+        except Exception, exc:
+            msg = 'failed to connecto to %s:%s - %s' % (host, port, str(exc))
+            self.module.fail_json(msg=msg)
+
+    def send(self, commands):
+        return self.shell.send(commands)
+
+
+class NetworkModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(NetworkModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def _load_params(self):
+        params = super(NetworkModule, self)._load_params()
+        provider = params.get('provider') or dict()
+        for key, value in provider.items():
+            if key in NET_COMMON_ARGS.keys():
+                params[key] = value
+        return params
+
+    def connect(self):
+        self.connection = Cli(self)
+        self.connection.connect()
+        if self.connection.shell._matched_prompt.strip().endswith('%'):
+            self.execute('cli')
+        self.execute('set cli screen-length 0')
+
+    def configure(self, commands):
+        commands = to_list(commands)
+        commands.insert(0, 'configure')
+        commands.append('commit and-quit')
+        responses = self.execute(commands)
+        responses.pop(0)
+        responses.pop()
+        return responses
+
+    def execute(self, commands, **kwargs):
+        return self.connection.send(commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=4)
+
+    def get_config(self):
+        cmd = 'show configuration'
+        return self.execute(cmd)[0]
+
+def get_module(**kwargs):
+    """Return instance of NetworkModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+    kwargs['check_invalid_arguments'] = False
+
+    module = NetworkModule(**kwargs)
+
+    # HAS_PARAMIKO is set by module_utils/shell.py
+    if not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    module.connect()
+    return module
+

lib/ansible/module_utils/known_hosts.py

@@ -28,7 +28,11 @@
 
 import os
 import hmac
-import urlparse
+
+try:
+    import urlparse
+except ImportError:
+    import urllib.parse as urlparse
 
 try:
     from hashlib import sha1
@@ -74,12 +78,12 @@ def get_fqdn(repo_url):
     if "@" in repo_url and "://" not in repo_url:
         # most likely an user@host:path or user@host/path type URL
         repo_url = repo_url.split("@", 1)[1]
-        if ":" in repo_url:
-            repo_url = repo_url.split(":")[0]
-            result = repo_url
+        if repo_url.startswith('['):
+            result = repo_url.split(']', 1)[0] + ']'
+        elif ":" in repo_url:
+            result = repo_url.split(":")[0]
         elif "/" in repo_url:
-            repo_url = repo_url.split("/")[0]
-            result = repo_url
+            result = repo_url.split("/")[0]
     elif "://" in repo_url:
         # this should be something we can parse with urlparse
         parts = urlparse.urlparse(repo_url)
@@ -87,11 +91,13 @@ def get_fqdn(repo_url):
         # ensure we actually have a parts[1] before continuing.
         if parts[1] != '':
             result = parts[1]
-            if ":" in result:
-                result = result.split(":")[0]
             if "@" in result:
                 result = result.split("@", 1)[1]
 
+            if result[0].startswith('['):
+                result = result.split(']', 1)[0] + ']'
+            elif ":" in result:
+                result = result.split(":")[0]
     return result
 
 def check_hostkey(module, fqdn):
@@ -113,6 +119,7 @@ def not_in_host_file(self, host):
     host_file_list.append(user_host_file)
     host_file_list.append("/etc/ssh/ssh_known_hosts")
     host_file_list.append("/etc/ssh/ssh_known_hosts2")
+    host_file_list.append("/etc/openssh/ssh_known_hosts")
 
     hfiles_not_found = 0
     for hf in host_file_list:
@@ -169,7 +176,7 @@ def add_host_key(module, fqdn, key_type="rsa", create_dir=False):
     if not os.path.exists(user_ssh_dir):
         if create_dir:
             try:
-                os.makedirs(user_ssh_dir, 0700)
+                os.makedirs(user_ssh_dir, int('700', 8))
             except:
                 module.fail_json(msg="failed to create host key directory: %s" % user_ssh_dir)
         else:

lib/ansible/module_utils/mysql.py

@@ -0,0 +1,66 @@
+# This code is part of Ansible, but is an independent component.
+# This particular file snippet, and this file snippet only, is BSD licensed.
+# Modules you write using this snippet, which is embedded dynamically by Ansible
+# still belong to the author of the module, and may assign their own license
+# to the complete work.
+#
+# Copyright (c), Jonathan Mainguy <jon@soh.re>, 2015
+# Most of this was originally added by Sven Schliesing @muffl0n in the mysql_user.py module
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without modification,
+# are permitted provided that the following conditions are met:
+#
+#    * Redistributions of source code must retain the above copyright
+#      notice, this list of conditions and the following disclaimer.
+#    * Redistributions in binary form must reproduce the above copyright notice,
+#      this list of conditions and the following disclaimer in the documentation
+#      and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+# IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
+# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE
+# USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+
+
+def mysql_connect(module, login_user=None, login_password=None, config_file='', ssl_cert=None, ssl_key=None, ssl_ca=None, db=None, cursor_class=None):
+    config = {
+        'host': module.params['login_host'],
+        'ssl': {
+            }
+    }
+
+    if module.params['login_unix_socket']:
+        config['unix_socket'] = module.params['login_unix_socket']
+    else:
+        config['port'] = module.params['login_port']
+
+    if os.path.exists(config_file):
+        config['read_default_file'] = config_file
+
+    # If login_user or login_password are given, they should override the
+    # config file
+    if login_user is not None:
+        config['user'] = login_user
+    if login_password is not None:
+        config['passwd'] = login_password
+    if ssl_cert is not None:
+        config['ssl']['cert'] = ssl_cert
+    if ssl_key is not None:
+        config['ssl']['key'] = ssl_key
+    if ssl_ca is not None:
+        config['ssl']['ca'] = ssl_ca
+    if db is not None:
+        config['db'] = db
+
+    db_connection = MySQLdb.connect(**config)
+    if cursor_class is not None:
+        return db_connection.cursor(cursorclass=MySQLdb.cursors.DictCursor)
+    else:
+        return db_connection.cursor()

lib/ansible/module_utils/netcfg.py

@@ -0,0 +1,170 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import re
+import collections
+
+class ConfigLine(object):
+
+    def __init__(self, text):
+        self.text = text
+        self.children = list()
+        self.parents = list()
+        self.raw = None
+
+    def __str__(self):
+        return self.raw
+
+    def __eq__(self, other):
+        if self.text == other.text:
+            return self.parents == other.parents
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+
+def parse(lines, indent):
+    toplevel = re.compile(r'\S')
+    childline = re.compile(r'^\s*(.+)$')
+    repl = r'([{|}|;])'
+
+    ancestors = list()
+    config = list()
+
+    for line in str(lines).split('\n'):
+        text = str(re.sub(repl, '', line)).strip()
+
+        cfg = ConfigLine(text)
+        cfg.raw = line
+
+        if not text or text[0] in ['!', '#']:
+            continue
+
+        # handle top level commands
+        if toplevel.match(line):
+            ancestors = [cfg]
+
+        # handle sub level commands
+        else:
+            match = childline.match(line)
+            line_indent = match.start(1)
+            level = int(line_indent / indent)
+            parent_level = level - 1
+
+            cfg.parents = ancestors[:level]
+
+            if level > len(ancestors):
+                config.append(cfg)
+                continue
+
+            for i in range(level, len(ancestors)):
+                ancestors.pop()
+
+            ancestors.append(cfg)
+            ancestors[parent_level].children.append(cfg)
+
+        config.append(cfg)
+
+    return config
+
+
+class Conditional(object):
+    ''' 
+    Used in command modules to evaluate waitfor conditions
+    ''' 
+
+    OPERATORS = {
+        'eq': ['eq', '=='],
+        'neq': ['neq', 'ne', '!='],
+        'gt': ['gt', '>'],
+        'ge': ['ge', '>='],
+        'lt': ['lt', '<'],
+        'le': ['le', '<='],
+        'contains': ['contains']
+    }
+
+    def __init__(self, conditional):
+        self.raw = conditional
+
+        key, op, val = shlex.split(conditional)
+        self.key = key
+        self.func = self.func(op)
+        self.value = self._cast_value(val)
+
+    def __call__(self, data):
+        try:
+            value = self.get_value(dict(result=data))
+            return self.func(value)
+        except Exception:
+            raise ValueError(self.key)
+
+    def _cast_value(self, value):
+        if value in BOOLEANS_TRUE:
+            return True
+        elif value in BOOLEANS_FALSE:
+            return False
+        elif re.match(r'^\d+\.d+$', value):
+            return float(value)
+        elif re.match(r'^\d+$', value):
+            return int(value)
+        else:
+            return unicode(value)
+
+    def func(self, oper):
+        for func, operators in self.OPERATORS.items():
+            if oper in operators:
+                return getattr(self, func)
+        raise AttributeError('unknown operator: %s' % oper)
+
+    def get_value(self, result):
+        for key in self.key.split('.'):
+            match = re.match(r'^(.+)\[(\d+)\]', key)
+            if match:
+                key, index = match.groups()
+                result = result[key][int(index)]
+            else:
+                result = result.get(key)
+        return result
+
+    def number(self, value):
+        if '.' in str(value):
+            return float(value)
+        else:
+            return int(value)
+
+    def eq(self, value):
+        return value == self.value
+
+    def neq(self, value):
+        return value != self.value
+
+    def gt(self, value):
+        return self.number(value) > self.value
+
+    def ge(self, value):
+        return self.number(value) >= self.value
+
+    def lt(self, value):
+        return self.number(value) < self.value
+
+    def le(self, value):
+        return self.number(value) <= self.value
+
+    def contains(self, value):
+        return self.value in value

lib/ansible/module_utils/nxapi.py

@@ -1,130 +0,0 @@
-#
-# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-#
-"""
-This module adds support for Cisco NXAPI to Ansible shared
-module_utils.  It builds on module_utils/urls.py to provide
-NXAPI support over HTTP/S which is required for proper operation.
-
-In order to use this module, include it as part of a custom
-module as shown below.
-
-** Note: The order of the import statements does matter. **
-
-from ansible.module_utils.basic import *
-from ansible.module_utils.urls import *
-from ansible.module_utils.nxapi import *
-
-The nxapi module provides the following common argument spec:
-
-    * host (str) - [Required] The IPv4 address or FQDN of the network device
-
-    * port (str) - Overrides the default port to use for the HTTP/S
-        connection.  The default values are 80 for HTTP and
-        443 for HTTPS
-
-    * url_username (str) - [Required] The username to use to authenticate
-        the HTTP/S connection.  Aliases: username
-
-    * url_password (str) - [Required] The password to use to authenticate
-        the HTTP/S connection.  Aliases: password
-
-    * use_ssl (bool) - Specifies whether or not to use an encrypted (HTTPS)
-        connection or not.  The default value is False.
-
-    * command_type (str) - The type of command to send to the remote
-        device.  Valid values in `cli_show`, `cli_show_ascii`, 'cli_conf`
-        and `bash`.  The default value is `cli_show_ascii`
-
-In order to communicate with Cisco NXOS devices, the NXAPI feature
-must be enabled and configured on the device.
-
-"""
-
-NXAPI_COMMAND_TYPES = ['cli_show', 'cli_show_ascii', 'cli_conf', 'bash']
-
-def nxapi_argument_spec(spec=None):
-    """Creates an argument spec for working with NXAPI
-    """
-    arg_spec = url_argument_spec()
-    arg_spec.update(dict(
-        host=dict(required=True),
-        port=dict(),
-        url_username=dict(required=True, aliases=['username']),
-        url_password=dict(required=True, aliases=['password']),
-        use_ssl=dict(default=False, type='bool'),
-        command_type=dict(default='cli_show_ascii', choices=NXAPI_COMMAND_TYPES)
-    ))
-    if spec:
-        arg_spec.update(spec)
-    return arg_spec
-
-def nxapi_url(module):
-    """Constructs a valid NXAPI url
-    """
-    if module.params['use_ssl']:
-        proto = 'https'
-    else:
-        proto = 'http'
-    host = module.params['host']
-    url = '{}://{}'.format(proto, host)
-    port = module.params['port']
-    if module.params['port']:
-        url = '{}:{}'.format(url, module.params['port'])
-    url = '{}/ins'.format(url)
-    return url
-
-def nxapi_body(commands, command_type, **kwargs):
-    """Encodes a NXAPI JSON request message
-    """
-    if isinstance(commands, (list, set, tuple)):
-        commands = ' ;'.join(commands)
-
-    msg = {
-        'version': kwargs.get('version') or '1.2',
-        'type': command_type,
-        'chunk': kwargs.get('chunk') or '0',
-        'sid': kwargs.get('sid'),
-        'input': commands,
-        'output_format': 'json'
-    }
-
-    return dict(ins_api=msg)
-
-def nxapi_command(module, commands, command_type=None, **kwargs):
-    """Sends the list of commands to the device over NXAPI
-    """
-    url = nxapi_url(module)
-
-    command_type = command_type or module.params['command_type']
-
-    data = nxapi_body(commands, command_type)
-    data = module.jsonify(data)
-
-    headers = {'Content-Type': 'text/json'}
-
-    response, headers = fetch_url(module, url, data=data, headers=headers,
-                                  method='POST')
-
-    status = kwargs.get('status') or 200
-    if headers['status'] != status:
-        module.fail_json(**headers)
-
-    response = module.from_json(response.read())
-    return response, headers
-

lib/ansible/module_utils/nxos.py

@@ -0,0 +1,235 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+NET_PASSWD_RE = re.compile(r"[\r\n]?password: $", re.I)
+
+NET_COMMON_ARGS = dict(
+    host=dict(required=True),
+    port=dict(type='int'),
+    username=dict(required=True),
+    password=dict(no_log=True),
+    transport=dict(default='cli', choices=['cli', 'nxapi']),
+    use_ssl=dict(default=False, type='bool'),
+    provider=dict()
+)
+
+NXAPI_COMMAND_TYPES = ['cli_show', 'cli_show_ascii', 'cli_conf', 'bash']
+NXAPI_ENCODINGS = ['json', 'xml']
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+class Nxapi(object):
+
+    def __init__(self, module):
+        self.module = module
+
+        # sets the module_utils/urls.py req parameters
+        self.module.params['url_username'] = module.params['username']
+        self.module.params['url_password'] = module.params['password']
+
+        self.url = None
+        self.enable = None
+
+    def _get_body(self, commands, command_type, encoding, version='1.2', chunk='0', sid=None):
+        """Encodes a NXAPI JSON request message
+        """
+        if isinstance(commands, (list, set, tuple)):
+            commands = ' ;'.join(commands)
+
+        if encoding not in NXAPI_ENCODINGS:
+            self.module.fail_json("Invalid encoding. Received %s. Expected one of %s" %
+                (encoding, ','.join(NXAPI_ENCODINGS)))
+
+        msg = {
+            'version': version,
+            'type': command_type,
+            'chunk': chunk,
+            'sid': sid,
+            'input': commands,
+            'output_format': encoding
+        }
+        return dict(ins_api=msg)
+
+    def connect(self):
+        host = self.module.params['host']
+        port = self.module.params['port']
+
+        if self.module.params['use_ssl']:
+            proto = 'https'
+            if not port:
+                port = 443
+        else:
+            proto = 'http'
+            if not port:
+                port = 80
+
+        self.url = '%s://%s:%s/ins' % (proto, host, port)
+
+    def send(self, commands, command_type='cli_show_ascii', encoding='json'):
+        """Send commands to the device.
+        """
+        clist = to_list(commands)
+
+        if command_type not in NXAPI_COMMAND_TYPES:
+            self.module.fail_json(msg="Invalid command_type. Received %s. Expected one of %s." %
+                (command_type, ','.join(NXAPI_COMMAND_TYPES)))
+
+        data = self._get_body(clist, command_type, encoding)
+        data = self.module.jsonify(data)
+
+        headers = {'Content-Type': 'application/json'}
+
+        response, headers = fetch_url(self.module, self.url, data=data,
+                headers=headers, method='POST')
+
+        if headers['status'] != 200:
+            self.module.fail_json(**headers)
+
+        response = self.module.from_json(response.read())
+        result = list()
+
+        try:
+            output = response['ins_api']['outputs']['output']
+            if isinstance(output, list):
+                for item in response['ins_api']['outputs']['output']:
+                    if item['code'] != '200':
+                        self.module.fail_json(msg=item['msg'], command=item['input'],
+                                code=item['code'])
+                    else:
+                        result.append(item['body'])
+            elif output['code'] != '200':
+                self.module.fail_json(msg=item['msg'], command=item['input'],
+                        code=item['code'])
+            else:
+                result.append(output['body'])
+        except Exception:
+            self.module.fail_json(**headers)
+
+        return result
+
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+
+        try:
+            self.shell.open(host, port=port, username=username, password=password)
+        except Exception, exc:
+            msg = 'failed to connecto to %s:%s - %s' % (host, port, str(exc))
+            self.module.fail_json(msg=msg)
+
+    def send(self, commands, encoding='text'):
+        return self.shell.send(commands)
+
+
+class NetworkModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(NetworkModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def _load_params(self):
+        params = super(NetworkModule, self)._load_params()
+        provider = params.get('provider') or dict()
+        for key, value in provider.items():
+            if key in NET_COMMON_ARGS.keys():
+                if not params.get(key) and value is not None:
+                    params[key] = value
+        return params
+
+    def connect(self):
+        if self.params['transport'] == 'nxapi':
+            self.connection = Nxapi(self)
+        else:
+            self.connection = Cli(self)
+
+        self.connection.connect()
+        if self.params['transport'] == 'cli':
+            self.execute('terminal length 0')
+
+    def configure(self, commands):
+        commands = to_list(commands)
+        if self.params['transport'] == 'cli':
+            commands.insert(0, 'configure')
+            responses = self.execute(commands)
+            responses.pop(0)
+        else:
+            responses = self.execute(commands, command_type='cli_conf')
+        return responses
+
+    def execute(self, commands, **kwargs):
+        try:
+            return self.connection.send(commands, **kwargs)
+        except Exception, exc:
+            self.fail_json(msg=exc.message, commands=commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=2)
+
+    def get_config(self):
+        cmd = 'show running-config'
+        if self.params.get('include_defaults'):
+            cmd += ' all'
+
+        response = self.execute(cmd)
+        return response[0]
+
+
+def get_module(**kwargs):
+    """Return instance of NetworkModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+
+    module = NetworkModule(**kwargs)
+
+    # HAS_PARAMIKO is set by module_utils/shell.py
+    if module.params['transport'] == 'cli' and not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    module.connect()
+    return module

lib/ansible/module_utils/openswitch.py

@@ -0,0 +1,246 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+import time
+import json
+
+try:
+    from runconfig import runconfig
+    from opsrest.settings import settings
+    from opsrest.manager import OvsdbConnectionManager
+    from opslib import restparser
+    HAS_OPS = True
+except ImportError:
+    HAS_OPS = False
+
+NET_PASSWD_RE = re.compile(r"[\r\n]?password: $", re.I)
+
+NET_COMMON_ARGS = dict(
+    host=dict(),
+    port=dict(type='int'),
+    username=dict(),
+    password=dict(no_log=True),
+    transport=dict(default='ssh', choices=['ssh', 'cli', 'rest']),
+)
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+def get_idl():
+    manager = OvsdbConnectionManager(settings.get('ovs_remote'),
+                                     settings.get('ovs_schema'))
+    manager.start()
+    idl = manager.idl
+
+    init_seq_no = 0
+    while (init_seq_no == idl.change_seqno):
+        idl.run()
+        time.sleep(1)
+
+    return idl
+
+def get_schema():
+    return restparser.parseSchema(settings.get('ext_schema'))
+
+def get_runconfig():
+    idl = get_idl()
+    schema = get_schema()
+    return runconfig.RunConfigUtil(idl, schema)
+
+class Response(object):
+
+    def __init__(self, resp, hdrs):
+        self.body = resp.read()
+        self.headers = hdrs
+
+    @property
+    def json(self):
+        try:
+            return json.loads(self.body)
+        except ValueError:
+            return None
+
+class Rest(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.baseurl = None
+
+    def connect(self):
+        host = self.module.params['host']
+        port = self.module.params['port']
+
+        if self.module.params['use_ssl']:
+            proto = 'https'
+            if not port:
+                port = 443
+        else:
+            proto = 'http'
+            if not port:
+                port = 80
+
+        self.baseurl = '%s://%s:%s/rest/v1' % (proto, host, port)
+
+    def _url_builder(self, path):
+        if path[0] == '/':
+            path = path[1:]
+        return '%s/%s' % (self.baseurl, path)
+
+    def send(self, method, path, data=None, headers=None):
+        url = self._url_builder(path)
+        data = self.module.jsonify(data)
+
+        if headers is None:
+            headers = dict()
+        headers.update({'Content-Type': 'application/json'})
+
+        resp, hdrs = fetch_url(self.module, url, data=data, headers=headers,
+                method=method)
+
+        return Response(resp, hdrs)
+
+    def get(self, path, data=None, headers=None):
+        return self.send('GET', path, data, headers)
+
+    def put(self, path, data=None, headers=None):
+        return self.send('PUT', path, data, headers)
+
+    def post(self, path, data=None, headers=None):
+        return self.send('POST', path, data, headers)
+
+    def delete(self, path, data=None, headers=None):
+        return self.send('DELETE', path, data, headers)
+
+class Cli(object):
+
+    def __init__(self, module):
+        self.module = module
+        self.shell = None
+
+    def connect(self, **kwargs):
+        host = self.module.params['host']
+        port = self.module.params['port'] or 22
+
+        username = self.module.params['username']
+        password = self.module.params['password']
+
+        self.shell = Shell()
+        self.shell.open(host, port=port, username=username, password=password)
+
+    def send(self, commands, encoding='text'):
+        return self.shell.send(commands)
+
+class OpsModule(AnsibleModule):
+
+    def __init__(self, *args, **kwargs):
+        super(OpsModule, self).__init__(*args, **kwargs)
+        self.connection = None
+        self._config = None
+        self._runconfig = None
+
+    @property
+    def config(self):
+        if not self._config:
+            self._config = self.get_config()
+        return self._config
+
+    def connect(self):
+        if self.params['transport'] == 'rest':
+            self.connection = Rest(self)
+        elif self.params['transport'] == 'cli':
+            self.connection = Cli(self)
+
+        try:
+            self.connection.connect()
+        except Exception, exc:
+            self.fail_json(msg=exc.message)
+
+    def configure(self, config):
+        if self.params['transport'] == 'cli':
+            commands = to_list(config)
+            commands.insert(0, 'configure terminal')
+            responses = self.execute(commands)
+            responses.pop(0)
+            return responses
+        elif self.params['transport'] == 'rest':
+            path = '/system/full-configuration'
+            return self.connection.put(path, data=config)
+        else:
+            if not self._runconfig:
+                self._runconfig = get_runconfig()
+            self._runconfig.write_config_to_db(config)
+
+    def execute(self, commands, **kwargs):
+        try:
+            return self.connection.send(commands, **kwargs)
+        except Exception, exc:
+            self.fail_json(msg=exc.message, commands=commands)
+
+    def disconnect(self):
+        self.connection.close()
+
+    def parse_config(self, cfg):
+        return parse(cfg, indent=4)
+
+    def get_config(self):
+        if self.params['transport'] == 'cli':
+            return self.execute('show running-config')[0]
+
+        elif self.params['transport'] == 'rest':
+            resp = self.connection.get('/system/full-configuration')
+            return resp.json
+
+        else:
+            if not self._runconfig:
+                self._runconfig = get_runconfig()
+            return self._runconfig.get_running_config()
+
+
+def get_module(**kwargs):
+    """Return instance of OpsModule
+    """
+    argument_spec = NET_COMMON_ARGS.copy()
+    if kwargs.get('argument_spec'):
+        argument_spec.update(kwargs['argument_spec'])
+    kwargs['argument_spec'] = argument_spec
+    kwargs['check_invalid_arguments'] = False
+
+    module = OpsModule(**kwargs)
+
+    if not HAS_OPS and module.params['transport'] == 'ssh':
+        module.fail_json(msg='could not import ops library')
+
+    # HAS_PARAMIKO is set by module_utils/shell.py
+    if module.params['transport'] == 'cli' and not HAS_PARAMIKO:
+        module.fail_json(msg='paramiko is required but does not appear to be installed')
+
+    # copy in values from local action.
+    params = json_dict_unicode_to_bytes(json.loads(MODULE_COMPLEX_ARGS))
+    for key, value in params.iteritems():
+        module.params[key] = value
+
+    if module.params['transport'] in ['cli', 'rest']:
+        module.connect()
+
+    return module
+

lib/ansible/module_utils/shell.py

@@ -0,0 +1,199 @@
+#
+# (c) 2015 Peter Sprygada, <psprygada@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+import re
+import socket
+
+from StringIO import StringIO
+
+try:
+    import paramiko
+    HAS_PARAMIKO = True
+except ImportError:
+    HAS_PARAMIKO = False
+
+
+ANSI_RE = re.compile(r'(\x1b\[\?1h\x1b=)')
+
+CLI_PROMPTS_RE = [
+    re.compile(r'[\r\n]?[a-zA-Z]{1}[a-zA-Z0-9-]*[>|#](?:\s*)$'),
+    re.compile(r'[\r\n]?[a-zA-Z]{1}[a-zA-Z0-9-]*\(.+\)#(?:\s*)$')
+]
+
+CLI_ERRORS_RE = [
+    re.compile(r"% ?Error"),
+    re.compile(r"^% \w+", re.M),
+    re.compile(r"% ?Bad secret"),
+    re.compile(r"invalid input", re.I),
+    re.compile(r"(?:incomplete|ambiguous) command", re.I),
+    re.compile(r"connection timed out", re.I),
+    re.compile(r"[^\r\n]+ not found", re.I),
+    re.compile(r"'[^']' +returned error code: ?\d+"),
+    re.compile(r"syntax error"),
+    re.compile(r"unknown command")
+]
+
+def to_list(val):
+    if isinstance(val, (list, tuple)):
+        return list(val)
+    elif val is not None:
+        return [val]
+    else:
+        return list()
+
+class ShellError(Exception):
+
+    def __init__(self, msg, command=None):
+        super(ShellError, self).__init__(msg)
+        self.message = msg
+        self.command = command
+
+class Command(object):
+
+    def __init__(self, command, prompt=None, response=None):
+        self.command = command
+        self.prompt = prompt
+        self.response = response
+
+    def __str__(self):
+        return self.command
+
+class Shell(object):
+
+    def __init__(self):
+        self.ssh = None
+        self.shell = None
+
+        self._matched_prompt = None
+
+        self.prompts = list()
+        self.prompts.extend(CLI_PROMPTS_RE)
+
+        self.errors = list()
+        self.errors.extend(CLI_ERRORS_RE)
+
+    def open(self, host, port=22, username=None, password=None,
+            timeout=10, key_filename=None, pkey=None, look_for_keys=None,
+            allow_agent=False):
+
+        self.ssh = paramiko.SSHClient()
+        self.ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+        use_keys = password is None
+
+        self.ssh.connect(host, port=port, username=username, password=password,
+                    timeout=timeout, look_for_keys=look_for_keys, pkey=pkey,
+                    key_filename=key_filename, allow_agent=allow_agent)
+
+        self.shell = self.ssh.invoke_shell()
+        self.shell.settimeout(10)
+        self.receive()
+
+    def strip(self, data):
+        return ANSI_RE.sub('', data)
+
+    def receive(self, cmd=None):
+        recv = StringIO()
+
+        while True:
+            data = self.shell.recv(200)
+
+            recv.write(data)
+            recv.seek(recv.tell() - 200)
+
+            window = self.strip(recv.read())
+
+            if isinstance(cmd, Command):
+                self.handle_input(window, prompt=cmd.prompt,
+                                  response=cmd.response)
+
+            try:
+                if self.read(window):
+                    resp = self.strip(recv.getvalue())
+                    return self.sanitize(cmd, resp)
+            except ShellError, exc:
+                exc.command = cmd
+                raise
+
+    def send(self, commands):
+        responses = list()
+        try:
+            for command in to_list(commands):
+                cmd = '%s\r' % str(command)
+                self.shell.sendall(cmd)
+                responses.append(self.receive(command))
+        except socket.timeout, exc:
+            raise ShellError("timeout trying to send command", cmd)
+        return responses
+
+    def close(self):
+        self.shell.close()
+
+    def handle_input(self, resp, prompt, response):
+        if not prompt or not response:
+            return
+
+        prompt = to_list(prompt)
+        response = to_list(response)
+
+        for pr, ans in zip(prompt, response):
+            match = pr.search(resp)
+            if match:
+                cmd = '%s\r' % ans
+                self.shell.sendall(cmd)
+
+    def sanitize(self, cmd, resp):
+        cleaned = []
+        for line in resp.splitlines():
+            if line.startswith(str(cmd)) or self.read(line):
+                continue
+            cleaned.append(line)
+        return "\n".join(cleaned)
+
+    def read(self, response):
+        for regex in self.errors:
+            if regex.search(response):
+                raise ShellError('%s' % response)
+
+        for regex in self.prompts:
+            match = regex.search(response)
+            if match:
+                self._matched_prompt = match.group()
+                return True
+
+def get_cli_connection(module):
+    host = module.params['host']
+    port = module.params['port']
+    if not port:
+        port = 22
+
+    username = module.params['username']
+    password = module.params['password']
+
+    try:
+        cli = Cli()
+        cli.open(host, port=port, username=username, password=password)
+    except paramiko.ssh_exception.AuthenticationException, exc:
+        module.fail_json(msg=exc.message)
+    except socket.error, exc:
+        host = '%s:%s' % (host, port)
+        module.fail_json(msg=exc.strerror, errno=exc.errno, host=host)
+    except socket.timeout:
+        module.fail_json(msg='socket timed out')
+
+    return cli

lib/ansible/module_utils/urls.py

@@ -310,36 +310,45 @@ class NoSSLError(SSLValidationError):
     """Needed to connect to an HTTPS url but no ssl library available to verify the certificate"""
     pass
 
+# Some environments (Google Compute Engine's CoreOS deploys) do not compile
+# against openssl and thus do not have any HTTPS support.
+CustomHTTPSConnection = CustomHTTPSHandler = None
+if hasattr(httplib, 'HTTPSConnection') and hasattr(urllib2, 'HTTPSHandler'):
+    class CustomHTTPSConnection(httplib.HTTPSConnection):
+        def __init__(self, *args, **kwargs):
+            httplib.HTTPSConnection.__init__(self, *args, **kwargs)
+            if HAS_SSLCONTEXT:
+                self.context = create_default_context()
+                if self.cert_file:
+                    self.context.load_cert_chain(self.cert_file, self.key_file)
 
-class CustomHTTPSConnection(httplib.HTTPSConnection):
-    def __init__(self, *args, **kwargs):
-        httplib.HTTPSConnection.__init__(self, *args, **kwargs)
-        if HAS_SSLCONTEXT:
-            self.context = create_default_context()
-            if self.cert_file:
-                self.context.load_cert_chain(self.cert_file, self.key_file)
+        def connect(self):
+            "Connect to a host on a given (SSL) port."
 
-    def connect(self):
-        "Connect to a host on a given (SSL) port."
+            if hasattr(self, 'source_address'):
+                sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address)
+            else:
+                sock = socket.create_connection((self.host, self.port), self.timeout)
 
-        if hasattr(self, 'source_address'):
-            sock = socket.create_connection((self.host, self.port), self.timeout, self.source_address)
-        else:
-            sock = socket.create_connection((self.host, self.port), self.timeout)
-        if self._tunnel_host:
-            self.sock = sock
-            self._tunnel()
-        if HAS_SSLCONTEXT:
-            self.sock = self.context.wrap_socket(sock, server_hostname=self.host)
-        else:
-            self.sock = ssl.wrap_socket(sock, keyfile=self.key_file, certfile=self.cert_file, ssl_version=PROTOCOL)
+            server_hostname = self.host
+            # Note: self._tunnel_host is not available on py < 2.6 but this code
+            # isn't used on py < 2.6 (lack of create_connection)
+            if self._tunnel_host:
+                self.sock = sock
+                self._tunnel()
+                server_hostname = self._tunnel_host
 
-class CustomHTTPSHandler(urllib2.HTTPSHandler):
+            if HAS_SSLCONTEXT:
+                self.sock = self.context.wrap_socket(sock, server_hostname=server_hostname)
+            else:
+                self.sock = ssl.wrap_socket(sock, keyfile=self.key_file, certfile=self.cert_file, ssl_version=PROTOCOL)
 
-    def https_open(self, req):
-        return self.do_open(CustomHTTPSConnection, req)
+    class CustomHTTPSHandler(urllib2.HTTPSHandler):
 
-    https_request = urllib2.AbstractHTTPHandler.do_request_
+        def https_open(self, req):
+            return self.do_open(CustomHTTPSConnection, req)
+
+        https_request = urllib2.AbstractHTTPHandler.do_request_
 
 def generic_urlparse(parts):
     '''
@@ -373,7 +382,10 @@ def generic_urlparse(parts):
         # get the username, password, etc.
         try:
             netloc_re = re.compile(r'^((?:\w)+(?::(?:\w)+)?@)?([A-Za-z0-9.-]+)(:\d+)?$')
-            (auth, hostname, port) = netloc_re.match(parts[1])
+            match = netloc_re.match(parts[1])
+            auth = match.group(1)
+            hostname = match.group(2)
+            port = match.group(3)
             if port:
                 # the capture group for the port will include the ':',
                 # so remove it and convert the port to an integer
@@ -383,6 +395,8 @@ def generic_urlparse(parts):
                 # and then split it up based on the first ':' found
                 auth = auth[:-1]
                 username, password = auth.split(':', 1)
+            else:
+                username = password = None
             generic_parts['username'] = username
             generic_parts['password'] = password
             generic_parts['hostname'] = hostname
@@ -390,7 +404,7 @@ def generic_urlparse(parts):
         except:
             generic_parts['username'] = None
             generic_parts['password'] = None
-            generic_parts['hostname'] = None
+            generic_parts['hostname'] = parts[1]
             generic_parts['port']     = None
     return generic_parts
 
@@ -532,7 +546,8 @@ class SSLValidationHandler(urllib2.BaseHandler):
             s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
             if https_proxy:
                 proxy_parts = generic_urlparse(urlparse.urlparse(https_proxy))
-                s.connect((proxy_parts.get('hostname'), proxy_parts.get('port')))
+                port = proxy_parts.get('port') or 443
+                s.connect((proxy_parts.get('hostname'), port))
                 if proxy_parts.get('scheme') == 'http':
                     s.sendall(self.CONNECT_COMMAND % (self.hostname, self.port))
                     if proxy_parts.get('username'):
@@ -542,7 +557,7 @@ class SSLValidationHandler(urllib2.BaseHandler):
                     connect_result = s.recv(4096)
                     self.validate_proxy_response(connect_result)
                     if context:
-                        ssl_s = context.wrap_socket(s, server_hostname=proxy_parts.get('hostname'))
+                        ssl_s = context.wrap_socket(s, server_hostname=self.hostname)
                     else:
                         ssl_s = ssl.wrap_socket(s, ca_certs=tmp_ca_cert_path, cert_reqs=ssl.CERT_REQUIRED, ssl_version=PROTOCOL)
                         match_hostname(ssl_s.getpeercert(), self.hostname)
@@ -661,8 +676,9 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
         handlers.append(proxyhandler)
 
     # pre-2.6 versions of python cannot use the custom https
-    # handler, since the socket class is lacking this method
-    if hasattr(socket, 'create_connection'):
+    # handler, since the socket class is lacking create_connection.
+    # Some python builds lack HTTPS support.
+    if hasattr(socket, 'create_connection') and CustomHTTPSHandler:
         handlers.append(CustomHTTPSHandler)
 
     opener = urllib2.build_opener(*handlers)

lib/ansible/new_inventory/__init__.py

@@ -1,344 +0,0 @@
-# (c) 2012-2014, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-#############################################
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-import re
-import sys
-from ansible import constants as C
-from ansible.inventory.group import Group
-from .host import Host
-from ansible.plugins.inventory.aggregate import InventoryAggregateParser
-from ansible import errors
-
-class Inventory:
-    '''
-    Create hosts and groups from inventory
-
-    Retrieve the hosts and groups that ansible knows about from this class.
-
-    Retrieve raw variables (non-expanded) from the Group and Host classes
-    returned from here.
-    '''
-
-    def __init__(self, inventory_list=C.DEFAULT_HOST_LIST):
-        '''
-        :kwarg inventory_list: A list of inventory sources.  This may be file
-            names which will be parsed as ini-like files, executable scripts
-            which return inventory data as json, directories of both of the above,
-            or hostnames.  Files and directories are 
-        :kwarg vault_password: Password to use if any of the inventory sources
-            are in an ansible vault
-        '''
-
-        self._restricted_to  = None
-        self._filter_pattern = None
-
-        parser = InventoryAggregateParser(inventory_list)
-        parser.parse()
-
-        self._basedir = parser.basedir
-        self._hosts   = parser.hosts
-        self._groups  = parser.groups
-
-    def get_hosts(self):
-        '''
-        Return the list of hosts, after filtering based on any set pattern
-        and restricting the results based on the set host restrictions.
-        '''
-
-        if self._filter_pattern:
-            hosts = self._filter_hosts()
-        else:
-            hosts = self._hosts[:]
-
-        if self._restricted_to is not None:
-            # this will preserve the order of hosts after intersecting them
-            res_set = set(hosts).intersection(self._restricted_to)
-            return [h for h in hosts if h in res_set]
-        else:
-            return hosts[:]
-
-    def get_groups(self):
-        '''
-        Retrieve the Group objects known to the Inventory
-        '''
-
-        return self._groups[:]
-
-    def get_host(self, hostname):
-        '''
-        Retrieve the Host object for a hostname
-        '''
-
-        for host in self._hosts:
-            if host.name == hostname:
-                return host
-
-        return None
-
-    def get_group(self, groupname):
-        '''
-        Retrieve the Group object for a groupname
-        '''
-
-        for group in self._groups:
-            if group.name == groupname:
-                return group
-
-        return None
-
-    def add_group(self, group):
-        '''
-        Add a new group to the inventory
-        '''
-
-        if group not in self._groups:
-            self._groups.append(group)
-
-    def set_filter_pattern(self, pattern='all'):
-        '''
-        Sets a pattern upon which hosts/groups will be filtered.
-        This pattern can contain logical groupings such as unions,
-        intersections and negations using special syntax.
-        '''
-
-        self._filter_pattern = pattern
-
-    def set_host_restriction(self, restriction):
-        '''
-        Restrict operations to hosts in the given list
-        '''
-
-        assert isinstance(restriction, list)
-        self._restricted_to = restriction[:]
-
-    def remove_host_restriction(self):
-        '''
-        Remove the restriction on hosts, if any.
-        '''
-
-        self._restricted_to = None
-
-    def _filter_hosts(self):
-        """
-        Limits inventory results to a subset of inventory that matches a given
-        list of patterns, such as to select a subset of a hosts selection that also
-        belongs to a certain geographic group or numeric slice.
-
-        Corresponds to --limit parameter to ansible-playbook
-
-        :arg patterns: The pattern to limit with.  If this is None it
-            clears the subset.  Multiple patterns may be specified as a comma,
-            semicolon, or colon separated string.
-        """
-
-        hosts = []
-
-        pattern_regular = []
-        pattern_intersection = []
-        pattern_exclude = []
-
-        patterns = self._pattern.replace(";",":").split(":")
-        for p in patterns:
-            if p.startswith("!"):
-                pattern_exclude.append(p)
-            elif p.startswith("&"):
-                pattern_intersection.append(p)
-            elif p:
-                pattern_regular.append(p)
-
-        # if no regular pattern was given, hence only exclude and/or intersection
-        # make that magically work
-        if pattern_regular == []:
-            pattern_regular = ['all']
-
-        # when applying the host selectors, run those without the "&" or "!"
-        # first, then the &s, then the !s.
-        patterns = pattern_regular + pattern_intersection + pattern_exclude
-
-        for p in patterns:
-            intersect = False
-            negate    = False
-            if p.startswith('&'):
-                intersect = True
-            elif p.startswith('!'):
-                p = p[1:]
-                negate = True
-
-            target = self._resolve_pattern(p)
-            if isinstance(target, Host):
-                if negate and target in hosts:
-                    # remove it
-                    hosts.remove(target)
-                elif target not in hosts:
-                    # for both union and intersections, we just append it
-                    hosts.append(target)
-            else:
-                if intersect:
-                    hosts = [ h for h in hosts if h not in target ]
-                elif negate:
-                    hosts = [ h for h in hosts if h in target ]
-                else:
-                    to_append = [ h for h in target if h.name not in [ y.name for y in hosts ] ]
-                    hosts.extend(to_append)
-
-        return hosts
-
-    def _resolve_pattern(self, pattern):
-        target = self.get_host(pattern)
-        if target:
-            return target
-        else:
-            (name, enumeration_details) = self._enumeration_info(pattern)
-            hpat = self._hosts_in_unenumerated_pattern(name)
-            result = self._apply_ranges(pattern, hpat)
-            return result
-
-    def _enumeration_info(self, pattern):
-        """
-        returns (pattern, limits) taking a regular pattern and finding out
-        which parts of it correspond to start/stop offsets.  limits is
-        a tuple of (start, stop) or None
-        """
-
-        # Do not parse regexes for enumeration info
-        if pattern.startswith('~'):
-            return (pattern, None)
-
-        # The regex used to match on the range, which can be [x] or [x-y].
-        pattern_re = re.compile("^(.*)\[([-]?[0-9]+)(?:(?:-)([0-9]+))?\](.*)$")
-        m = pattern_re.match(pattern)
-        if m:
-            (target, first, last, rest) = m.groups()
-            first = int(first)
-            if last:
-                if first < 0:
-                    raise errors.AnsibleError("invalid range: negative indices cannot be used as the first item in a range")
-                last = int(last)
-            else:
-                last = first
-            return (target, (first, last))
-        else:
-            return (pattern, None)
-
-    def _apply_ranges(self, pat, hosts):
-        """
-        given a pattern like foo, that matches hosts, return all of hosts
-        given a pattern like foo[0:5], where foo matches hosts, return the first 6 hosts
-        """ 
-
-        # If there are no hosts to select from, just return the
-        # empty set. This prevents trying to do selections on an empty set.
-        # issue#6258
-        if not hosts:
-            return hosts
-
-        (loose_pattern, limits) = self._enumeration_info(pat)
-        if not limits:
-            return hosts
-
-        (left, right) = limits
-
-        if left == '':
-            left = 0
-        if right == '':
-            right = 0
-        left=int(left)
-        right=int(right)
-        try:
-            if left != right:
-                return hosts[left:right]
-            else:
-                return [ hosts[left] ]
-        except IndexError:
-            raise errors.AnsibleError("no hosts matching the pattern '%s' were found" % pat)
-
-    def _hosts_in_unenumerated_pattern(self, pattern):
-        """ Get all host names matching the pattern """
-
-        results = []
-        hosts = []
-        hostnames = set()
-
-        # ignore any negative checks here, this is handled elsewhere
-        pattern = pattern.replace("!","").replace("&", "")
-
-        def __append_host_to_results(host):
-            if host not in results and host.name not in hostnames:
-                hostnames.add(host.name)
-                results.append(host)
-
-        groups = self.get_groups()
-        for group in groups:
-            if pattern == 'all':
-                for host in group.get_hosts():
-                    __append_host_to_results(host)
-            else:
-                if self._match(group.name, pattern):
-                    for host in group.get_hosts():
-                        __append_host_to_results(host)
-                else:
-                    matching_hosts = self._match_list(group.get_hosts(), 'name', pattern)
-                    for host in matching_hosts:
-                        __append_host_to_results(host)
-
-        if pattern in ["localhost", "127.0.0.1"] and len(results) == 0:
-            new_host = self._create_implicit_localhost(pattern)
-            results.append(new_host)
-        return results
-
-    def _create_implicit_localhost(self, pattern):
-        new_host = Host(pattern)
-        new_host._connection = 'local'
-        new_host.set_variable("ansible_python_interpreter", sys.executable)
-        ungrouped = self.get_group("ungrouped")
-        if ungrouped is None:
-            self.add_group(Group('ungrouped'))
-            ungrouped = self.get_group('ungrouped')
-            self.get_group('all').add_child_group(ungrouped)
-        ungrouped.add_host(new_host)
-        return new_host
-
-    def is_file(self):
-        '''
-        Did inventory come from a file?
-
-        :returns: True if the inventory is file based, False otherwise
-        '''
-        pass
-
-    def src(self):
-        '''
-        What's the complete path to the inventory file?
-
-        :returns: Complete path to the inventory file.  None if inventory is
-            not file-based
-        '''
-        pass
-
-    def basedir(self):
-        '''
-        What directory from which the inventory was read.
-        '''
-
-        return self._basedir
-

lib/ansible/new_inventory/host.py

@@ -1,51 +0,0 @@
-# (c) 2012-2014, Michael DeHaan <michael.dehaan@gmail.com>
-#
-# This file is part of Ansible
-#
-# Ansible is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# Ansible is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
-
-# Make coding more python3-ish
-from __future__ import (absolute_import, division, print_function)
-__metaclass__ = type
-
-class Host:
-    def __init__(self, name):
-        self._name         = name
-        self._connection   = None
-        self._ipv4_address = ''
-        self._ipv6_address = ''
-        self._port         = 22
-        self._vars         = dict()
-
-    def __repr__(self):
-        return self.get_name()
-
-    def get_name(self):
-        return self._name
-
-    def get_groups(self):
-        return []
-
-    def set_variable(self, name, value):
-        ''' sets a variable for this host '''
-
-        self._vars[name] = value
-
-    def get_vars(self):
-        ''' returns all variables for this host '''
-
-        all_vars = self._vars.copy()
-        all_vars.update(dict(inventory_hostname=self._name))
-        return all_vars
-

lib/ansible/parsing/mod_args.py

@@ -21,7 +21,7 @@ __metaclass__ = type
 
 from ansible.compat.six import iteritems, string_types
 
-from ansible.errors import AnsibleParserError
+from ansible.errors import AnsibleParserError,AnsibleError
 from ansible.plugins import module_loader
 from ansible.parsing.splitter import parse_kv, split_args
 from ansible.template import Templar
@@ -137,7 +137,16 @@ class ModuleArgsParser:
         # than those which may be parsed/normalized next
         final_args = dict()
         if additional_args:
-            final_args.update(additional_args)
+            if isinstance(additional_args, string_types):
+                templar = Templar(loader=None)
+                if templar._contains_vars(additional_args):
+                    final_args['_variable_params'] = additional_args
+                else:
+                    raise AnsibleParserError("Complex args containing variables cannot use bare variables, and must use the full variable style ('{{var_name}}')")
+            elif isinstance(additional_args, dict):
+                final_args.update(additional_args)
+            else:
+                raise AnsibleParserError('Complex args must be a dictionary or variable string ("{{var}}").')
 
         # how we normalize depends if we figured out what the module name is
         # yet.  If we have already figured it out, it's an 'old style' invocation.
@@ -155,6 +164,13 @@ class ModuleArgsParser:
                     tmp_args = parse_kv(tmp_args)
                 args.update(tmp_args)
 
+        # only internal variables can start with an underscore, so
+        # we don't allow users to set them directy in arguments
+        if args and action not in ('command', 'shell', 'script', 'raw'):
+            for arg in args:
+                if arg.startswith('_ansible_'):
+                    raise AnsibleError("invalid parameter specified for action '%s': '%s'" % (action, arg))
+
         # finally, update the args we're going to return with the ones
         # which were normalized above
         if args:
@@ -206,18 +222,21 @@ class ModuleArgsParser:
         action = None
         args = None
 
+        actions_allowing_raw = ('command', 'shell', 'script', 'raw')
         if isinstance(thing, dict):
             # form is like:  copy: { src: 'a', dest: 'b' } ... common for structured (aka "complex") args
             thing = thing.copy()
             if 'module' in thing:
-                action = thing['module']
+                action, module_args = self._split_module_string(thing['module'])
                 args = thing.copy()
+                check_raw = action in actions_allowing_raw
+                args.update(parse_kv(module_args, check_raw=check_raw))
                 del args['module']
 
         elif isinstance(thing, string_types):
             # form is like:  copy: src=a dest=b ... common shorthand throughout ansible
             (action, args) = self._split_module_string(thing)
-            check_raw = action in ('command', 'shell', 'script', 'raw')
+            check_raw = action in actions_allowing_raw
             args = parse_kv(args, check_raw=check_raw)
 
         else:
@@ -268,7 +287,7 @@ class ModuleArgsParser:
             if item in module_loader or item == 'meta' or item == 'include':
                 # finding more than one module name is a problem
                 if action is not None:
-                    raise AnsibleParserError("conflicting action statements", obj=self._task_ds)
+                    raise AnsibleParserError("conflicting action statements (%s, %s)" % (action, item), obj=self._task_ds)
                 action = item
                 thing = value
                 action, args = self._normalize_parameters(value, action=action, additional_args=additional_args)
@@ -282,7 +301,7 @@ class ModuleArgsParser:
                         obj=self._task_ds)
 
             else:
-                raise AnsibleParserError("no action detected in task", obj=self._task_ds)
+                raise AnsibleParserError("no action detected in task. This often indicates a misspelled module name, or incorrect module path.", obj=self._task_ds)
         elif args.get('_raw_params', '') != '' and action not in RAW_PARAM_MODULES:
             templar = Templar(loader=None)
             raw_params = args.pop('_raw_params')

lib/ansible/parsing/splitter.py

@@ -65,8 +65,8 @@ def parse_kv(args, check_raw=False):
                 raise
 
         raw_params = []
-        for x in vargs:
-            x = _decode_escapes(x)
+        for orig_x in vargs:
+            x = _decode_escapes(orig_x)
             if "=" in x:
                 pos = 0
                 try:
@@ -83,19 +83,14 @@ def parse_kv(args, check_raw=False):
                 k = x[:pos]
                 v = x[pos + 1:]
 
-                # only internal variables can start with an underscore, so
-                # we don't allow users to set them directy in arguments
-                if k.startswith('_'):
-                    raise AnsibleError("invalid parameter specified: '%s'" % k)
-
                 # FIXME: make the retrieval of this list of shell/command
                 #        options a function, so the list is centralized
                 if check_raw and k not in ('creates', 'removes', 'chdir', 'executable', 'warn'):
-                    raw_params.append(x)
+                    raw_params.append(orig_x)
                 else:
                     options[k.strip()] = unquote(v.strip())
             else:
-                raw_params.append(x)
+                raw_params.append(orig_x)
 
         # recombine the free-form params, if any were found, and assign
         # them to a special option for use later by the shell/command module
@@ -261,6 +256,6 @@ def split_args(args):
     # If we're done and things are not at zero depth or we're still inside quotes,
     # raise an error to indicate that the args were unbalanced
     if print_depth or block_depth or comment_depth or inside_quotes:
-        raise Exception("error while splitting arguments, either an unbalanced jinja2 block or quotes")
+        raise AnsibleError("error while splitting arguments, either an unbalanced jinja2 block or quotes")
 
     return params

lib/ansible/parsing/utils/addresses.py

@@ -20,6 +20,7 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import re
+from ansible.errors import AnsibleParserError, AnsibleError
 
 # Components that match a numeric or alphanumeric begin:end or begin:end:step
 # range expression inside square brackets.
@@ -162,6 +163,7 @@ patterns = {
             $
         '''.format(label=label), re.X|re.I|re.UNICODE
     ),
+
 }
 
 def parse_address(address, allow_ranges=False):
@@ -183,8 +185,8 @@ def parse_address(address, allow_ranges=False):
     # First, we extract the port number if one is specified.
 
     port = None
-    for type in ['bracketed_hostport', 'hostport']:
-        m = patterns[type].match(address)
+    for matching in ['bracketed_hostport', 'hostport']:
+        m = patterns[matching].match(address)
         if m:
             (address, port) = m.groups()
             port = int(port)
@@ -194,22 +196,20 @@ def parse_address(address, allow_ranges=False):
     # numeric ranges, or a hostname with alphanumeric ranges.
 
     host = None
-    for type in ['ipv4', 'ipv6', 'hostname']:
-        m = patterns[type].match(address)
+    for matching in ['ipv4', 'ipv6', 'hostname']:
+        m = patterns[matching].match(address)
         if m:
             host = address
             continue
 
     # If it isn't any of the above, we don't understand it.
-
     if not host:
-        return (None, None)
-
-    # If we get to this point, we know that any included ranges are valid. If
-    # the caller is prepared to handle them, all is well. Otherwise we treat
-    # it as a parse failure.
+        raise AnsibleError("Not a valid network hostname: %s" % address)
 
+    # If we get to this point, we know that any included ranges are valid.
+    # If the caller is prepared to handle them, all is well.
+    # Otherwise we treat it as a parse failure.
     if not allow_ranges and '[' in host:
-        return (None, None)
+        raise AnsibleParserError("Detected range in host but was asked to ignore ranges")
 
     return (host, port)

lib/ansible/parsing/vault/__init__.py

@@ -22,6 +22,7 @@ import shlex
 import shutil
 import sys
 import tempfile
+import random
 from io import BytesIO
 from subprocess import call
 from ansible.errors import AnsibleError
@@ -70,7 +71,7 @@ try:
 except ImportError:
     pass
 
-from ansible.compat.six import PY3, byte2int
+from ansible.compat.six import PY3
 from ansible.utils.unicode import to_unicode, to_bytes
 
 HAS_ANY_PBKDF2HMAC = HAS_PBKDF2 or HAS_PBKDF2HMAC
@@ -220,21 +221,91 @@ class VaultEditor:
     def __init__(self, password):
         self.vault = VaultLib(password)
 
+    def _shred_file_custom(self, tmp_path):
+        """"Destroy a file, when shred (core-utils) is not available
+
+        Unix `shred' destroys files "so that they can be recovered only with great difficulty with
+        specialised hardware, if at all". It is based on the method from the paper
+        "Secure Deletion of Data from Magnetic and Solid-State Memory",
+        Proceedings of the Sixth USENIX Security Symposium (San Jose, California, July 22-25, 1996).
+
+        We do not go to that length to re-implement shred in Python; instead, overwriting with a block
+        of random data should suffice.
+
+        See https://github.com/ansible/ansible/pull/13700 .
+        """
+
+        file_len = os.path.getsize(tmp_path)
+
+        if file_len > 0: # avoid work when file was empty
+            max_chunk_len = min(1024*1024*2, file_len)
+
+            passes = 3
+            with open(tmp_path,  "wb") as fh:
+                for _ in range(passes):
+                    fh.seek(0,  0)
+                    # get a random chunk of data, each pass with other length
+                    chunk_len = random.randint(max_chunk_len//2, max_chunk_len)
+                    data = os.urandom(chunk_len)
+
+                    for _ in range(0, file_len // chunk_len):
+                        fh.write(data)
+                    fh.write(data[:file_len % chunk_len])
+
+                    assert(fh.tell() == file_len) # FIXME remove this assert once we have unittests to check its accuracy
+                    os.fsync(fh)
+
+
+    def _shred_file(self, tmp_path):
+        """Securely destroy a decrypted file
+
+        Note standard limitations of GNU shred apply (For flash, overwriting would have no effect
+        due to wear leveling; for other storage systems, the async kernel->filesystem->disk calls never
+        guarantee data hits the disk; etc). Furthermore, if your tmp dirs is on tmpfs (ramdisks),
+        it is a non-issue.
+
+        Nevertheless, some form of overwriting the data (instead of just removing the fs index entry) is
+        a good idea. If shred is not available (e.g. on windows, or no core-utils installed), fall back on
+        a custom shredding method.
+        """
+
+        if not os.path.isfile(tmp_path):
+            # file is already gone
+            return
+
+        try:
+            r = call(['shred', tmp_path])
+        except OSError:
+            # shred is not available on this system, or some other error occured.
+            r = 1
+
+        if r != 0:
+            # we could not successfully execute unix shred; therefore, do custom shred.
+            self._shred_file_custom(tmp_path)
+
+        os.remove(tmp_path)
+
     def _edit_file_helper(self, filename, existing_data=None, force_save=False):
 
         # Create a tempfile
         _, tmp_path = tempfile.mkstemp()
 
         if existing_data:
-            self.write_data(existing_data, tmp_path)
+            self.write_data(existing_data, tmp_path, shred=False)
 
         # drop the user into an editor on the tmp file
-        call(self._editor_shell_command(tmp_path))
+        try:
+            call(self._editor_shell_command(tmp_path))
+        except:
+            # whatever happens, destroy the decrypted file
+            self._shred_file(tmp_path)
+            raise
+
         tmpdata = self.read_data(tmp_path)
 
         # Do nothing if the content has not changed
         if existing_data == tmpdata and not force_save:
-            os.remove(tmp_path)
+            self._shred_file(tmp_path)
             return
 
         # encrypt new data and write out to tmp
@@ -258,7 +329,7 @@ class VaultEditor:
 
         ciphertext = self.read_data(filename)
         plaintext = self.vault.decrypt(ciphertext)
-        self.write_data(plaintext, output_file or filename)
+        self.write_data(plaintext, output_file or filename, shred=False)
 
     def create_file(self, filename):
         """ create a new encrypted file """
@@ -323,13 +394,21 @@ class VaultEditor:
 
         return data
 
-    def write_data(self, data, filename):
+    def write_data(self, data, filename, shred=True):
+        """write data to given path
+        
+        if shred==True, make sure that the original data is first shredded so 
+        that is cannot be recovered
+        """
         bytes = to_bytes(data, errors='strict')
         if filename == '-':
             sys.stdout.write(bytes)
         else:
             if os.path.isfile(filename):
-                os.remove(filename)
+                if shred:
+                    self._shred_file(filename)
+                else:
+                    os.remove(filename)
             with open(filename, "wb") as fh:
                 fh.write(bytes)
 
@@ -338,6 +417,7 @@ class VaultEditor:
         # overwrite dest with src
         if os.path.isfile(dest):
             prev = os.stat(dest)
+            # old file 'dest' was encrypted, no need to _shred_file
             os.remove(dest)
         shutil.move(src, dest)
 

lib/ansible/parsing/yaml/dumper.py

@@ -22,7 +22,7 @@ __metaclass__ = type
 import yaml
 from ansible.compat.six import PY3
 
-from ansible.parsing.yaml.objects import AnsibleUnicode
+from ansible.parsing.yaml.objects import AnsibleUnicode, AnsibleSequence, AnsibleMapping
 from ansible.vars.hostvars import HostVars
 
 class AnsibleDumper(yaml.SafeDumper):
@@ -50,3 +50,13 @@ AnsibleDumper.add_representer(
     represent_hostvars,
 )
 
+AnsibleDumper.add_representer(
+    AnsibleSequence,
+    yaml.representer.SafeRepresenter.represent_list,
+)
+
+AnsibleDumper.add_representer(
+    AnsibleMapping,
+    yaml.representer.SafeRepresenter.represent_dict,
+)
+