New release v2.2.0.0-1

* Exception.message gone in 3.x
(cherry picked from commit 4a067c3f50)

CHANGELOG.md

@@ -1,10 +1,12 @@
 Ansible Changes By Release
 ==========================
 
-## 2.2 "The Battle of Evermore" - ACTIVE DEVELOPMENT
+## 2.2 "The Battle of Evermore" - 11-01-2016
 
 ###Major Changes:
 
+* Security fix for CVE-2016-8628 - Command injection by compromised server via fact variables. In some situations, facts returned by modules could overwrite connection-based facts or some other special variables, leading to injected commands running on the Ansible controller as the user running Ansible (or via escalated permissions).
+* Security fix for CVE-2016-8614 - apt_key module not properly validating keys in some situations.
 * Added the `listen` feature for modules. This feature allows tasks to more easily notify multiple handlers, as well as making it easier for handlers from decoupled roles to be notified.
 * Major performance improvements.
 * Added support for binary modules
@@ -33,16 +35,15 @@ Ansible Changes By Release
 * Tech Preview: Work has been done to get Ansible running under Python3.  This work is not complete enough to depend upon in production environments but it is enough to begin testing it.
   * Most of the controller side should now work.  Users should be able to run python3 /usr/bin/ansible and python3 /usr/bin/ansible-playbook and have core features of ansible work.
   * A few of the most essential modules have been audited and are known to work.  Others work out of the box.
-  * We are using unit and integration tests to help us port code and not regress later.  Even if you are not famiriar with python you can still help by contributing integration tests (just ansible roles) that exercise more of the code to make sure it continues to run on both Python2 and Python3.
+  * We are using unit and integration tests to help us port code and not regress later.  Even if you are not familiar with python you can still help by contributing integration tests (just ansible roles) that exercise more of the code to make sure it continues to run on both Python2 and Python3.
   * scp_if_ssh now supports True, False and "smart". "smart" is the default and will retry failed sftp transfers with scp.
 * Network:
-  * Refactored all network modules to remove dulicate code and take advantage of Ansiballz implementation
+  * Refactored all network modules to remove duplicate code and take advantage of Ansiballz implementation
   * All functionality from *_template network modules have been combined into *_config module
   * Network *_command modules not longer allow configuration mode statements
 
 ####New Modules:
 - apache2_mod_proxy
-- archive
 - asa
   * asa_acl
   * asa_command
@@ -283,6 +284,13 @@ Ansible Changes By Release
 * Fix a problem with the pip module updating the python pip package itself.
 * ansible_play_hosts is a new magic variable to provide a list of hosts in scope for the current play. Unlike play_hosts it is not subject to the 'serial' keyword.
 * ansible_play_batch is a new magic variable meant to substitute the current play_hosts.
+* The subversion module from core now marks its password parameter as no_log so
+  the password is obscured when logging.
+* The postgresql_lang and postgresql_ext modules from extras now mark
+  login_password as no_log so the password is obscured when logging.
+* Fix for yum module incorrectly thinking it succeeded in installing packages
+* Make the default ansible_managed template string into a static string since
+  all of the replacable values lead to non-idempotent behaviour.
 
 ###For custom front ends using the API:
 * ansible.parsing.vault:
@@ -292,7 +300,7 @@ Ansible Changes By Release
     Ansible.  The feature it was intended to support has now been implemented
     without using this.
   * VaultAES, the older, insecure encrypted format that debuted in Ansible-1.5
-    and was relaced by VaultAES256 less than a week later, now has a deprecation
+    and was replaced by VaultAES256 less than a week later, now has a deprecation
     warning.  **It will be removed in 2.3**.  In the unlikely event that you
     wrote a vault file in that 1 week window and have never modified the file
     since (ansible-vault automatically re-encrypts the file using VaultAES256
@@ -308,7 +316,6 @@ Ansible Changes By Release
 ###Deprecations
 Notice given that the following will be removed in Ansible 2.4:
 * Modules
-  * asa_template
   * eos_template
   * ios_template
   * iosxr_template

README.md

@@ -1,5 +1,5 @@
 [![PyPI version](https://img.shields.io/pypi/v/ansible.svg)](https://pypi.python.org/pypi/ansible)
-[![Build Status](https://api.shippable.com/projects/573f79d02a8192902e20e34b/badge?branch=devel)](https://app.shippable.com/projects/573f79d02a8192902e20e34b)
+[![Build Status](https://api.shippable.com/projects/573f79d02a8192902e20e34b/badge?branch=stable-2.2)](https://app.shippable.com/projects/573f79d02a8192902e20e34b)
 
 
 Ansible
@@ -7,13 +7,13 @@ Ansible
 
 Ansible is a radically simple IT automation system.  It handles configuration-management, application deployment, cloud provisioning, ad-hoc task-execution, and multinode orchestration - including trivializing things like zero downtime rolling updates with load balancers.
 
-Read the documentation and more at http://ansible.com/
+Read the documentation and more at https://ansible.com/
 
 Many users run straight from the development branch (it's generally fine to do so), but you might also wish to consume a release.  
 
-You can find instructions [here](http://docs.ansible.com/intro_getting_started.html) for a variety of platforms.  If you decide to go with the development branch, be sure to run `git submodule update --init --recursive` after doing a checkout. 
+You can find instructions [here](https://docs.ansible.com/intro_getting_started.html) for a variety of platforms.  If you decide to go with the development branch, be sure to run `git submodule update --init --recursive` after doing a checkout. 
 
-If you want to download a tarball of a release, go to [releases.ansible.com](http://releases.ansible.com/ansible), though most users use `yum` (using the EPEL instructions linked above), `apt` (using the PPA instructions linked above), or `pip install ansible`.
+If you want to download a tarball of a release, go to [releases.ansible.com](https://releases.ansible.com/ansible), though most users use `yum` (using the EPEL instructions linked above), `apt` (using the PPA instructions linked above), or `pip install ansible`.
 
 Design Principles
 =================
@@ -31,11 +31,11 @@ Design Principles
 Get Involved
 ============
 
-   * Read [Community Information](http://docs.ansible.com/community.html) for all kinds of ways to contribute to and interact with the project, including mailing list information and how to submit bug reports and code to Ansible.  
+   * Read [Community Information](https://docs.ansible.com/community.html) for all kinds of ways to contribute to and interact with the project, including mailing list information and how to submit bug reports and code to Ansible.  
    * All code submissions are done through pull requests.  Take care to make sure no merge commits are in the submission, and use `git rebase` vs `git merge` for this reason.  If submitting a large code change (other than modules), it's probably a good idea to join ansible-devel and talk about what you would like to do or add first and to avoid duplicate efforts.  This not only helps everyone know what's going on, it also helps save time and effort if we decide some changes are needed.
-   * Users list: [ansible-project](http://groups.google.com/group/ansible-project)
-   * Development list: [ansible-devel](http://groups.google.com/group/ansible-devel)
-   * Announcement list: [ansible-announce](http://groups.google.com/group/ansible-announce) - read only
+   * Users list: [ansible-project](https://groups.google.com/group/ansible-project)
+   * Development list: [ansible-devel](https://groups.google.com/group/ansible-devel)
+   * Announcement list: [ansible-announce](https://groups.google.com/group/ansible-announce) - read only
    * irc.freenode.net: #ansible
 
 Branch Info
@@ -45,13 +45,13 @@ Branch Info
    * The devel branch corresponds to the release actively under development.
    * As of 1.8, modules are kept in different repos, you'll want to follow [core](https://github.com/ansible/ansible-modules-core) and [extras](https://github.com/ansible/ansible-modules-extras)
    * Various release-X.Y branches exist for previous releases.
-   * We'd love to have your contributions, read [Community Information](http://docs.ansible.com/community.html) for notes on how to get started.
+   * We'd love to have your contributions, read [Community Information](https://docs.ansible.com/community.html) for notes on how to get started.
 
 Authors
 =======
 
 Ansible was created by [Michael DeHaan](https://github.com/mpdehaan) (michael.dehaan/gmail/com) and has contributions from over 1000 users (and growing).  Thanks everyone!
 
-Ansible is sponsored by [Ansible, Inc](http://ansible.com)
+Ansible is sponsored by [Ansible, Inc](https://ansible.com)
 
 

RELEASES.txt

@@ -1,74 +1,74 @@
 Ansible Releases at a Glance
 ============================
 
-Active Development
-++++++++++++++++++
-
-2.2    TBD - in progress
-
-Released
-++++++++
-
-2.1.0  "The Song Remains the Same" in progress
-2.0.2  "Over the Hills and Far Away" 04-19-2016
-2.0.1  "Over the Hills and Far Away" 02-24-2016
-2.0.0  "Over the Hills and Far Away" 01-12-2016
-1.9.6  "Dancing In the Streets" 04-15-2016
-1.9.5  "Dancing In the Streets" 03-21-2016
-1.9.4  "Dancing In the Streets" 10-09-2015
-1.9.3  "Dancing In the Streets" 09-03-2015
-1.9.2  "Dancing In the Streets" 06-24-2015
-1.9.1  "Dancing In the Streets" 04-27-2015
-1.9.0  "Dancing In the Streets" 03-25-2015
-1.8.4  "You Really Got Me" ---- 02-19-2015
-1.8.3  "You Really Got Me" ---- 02-17-2015
-1.8.2  "You Really Got Me" ---- 12-04-2014
-1.8.1  "You Really Got Me" ---- 11-26-2014
-1.7.2  "Summer Nights" -------- 09-24-2014
-1.7.1  "Summer Nights" -------- 08-14-2014
-1.7    "Summer Nights" -------- 08-06-2014
-1.6.10 "The Cradle Will Rock" - 07-25-2014
-1.6.9  "The Cradle Will Rock" - 07-24-2014
-1.6.8  "The Cradle Will Rock" - 07-22-2014
-1.6.7  "The Cradle Will Rock" - 07-21-2014
-1.6.6  "The Cradle Will Rock" - 07-01-2014
-1.6.5  "The Cradle Will Rock" - 06-25-2014
-1.6.4  "The Cradle Will Rock" - 06-25-2014
-1.6.3  "The Cradle Will Rock" - 06-09-2014
-1.6.2  "The Cradle Will Rock" - 05-23-2014
-1.6.1  "The Cradle Will Rock" - 05-07-2014
-1.6    "The Cradle Will Rock" - 05-05-2014 
-1.5.5  "Love Walks In" -------- 04-18-2014
-1.5.4  "Love Walks In" -------- 04-01-2014
-1.5.3  "Love Walks In" -------- 03-13-2014
-1.5.2  "Love Walks In" -------- 03-11-2014
-1.5.1  "Love Walks In" -------- 03-10-2014
-1.5    "Love Walks In" -------- 02-28-2014
-1.4.5  "Could This Be Magic?" - 02-12-2014
-1.4.4  "Could This Be Magic?" - 01-06-2014
-1.4.3  "Could This Be Magic?" - 12-20-2013
-1.4.2  "Could This Be Magic?" - 12-18-2013
-1.4.1  "Could This Be Magic?" - 11-27-2013
-1.4    "Could This Be Magic?" - 11-21-2013
-1.3.4  "Top of the World" ----- 10-29-2013
-1.3.3  "Top of the World" ----- 10-09-2013
-1.3.2  "Top of the World" ----- 09-19-2013
-1.3.1  "Top of the World" ----- 09-16-2013
-1.3    "Top of the World" ----- 09-13-2013
-1.2.3  "Hear About It Later" -- 08-21-2013
-1.2.2  "Hear About It Later" -- 07-05-2013
-1.2.1  "Hear About It Later" -- 07-04-2013
-1.2    "Right Now" ------------ 06-10-2013
-1.1    "Mean Street" ---------- 04-02-2013
-1.0    "Eruption" ------------- 02-01-2013
-0.9    "Dreams" --------------- 11-30-2012
-0.8    "Cathedral" ------------ 10-19-2012
-0.7    "Panama" --------------- 09-06-2012
-0.6    "Cabo" ----------------- 08-06-2012
-0.5    "Amsterdam" ------------ 07-04-2012
-0.4    "Unchained" ------------ 05-23-2012
-0.3    "Baluchitherium" ------- 04-23-2012
-0.0.2  Untitled
-0.0.1  Untitled
-
+VERSION  RELEASE     CODE NAME
+++++++++++++++++++++++++++++++
 
+2.2.0    11-01-2016  "The Battle of Evermore"
+2.1.3    TBD         "The Song Remains the Same"
+2.1.2    09-29-2016  "The Song Remains the Same"
+2.1.1    07-28-2016  "The Song Remains the Same"
+2.1.0    05-25-2016  "The Song Remains the Same"
+2.0.2    04-19-2016  "Over the Hills and Far Away"
+2.0.1    02-24-2016  "Over the Hills and Far Away"
+2.0.0    01-12-2016  "Over the Hills and Far Away"
+1.9.6    04-15-2016  "Dancing In the Streets"
+1.9.5    03-21-2016  "Dancing In the Streets"
+1.9.4    10-09-2015  "Dancing In the Streets"
+1.9.3    09-03-2015  "Dancing In the Streets"
+1.9.2    06-24-2015  "Dancing In the Streets"
+1.9.1    04-27-2015  "Dancing In the Streets"
+1.9.0    03-25-2015  "Dancing In the Streets"
+1.8.4    02-19-2015  "You Really Got Me"
+1.8.3    02-17-2015  "You Really Got Me"
+1.8.2    12-04-2014  "You Really Got Me"
+1.8.1    11-26-2014  "You Really Got Me"
+1.8.0    ?           "You Really Got Me"
+1.7.2    09-24-2014  "Summer Nights"
+1.7.1    08-14-2014  "Summer Nights"
+1.7.0    08-06-2014  "Summer Nights"
+1.6.10   07-25-2014  "The Cradle Will Rock"
+1.6.9    07-24-2014  "The Cradle Will Rock"
+1.6.8    07-22-2014  "The Cradle Will Rock"
+1.6.7    07-21-2014  "The Cradle Will Rock"
+1.6.6    07-01-2014  "The Cradle Will Rock"
+1.6.5    06-25-2014  "The Cradle Will Rock"
+1.6.4    06-25-2014  "The Cradle Will Rock"
+1.6.3    06-09-2014  "The Cradle Will Rock"
+1.6.2    05-23-2014  "The Cradle Will Rock"
+1.6.1    05-07-2014  "The Cradle Will Rock"
+1.6.0    05-05-2014  "The Cradle Will Rock"
+1.5.5    04-18-2014  "Love Walks In"
+1.5.4    04-01-2014  "Love Walks In"
+1.5.3    03-13-2014  "Love Walks In"
+1.5.2    03-11-2014  "Love Walks In"
+1.5.1    03-10-2014  "Love Walks In"
+1.5.0    02-28-2014  "Love Walks In"
+1.4.5    02-12-2014  "Could This Be Magic?"
+1.4.4    01-06-2014  "Could This Be Magic?"
+1.4.3    12-20-2013  "Could This Be Magic?"
+1.4.2    12-18-2013  "Could This Be Magic?"
+1.4.1    11-27-2013  "Could This Be Magic?"
+1.4.0    11-21-2013  "Could This Be Magic?"
+1.3.4    10-29-2013  "Top of the World"
+1.3.3    10-09-2013  "Top of the World"
+1.3.2    09-19-2013  "Top of the World"
+1.3.1    09-16-2013  "Top of the World"
+1.3.0    09-13-2013  "Top of the World"
+1.2.3    08-21-2013  "Right Now"
+1.2.2    07-05-2013  "Right Now"
+1.2.1    07-04-2013  "Right Now"
+1.2.0    06-10-2013  "Right Now"
+1.1.0    04-02-2013  "Mean Street"
+1.0.0    02-01-2013  "Eruption"
+0.9.0    11-30-2012  "Dreams"
+0.8.0    10-19-2012  "Cathedral"
+0.7.0    09-06-2012  "Panama"
+0.6.0    08-06-2012  "Cabo"
+0.5.0    07-04-2012  "Amsterdam"
+0.4.0    05-23-2012  "Unchained"
+0.3.0    04-23-2012  "Baluchitherium"
+0.2.0    ?           "Untitled"
+0.1.0    ?           "Untitled"
+0.0.2    ?           "Untitled"
+0.0.1    ?           "Untitled"

VERSION

@@ -1 +1 @@
-2.2.0 0.0.devel
+2.2.0.0 1

bin/ansible

@@ -126,9 +126,11 @@ if __name__ == '__main__':
         have_cli_options = cli is not None and cli.options is not None
         display.error("Unexpected Exception: %s" % to_text(e), wrap_text=False)
         if not have_cli_options or have_cli_options and cli.options.verbosity > 2:
-            display.display(u"the full traceback was:\n\n%s" % to_text(traceback.format_exc()))
+            log_only = False
         else:
             display.display("to see the full traceback, use -vvv")
+            log_only = True 
+        display.display(u"the full traceback was:\n\n%s" % to_text(traceback.format_exc()), log_only=log_only)
         exit_code = 250
     finally:
         # Remove ansible tempdir

contrib/inventory/digital_ocean.py

@@ -260,7 +260,7 @@ or environment variables (DO_API_TOKEN)\n''')
 
         # Private IP Address
         if config.has_option('digital_ocean', 'use_private_network'):
-            self.use_private_network = config.get('digital_ocean', 'use_private_network')
+            self.use_private_network = config.getboolean('digital_ocean', 'use_private_network')
 
         # Group variables
         if config.has_option('digital_ocean', 'group_variables'):

contrib/inventory/ovirt.py

@@ -211,7 +211,7 @@ class OVirtInventory(object):
             'ovirt_uuid': inst.get_id(),
             'ovirt_id': inst.get_id(),
             'ovirt_image': inst.get_os().get_type(),
-            'ovirt_machine_type': inst.get_instance_type(),
+            'ovirt_machine_type': self.get_machine_type(inst),
             'ovirt_ips': ips,
             'ovirt_name': inst.get_name(),
             'ovirt_description': inst.get_description(),
@@ -230,6 +230,11 @@ class OVirtInventory(object):
         """
         return [x.get_name() for x in inst.get_tags().list()]
 
+    def get_machine_type(self,inst):
+        inst_type = inst.get_instance_type()
+        if inst_type:
+            return self.driver.instancetypes.get(id=inst_type.id).name
+
     # noinspection PyBroadException,PyUnusedLocal
     def get_instance(self, instance_name):
         """Gets details about a specific instance """

docsite/rst/community.rst

@@ -251,7 +251,7 @@ Tower Support Questions
 
 Ansible `Tower <http://ansible.com/tower>`_ is a UI, Server, and REST endpoint for Ansible, produced by Ansible, Inc.
 
-If you have a question about Tower, email `support@ansible.com <mailto:support@ansible.com>`_ rather than using the IRC
+If you have a question about Tower, visit `support.ansible.com <https://support.ansible.com/>`_ rather than using the IRC
 channel or the general project mailing list.
 
 IRC Channel

docsite/rst/intro_configuration.rst

@@ -443,6 +443,20 @@ implications and wish to disable it, you may do so here by setting the value to
 
     host_key_checking=True
 
+.. _internal_poll_interval:
+
+internal_poll_interval
+======================
+
+.. versionadded:: 2.2
+
+This sets the interval (in seconds) of Ansible internal processes polling each other.
+Lower values improve performance with large playbooks at the expense of extra CPU load.
+Higher values are more suitable for Ansible usage in automation scenarios, when UI responsiveness is not required but CPU usage might be a concern.
+Default corresponds to the value hardcoded in Ansible ≤ 2.1::
+
+    internal_poll_interval=0.001
+
 .. _inventory_file:
 
 inventory

examples/ansible.cfg

@@ -70,6 +70,9 @@
 #task_includes_static = True
 #handler_includes_static = True
 
+# Controls if a missing handler for a notification event is an error or a warning
+#error_on_missing_handler = True
+
 # change this for alternative sudo implementations
 #sudo_exe = sudo
 
@@ -121,8 +124,9 @@
 # templates indicates to users editing templates files will be replaced.
 # replacing {file}, {host} and {uid} and strftime codes with proper values.
 #ansible_managed = Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}
-# This short version is better used in templates as it won't flag the file as changed every run.
-#ansible_managed = Ansible managed: {file} on {host}
+# {file}, {host}, {uid}, and the timestamp can all interfere with idempotence
+# in some situations so the default is a static string:
+#ansible_managed = Ansible managed
 
 # by default, ansible-playbook will display "Skipping [host]" if it determines a task
 # should not be run on a host.  Set this to "False" if you don't want to see these "Skipping"

hacking/env-setup.fish

@@ -4,16 +4,16 @@
 set HACKING_DIR (dirname (status -f))
 set FULL_PATH (python -c "import os; print(os.path.realpath('$HACKING_DIR'))")
 set ANSIBLE_HOME (dirname $FULL_PATH)
-set PREFIX_PYTHONPATH $ANSIBLE_HOME/lib 
-set PREFIX_PATH $ANSIBLE_HOME/bin 
+set PREFIX_PYTHONPATH $ANSIBLE_HOME/lib
+set PREFIX_PATH $ANSIBLE_HOME/bin
 set PREFIX_MANPATH $ANSIBLE_HOME/docs/man
 
 # set quiet flag
-if set -q argv
+if test (count $argv) -ge 1
     switch $argv
-    case '-q' '--quiet'
-        set QUIET "true"
-    case '*'
+        case '-q' '--quiet'
+            set QUIET "true"
+        case '*'
     end
 end
 
@@ -49,15 +49,14 @@ set -gx ANSIBLE_LIBRARY $ANSIBLE_HOME/library
 
 # Generate egg_info so that pkg_resources works
 pushd $ANSIBLE_HOME
+if test -e $PREFIX_PYTHONPATH/ansible*.egg-info
+    rm -r $PREFIX_PYTHONPATH/ansible*.egg-info
+end
 if [ $QUIET ]
     python setup.py -q egg_info
 else
     python setup.py egg_info
 end
-if test -e $PREFIX_PYTHONPATH/ansible*.egg-info
-    rm -r $PREFIX_PYTHONPATH/ansible*.egg-info
-end
-mv ansible*egg-info $PREFIX_PYTHONPATH
 find . -type f -name "*.pyc" -delete
 popd
 

hacking/test-module

@@ -105,6 +105,10 @@ def boilerplate_module(modfile, args, interpreter, check, destfile):
     #included_boilerplate = module_data.find(module_common.REPLACER) != -1 or module_data.find("import ansible.module_utils") != -1
 
     complex_args = {}
+
+    # default selinux fs list is pass in as _ansible_selinux_special_fs arg
+    complex_args['_ansible_selinux_special_fs'] = C.DEFAULT_SELINUX_SPECIAL_FS
+
     if args.startswith("@"):
         # Argument is a YAML file (JSON is a subset of YAML)
         complex_args = utils_vars.combine_vars(complex_args, loader.load_from_file(args[1:]))

lib/ansible/cli/doc.py

@@ -136,7 +136,8 @@ class DocCLI(CLI):
                 display.vvv(traceback.print_exc())
                 raise AnsibleError("module %s missing documentation (or could not parse documentation): %s\n" % (module, str(e)))
 
-        self.pager(text)
+        if text:
+            self.pager(text)
         return 0
 
     def find_modules(self, path):

lib/ansible/constants.py

@@ -146,7 +146,7 @@ DEFAULT_COW_WHITELIST = ['bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'd
 DEFAULTS='defaults'
 
 # FIXME: add deprecation warning when these get set
-#### DEPRECATED VARS #### 
+#### DEPRECATED VARS ####
 # use more sanely named 'inventory'
 DEPRECATED_HOST_LIST  = get_config(p, DEFAULTS, 'hostfile', 'ANSIBLE_HOSTS', '/etc/ansible/hosts', ispath=True)
 # this is not used since 0.5 but people might still have in config
@@ -176,7 +176,7 @@ DEFAULT_VAULT_PASSWORD_FILE = get_config(p, DEFAULTS, 'vault_password_file', 'AN
 DEFAULT_TRANSPORT         = get_config(p, DEFAULTS, 'transport',        'ANSIBLE_TRANSPORT',        'smart')
 DEFAULT_SCP_IF_SSH        = get_config(p, 'ssh_connection', 'scp_if_ssh',       'ANSIBLE_SCP_IF_SSH',       'smart')
 DEFAULT_SFTP_BATCH_MODE   = get_config(p, 'ssh_connection', 'sftp_batch_mode', 'ANSIBLE_SFTP_BATCH_MODE', True, boolean=True)
-DEFAULT_MANAGED_STR       = get_config(p, DEFAULTS, 'ansible_managed',  None,           'Ansible managed: {file} modified on %Y-%m-%d %H:%M:%S by {uid} on {host}')
+DEFAULT_MANAGED_STR       = get_config(p, DEFAULTS, 'ansible_managed',  None,           'Ansible managed')
 DEFAULT_SYSLOG_FACILITY   = get_config(p, DEFAULTS, 'syslog_facility',  'ANSIBLE_SYSLOG_FACILITY', 'LOG_USER')
 DEFAULT_KEEP_REMOTE_FILES = get_config(p, DEFAULTS, 'keep_remote_files', 'ANSIBLE_KEEP_REMOTE_FILES', False, boolean=True)
 DEFAULT_HASH_BEHAVIOUR    = get_config(p, DEFAULTS, 'hash_behaviour', 'ANSIBLE_HASH_BEHAVIOUR', 'replace')
@@ -190,6 +190,8 @@ DEFAULT_LOG_PATH          = get_config(p, DEFAULTS, 'log_path',           'ANSIB
 DEFAULT_FORCE_HANDLERS    = get_config(p, DEFAULTS, 'force_handlers', 'ANSIBLE_FORCE_HANDLERS', False, boolean=True)
 DEFAULT_INVENTORY_IGNORE  = get_config(p, DEFAULTS, 'inventory_ignore_extensions', 'ANSIBLE_INVENTORY_IGNORE', ["~", ".orig", ".bak", ".ini", ".cfg", ".retry", ".pyc", ".pyo"], islist=True)
 DEFAULT_VAR_COMPRESSION_LEVEL = get_config(p, DEFAULTS, 'var_compression_level', 'ANSIBLE_VAR_COMPRESSION_LEVEL', 0, integer=True)
+DEFAULT_INTERNAL_POLL_INTERVAL = get_config(p, DEFAULTS, 'internal_poll_interval', None, 0.001, floating=True)
+ERROR_ON_MISSING_HANDLER  = get_config(p, DEFAULTS, 'error_on_missing_handler', 'ANSIBLE_ERROR_ON_MISSING_HANDLER', True, boolean=True)
 
 # static includes
 DEFAULT_TASK_INCLUDES_STATIC    = get_config(p, DEFAULTS, 'task_includes_static', 'ANSIBLE_TASK_INCLUDES_STATIC', False, boolean=True)

lib/ansible/executor/action_write_locks.py

@@ -1,4 +1,4 @@
-# (c) 2016 - Red Hat, Inc. <support@ansible.com>
+# (c) 2016 - Red Hat, Inc. <info@ansible.com>
 #
 # This file is part of Ansible
 #

lib/ansible/executor/module_common.py

@@ -212,12 +212,12 @@ def debug(command, zipped_mod, json_params):
                 directory = os.path.dirname(dest_filename)
                 if not os.path.exists(directory):
                     os.makedirs(directory)
-                f = open(dest_filename, 'w')
+                f = open(dest_filename, 'wb')
                 f.write(z.read(filename))
                 f.close()
 
         # write the args file
-        f = open(args_path, 'w')
+        f = open(args_path, 'wb')
         f.write(json_params)
         f.close()
 

lib/ansible/executor/play_iterator.py

@@ -28,6 +28,7 @@ from ansible import constants as C
 from ansible.errors import AnsibleError
 from ansible.playbook.block import Block
 from ansible.playbook.task import Task
+from ansible.playbook.role_include import IncludeRole
 
 from ansible.utils.boolean import boolean
 
@@ -80,7 +81,7 @@ class HostState:
                         ret.append(states[i])
                 return "|".join(ret)
 
-        return "HOST STATE: block=%d, task=%d, rescue=%d, always=%d, role=%s, run_state=%s, fail_state=%s, pending_setup=%s, tasks child state? %s, rescue child state? %s, always child state? %s, did start at task? %s" % (
+        return "HOST STATE: block=%d, task=%d, rescue=%d, always=%d, role=%s, run_state=%s, fail_state=%s, pending_setup=%s, tasks child state? (%s), rescue child state? (%s), always child state? (%s), did start at task? %s" % (
             self.cur_block,
             self.cur_regular_task,
             self.cur_rescue_task,
@@ -265,9 +266,18 @@ class PlayIterator:
             else:
                 return old_s.cur_dep_chain != task.get_dep_chain()
 
+        def _role_is_child(r):
+            parent = task._parent
+            while parent:
+                if hasattr(parent, '_role') and parent._role == r and isinstance(parent, IncludeRole):
+                    return True
+                parent = parent._parent
+            return False
+
         if task and task._role:
             # if we had a current role, mark that role as completed
-            if s.cur_role and _roles_are_different(task._role, s.cur_role) and host.name in s.cur_role._had_task_run and not peek:
+            if s.cur_role and _roles_are_different(task._role, s.cur_role) and host.name in s.cur_role._had_task_run and \
+               not _role_is_child(s.cur_role) and not peek:
                 s.cur_role._completed[host.name] = True
             s.cur_role = task._role
             s.cur_dep_chain = task.get_dep_chain()
@@ -499,6 +509,10 @@ class PlayIterator:
     def _check_failed_state(self, state):
         if state is None:
             return False
+        elif state.run_state == self.ITERATING_RESCUE and self._check_failed_state(state.rescue_child_state):
+            return True
+        elif state.run_state == self.ITERATING_ALWAYS and self._check_failed_state(state.always_child_state):
+            return True
         elif state.fail_state != self.FAILED_NONE:
             if state.run_state == self.ITERATING_RESCUE and state.fail_state&self.FAILED_RESCUE == 0:
                 return False
@@ -512,10 +526,6 @@ class PlayIterator:
                 return False
             else:
                 return True
-        elif state.run_state == self.ITERATING_RESCUE and self._check_failed_state(state.rescue_child_state):
-            return True
-        elif state.run_state == self.ITERATING_ALWAYS and self._check_failed_state(state.always_child_state):
-            return True
         return False
 
     def is_failed(self, host):

lib/ansible/executor/task_executor.py

@@ -85,9 +85,6 @@ class TaskExecutor:
         display.debug("in run()")
 
         try:
-            # get search path for this task to pass to lookup plugins
-            self._job_vars['ansible_search_path'] = self._task.get_search_path()
-
             items = self._get_loop_items()
             if items is not None:
                 if len(items) > 0:
@@ -173,6 +170,10 @@ class TaskExecutor:
                 old_vars[k] = self._job_vars[k]
             self._job_vars[k] = play_context_vars[k]
 
+        # get search path for this task to pass to lookup plugins
+        self._job_vars['ansible_search_path'] = self._task.get_search_path()
+
+
         templar = Templar(loader=self._loader, shared_loader_obj=self._shared_loader_obj, variables=self._job_vars)
         items = None
         if self._task.loop:
@@ -212,6 +213,11 @@ class TaskExecutor:
             for idx, item in enumerate(items):
                 if item is not None and not isinstance(item, UnsafeProxy):
                     items[idx] = UnsafeProxy(item)
+
+        # ensure basedir is always in (dwim already searches here but we need to display it)
+        if self._loader.get_basedir() not in self._job_vars['ansible_search_path']:
+            self._job_vars['ansible_search_path'].append(self._loader.get_basedir())
+
         return items
 
     def _run_loop(self, items):
@@ -416,14 +422,10 @@ class TaskExecutor:
             include_file = templar.template(include_file)
             return dict(include=include_file, include_variables=include_variables)
 
-        # TODO: not needed?
         # if this task is a IncludeRole, we just return now with a success code so the main thread can expand the task list for the given host
         elif self._task.action == 'include_role':
             include_variables = self._task.args.copy()
-            role = templar.template(self._task._role_name)
-            if not role:
-                return dict(failed=True, msg="No role was specified to include")
-            return dict(include_role=role, include_variables=include_variables)
+            return dict(include_role=self._task, include_variables=include_variables)
 
         # Now we do final validation on the task, which sets all fields to their final values.
         self._task.post_validate(templar=templar)
@@ -499,7 +501,7 @@ class TaskExecutor:
                 vars_copy[self._task.register] = wrap_var(result.copy())
 
             if self._task.async > 0:
-                if self._task.poll > 0:
+                if self._task.poll > 0 and not result.get('skipped'):
                     result = self._poll_async_result(result=result, templar=templar, task_vars=vars_copy)
 
                 # ensure no log is preserved

lib/ansible/executor/task_queue_manager.py

@@ -222,7 +222,7 @@ class TaskQueueManager:
         )
 
         # Fork # of forks, # of hosts or serial, whichever is lowest
-        num_hosts = len(self._inventory.get_hosts(new_play.hosts))
+        num_hosts = len(self._inventory.get_hosts(new_play.hosts, ignore_restrictions=True))
 
         max_serial = 0
         if new_play.serial:

lib/ansible/executor/task_result.py

@@ -62,11 +62,13 @@ class TaskResult:
         return self._check_key('unreachable')
 
     def _check_key(self, key):
-        if self._result.get('results', []):
+        '''get a specific key from the result or it's items'''
+
+        if isinstance(self._result, dict) and key in self._result:
+            return self._result.get(key, False)
+        else:
             flag = False
             for res in self._result.get('results', []):
                 if isinstance(res, dict):
                     flag |= res.get(key, False)
             return flag
-        else:
-            return self._result.get(key, False)

lib/ansible/inventory/dir.py

@@ -25,6 +25,7 @@ import os
 from ansible import constants as C
 from ansible.errors import AnsibleError
 from ansible.utils.vars import combine_vars
+from ansible.module_utils._text import to_native
 
 #FIXME: make into plugins
 from ansible.inventory.ini import InventoryParser as InventoryINIParser
@@ -44,11 +45,10 @@ def get_file_parser(hostsfile, groups, loader):
     parser = None
 
     try:
-        inv_file = open(hostsfile)
-        first_line = inv_file.readlines()[0]
-        inv_file.close()
-        if first_line.startswith('#!'):
-            shebang_present = True
+        with open(hostsfile, 'rb') as inv_file:
+            initial_chars = inv_file.read(2)
+            if initial_chars.startswith(b'#!'):
+                shebang_present = True
     except:
         pass
 
@@ -59,7 +59,7 @@ def get_file_parser(hostsfile, groups, loader):
             parser = InventoryScript(loader=loader, groups=groups, filename=hostsfile)
             processed = True
         except Exception as e:
-            myerr.append(str(e))
+            myerr.append('Attempted to execute "%s" as inventory script: %s' % (hostsfile, to_native(e)))
     elif shebang_present:
 
         myerr.append("The inventory file \'%s\' looks like it should be an executable inventory script, but is not marked executable. Perhaps you want to correct this with `chmod +x %s`?" % (hostsfile, hostsfile))
@@ -70,7 +70,7 @@ def get_file_parser(hostsfile, groups, loader):
             parser = InventoryYAMLParser(loader=loader, groups=groups, filename=hostsfile)
             processed = True
         except Exception as e:
-            myerr.append(str(e))
+            myerr.append('Attempted to read "%s" as YAML: %s' % (to_native(hostsfile), to_native(e)))
 
     # ini
     if not processed and not shebang_present:
@@ -78,7 +78,7 @@ def get_file_parser(hostsfile, groups, loader):
             parser = InventoryINIParser(loader=loader, groups=groups, filename=hostsfile)
             processed = True
         except Exception as e:
-            myerr.append(str(e))
+            myerr.append('Attempted to read "%s" as ini file: %s ' % (to_native(hostsfile), to_native(e)))
 
     if not processed and myerr:
         raise AnsibleError('\n'.join(myerr))

lib/ansible/inventory/ini.py

@@ -40,7 +40,6 @@ class InventoryParser(object):
     """
 
     def __init__(self, loader, groups, filename=C.DEFAULT_HOST_LIST):
-        self._loader = loader
         self.filename = filename
 
         # Start with an empty host list and whatever groups we're passed in
@@ -53,12 +52,17 @@ class InventoryParser(object):
         # Read in the hosts, groups, and variables defined in the
         # inventory file.
 
-        if loader:
-            (data, private) = loader._get_file_contents(filename)
-        else:
-            with open(filename) as fh:
-                data = to_text(fh.read())
-        data = data.split('\n')
+        with open(filename, 'rb') as fh:
+            data = fh.read()
+            try:
+                # Faster to do to_text once on a long string than many
+                # times on smaller strings
+                data = to_text(data, errors='surrogate_or_strict')
+                data = [line for line in data.splitlines() if not (line.startswith(u';') or line.startswith(u'#'))]
+            except UnicodeError:
+                # Skip comment lines here to avoid potential undecodable
+                # errors in comments: https://github.com/ansible/ansible/issues/17593
+                data = [to_text(line, errors='surrogate_or_strict') for line in data.splitlines() if not (line.startswith(b';') or line.startswith(b'#'))]
 
         self._parse(data)
 
@@ -88,8 +92,8 @@ class InventoryParser(object):
 
             line = line.strip()
 
-            # Skip empty lines and comments
-            if line == '' or line.startswith(";") or line.startswith("#"):
+            # Skip empty lines
+            if not line:
                 continue
 
             # Is this a [section] header? That tells us what group we're parsing

lib/ansible/module_utils/_text.py

@@ -173,12 +173,6 @@ def to_text(obj, encoding='utf-8', errors=None, nonstring='simplerepr'):
         else:
             errors = 'strict'
 
-    if errors is None:
-        if PY3:
-            errors = 'surrogateescape'
-        else:
-            errors = 'replace'
-
     if isinstance(obj, binary_type):
         return obj.decode(encoding, errors)
 

lib/ansible/module_utils/asa.py

@@ -35,8 +35,6 @@ from ansible.module_utils.network import to_list
 from ansible.module_utils.shell import CliBase
 from ansible.module_utils.netcli import Command
 
-add_argument('show_command', dict(default='show running-config',
-             choices=['show running-config', 'more system:running-config']))
 add_argument('context', dict(required=False))
 
 
@@ -88,10 +86,16 @@ class Cli(CliBase):
         responses = self.execute(cmds)
         return responses[1:]
 
-    def get_config(self, include_defaults=False):
+    def get_config(self, include=None):
+        if include not in [None, 'defaults', 'passwords']:
+            raise ValueError('include must be one of None, defaults, passwords')
         cmd = 'show running-config'
-        if include_defaults:
-            cmd += ' all'
+        if include == 'passwords':
+            cmd = 'more system:running-config'
+        elif include == 'defaults':
+            cmd = 'show running-config all'
+        else:
+            cmd = 'show running-config'
         return self.run_commands(cmd)[0]
 
     def load_config(self, commands):

lib/ansible/module_utils/azure_rm_common.py

@@ -112,14 +112,6 @@ AZURE_EXPECTED_VERSIONS = dict(
 
 AZURE_MIN_RELEASE = '2.0.0rc5'
 
-
-def check_client_version(client_name, client_version, expected_version):
-    # Pinning Azure modules to 2.0.0rc5.
-    if LooseVersion(client_version) != LooseVersion(expected_version):
-            self.fail("Installed {0} client version is {1}. The supported version is {2}. Try "
-                      "`pip install azure=={3}`".format(client_name, client_version, expected_version,
-                                                        AZURE_MIN_RELEASE))
-
 class AzureRMModuleBase(object):
 
     def __init__(self, derived_arg_spec, bypass_checks=False, no_log=False,
@@ -192,6 +184,13 @@ class AzureRMModuleBase(object):
         res = self.exec_module(**self.module.params)
         self.module.exit_json(**res)
 
+    def check_client_version(self, client_name, client_version, expected_version):
+        # Pinning Azure modules to 2.0.0rc5.
+        if LooseVersion(client_version) < LooseVersion(expected_version):
+            self.fail("Installed {0} client version is {1}. The supported version is {2}. Try "
+                      "`pip install azure>={3}`".format(client_name, client_version, expected_version,
+                                                        AZURE_MIN_RELEASE))
+
     def exec_module(self, **kwargs):
         self.fail("Error: {0} failed to implement exec_module method.".format(self.__class__.__name__))
 
@@ -574,7 +573,7 @@ class AzureRMModuleBase(object):
     def storage_client(self):
         self.log('Getting storage client...')
         if not self._storage_client:
-            check_client_version('storage', storage_client_version, AZURE_EXPECTED_VERSIONS['storage_client_version'])
+            self.check_client_version('storage', storage_client_version, AZURE_EXPECTED_VERSIONS['storage_client_version'])
             self._storage_client = StorageManagementClient(self.azure_credentials, self.subscription_id)
             self._register('Microsoft.Storage')
         return self._storage_client
@@ -583,7 +582,7 @@ class AzureRMModuleBase(object):
     def network_client(self):
         self.log('Getting network client')
         if not self._network_client:
-            check_client_version('network', network_client_version, AZURE_EXPECTED_VERSIONS['network_client_version'])
+            self.check_client_version('network', network_client_version, AZURE_EXPECTED_VERSIONS['network_client_version'])
             self._network_client = NetworkManagementClient(self.azure_credentials, self.subscription_id)
             self._register('Microsoft.Network')
         return self._network_client
@@ -592,7 +591,7 @@ class AzureRMModuleBase(object):
     def rm_client(self):
         self.log('Getting resource manager client')
         if not self._resource_client:
-            check_client_version('resource', resource_client_version, AZURE_EXPECTED_VERSIONS['resource_client_version'])
+            self.check_client_version('resource', resource_client_version, AZURE_EXPECTED_VERSIONS['resource_client_version'])
             self._resource_client = ResourceManagementClient(self.azure_credentials, self.subscription_id)
         return self._resource_client
 
@@ -600,7 +599,7 @@ class AzureRMModuleBase(object):
     def compute_client(self):
         self.log('Getting compute client')
         if not self._compute_client:
-            check_client_version('compute', compute_client_version, AZURE_EXPECTED_VERSIONS['compute_client_version'])
+            self.check_client_version('compute', compute_client_version, AZURE_EXPECTED_VERSIONS['compute_client_version'])
             self._compute_client = ComputeManagementClient(self.azure_credentials, self.subscription_id)
             self._register('Microsoft.Compute')
         return self._compute_client

lib/ansible/module_utils/basic.py

@@ -5,6 +5,7 @@
 # to the complete work.
 #
 # Copyright (c), Michael DeHaan <michael.dehaan@gmail.com>, 2012-2013
+# Copyright (c), Toshio Kuratomi <tkuratomi@ansible.com> 2016
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without modification,
@@ -1959,13 +1960,21 @@ class AnsibleModule(object):
                 except (OSError, IOError):
                     e = get_exception()
                     self.fail_json(msg='The destination directory (%s) is not writable by the current user. Error was: %s' % (os.path.dirname(dest), e))
+                except TypeError:
+                    # We expect that this is happening because python3.4.x and
+                    # below can't handle byte strings in mkstemp().  Traceback
+                    # would end in something like:
+                    #     file = _os.path.join(dir, pre + name + suf)
+                    # TypeError: can't concat bytes to str
+                    self.fail_json(msg='Failed creating temp file for atomic move.  This usually happens when using Python3 less than Python3.5.  Please use Python2.x or Python3.5 or greater.', exception=sys.exc_info())
+
                 b_tmp_dest_name = to_bytes(tmp_dest_name, errors='surrogate_or_strict')
 
                 try:
                     try:
                         # close tmp file handle before file operations to prevent text file busy errors on vboxfs synced folders (windows host)
                         os.close(tmp_dest_fd)
-                        # leaves tmp file behind when sudo and  not root
+                        # leaves tmp file behind when sudo and not root
                         if switched_user and os.getuid() != 0:
                             # cleanup will happen by 'rm' of tempdir
                             # copy2 will preserve some metadata
@@ -2030,7 +2039,7 @@ class AnsibleModule(object):
       else:
           self.fail_json(msg='Could not replace file: %s to %s: %s' % (src, dest, exception))
 
-    def run_command(self, args, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None, environ_update=None, umask=None):
+    def run_command(self, args, check_rc=False, close_fds=True, executable=None, data=None, binary_data=False, path_prefix=None, cwd=None, use_unsafe_shell=False, prompt_regex=None, environ_update=None, umask=None, encoding='utf-8', errors='surrogate_or_strict'):
         '''
         Execute a command, returns rc, stdout, and stderr.
 
@@ -2052,8 +2061,27 @@ class AnsibleModule(object):
         :kw prompt_regex: Regex string (not a compiled regex) which can be
             used to detect prompts in the stdout which would otherwise cause
             the execution to hang (especially if no input data is specified)
-        :kwarg environ_update: dictionary to *update* os.environ with
+        :kw environ_update: dictionary to *update* os.environ with
         :kw umask: Umask to be used when running the command. Default None
+        :kw encoding: Since we return native strings, on python3 we need to
+            know the encoding to use to transform from bytes to text.  If you
+            want to always get bytes back, use encoding=None.  The default is
+            "utf-8".  This does not affect transformation of strings given as
+            args.
+        :kw errors: Since we return native strings, on python3 we need to
+            transform stdout and stderr from bytes to text.  If the bytes are
+            undecodable in the ``encoding`` specified, then use this error
+            handler to deal with them.  The default is ``surrogate_or_strict``
+            which means that the bytes will be decoded using the
+            surrogateescape error handler if available (available on all
+            python3 versions we support) otherwise a UnicodeError traceback
+            will be raised.  This does not affect transformations of strings
+            given as args.
+        :returns: A 3-tuple of return code (integer), stdout (native string),
+            and stderr (native string).  On python2, stdout and stderr are both
+            byte strings.  On python3, stdout and stderr are text strings converted
+            according to the encoding and errors parameters.  If you want byte
+            strings on python3, use encoding=None to turn decoding to text off.
         '''
 
         shell = False
@@ -2188,11 +2216,7 @@ class AnsibleModule(object):
         try:
 
             if self._debug:
-                if isinstance(args, list):
-                    running = ' '.join(args)
-                else:
-                    running = args
-                self.log('Executing: ' + running)
+                self.log('Executing: ' + clean_args)
             cmd = subprocess.Popen(args, **kwargs)
 
             # the communication logic here is essentially taken from that
@@ -2225,6 +2249,10 @@ class AnsibleModule(object):
                 # if we're checking for prompts, do it now
                 if prompt_re:
                     if prompt_re.search(stdout) and not data:
+                        if encoding:
+                            stdout = to_native(stdout, encoding=encoding, errors=errors)
+                        else:
+                            stdout = stdout
                         return (257, stdout, "A prompt was encountered while running a command, but no input data was specified")
                 # only break out if no pipes are left to read or
                 # the pipes are completely read and
@@ -2268,6 +2296,9 @@ class AnsibleModule(object):
         # reset the pwd
         os.chdir(prev_dir)
 
+        if encoding is not None:
+            return (rc, to_native(stdout, encoding=encoding, errors=errors),
+                    to_native(stderr, encoding=encoding, errors=errors))
         return (rc, stdout, stderr)
 
     def append_to_file(self, filename, str):

lib/ansible/module_utils/docker_common.py

@@ -446,8 +446,7 @@ class AnsibleDockerClient(Client):
         '''
         self.log("Pulling image %s:%s" % (name, tag))
         try:
-            for line in self.pull(name, tag=tag, stream=True):
-                line = json.loads(line)
+            for line in self.pull(name, tag=tag, stream=True, decode=True):
                 self.log(line, pretty_print=True)
                 if line.get('error'):
                     if line.get('errorDetail'):

lib/ansible/module_utils/eos.py

@@ -35,6 +35,7 @@ from ansible.module_utils.network import add_argument, register_transport, to_li
 from ansible.module_utils.netcli import Command
 from ansible.module_utils.shell import CliBase
 from ansible.module_utils.urls import fetch_url, url_argument_spec
+from ansible.module_utils._text import to_native
 
 EAPI_FORMATS = ['json', 'text']
 
@@ -50,7 +51,7 @@ class EosConfigMixin(object):
         cmds = ['configure terminal']
         cmds.extend(to_list(commands))
         cmds.append('end')
-        responses = self.execute(commands)
+        responses = self.execute(cmds)
         return responses[1:-1]
 
     def get_config(self, include_defaults=False, **kwargs):
@@ -60,6 +61,12 @@ class EosConfigMixin(object):
         return self.execute([cmd])[0]
 
     def load_config(self, config, commit=False, replace=False):
+        if self.supports_sessions():
+            return self.load_config_session(config, commit, replace)
+        else:
+            return self.configure(config)
+
+    def load_config_session(self, config, commit=False, replace=False):
         """ Loads the configuration into the remote device
         """
         session = 'ansible_%s' % int(time.time())
@@ -82,7 +89,7 @@ class EosConfigMixin(object):
                 self.execute(['no configure session %s' % session])
         except NetworkError:
             exc = get_exception()
-            if 'timeout trying to send command' in exc.message:
+            if 'timeout trying to send command' in to_native(exc):
                 # try to get control back and get out of config mode
                 if isinstance(self, Cli):
                     self.execute(['\x03', 'end'])
@@ -116,6 +123,17 @@ class EosConfigMixin(object):
         commands = ['configure session %s' % session, 'abort']
         self.execute(commands)
 
+    def supports_sessions(self):
+        try:
+            if isinstance(self, Eapi):
+                self.execute(['show configuration sessions'], output='text')
+            else:
+                self.execute('show configuration sessions')
+            return True
+        except NetworkError:
+            return False
+
+
 
 class Eapi(EosConfigMixin):
 

lib/ansible/module_utils/facts.py

@@ -34,23 +34,9 @@ import pwd
 
 from ansible.module_utils.basic import get_all_subclasses
 from ansible.module_utils.six import PY3, iteritems
+from ansible.module_utils.six.moves import configparser, StringIO
 from ansible.module_utils._text import to_native
 
-# py2 vs py3; replace with six via ansiballz
-try:
-    # python2
-    import ConfigParser as configparser
-except ImportError:
-    # python3
-    import configparser
-
-try:
-    # python2
-    from StringIO import StringIO
-except ImportError:
-    # python3
-    from io import StringIO
-
 try:
     # python2
     from string import maketrans
@@ -261,9 +247,8 @@ class Facts(object):
                 # try to read it as json first
                 # if that fails read it with ConfigParser
                 # if that fails, skip it
-                rc, out, err = self.module.run_command(fn)
                 try:
-                    out = out.decode('utf-8', 'strict')
+                    rc, out, err = self.module.run_command(fn)
                 except UnicodeError:
                     fact = 'error loading fact - output of running %s was not utf-8' % fn
                     local[fact_base] = fact
@@ -394,9 +379,8 @@ class Facts(object):
     def get_lsb_facts(self):
         lsb_path = self.module.get_bin_path('lsb_release')
         if lsb_path:
-            rc, out, err = self.module.run_command([lsb_path, "-a"])
+            rc, out, err = self.module.run_command([lsb_path, "-a"], errors='surrogate_or_replace')
             if rc == 0:
-                out = out.decode('utf-8', 'replace')
                 self.facts['lsb'] = {}
                 for line in out.split('\n'):
                     if len(line) < 1 or ':' not in line:
@@ -466,8 +450,7 @@ class Facts(object):
     def get_caps_facts(self):
         capsh_path = self.module.get_bin_path('capsh')
         if capsh_path:
-            rc, out, err = self.module.run_command([capsh_path, "--print"])
-            out = out.decode('utf-8', 'replace')
+            rc, out, err = self.module.run_command([capsh_path, "--print"], errors='surrogate_or_replace')
             enforced_caps = []
             enforced = 'NA'
             for line in out.split('\n'):
@@ -650,7 +633,7 @@ class Distribution(object):
         SLC = 'RedHat', Ascendos = 'RedHat', CloudLinux = 'RedHat', PSBM = 'RedHat',
         OracleLinux = 'RedHat', OVS = 'RedHat', OEL = 'RedHat', Amazon = 'RedHat',
         XenServer = 'RedHat', Ubuntu = 'Debian', Debian = 'Debian', Raspbian = 'Debian', Slackware = 'Slackware', SLES = 'Suse',
-        SLED = 'Suse', openSUSE = 'Suse', SuSE = 'Suse', SLES_SAP = 'Suse', Gentoo = 'Gentoo', Funtoo = 'Gentoo',
+        SLED = 'Suse', openSUSE = 'Suse', openSUSE_Tumbleweed = 'Suse', SuSE = 'Suse', SLES_SAP = 'Suse', SUSE_LINUX = 'Suse', Gentoo = 'Gentoo', Funtoo = 'Gentoo',
         Archlinux = 'Archlinux', Manjaro = 'Archlinux', Mandriva = 'Mandrake', Mandrake = 'Mandrake', Altlinux = 'Altlinux',
         Solaris = 'Solaris', Nexenta = 'Solaris', OmniOS = 'Solaris', OpenIndiana = 'Solaris',
         SmartOS = 'Solaris', AIX = 'AIX', Alpine = 'Alpine', MacOSX = 'Darwin',
@@ -1026,11 +1009,11 @@ class LinuxHardware(Hardware):
             key = data[0]
             if key in self.ORIGINAL_MEMORY_FACTS:
                 val = data[1].strip().split(' ')[0]
-                self.facts["%s_mb" % key.lower()] = int(val) / 1024
+                self.facts["%s_mb" % key.lower()] = int(val) // 1024
 
             if key in self.MEMORY_FACTS:
                  val = data[1].strip().split(' ')[0]
-                 memstats[key.lower()] = int(val) / 1024
+                 memstats[key.lower()] = int(val) // 1024
 
         if None not in (memstats.get('memtotal'), memstats.get('memfree')):
             memstats['real:used'] = memstats['memtotal'] - memstats['memfree']
@@ -1268,7 +1251,7 @@ class LinuxHardware(Hardware):
     def _run_findmnt(self, findmnt_path):
         args = ['--list', '--noheadings', '--notruncate']
         cmd = [findmnt_path] + args
-        rc, out, err = self.module.run_command(cmd)
+        rc, out, err = self.module.run_command(cmd, errors='surrogate_or_replace')
         return rc, out, err
 
     def _find_bind_mounts(self):
@@ -1280,7 +1263,6 @@ class LinuxHardware(Hardware):
         rc, out, err = self._run_findmnt(findmnt_path)
         if rc != 0:
             return bind_mounts
-        out = out.decode('utf-8', 'replace')
 
         # find bind mounts, in case /etc/mtab is a symlink to /proc/mounts
         for line in out.splitlines():
@@ -1359,8 +1341,7 @@ class LinuxHardware(Hardware):
         self.facts['devices'] = {}
         lspci = self.module.get_bin_path('lspci')
         if lspci:
-            rc, pcidata, err = self.module.run_command([lspci, '-D'])
-            pcidata = pcidata.decode('utf-8', 'replace')
+            rc, pcidata, err = self.module.run_command([lspci, '-D'], errors='surrogate_or_replace')
         else:
             pcidata = None
 
@@ -1557,10 +1538,10 @@ class SunOSHardware(Hardware):
         reserved = int(out.split()[5][:-1])
         used = int(out.split()[8][:-1])
         free = int(out.split()[10][:-1])
-        self.facts['swapfree_mb'] = free / 1024
-        self.facts['swaptotal_mb'] = (free + used) / 1024
-        self.facts['swap_allocated_mb'] = allocated / 1024
-        self.facts['swap_reserved_mb'] = reserved / 1024
+        self.facts['swapfree_mb'] = free // 1024
+        self.facts['swaptotal_mb'] = (free + used) // 1024
+        self.facts['swap_allocated_mb'] = allocated // 1024
+        self.facts['swap_reserved_mb'] = reserved // 1024
 
     @timeout(10)
     def get_mount_facts(self):
@@ -1631,8 +1612,8 @@ class OpenBSDHardware(Hardware):
         #  0 0 0  47512   28160   51   0   0   0   0   0   1   0  116    89   17  0  1 99
         rc, out, err = self.module.run_command("/usr/bin/vmstat")
         if rc == 0:
-            self.facts['memfree_mb'] = int(out.splitlines()[-1].split()[4]) / 1024
-            self.facts['memtotal_mb'] = int(self.sysctl['hw.usermem']) / 1024 / 1024
+            self.facts['memfree_mb'] = int(out.splitlines()[-1].split()[4]) // 1024
+            self.facts['memtotal_mb'] = int(self.sysctl['hw.usermem']) // 1024 // 1024
 
         # Get swapctl info. swapctl output looks like:
         # total: 69268 1K-blocks allocated, 0 used, 69268 available
@@ -1642,8 +1623,8 @@ class OpenBSDHardware(Hardware):
         if rc == 0:
             swaptrans = maketrans(' ', ' ')
             data = out.split()
-            self.facts['swapfree_mb'] = int(data[-2].translate(swaptrans, "kmg")) / 1024
-            self.facts['swaptotal_mb'] = int(data[1].translate(swaptrans, "kmg")) / 1024
+            self.facts['swapfree_mb'] = int(data[-2].translate(swaptrans, "kmg")) // 1024
+            self.facts['swaptotal_mb'] = int(data[1].translate(swaptrans, "kmg")) // 1024
 
     def get_processor_facts(self):
         processor = []
@@ -1718,8 +1699,8 @@ class FreeBSDHardware(Hardware):
                 pagecount = int(data[1])
             if 'vm.stats.vm.v_free_count' in line:
                 freecount = int(data[1])
-        self.facts['memtotal_mb'] = pagesize * pagecount / 1024 / 1024
-        self.facts['memfree_mb'] = pagesize * freecount / 1024 / 1024
+        self.facts['memtotal_mb'] = pagesize * pagecount // 1024 // 1024
+        self.facts['memfree_mb'] = pagesize * freecount // 1024 // 1024
         # Get swapinfo.  swapinfo output looks like:
         # Device          1M-blocks     Used    Avail Capacity
         # /dev/ada0p3        314368        0   314368     0%
@@ -1730,8 +1711,8 @@ class FreeBSDHardware(Hardware):
             lines.pop()
         data = lines[-1].split()
         if data[0] != 'Device':
-            self.facts['swaptotal_mb'] = int(data[1]) / 1024
-            self.facts['swapfree_mb'] = int(data[3]) / 1024
+            self.facts['swaptotal_mb'] = int(data[1]) // 1024
+            self.facts['swapfree_mb'] = int(data[3]) // 1024
 
     @timeout(10)
     def get_mount_facts(self):
@@ -1860,7 +1841,7 @@ class NetBSDHardware(Hardware):
             key = data[0]
             if key in NetBSDHardware.MEMORY_FACTS:
                 val = data[1].strip().split(' ')[0]
-                self.facts["%s_mb" % key.lower()] = int(val) / 1024
+                self.facts["%s_mb" % key.lower()] = int(val) // 1024
 
     @timeout(10)
     def get_mount_facts(self):
@@ -1930,8 +1911,8 @@ class AIX(Hardware):
                 pagecount = int(data[0])
             if 'free pages' in line:
                 freecount = int(data[0])
-        self.facts['memtotal_mb'] = pagesize * pagecount / 1024 / 1024
-        self.facts['memfree_mb'] = pagesize * freecount / 1024 / 1024
+        self.facts['memtotal_mb'] = pagesize * pagecount // 1024 // 1024
+        self.facts['memfree_mb'] = pagesize * freecount // 1024 // 1024
         # Get swapinfo.  swapinfo output looks like:
         # Device          1M-blocks     Used    Avail Capacity
         # /dev/ada0p3        314368        0   314368     0%
@@ -2072,12 +2053,12 @@ class HPUX(Hardware):
         pagesize = 4096
         rc, out, err = self.module.run_command("/usr/bin/vmstat | tail -1", use_unsafe_shell=True)
         data = int(re.sub(' +',' ',out).split(' ')[5].strip())
-        self.facts['memfree_mb'] = pagesize * data / 1024 / 1024
+        self.facts['memfree_mb'] = pagesize * data // 1024 // 1024
         if self.facts['architecture'] == '9000/800':
             try:
                 rc, out, err = self.module.run_command("grep Physical /var/adm/syslog/syslog.log")
                 data = re.search('.*Physical: ([0-9]*) Kbytes.*',out).groups()[0].strip()
-                self.facts['memtotal_mb'] = int(data) / 1024
+                self.facts['memtotal_mb'] = int(data) // 1024
             except AttributeError:
                 #For systems where memory details aren't sent to syslog or the log has rotated, use parsed
                 #adb output. Unfortunately /dev/kmem doesn't have world-read, so this only works as root.
@@ -2168,11 +2149,11 @@ class Darwin(Hardware):
             self.facts['processor_cores'] = self.sysctl['hw.physicalcpu']
 
     def get_memory_facts(self):
-        self.facts['memtotal_mb'] = int(self.sysctl['hw.memsize']) / 1024 / 1024
+        self.facts['memtotal_mb'] = int(self.sysctl['hw.memsize']) // 1024 // 1024
 
         rc, out, err = self.module.run_command("sysctl hw.usermem")
         if rc == 0:
-            self.facts['memfree_mb'] = int(out.splitlines()[-1].split()[1]) / 1024 / 1024
+            self.facts['memfree_mb'] = int(out.splitlines()[-1].split()[1]) // 1024 // 1024
 
 
 class Network(Facts):
@@ -2254,8 +2235,7 @@ class LinuxNetwork(Network):
                 continue
             if v == 'v6' and not socket.has_ipv6:
                 continue
-            rc, out, err = self.module.run_command(command[v])
-            out = out.decode('utf-8', 'replace')
+            rc, out, err = self.module.run_command(command[v], errors='surrogate_or_replace')
             if not out:
                 # v6 routing may result in
                 #   RTNETLINK answers: Invalid argument
@@ -2425,12 +2405,10 @@ class LinuxNetwork(Network):
             ip_path = self.module.get_bin_path("ip")
 
             args = [ip_path, 'addr', 'show', 'primary', device]
-            rc, stdout, stderr = self.module.run_command(args)
-            primary_data = stdout.decode('utf-8', 'replace')
+            rc, primary_data, stderr = self.module.run_command(args, errors='surrogate_or_replace')
 
             args = [ip_path, 'addr', 'show', 'secondary', device]
-            rc, stdout, stderr = self.module.run_command(args)
-            secondary_data = stdout.decode('utf-8', 'decode')
+            rc, secondary_data, stderr = self.module.run_command(args, errors='surrogate_or_replace')
 
             parse_ip_output(primary_data)
             parse_ip_output(secondary_data, secondary=True)
@@ -2452,8 +2430,7 @@ class LinuxNetwork(Network):
         ethtool_path = self.module.get_bin_path("ethtool")
         if ethtool_path:
             args = [ethtool_path, '-k', device]
-            rc, stdout, stderr = self.module.run_command(args)
-            stdout = stdout.decode('utf-8', 'replace')
+            rc, stdout, stderr = self.module.run_command(args, errors='surrogate_or_replace')
             if rc == 0:
                 for line in stdout.strip().split('\n'):
                     if not line or line.endswith(":"):

lib/ansible/module_utils/lxd.py

@@ -121,6 +121,8 @@ class LXDClient(object):
             if resp_type == 'error':
                 if ok_error_codes is not None and resp_json['error_code'] in ok_error_codes:
                     return resp_json
+                if resp_json['error'] == "Certificate already in trust store":
+                    return resp_json
                 self._raise_err_from_json(resp_json)
             return resp_json
         except socket.error as e:

lib/ansible/module_utils/network.py

@@ -32,6 +32,7 @@ from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.basic import env_fallback, get_exception
 from ansible.module_utils.netcli import Cli, Command
 from ansible.module_utils.netcfg import Config
+from ansible.module_utils._text import to_native
 
 NET_TRANSPORT_ARGS = dict(
     host=dict(required=True),
@@ -105,7 +106,7 @@ class NetworkModule(AnsibleModule):
             self.fail_json(msg='Unknown transport or no default transport specified')
         except (TypeError, NetworkError):
             exc = get_exception()
-            self.fail_json(msg=exc.message)
+            self.fail_json(msg=to_native(exc))
 
         if connect_on_load:
             self.connect()
@@ -147,17 +148,20 @@ class NetworkModule(AnsibleModule):
                 self.connection.connect(self.params)
                 if self.params['authorize']:
                     self.connection.authorize(self.params)
+                self.log('connected to %s:%s using %s' % (self.params['host'],
+                         self.params['port'], self.params['transport']))
         except NetworkError:
             exc = get_exception()
-            self.fail_json(msg=exc.message)
+            self.fail_json(msg=to_native(exc))
 
     def disconnect(self):
         try:
             if self.connected:
                 self.connection.disconnect()
+            self.log('disconnected from %s' % self.params['host'])
         except NetworkError:
             exc = get_exception()
-            self.fail_json(msg=exc.message)
+            self.fail_json(msg=to_native(exc))
 
 def register_transport(transport, default=False):
     def register(cls):

lib/ansible/module_utils/shell.py

@@ -21,11 +21,6 @@ import re
 import socket
 import time
 
-# py2 vs py3; replace with six via ansiballz
-try:
-    from StringIO import StringIO
-except ImportError:
-    from io import StringIO
 
 try:
     import paramiko
@@ -36,6 +31,8 @@ except ImportError:
 
 from ansible.module_utils.basic import get_exception
 from ansible.module_utils.network import NetworkError
+from ansible.module_utils.six.moves import StringIO
+from ansible.module_utils._text import to_native
 
 ANSI_RE = [
     re.compile(r'(\x1b\[\?1h\x1b=)'),
@@ -55,7 +52,6 @@ class ShellError(Exception):
 
     def __init__(self, msg, command=None):
         super(ShellError, self).__init__(msg)
-        self.message = msg
         self.command = command
 
 
@@ -106,6 +102,8 @@ class Shell(object):
             raise ShellError("unable to resolve host name")
         except AuthenticationException:
             raise ShellError('Unable to authenticate to remote device')
+        except socket.timeout:
+            raise ShellError("timeout trying to connect to remote device")
         except socket.error:
             exc = get_exception()
             if exc.errno == 60:
@@ -157,7 +155,7 @@ class Shell(object):
             raise ShellError("timeout trying to send command: %s" % cmd)
         except socket.error:
             exc = get_exception()
-            raise ShellError("problem sending command to host: %s" % exc.message)
+            raise ShellError("problem sending command to host: %s" % to_native(exc))
         return responses
 
     def close(self):
@@ -230,7 +228,7 @@ class CliBase(object):
         except ShellError:
             exc = get_exception()
             raise NetworkError(
-                msg='failed to connect to %s:%s' % (host, port), exc=str(exc)
+                msg='failed to connect to %s:%s' % (host, port), exc=to_native(exc)
             )
 
         self._connected = True
@@ -249,7 +247,7 @@ class CliBase(object):
             return self.shell.send(commands)
         except ShellError:
             exc = get_exception()
-            raise NetworkError(exc.message, commands=commands)
+            raise NetworkError(to_native(exc), commands=commands)
 
     def run_commands(self, commands):
         return self.execute(to_list(commands))

lib/ansible/module_utils/urls.py

@@ -105,8 +105,6 @@ import platform
 import tempfile
 import base64
 
-from ansible.module_utils.basic import get_distribution, get_exception
-
 try:
     import httplib
 except ImportError:
@@ -115,7 +113,9 @@ except ImportError:
 
 import ansible.module_utils.six.moves.urllib.request as urllib_request
 import ansible.module_utils.six.moves.urllib.error as urllib_error
+from ansible.module_utils.basic import get_distribution, get_exception
 from ansible.module_utils.six import b
+from ansible.module_utils._text import to_bytes
 
 try:
     # python3
@@ -672,9 +672,14 @@ class SSLValidationHandler(urllib_request.BaseHandler):
                     s.sendall(self.CONNECT_COMMAND % (self.hostname, self.port))
                     if proxy_parts.get('username'):
                         credentials = "%s:%s" % (proxy_parts.get('username',''), proxy_parts.get('password',''))
-                        s.sendall('Proxy-Authorization: Basic %s\r\n' % credentials.encode('base64').strip())
-                    s.sendall('\r\n')
-                    connect_result = s.recv(4096)
+                        s.sendall(b('Proxy-Authorization: Basic %s\r\n') % base64.b64encode(to_bytes(credentials, errors='surrogate_or_strict')).strip())
+                    s.sendall(b('\r\n'))
+                    connect_result = b("")
+                    while connect_result.find(b("\r\n\r\n")) <= 0:
+                        connect_result += s.recv(4096)
+                        # 128 kilobytes of headers should be enough for everyone.
+                        if len(connect_result) > 131072:
+                            raise ProxyError('Proxy sent too verbose headers. Only 128KiB allowed.')
                     self.validate_proxy_response(connect_result)
                     if context:
                         ssl_s = context.wrap_socket(s, server_hostname=self.hostname)
@@ -878,7 +883,10 @@ def open_url(url, data=None, headers=None, method=None, use_proxy=True,
 
 
 def basic_auth_header(username, password):
-    return "Basic %s" % base64.b64encode("%s:%s" % (username, password))
+    """Takes a username and password and returns a byte string suitable for
+    using as value of an Authorization header to do basic auth.
+    """
+    return b("Basic %s") % base64.b64encode(to_bytes("%s:%s" % (username, password), errors='surrogate_or_strict'))
 
 
 def url_argument_spec():

lib/ansible/parsing/dataloader.py

@@ -313,7 +313,11 @@ class DataLoader():
                         search.append(os.path.join(os.path.dirname(b_mydir), b_dirname, b_source))
                         search.append(os.path.join(b_mydir, b_source))
                     else:
-                        search.append(os.path.join(b_upath, b_dirname, b_source))
+                        # don't add dirname if user already is using it in source
+                        if b_source.split(b'/')[0] == b_dirname:
+                            search.append(os.path.join(b_upath, b_source))
+                        else:
+                            search.append(os.path.join(b_upath, b_dirname, b_source))
                         search.append(os.path.join(b_upath, b'tasks', b_source))
                 elif b_dirname not in b_source.split(b'/'):
                     # don't add dirname if user already is using it in source

lib/ansible/parsing/vault/__init__.py

@@ -83,9 +83,9 @@ try:
 except ImportError:
     pass
 except Exception as e:
-    display.warning("Optional dependency 'cryptography' raised an exception, falling back to 'Crypto'")
+    display.vvvv("Optional dependency 'cryptography' raised an exception, falling back to 'Crypto'.")
     import traceback
-    display.debug("Traceback from import of cryptography was {0}".format(traceback.format_exc()))
+    display.vvvv("Traceback from import of cryptography was {0}".format(traceback.format_exc()))
 
 HAS_ANY_PBKDF2HMAC = HAS_PBKDF2 or HAS_PBKDF2HMAC
 

lib/ansible/playbook/helpers.py

@@ -197,7 +197,7 @@ def load_list_of_tasks(ds, play, block=None, role=None, task_include=None, use_h
                         # the same fashion used by the on_include callback. We also do it here,
                         # because the recursive nature of helper methods means we may be loading
                         # nested includes, and we want the include order printed correctly
-                        display.display("statically included: %s" % include_file, color=C.COLOR_SKIP)
+                        display.vv("statically included: %s" % include_file)
                     except AnsibleFileNotFound:
                         if t.static or \
                            C.DEFAULT_TASK_INCLUDES_STATIC or \

lib/ansible/playbook/play_context.py

@@ -31,6 +31,7 @@ import string
 from ansible.compat.six import iteritems, string_types
 from ansible import constants as C
 from ansible.errors import AnsibleError
+from ansible.module_utils._text import to_bytes
 from ansible.playbook.attribute import FieldAttribute
 from ansible.playbook.base import Base
 from ansible.utils.boolean import boolean
@@ -84,38 +85,39 @@ MAGIC_VARIABLE_MAPPING = dict(
    module_compression = ('ansible_module_compression',),
 )
 
-SU_PROMPT_LOCALIZATIONS = [
-    'Password',
-    '암호',
-    'パスワード',
-    'Adgangskode',
-    'Contraseña',
-    'Contrasenya',
-    'Hasło',
-    'Heslo',
-    'Jelszó',
-    'Lösenord',
-    'Mật khẩu',
-    'Mot de passe',
-    'Parola',
-    'Parool',
-    'Pasahitza',
-    'Passord',
-    'Passwort',
-    'Salasana',
-    'Sandi',
-    'Senha',
-    'Wachtwoord',
-    'ססמה',
-    'Лозинка',
-    'Парола',
-    'Пароль',
-    'गुप्तशब्द',
-    'शब्दकूट',
-    'సంకేతపదము',
-    'හස්පදය',
-    '密码',
-    '密碼',
+b_SU_PROMPT_LOCALIZATIONS = [
+    to_bytes('Password'),
+    to_bytes('암호'),
+    to_bytes('パスワード'),
+    to_bytes('Adgangskode'),
+    to_bytes('Contraseña'),
+    to_bytes('Contrasenya'),
+    to_bytes('Hasło'),
+    to_bytes('Heslo'),
+    to_bytes('Jelszó'),
+    to_bytes('Lösenord'),
+    to_bytes('Mật khẩu'),
+    to_bytes('Mot de passe'),
+    to_bytes('Parola'),
+    to_bytes('Parool'),
+    to_bytes('Pasahitza'),
+    to_bytes('Passord'),
+    to_bytes('Passwort'),
+    to_bytes('Salasana'),
+    to_bytes('Sandi'),
+    to_bytes('Senha'),
+    to_bytes('Wachtwoord'),
+    to_bytes('ססמה'),
+    to_bytes('Лозинка'),
+    to_bytes('Парола'),
+    to_bytes('Пароль'),
+    to_bytes('गुप्तशब्द'),
+    to_bytes('शब्दकूट'),
+    to_bytes('సంకేతపదము'),
+    to_bytes('හස්පදය'),
+    to_bytes('密码'),
+    to_bytes('密碼'),
+    to_bytes('口令'),
 ]
 
 TASK_ATTRIBUTE_OVERRIDES = (
@@ -515,21 +517,24 @@ class PlayContext(Base):
             elif self.become_method == 'su':
 
                 # passing code ref to examine prompt as simple string comparisson isn't good enough with su
-                def detect_su_prompt(data):
-                    SU_PROMPT_LOCALIZATIONS_RE = re.compile("|".join(['(\w+\'s )?' + x + ' ?: ?' for x in SU_PROMPT_LOCALIZATIONS]), flags=re.IGNORECASE)
-                    return bool(SU_PROMPT_LOCALIZATIONS_RE.match(data))
+                def detect_su_prompt(b_data):
+                    b_password_string = b"|".join([b'(\w+\'s )?' + x for x in b_SU_PROMPT_LOCALIZATIONS])
+                    # Colon or unicode fullwidth colon
+                    b_password_string = b_password_string + to_bytes(u' ?(:|：) ?')
+                    b_SU_PROMPT_LOCALIZATIONS_RE = re.compile(b_password_string, flags=re.IGNORECASE)
+                    return bool(b_SU_PROMPT_LOCALIZATIONS_RE.match(b_data))
                 prompt = detect_su_prompt
 
                 becomecmd = '%s %s %s -c %s' % (exe, flags, self.become_user, pipes.quote(command))
 
             elif self.become_method == 'pbrun':
 
-                prompt='assword:'
-                becomecmd = '%s -b %s -u %s %s' % (exe, flags, self.become_user, success_cmd)
+                prompt='Password:'
+                becomecmd = '%s %s -u %s %s' % (exe, flags, self.become_user, success_cmd)
 
             elif self.become_method == 'ksu':
-                def detect_ksu_prompt(data):
-                    return re.match("Kerberos password for .*@.*:", data)
+                def detect_ksu_prompt(b_data):
+                    return re.match(b"Kerberos password for .*@.*:", b_data)
 
                 prompt = detect_ksu_prompt
                 becomecmd = '%s %s %s -e %s' % (exe, self.become_user, flags, command)

lib/ansible/playbook/role/__init__.py

@@ -207,6 +207,8 @@ class Role(Base, Become, Conditional, Taggable):
             main_file = self._resolve_main(file_path, main)
             if self._loader.path_exists(main_file):
                 return self._loader.load_from_file(main_file)
+            elif main is not None:
+                raise AnsibleParserError("Could not find specified file in role: %s" % main)
         return None
 
     def _resolve_main(self, basepath, main=None):

lib/ansible/playbook/role/definition.py

@@ -138,18 +138,22 @@ class RoleDefinition(Base, Become, Conditional, Taggable):
         # we always start the search for roles in the base directory of the playbook
         role_search_paths = [
             os.path.join(self._loader.get_basedir(), u'roles'),
-            self._loader.get_basedir(),
         ]
 
         # also search in the configured roles path
         if C.DEFAULT_ROLES_PATH:
             role_search_paths.extend(C.DEFAULT_ROLES_PATH)
 
-        # finally, append the roles basedir, if it was set, so we can
+        # next, append the roles basedir, if it was set, so we can
         # search relative to that directory for dependent roles
         if self._role_basedir:
             role_search_paths.append(self._role_basedir)
 
+        # finally as a last resort we look in the current basedir as set
+        # in the loader (which should be the playbook dir itself) but without
+        # the roles/ dir appended
+        role_search_paths.append(self._loader.get_basedir())
+
         # create a templar class to template the dependency names, in
         # case they contain variables
         if self._variable_manager is not None:

lib/ansible/playbook/role/include.py

@@ -28,6 +28,7 @@ from ansible.parsing.yaml.objects import AnsibleBaseYAMLObject
 from ansible.playbook.attribute import Attribute, FieldAttribute
 from ansible.playbook.role.definition import RoleDefinition
 from ansible.playbook.role.requirement import RoleRequirement
+from ansible.module_utils._text import to_native
 
 
 __all__ = ['RoleInclude']
@@ -49,7 +50,9 @@ class RoleInclude(RoleDefinition):
     @staticmethod
     def load(data, play, current_role_path=None, parent_role=None, variable_manager=None, loader=None):
 
-        assert isinstance(data, string_types) or isinstance(data, dict) or isinstance(data, AnsibleBaseYAMLObject)
+        if not (isinstance(data, string_types) or isinstance(data, dict) or isinstance(data, AnsibleBaseYAMLObject)):
+            raise AnsibleParserError("Invalid role definition: %s" % to_native(data))
+
         if isinstance(data, string_types) and ',' in data:
             data = RoleRequirement.role_spec_parse(data)
 

lib/ansible/playbook/role_include.py

@@ -21,6 +21,7 @@ __metaclass__ = type
 
 from os.path import basename
 
+from ansible.errors import AnsibleParserError
 from ansible.playbook.attribute import FieldAttribute
 from ansible.playbook.task import Task
 from ansible.playbook.role import Role
@@ -46,6 +47,7 @@ class IncludeRole(Task):
     # ATTRIBUTES
 
     # private as this is a 'module options' vs a task property
+    _allow_duplicates = FieldAttribute(isa='bool', default=True, private=True)
     _static = FieldAttribute(isa='bool', default=None, private=True)
     _private = FieldAttribute(isa='bool', default=None, private=True)
 
@@ -56,6 +58,7 @@ class IncludeRole(Task):
         self.statically_loaded = False
         self._from_files = {}
         self._parent_role = role
+        self._role_name = None
 
 
     def get_block_list(self, play=None, variable_manager=None, loader=None):
@@ -66,11 +69,12 @@ class IncludeRole(Task):
         else:
             myplay = play
 
-        ri = RoleInclude.load(self.name, play=myplay, variable_manager=variable_manager, loader=loader)
+        ri = RoleInclude.load(self._role_name, play=myplay, variable_manager=variable_manager, loader=loader)
         ri.vars.update(self.vars)
 
         # build role
         actual_role = Role.load(ri, myplay, parent_role=self._parent_role, from_files=self._from_files)
+        actual_role._metadata.allow_duplicates = self.allow_duplicates
 
         # compile role
         blocks = actual_role.compile(play=myplay)
@@ -89,18 +93,24 @@ class IncludeRole(Task):
 
         ir = IncludeRole(block, role, task_include=task_include).load_data(data, variable_manager=variable_manager, loader=loader)
 
-        # set built in's
-        attributes = frozenset(ir._valid_attrs.keys())
-        for builtin in attributes:
-            if ir.args.get(builtin):
-                setattr(ir, builtin, ir.args.get(builtin))
+        ### Process options
+        # name is needed, or use role as alias
+        ir._role_name = ir.args.get('name', ir.args.get('role'))
+        if ir._role_name is None:
+            raise AnsibleParserError("'name' is a required field for include_role.")
 
         # build options for role includes
         for key in ['tasks', 'vars', 'defaults']:
-            from_key = key + '_from'
+            from_key ='%s_from' % key
             if  ir.args.get(from_key):
                 ir._from_files[key] = basename(ir.args.get(from_key))
 
+        #FIXME: find a way to make this list come from object ( attributes does not work as per below)
+        # manual list as otherwise the options would set other task parameters we don't want.
+        for option in ['static', 'private', 'allow_duplicates']:
+            if option in ir.args:
+                setattr(ir, option, ir.args.get(option))
+
         return ir.load_data(data, variable_manager=variable_manager, loader=loader)
 
     def copy(self, exclude_parent=False, exclude_tasks=False):
@@ -109,6 +119,7 @@ class IncludeRole(Task):
         new_me.statically_loaded = self.statically_loaded
         new_me._from_files = self._from_files.copy()
         new_me._parent_role = self._parent_role
+        new_me._role_name   = self._role_name
 
         return new_me
 

lib/ansible/plugins/__init__.py

@@ -373,6 +373,7 @@ class PluginLoader:
     def all(self, *args, **kwargs):
         ''' instantiates all plugins with the same arguments '''
 
+        path_only = kwargs.pop('path_only', False)
         class_only = kwargs.pop('class_only', False)
         all_matches = []
         found_in_cache = True
@@ -385,6 +386,10 @@ class PluginLoader:
             if '__init__' in name:
                 continue
 
+            if path_only:
+                yield path
+                continue
+
             if path not in self._module_cache:
                 self._module_cache[path] = self._load_module_source(name, path)
                 found_in_cache = False

lib/ansible/plugins/action/__init__.py

@@ -38,6 +38,7 @@ from ansible.executor.module_common import modify_module
 from ansible.module_utils._text import to_bytes, to_native, to_text
 from ansible.module_utils.json_utils import _filter_non_json_lines
 from ansible.parsing.utils.jsonify import jsonify
+from ansible.playbook.play_context import MAGIC_VARIABLE_MAPPING
 from ansible.release import __version__
 
 
@@ -670,6 +671,32 @@ class ActionBase(with_metaclass(ABCMeta, object)):
                 display.warning(w)
             data = json.loads(filtered_output)
             data['_ansible_parsed'] = True
+            if 'ansible_facts' in data and isinstance(data['ansible_facts'], dict):
+                remove_keys = set()
+                fact_keys = set(data['ansible_facts'].keys())
+                # first we add all of our magic variable names to the set of
+                # keys we want to remove from facts
+                for magic_var in MAGIC_VARIABLE_MAPPING:
+                    remove_keys.update(fact_keys.intersection(MAGIC_VARIABLE_MAPPING[magic_var]))
+                # next we remove any connection plugin specific vars
+                for conn_path in self._shared_loader_obj.connection_loader.all(path_only=True):
+                    try:
+                        conn_name = os.path.splitext(os.path.basename(conn_path))[0]
+                        re_key = re.compile('^ansible_%s_' % conn_name)
+                        for fact_key in fact_keys:
+                            if re_key.match(fact_key):
+                                remove_keys.add(fact_key)
+                    except AttributeError:
+                        pass
+                # finally, we search for interpreter keys to remove
+                re_interp = re.compile('^ansible_.*_interpreter$')
+                for fact_key in fact_keys:
+                    if re_interp.match(fact_key):
+                        remove_keys.add(fact_key)
+                # then we remove them (except for ssh host keys)
+                for r_key in remove_keys:
+                    if not r_key.startswith('ansible_ssh_host_key_'):
+                        del data['ansible_facts'][r_key]
         except ValueError:
             # not valid json, lets try to capture error
             data = dict(failed=True, _ansible_parsed=False)
@@ -721,8 +748,8 @@ class ActionBase(with_metaclass(ABCMeta, object)):
 
         # Change directory to basedir of task for command execution when connection is local
         if self._connection.transport == 'local':
-            os.chdir(self._loader.get_basedir())
             cwd = os.getcwd()
+            os.chdir(self._loader.get_basedir())
         try:
             rc, stdout, stderr = self._connection.exec_command(cmd, in_data=in_data, sudoable=sudoable)
         finally:
@@ -817,6 +844,7 @@ class ActionBase(with_metaclass(ABCMeta, object)):
             to get back the first existing file found.
         '''
 
+        # dwim already deals with playbook basedirs
         path_stack = self._task.get_search_path()
 
         result = self._loader.path_dwim_relative_stack(path_stack, dirname, needle)

lib/ansible/plugins/action/synchronize.py

@@ -114,7 +114,7 @@ class ActionModule(ActionBase):
         # connection to the remote host
         if 'ansible_syslog_facility' in task_vars:
             del task_vars['ansible_syslog_facility']
-        for key in task_vars.keys():
+        for key in list(task_vars.keys()):
             if key.startswith("ansible_") and key.endswith("_interpreter"):
                 del task_vars[key]
 

lib/ansible/plugins/connection/local.py

@@ -90,7 +90,7 @@ class Connection(ConnectionBase):
         if self._play_context.prompt and sudoable:
             fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) | os.O_NONBLOCK)
             fcntl.fcntl(p.stderr, fcntl.F_SETFL, fcntl.fcntl(p.stderr, fcntl.F_GETFL) | os.O_NONBLOCK)
-            become_output = ''
+            become_output = b''
             while not self.check_become_success(become_output) and not self.check_password_prompt(become_output):
 
                 rfd, wfd, efd = select.select([p.stdout, p.stderr], [], [p.stdout, p.stderr], self._play_context.timeout)
@@ -100,13 +100,13 @@ class Connection(ConnectionBase):
                     chunk = p.stderr.read()
                 else:
                     stdout, stderr = p.communicate()
-                    raise AnsibleError('timeout waiting for privilege escalation password prompt:\n' + become_output)
+                    raise AnsibleError('timeout waiting for privilege escalation password prompt:\n' + to_native(become_output))
                 if not chunk:
                     stdout, stderr = p.communicate()
-                    raise AnsibleError('privilege output closed while waiting for password prompt:\n' + become_output)
+                    raise AnsibleError('privilege output closed while waiting for password prompt:\n' + to_native(become_output))
                 become_output += chunk
             if not self.check_become_success(become_output):
-                p.stdin.write(self._play_context.become_pass + '\n')
+                p.stdin.write(to_bytes(self._play_context.become_pass, errors='surrogate_or_strict') + b'\n')
             fcntl.fcntl(p.stdout, fcntl.F_SETFL, fcntl.fcntl(p.stdout, fcntl.F_GETFL) & ~os.O_NONBLOCK)
             fcntl.fcntl(p.stderr, fcntl.F_SETFL, fcntl.fcntl(p.stderr, fcntl.F_GETFL) & ~os.O_NONBLOCK)
 

lib/ansible/plugins/connection/paramiko_ssh.py

@@ -310,7 +310,7 @@ class Connection(ConnectionBase):
 
                 if passprompt:
                     if self._play_context.become and self._play_context.become_pass:
-                        chan.sendall(self._play_context.become_pass + '\n')
+                        chan.sendall(to_bytes(self._play_context.become_pass) + b'\n')
                     else:
                         raise AnsibleError("A password is reqired but none was supplied")
                 else:

lib/ansible/plugins/connection/ssh.py

@@ -599,6 +599,55 @@ class Connection(ConnectionBase):
 
         return (returncode, stdout, stderr)
 
+    def _file_transport_command(self, in_path, out_path, sftp_action):
+        # scp and sftp require square brackets for IPv6 addresses, but
+        # accept them for hostnames and IPv4 addresses too.
+        host = '[%s]' % self.host
+
+        # since this can be a non-bool now, we need to handle it correctly
+        scp_if_ssh = C.DEFAULT_SCP_IF_SSH
+        if not isinstance(scp_if_ssh, bool):
+            scp_if_ssh = scp_if_ssh.lower()
+            if scp_if_ssh in BOOLEANS:
+                scp_if_ssh = boolean(scp_if_ssh)
+            elif scp_if_ssh != 'smart':
+                raise AnsibleOptionsError('scp_if_ssh needs to be one of [smart|True|False]')
+
+        # create a list of commands to use based on config options
+        methods = ['sftp']
+        if scp_if_ssh == 'smart':
+            methods.append('scp')
+        elif scp_if_ssh:
+            methods = ['scp']
+
+        success = False
+        res = None
+        for method in methods:
+            if method == 'sftp':
+                cmd = self._build_command('sftp', to_bytes(host))
+                in_data = u"{0} {1} {2}\n".format(sftp_action, pipes.quote(in_path), pipes.quote(out_path))
+            elif method == 'scp':
+                cmd = self._build_command('scp', in_path, u'{0}:{1}'.format(host, pipes.quote(out_path)))
+                in_data = None
+
+            in_data = to_bytes(in_data, nonstring='passthru')
+            (returncode, stdout, stderr) = self._run(cmd, in_data, checkrc=False)
+            # Check the return code and rollover to next method if failed
+            if returncode == 0:
+                success = True
+                break
+            else:
+                # If not in smart mode, the data will be printed by the raise below
+                if scp_if_ssh == 'smart':
+                    display.warning(msg='%s transfer mechanism failed on %s. Use ANSIBLE_DEBUG=1 to see detailed information' % (method, host))
+                    display.debug(msg='%s' % to_native(stdout))
+                    display.debug(msg='%s' % to_native(stderr))
+                res = (returncode, stdout, stderr)
+
+        if not success:
+            raise AnsibleError("failed to transfer file to {0}:\n{1}\n{2}"\
+                    .format(to_native(out_path), to_native(res[1]), to_native(res[2])))
+
     #
     # Main public methods
     #
@@ -655,53 +704,7 @@ class Connection(ConnectionBase):
         if not os.path.exists(to_bytes(in_path, errors='surrogate_or_strict')):
             raise AnsibleFileNotFound("file or module does not exist: {0}".format(to_native(in_path)))
 
-        # scp and sftp require square brackets for IPv6 addresses, but
-        # accept them for hostnames and IPv4 addresses too.
-        host = '[%s]' % self.host
-
-        # since this can be a non-bool now, we need to handle it correctly
-        scp_if_ssh = C.DEFAULT_SCP_IF_SSH
-        if not isinstance(scp_if_ssh, bool):
-            scp_if_ssh = scp_if_ssh.lower()
-            if scp_if_ssh in BOOLEANS:
-                scp_if_ssh = boolean(scp_if_ssh)
-            elif scp_if_ssh != 'smart':
-                raise AnsibleOptionsError('scp_if_ssh needs to be one of [smart|True|False]')
-
-        # create a list of commands to use based on config options
-        methods = ['sftp']
-        if scp_if_ssh == 'smart':
-            methods.append('scp')
-        elif scp_if_ssh:
-            methods = ['scp']
-
-        success = False
-        res = None
-        for method in methods:
-            if method == 'sftp':
-                cmd = self._build_command('sftp', to_bytes(host))
-                in_data = u"put {0} {1}\n".format(pipes.quote(in_path), pipes.quote(out_path))
-            elif method == 'scp':
-                cmd = self._build_command('scp', in_path, u'{0}:{1}'.format(host, pipes.quote(out_path)))
-                in_data = None
-
-            in_data = to_bytes(in_data, nonstring='passthru')
-            (returncode, stdout, stderr) = self._run(cmd, in_data, checkrc=False)
-            # Check the return code and rollover to next method if failed
-            if returncode == 0:
-                success = True
-                break
-            else:
-                # If not in smart mode, the data will be printed by the raise below 
-                if scp_if_ssh == 'smart':
-                    display.warning(msg='%s transfer mechanism failed on %s. Use ANSIBLE_DEBUG=1 to see detailed information' % (method, host))
-                    display.debug(msg='%s' % to_native(stdout))
-                    display.debug(msg='%s' % to_native(stderr))
-                res = (returncode, stdout, stderr)
-
-        if not success:
-            raise AnsibleError("failed to transfer file to {0}:\n{1}\n{2}"\
-                    .format(to_native(out_path), to_native(res[1]), to_native(res[2])))
+        self._file_transport_command(in_path, out_path, 'put')
 
     def fetch_file(self, in_path, out_path):
         ''' fetch a file from remote to local '''
@@ -709,23 +712,7 @@ class Connection(ConnectionBase):
         super(Connection, self).fetch_file(in_path, out_path)
 
         display.vvv(u"FETCH {0} TO {1}".format(in_path, out_path), host=self.host)
-
-        # scp and sftp require square brackets for IPv6 addresses, but
-        # accept them for hostnames and IPv4 addresses too.
-        host = '[%s]' % self.host
-
-        if C.DEFAULT_SCP_IF_SSH:
-            cmd = self._build_command('scp', u'{0}:{1}'.format(host, pipes.quote(in_path)), out_path)
-            in_data = None
-        else:
-            cmd = self._build_command('sftp', host)
-            in_data = u"get {0} {1}\n".format(pipes.quote(in_path), pipes.quote(out_path))
-
-        in_data = to_bytes(in_data, nonstring='passthru')
-        (returncode, stdout, stderr) = self._run(cmd, in_data)
-
-        if returncode != 0:
-            raise AnsibleError("failed to transfer file from {0}:\n{1}\n{2}".format(in_path, stdout, stderr))
+        self._file_transport_command(in_path, out_path, 'get')
 
     def close(self):
         # If we have a persistent ssh connection (ControlPersist), we can ask it

lib/ansible/plugins/filter/core.py

@@ -67,7 +67,7 @@ class AnsibleJSONEncoder(json.JSONEncoder):
         if isinstance(o, HostVars):
             return dict(o)
         else:
-            return json.JSONEncoder.default(o)
+            return super(AnsibleJSONEncoder, self).default(o)
 
 def to_yaml(a, *args, **kw):
     '''Make verbose, human readable yaml'''
@@ -257,7 +257,11 @@ def get_encrypted_password(password, hashtype='sha512', salt=None):
             saltstring =  "$%s$%s" % (cryptmethod[hashtype],salt)
             encrypted = crypt.crypt(password, saltstring)
         else:
-            cls = getattr(passlib.hash, '%s_crypt' % hashtype)
+            if hashtype == 'blowfish':
+                cls = passlib.hash.bcrypt;
+            else:
+                cls = getattr(passlib.hash, '%s_crypt' % hashtype)
+
             encrypted = cls.encrypt(password, salt=salt)
 
         return encrypted
@@ -387,6 +391,46 @@ def extract(item, container, morekeys=None):
 
     return value
 
+def failed(*a, **kw):
+    ''' Test if task result yields failed '''
+    item = a[0]
+    if type(item) != dict:
+        raise errors.AnsibleFilterError("|failed expects a dictionary")
+    rc = item.get('rc',0)
+    failed = item.get('failed',False)
+    if rc != 0 or failed:
+        return True
+    else:
+        return False
+
+def success(*a, **kw):
+    ''' Test if task result yields success '''
+    return not failed(*a, **kw)
+
+def changed(*a, **kw):
+    ''' Test if task result yields changed '''
+    item = a[0]
+    if type(item) != dict:
+        raise errors.AnsibleFilterError("|changed expects a dictionary")
+    if not 'changed' in item:
+        changed = False
+        if ('results' in item    # some modules return a 'results' key
+                and type(item['results']) == list
+                and type(item['results'][0]) == dict):
+            for result in item['results']:
+                changed = changed or result.get('changed', False)
+    else:
+        changed = item.get('changed', False)
+    return changed
+
+def skipped(*a, **kw):
+    ''' Test if task result yields skipped '''
+    item = a[0]
+    if type(item) != dict:
+        raise errors.AnsibleFilterError("|skipped expects a dictionary")
+    skipped = item.get('skipped', False)
+    return skipped
+
 class FilterModule(object):
     ''' Ansible core jinja2 filters '''
 
@@ -467,4 +511,18 @@ class FilterModule(object):
 
             # array and dict lookups
             'extract': extract,
+
+            # failure testing
+            'failed'    : failed,
+            'failure'   : failed,
+            'success'   : success,
+            'succeeded' : success,
+
+            # changed testing
+            'changed' : changed,
+            'change'  : changed,
+
+            # skip testing
+            'skipped' : skipped,
+            'skip'    : skipped,
         }

lib/ansible/plugins/lookup/__init__.py

@@ -102,7 +102,7 @@ class LookupBase(with_metaclass(ABCMeta, object)):
         """
         pass
 
-    def find_file_in_search_path(self, myvars, subdir, needle):
+    def find_file_in_search_path(self, myvars, subdir, needle, ignore_missing=False):
         '''
         Return a file (needle) in the task's expected search path.
         '''
@@ -113,7 +113,7 @@ class LookupBase(with_metaclass(ABCMeta, object)):
             paths = self.get_basedir(myvars)
 
         result = self._loader.path_dwim_relative_stack(paths, subdir, needle)
-        if result is None:
+        if result is None and not ignore_missing:
             self._display.warning("Unable to find '%s' in expected paths." % needle)
 
         return result

lib/ansible/plugins/lookup/first_found.py

@@ -181,7 +181,7 @@ class LookupModule(LookupBase):
              # get subdir if set by task executor, default to files otherwise
             subdir = getattr(self, '_subdir', 'files')
             path = None
-            path = self.find_file_in_search_path(variables, subdir, fn)
+            path = self.find_file_in_search_path(variables, subdir, fn, ignore_missing=True)
             if path is not None:
                 return [path]
         else:

lib/ansible/plugins/lookup/password.py

@@ -248,8 +248,7 @@ class LookupModule(LookupBase):
                 _write_password_file(b_path, content)
 
             if params['encrypt']:
-                password = do_encrypt(plaintext_password, params['encrypt'],
-                                      salt=to_bytes(salt, encoding='ascii', errors='strict'))
+                password = do_encrypt(plaintext_password, params['encrypt'], salt=salt)
                 ret.append(password)
             else:
                 ret.append(plaintext_password)

lib/ansible/plugins/strategy/__init__.py

@@ -26,6 +26,7 @@ from collections import deque
 from multiprocessing import Lock
 from jinja2.exceptions import UndefinedError
 
+from ansible import constants as C
 from ansible.compat.six.moves import queue as Queue
 from ansible.compat.six import iteritems, string_types
 from ansible.errors import AnsibleError, AnsibleParserError, AnsibleUndefinedVariable
@@ -71,14 +72,12 @@ class SharedPluginLoaderObj:
 
 _sentinel = object()
 def results_thread_main(strategy):
-    #print("RESULT THREAD STARTING: %s" % threading.current_thread())
     while True:
         try:
             result = strategy._final_q.get()
             if type(result) == object:
                 break
             else:
-                #print("result in thread is: %s" % result._result)
                 strategy._results_lock.acquire()
                 strategy._results.append(result)
                 strategy._results_lock.release()
@@ -86,7 +85,6 @@ def results_thread_main(strategy):
             break
         except Queue.Empty:
             pass
-    #print("RESULT THREAD EXITED: %s" % threading.current_thread())
 
 class StrategyBase:
 
@@ -121,7 +119,7 @@ class StrategyBase:
         self._results = deque()
         self._results_lock = threading.Condition(threading.Lock())
 
-        #print("creating thread for strategy %s" % id(self))
+        # create the result processing thread for reading results in the background
         self._results_thread = threading.Thread(target=results_thread_main, args=(self,))
         self._results_thread.daemon = True
         self._results_thread.start()
@@ -316,7 +314,6 @@ class StrategyBase:
                 continue
 
             if original_task.register:
-                #print("^ REGISTERING RESULT %s" % original_task.register)
                 if original_task.run_once:
                     host_list = [host for host in self._inventory.get_hosts(iterator._play.hosts) if host.name not in self._tqm._unreachable_hosts]
                 else:
@@ -391,40 +388,43 @@ class StrategyBase:
                             # So, per the docs, we reassign the list so the proxy picks up and
                             # notifies all other threads
                             for handler_name in result_item['_ansible_notify']:
+                                found = False
                                 # Find the handler using the above helper.  First we look up the
                                 # dependency chain of the current task (if it's from a role), otherwise
                                 # we just look through the list of handlers in the current play/all
                                 # roles and use the first one that matches the notify name
+                                target_handler = search_handler_blocks(handler_name, iterator._play.handlers)
+                                if target_handler is not None:
+                                    found = True
+                                    if original_host not in self._notified_handlers[target_handler]:
+                                        self._notified_handlers[target_handler].append(original_host)
+                                        # FIXME: should this be a callback?
+                                        display.vv("NOTIFIED HANDLER %s" % (handler_name,))
+                                else:
+                                    # As there may be more than one handler with the notified name as the
+                                    # parent, so we just keep track of whether or not we found one at all
+                                    for target_handler in self._notified_handlers:
+                                        if parent_handler_match(target_handler, handler_name):
+                                            self._notified_handlers[target_handler].append(original_host)
+                                            display.vv("NOTIFIED HANDLER %s" % (target_handler.get_name(),))
+                                            found = True
+
                                 if handler_name in self._listening_handlers:
                                     for listening_handler_name in self._listening_handlers[handler_name]:
                                         listening_handler = search_handler_blocks(listening_handler_name, iterator._play.handlers)
-                                        if listening_handler is None:
-                                            raise AnsibleError("The requested handler listener '%s' was not found in any of the known handlers" % listening_handler_name)
+                                        if listening_handler is not None:
+                                            found = True
                                         if original_host not in self._notified_handlers[listening_handler]:
                                             self._notified_handlers[listening_handler].append(original_host)
                                             display.vv("NOTIFIED HANDLER %s" % (listening_handler_name,))
 
-                                else:
-                                    target_handler = search_handler_blocks(handler_name, iterator._play.handlers)
-                                    if target_handler is not None:
-                                        if original_host not in self._notified_handlers[target_handler]:
-                                            self._notified_handlers[target_handler].append(original_host)
-                                            # FIXME: should this be a callback?
-                                            display.vv("NOTIFIED HANDLER %s" % (handler_name,))
-                                    else:
-                                        # As there may be more than one handler with the notified name as the
-                                        # parent, so we just keep track of whether or not we found one at all
-                                        found = False
-                                        for target_handler in self._notified_handlers:
-                                            if parent_handler_match(target_handler, handler_name):
-                                                self._notified_handlers[target_handler].append(original_host)
-                                                display.vv("NOTIFIED HANDLER %s" % (target_handler.get_name(),))
-                                                found = True
-
-                                        # and if none were found, then we raise an error
-                                        if not found:
-                                            raise AnsibleError("The requested handler '%s' was found in neither the main handlers list nor the listening handlers list" % handler_name)
-
+                                # and if none were found, then we raise an error
+                                if not found:
+                                  msg = "The requested handler '%s' was not found in either the main handlers list nor in the listening handlers list" % handler_name
+                                  if C.ERROR_ON_MISSING_HANDLER:
+                                      raise AnsibleError(msg)
+                                  else:
+                                      display.warning(msg)
 
                     if 'add_host' in result_item:
                         # this task added a new host (add_host module)
@@ -533,6 +533,8 @@ class StrategyBase:
 
             results = self._process_pending_results(iterator)
             ret_results.extend(results)
+            if self._pending_results > 0:
+                time.sleep(C.DEFAULT_INTERNAL_POLL_INTERVAL)
 
         display.debug("no more pending results, returning what we have")
 

lib/ansible/plugins/strategy/free.py

@@ -21,6 +21,7 @@ __metaclass__ = type
 
 import time
 
+from ansible import constants as C
 from ansible.errors import AnsibleError
 from ansible.playbook.included_file import IncludedFile
 from ansible.plugins import action_loader
@@ -55,7 +56,7 @@ class StrategyModule(StrategyBase):
         # the last host to be given a task
         last_host = 0
 
-        result = True
+        result = self._tqm.RUN_OK
 
         work_to_do = True
         while work_to_do and not self._tqm._terminated:
@@ -174,7 +175,7 @@ class StrategyModule(StrategyBase):
                     variable_manager=self._variable_manager
                 )
             except AnsibleError as e:
-                return False
+                return self._tqm.RUN_ERROR
 
             if len(included_files) > 0:
                 all_blocks = dict((host, []) for host in hosts_left)
@@ -202,7 +203,7 @@ class StrategyModule(StrategyBase):
                 display.debug("done adding collected blocks to iterator")
 
             # pause briefly so we don't spin lock
-            time.sleep(0.001)
+            time.sleep(C.DEFAULT_INTERNAL_POLL_INTERVAL)
 
         # collect all the final results
         results = self._wait_on_pending_results(iterator)

lib/ansible/plugins/strategy/linear.py

@@ -295,13 +295,12 @@ class StrategyModule(StrategyBase):
                             if 'skipped' in include_result and include_result['skipped'] or 'failed' in include_result and include_result['failed']:
                                 continue
 
-                            role_vars = include_result.get('include_variables', dict())
-                            if loop_var and loop_var in include_result:
-                                role_vars[loop_var] = include_result[loop_var]
-
                             display.debug("generating all_blocks data for role")
                             new_ir = hr._task.copy()
-                            new_ir.args.update(role_vars)
+                            new_ir.vars.update(include_result.get('include_variables', dict()))
+                            if loop_var and loop_var in include_result:
+                                new_ir.vars[loop_var] = include_result[loop_var]
+
                             all_role_blocks.extend(new_ir.get_block_list(play=iterator._play, variable_manager=self._variable_manager, loader=self._loader))
 
                 if len(all_role_blocks) > 0:
@@ -395,7 +394,8 @@ class StrategyModule(StrategyBase):
                 if any_errors_fatal and (len(failed_hosts) > 0 or len(unreachable_hosts) > 0):
                     for host in hosts_left:
                         (s, _) = iterator.get_next_task_for_host(host, peek=True)
-                        if s.run_state != iterator.ITERATING_RESCUE:
+                        if s.run_state != iterator.ITERATING_RESCUE or \
+                           s.run_state == iterator.ITERATING_RESCUE and s.fail_state & iterator.FAILED_RESCUE != 0:
                             self._tqm._failed_hosts[host.name] = True
                             result |= self._tqm.RUN_FAILED_BREAK_PLAY
                 display.debug("done checking for any_errors_fatal")
@@ -415,6 +415,13 @@ class StrategyModule(StrategyBase):
                         result |= self._tqm.RUN_FAILED_BREAK_PLAY
                 display.debug("done checking for max_fail_percentage")
 
+                display.debug("checking to see if all hosts have failed and the running result is not ok")
+                if result != self._tqm.RUN_OK and len(self._tqm._failed_hosts) >= len(hosts_left):
+                    display.debug("^ not ok, so returning result now")
+                    self._tqm.send_callback('v2_playbook_on_no_hosts_remaining')
+                    return result
+                display.debug("done checking to see if all hosts have failed")
+
             except (IOError, EOFError) as e:
                 display.debug("got IOError/EOFError in task loop: %s" % e)
                 # most likely an abort, return failed

lib/ansible/plugins/test/core.py

@@ -121,17 +121,13 @@ class TestModule(object):
         return {
             # failure testing
             'failed'    : failed,
-            'failure'   : failed,
-            'success'   : success,
             'succeeded' : success,
 
             # changed testing
             'changed' : changed,
-            'change'  : changed,
 
             # skip testing
             'skipped' : skipped,
-            'skip'    : skipped,
 
             # regex
             'match': match,

lib/ansible/release.py

@@ -19,5 +19,5 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-__version__ = '2.2.0'
+__version__ = '2.2.0.0'
 __author__  = 'Ansible, Inc.'

lib/ansible/utils/cmd_functions.py

@@ -24,7 +24,7 @@ import shlex
 import subprocess
 import select
 
-from ansible.compat.six import PY2
+from ansible.compat.six import PY2, PY3
 from ansible.module_utils._text import to_bytes
 
 def run_cmd(cmd, live=False, readsize=10):
@@ -51,7 +51,11 @@ def run_cmd(cmd, live=False, readsize=10):
         if p.stdout in rfd:
             dat = os.read(p.stdout.fileno(), readsize)
             if live:
-                sys.stdout.write(dat)
+                # On python3, stdout has a codec to go from text type to bytes
+                if PY3:
+                    sys.stdout.buffer.write(dat)
+                else:
+                    sys.stdout.write(dat)
             stdout += dat
             if dat == b'':
                 rpipes.remove(p.stdout)
@@ -59,7 +63,11 @@ def run_cmd(cmd, live=False, readsize=10):
             dat = os.read(p.stderr.fileno(), readsize)
             stderr += dat
             if live:
-                sys.stdout.write(dat)
+                # On python3, stdout has a codec to go from text type to bytes
+                if PY3:
+                    sys.stdout.buffer.write(dat)
+                else:
+                    sys.stdout.write(dat)
             if dat == b'':
                 rpipes.remove(p.stderr)
         # only break out if we've emptied the pipes, or there is nothing to

lib/ansible/utils/encrypt.py

@@ -67,7 +67,7 @@ except ImportError:
 
 from ansible import constants as C
 from ansible.errors import AnsibleError
-from ansible.module_utils._text import to_text
+from ansible.module_utils._text import to_text, to_bytes
 
 __all__ = ['do_encrypt']
 
@@ -84,6 +84,10 @@ def do_encrypt(result, encrypt, salt_size=None, salt=None):
         if salt_size:
             result = crypt.encrypt(result, salt_size=salt_size)
         elif salt:
+            if crypt._salt_is_bytes:
+                salt = to_bytes(salt, encoding='ascii', errors='strict')
+            else:
+                salt = to_text(salt, encoding='ascii', errors='strict')
             result = crypt.encrypt(result, salt=salt)
         else:
             result = crypt.encrypt(result)

lib/ansible/utils/helpers.py

@@ -1,4 +1,4 @@
-# (c) 2016, Ansible by Red Hat <support@ansible.com>
+# (c) 2016, Ansible by Red Hat <info@ansible.com>
 #
 # This file is part of Ansible
 #

lib/ansible/utils/module_docs_fragments/asa.py

@@ -90,18 +90,6 @@ options:
         met either by individual arguments or values in this dict.
     required: false
     default: null
-  show_command:
-    description:
-      - Specifies which command will be used to get the current configuration.
-        By default the C(show running-config) command will be used, this command
-        masks some passwords. For example passwords for VPN. If you need to
-        match against masked passwords use C(more system:running-config).
-        Note that the C(more system:running-config) only works in the system
-        context if you are running the ASA in multiple context mode.
-        before sending any commands.
-    required: false
-    default: show running-config
-    choices: ['show running-config', 'more system:running-config']
   context:
     description:
       - Specifies which context to target if you are running in the ASA in

lib/ansible/vars/__init__.py

@@ -459,7 +459,7 @@ class VariableManager:
 
             templar.set_available_variables(vars_copy)
             delegated_host_name = templar.template(task.delegate_to, fail_on_undefined=False)
-            if not delegated_host_name:
+            if delegated_host_name is None:
                 raise AnsibleError(message="Undefined delegate_to host for task:", obj=task._ds)
             if delegated_host_name in delegated_host_vars:
                 # no need to repeat ourselves, as the delegate_to value

packaging/debian/changelog

@@ -2,76 +2,83 @@ ansible (%VERSION%-%RELEASE%~%DIST%) %DIST%; urgency=low
 
   * %VERSION% release
 
- -- Ansible, Inc. <support@ansible.com>  %DATE%
+ -- Ansible, Inc. <info@ansible.com>  %DATE%
+
+ansible (2.2.0.0) unstable; urgency=low
+
+  * 2.2.0.0
+
+ -- Ansible, Inc. <info@ansible.com>  Mon, 31 Oct 2016 22:20:37 -0500
+
 
 ansible (2.0.1.0) unstable; urgency=low
 
   * 2.0.1.0
 
- -- Ansible, Inc. <support@ansible.com>  Wed, 24 Feb 2016 18:28:59 -0500
+ -- Ansible, Inc. <info@ansible.com>  Wed, 24 Feb 2016 18:28:59 -0500
 
 
 ansible (2.0.0.2) unstable; urgency=low
 
   * 2.0.0.2
 
- -- Ansible, Inc. <support@ansible.com>  Thu, 14 Jan 2016 17:17:41 -0500
+ -- Ansible, Inc. <info@ansible.com>  Thu, 14 Jan 2016 17:17:41 -0500
 
 
 ansible (2.0.0.1) unstable; urgency=low
 
   * 2.0.0.1
 
- -- Ansible, Inc. <support@ansible.com>  Tue, 12 Jan 2016 17:53:29 -0500
+ -- Ansible, Inc. <info@ansible.com>  Tue, 12 Jan 2016 17:53:29 -0500
 
 
 ansible (2.0.0.0) unstable; urgency=low
 
   * 2.0.0.0
 
- -- Ansible, Inc. <support@ansible.com>  Tue, 12 Jan 2016 08:33:59 -0500
+ -- Ansible, Inc. <info@ansible.com>  Tue, 12 Jan 2016 08:33:59 -0500
 
 ansible (1.9.4) unstable; urgency=low
 
   * 1.9.4
 
- -- Ansible, Inc. <support@ansible.com>  Fri, 09 Oct 2015 15:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Fri, 09 Oct 2015 15:00:00 -0500
 
 ansible (1.9.3) unstable; urgency=low
 
   * 1.9.3
 
- -- Ansible, Inc. <support@ansible.com>  Thu, 03 Sep 2015 18:30:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Thu, 03 Sep 2015 18:30:00 -0500
 
 ansible (1.9.2) unstable; urgency=low
 
   * 1.9.2
 
- -- Ansible, Inc. <support@ansible.com>  Wed, 24 Jun 2015 14:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Wed, 24 Jun 2015 14:00:00 -0500
 
 ansible (1.9.1) unstable; urgency=low
 
   * 1.9.1
 
- -- Ansible, Inc. <support@ansible.com>  Mon, 27 Apr 2015 17:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Mon, 27 Apr 2015 17:00:00 -0500
 
 ansible (1.9.0.1) unstable; urgency=low
 
   * 1.9.0.1
 
- -- Ansible, Inc. <support@ansible.com>  Wed, 25 Mar 2015 15:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Wed, 25 Mar 2015 15:00:00 -0500
 
 ansible (1.8.4) unstable; urgency=low
 
   * 1.8.4
 
- -- Ansible, Inc. <support@ansible.com>  Thu, 19 Feb 2015 12:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Thu, 19 Feb 2015 12:00:00 -0500
 
 ansible (1.8.3) unstable; urgency=low
 
   * 1.8.3
 
- -- Ansible, Inc. <support@ansible.com>  Tue, 17 Feb 2015 16:00:00 -0500
+ -- Ansible, Inc. <info@ansible.com>  Tue, 17 Feb 2015 16:00:00 -0500
 
 ansible (1.8.2) unstable; urgency=low
 

packaging/debian/control

@@ -2,7 +2,7 @@ Source: ansible
 Section: admin
 Priority: optional
 Standards-Version: 3.9.3
-Maintainer: Ansible, Inc. <support@ansible.com>
+Maintainer: Ansible, Inc. <info@ansible.com>
 Build-Depends: cdbs, debhelper (>= 5.0.0), asciidoc, python, dh-python | python-support, python-setuptools
 Homepage: http://ansible.github.com/
 

packaging/release/release.yml

@@ -7,7 +7,7 @@
     release_date: "{{lookup('pipe', 'date +\"%m-%d-%Y\"')}}"
     rpm_spec_line: |
 
-      * {{lookup('pipe', 'date +"%a %b %d %Y"')}} Ansible, Inc. <support@ansible.com> - {{ansible_release_version}}-{{ansible_release_string}}
+      * {{lookup('pipe', 'date +"%a %b %d %Y"')}} Ansible, Inc. <info@ansible.com> - {{ansible_release_version}}-{{ansible_release_string}}
       - Release {{ansible_release_version}}-{{ansible_release_string}}
     deb_changelog_line: |
 
@@ -15,7 +15,7 @@
 
         * {{ansible_release_version}}
 
-       -- Ansible, Inc. <support@ansible.com>  {{lookup('pipe', 'date -R')}}
+       -- Ansible, Inc. <info@ansible.com>  {{lookup('pipe', 'date -R')}}
 
   vars_prompt:
   - name: ansible_release_branch
@@ -99,7 +99,7 @@
         dest: "{{release_dir}}/packaging/debian/changelog"
         regexp: "^ansible ({{ansible_release_version}})"
         line: "{{deb_changelog_line}}"
-        insertafter: "-- Ansible, Inc. <support@ansible.com>  %DATE%"
+        insertafter: "-- Ansible, Inc. <info@ansible.com>  %DATE%"
     - name: Update RELEASES.txt
       template:
         dest: "{{release_dir}}/RELEASES.txt"

packaging/rpm/ansible.spec

@@ -125,37 +125,40 @@ rm -rf %{buildroot}
 
 %changelog
 
-* Wed Feb 24 2016 Ansible, Inc. <support@ansible.com> - 2.0.1.0-1
+* Mon Oct 31 2016 Ansible, Inc. <info@ansible.com> - 2.2.0.0-1
+- Release 2.2.0.0-1
+
+* Wed Feb 24 2016 Ansible, Inc. <info@ansible.com> - 2.0.1.0-1
 - Release 2.0.1.0-1
 
-* Thu Jan 14 2016 Ansible, Inc. <support@ansible.com> - 2.0.0.2-1
+* Thu Jan 14 2016 Ansible, Inc. <info@ansible.com> - 2.0.0.2-1
 - Release 2.0.0.2-1
 
-* Tue Jan 12 2016 Ansible, Inc. <support@ansible.com> - 2.0.0.1-1
+* Tue Jan 12 2016 Ansible, Inc. <info@ansible.com> - 2.0.0.1-1
 - Release 2.0.0.1-1
 
-* Tue Jan 12 2016 Ansible, Inc. <support@ansible.com> - 2.0.0.0-1
+* Tue Jan 12 2016 Ansible, Inc. <info@ansible.com> - 2.0.0.0-1
 - Release 2.0.0.0-1
 
-* Fri Oct 09 2015 Ansible, Inc. <support@ansible.com> - 1.9.4
+* Fri Oct 09 2015 Ansible, Inc. <info@ansible.com> - 1.9.4
 - Release 1.9.4
 
-* Thu Sep 03 2015 Ansible, Inc. <support@ansible.com> - 1.9.3
+* Thu Sep 03 2015 Ansible, Inc. <info@ansible.com> - 1.9.3
 - Release 1.9.3
 
-* Wed Jun 24 2015 Ansible, Inc. <support@ansible.com> - 1.9.2
+* Wed Jun 24 2015 Ansible, Inc. <info@ansible.com> - 1.9.2
 - Release 1.9.2
 
-* Mon Apr 27 2015 Ansible, Inc. <support@ansible.com> - 1.9.1
+* Mon Apr 27 2015 Ansible, Inc. <info@ansible.com> - 1.9.1
 - Release 1.9.1
 
-* Wed Mar 25 2015 Ansible, Inc. <support@ansible.com> - 1.9.0
+* Wed Mar 25 2015 Ansible, Inc. <info@ansible.com> - 1.9.0
 - Release 1.9.0
 
-* Thu Feb 19 2015 Ansible, Inc. <support@ansible.com> - 1.8.4
+* Thu Feb 19 2015 Ansible, Inc. <info@ansible.com> - 1.8.4
 - Release 1.8.4
 
-* Tue Feb 17 2015 Ansible, Inc. <support@ansible.com> - 1.8.3
+* Tue Feb 17 2015 Ansible, Inc. <info@ansible.com> - 1.8.3
 - Release 1.8.3
 
 * Thu Dec 04 2014 Michael DeHaan <michael@ansible.com> - 1.8.2

setup.py

@@ -17,7 +17,7 @@ setup(name='ansible',
       version=__version__,
       description='Radically simple IT automation',
       author=__author__,
-      author_email='support@ansible.com',
+      author_email='info@ansible.com',
       url='http://ansible.com/',
       license='GPLv3',
       # Ansible will also make use of a system copy of python-six if installed but use a

test/code-smell/use-compat-six.sh

@@ -2,8 +2,13 @@
 
 # Do we want to check dynamic inventory, bin, etc?
 BASEDIR=${1-"lib"}
+# REGEX of individual files where the import of six is not a problem
+# digital_ocean is checking for six because dopy doesn't specify the
+#   requirement on six so it needs to try importing six to give the correct error
+#   message
+WHITELIST='(lib/ansible/modules/core/cloud/digital_ocean/digital_ocean.py)'
 
-SIX_USERS=$(find "$BASEDIR" -name '*.py' -exec grep -wH six \{\} \;|grep import |grep -v ansible.compat| grep -v ansible.module_utils.six)
+SIX_USERS=$(find "$BASEDIR" -name '*.py' -exec grep -wH six \{\} \;|grep import |grep -v ansible.compat| grep -v ansible.module_utils.six| egrep -v "^$WHITELIST:")
 if test -n "$SIX_USERS" ; then
   printf "$SIX_USERS"
   exit 1

test/integration/destructive.yml

@@ -9,6 +9,7 @@
     - { role: test_pip, tags: test_pip, when: ansible_fips != True }
     - { role: test_gem, tags: test_gem }
     - { role: test_yum, tags: test_yum, when: ansible_python.version.major == '2' }
+    - { role: test_dnf, tags: test_dnf }
     - { role: test_apt, tags: test_apt }
     - { role: test_apt_repository, tags: [test_apt_repository, test_apt_key] }
     - { role: test_postgresql, tags: [test_postgresql, test_postgresql_db, test_postgresql_privs, test_postgresql_user, needs_privileged] }

test/integration/integration_config.yml

@@ -2,7 +2,7 @@
 win_output_dir: 'C:\ansible_testing'
 output_dir: ~/ansible_testing
 non_root_test_user: ansible
-pip_test_package: epdb
+pip_test_package: isort
 
 # variables used in precedence tests, here passed to -e
 etest: 'from -e'

test/integration/roles/test_dnf/meta/main.yml

@@ -0,0 +1,2 @@
+dependencies:
+  - prepare_tests

test/integration/roles/test_dnf/tasks/dnf.yml

@@ -0,0 +1,203 @@
+# Ensure that python2-dnf is installed (This test is only like this in
+# stable-2.2 because we have not backported the feature that ensures the python
+# bindings are installed)
+- name:
+  command: dnf install -y python2-dnf
+
+# UNINSTALL
+#   With 'python2-dnf' uninstalled, the first call to 'dnf' should install
+#   python2-dnf.
+- name: uninstall sos
+  dnf:
+    name: sos
+    state: removed
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_result
+
+- debug: var=dnf_result
+- debug: var=rpm_result
+
+- name: verify uninstallation of sos
+  assert:
+    that:
+        - "not dnf_result.failed | default(False)"
+        - "rpm_result.rc == 1"
+
+# UNINSTALL AGAIN
+- name: uninstall sos
+  dnf:
+    name: sos
+    state: removed
+  register: dnf_result
+
+- name: verify no change on re-uninstall
+  assert:
+    that:
+        - "not dnf_result.changed"
+
+# INSTALL
+- name: install sos
+  dnf:
+    name: sos
+    state: present
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_result
+
+- debug: var=dnf_result
+- debug: var=rpm_result
+
+- name: verify installation of sos
+  assert:
+    that:
+        - "not dnf_result.failed | default(False)"
+        - "dnf_result.changed"
+        - "rpm_result.rc == 0"
+
+- name: verify dnf module outputs
+  assert:
+    that:
+        - "'changed' in dnf_result"
+        - "'results' in dnf_result"
+
+# INSTALL AGAIN
+- name: install sos again
+  dnf:
+    name: sos
+    state: present
+  register: dnf_result
+
+- name: verify no change on second install
+  assert:
+    that:
+        - "not dnf_result.changed"
+
+# Multiple packages
+- name: uninstall sos and sharutils
+  dnf: name=sos,sharutils state=removed
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_sos_result
+
+- name: check sharutils with rpm
+  shell: rpm -q sharutils
+  failed_when: False
+  register: rpm_sharutils_result
+
+- name: verify packages installed
+  assert:
+    that:
+        - "rpm_sos_result.rc != 0"
+        - "rpm_sharutils_result.rc != 0"
+
+- name: install sos and sharutils as comma separated
+  dnf: name=sos,sharutils state=present
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_sos_result
+
+- name: check sharutils with rpm
+  shell: rpm -q sharutils
+  failed_when: False
+  register: rpm_sharutils_result
+
+- name: verify packages installed
+  assert:
+    that:
+        - "not dnf_result.failed | default(False)"
+        - "dnf_result.changed"
+        - "rpm_sos_result.rc == 0"
+        - "rpm_sharutils_result.rc == 0"
+
+- name: uninstall sos and sharutils
+  dnf: name=sos,sharutils state=removed
+  register: dnf_result
+
+- name: install sos and sharutils as list
+  dnf:
+    name:
+      - sos
+      - sharutils
+    state: present
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_sos_result
+
+- name: check sharutils with rpm
+  shell: rpm -q sharutils
+  failed_when: False
+  register: rpm_sharutils_result
+
+- name: verify packages installed
+  assert:
+    that:
+        - "not dnf_result.failed | default(False)"
+        - "dnf_result.changed"
+        - "rpm_sos_result.rc == 0"
+        - "rpm_sharutils_result.rc == 0"
+
+- name: uninstall sos and sharutils
+  dnf:
+    name: "sos,sharutils"
+    state: removed
+  register: dnf_result
+
+- name: install sos and sharutils as comma separated with spaces
+  dnf:
+    name: "sos, sharutils"
+    state: present
+  register: dnf_result
+
+- name: check sos with rpm
+  shell: rpm -q sos
+  failed_when: False
+  register: rpm_sos_result
+
+- name: check sos with rpm
+  shell: rpm -q sharutils
+  failed_when: False
+  register: rpm_sharutils_result
+
+- name: verify packages installed
+  assert:
+    that:
+        - "not dnf_result.failed | default(False)"
+        - "dnf_result.changed"
+        - "rpm_sos_result.rc == 0"
+        - "rpm_sharutils_result.rc == 0"
+
+- name: uninstall sos and sharutils
+  dnf:
+    name:
+      - sos
+      - sharutils
+    state: removed
+
+- name: install non-existent rpm 
+  dnf:
+    name: "{{ item }}"
+  with_items:
+    - does-not-exist
+  register: non_existent_rpm
+  ignore_errors: True
+
+- name: check non-existent rpm install failed
+  assert:
+    that:
+      - non_existent_rpm|failed

test/integration/roles/test_dnf/tasks/main.yml

@@ -0,0 +1,23 @@
+# test code for the yum module
+# (c) 2014, James Tanner <tanner.jc@gmail.com>
+
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# Note: We install the yum package onto Fedora so that this will work on dnf systems
+# We want to test that for people who don't want to upgrade their systems.
+- include: 'dnf.yml'
+  when: (ansible_distribution in ['RedHat', 'CentOS', 'ScientificLinux'] and False) or (ansible_distribution in ['Fedora'] and ansible_distribution_major_version >= 23)
+

test/integration/roles/test_filters/files/foo.txt

@@ -11,7 +11,7 @@ Dumping the same structure to YAML
 
 Dumping the same structure to JSON, but don't pretty print
 
-["this is a list element", {"this": "is a hash element in a list", "where": "endor", "warp": 9}]
+["this is a list element", {"this": "is a hash element in a list", "warp": 9, "where": "endor"}]
 
 Dumping the same structure to YAML, but don't pretty print
 

test/integration/roles/test_filters/templates/foo.j2

@@ -7,7 +7,7 @@ Dumping the same structure to YAML
 
 Dumping the same structure to JSON, but don't pretty print
 
-{{ some_structure | to_json }}
+{{ some_structure | to_json(sort_keys=true) }}
 
 Dumping the same structure to YAML, but don't pretty print
 

test/integration/roles/test_git/tasks/main.yml

@@ -31,6 +31,7 @@
     repo_update_url_1: 'https://github.com/ansible-test-robinro/git-test-old'
     repo_update_url_2: 'https://github.com/ansible-test-robinro/git-test-new'
     repo_depth_url: 'https://github.com/ansible-test-robinro/git-test-shallow-depth'
+    repo_verify: 'https://github.com/pixelrebel/ansible-git-test.git'
     known_host_files:
       - "{{ lookup('env','HOME') }}/.ssh/known_hosts"
       - '/etc/ssh/ssh_known_hosts'
@@ -46,6 +47,10 @@
   shell: git --version | grep 'git version' | sed 's/git version //'
   register: git_version
 
+- name: get gpg version
+  shell: gpg --version 2>1 | head -1 | sed -e 's/gpg (GnuPG) //'
+  register: gpg_version
+
 - name: set dummy git config
   shell: git config --global user.email "noreply@example.com"; git config --global user.name "Ansible Test Runner"
 
@@ -659,3 +664,57 @@
 - name: clear checkout_dir
   file: state=absent path={{ checkout_dir }}
 
+
+  # Test for tag verification
+  # clone a repo checkout signed tag, verify tag
+
+- name: Import Jamie Evans GPG key
+  command: gpg --keyserver pgp.mit.edu --recv-key 61107C8E
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))
+  
+- name: Copy ownertrust
+  copy: "content='2D55902D66FEEBCEA4447C93E79A36DA61107C8E:6:\n' dest=/tmp/ownertrust-git.txt"
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))
+
+- name: Import ownertrust
+  command: gpg --import-ownertrust /tmp/ownertrust-git.txt
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))
+
+- name: Clone signed repo and verify tag
+  git: repo={{ repo_verify }} dest={{ checkout_dir }} version=v0.0 verify_commit=yes
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))
+
+- name: Remove Jamie Evans GPG key
+  command: gpg --batch --yes --delete-key 61107C8E
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))
+
+- name: Clean up files
+  file: path="{{ item }}" state=absent
+  with_items:
+    - "{{ checkout_dir }}"
+    - /tmp/ownertrust-git.txt
+  when: >
+    not gpg_version.stderr and
+    gpg_version.stdout and
+    (git_version.stdout | version_compare("2.1.0", '>=') or
+    gpg_version.stdout | version_compare("1.4.16", '>='))

test/integration/roles/test_uri/files/testserver.py

@@ -1,7 +1,17 @@
-import mimetypes
+import sys
 
 if __name__ == '__main__':
-    mimetypes.init()
-    mimetypes.add_type('application/json', '.json')
-    import SimpleHTTPServer
-    SimpleHTTPServer.test()
+    if sys.version_info[0] >= 3:
+        import http.server
+        import socketserver
+        PORT = int(sys.argv[1])
+        Handler = http.server.SimpleHTTPRequestHandler
+        httpd = socketserver.TCPServer(("", PORT), Handler)
+        httpd.serve_forever()
+    else:
+        import mimetypes
+        mimetypes.init()
+        mimetypes.add_type('application/json', '.json')
+        import SimpleHTTPServer
+        SimpleHTTPServer.test()
+

test/integration/roles/test_win_async_wrapper/tasks/main.yml

@@ -138,33 +138,31 @@
     - asyncresult | failed == true
     - asyncresult.msg is search('failing via exception')
 
+- name: loop async success
+  async_test:
+    sleep_delay_sec: 3
+  async: 10
+  poll: 0
+  with_sequence: start=1 end=4
+  register: async_many
 
-# this test is only flaky under shippable/AWS- the last iteration "until's" forever, cannot repro locally
-#- name: loop async success
-#  async_test:
-#    sleep_delay_sec: 3
-#  async: 10
-#  poll: 0
-#  with_sequence: start=1 end=4
-#  register: async_many
-#
-#- name: wait for completion
-#  async_status:
-#    jid: "{{ item }}"
-#  register: asyncout
-#  until: asyncout.finished == 1
-#  retries: 10
-#  delay: 1
-#  with_items: "{{ async_many.results | map(attribute='ansible_job_id') | list }}"
-#
-#- name: validate results
-#  assert:
-#    that:
-#    - item.finished == 1
-#    - item.slept_sec == 3
-#    - item.changed == true
-#    - item.ansible_job_id is match('\d+\.\d+')
-#  with_items: "{{ asyncout.results }}"
+- name: wait for completion
+  async_status:
+    jid: "{{ item }}"
+  register: asyncout
+  until: asyncout.finished == 1
+  retries: 10
+  delay: 1
+  with_items: "{{ async_many.results | map(attribute='ansible_job_id') | list }}"
+
+- name: validate results
+  assert:
+    that:
+    - item.finished == 1
+    - item.slept_sec == 3
+    - item.changed == true
+    - item.ansible_job_id is match('\d+\.\d+')
+  with_items: "{{ asyncout.results }}"
 
 # this part of the test is flaky- Windows PIDs are reused aggressively, so this occasionally fails due to a new process with the same ID
 # FUTURE: consider having the test module hook to a kernel object we can poke at that gets signaled/released on exit

test/integration/roles/test_win_command/tasks/main.yml

@@ -124,11 +124,14 @@
     - cmdout|failed
     - cmdout.rc == 254
 
-- name: write large buffer to stdout
-  win_command: powershell /c "$ba = New-Object byte[] 16384; (New-Object System.Random 32).NextBytes($ba); [Convert]::ToBase64String($ba) | Write-Output"
+- name: interleave large writes between stdout/stderr (check for buffer consumption deadlock)
+  win_command: powershell /c "$ba = New-Object byte[] 4096; (New-Object System.Random 32).NextBytes($ba); $text = [Convert]::ToBase64String($ba); Write-Output startout; Write-Error starterror; Write-Error $text; Write-Output $text; Write-Error $text; Write-Output $text; Write-Error $text; Write-Output $text; Write-Output doneout Write-Error doneerror"
   register: cmdout
 
-# TODO: fix small buffer deadlock on large write to stderr before stdout has been consumed
-#- name: write large buffer to stderr
-#  win_shell: $ba = New-Object byte[] 16384; (New-Object System.Random 32).NextBytes($ba); [Convert]::ToBase64String($ba) | Write-Error; Write-Output test
-#  register: cmdout
+- name: ensure that the entirety of both streams were read
+  assert:
+    that:
+    - cmdout.stdout is search("startout")
+    - cmdout.stdout is search("doneout")
+    - cmdout.stderr is search("starterror")
+    - cmdout.stderr is search("doneerror")

test/integration/roles/test_win_shell/tasks/main.yml

@@ -165,11 +165,14 @@
     - shellout.stdout_lines == ["line1 ", "line2"]
     - shellout.stderr == ""
 
-- name: write large buffer to stdout
-  win_shell: $ba = New-Object byte[] 16384; (New-Object System.Random 32).NextBytes($ba); [Convert]::ToBase64String($ba) | Write-Output
+- name: interleave large writes between stdout/stderr (check for buffer consumption deadlock)
+  win_shell: $ba = New-Object byte[] 4096; (New-Object System.Random 32).NextBytes($ba); $text = [Convert]::ToBase64String($ba); Write-Output startout; Write-Error starterror; Write-Error $text; Write-Output $text; Write-Error $text; Write-Output $text; Write-Error $text; Write-Output $text; Write-Output doneout Write-Error doneerror
   register: shellout
 
-# TODO: fix small buffer deadlock on large write to stderr before stdout has been consumed
-#- name: write large buffer to stderr
-#  win_shell: $ba = New-Object byte[] 16384; (New-Object System.Random 32).NextBytes($ba); [Convert]::ToBase64String($ba) | Write-Error; Write-Output test
-#  register: shellout
+- name: ensure that the entirety of both streams were read
+  assert:
+    that:
+    - shellout.stdout is search("startout")
+    - shellout.stdout is search("doneout")
+    - shellout.stderr is search("starterror")
+    - shellout.stderr is search("doneerror")

test/sanity/validate-modules/README.rst

@@ -0,0 +1,70 @@
+validate-modules
+================
+
+Python program to help test or validate Ansible modules.
+
+Originally developed by Matt Martz (@sivel)
+
+Usage
+~~~~~
+
+.. code:: shell
+
+    cd /path/to/ansible/source
+    source hacking/env-setup
+    test/sanity/validate-modules/validate-modules /path/to/modules
+
+Help
+~~~~
+
+.. code:: shell
+
+    usage: validate-modules [-h] [-w] [--exclude EXCLUDE] modules
+
+    positional arguments:
+      modules            Path to module or module directory
+
+    optional arguments:
+      -h, --help         show this help message and exit
+      -w, --warnings     Show warnings
+      --exclude EXCLUDE  RegEx exclusion pattern
+
+Current Validations
+===================
+
+Modules
+~~~~~~~
+
+Errors
+^^^^^^
+
+#. Interpreter line is not ``#!/usr/bin/python``
+#. ``main()`` not at the bottom of the file
+#. Module does not import ``ansible.module_utils.basic``
+#. Missing ``DOCUMENTATION``
+#. Documentation is invalid YAML
+#. Invalid schema for ``DOCUMENTATION``
+#. Missing ``EXAMPLES``
+#. Invalid Python Syntax
+#. Tabbed indentation
+#. Use of ``sys.exit()`` instead of ``exit_json`` or ``fail_json``
+#. Missing GPLv3 license header in module
+#. PowerShell module missing ``WANT_JSON``
+#. PowerShell module missing ``POWERSHELL_COMMON``
+#. New modules have the correct ``version_added``
+#. New arguments have the correct ``version_added``
+#. Modules should not import requests, instead use ``ansible.module_utils.urls``
+#. Missing ``RETURN`` for new modules
+
+Warnings
+^^^^^^^^
+
+#. Try/Except ``HAS_`` expression missing
+#. Missing ``RETURN`` for existing modules
+#. ``import json`` found
+#. Module contains duplicate globals from basic.py
+
+Module Directories (Python Packages)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+#. Missing ``__init__.py``

test/sanity/validate-modules/__init__.py

@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2015 Matt Martz <matt@sivel.net>
+# Copyright (C) 2015 Rackspace US, Inc.
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+__version__ = '0.0.1b'

test/sanity/validate-modules/module_args.py

@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2016 Matt Martz <matt@sivel.net>
+# Copyright (C) 2016 Rackspace US, Inc.
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import imp
+import sys
+
+from modulefinder import ModuleFinder
+
+import mock
+
+
+MODULE_CLASSES = [
+    'ansible.module_utils.basic.AnsibleModule',
+    'ansible.module_utils.vca.VcaAnsibleModule',
+    'ansible.module_utils.nxos.NetworkModule',
+    'ansible.module_utils.eos.NetworkModule',
+    'ansible.module_utils.ios.NetworkModule',
+    'ansible.module_utils.iosxr.NetworkModule',
+    'ansible.module_utils.junos.NetworkModule',
+    'ansible.module_utils.openswitch.NetworkModule',
+]
+
+
+class AnsibleModuleCallError(RuntimeError):
+    pass
+
+
+def add_mocks(filename):
+    gp = mock.patch('ansible.module_utils.basic.get_platform').start()
+    gp.return_value = 'linux'
+
+    module_mock = mock.MagicMock()
+    mocks = []
+    for module_class in MODULE_CLASSES:
+        mocks.append(
+            mock.patch('ansible.module_utils.basic.AnsibleModule',
+                       new=module_mock)
+        )
+    for m in mocks:
+        p = m.start()
+        p.side_effect = AnsibleModuleCallError()
+
+    finder = ModuleFinder()
+    try:
+        finder.run_script(filename)
+    except:
+        pass
+
+    sys_mock = mock.MagicMock()
+    sys_mock.__version__ = '0.0.0'
+    sys_mocks = []
+    for module, sources in finder.badmodules.items():
+        if module in sys.modules:
+            continue
+        if [s for s in sources if s[:7] in ['ansible', '__main_']]:
+            parts = module.split('.')
+            for i in range(len(parts)):
+                dotted = '.'.join(parts[:i+1])
+                sys.modules[dotted] = sys_mock
+                sys_mocks.append(dotted)
+
+    return module_mock, mocks, sys_mocks
+
+
+def remove_mocks(mocks, sys_mocks):
+    for m in mocks:
+        m.stop()
+
+    for m in sys_mocks:
+        try:
+            del sys.modules[m]
+        except KeyError:
+            pass
+
+
+def get_argument_spec(filename):
+    module_mock, mocks, sys_mocks = add_mocks(filename)
+
+    try:
+        mod = imp.load_source('module', filename)
+        if not module_mock.call_args:
+            mod.main()
+    except AnsibleModuleCallError:
+        pass
+    except Exception:
+        # We can probably remove this branch, it is here for use while testing
+        pass
+
+    remove_mocks(mocks, sys_mocks)
+
+    try:
+        args, kwargs = module_mock.call_args
+        try:
+            return kwargs['argument_spec']
+        except KeyError:
+            return args[0]
+    except TypeError:
+        return {}

test/sanity/validate-modules/schema.py

@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2015 Matt Martz <matt@sivel.net>
+# Copyright (C) 2015 Rackspace US, Inc.
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from voluptuous import ALLOW_EXTRA, Any, Required, Schema
+
+option_schema = Schema(
+    {
+        Required('description'): Any(basestring, [basestring]),
+        'required': bool,
+        'choices': list,
+        'aliases': list,
+        'version_added': Any(basestring, float)
+    },
+    extra=ALLOW_EXTRA
+)
+
+doc_schema = Schema(
+    {
+        Required('module'): basestring,
+        'short_description': basestring,
+        'description': Any(basestring, [basestring]),
+        'version_added': Any(basestring, float),
+        'author': Any(None, basestring, [basestring]),
+        'notes': Any(None, [basestring]),
+        'requirements': [basestring],
+        'options': Any(None, dict),
+        'extends_documentation_fragment': Any(basestring, [basestring])
+    },
+    extra=ALLOW_EXTRA
+)

test/sanity/validate-modules/utils.py

@@ -0,0 +1,68 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2015 Matt Martz <matt@sivel.net>
+# Copyright (C) 2015 Rackspace US, Inc.
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import ast
+import sys
+
+# We only use StringIO, since we cannot setattr on cStringIO
+from StringIO import StringIO
+
+
+def find_globals(g, tree):
+    """Uses AST to find globals in an ast tree"""
+    for child in tree:
+        if hasattr(child, 'body') and isinstance(child.body, list):
+            find_globals(g, child.body)
+        elif isinstance(child, (ast.FunctionDef, ast.ClassDef)):
+            g.add(child.name)
+            continue
+        elif isinstance(child, ast.Assign):
+            try:
+                g.add(child.targets[0].id)
+            except (IndexError, AttributeError):
+                pass
+        elif isinstance(child, ast.Import):
+            g.add(child.names[0].name)
+        elif isinstance(child, ast.ImportFrom):
+            for name in child.names:
+                g_name = name.asname or name.name
+                if g_name == '*':
+                    continue
+                g.add(g_name)
+
+
+class CaptureStd():
+    """Context manager to handle capturing stderr and stdout"""
+
+    def __enter__(self):
+        self.sys_stdout = sys.stdout
+        self.sys_stderr = sys.stderr
+        sys.stdout = self.stdout = StringIO()
+        sys.stderr = self.stderr = StringIO()
+        setattr(sys.stdout, 'encoding', self.sys_stdout.encoding)
+        setattr(sys.stderr, 'encoding', self.sys_stderr.encoding)
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        sys.stdout = self.sys_stdout
+        sys.stderr = self.sys_stderr
+
+    def get(self):
+        """Return ``(stdout, stderr)``"""
+
+        return self.stdout.getvalue(), self.stderr.getvalue()

test/sanity/validate-modules/validate-modules

@@ -0,0 +1,680 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2015 Matt Martz <matt@sivel.net>
+# Copyright (C) 2015 Rackspace US, Inc.
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import print_function
+
+import abc
+import argparse
+import ast
+import os
+import re
+import sys
+
+from distutils.version import StrictVersion
+from fnmatch import fnmatch
+
+from ansible import __version__ as ansible_version
+from ansible.executor.module_common import REPLACER_WINDOWS
+from ansible.plugins import module_loader
+from ansible.utils.module_docs import BLACKLIST_MODULES, get_docstring
+
+from module_args import get_argument_spec
+
+from schema import doc_schema, option_schema
+
+from utils import CaptureStd
+
+import yaml
+
+
+BLACKLIST_DIRS = frozenset(('.git', 'test', '.github', '.idea'))
+INDENT_REGEX = re.compile(r'([\t]*)')
+BLACKLIST_IMPORTS = {
+    'requests': {
+        'new_only': True,
+        'msg': ('requests import found, should use '
+                'ansible.module_utils.urls instead')
+    },
+    'boto(?:\.|$)': {
+        'new_only': True,
+        'msg': ('boto import found, new modules should use boto3')
+    },
+}
+
+
+class Validator(object):
+    """Validator instances are intended to be run on a single object.  if you
+    are scanning multiple objects for problems, you'll want to have a separate
+    Validator for each one."""
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self):
+        self.reset()
+
+    def reset(self):
+        """Reset the test results"""
+        self.errors = []
+        self.warnings = []
+        self.traces = []
+        self.warning_traces = []
+
+    @abc.abstractproperty
+    def object_name(self):
+        """Name of the object we validated"""
+        pass
+
+    @abc.abstractproperty
+    def object_path(self):
+        """Path of the object we validated"""
+        pass
+
+    @abc.abstractmethod
+    def validate(self, reset=True):
+        """Run this method to generate the test results"""
+        if reset:
+            self.reset()
+
+    def report(self, warnings=False):
+        """Print out the test results"""
+        if self.errors or (warnings and self.warnings):
+            print('=' * 76)
+            print(self.object_path)
+            print('=' * 76)
+
+        ret = []
+
+        traces = self.traces[:]
+        if warnings and self.warnings:
+            traces.extend(self.warning_traces)
+
+        for trace in traces:
+            print('TRACE:')
+            print('\n    '.join(('    %s' % trace).splitlines()))
+        for error in self.errors:
+            print('ERROR: %s' % error)
+            ret.append(1)
+        if warnings:
+            for warning in self.warnings:
+                print('WARNING: %s' % warning)
+                # ret.append(1)  # Don't incrememt exit status for warnings
+
+        if self.errors or (warnings and self.warnings):
+            print()
+
+        return len(ret)
+
+
+class ModuleValidator(Validator):
+    BLACKLIST_PATTERNS = ('.git*', '*.pyc', '*.pyo', '.*', '*.md', '*.txt')
+    BLACKLIST_FILES = frozenset(('.git', '.gitignore', '.travis.yml',
+                                 'shippable.yml',
+                                 '.gitattributes', '.gitmodules', 'COPYING',
+                                 '__init__.py', 'VERSION', 'test-docs.sh'))
+    BLACKLIST = BLACKLIST_FILES.union(BLACKLIST_MODULES)
+
+    PS_DOC_BLACKLIST = frozenset((
+        'async_status.ps1',
+        'slurp.ps1',
+        'setup.ps1'
+    ))
+
+    def __init__(self, path, analyze_arg_spec=False):
+        super(ModuleValidator, self).__init__()
+
+        self.path = path
+        self.basename = os.path.basename(self.path)
+        self.name, _ = os.path.splitext(self.basename)
+
+        self.analyze_arg_spec = analyze_arg_spec
+
+        self._python_module_override = False
+
+        with open(path) as f:
+            self.text = f.read()
+        self.length = len(self.text.splitlines())
+        try:
+            self.ast = ast.parse(self.text)
+        except:
+            self.ast = None
+
+    @property
+    def object_name(self):
+        return self.basename
+
+    @property
+    def object_path(self):
+        return self.path
+
+    def _python_module(self):
+        if self.path.endswith('.py') or self._python_module_override:
+            return True
+        return False
+
+    def _powershell_module(self):
+        if self.path.endswith('.ps1'):
+            return True
+        return False
+
+    def _just_docs(self):
+        try:
+            for child in self.ast.body:
+                if not isinstance(child, ast.Assign):
+                    return False
+            return True
+        except AttributeError:
+            return False
+
+    def _is_new_module(self):
+        if self.name.startswith("_") and not os.path.islink(self.name):
+            # This is a deprecated module, so look up the *old* name
+            return not module_loader.has_plugin(self.name[1:])
+        else:
+            return not module_loader.has_plugin(self.name)
+
+    def _check_interpreter(self, powershell=False):
+        if powershell:
+            if not self.text.startswith('#!powershell\n'):
+                self.errors.append('Interpreter line is not "#!powershell"')
+            return
+
+        if not self.text.startswith('#!/usr/bin/python'):
+            self.errors.append('Interpreter line is not "#!/usr/bin/python"')
+
+    def _check_for_sys_exit(self):
+        if 'sys.exit(' in self.text:
+            self.errors.append('sys.exit() call found. Should be '
+                               'exit_json/fail_json')
+
+    def _check_for_gpl3_header(self):
+        if ('GNU General Public License' not in self.text and
+                'version 3' not in self.text):
+            self.errors.append('GPLv3 license header not found')
+
+    def _check_for_tabs(self):
+        for line_no, line in enumerate(self.text.splitlines()):
+            indent = INDENT_REGEX.search(line)
+            if indent and '\t' in line:
+                index = line.index('\t')
+                self.errors.append('indentation contains tabs. line %d '
+                                   'column %d' % (line_no + 1, index))
+
+    def _find_blacklist_imports(self):
+        for child in self.ast.body:
+            names = []
+            if isinstance(child, ast.Import):
+                names.extend(child.names)
+            elif isinstance(child, ast.TryExcept):
+                bodies = child.body
+                for handler in child.handlers:
+                    bodies.extend(handler.body)
+                for grandchild in bodies:
+                    if isinstance(grandchild, ast.Import):
+                        names.extend(grandchild.names)
+            for name in names:
+                for blacklist_import, options in BLACKLIST_IMPORTS.items():
+                    if re.search(blacklist_import, name.name):
+                        msg = options['msg']
+                        new_only = options['new_only']
+                        if self._is_new_module() and new_only:
+                            self.errors.append(msg)
+                        elif not new_only:
+                            self.errors.append(msg)
+
+    def _find_module_utils(self, main):
+        linenos = []
+        found_basic = False
+        for child in self.ast.body:
+            if isinstance(child, (ast.Import, ast.ImportFrom)):
+                names = []
+                try:
+                    names.append(child.module)
+                    if child.module.endswith('.basic'):
+                        found_basic = True
+                except AttributeError:
+                    pass
+                names.extend([n.name for n in child.names])
+
+                if [n for n in names if n.startswith('ansible.module_utils')]:
+                    linenos.append(child.lineno)
+
+                    for name in child.names:
+                        if (isinstance(name, ast.alias) and
+                                name.name == 'basic'):
+                            found_basic = True
+
+        if not linenos:
+            self.errors.append('Did not find a module_utils import')
+        elif not found_basic:
+            self.warnings.append('Did not find "ansible.module_utils.basic" '
+                               'import')
+
+        return linenos
+
+    def _find_main_call(self):
+        lineno = False
+        if_bodies = []
+        for child in self.ast.body:
+            if isinstance(child, ast.If):
+                try:
+                    if child.test.left.id == '__name__':
+                        if_bodies.extend(child.body)
+                except AttributeError:
+                    pass
+
+        bodies = self.ast.body
+        bodies.extend(if_bodies)
+
+        for child in bodies:
+            if isinstance(child, ast.Expr):
+                if isinstance(child.value, ast.Call):
+                    if (isinstance(child.value.func, ast.Name) and
+                            child.value.func.id == 'main'):
+                        lineno = child.lineno
+                        if lineno < self.length - 1:
+                            self.errors.append('Call to main() not the last '
+                                               'line')
+
+        if not lineno:
+            self.errors.append('Did not find a call to main')
+
+        return lineno or 0
+
+    def _find_has_import(self):
+        for child in self.ast.body:
+            found_try_except_import = False
+            found_has = False
+            if isinstance(child, ast.TryExcept):
+                bodies = child.body
+                for handler in child.handlers:
+                    bodies.extend(handler.body)
+                for grandchild in bodies:
+                    if isinstance(grandchild, ast.Import):
+                        found_try_except_import = True
+                    if isinstance(grandchild, ast.Assign):
+                        for target in grandchild.targets:
+                            if target.id.lower().startswith('has_'):
+                                found_has = True
+            if found_try_except_import and not found_has:
+                self.warnings.append('Found Try/Except block without HAS_ '
+                                     'assginment')
+
+    def _find_ps_replacers(self):
+        if 'WANT_JSON' not in self.text:
+            self.errors.append('WANT_JSON not found in module')
+
+        if REPLACER_WINDOWS not in self.text:
+            self.errors.append('"%s" not found in module' % REPLACER_WINDOWS)
+
+    def _find_ps_docs_py_file(self):
+        if self.object_name in self.PS_DOC_BLACKLIST:
+            return
+        py_path = self.path.replace('.ps1', '.py')
+        if not os.path.isfile(py_path):
+            self.errors.append('Missing python documentation file')
+
+    def _get_docs(self):
+        docs = {
+            'DOCUMENTATION': {
+                'value': None,
+                'lineno': 0
+            },
+            'EXAMPLES': {
+                'value': None,
+                'lineno': 0
+            },
+            'RETURN': {
+                'value': None,
+                'lineno': 0
+            }
+        }
+        for child in self.ast.body:
+            if isinstance(child, ast.Assign):
+                for grandchild in child.targets:
+                    if grandchild.id == 'DOCUMENTATION':
+                        docs['DOCUMENTATION']['value'] = child.value.s
+                        docs['DOCUMENTATION']['lineno'] = child.lineno
+                    elif grandchild.id == 'EXAMPLES':
+                        docs['EXAMPLES']['value'] = child.value.s[1:]
+                        docs['EXAMPLES']['lineno'] = child.lineno
+                    elif grandchild.id == 'RETURN':
+                        docs['RETURN']['value'] = child.value.s
+                        docs['RETURN']['lineno'] = child.lineno
+
+        return docs
+
+    def _validate_docs_schema(self, doc):
+        errors = []
+        try:
+            doc_schema(doc)
+        except Exception as e:
+            errors.extend(e.errors)
+
+        options = doc.get('options', {})
+        for key, option in (options or {}).iteritems():
+            try:
+                option_schema(option)
+            except Exception as e:
+                for error in e.errors:
+                    error.path[:0] = ['options', key]
+                errors.extend(e.errors)
+
+        for error in errors:
+            path = [str(p) for p in error.path]
+            self.errors.append('DOCUMENTATION.%s: %s' %
+                               ('.'.join(path), error.error_message))
+
+    def _validate_docs(self):
+        doc_info = self._get_docs()
+        try:
+            doc = yaml.safe_load(doc_info['DOCUMENTATION']['value'])
+        except yaml.YAMLError as e:
+            doc = None
+            # This offsets the error line number to where the
+            # DOCUMENTATION starts so we can just go to that line in the
+            # module
+            e.problem_mark.line += (
+                doc_info['DOCUMENTATION']['lineno'] - 1
+            )
+            e.problem_mark.name = '%s.DOCUMENTATION' % self.name
+            self.traces.append(e)
+            self.errors.append('DOCUMENTATION is not valid YAML. Line %d '
+                               'column %d' %
+                               (e.problem_mark.line + 1,
+                                e.problem_mark.column + 1))
+        except AttributeError:
+            self.errors.append('No DOCUMENTATION provided')
+        else:
+            with CaptureStd():
+                try:
+                    get_docstring(self.path, verbose=True)
+                except AssertionError:
+                    fragment = doc['extends_documentation_fragment']
+                    self.errors.append('DOCUMENTATION fragment missing: %s' %
+                                       fragment)
+                except Exception as e:
+                    self.traces.append(e)
+                    self.errors.append('Unknown DOCUMENTATION error, see '
+                                       'TRACE')
+
+            self._validate_docs_schema(doc)
+            self._check_version_added(doc)
+            self._check_for_new_args(doc)
+
+        if not bool(doc_info['EXAMPLES']['value']):
+            self.errors.append('No EXAMPLES provided')
+
+        if not bool(doc_info['RETURN']['value']):
+            if self._is_new_module():
+                self.errors.append('No RETURN documentation provided')
+            else:
+                self.warnings.append('No RETURN provided')
+        else:
+            try:
+                yaml.safe_load(doc_info['RETURN']['value'])
+            except yaml.YAMLError as e:
+                e.problem_mark.line += (
+                    doc_info['RETURN']['lineno'] - 1
+                )
+                e.problem_mark.name = '%s.RETURN' % self.name
+                self.errors.append('RETURN is not valid YAML. Line %d '
+                                   'column %d' %
+                                   (e.problem_mark.line + 1,
+                                    e.problem_mark.column + 1))
+                self.traces.append(e)
+
+    def _check_version_added(self, doc):
+        if not self._is_new_module():
+            return
+
+        try:
+            version_added = StrictVersion(str(doc.get('version_added', '0.0')))
+        except ValueError:
+            version_added = doc.get('version_added', '0.0')
+            self.errors.append('version_added is not a valid version '
+                               'number: %r' % version_added)
+            return
+
+        should_be = '.'.join(ansible_version.split('.')[:2])
+        strict_ansible_version = StrictVersion(should_be)
+
+        if (version_added < strict_ansible_version or
+                strict_ansible_version < version_added):
+            self.errors.append('version_added should be %s. Currently %s' %
+                               (should_be, version_added))
+
+    def _validate_argument_spec(self):
+        if not self.analyze_arg_spec:
+            return
+        spec = get_argument_spec(self.path)
+        for arg, data in spec.items():
+            if data.get('required') and data.get('default', object) != object:
+                self.errors.append('"%s" is marked as required but specifies '
+                                   'a default. Arguments with a default '
+                                   'should not be marked as required' % arg)
+
+    def _check_for_new_args(self, doc):
+        if self._is_new_module():
+            return
+
+        with CaptureStd():
+            try:
+                existing = module_loader.find_plugin(self.name, mod_type='.py')
+                existing_doc, _, _ = get_docstring(existing, verbose=True)
+                existing_options = existing_doc.get('options', {})
+            except AssertionError:
+                fragment = doc['extends_documentation_fragment']
+                self.warnings.append('Pre-existing DOCUMENTATION fragment '
+                                     'missing: %s' % fragment)
+                return
+            except Exception as e:
+                self.warning_traces.append(e)
+                self.warnings.append('Unknown pre-existing DOCUMENTATION '
+                                     'error, see TRACE. Submodule refs may '
+                                     'need updated')
+                return
+
+        try:
+            mod_version_added = StrictVersion(
+                str(existing_doc.get('version_added', '0.0'))
+            )
+        except ValueError:
+            mod_version_added = StrictVersion('0.0')
+
+        options = doc.get('options', {})
+
+        should_be = '.'.join(ansible_version.split('.')[:2])
+        strict_ansible_version = StrictVersion(should_be)
+
+        for option, details in options.iteritems():
+            new = not bool(existing_options.get(option))
+            if not new:
+                continue
+
+            try:
+                version_added = StrictVersion(
+                    str(details.get('version_added', '0.0'))
+                )
+            except ValueError:
+                version_added = details.get('version_added', '0.0')
+                self.errors.append('version_added for new option (%s) '
+                                   'is not a valid version number: %r' %
+                                   (option, version_added))
+                continue
+            except:
+                # If there is any other exception it should have been caught
+                # in schema validation, so we won't duplicate errors by
+                # listing it again
+                continue
+
+            if (strict_ansible_version != mod_version_added and
+                    (version_added < strict_ansible_version or
+                     strict_ansible_version < version_added)):
+                self.errors.append('version_added for new option (%s) should '
+                                   'be %s. Currently %s' %
+                                   (option, should_be, version_added))
+
+    def validate(self):
+        super(ModuleValidator, self).validate()
+
+        # Blacklists -- these files are not checked
+        if not frozenset((self.basename,
+                          self.name)).isdisjoint(self.BLACKLIST):
+            return
+        for pat in self.BLACKLIST_PATTERNS:
+            if fnmatch(self.basename, pat):
+                return
+
+#        if self._powershell_module():
+#            self.warnings.append('Cannot check powershell modules at this '
+#                                 'time.  Skipping')
+#            return
+        if not self._python_module() and not self._powershell_module():
+            self.errors.append('Official Ansible modules must have a .py '
+                               'extension for python modules or a .ps1 '
+                               'for powershell modules')
+            self._python_module_override = True
+
+        if self._python_module() and self.ast is None:
+            self.errors.append('Python SyntaxError while parsing module')
+            try:
+                compile(self.text, self.path, 'exec')
+            except Exception as e:
+                self.traces.append(e)
+            return
+
+        if self._python_module():
+            self._validate_docs()
+
+        if self._python_module() and not self._just_docs():
+            self._validate_argument_spec()
+            self._check_for_sys_exit()
+            self._find_blacklist_imports()
+            main = self._find_main_call()
+            self._find_module_utils(main)
+            self._find_has_import()
+            self._check_for_tabs()
+
+        if self._powershell_module():
+            self._find_ps_replacers()
+            self._find_ps_docs_py_file()
+
+        self._check_for_gpl3_header()
+        if not self._just_docs():
+            self._check_interpreter(powershell=self._powershell_module())
+
+
+class PythonPackageValidator(Validator):
+    BLACKLIST_FILES = frozenset(('__pycache__',))
+
+    def __init__(self, path):
+        super(PythonPackageValidator, self).__init__()
+
+        self.path = path
+        self.basename = os.path.basename(path)
+
+    @property
+    def object_name(self):
+        return self.basename
+
+    @property
+    def object_path(self):
+        return self.path
+
+    def validate(self):
+        super(PythonPackageValidator, self).validate()
+
+        if self.basename in self.BLACKLIST_FILES:
+            return
+
+        init_file = os.path.join(self.path, '__init__.py')
+        if not os.path.exists(init_file):
+            self.errors.append('Ansible module subdirectories must contain an '
+                               '__init__.py')
+
+
+def re_compile(value):
+    """
+    Argparse expects things to raise TypeError, re.compile raises an re.error
+    exception
+
+    This function is a shorthand to convert the re.error exception to a
+    TypeError
+    """
+
+    try:
+        return re.compile(value)
+    except re.error as e:
+        raise TypeError(e)
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('modules', nargs='+',
+                        help='Path to module or module directory')
+    parser.add_argument('-w', '--warnings', help='Show warnings',
+                        action='store_true')
+    parser.add_argument('--exclude', help='RegEx exclusion pattern',
+                        type=re_compile)
+    parser.add_argument('--arg-spec', help='Analyze module argument spec',
+                        action='store_true', default=False)
+    args = parser.parse_args()
+
+    args.modules[:] = [m.rstrip('/') for m in args.modules]
+
+    exit = []
+
+    for module in args.modules:
+        if os.path.isfile(module):
+            path = module
+            if args.exclude and args.exclude.search(path):
+                sys.exit(0)
+            mv = ModuleValidator(path, analyze_arg_spec=args.arg_spec)
+            mv.validate()
+            exit.append(mv.report(args.warnings))
+
+        for root, dirs, files in os.walk(module):
+            basedir = root[len(module)+1:].split('/', 1)[0]
+            if basedir in BLACKLIST_DIRS:
+                continue
+            for dirname in dirs:
+                if root == module and dirname in BLACKLIST_DIRS:
+                    continue
+                path = os.path.join(root, dirname)
+                if args.exclude and args.exclude.search(path):
+                    continue
+                pv = PythonPackageValidator(path)
+                pv.validate()
+                exit.append(pv.report(args.warnings))
+
+            for filename in files:
+                path = os.path.join(root, filename)
+                if args.exclude and args.exclude.search(path):
+                    continue
+                mv = ModuleValidator(path, analyze_arg_spec=args.arg_spec)
+                mv.validate()
+                exit.append(mv.report(args.warnings))
+
+    sys.exit(sum(exit))
+
+
+if __name__ == '__main__':
+    try:
+        main()
+    except KeyboardInterrupt:
+        pass

test/units/module_utils/basic/test_run_command.py

@@ -25,6 +25,7 @@ import sys
 import time
 from io import BytesIO, StringIO
 
+from ansible.compat.six import PY3
 from ansible.compat.tests import unittest
 from ansible.compat.tests.mock import call, MagicMock, Mock, patch, sentinel
 
@@ -173,13 +174,24 @@ class TestAnsibleModuleRunCommand(unittest.TestCase):
         self.cmd_out[sentinel.stdout] = BytesIO(b'hello')
         (rc, stdout, stderr) = self.module.run_command('/bin/cat hello.txt')
         self.assertEqual(rc, 0)
-        self.assertEqual(stdout, b'hello')
+        # module_utils function.  On py3 it returns text and py2 it returns
+        # bytes because it's returning native strings
+        if PY3:
+            self.assertEqual(stdout, u'hello')
+        else:
+            self.assertEqual(stdout, b'hello')
 
     def test_utf8_output(self):
         self.cmd_out[sentinel.stdout] = BytesIO(u'Žarn§'.encode('utf-8'))
         self.cmd_out[sentinel.stderr] = BytesIO(u'لرئيسية'.encode('utf-8'))
         (rc, stdout, stderr) = self.module.run_command('/bin/something_ugly')
         self.assertEqual(rc, 0)
-        self.assertEqual(stdout.decode('utf-8'), u'Žarn§')
-        self.assertEqual(stderr.decode('utf-8'), u'لرئيسية')
+        # module_utils function.  On py3 it returns text and py2 it returns
+        # bytes because it's returning native strings
+        if PY3:
+            self.assertEqual(stdout, u'Žarn§')
+            self.assertEqual(stderr, u'لرئيسية')
+        else:
+            self.assertEqual(stdout.decode('utf-8'), u'Žarn§')
+            self.assertEqual(stderr.decode('utf-8'), u'لرئيسية')
 

test/units/module_utils/test_distribution_version.py

@@ -172,6 +172,24 @@ ID_LIKE="suse"
                    'os_family': u'Suse',
                    'distribution_version': u'13.2'}
     },
+    {
+        "platform.dist": [
+            "", 
+            "", 
+            ""
+        ], 
+        "input": {
+            "/etc/os-release": "NAME=\"openSUSE Tumbleweed\"\n# VERSION=\"20160917\"\nID=opensuse\nID_LIKE=\"suse\"\nVERSION_ID=\"20160917\"\nPRETTY_NAME=\"openSUSE Tumbleweed\"\nANSI_COLOR=\"0;32\"\nCPE_NAME=\"cpe:/o:opensuse:tumbleweed:20160917\"\nBUG_REPORT_URL=\"https://bugs.opensuse.org\"\nHOME_URL=\"https://www.opensuse.org/\"\n"
+        }, 
+        "name": "openSUSE Tumbleweed 20160917", 
+        "result": {
+            "distribution_release": "NA", 
+            "distribution": "openSUSE Tumbleweed", 
+            "distribution_major_version": "NA", 
+            "os_family": "Suse", 
+            "distribution_version": "20160917"
+        }
+    },
     { # see https://github.com/ansible/ansible/issues/14837
         "name": "SLES 11.3",
         "input": {

test/units/module_utils/test_facts.py

@@ -405,7 +405,7 @@ MTAB_ENTRIES = \
 
 BIND_MOUNTS = ['/not/a/real/bind_mount']
 
-FINDMNT_OUTPUT = b"""
+FINDMNT_OUTPUT = u"""
 /sys                            sysfs                                sysfs           rw,nosuid,nodev,noexec,relatime,seclabel
 /proc                           proc                                 proc            rw,nosuid,nodev,noexec,relatime
 /dev                            devtmpfs                             devtmpfs        rw,nosuid,seclabel,size=8044400k,nr_inodes=2011100,mode=755

test/units/playbook/test_play_context.py

@@ -157,7 +157,7 @@ class TestPlayContext(unittest.TestCase):
 
         play_context.become_method = 'pbrun'
         cmd = play_context.make_become_cmd(cmd=default_cmd, executable="/bin/bash")
-        self.assertEqual(cmd, """%s -b %s -u %s 'echo %s; %s'""" % (pbrun_exe, pbrun_flags, play_context.become_user, play_context.success_key, default_cmd))
+        self.assertEqual(cmd, """%s %s -u %s 'echo %s; %s'""" % (pbrun_exe, pbrun_flags, play_context.become_user, play_context.success_key, default_cmd))
 
         play_context.become_method = 'pfexec'
         cmd = play_context.make_become_cmd(cmd=default_cmd, executable="/bin/bash")

test/units/plugins/connections/test_connection_ssh.py

@@ -308,6 +308,19 @@ class TestConnectionBaseClass(unittest.TestCase):
         conn._run.return_value = (0, '', '')
         conn.host = "some_host"
 
+        # Test with C.DEFAULT_SCP_IF_SSH set to smart
+        # Test when SFTP works
+        C.DEFAULT_SCP_IF_SSH = 'smart'
+        expected_in_data = b' '.join((b'put', to_bytes(pipes.quote('/path/to/in/file')), to_bytes(pipes.quote('/path/to/dest/file')))) + b'\n'
+        conn.put_file('/path/to/in/file', '/path/to/dest/file')
+        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+
+        # Test when SFTP doesn't work but SCP does
+        conn._run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
+        conn.put_file('/path/to/in/file', '/path/to/dest/file')
+        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._run.side_effect = None
+
         # test with C.DEFAULT_SCP_IF_SSH enabled
         C.DEFAULT_SCP_IF_SSH = True
         conn.put_file('/path/to/in/file', '/path/to/dest/file')
@@ -328,6 +341,7 @@ class TestConnectionBaseClass(unittest.TestCase):
         conn.put_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
         conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
+
         # test that a non-zero rc raises an error
         conn._run.return_value = (1, 'stdout', 'some errors')
         self.assertRaises(AnsibleError, conn.put_file, '/path/to/bad/file', '/remote/path/to/file')
@@ -348,25 +362,38 @@ class TestConnectionBaseClass(unittest.TestCase):
         conn._run.return_value = (0, '', '')
         conn.host = "some_host"
 
+        # Test with C.DEFAULT_SCP_IF_SSH set to smart
+        # Test when SFTP works
+        C.DEFAULT_SCP_IF_SSH = 'smart'
+        expected_in_data = b' '.join((b'get', to_bytes(pipes.quote('/path/to/in/file')), to_bytes(pipes.quote('/path/to/dest/file')))) + b'\n'
+        conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
+        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
+
+        # Test when SFTP doesn't work but SCP does
+        conn._run.side_effect = [(1, 'stdout', 'some errors'), (0, '', '')]
+        conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
+        conn._run.assert_called_with('some command to run', None, checkrc=False)
+        conn._run.side_effect = None
+
         # test with C.DEFAULT_SCP_IF_SSH enabled
         C.DEFAULT_SCP_IF_SSH = True
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', None)
+        conn._run.assert_called_with('some command to run', None, checkrc=False)
 
         conn.fetch_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', None)
+        conn._run.assert_called_with('some command to run', None, checkrc=False)
 
         # test with C.DEFAULT_SCP_IF_SSH disabled
         C.DEFAULT_SCP_IF_SSH = False
         expected_in_data = b' '.join((b'get', to_bytes(pipes.quote('/path/to/in/file')), to_bytes(pipes.quote('/path/to/dest/file')))) + b'\n'
         conn.fetch_file('/path/to/in/file', '/path/to/dest/file')
-        conn._run.assert_called_with('some command to run', expected_in_data)
+        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         expected_in_data = b' '.join((b'get',
             to_bytes(pipes.quote('/path/to/in/file/with/unicode-fö〩')),
             to_bytes(pipes.quote('/path/to/dest/file/with/unicode-fö〩')))) + b'\n'
         conn.fetch_file(u'/path/to/in/file/with/unicode-fö〩', u'/path/to/dest/file/with/unicode-fö〩')
-        conn._run.assert_called_with('some command to run', expected_in_data)
+        conn._run.assert_called_with('some command to run', expected_in_data, checkrc=False)
 
         # test that a non-zero rc raises an error
         conn._run.return_value = (1, 'stdout', 'some errors')

test/units/vars/test_variable_manager.py

@@ -20,17 +20,20 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 from collections import defaultdict
-from six import iteritems
 
+from ansible.compat.six import iteritems
+from ansible.compat.six.moves import builtins
 from ansible.compat.tests import unittest
-from ansible.compat.tests.mock import patch, MagicMock
+from ansible.compat.tests.mock import MagicMock, mock_open, patch
 from ansible.inventory import Inventory
 from ansible.playbook.play import Play
-from ansible.vars import VariableManager
 
 from units.mock.loader import DictDataLoader
 from units.mock.path import mock_unfrackpath_noop
 
+from ansible.vars import VariableManager
+
+
 class TestVariableManager(unittest.TestCase):
 
     def setUp(self):
@@ -192,9 +195,7 @@ class TestVariableManager(unittest.TestCase):
         v = VariableManager()
         v._fact_cache = defaultdict(dict)
 
-        fake_loader = DictDataLoader({
-            # inventory1
-            '/etc/ansible/inventory1': """
+        inventory1_filedata = """
             [group2:children]
             group1
 
@@ -206,8 +207,11 @@ class TestVariableManager(unittest.TestCase):
 
             [group2:vars]
             group_var = group_var_from_inventory_group2
-            """,
+            """
 
+        fake_loader = DictDataLoader({
+            # inventory1
+            '/etc/ansible/inventory1': inventory1_filedata,
             # role defaults_only1
             '/etc/ansible/roles/defaults_only1/defaults/main.yml': """
             default_var: "default_var_from_defaults_only1"
@@ -231,7 +235,8 @@ class TestVariableManager(unittest.TestCase):
         })
 
         mock_basedir.return_value = './'
-        inv1 = Inventory(loader=fake_loader, variable_manager=v, host_list='/etc/ansible/inventory1')
+        with patch.object(builtins, 'open', mock_open(read_data=inventory1_filedata)):
+            inv1 = Inventory(loader=fake_loader, variable_manager=v, host_list='/etc/ansible/inventory1')
         inv1.set_playbook_basedir('./')
 
         play1 = Play.load(dict(

test/utils/shippable/modules/generate-tests

@@ -230,12 +230,12 @@ def get_role_tags(playbook_path):
     return tags
 
 
-def get_changed_paths(git_root, branch='devel'):
+def get_changed_paths(git_root, branch='origin/stable-2.2'):
     """Get file paths changed in current branch vs given branch.
 
     Args:
         git_root: The root of the git clone.
-        branch: The branch to compare against (default: devel)
+        branch: The branch to compare against (default: origin/stable-2.2)
 
     Returns: List of file paths changed.
     """

test/utils/shippable/python3-test-tag-blacklist.txt

@@ -1,13 +1,3 @@
-test_apache2_module
-test_authorized_key
-test_filters
-test_gem
-test_git
 test_hg
-test_lookups
-test_pip
 test_service
-test_subversion
-test_synchronize
-test_unarchive
 test_uri

test/utils/shippable/python3-test-target-blacklist.txt

@@ -1 +0,0 @@
-s/ pull / /

test/utils/shippable/sanity-requirements.txt

@@ -2,3 +2,5 @@ tox
 pyyaml
 jinja2
 setuptools
+mock
+voluptuous==0.8.8