New release v2.4.1.0-1

(cherry picked from commit d7d1402bb7)

.github/ISSUE_TEMPLATE.md

@@ -13,7 +13,7 @@ Also test if the latest release, and master branch are affected too.
 <!--- Name of the module/plugin/task/feature -->
 
 ##### ANSIBLE VERSION
-<!--- Paste verbatim output from “ansible --version” between quotes below -->
+<!--- Paste verbatim output from "ansible --version" between quotes below -->
 ```
 
 ```
@@ -27,7 +27,7 @@ Mention any settings you have changed/added/removed in ansible.cfg
 ##### OS / ENVIRONMENT
 <!---
 Mention the OS you are running Ansible from, and the OS you are
-managing, or say “N/A” for anything that is not platform-specific.
+managing, or say "N/A" for anything that is not platform-specific.
 Also mention the specific version of what you are trying to control,
 e.g. if this is a network bug the version of firmware on the network device.
 -->

.github/PULL_REQUEST_TEMPLATE.md

@@ -18,7 +18,7 @@ the change does.
 <!--- Name of the module/plugin/module/task -->
 
 ##### ANSIBLE VERSION
-<!--- Paste verbatim output from “ansible --version” between quotes below -->
+<!--- Paste verbatim output from "ansible --version" between quotes below -->
 ```
 
 ```

.gitignore

@@ -38,6 +38,8 @@ docs/docsite/rst/partner_maintained.rst
 docs/docsite/rst/playbook_keywords.rst
 docs/docsite/rst/playbooks_directives.rst
 docs/docsite/*.html
+docs/docsite/rst/ansible.rst
+docs/docsite/rst/ansible-*.rst
 docs/docsite/_static/*.gif
 docs/docsite/_static/*.png
 docs/docsite/_static/websupport.js

CHANGELOG.md

@@ -1,7 +1,152 @@
 Ansible Changes By Release
 ==========================
 
-## 2.4 "Dancing Days" - ACTIVE DEVELOPMENT
+<a id="2.4.1"></a>
+
+## 2.4.1 "Dancing Days" - TBD
+
+### Bugfixes
+
+* Security fix for CVE-2017-7550 the jenkins_plugin module was logging the jenkins
+  server password if the url_password was passed via the params field:
+  https://github.com/ansible/ansible/pull/30875
+* Update openssl\* module documentation to show openssl-0.16 is the minimum version
+* Fix openssl_certificate's csr handling
+* Python-3 fixes
+  * Fix openssl_certificate parameter assertion on Python3
+  * Fix for python3 and nonascii strings in inventory plugins (https://github.com/ansible/ansible/pull/30666)
+  * Fix missing urllib in iam_policy
+  * Fix crypttab module for bytes<=>text string mismatch ( https://github.com/ansible/ansible/pull/30457 )
+  * Fix lxc_container module combining bytes with text ( https://github.com/ansible/ansible/pull/30572 )
+  * Fix map doesn't return a list on python3 in ec2_snapshot_facts module (https://github.com/ansible/ansible/pull/30606)
+  * Fix uri (and other url retrieving) modules when used with a proxy. (https://github.com/ansible/ansible/issues/31109)
+  * Fix azure_rm dynamic inventory script ConfigParser usage.
+* Fix for win_file to respect check mode when deleting directories
+* Fix for Ansible.ModuleUtils.Legacy.psm1 to return list params correctly
+* Fix for a proper logout in the module ovirt_vms
+* Fixed docs for 'password' lookup
+* Corrected and added missing feature and porting docs for 2.4
+* Fix for Ansible.ModuleUtils.CamelConversion to handle empty lists and lists with one entry
+* Fix nxos terminal regex to parse username correctly.
+* Fix colors for selective callback
+* Fix for 'New password' prompt on 'ansible-vault edit' (https://github.com/ansible/ansible/issues/30491)
+* Fix for 'ansible-vault encrypt' with vault_password_file in config and --ask-vault-pass cli (https://github.com/ansible/ansible/pull/30514#pullrequestreview-63395903)
+* updated porting guide with notes for callbacks and config
+* Added backwards compatiblity shim for callbacks that do not inherit from CallbackBase
+* Corrected issue with configuration and multiple ini entries being overwriten even when not set.
+* backported fix for doc generation (plugin_formatter)
+* Fix ec2_lc module for an unknown parameter name (https://github.com/ansible/ansible/pull/30573)
+* Change configuration of defaults to use standard jinja2 instead of custom
+  eval() for using variables in the default field of config (https://github.com/ansible/ansible/pull/30650)
+* added missing entry in chlog deprecation
+* Fixed precedence and values for become flags and executable settings
+* Fix for win_domain_membership to throw more helpful error messages and check/fix when calling WMI function after changing workgroup
+* Fix for win_power_plan to compare the OS version's correctly and work on Windows 10/Server 2016
+* Fix module doc for typo in telnet command option
+* Fix OpenBSD pkg_mgr fact (https://github.com/ansible/ansible/issues/30623)
+* Fix encoding error when there are nonascii values in the path to the ssh binary
+* removed YAML inventory group name validation, broke existing setups and should be global in any case, and configurable
+* performance improvment for inventory, had slown down considerably from 2.3
+* Fix cpu facts on sparc64 (https://github.com/ansible/ansible/pull/30261)
+* Fix ansible_distribution fact for Arch linux (https://github.com/ansible/ansible/issues/30600)
+* remove print statements from play_context/become
+* Fix vault errors after 'ansible-vault edit' (https://github.com/ansible/ansible/issues/30575)
+* updated api doc example to match api changes
+* corrected issues with slack callback plugin
+* it is import_playbook .. not import_plays .. docs now reflect this
+* fixed typo and missed include/import conversion in import_tasks docs
+* updated porting docs with note about inventory_dir
+* removed extension requirement for yaml inventory plugin to restore previous behaviour
+* fixed ansible-pull to now correctly deal with inventory
+* corrected dig lookup docs
+* fix type handling for sensu_silence so the module works
+* added fix for win_iis_webapppool to correctly handle array elements
+* Fix bugs caused by lack of collector ordering like service_mgr being incorrect (https://github.com/ansible/ansible/issues/30753)
+* Fix os_image when the id parameter is not set in the task. ( https://github.com/ansible/ansible/pull/29147 )
+* Fix for the winrm connection to use proper task vars
+* removed typo from dig lookup docs
+* Updated win_chocolatey example to be clearer around what should be used with become
+* Fix for copy module when permissions are changed but the file contents are not ( https://github.com/ansible/ansible/issues/30556 )
+* corrected YAML_FILENAME_EXTENSIONS ini setter as key/section were swapped
+* Better error message when a yaml inventory is invalid
+* avoid include_Xs conflating vars with options
+* Fix aws_s3 module handling `encrypt` option (https://github.com/ansible/ansible/pull/31203)
+* Fix for win_msg to document and show error when message is greater than 255 characters
+* Fix for win_dotnet_ngen to work after recent regression
+* fixed backwards compat method for config
+* removed docs for prematurely added ssh specific pipelining settings
+* fixed redis cache typo
+* Fix AttributeError during inventory group deserialization (https://github.com/ansible/ansible/issues/30903)
+* Fix 'ansible-vault encrypt --output=-' (https://github.com/ansible/ansible/issues/30550)
+* restore pre 2.4 pipeline configuration options (env and ini)
+* Fix win_copy regression: handling of vault-encrypted source files (https://github.com/ansible/ansible/pull/31084)
+* Updated return values for win_reg_stat to correctly show what is being returned (https://github.com/ansible/ansible/pull/31252)
+* reduced normal error redundancy and verbosity, display on increased and when needed
+* Give an informative error instead of a traceback if include_vars dir is file instead of directory (https://github.com/ansible/ansible/pull/31157)
+* Fix monit module's version check for color support (https://github.com/ansible/ansible/pull/31212)
+* Make `elasticsearch_plugin` module work with both 2.x and 5.x (https://github.com/ansible/ansible/issues/21989)
+* Fix for become on Windows to handle ignored errors (https://github.com/ansible/ansible/issues/30468)
+* Fix removal of newlines when writing SELinux config (https://github.com/ansible/ansible/issues/30618)
+* clarified extension requirement for constructed inv plugin
+* really turn off inventory caching, toggle will be added in 2.5
+* for inventory sources, dont follow symlinks to calculate base directory, used for group/host_vars
+* Port the uptime.py example script to the new inventory API.
+* inventory_file variable again returns full path, not just basename
+* added info about cwd group/host vars to porting guide
+* Fix name parsing out of envra in the yum module
+* give user friendly error on badly formatted yaml inventory source
+* Fix any_errors_fatal setting in playbooks.
+* Fix setting of ssh-extra-args from the cli (https://github.com/ansible/ansible/pull/31326)
+* Change SELinux fact behavior to always return a dictionary. (https://github.com/ansible/ansible/issues/18692)
+* Revert a fix for using non /bin/sh shells for modules' running commands as
+  this was causing output from commands to change, thus breaking playbooks.
+  See the original bug for details and links to the eventual fix:
+  https://github.com/ansible/ansible/issues/24169
+* Do not log data field in `docker_secrets` module (https://github.com/ansible/ansible/pull/31366)
+* Fix rpm_key taking the wrong 8 chars from the keyid (https://github.com/ansible/ansible/pull/31045)
+* chown errors now more informative
+* Fix for win_copy to copy a source file that has invalid windows characters in the filename, the dest still must be have valid windows characters (https://github.com/ansible/ansible/issues/31336#issuecomment-334649927)
+* Fix systemd module to not run daemon-reload in check mode.
+* fixed some parsing and selection issues with inventory manager, fixed minor bugs in yaml and constructed plugins
+* Fix the ping module documentation to reference win_ping instead of itself: https://github.com/ansible/ansible/pull/31444
+* Fix for ec2_win_password to allow blank key_passphrase again (https://github.com/ansible/ansible/pull/28791)
+* added toggle for vars_plugin behaviour to execute relative to playbook, set default to revert to previous way.
+* Fix for win_copy to not remove destination file on change when in check mode (https://github.com/ansible/ansible/pull/31469)
+* Fix include_role usage of role_name (https://github.com/ansible/ansible/pull/31463)
+* Fix service and package forcing a second run of the setup module to function (https://github.com/ansible/ansible/issues/31485)
+* Better error message when attempting to use include or import with /usr/bin/ansible (https://github.com/ansible/ansible/pull/31492/)
+* Fix `sysctl` module to remove etries when `state=absent` (https://github.com/ansible/ansible/issues/29920)
+* Fix for ec2_group to avoid trying to iterate over None (https://github.com/ansible/ansible/pull/31531)
+* Fix for ec2_group for a possible KeyError bug (https://github.com/ansible/ansible/pull/31540)
+* Fix for the rpm_key module when importing the first gpg key on a system (https://github.com/ansible/ansible/pull/31514)
+* Fix for aws_s3 metadata to use the correct parameters when uploading a file (https://github.com/ansible/ansible/issues/31232)
+* Fix for the yum module when installing from file/url crashes (https://github.com/ansible/ansible/pull/31529)
+* Improved error messaging for Windows become/runas when username is bogus (https://github.com/ansible/ansible/pull/31551)
+* Fix rollback feature in junos_config to now allow configuration rollback on device (https://github.com/ansible/ansible/pull/31424)
+* Remove command executed log from ansible-connection (https://github.com/ansible/ansible/pull/31581)
+* Fix relative paths to be relative to config file when there is no playbook available (https://github.com/ansible/ansible/issues/31533)
+* Fix Inventory plugins to use the configured inventory plugin path (https://github.com/ansible/ansible/issues/31605)
+* Fix include task to be dynamic (https://github.com/ansible/ansible/issues/31593)
+* A couple fixes to the test process to account for new testing resources in
+  our ci system and an upstream cryptography update that didn't work with
+  pip-8.x
+* Document backup_path in a few dellos modules and vyos_config (https://github.com/ansible/ansible/issues/31844)
+* Fix for vmware_vm_facts with dangling inaccessible VM which don't have MAC addresses (https://github.com/ansible/ansible/pull/31629)
+* Fix for win_regedit sending extra data that could confuse ansible's result parsing (https://github.com/ansible/ansible/pull/31813)
+* Fix git module to correctly cleanup temporary dirs (https://github.com/ansible/ansible/pull/31541)
+* Fix for modules which use atomic_move() to rename files raising an exception
+  if a file could not be opened.  Fix will return a nice error message instead:
+  https://github.com/ansible/ansible/issues/31786
+* Fix ansible-doc and ansible-console module-path option (https://github.com/ansible/ansible/pull/31744)
+* Fix for hostname module on RHEL 7.5 (https://github.com/ansible/ansible/issues/31811)
+
+### Known Bugs
+* Implicit localhost is getting ansible_connection from all:vars instead of
+  from the implicit localhost definition (https://github.com/ansible/ansible/issues/31420)
+
+<a id="2.4"></a>
+
+## 2.4 "Dancing Days" - 2017-09-19
 
 ### Major Changes
 
@@ -9,8 +154,6 @@ Ansible Changes By Release
 * New import/include keywords to replace the old bare `include` directives. The use of `static: {yes|no}` on such includes is now deprecated.
     - Using `import_*` (`import_playbook`, `import_tasks`, `import_role`) directives are static.
     - Using `include_*` (`include_tasks`, `include_role`) directives are dynamic.
-* Added fact namespacing, from now on facts will be available under `ansible_facts` namespace (i.e. `ansible_facts.ansible_os_distribution`).
-  They will continue to be added into the main namespace directly, but now a configuration toggle to disable this, currently off by default, in the future it will be on by default.
   This is done to avoid collisions and possible security issues as facts come from the remote targets and they might be compromised.
 * New `order` play level keyword that allows the user to change the order in which Ansible processes hosts when dispatching tasks.
 * Users can now set group merge priority for groups of the same depth (parent child relationship), using the new `ansible_group_priority` variable, when values are the same or don't exist it will fallback to the previous sorting by name'.
@@ -22,14 +165,14 @@ Ansible Changes By Release
   - New inventory plugins for creating inventory
   - Old inventory formats are still supported via plugins
   - Inline host_list is also an inventory plugin, an example alternative `advanced_host_list` is also provided (it supports ranges)
-  - New configuration option to list enabled plugins and precedence order: `whitelist_inventory` in ansible.cfg
+  - New configuration option to list enabled plugins and precedence order `[inventory]enable_plugins` in ansible.cfg
   - vars_plugins have been reworked, they are now run from Vars manager and API has changed (need docs)
   - Loading group_vars/host_vars is now a vars plugin and can be overridden
   - It is now possible to specify mulitple inventory sources in the command line (-i /etc/hosts1 -i /opt/hosts2)
   - Inventory plugins can use the cache plugin (i.e. virtualbox) and is affected by `meta: refresh_inventory`
   - Group variable precedence is now configurable via new 'precedence' option in ansible.cfg (needs docs)
   - Improved warnings and error messages across the board
-* Configuration has been changed from a hardcoded into the constants module to dynamically loaded from yaml definitions
+* Configuration has been changed from a hardcoded listing in the constants module to dynamically loaded from yaml definitions
   - Also added an ansible-config CLI to allow for listing config options and dumping current config (including origin)
   - TODO: build upon this to add many features detailed in ansible-config proposal https://github.com/ansible/proposals/issues/35
 * Windows modules now support the use of multiple shared module_utils files in the form of Powershell modules (.psm1), via `#Requires -Module Ansible.ModuleUtils.Whatever.psm1`
@@ -49,30 +192,33 @@ Ansible Changes By Release
   moved to `ansible.utils.unsafe_proxy` to avoid a circular import.
 * The win_get_url module has the dictionary 'win_get_url' in its results deprecated,
   its content is now also available directly in the resulting output, like other modules.
+* previouslly deprecated 'hostfile' config settings have been 're-deprecated' as before the code did not warn about deprecated configuration settings
+, but it does now.
 
 #### Deprecated Modules (to be removed in 2.8):
-* ec2_facts: replaced by ec2_metadata_facts
+* azure: use M(azure_rm_virtualmachine) instead
 * cs_nic: replaced by cs_instance_nic_secondaryip, also see new module cs_instance_nic for managing nics
+* ec2_facts: replaced by ec2_metadata_facts
+* ec2_remote_facts: replaced by ec2_instance_facts
 * panos_address: use M(panos_object) instead
-* panos_service: use M(panos_object) instead
-* panos_security_policy: use M(panos_security_rule) instead
 * panos_nat_policy: use M(panos_nat_rule) instead
+* panos_security_policy: use M(panos_security_rule) instead
+* panos_service: use M(panos_object) instead
 * s3: replaced by aws_s3
-* ec2_remote_facts: replaced by 
 * win_msi: use M(win_package) instead
 
-#### Removed Modules (previouslly deprecated):
+#### Removed Modules (previously deprecated):
 * eos_template: use eos_config instead
 * ios_template: use ios_config instead
 * iosxr_template: use iosxr_config instead
 * junos_template: use junos_config instead
 * nxos_template: use nxos_config instead
-* ops_template: use ops_config instead
 * openswitch
+* ops_template: use ops_config instead
 
 
 ### Minor Changes
-* Removed previously deprecated config option `hostfile` and env var `ANSIBLE_HOSTS`
+* Now deprecated configuration options issue warnings when set.
 * Removed unused and deprecated config option `pattern`
 * Updated the copy of six bundled for modules to use from 1.4.1 to 1.10.0
 * The `include_dir` var is not a global anymore, as we now allow multiple inventory sources, it is now host dependant.
@@ -130,21 +276,51 @@ Ansible Changes By Release
     - extensions of files to ignore when using inventory directory
 	- patterns of flies to ignore when using inventory directory
 	- option to toggle failed inventory source parsing between an error or a warning
+* More fixes for Python 3 across the code base.
+* win_shell and win_command modules now properly preserve quoted arguments passed on the command-line. Tasks that attempted to work around the issue by adding extra quotes/escaping may need to be reworked. See https://github.com/ansible/ansible/issues/23019 for additional detail.
+* All configuration paths are now relative to the `ansible.cfg` file used.
+* By user request, a 'configuration macro' (``CWD``) is available to force configured paths to be relative to the current working directory. Please note that this is unsafe and not recommended.
+
 
 #### New Callbacks:
 - full_skip
 - profile_roles
 - stderr
 
+#### New Connection plugins:
+- buildah
+- saltstack
+
 #### New Filters:
+- ipaddr filter gained several new suboptions
+  - first_usable
+  - ip/prefix
+  - ip_netmask
+  - last_usable
+  - next_usable
+  - network_id
+  - network/prefix
+  - network_netmask
+  - network_wildcard
+  - previous_usable
+  - range_usable
+  - size_usable
+  - wildcard
+- next_nth_usable
+- network_in_network
+- network_in_usable
+- previous_nth_usable
 - parse_cli
 - parse_cli_textfsm
+- strftime
+- urlsplit
 
 #### New Inventory Plugins:
 - advanced_host_list
 - constructed
 - host_list
 - ini
+- openstack
 - script
 - virtualbox
 - yaml
@@ -152,11 +328,22 @@ Ansible Changes By Release
 #### New Inventory scripts:
 - lxd
 
-#### New: Tests
+#### New Lookups:
+- chef_databag
+- cyberarkpassword
+- hiera
+
+#### New Tests:
 - any : true if any element is true
 - all: true if all elements are true
 
 ### Module Notes
+- By mistake, an early version of elb_classic_lb, elb_instance, and elb_classic_lb_facts modules
+  were released and marked as stableinterface.  These are now marked as preview in 2.4.1 and their
+  parameters and return values may change in 2.5.0.  Part of this mistake included deprecating the
+  ec2_elb_lb, ec2_lb, and ec2_elb_facts modules prematurely.  These modules won't be deprecated
+  until the replacements above have a stableinterface and the erroneous deprecation has been fixed
+  in 2.4.1.
 - The docker_container module has gained a new option, `working_dir` which allows
   specifying the working directory for the command being run in the image.
 - The ec2_win_password module now requires the cryptography python module be installed to run
@@ -578,6 +765,141 @@ Ansible Changes By Release
   * win_wait_for
   * win_wakeonlan
 
+<a id="2.3.3"></a>
+
+## 2.3.3 "Ramble On" - TBD
+
+### Bugfixes
+* Fix alternatives module handlling of non existing options
+* Fix synchronize traceback with the docker connection plugin
+* Do not escape backslashes in the template lookup plugin to mirror what the template module does
+* Fix the expires option of the postgresq_user module
+* Fix for win_acl when settings permissions on registry objects that use `ALL APPLICATION PACKAGES` and `ALL RESTRICTED APPLICATION PACKAGES`
+* Python3 fixes
+  * asorted azure modules
+  * pause module
+  * hacking/env-setup script
+  * Fix traceback when checking for passwords in logged strings when logging executed commands.
+  * docker_login module
+  * Workaround python-libselinux API change in the seboolean module
+  * digital_ocean_tag module
+  * Fix the zip filter
+  * Fix user module combining bytes and text
+  * Fix for security groups in the amazon efs module
+  * Fix for the jail connection plugin not finding the named jail
+  * Fix for blockinfile's parameters insertbefore and insertafter
+* ios_config: Fix traceback when the efaults parameter is not set
+* iosxr_config: Fixed unicode error when UTF-8 characters are in configs
+* Fix check mode in archive module
+* Fix UnboundLocalError in check mode in cs_role module
+* Fix to always use lowercase hostnames for host keys in known_hosts module
+* Added missing return results for win_stat
+* Fix rabbitmq modules to give a helpful error if requests is not installed
+* Fix yum module not deleting rpms that it downloaded
+* Fix yum module failing with a URL to an rpm
+* Fix file module inappropriately expanding literal dollar signs in a path read
+  from the filesystem as an environment variable.
+* Fix the ssh "smart" transport setting which automatically selects the best means of
+  transferring files over ssh (sftp, ssh, piped).
+* Fix authentication by api_key parameter in exoscale modules.
+* vmware module_utils shared code ssl/validate_certs fixes in connection logic
+* allow 'bridge' facts to work for certain containers that create conflicting ones with connection plugins
+* Fix for win_get_url to use TLS 1.2/1.1 if it is available on the host
+* Fix for the filetree lookup with non-ascii group names
+* Better message for invalid keywords/options in task due to undefined expressions
+* Fixed check mode for enable on Solaris for service module
+* Fix cloudtrail module to allow AWS profiles other than the default
+* Fix an encoding issue with secret (password) vars_prompts
+
+<a id="2.3.2"></a>
+
+## 2.3.2 "Ramble On" - 2017-08-04
+
+### Bugfixes
+* Fix partend i18n issues
+* fixed handling of extra vars for tower_job_template (#25272)
+* Python3 bugfixes
+  * Fix sorting of ec2 policies
+  * Fix digital_ocean dynamic inventory script
+  * Fix for the docker connection plugin
+  * Fix pip module when using python3's pyvenv and python3 -m venv to create virtualenvs
+* Fix for the AnsiBallZ wrapper so that it gives a better error message when
+  there's not enough disk space to create its tempdir.
+* Fix so ansilbe-galaxy install --force with unversioned roles will once again
+  overwrite old versions.
+* Fix for RabbitMQ 3.6.7 endpoint return code changing.
+* Fix for Foreman organization creation
+* fixed incorrect fail_json ref in rpm_key
+* Corrected requried on hash_name for dynamodb_table
+* Fix for fetch action plugin not validating correctly
+* Avoid vault view writing display to logs
+* htpasswd: fix passlib module version comparison
+* Fix for flowdock error message when external_user_name is missing
+* fixed corner case for delegate_to, loops and delegate_facts
+* fixed wait_for python2.4/2.5 compatibility (this is last version this is needed)
+* fix for adhoc not obeying callback options
+* fix for win_find where it fails to recursively scan empty nested directories
+* fix non-pipelined code paths for Windows (eg, ANSIBLE_KEEP_REMOTE_FILES, non-pipelined connection plugins)
+* fix for win_updates where args and check mode were ignored due to common code change
+* fix for unprivileged users to Windows runas become method
+* fix starttls code path for mail module
+* fix missing LC_TYPE in parted module
+* fix CN parsing with OpenSSL 1.1 in letsencrypt module
+* fix params assignment in jabber module
+* fix TXT record type handling in exo_dns_record module
+* fix message queue message ttl can't be 0 in rabbitmq_queue module
+* CloudStack bugfixes:
+  * fix template upload for users in cs_template module, change default to is_routing=None
+  * several fixes in cs_host module fixes hypervisor handling
+  * fix network param ignored due typo in cs_nic module
+  * fix missing type bool in module cs_zone
+  * fix KeyError: 'sshkeypair' in cs_instance module for CloudStack v4.5 and before
+* fix for win_chocolatey where trying to upgrade all the packages as per the example docs fails
+* fix for win_chocolatey where it did not fail if the version set did not exist
+* fix for win_regedit always changing a reg key if the dword values set is a hex
+* fix for wait_for on non-Linux systems with newer versions of psutil
+* fix eos_banner code and test issues
+* run tearup and teardown of EAPI service only on EAPI tests
+* fix eos_config tests so only Eth1 and Eth2 are used
+* Fix for potential bug when using legacy inventory vars for configuring the su password.
+* Fix crash in file module when directories contain non-utf8 filenames
+* Fix for dnf groupinstall with dnf-2.x
+* Fix seboolean module for incompatibility in newer Python3 selinux bindings
+* Optimization for inventory, no need to dedup at every stage, its redundant and slow
+* Fix fact gathering for package and service action plugins
+* make random_choice more error resilient (#27380)
+* ensure prefix in plugin loading to avoid conflicts
+* fix for a small number of modules (tempfile, possibly copy) which could fail
+  if the tempdir on the remote box was a symlink
++ fix non-pipelined code paths for Windows (eg, ANSIBLE_KEEP_REMOTE_FILES, non-pipelined connection plugins)
+* fix for win_updates where args and check mode were ignored due to common code change
+
+<a id="2.3.1"></a>
+
+## 2.3.1 "Ramble On" - 2017-06-01
+
+
+### Bugfixes
+* Security fix for CVE-2017-7481 - data for lookup plugins used as variables was not being correctly marked as "unsafe".
+* Fix default value of fetch module's validate_checksum to be True
+* Added fix for "meta: refresh_connection" not working with default 'smart' connection.
+* Fix template so that the --diff command line option works when the destination is a directory
+* Fix python3 bugs in pam_limits
+* Fix unbound error when using module deprecation as a single string
+* Several places in which error handling was broken due to bad conversions or just typos
+* Fix to user module for appending/setting groups on OpenBSD (flags were reversed)
+* assemble fix to use safer os.join.path, avoids charset issues
+* fixed issue with solaris facts and i18n
+* added python2.4 compatiblity fix to sysctl module
+* Fix comparison of exisiting container security opts in the docker_container module
+* fixed service module invocation of insserv on certain platforms
+* Fix traceback in os_user in an error case.
+* Fix docker container to restart a container when changing to fewer exposed ports
+* Fix tracebacks in docker_network
+* Fixes to detection of updated docker images
+* Handle detection of docker image changes when published ports is changed
+* Fix for docker_container restarting images when links list is empty.
+
 <a id="2.3"></a>
 
 ## 2.3 "Ramble On" - 2017-04-12
@@ -613,6 +935,7 @@ Moving to Ansible 2.3 guide http://docs.ansible.com/ansible/porting_guide_2.3.ht
 * 'service' tasks can now use async again, we had lost this capability when changed into an action plugin.
 * made any_errors_fatal inheritable from play to task and all other objects in between.
 * many small performance improvements in inventory and variable handling and in task execution.
+* Added a retry class to the ec2_asg module since customers were running into throttling errors (AWSRetry is a solution for modules using boto3 which isn't applicable here).
 
 ### Deprecations
 * Specifying --tags (or --skip-tags) multiple times on the command line
@@ -844,6 +1167,14 @@ Moving to Ansible 2.3 guide http://docs.ansible.com/ansible/porting_guide_2.3.ht
   * panos_service
 - postgresql_schema
 - proxmox_kvm
+- proxysql:
+  * proxysql_backend_servers
+  * proxysql_global_variables
+  * proxysql_manage_config
+  * proxysql_mysql_users
+  * proxysql_query_rules
+  * proxysql_replication_hostgroups
+  * proxysql_scheduler
 - pubnub_blocks
 - pulp_repo
 - runit
@@ -1640,7 +1971,7 @@ Module fixes:
   This re-executes inventory scripts, but does not force them to ignore any cache they might use.
 * New delegate_facts directive, a boolean that allows you to apply facts to the delegated host (true/yes) instead of the inventory_hostname (no/false) which is the default and previous behaviour.
 * local connections now work with 'su' as a privilege escalation method
-* Ansible 2.0 has deprecated the “ssh” from ansible_ssh_user, ansible_ssh_host, and ansible_ssh_port to become ansible_user, ansible_host, and ansible_port.
+* Ansible 2.0 has deprecated the "ssh" from ansible_ssh_user, ansible_ssh_host, and ansible_ssh_port to become ansible_user, ansible_host, and ansible_port.
 * New ssh configuration variables (`ansible_ssh_common_args`, `ansible_ssh_extra_args`) can be used to configure a
   per-group or per-host ssh ProxyCommand or set any other ssh options.
   `ansible_ssh_extra_args` is used to set options that are accepted only by ssh (not sftp or scp, which have their own analogous settings).

Makefile

@@ -29,6 +29,7 @@ ASCII2HTMLMAN = a2x -L -D docs/html/man/ -d manpage -f xhtml
 else
 ASCII2MAN = @echo "ERROR: AsciiDoc 'a2x' command is not installed but is required to build $(MANPAGES)" && exit 1
 endif
+GENERATE_CLI = docs/bin/generate_man.py
 
 PYTHON=python
 SITELIB = $(shell $(PYTHON) -c "from distutils.sysconfig import get_python_lib; print get_python_lib()")
@@ -343,8 +344,9 @@ webdocs:
 
 .PHONY: generate_asciidoc
 generate_asciidoc: lib/ansible/cli/*.py
-	mkdir -p ./docs/man/man1/
-	PYTHONPATH=./lib ./docs/bin/generate_man.py
+	mkdir -p ./docs/man/man1/ ; \
+	PYTHONPATH=./lib $(GENERATE_CLI) --template-file=docs/templates/man.j2 --output-dir=docs/man/man1/ --output-format man lib/ansible/cli/*.py
+
 
 docs: generate_asciidoc
 	make $(MANPAGES)

RELEASES.txt

@@ -4,7 +4,9 @@ Ansible Releases at a Glance
 VERSION  RELEASE     CODE NAME
 ++++++++++++++++++++++++++++++
 
-2.4.0    TBD         "Dancing Days"
+2.5.0    TBD         "Kashmir"
+2.4.1    10-25-2017  "Dancing Days"
+2.4.0    09-19-2017  "Dancing Days"
 2.3.2    08-04-2017  "Ramble On"
 2.3.1    06-01-2017  "Ramble On"
 2.3.0    04-12-2017  "Ramble On"

VERSION

@@ -1 +1 @@
-2.4.0 0.0.devel
+2.4.1.0 1

bin/ansible-connection

@@ -214,7 +214,6 @@ class Server():
 
     def do_EXEC(self, data):
         cmd = data.split(b'EXEC: ')[1]
-        display.display('Command executed: %s' % cmd, log_only=True)
         return self.connection.exec_command(cmd)
 
     def do_PUT(self, data):

contrib/inventory/azure_rm.py

@@ -187,14 +187,18 @@ Version: 1.0.0
 '''
 
 import argparse
-import ConfigParser
 import json
 import os
 import re
 import sys
 import inspect
-import traceback
 
+try:
+    # python2
+    import ConfigParser as cp
+except ImportError:
+    # python3
+    import configparser as cp
 
 from packaging.version import Version
 
@@ -326,7 +330,7 @@ class AzureRM(object):
         path = expanduser("~")
         path += "/.azure/credentials"
         try:
-            config = ConfigParser.ConfigParser()
+            config = cp.ConfigParser()
             config.read(path)
         except Exception as exc:
             self.fail("Failed to access {0}. Check that the file exists and you have read "
@@ -795,7 +799,7 @@ class AzureInventory(object):
         config = None
         settings = None
         try:
-            config = ConfigParser.ConfigParser()
+            config = cp.ConfigParser()
             config.read(path)
         except:
             pass

contrib/inventory/docker.py

@@ -371,7 +371,6 @@ HAS_DOCKER_PY = True
 HAS_DOCKER_ERROR = False
 
 try:
-    from docker import Client
     from docker.errors import APIError, TLSParameterError
     from docker.tls import TLSConfig
     from docker.constants import DEFAULT_TIMEOUT_SECONDS, DEFAULT_DOCKER_API_VERSION
@@ -379,6 +378,19 @@ except ImportError as exc:
     HAS_DOCKER_ERROR = str(exc)
     HAS_DOCKER_PY = False
 
+# Client has recently been split into DockerClient and APIClient
+try:
+    from docker import Client
+except ImportError as exc:
+    try:
+        from docker import APIClient as Client
+    except ImportError as exc:
+        HAS_DOCKER_ERROR = str(exc)
+        HAS_DOCKER_PY = False
+
+        class Client:
+            pass
+
 DEFAULT_DOCKER_HOST = 'unix://var/run/docker.sock'
 DEFAULT_TLS = False
 DEFAULT_TLS_VERIFY = False
@@ -779,6 +791,10 @@ class DockerInventory(object):
         if config_path:
             try:
                 config_file = os.path.abspath(config_path)
+                # default config path is docker.yml in same directory as this script
+                # old behaviour is docker.yml in current directory. Handle both.
+                if not os.path.exists(config_file):
+                    config_file = os.path.abspath(os.path.basename(config_path))
             except:
                 config_file = None
 
@@ -813,7 +829,7 @@ class DockerInventory(object):
         # Parse command line arguments
 
         basename = os.path.splitext(os.path.basename(__file__))[0]
-        default_config = basename + '.yml'
+        default_config = os.path.join(os.path.dirname(__file__), basename + '.yml')
 
         parser = argparse.ArgumentParser(
             description='Return Ansible inventory for one or more Docker hosts.')

docs/bin/generate_man.py

@@ -1,6 +1,8 @@
 #!/usr/bin/env python
 
+import optparse
 import os
+import pprint
 import sys
 
 from jinja2 import Environment, FileSystemLoader
@@ -8,6 +10,46 @@ from jinja2 import Environment, FileSystemLoader
 from ansible.module_utils._text import to_bytes
 
 
+def generate_parser():
+    p = optparse.OptionParser(
+        version='%prog 1.0',
+        usage='usage: %prog [options]',
+        description='Generate cli documentation from cli docstrings',
+    )
+
+    p.add_option("-t", "--template-file", action="store", dest="template_file", default="../templates/man.j2", help="path to jinja2 template")
+    p.add_option("-o", "--output-dir", action="store", dest="output_dir", default='/tmp/', help="Output directory for rst files")
+    p.add_option("-f", "--output-format", action="store", dest="output_format", default='man', help="Output format for docs (the default 'man' or 'rst')")
+    return p
+
+
+# from https://www.python.org/dev/peps/pep-0257/
+def trim_docstring(docstring):
+    if not docstring:
+        return ''
+    # Convert tabs to spaces (following the normal Python rules)
+    # and split into a list of lines:
+    lines = docstring.expandtabs().splitlines()
+    # Determine minimum indentation (first line doesn't count):
+    indent = sys.maxint
+    for line in lines[1:]:
+        stripped = line.lstrip()
+        if stripped:
+            indent = min(indent, len(line) - len(stripped))
+    # Remove indentation (first line is special):
+    trimmed = [lines[0].strip()]
+    if indent < sys.maxint:
+        for line in lines[1:]:
+            trimmed.append(line[indent:].rstrip())
+    # Strip off trailing and leading blank lines:
+    while trimmed and not trimmed[-1]:
+        trimmed.pop()
+    while trimmed and not trimmed[0]:
+        trimmed.pop(0)
+    # Return a single string:
+    return '\n'.join(trimmed)
+
+
 def get_options(optlist):
     ''' get actual options '''
 
@@ -24,107 +66,215 @@ def get_options(optlist):
     return opts
 
 
+def get_option_groups(option_parser):
+    groups = []
+    for option_group in option_parser.option_groups:
+        group_info = {}
+        group_info['desc'] = option_group.get_description()
+        group_info['options'] = option_group.option_list
+        group_info['group_obj'] = option_group
+        groups.append(group_info)
+    return groups
+
+
 def opt_doc_list(cli):
     ''' iterate over options lists '''
 
     results = []
-    for optg in cli.parser.option_groups:
-        results.extend(get_options(optg.option_list))
+    for option_group in cli.parser.option_groups:
+        results.extend(get_options(option_group.option_list))
 
     results.extend(get_options(cli.parser.option_list))
 
     return results
 
 
-def opts_docs(cli, name):
+# def opts_docs(cli, name):
+def opts_docs(cli_class_name, cli_module_name):
     ''' generate doc structure from options '''
 
-    # cli name
-    if '-' in name:
-        name = name.split('-')[1]
-    else:
-        name = 'adhoc'
+    cli_name = 'ansible-%s' % cli_module_name
+    if cli_module_name == 'adhoc':
+        cli_name = 'ansible'
 
-    # cli info
+    # WIth no action/subcommand
+    # shared opts set
+    # instantiate each cli and ask its options
+    cli_klass = getattr(__import__("ansible.cli.%s" % cli_module_name,
+                                   fromlist=[cli_class_name]), cli_class_name)
+    cli = cli_klass([])
+
+    # parse the common options
+    try:
+        cli.parse()
+    except:
+        pass
+
+    # base/common cli info
     docs = {
-        'cli': name,
+        'cli': cli_module_name,
+        'cli_name': cli_name,
         'usage': cli.parser.usage,
         'short_desc': cli.parser.description,
-        'long_desc': cli.__doc__,
+        'long_desc': trim_docstring(cli.__doc__),
+        'actions': {},
     }
+    option_info = {'option_names': [],
+                   'options': [],
+                   'groups': []}
 
+    for extras in ('ARGUMENTS'):
+        if hasattr(cli, extras):
+            docs[extras.lower()] = getattr(cli, extras)
+
+    common_opts = opt_doc_list(cli)
+    groups_info = get_option_groups(cli.parser)
+    shared_opt_names = []
+    for opt in common_opts:
+        shared_opt_names.extend(opt.get('options', []))
+
+    option_info['options'] = common_opts
+    option_info['option_names'] = shared_opt_names
+
+    option_info['groups'].extend(groups_info)
+
+    docs.update(option_info)
+
+    # now for each action/subcommand
     # force populate parser with per action options
-    if cli.VALID_ACTIONS:
-        docs['actions'] = {}
+
+    # use class attrs not the attrs on a instance (not that it matters here...)
+    for action in getattr(cli_klass, 'VALID_ACTIONS', ()):
+        # instantiate each cli and ask its options
+        action_cli_klass = getattr(__import__("ansible.cli.%s" % cli_module_name,
+                                              fromlist=[cli_class_name]), cli_class_name)
+        # init with args with action added?
+        cli = action_cli_klass([])
+        cli.args.append(action)
+
+        try:
+            cli.parse()
+        except:
+            pass
+
+        # FIXME/TODO: needed?
         # avoid dupe errors
         cli.parser.set_conflict_handler('resolve')
-        for action in cli.VALID_ACTIONS:
-            cli.args.append(action)
-            cli.set_action()
-            docs['actions'][action] = getattr(cli, 'execute_%s' % action).__doc__
+
+        cli.set_action()
+
+        action_info = {'option_names': [],
+                       'options': []}
+        # docs['actions'][action] = {}
+        # docs['actions'][action]['name'] = action
+        action_info['name'] = action
+        action_info['desc'] = trim_docstring(getattr(cli, 'execute_%s' % action).__doc__)
+
+        # docs['actions'][action]['desc'] = getattr(cli, 'execute_%s' % action).__doc__.strip()
+        action_doc_list = opt_doc_list(cli)
+
+        uncommon_options = []
+        for action_doc in action_doc_list:
+            # uncommon_options = []
+
+            option_aliases = action_doc.get('options', [])
+            for option_alias in option_aliases:
+
+                if option_alias in shared_opt_names:
+                    continue
+
+                # TODO: use set
+                if option_alias not in action_info['option_names']:
+                    action_info['option_names'].append(option_alias)
+
+                if action_doc in action_info['options']:
+                    continue
+
+                uncommon_options.append(action_doc)
+
+            action_info['options'] = uncommon_options
+
+        docs['actions'][action] = action_info
 
     docs['options'] = opt_doc_list(cli)
-
     return docs
 
+
 if __name__ == '__main__':
 
-    template_file = 'man.j2'
+    parser = generate_parser()
 
+    options, args = parser.parse_args()
+
+    template_file = options.template_file
+    template_path = os.path.expanduser(template_file)
+    template_dir = os.path.abspath(os.path.dirname(template_path))
+    template_basename = os.path.basename(template_file)
+
+    output_dir = os.path.abspath(options.output_dir)
+    output_format = options.output_format
+
+    cli_modules = args
+
+    # various cli parsing things checks sys.argv if the 'args' that are passed in are []
+    # so just remove any args so the cli modules dont try to parse them resulting in warnings
+    sys.argv = [sys.argv[0]]
     # need to be in right dir
     os.chdir(os.path.dirname(__file__))
 
     allvars = {}
     output = {}
     cli_list = []
-    for binary in os.listdir('../../lib/ansible/cli'):
+    cli_bin_name_list = []
+
+    # for binary in os.listdir('../../lib/ansible/cli'):
+    for cli_module_name in cli_modules:
+        binary = os.path.basename(os.path.expanduser(cli_module_name))
 
         if not binary.endswith('.py'):
             continue
         elif binary == '__init__.py':
             continue
 
-        libname = os.path.splitext(binary)[0]
-        print("Found CLI %s" % libname)
+        cli_name = os.path.splitext(binary)[0]
 
-        if libname == 'adhoc':
-            myclass = 'AdHocCLI'
-            output[libname] = 'ansible.1.asciidoc.in'
+        if cli_name == 'adhoc':
+            cli_class_name = 'AdHocCLI'
+            # myclass = 'AdHocCLI'
+            output[cli_name] = 'ansible.1.asciidoc.in'
+            cli_bin_name = 'ansible'
         else:
-            myclass = "%sCLI" % libname.capitalize()
-            output[libname] = 'ansible-%s.1.asciidoc.in' % libname
+            # myclass = "%sCLI" % libname.capitalize()
+            cli_class_name = "%sCLI" % cli_name.capitalize()
+            output[cli_name] = 'ansible-%s.1.asciidoc.in' % cli_name
+            cli_bin_name = 'ansible-%s' % cli_name
 
-        # instantiate each cli and ask its options
-        mycli = getattr(__import__("ansible.cli.%s" % libname, fromlist=[myclass]), myclass)
-        cli_object = mycli([])
-        try:
-            cli_object.parse()
-        except:
-            # no options passed, we expect errors
-            pass
-
-        allvars[libname] = opts_docs(cli_object, libname)
-
-        for extras in ('ARGUMENTS'):
-            if hasattr(cli_object, extras):
-                allvars[libname][extras.lower()] = getattr(cli_object, extras)
+        # FIXME:
+        allvars[cli_name] = opts_docs(cli_class_name, cli_name)
+        cli_bin_name_list.append(cli_bin_name)
 
     cli_list = allvars.keys()
-    for libname in cli_list:
+
+    doc_name_formats = {'man': '%s.1.asciidoc.in',
+                        'rst': '%s.rst'}
+
+    for cli_name in cli_list:
 
         # template it!
-        env = Environment(loader=FileSystemLoader('../templates'))
-        template = env.get_template('man.j2')
+        env = Environment(loader=FileSystemLoader(template_dir))
+        template = env.get_template(template_basename)
 
         # add rest to vars
-        tvars = allvars[libname]
+        tvars = allvars[cli_name]
         tvars['cli_list'] = cli_list
-        tvars['cli'] = libname
+        tvars['cli_bin_name_list'] = cli_bin_name_list
+        tvars['cli'] = cli_name
         if '-i' in tvars['options']:
             print('uses inventory')
 
         manpage = template.render(tvars)
-        filename = '../man/man1/%s' % output[libname]
+        filename = os.path.join(output_dir, doc_name_formats[output_format] % tvars['cli_name'])
+
         with open(filename, 'wb') as f:
             f.write(to_bytes(manpage))
-            print("Wrote man docs to %s" % filename)
+            print("Wrote doc to %s" % filename)

docs/bin/plugin_formatter.py

@@ -120,6 +120,9 @@ def write_data(text, output_dir, outputname, module=None):
     if output_dir is not None:
         if module:
             outputname = outputname % module
+
+        if not os.path.exists(output_dir):
+            os.makedirs(output_dir)
         fname = os.path.join(output_dir, outputname)
         fname = fname.replace(".py", "")
         with open(fname, 'wb') as f:

docs/docsite/Makefile

@@ -4,6 +4,7 @@ FORMATTER=../bin/plugin_formatter.py
 TESTING_FORMATTER=../bin/testing_formatter.sh
 DUMPER=../bin/dump_keywords.py
 CONFIG_DUMPER=../bin/dump_config.py
+GENERATE_CLI=../bin/generate_man.py
 ifeq ($(shell echo $(OS) | egrep -ic 'Darwin|FreeBSD|OpenBSD|DragonFly'),1)
 CPUS ?= $(shell sysctl hw.ncpu|awk '{print $$2}')
 else
@@ -19,7 +20,8 @@ all: docs
 
 docs: clean htmldocs
 
-htmldocs: testing keywords modules staticmin config
+htmldocs: testing keywords modules staticmin cli config
+
 	CPUS=$(CPUS) $(MAKE) -f Makefile.sphinx html
 
 webdocs: docs
@@ -45,9 +47,14 @@ clean:
 	-rm rst/*_maintained.rst
 	-rm rst/playbooks_directives.rst
 	-rm rst/playbooks_keywords.rst
+#	-rm rst/cli/ansible*.rst
 
 .PHONEY: docs clean
 
+# TODO: make generate_man output dir cli option
+cli: $(GENERATE_CLI)
+	PYTHONPATH=../../lib $(GENERATE_CLI) --template-file=../templates/cli_rst.j2 --output-dir=rst/ --output-format rst ../../lib/ansible/cli/*.py
+
 keywords: $(FORMATTER) ../templates/playbooks_keywords.rst.j2
 	PYTHONPATH=../../lib $(DUMPER) --template-dir=../templates --output-dir=rst/ -d ./keyword_desc.yml
 
@@ -57,7 +64,7 @@ config:
 modules: $(FORMATTER) ../templates/plugin.rst.j2
 # Limit building of module documentation if requested.
 ifdef MODULES
-	PYTHONPATH=../../lib $(FORMATTER) -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/ -l $(MODULES)	
+	PYTHONPATH=../../lib $(FORMATTER) -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/ -l $(MODULES)
 else
 	PYTHONPATH=../../lib $(FORMATTER) -t rst --template-dir=../templates --module-dir=../../lib/ansible/modules -o rst/
 endif

docs/docsite/_themes/srtd/layout.html

@@ -160,7 +160,7 @@
                 <div class="DocSiteProduct-logoText">
                     Ansible  
                     <div class="DocSiteProduct-CurrentVersion" align="right">
-                      devel
+                      v2.4
                   </div>
                 </div>
             </div>

docs/docsite/keyword_desc.yml

@@ -1,55 +1,55 @@
 accelerate: DEPRECATED, set to True to use accelerate connection plugin.
-accelerate_ipv6: DEPRECATED, set to True to force accelerate plugin to use ipv6 for it's connection.
+accelerate_ipv6: "DEPRECATED, set to True to force accelerate plugin to use ipv6 for it's connection."
 accelerate_port: DEPRECATED, set to override default port use for accelerate connection.
-action: The 'action' to execute for a task, it normally translates into a C(module) or action plugin.
+action: "The 'action' to execute for a task, it normally translates into a C(module) or action plugin."
 args: DEPRECATED, A secondary way to add arguments into a task, it takes a dictionary in which keys map to options and values .. well you get it.
 always: List of tasks, in a block, that execute no matter if there is an error in the block or not.
-always_run: DEPRECATED, forces a task to run even in check mode, use check_mode directive instead.
+always_run: DEPRECATED, forces a task to run even in check mode, use :term:`check_mode` directive instead.
 any_errors_fatal: Force any un-handled task errors on any host to propagate to all hosts and end the play.
 async: Run a task asyncronouslly if the C(action) supports this.
-become: Boolean that controls if privilege escalation is used or not on Task execution.
-become_flags: A string of flag(s) to pass to the privilege escalation program when ``become`` is True.
+become: Boolean that controls if privilege escalation is used or not on :term:`Task` execution.
+become_flags: A string of flag(s) to pass to the privilege escalation program when :term:`become` is True.
 become_method: Which method of privilege escalation to use. i.e. sudo/su/etc.
-become_user: User that you 'become' after using privilege escalation, the remote/login user must have permissions to become this user.
+become_user: "User that you 'become' after using privilege escalation, the remote/login user must have permissions to become this user."
 block: List of tasks in a block.
-changed_when: Conditional expression that overrides the task's normal 'changed' status.
-check_mode: A boolean that controls if a task is executed in 'check' mode
+changed_when: "Conditional expression that overrides the task's normal 'changed' status."
+check_mode: "A boolean that controls if a task is executed in 'check' mode"
 connection: Allows you to change the connection plugin used for tasks to execute on the target.
-delay: Number of seconds to delay between retries, this setting only makes sense when used with 'until'.
+delay: Number of seconds to delay between retries, this setting is only used in combination with :term:`until`.
 delegate_facts: Boolean that allows you to apply facts to delegated host instead of inventory_hostname.
 delegate_to: Host to execute task instead of the target (inventory_hostname), connection vars from the delegated host will also be used for the task.
-diff: Toggle to make tasks return 'diff' information or not.
+diff: "Toggle to make tasks return 'diff' information or not."
 environment: A dictionary that gets converted into environment vars to be provided for the task upon execution.
-fact_path: Set the fact path option for the fact gathering plugin controlled by ``gather_facts``.
-failed_when: Conditional expression that overrides the task's normal 'failed' status.
+fact_path: Set the fact path option for the fact gathering plugin controlled by :term:`gather_facts`.
+failed_when: "Conditional expression that overrides the task's normal 'failed' status."
 force_handlers: Will force notified handler execution for hosts even if they failed during the play, it will not trigger if the play itself fails.
-gather_facts: A boolean that controls if the play will automatically run the 'setup' task to gather facts for the hosts.
-gather_subset: Allows you to pass subset options to the  fact gathering plugin controlled by ``gather_facts``.
-gather_timeout: Allows you to set the timeout for the fact gathering plugin controlled by ``gather_facts``.
-handlers: A section with tasks that are treated as handlers, these won't get executed normally, only when notified. After each section of tasks is complete.
-hosts: A list of groups, hosts or host pattern that translates into a list of hosts that are the play's target.
+gather_facts: "A boolean that controls if the play will automatically run the 'setup' task to gather facts for the hosts."
+gather_subset: Allows you to pass subset options to the  fact gathering plugin controlled by :term:`gather_facts`.
+gather_timeout: Allows you to set the timeout for the fact gathering plugin controlled by :term:`gather_facts`.
+handlers: "A section with tasks that are treated as handlers, these won't get executed normally, only when notified. After each section of tasks is complete."
+hosts: "A list of groups, hosts or host pattern that translates into a list of hosts that are the play's target."
 ignore_errors: Boolean that allows you to ignore task failures and continue with play. It does not affect connection errors.
-loop_control: Several keys here allow you to modify/set loop behaviour in a task see http://docs.ansible.com/ansible/latest/playbooks_loops.html#loop-control for details.
+loop_control: "Several keys here allow you to modify/set loop behaviour in a task see http://docs.ansible.com/ansible/latest/playbooks_loops.html#loop-control for details."
 max_fail_percentage: can be used to abort the run after a given percentage of hosts in the current batch has failed.
-name: It's a name, works mostly for documentation, in the case of tasks/handlers it can be an identifier.
+name: "It's a name, works mostly for documentation, in the case of tasks/handlers it can be an identifier."
 no_log: Boolean that controls information disclosure.
-notify: list of handlers to notify when the task returns a 'changed=True' status.
+notify: "list of handlers to notify when the task returns a 'changed=True' status."
 order: Controls the sorting of hosts as they are used for executing the play. Possible values are inventory (default), sorted, reverse_sorted, reverse_inventory and shuffle.
 poll: Sets the polling interval in seconds for async tasks (default 10s).
 port: Used to override the default port used in a connection.
-post_tasks: A list of tasks to execute after the ``tasks`` section.
-pre_tasks: A list of tasks to execute before ``roles``.
+post_tasks: A list of tasks to execute after the :term:`tasks` section.
+pre_tasks: A list of tasks to execute before :term:`roles`.
 remote_user: User used to log into the target via the connection plugin. AKA login user.
 register: Name of variable that will contain task status and module return data.
-rescue: List of tasks in a block that run if there is a task error in the main ``block`` list.
-retries: Number of retries before giving up in a 'until' loop, this setting is ignored on it's own.
+rescue: List of tasks in a :term:`block` that run if there is a task error in the main :term:`block` list.
+retries: "Number of retries before giving up in a :term:`until` loop. This setting is only used in combination with :term:`until`."
 roles: List of roles to be imported into the play
 run_once: Boolean that will bypass the host loop, forcing the task to execute on the first host available and will also apply any facts to all active hosts.
 serial: Defines the 'batch' of hosts to execute the current play until the end.
 strategy: Allows you to choose the connection plugin to use for the play.
 tags: Tags applied to the task or included tasks, this allows selecting subsets of tasks from the command line.
-tasks: Main list of tasks to execute in the play, they run after ``roles`` and before ``post_tasks``.
-until: This keyword implies a 'retry loop' that will go on until the condition supplied here is met or we hit the retry limit.
+tasks: Main list of tasks to execute in the play, they run after :term:`roles` and before :term:`post_tasks`.
+until: "This keyword implies a ':term:`retries` loop' that will go on until the condition supplied here is met or we hit the :term:`retries` limit."
 vars: Dictionary/map of variables
 vars_files: List of files that contain vars to include in the play.
 vars_prompt: list of variables to prompt for.

docs/docsite/rst/command_line_tools.rst

@@ -0,0 +1,15 @@
+Command Line Tools
+==================
+
+
+.. toctree:: :maxdepth: 1
+
+    ansible
+    ansible-playbook
+    ansible-vault
+    ansible-galaxy
+    ansible-console
+    ansible-config
+    ansible-doc
+    ansible-inventory
+    ansible-pull

docs/docsite/rst/committer_guidelines.rst

@@ -3,7 +3,7 @@ Committers Guidelines (for people with commit rights to Ansible on GitHub)
 
 These are the guidelines for people with commit access to Ansible. Committers are essentially acting as members of the Ansible Core team, although not necessarily as an employee of Ansible and Red Hat. Please read the guidelines before you commit.
 
-These guidelines apply to everyone. At the same time, this ISN’T a process document. So just use good judgement. You’ve been given commit access because we trust your judgement.
+These guidelines apply to everyone. At the same time, this ISN'T a process document. So just use good judgement. You've been given commit access because we trust your judgement.
 
 That said, use the trust wisely. 
 
@@ -19,18 +19,18 @@ Any other new features and changes to high level design should go through the pr
 Our Workflow on GitHub
 ======================
 
-As a committer, you may already know this, but our workflow forms a lot of our team policies. Please ensure you’re aware of the following workflow steps:
+As a committer, you may already know this, but our workflow forms a lot of our team policies. Please ensure you're aware of the following workflow steps:
 
 * Fork the repository upon which you want to do some work to your own personal repository
 * Work on the specific branch upon which you need to commit
-* Create a Pull Request back to the Ansible repository and tag the people you would like to review; assign someone as the primary “owner” of your request
+* Create a Pull Request back to the Ansible repository and tag the people you would like to review; assign someone as the primary "owner" of your request
 * Adjust code as necessary based on the Comments provided
 * Ask someone on the Core Team to do a final review and merge
 
 Addendum to workflow for Committers:
 ------------------------------------
 
-The Core Team is aware that this can be a difficult process at times. Sometimes, the team breaks the rules: Direct commits, merging their own PRs. This section is a set of guidelines. If you’re changing a comma in a doc, or making a very minor change, you can use your best judgement. This is another trust thing. The process is critical for any major change, but for little things or getting something done quickly, use your best judgement and make sure people on the team are aware of your work.
+The Core Team is aware that this can be a difficult process at times. Sometimes, the team breaks the rules: Direct commits, merging their own PRs. This section is a set of guidelines. If you're changing a comma in a doc, or making a very minor change, you can use your best judgement. This is another trust thing. The process is critical for any major change, but for little things or getting something done quickly, use your best judgement and make sure people on the team are aware of your work.
 
 Roles on Core
 =============
@@ -41,12 +41,12 @@ General Rules
 =============
 Individuals with direct commit access to ansible/ansible are entrusted with powers that allow them to do a broad variety of things--probably more than we can write down. Rather than rules, treat these as general *guidelines*, individuals with this power are expected to use their best judgement. 
 
-* Don’t
+* Don't
 
   - Commit directly.
   - Merge your own PRs. Someone else should have a chance to review and approve the PR merge. If you are a Core Committer, you have a small amount of leeway here for very minor changes.
   - Forget about alternate environments. Consider the alternatives--yes, people have bad environments, but they are the ones who need us the most.
-  - Drag your community team members down. Always discuss the technical merits, but you should never address the person’s limitations (you can later go for beers and call them idiots, but not in IRC/Github/etc.).
+  - Drag your community team members down. Always discuss the technical merits, but you should never address the person's limitations (you can later go for beers and call them idiots, but not in IRC/Github/etc.).
   - Forget about the maintenance burden. Some things are really cool to have, but they might not be worth shoehorning in if the maintenance burden is too great.
   - Break playbooks. Always keep backwards compatibility in mind.
   - Forget to keep it simple. Complexity breeds all kinds of problems.
@@ -55,7 +55,7 @@ Individuals with direct commit access to ansible/ansible are entrusted with powe
 
   - Squash, avoid merges whenever possible, use github's squash commits or cherry pick if needed (bisect thanks you).
   - Be active. Committers who have no activity on the project (through merges, triage, commits, etc.) will have their permissions suspended.
-  - Consider backwards compatibility (goes back to "don’t break existing playbooks").
+  - Consider backwards compatibility (goes back to "don't break existing playbooks").
   - Write tests. PRs with tests are looked at with more priority than PRs without tests that should have them included. While not all changes require tests, be sure to add them for bug fixes or functionality changes.
   - Discuss with other committers, specially when you are unsure of something.
   - Document! If your PR is a new feature or a change to behavior, make sure you've updated all associated documentation or have notified the right people to do so. It also helps to add the version of Core against which this documentation is compatible (to avoid confusion with stable versus devel docs, for backwards compatibility, etc.).

docs/docsite/rst/community.rst

@@ -2,7 +2,7 @@ Community Information & Contributing
 ````````````````````````````````````
 
 Ansible is an open source project designed to bring together administrators and developers of all kinds to collaborate on building
-IT automation solutions that work well for them.   
+IT automation solutions that work well for them.
 
 Should you wish to get more involved -- whether in terms of just asking a question, helping other users, introducing new people to Ansible, or helping with the software or documentation, we welcome your contributions to the project.
 
@@ -16,7 +16,7 @@ I've Got A Question
 
 We're happy to help!
 
-Ansible questions are best asked on the `Ansible Google Group Mailing List <http://groups.google.com/group/ansible-project>`_.  
+Ansible questions are best asked on the `Ansible Google Group Mailing List <http://groups.google.com/group/ansible-project>`_.
 
 This is a very large high-traffic list for answering questions and sharing tips
 and tricks. Anyone can join, and email delivery is optional if you just want to read the group online.  To cut down on spam, your first post is moderated, though posts are approved quickly.
@@ -42,10 +42,10 @@ This is a low-traffic read-only list, where we'll share release announcements an
 I'd Like To Help Share and Promote Ansible
 ------------------------------------------
 
-You can help share Ansible with others by telling friends and colleagues, writing a blog post, 
-or presenting at user groups (like DevOps groups or the local LUG).  
+You can help share Ansible with others by telling friends and colleagues, writing a blog post,
+or presenting at user groups (like DevOps groups or the local LUG).
 
-You are also welcome to share slides on speakerdeck, sign up for a free account and tag it “Ansible”. On Twitter, 
+You are also welcome to share slides on speakerdeck, sign up for a free account and tag it "Ansible". On Twitter,
 you can also share things with #ansible and may wish to `follow us <https://twitter.com/ansible>`_.
 
 I'd Like To Help Ansible Move Faster
@@ -73,31 +73,31 @@ more quickly.
 Do not use the issue tracker for "how do I do this" type questions.  These are great candidates
 for IRC or the mailing list instead where things are likely to be more of a discussion.
 
-To be respectful of reviewers' time and allow us to help everyone efficiently, please 
+To be respectful of reviewers' time and allow us to help everyone efficiently, please
 provide minimal well-reduced and well-commented examples versus sharing your entire production
-playbook.  Include playbook snippets and output where possible.  
+playbook.  Include playbook snippets and output where possible.
 
 When sharing YAML in playbooks, formatting can be preserved by using `code blocks <https://help.github.com/articles/creating-and-highlighting-code-blocks/>`_.
 
 For multiple-file content, we encourage use of gist.github.com.  Online pastebin content can expire, so it's nice to have things around for a longer term if they
 are referenced in a ticket.
 
-If you are not sure if something is a bug yet, you are welcome to ask about something on 
-the mailing list or IRC first.  
+If you are not sure if something is a bug yet, you are welcome to ask about something on
+the mailing list or IRC first.
 
-As we are a very high volume project, if you determine that 
+As we are a very high volume project, if you determine that
 you do have a bug, please be sure to open the issue yourself to ensure we have a record of
-it. Don’t rely on someone else in the community to file the bug report for you.
+it. Don't rely on someone else in the community to file the bug report for you.
 
 It may take some time to get to your report, see our information about priority flags below.
 
 I'd Like To Help With Documentation
 -----------------------------------
 
-Ansible documentation is a community project too!  
+Ansible documentation is a community project too!
 
-If you would like to help with the 
-documentation, whether correcting a typo or improving a section, or maybe even 
+If you would like to help with the
+documentation, whether correcting a typo or improving a section, or maybe even
 documenting a new feature, submit a GitHub pull request to  the code that
 lives in the ``docsite/rst`` subdirectory of the project for most pages, and there is an "Edit on GitHub"
 link up on those.
@@ -107,9 +107,9 @@ Module documentation is generated from a ``DOCUMENTATION`` structure embedded in
 For more information on module documentation see `How to document modules <http://docs.ansible.com/ansible/dev_guide/developing_modules_documenting.html>`_.
 
 Aside from modules, the main docs are in restructured text
-format.  
+format.
 
-If you aren’t comfortable with restructured text, you can also open a ticket on 
+If you aren't comfortable with restructured text, you can also open a ticket on
 GitHub about any errors you spot or sections you would like to see added. For more information
 on creating pull requests, please refer to the
 `GitHub help guide <https://help.github.com/articles/using-pull-requests>`_.
@@ -155,7 +155,7 @@ If you make a mistake you do not need to close your PR. Instead, create a clean
 with ``--force`` to overwrite the existing branch (permissible in this case as no one else should be using that
 branch as reference). Code comments won't be lost, they just won't be attached to the existing branch.
 
-We’ll then review your contributions and engage with you about questions and  so on.
+We'll then review your contributions and engage with you about questions and  so on.
 
 Because we have a very large and active community it may take awhile to get your contributions
 in!  See the notes about priorities in a later section for understanding our work queue.
@@ -172,14 +172,14 @@ are interested in writing new modules to be included in the core Ansible distrib
 to the `module development documentation <http://docs.ansible.com/developing_modules.html>`_.
 
 Ansible's aesthetic encourages simple, readable code and consistent,
-conservatively extending, backwards-compatible improvements. When contributing code to Ansible, 
+conservatively extending, backwards-compatible improvements. When contributing code to Ansible,
 please observe the following guidelines:
 
 - Code developed for Ansible needs to support Python2-2.6 or higher and Python3-3.5 or higher.
 - Use a 4-space indent, not  tabs.
 - We do not enforce 80 column lines; up to 160 columns are acceptable.
 - We do not accept 'style only' pull requests unless the code is nearly unreadable.
-- We are "PEP8ish", but not strictly compliant, see :doc:`testing_pep8` for more information.
+- We are not strictly compliant with PEP8. See :doc:`dev_guide/testing_pep8` for more information.
 
 You can also contribute by testing and revising other requests, especially if it is one you are interested
 in using. Please keep your comments clear and to the point, courteous and constructive, tickets are not a
@@ -228,7 +228,7 @@ To subscribe to a group from a non-google account, you can send an email to the
 IRC Meetings
 ------------
 
-The Ansible community holds regular IRC meetings on various topics, and anyone who is interested is invited to 
+The Ansible community holds regular IRC meetings on various topics, and anyone who is interested is invited to
 participate. For more information about Ansible meetings, consult the `meeting schedule and agenda page <https://github.com/ansible/community/blob/master/meetings/README.md>`_.
 
 Release Numbering
@@ -286,7 +286,7 @@ These labels don't really have a definition - they are a simple ordering.  Howev
 affecting a major module (yum, apt, etc) is likely to be prioritized higher than a module
 affecting a smaller number of users.
 
-Since we place a strong emphasis on testing and code review, it may take a few months for a minor feature to get merged. This doesn't mean that we'll be exhausting all of the higher-priority queues before getting to your ticket; we will also take periodic sweeps through the lower priority queues and give them some attention as well, particularly in the area of new module changes. 
+Since we place a strong emphasis on testing and code review, it may take a few months for a minor feature to get merged. This doesn't mean that we'll be exhausting all of the higher-priority queues before getting to your ticket; we will also take periodic sweeps through the lower priority queues and give them some attention as well, particularly in the area of new module changes.
 
 
 Every bit of effort helps - if you're wishing to expedite the inclusion of a P3 feature pull request for instance, the best thing you can do is help close P2 bug reports.

docs/docsite/rst/community/development_process.rst

@@ -41,9 +41,9 @@ Each module has at least one assigned maintainer, listed in a `maintainer's file
 .. _Ansibullbot: https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md
 .. _maintainer's file: https://github.com/ansible/ansible/blob/devel/.github/BOTMETA.yml
 
-Some modules have no community maintainers assigned. In this case, the maintainer is listed as ``$team_ansible``. Ultimately, it’s our goal to have at least one community maintainer for every module.
+Some modules have no community maintainers assigned. In this case, the maintainer is listed as ``$team_ansible``. Ultimately, it's our goal to have at least one community maintainer for every module.
 
-The maintainer’s job is to review PRs and decide whether that PR should be merged (``shipit``) or revised (``needs_revision``).
+The maintainer's job is to review PRs and decide whether that PR should be merged (``shipit``) or revised (``needs_revision``).
 
 The ultimate goal of any pull request is to reach **shipit** status, where the Core team then decides whether the PR is ready to be merged. Not every PR that reaches the **shipit** label is actually ready to be merged, but the better our reviewers are, and the better our guidelines are, the more likely it will be that a PR that reaches **shipit** will be mergeable.
 
@@ -54,7 +54,7 @@ Workflow
 
 Ansibullbot runs continuously. You can generally expect to see changes to your issue or pull request within thirty minutes. Ansibullbot examines every open pull request in the repositories, and enforces state roughly according to the following workflow:
 
--  If a pull request has no workflow labels, it’s considered **new**. Files in the pull request are identified, and the maintainers of those files are pinged by the bot, along with instructions on how to review the pull request. (Note: sometimes we strip labels from a pull request to “reboot” this process.)
+-  If a pull request has no workflow labels, it's considered **new**. Files in the pull request are identified, and the maintainers of those files are pinged by the bot, along with instructions on how to review the pull request. (Note: sometimes we strip labels from a pull request to "reboot" this process.)
 -  If the module maintainer is not ``$team_ansible``, the pull request then goes into the **community_review** state.
 -  If the module maintainer is ``$team_ansible``, the pull request then goes into the **core_review** state (and probably sits for a while).
 -  If the pull request is in **community_review** and has received comments from the maintainer:
@@ -110,4 +110,4 @@ Special Labels
 
 -  **new_plugin**: this is for new modules or plugins that are not yet in Ansible.
 
-   **Note:** `new_plugin` kicks off a completely separate process, and frankly it doesn’t work very well at present. We’re working our best to improve this process.
+   **Note:** `new_plugin` kicks off a completely separate process, and frankly it doesn't work very well at present. We're working our best to improve this process.

docs/docsite/rst/community/reporting_bugs_and_features.rst

@@ -25,7 +25,7 @@ For multiple-file content, we encourage use of gist.github.com.  Online pastebin
 
 If you are not sure if something is a bug yet, you are welcome to ask about something on the mailing list or IRC first.
 
-As we are a very high volume project, if you determine that you do have a bug, please be sure to open the issue yourself to ensure we have a record of it. Don’t rely on someone else in the community to file the bug report for you.
+As we are a very high volume project, if you determine that you do have a bug, please be sure to open the issue yourself to ensure we have a record of it. Don't rely on someone else in the community to file the bug report for you.
 
 Requesting a feature
 ====================

docs/docsite/rst/dev_guide/developing_api.rst

@@ -44,8 +44,8 @@ In 2.0 things get a bit more complicated to start, but you end up with much more
     import json
     from collections import namedtuple
     from ansible.parsing.dataloader import DataLoader
-    from ansible.vars import VariableManager
-    from ansible.inventory import Inventory
+    from ansible.vars.manager import VariableManager
+    from ansible.inventory.manager import InventoryManager
     from ansible.playbook.play import Play
     from ansible.executor.task_queue_manager import TaskQueueManager
     from ansible.plugins.callback import CallbackBase
@@ -63,21 +63,21 @@ In 2.0 things get a bit more complicated to start, but you end up with much more
             This method could store the result in an instance attribute for retrieval later
             """
             host = result._host
-            print json.dumps({host.name: result._result}, indent=4)
+            print(json.dumps({host.name: result._result}, indent=4))
 
-    Options = namedtuple('Options', ['connection', 'module_path', 'forks', 'become', 'become_method', 'become_user', 'check'])
+    Options = namedtuple('Options', ['connection', 'module_path', 'forks', 'become', 'become_method', 'become_user', 'check', 'diff'])
     # initialize needed objects
-    variable_manager = VariableManager()
     loader = DataLoader()
-    options = Options(connection='local', module_path='/path/to/mymodules', forks=100, become=None, become_method=None, become_user=None, check=False)
+    options = Options(connection='local', module_path='/path/to/mymodules', forks=100, become=None, become_method=None, become_user=None, check=False,
+                      diff=False)
     passwords = dict(vault_pass='secret')
 
     # Instantiate our ResultCallback for handling results as they come in
     results_callback = ResultCallback()
 
     # create inventory and pass to var manager
-    inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list='localhost')
-    variable_manager.set_inventory(inventory)
+    inventory = InventoryManager(loader=loader, sources=['localhost'])
+    variable_manager = VariableManager(loader=loader, inventory=inventory)
 
     # create play with tasks
     play_source =  dict(

docs/docsite/rst/dev_guide/developing_modules_checklist.rst

@@ -55,7 +55,7 @@ The complete module metadata specification is here: `Ansible metadata block <htt
     * Examples--include them whenever possible and make sure they are reproducible.
     * Document the return structure of the module. Refer to :ref:`common_return_values` and :ref:`module_documenting` for additional information.
 * Predictable user interface: This is a particularly important section as it is also an area where we need significant improvements.
-    * Name consistency across modules (we’ve gotten better at this, but we still have many deviations).
+    * Name consistency across modules (we've gotten better at this, but we still have many deviations).
     * Declarative operation (not CRUD)--this makes it easy for a user not to care what the existing state is, just about the final state. ``started/stopped``, ``present/absent``--don't overload options too much. It is preferable to add a new, simple option than to add choices/states that don't fit with existing ones.
     * Keep options small, having them take large data structures might save us a few tasks, but adds a complex requirement that we cannot easily validate before passing on to the module.
     * Allow an "expert mode". This may sound like the absolute opposite of the previous one, but it is always best to let expert users deal with complex data. This requires different modules in some cases, so that you end up having one (1) expert module and several 'piecemeal' ones (ec2_vpc_net?). The reason for this is not, as many users express, because it allows a single task and keeps plays small (which just moves the data complexity into vars files, leaving you with a slightly different structure in another YAML file). It does, however, allow for a more 'atomic' operation against the underlying APIs and services.

docs/docsite/rst/dev_guide/developing_modules_documenting.rst

@@ -109,11 +109,11 @@ Structure
 Fields
 ^^^^^^
 
-:metadata_version: An “X.Y” formatted string. X and Y are integers which
+:metadata_version: An "X.Y" formatted string. X and Y are integers which
    define the metadata format version. Modules shipped with Ansible are
    tied to an Ansible release, so we will only ship with a single version
-   of the metadata. We’ll increment Y if we add fields or legal values
-   to an existing field. We’ll increment X if we remove fields or values
+   of the metadata. We'll increment Y if we add fields or legal values
+   to an existing field. We'll increment X if we remove fields or values
    or change the type or meaning of a field.
    Current metadata_version is "1.1"
 :supported_by: This field records who supports the module.
@@ -130,13 +130,13 @@ Fields
    `Modules Support <http://docs.ansible.com/ansible/modules_support.html>`_.
 
 :status: This field records information about the module that is
-   important to the end user. It’s a list of strings. The default value
-   is a single element list [“preview”]. The following strings are valid
+   important to the end user. It's a list of strings. The default value
+   is a single element list ["preview"]. The following strings are valid
    statuses and have the following meanings:
 
-   :stableinterface: This means that the module’s parameters are
+   :stableinterface: This means that the module's parameters are
       stable. Every effort will be made not to remove parameters or change
-      their meaning. It is not a rating of the module’s code quality.
+      their meaning. It is not a rating of the module's code quality.
    :preview: This module is a tech preview. This means it may be
       unstable, the parameters may change, or it may require libraries or
       web services that are themselves subject to incompatible changes.
@@ -223,6 +223,8 @@ The following fields can be used and are all required unless specified otherwise
     * If `required` is false/missing, `default` may be specified (assumed 'null' if missing).
     * Ensure that the default parameter in the docs matches the default parameter in the code.
     * The default option must not be listed as part of the description.
+    * If the option is a boolean value, you can use any of the boolean values recognized by Ansible:
+      (such as true/false or yes/no).  Choose the one that reads better in the context of the option.
   :choices:
     List of option values. Should be absent if empty.
   :type:

docs/docsite/rst/dev_guide/developing_modules_in_groups.rst

@@ -26,7 +26,7 @@ Although it's tempting to get straight into coding, there are a few things to be
 * Shared code can be placed into ``lib/ansible/module_utils/``
 * Shared documentation (for example describing common arguments) can be placed in ``lib/ansible/utils/module_docs_fragments/``.
 * With great power comes great responsibility: Ansible module maintainers have a duty to help keep modules up to date. As with all successful community projects, module maintainers should keep a watchful eye for reported issues and contributions.
-* Although not required, unit and/or integration tests are strongly recommended. Unit tests are especially valuable when external resources (such as cloud or network devices) are required. For more information see :doc:`testing` and the `Testing Working Group <https://github.com/ansible/community/blob/master/meetings/README.md>`_.
+* Although not required, unit and/or integration tests are strongly recommended. Unit tests are especially valuable when external resources (such as cloud or network devices) are required. For more information see :doc:`dev_guide/testing` and the `Testing Working Group <https://github.com/ansible/community/blob/master/meetings/README.md>`_.
   * Starting with Ansible 2.4 all :doc:`../list_of_network_modules` MUST have unit tests.
 
 

docs/docsite/rst/dev_guide/developing_python3.rst

@@ -295,7 +295,7 @@ new exception-catching syntax which uses the ``as`` keyword:
 Do **not** use the following syntax as it will fail on every version of Python-3:
 
 .. This code block won't highlight because python2 isn't recognized. This is necessary to pass tests under python 3.
-.. code-block:: python2
+.. code-block:: none
 
     try:
         a = 2/0

docs/docsite/rst/dev_guide/index.rst

@@ -30,7 +30,6 @@ To get started, select one of the following topics.
    developing_core
    developing_python3
    developing_api
-   developing_test_pr
    developing_rebasing
    testing
    repomerge

docs/docsite/rst/dev_guide/overview_architecture.rst

@@ -25,7 +25,7 @@ Inventory
 
 By default, Ansible represents what machines it manages using a very simple INI file that puts all of your managed machines in groups of your own choosing.
 
-To add new machines, there is no additional SSL signing server involved, so there's never any hassle deciding why a particular machine didn’t get linked up due to obscure NTP or DNS issues.
+To add new machines, there is no additional SSL signing server involved, so there's never any hassle deciding why a particular machine didn't get linked up due to obscure NTP or DNS issues.
 
 If there's another source of truth in your infrastructure, Ansible can also plugin to that, such as drawing inventory, group, and variable information from sources like EC2, Rackspace, OpenStack, and more.
 
@@ -69,4 +69,4 @@ Here's what a simple playbook looks like::
 Extending Ansible with Plug-ins and the API
 ````````````````````````````````````````````
 
-Should you want to write your own, Ansible modules can be written in any language that can return JSON (Ruby, Python, bash, etc). Inventory can also plug in to any datasource by writing a program that speaks to that datasource and returns JSON. There's also various Python APIs for extending Ansible’s connection types (SSH is not the only transport possible), callbacks (how Ansible logs, etc), and even for adding new server side behaviors.
+Should you want to write your own, Ansible modules can be written in any language that can return JSON (Ruby, Python, bash, etc). Inventory can also plug in to any datasource by writing a program that speaks to that datasource and returns JSON. There's also various Python APIs for extending Ansible's connection types (SSH is not the only transport possible), callbacks (how Ansible logs, etc), and even for adding new server side behaviors.

docs/docsite/rst/dev_guide/style_guide/grammar_punctuation.rst

@@ -110,7 +110,7 @@ Never use "we" when writing. "We" aren't doing anything on the user side. Ansibl
 
 Hyphen
 ~~~~~~~~~~~~~~
-The hyphen’s primary function is the formation of certain compound terms. Do not use a hyphen unless it serves a purpose. If a compound adjective cannot be misread or, as with many psychological terms, its meaning is established, a hyphen is not necessary.
+The hyphen's primary function is the formation of certain compound terms. Do not use a hyphen unless it serves a purpose. If a compound adjective cannot be misread or, as with many psychological terms, its meaning is established, a hyphen is not necessary.
 
 Use hyphens to avoid ambiguity or confusion:
 

docs/docsite/rst/dev_guide/style_guide/spelling_word_choice.rst

@@ -78,7 +78,7 @@ Use to describes where to place options for a command, but not where to type the
 Daylight saving time (DST)
 +++++++++++++++++++++++++++++++
 
-Correct. Do not use daylight savings time. Daylight Saving Time (DST) is often misspelled “Daylight Savings”, with an “s” at the end. Other common variations are “Summer Time”and “Daylight-Saving Time”. (http://www.timeanddate.com/time/dst/daylight-savings-time.html)
+Correct. Do not use daylight savings time. Daylight Saving Time (DST) is often misspelled "Daylight Savings", with an "s" at the end. Other common variations are "Summer Time"and "Daylight-Saving Time". (http://www.timeanddate.com/time/dst/daylight-savings-time.html)
 
 
 Download
@@ -99,7 +99,7 @@ When used as a verb, fail over is two words since there can be different tenses
 
 Fewer
 +++++++++++++++++++
-Fewer is used with plural nouns. Think things you could count.  Time, money, distance, and weight are often listed as exceptions to the traditional “can you count it” rule, often thought of a singular amounts (the work will take less than 5 hours, for example).
+Fewer is used with plural nouns. Think things you could count.  Time, money, distance, and weight are often listed as exceptions to the traditional "can you count it" rule, often thought of a singular amounts (the work will take less than 5 hours, for example).
 
 File name
 +++++++++++++

docs/docsite/rst/dev_guide/style_guide/why_use.rst

@@ -13,8 +13,8 @@ This style guide incorporates current Ansible resources and information so that
 
    <blockquote class="note info">
 
-   “If you don't find it in the index, look very carefully through the entire catalogue.” 
-    ― Sears, Roebuck and Co., 1897 Sears Roebuck & Co. Catalogue 
+   "If you don't find it in the index, look very carefully through the entire catalogue."
+    ― Sears, Roebuck and Co., 1897 Sears Roebuck & Co. Catalogue
  
 .. raw:: html
 

docs/docsite/rst/dev_guide/testing/sanity/no-smart-quotes.rst

@@ -0,0 +1,4 @@
+Sanity Tests » no-smart-quotes
+==============================
+
+Smart quotes (``”“‘’``) should not be used.  Use plain ascii quotes (``"'``) instead.

docs/docsite/rst/dev_guide/testing/sanity/no-unicode-literals.rst

@@ -0,0 +1,16 @@
+Sanity Tests » no-unicode_literals
+==================================
+
+The use of :code:`from __future__ import unicode_literals` has been deemed an anti-pattern.  The
+problems with it are:
+
+* It makes it so one can't jump into the middle of a file and know whether a bare literal string is
+  a byte string or text string.  The programmer has to first check the top of the file to see if the
+  import is there.
+* It removes the ability to define native strings (a string which should be a byte string on python2
+  and a text string on python3) via a string literal.
+* It makes for more context switching.  A programmer could be reading one file which has
+  `unicode_literals` and know that bare string literals are text strings but then switch to another
+  file (perhaps tracing program execution into a third party library) and have to switch their
+  understanding of what bare string literals are.
+

docs/docsite/rst/galaxy.rst

@@ -152,7 +152,7 @@ Content of the *requirements.yml* file:
     # from galaxy
     - src: yatesr.timezone
 
-    - include: <path_to_requirements>/webserver.yml
+    - import_tasks: <path_to_requirements>/webserver.yml
 
 
 Content of the *webserver.yml* file:

docs/docsite/rst/glossary.rst

@@ -436,7 +436,7 @@ when a term comes up on the mailing list.
         labeled ``ntp``, and then run just the ``ntp`` steps to reconfigure
         the time server information on a remote host.
 
-    Tasks
+    Task
         :term:`Playbooks` exist to run tasks.  Tasks combine an :term:`action`
         (a module and its arguments) with a name and optionally some other
         keywords (like :term:`looping directives <loops>`).   :term:`Handlers`
@@ -444,6 +444,9 @@ when a term comes up on the mailing list.
         unless they are notified by name when a task reports an underlying
         change on a remote system.
 
+    Tasks
+        A list of :term:`Task`.
+
     Templates
         Ansible can easily transfer files to remote systems but often it is
         desirable to substitute variables in other files.  Variables may come

docs/docsite/rst/guide_azure.rst

@@ -40,11 +40,11 @@ There is now a detailed official tutorial describing `how to create a service pr
 
 After stepping through the tutorial you will have:
 
-* Your Client ID, which is found in the “client id” box in the “Configure” page of your application in the Azure portal
+* Your Client ID, which is found in the "client id" box in the "Configure" page of your application in the Azure portal
 * Your Secret key, generated when you created the application. You cannot show the key after creation.
-  If you lost the key, you must create a new one in the “Configure” page of your application.
-* And finally, a tenant ID. It’s a UUID (e.g. ABCDEFGH-1234-ABCD-1234-ABCDEFGHIJKL) pointing to the AD containing your
-  application. You will find it in the URL from within the Azure portal, or in the “view endpoints” of any given URL.
+  If you lost the key, you must create a new one in the "Configure" page of your application.
+* And finally, a tenant ID. It's a UUID (e.g. ABCDEFGH-1234-ABCD-1234-ABCDEFGHIJKL) pointing to the AD containing your
+  application. You will find it in the URL from within the Azure portal, or in the "view endpoints" of any given URL.
 
 
 Using Active Directory Username/Password

docs/docsite/rst/index.rst

@@ -16,7 +16,7 @@ We believe simplicity is relevant to all sizes of environments, so we design for
 Ansible manages machines in an agent-less manner. There is never a question of how to
 upgrade remote daemons or the problem of not being able to manage systems because daemons are uninstalled.  Because OpenSSH is one of the most peer-reviewed open source components, security exposure is greatly reduced. Ansible is decentralized--it relies on your existing OS credentials to control access to remote machines. If needed, Ansible can easily connect with Kerberos, LDAP, and other centralized authentication management systems.
 
-This documentation covers the current released version of Ansible (2.3) and also some development version features (2.4).  For recent features, we note in each section the version of Ansible where the feature was added.
+This documentation covers the current released version of Ansible (2.4) and also some development version features ('devel').  For recent features, we note in each section the version of Ansible where the feature was added.
 
 Ansible, Inc. releases a new major release of Ansible approximately every two months.  The core application evolves somewhat conservatively, valuing simplicity in language design and setup. However, the community around new modules and plugins being developed and contributed moves very quickly, typically adding 20 or so new modules in each release.
 
@@ -32,6 +32,7 @@ Ansible, Inc. releases a new major release of Ansible approximately every two mo
    modules
    modules_by_category
    vault
+   command_line_tools
    guides
    dev_guide/index
    tower

docs/docsite/rst/intro_adhoc.rst

@@ -195,7 +195,7 @@ Deploying From Source Control
 
 Deploy your webapp straight from git::
 
-    $ ansible webservers -m git -a "repo=git://foo.example.org/repo.git dest=/srv/myapp version=HEAD"
+    $ ansible webservers -m git -a "repo=https://foo.example.org/repo.git dest=/srv/myapp version=HEAD"
 
 Since Ansible modules can notify change handlers it is possible to
 tell Ansible to run specific tasks when the code is updated, such as

docs/docsite/rst/intro_bsd.rst

@@ -27,7 +27,7 @@ Bootstrapping BSD
 As mentioned above, you can bootstrap Ansible with the ``raw`` module and remotely install Python on targets. The following example installs Python 2.7 which includes the json library required for full functionality of Ansible.
 On your control machine you can simply execute the following for most versions of FreeBSD::
 
-    ansible -m raw -a “pkg install -y python27” mybsdhost1
+    ansible -m raw -a "pkg install -y python27" mybsdhost1
 
 Once this is done you can now use other Ansible modules apart from the ``raw`` module.
 

docs/docsite/rst/intro_configuration.rst

@@ -443,7 +443,7 @@ value is a TTL in seconds::
 fact_path
 =========
 
-This option allows you to globally configure a custom path for :ref:`_local_facts`:: for the implied `setup` task when using implied fact gathering.
+This option allows you to globally configure a custom path for :ref:`local_facts` for the implied `setup` task when using implied fact gathering.
 
   fact_path = /home/centos/ansible_facts.d
 
@@ -657,15 +657,15 @@ merge_multiple_cli_tags
 
 .. versionadded:: 2.3
 
-This allows changing how multiple :option:`--tags` and :option:`--skip-tags`
-arguments are handled on the command line.  Specifying :option:`--tags` more
-than once merges all of the :option:`--tags` options together.  If you want
-the pre-2.4.x behaviour where only the last value of :option:`--tags` is used,
-then set this to False.  The same holds true for :option:`--skip-tags`.
+This allows changing how multiple :option:`--tags <ansible-playbook --tags>` and :option:`--skip-tags <ansible-playbook --skip-tags>`
+arguments are handled on the command line.  Specifying :option:`--tags <ansible-playbook --tags>` more
+than once merges all of the :option:`--tags <ansible-playbook --tags>` options together.  If you want
+the pre-2.4.x behaviour where only the last value of :option:`--tags <ansible-playbook --tags>` is used,
+then set this to False.  The same holds true for :option:`--skip-tags <ansible-playbook --skip-tags>`.
 
 .. note:: The default value for this in 2.3 is False.  In 2.4, the
     default value is True.  After 2.8, the option will be removed.
-    Multiple :option:`--tags` and multiple :option:`--skip-tags` will always
+    Multiple :option:`--tags <ansible-playbook --tags>` and multiple :option:`--skip-tags <ansible-playbook --skip-tags>` will always
     be merged together.
 
 .. _module_lang:
@@ -814,26 +814,6 @@ always default to the current user if this is not defined::
     remote_user = root
 
 
-.. _restrict_facts_namespace:
-
-restrict_facts_namespace
-========================
-
-.. versionadded:: 2.4
-
-This allows restricting facts in their own namespace (under ansible_facts) instead of pushing them into the main.
-False by default. Can also be set via the environment variable :envvar:`ANSIBLE_RESTRICT_FACTS`. Using `ansible_system` as an example:
-
-When False::
-
-    - debug: var=ansible_system
-
-
-When True::
-
-    - debug: var=ansible_facts.ansible_system
-
-
 .. _retry_files_enabled:
 
 retry_files_enabled
@@ -1025,7 +1005,7 @@ As of 1.7 this file can also be a script. If you are using a script instead of a
 Privilege Escalation Settings
 -----------------------------
 
-Ansible can use existing privilege escalation systems to allow a user to execute tasks as another. As of 1.9 ‘become’ supersedes the old sudo/su, while still being backwards compatible.  Settings live under the [privilege_escalation] header.
+Ansible can use existing privilege escalation systems to allow a user to execute tasks as another. As of 1.9 'become' supersedes the old sudo/su, while still being backwards compatible.  Settings live under the [privilege_escalation] header.
 
 .. _become:
 

docs/docsite/rst/intro_installation.rst

@@ -104,10 +104,10 @@ Installing the Control Machine
 Latest Release Via Yum
 ++++++++++++++++++++++
 
-.. note:: We’ve changed how the Ansible community packages are distributed. 
+.. note:: We've changed how the Ansible community packages are distributed.
   For users of RHEL/CentOS/Scientific Linux version 7, the Ansible community RPM
   package will transition from the EPEL repository to the Extras channel.  There will be no
-  change for version 6 of RHEL/CentOS/Scientific Linux since Extras is not a part of version 6.   
+  change for version 6 of RHEL/CentOS/Scientific Linux since Extras is not a part of version 6.
 
 RPMs for RHEL7 are available from `the Extras channel <https://access.redhat.com/solutions/912213>`_.
 
@@ -115,16 +115,16 @@ RPMs for RHEL6 are available from yum for `EPEL
 <http://fedoraproject.org/wiki/EPEL>`_ 6 and currently supported
 Fedora distributions.
 
-Ansible will also have RPMs/YUM-repo available at `<https://releases.ansible.com/ansible/rpms/`.
+Ansible will also have RPMs/YUM-repo available at `<https://releases.ansible.com/ansible/rpm/`_.
 
 Ansible version 2.4 can manage earlier operating
 systems that contain Python 2.6 or higher.
 
-You can also build an RPM yourself.  From the root of a checkout or tarball, use the ``make rpm`` command to build an RPM you can distribute and install. 
+You can also build an RPM yourself.  From the root of a checkout or tarball, use the ``make rpm`` command to build an RPM you can distribute and install.
 
 .. code-block:: bash
 
-    $ git clone git://github.com/ansible/ansible.git 
+    $ git clone https://github.com/ansible/ansible.git
     $ cd ./ansible
     $ make rpm
     $ sudo rpm -Uvh ./rpm-build/ansible-*.noarch.rpm
@@ -263,7 +263,7 @@ Then install Ansible with [1]_::
 
 Or if you are looking for the latest development version::
 
-    pip install git+git://github.com/ansible/ansible.git@devel
+    pip install git+https://github.com/ansible/ansible.git@devel
 
 If you are installing on OS X Mavericks, you may encounter some noise from your compiler.  A workaround is to do the following::
 
@@ -305,7 +305,7 @@ To install from source, clone the Ansible git repository:
 
 .. code-block:: bash
 
-    $ git clone git://github.com/ansible/ansible.git --recursive
+    $ git clone https://github.com/ansible/ansible.git --recursive
     $ cd ./ansible
 
 Once git has cloned the Ansible repository, setup the Ansible environment:

docs/docsite/rst/intro_inventory.rst

@@ -11,7 +11,7 @@ which defaults to being saved in the location ``/etc/ansible/hosts``.
 You can specify a different inventory file using the ``-i <path>`` option on the command line.
 
 Not only is this inventory configurable, but you can also use multiple inventory files at the same time and
-pull inventory from dynamic or cloud sources, as described in :doc:`intro_dynamic_inventory`.
+pull inventory from dynamic or cloud sources or different formats (YAML, ini, etc), as described in :doc:`intro_dynamic_inventory`.
 Introduced in version 2.4, Ansible has inventory plugins to make this flexible and customizable.
 
 .. _inventoryformat:
@@ -38,12 +38,30 @@ For this example, the format for ``/etc/ansible/hosts`` is an INI-like (one of A
 The headings in brackets are group names, which are used in classifying systems
 and deciding what systems you are controlling at what times and for what purpose.
 
+A YAML version would look like:
+
+.. code-block:: yaml
+
+  all:
+    hosts:
+      mail.example.com
+    children:
+      webservers:
+        hosts:
+          foo.example.com:
+          bar.example.com:
+      dbservers:
+        hosts:
+          one.example.com:
+          two.example.com:
+          three.example.com:
+
+
 It is ok to put systems in more than one group, for instance a server could be both a webserver and a dbserver.
 If you do, note that variables will come from all of the groups they are a member of. Variable precedence is detailed in a later chapter.
 
-If you have hosts that run on non-standard SSH ports you can put the port number
-after the hostname with a colon.  Ports listed in your SSH config file won't be used with the `paramiko`
-connection but will be used with the `openssh` connection.
+If you have hosts that run on non-standard SSH ports you can put the port number after the hostname with a colon.
+Ports listed in your SSH config file won't be used with the `paramiko` connection but will be used with the `openssh` connection.
 
 To make things explicit, it is suggested that you set them if things are not running on the default port:
 
@@ -51,20 +69,33 @@ To make things explicit, it is suggested that you set them if things are not run
 
     badwolf.example.com:5309
 
-Suppose you have just static IPs and want to set up some aliases that live in your host file, or you are connecting through tunnels.  You can also describe hosts like this:
+Suppose you have just static IPs and want to set up some aliases that live in your host file, or you are connecting through tunnels.
+You can also describe hosts via variables:
+
+In INI:
 
 .. code-block:: ini
 
     jumper ansible_port=5555 ansible_host=192.0.2.50
 
-In the above example, trying to ansible against the host alias "jumper" (which may not even be a real hostname) will contact 192.0.2.50 on port 5555.  Note that this is using a feature of the inventory file to define some special variables.  Generally speaking this is not the best
-way to define variables that describe your system policy, but we'll share suggestions on doing this later.  We're just getting started.
+In YAML:
 
-.. note:: Values passed in using the ``key=value`` syntax are interpreted as Python literal structure (strings, numbers, tuples, lists, dicts,
-          booleans, None), alternatively as string. For example ``var=FALSE`` would create a string equal to 'FALSE'. Do not rely on types set
-          during definition, always make sure you specify type with a filter when needed when consuming the variable.
+.. code-block:: yaml
 
-Adding a lot of hosts?  If you have a lot of hosts following similar patterns you can do this rather than listing each hostname:
+    hosts:
+      jumper:
+        ansible_port: 5555
+        ansible_host: 192.0.2.50
+
+In the above example, trying to ansible against the host alias "jumper" (which may not even be a real hostname) will contact 192.0.2.50 on port 5555.
+Note that this is using a feature of the inventory file to define some special variables.
+Generally speaking, this is not the best way to define variables that describe your system policy, but we'll share suggestions on doing this later.
+
+.. note:: Values passed in the INI format using the ``key=value`` syntax are not interpreted as Python literal structure
+          (strings, numbers, tuples, lists, dicts, booleans, None), but as a string. For example ``var=FALSE`` would create a string equal to 'FALSE'.
+          Do not rely on types set during definition, always make sure you specify type with a filter when needed when consuming the variable.
+
+If you are adding a lot of hosts following similar patterns, you can do this rather than listing each hostname:
 
 .. code-block:: ini
 
@@ -91,15 +122,14 @@ You can also select the connection type and user on a per host basis:
    other1.example.com     ansible_connection=ssh        ansible_user=mpdehaan
    other2.example.com     ansible_connection=ssh        ansible_user=mdehaan
 
-As mentioned above, setting these in the inventory file is only a shorthand, and we'll discuss how to store them in individual files
-in the 'host_vars' directory a bit later on.
+As mentioned above, setting these in the inventory file is only a shorthand, and we'll discuss how to store them in individual files in the 'host_vars' directory a bit later on.
 
 .. _host_variables:
 
 Host Variables
 ++++++++++++++
 
-As alluded to above, it is easy to assign variables to hosts that will be used later in playbooks:
+As described above, it is easy to assign variables to hosts that will be used later in playbooks:
 
 .. code-block:: ini
 
@@ -112,7 +142,11 @@ As alluded to above, it is easy to assign variables to hosts that will be used l
 Group Variables
 +++++++++++++++
 
-Variables can also be applied to an entire group at once::
+Variables can also be applied to an entire group at once:
+
+The INI way:
+
+.. code-block:: ini
 
    [atlanta]
    host1
@@ -122,14 +156,30 @@ Variables can also be applied to an entire group at once::
    ntp_server=ntp.atlanta.example.com
    proxy=proxy.atlanta.example.com
 
-Be aware that this is only a convenient way to apply variables to multiple hosts at once; even though you can target hosts by group, variables are always flattened to the host level before a play is executed.
+The YAML version:
+
+.. code-block:: yaml
+
+    atlanta:
+      hosts:
+        host1:
+        host2:
+      vars:
+        ntp_server: ntp.atlanta.example.com
+        proxy: proxy.atlanta.example.com
+
+Be aware that this is only a convenient way to apply variables to multiple hosts at once; even though you can target hosts by group, **variables are always flattened to the host level** before a play is executed.
 
 .. _subgroups:
 
 Groups of Groups, and Group Variables
 +++++++++++++++++++++++++++++++++++++
 
-It is also possible to make groups of groups using the ``:children`` suffix. Just like above, you can apply variables using ``:vars``::
+It is also possible to make groups of groups using the ``:children`` suffix in INI or the ``children:`` entry in YAML.
+You can apply variables using ``:vars`` or ``vars:``:
+
+
+.. code-block:: ini
 
    [atlanta]
    host1
@@ -155,11 +205,38 @@ It is also possible to make groups of groups using the ``:children`` suffix. Jus
    southwest
    northwest
 
+.. code-block:: yaml
+
+  all:
+    children:
+      usa:
+        children:
+          southeast:
+            children:
+              atlanta:
+                hosts:
+                  host1:
+                  host2:
+              raleigh:
+                hosts:
+                  host2:
+                  host3:
+            vars:
+              some_server: foo.southeast.example.com
+              halon_system_timeout: 30
+              self_destruct_countdown: 60
+              escape_pods: 2
+         northeast:
+         northwest:
+         southwest:
+
 If you need to store lists or hash data, or prefer to keep host and group specific variables separate from the inventory file, see the next section.
 Child groups have a couple of properties to note:
 
- - First, any host that is member of a child group is automatically a member of the parent group.
- - Second, a child group's variables will have higher precedence (override) a parent group's variables.
+ - Any host that is member of a child group is automatically a member of the parent group.
+ - A child group's variables will have higher precedence (override) a parent group's variables.
+ - Groups can have multiple parents and children, but not circular relationships.
+ - Hosts can also be in multiple groups, but there will only be **one** instance of a host, merging the data from the multiple groups.
 
 .. _default_groups:
 
@@ -168,20 +245,20 @@ Default groups
 
 There are two default groups: ``all`` and ``ungrouped``. ``all`` contains every host.
 ``ungrouped`` contains all hosts that don't have another group aside from ``all``.
+Every host will always belong to at least 2 groups.
+Though ``all`` and ``ungrouped`` are always present, they can be implicit and not appear in group listings like ``group_names``.
 
 .. _splitting_out_vars:
 
 Splitting Out Host and Group Specific Data
 ++++++++++++++++++++++++++++++++++++++++++
 
-The preferred practice in Ansible is actually not to store variables in the main inventory file.
+The preferred practice in Ansible is to not store variables in the main inventory file.
 
-In addition to storing variables directly in the INI file, host
-and group variables can be stored in individual files relative to the
-inventory file.
+In addition to storing variables directly in the inventory file, host and group variables can be stored in individual files relative to the inventory file (not directory, it is always the file).
 
-These variable files are in YAML format. Valid file extensions include '.yml', '.yaml', '.json',
-or no file extension. See :doc:`YAMLSyntax` if you are new to YAML.
+These variable files are in YAML format. Valid file extensions include '.yml', '.yaml', '.json', or no file extension.
+See :doc:`YAMLSyntax` if you are new to YAML.
 
 Assuming the inventory file path is::
 
@@ -202,9 +279,9 @@ the 'raleigh' group might look like::
     ntp_server: acme.example.org
     database_server: storage.example.org
 
-It is ok if these files do not exist, as this is an optional feature.
+It is okay if these files do not exist, as this is an optional feature.
 
-As an advanced use-case, you can create *directories* named after your groups or hosts, and
+As an advanced use case, you can create *directories* named after your groups or hosts, and
 Ansible will read all the files in these directories. An example with the 'raleigh' group::
 
     /etc/ansible/group_vars/raleigh/db_settings
@@ -372,7 +449,7 @@ Here is an example of how to instantly deploy to created containers::
    :doc:`intro_adhoc`
        Examples of basic commands
    :doc:`playbooks`
-       Learning Ansible’s configuration, deployment, and orchestration language.
+       Learning Ansible's configuration, deployment, and orchestration language.
    `Mailing List <http://groups.google.com/group/ansible-project>`_
        Questions? Help? Ideas?  Stop by the list on Google Groups
    `irc.freenode.net <http://irc.freenode.net>`_

docs/docsite/rst/modules_support.rst

@@ -1,51 +1,50 @@
-Module Support
---------------
+Module Maintenance & Support
+----------------------------
 
 .. toctree:: :maxdepth: 1
 
-Ansible has many modules, but not all of them are maintained by the core project committers. Each module should have associated metadata that indicates which of the following categories they fall into. This should be visible in each module's documentation.
-
-Documentation updates for each module can also be edited directly in the module and by submitting a pull request to the module source code; just look for the "DOCUMENTATION" block in the source tree.
-
-If you believe you have found a bug in a module and are already running the latest stable or development version of Ansible, first look in the `issue tracker at github.com/ansible/ansible <https://github.com/ansible/ansible/issues>`_ to see if a bug has already been filed.  If not, we would be grateful if you would file one.
-
-Should you have a question rather than a bug report, inquiries are welcome on the `ansible-project google group <https://groups.google.com/forum/#!forum/ansible-project>`_ or on Ansible's "#ansible" channel, located on irc.freenode.net.
-
-For development-oriented topics, use the `ansible-devel google group <https://groups.google.com/forum/#!forum/ansible-devel>`_  or Ansible's ``#ansible`` and ``#ansible-devel`` channels, located on irc.freenode.net.  You should also read :doc:`community`, :doc:`dev_guide/testing` and :doc:`dev_guide/developing_modules`.
-
-The modules are hosted on GitHub in a subdirectory of the `ansible <https://github.com/ansible/ansible/tree/devel/lib/ansible/modules>`_ repo.
+To help identify maintainers and understand how the included modules are officially supported, each module now has associated metadata that provides additional clarity for maintenance and support.
 
 Core
 ````
 
-These are :doc:`modules maintained by the Ansible Core Team<core_maintained>`
-and will always ship with Ansible itself.
+:doc:`Core modules are maintained by the Ansible Engineering Team<core_maintained>`.
+These modules are integral to the basic foundations of the Ansible distribution.
 
 Network
 ```````
 
-These are :doc:`modules maintained by the Ansible Network Team<network_maintained>` in a relationship
-similar to how the Ansible Core Team maintains the Core modules.
+:doc:`Network modules are maintained by the Ansible Network Team<network_maintained>`. Please note there are :doc:`additional networking modules<list_of_network_modules>` that are categorized as Certified or Community not maintained by Ansible.
+
 
 Certified
 `````````
 
-Some examples of :doc:`Certified Modules<partner_maintained>` are those submitted by other companies. Maintainers of these types of modules must watch for any issues reported or pull requests raised against the module.
-
-The Ansible Core Team will review all modules becoming certified.  Core committers will review proposed changes to existing Certified Modules once the community maintainers of the module have approved the changes. Core committers will also ensure that any issues that arise due to Ansible engine changes will be remediated.
-Also, it is strongly recommended (but not presently required) for these types of modules to have unit tests.
-
-These modules are currently shipped with Ansible, but might be shipped separately in the future.
+Certified modules are part of a future planned program currently in development.
 
 Community
 `````````
 
-These are :doc:`modules maintained by the Ansible Community<community_maintained>`.  They **are not** supported by the Ansible Core Team or by companies/partners associated to the module.
+:doc:`Community modules are submitted and maintained by the Ansible community<community_maintained>`.  These modules are not maintained by Ansible, and are included as a convenience.
 
-They are still fully usable, but the response rate to issues is purely up to the community.  Best effort support will be provided but is not covered under any support contracts.
+Issue Reporting
+```````````````
 
-These modules are currently shipped with Ansible, but will most likely be shipped separately in the future.
+If you believe you have found a bug in a module and are already running the latest stable or development version of Ansible, first look at the `issue tracker in the Ansible repo <https://github.com/ansible/ansible/issues>`_ to see if an issue has already been filed. If not, please file one.
 
+Should you have a question rather than a bug report, inquiries are welcome on the `ansible-project Google group <https://groups.google.com/forum/#%21forum/ansible-project>`_ or on Ansible's "#ansible" channel, located on irc.freenode.net.
+
+For development-oriented topics, use the `ansible-devel Google group <https://groups.google.com/forum/#%21forum/ansible-devel>`_ or Ansible's #ansible and #ansible-devel channels, located on irc.freenode.net. You should also read :doc:`Community Information & Contributing <community>`, :doc:`Testing Ansible <dev_guide/testing>`, and :doc:`Developing Modules <dev_guide/developing_modules>`.
+
+The modules are hosted on GitHub in a subdirectory of the `Ansible <https://github.com/ansible/ansible/tree/devel/lib/ansible/modules>`_ repo.
+
+NOTE: If you have a Red Hat Ansible Engine product subscription, please follow the standard issue reporting process via the Red Hat Customer Portal.
+
+Support
+```````
+
+For more information on how included Ansible modules are supported by Red Hat,
+please refer to the following `knowledgebase article <https://access.redhat.com/articles/3166901>`_ as well as other resources on the `Red Hat Customer Portal. <https://access.redhat.com/>`_
 
 .. seealso::
 

docs/docsite/rst/network_debug_troubleshooting.rst

@@ -32,7 +32,7 @@ Errors generally fall into one of the following categories:
   * Can occur when trying to pull a large amount of data
   * May actually be masking a authentication issue
 :Playbook issues:
-  * Use of ``delegate_to``, instead of ``ProxyCommand``
+  * Use of ``delegate_to``, instead of ``ProxyCommand``. See :ref:`network proxy guide <network_delegate_to_vs_ProxyCommand>` for more information.
   * Not using ``connection: local``
 
 
@@ -51,7 +51,7 @@ Enabling Networking logging and how to read the logfile
 
 Ansible 2.3 features improved logging to help diagnose and troubleshoot issues regarding Ansible Networking modules.
 
-Because logging is very verbose it is disabled by default. It can be enabled via the :envvar:`ANSIBLE_LOG_PATH` and :envvar:`ANISBLE_DEBUG` options::
+Because logging is very verbose it is disabled by default. It can be enabled via the :envvar:`ANSIBLE_LOG_PATH` and :envvar:`ANSIBLE_DEBUG` options::
 
    # Specify the location for the log file
    export ANSIBLE_LOG_PATH=~/ansible.log
@@ -410,10 +410,10 @@ For example:
 
 Suggestions to resolve:
 
-Increase value of presistent connection idle timeout.
-Note: This value should be greater than SSH timeout ie. timeout value under defaults
-section in configuration file and less than the value of the persistent
-connection idle timeout (connect_timeout)
+Increase the value of the persistent connection idle timeout.
+Note: This value should be greater than the SSH timeout value (the timeout value under the defaults
+section in the configuration file) and less than the value of the persistent
+connection idle timeout (connect_timeout).
 
 .. code-block:: yaml
 
@@ -493,7 +493,7 @@ Add ``authorize: yes`` to the task. For example:
 If the user requires a password to go into privileged mode, this can be specified with ``auth_pass``; if ``auth_pass`` isn't set, the environment variable :envvar:`ANSIBLE_NET_AUTHORIZE` will be used instead.
 
 
-Add `authorize: yes` to the task. For example:
+Add ``authorize: yes`` to the task. For example:
 
 .. code-block:: yaml
 
@@ -506,40 +506,41 @@ Add `authorize: yes` to the task. For example:
   register: result
 
 
-.. delete_to not honoured
-   ----------------------
+Proxy Issues
+============
 
-   FIXME Do we get an error message
+ .. _network_delegate_to_vs_ProxyCommand:
 
-   FIXME Link to howto
+delegate_to vs ProxyCommand
+---------------------------
+
+The new connection framework for Network Modules in Ansible 2.3 that uses ``cli`` transport
+no longer supports the use of the ``delegate_to`` directive.
+In order to use a bastion or intermediate jump host to connect to network devices over ``cli``
+transport, network modules now support the use of ``ProxyCommand``.
+
+To use ``ProxyCommand``, configure the proxy settings in the Ansible inventory
+file to specify the proxy host.
+
+.. code-block:: ini
+
+    [nxos]
+    nxos01
+    nxos02
+
+    [nxos:vars]
+    ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion01"'
 
 
+With the configuration above, simply build and run the playbook as normal with
+no additional changes necessary.  The network module will now connect to the
+network device by first connecting to the host specified in
+``ansible_ssh_common_args``, which is ``bastion01`` in the above example.
 
 
-   fixmes
-   ======
+.. note:: Using ``ProxyCommand`` with passwords via variables
 
-   Error: "number of connection attempts exceeded, unable to connect to control socket"
-   ------------------------------------------------------------------------------------
+   By design, SSH doesn't support providing passwords via environment variables.
+   This is done to prevent secrets from leaking out, for example in ``ps`` output.
 
-   **Platforms:** Any
-
-   This occurs when Ansible wasn't able to connect to the remote device and obtain a shell with the timeout.
-
-
-   This information is available when :ref:`DEFAULT_LOG_PATH` is set see (FIXMELINKTOSECTION):
-
-   .. code-block:: yaml
-
-     less $ANSIBLE_LOG_PATH
-     2017-03-10 15:32:06,173 p=19677 u=fred |  connect retry timeout expired, unable to connect to control socket
-     2017-03-10 15:32:06,174 p=19677 u=fred |  persistent_connect_retry_timeout is 15 secs
-     2017-03-10 15:32:06,222 p=19669 u=fred |  fatal: [veos01]: FAILED! => {
-
-   Suggestions to resolve:
-
-   Do stuff For example:
-
-   .. code-block:: yaml
-
-   	Example stuff
+   We recommend using SSH Keys, and if needed an ssh-agent, rather than passwords, where ever possible.

docs/docsite/rst/playbooks_async.rst

@@ -94,7 +94,7 @@ of tasks running concurrently, you can do it this way::
           - 4
           - 5
         durations: "{{ item }}"
-      include: execute_batch.yml
+      include_tasks: execute_batch.yml
       with_items:
         - "{{ sleep_durations | batch(2) | list }}"
 

docs/docsite/rst/playbooks_best_practices.rst

@@ -216,15 +216,15 @@ variables from the file "group_vars/ec2_tag_class_webserver" automatically.
 Top Level Playbooks Are Separated By Role
 `````````````````````````````````````````
 
-In site.yml, we include a playbook that defines our entire infrastructure.  Note this is SUPER short, because it's just including
-some other playbooks.  Remember, playbooks are nothing more than lists of plays::
+In site.yml, we import a playbook that defines our entire infrastructure.  This is a very short example, because it's just importing
+some other playbooks::
 
     ---
     # file: site.yml
-    - include: webservers.yml
-    - include: dbservers.yml
+    - import_playbook: webservers.yml
+    - import_playbook: dbservers.yml
 
-In a file like webservers.yml (also at the top level), we simply map the configuration of the webservers group to the roles performed by the webservers group.  Also notice this is incredibly short.  For example::
+In a file like webservers.yml (also at the top level), we map the configuration of the webservers group to the roles performed by the webservers group::
 
     ---
     # file: webservers.yml

docs/docsite/rst/playbooks_conditionals.rst

@@ -154,13 +154,13 @@ there will be accessible to future tasks::
 
 .. _when_roles_and_includes:
 
-Applying 'when' to roles and includes
-`````````````````````````````````````
+Applying 'when' to roles, imports, and includes
+```````````````````````````````````````````````
 
 Note that if you have several tasks that all share the same conditional statement, you can affix the conditional
 to a task include statement as below.  All the tasks get evaluated, but the conditional is applied to each and every task::
 
-    - include: tasks/sometasks.yml
+    - import_tasks: tasks/sometasks.yml
       when: "'reticulating splines' in output"
 
 .. note:: In versions prior to 2.0 this worked with task includes but not playbook includes.  2.0 allows it to work with both.
@@ -174,6 +174,24 @@ Or with a role::
 You will note a lot of 'skipped' output by default in Ansible when using this approach on systems that don't match the criteria.
 Read up on the 'group_by' module in the :doc:`modules` docs for a more streamlined way to accomplish the same thing.
 
+When used with `include_*` tasks instead of imports, the conditional is applied _only_ to the include task itself and not any other
+tasks within the included file(s). A common situation where this distinction is important is as follows::
+
+    # include a file to define a variable when it is not already defined
+
+    # main.yml
+    - include_tasks: other_tasks.yml
+      when: x is not defined
+
+    # other_tasks.yml
+    - set_fact:
+        x: foo
+    - debug:
+        var: x
+
+In the above example, if ``import_tasks`` had been used instead both included tasks would have also been skipped. With ``include_tasks``
+instead, the tasks are executed as expected because the conditional is not applied to them.
+
 .. _conditional_imports:
 
 Conditional Imports

docs/docsite/rst/playbooks_intro.rst

@@ -345,7 +345,7 @@ a variable called ``vhost`` in the ``vars`` section, you could do this::
 Those same variables are usable in templates, which we'll get to later.
 
 Now in a very basic playbook all the tasks will be listed directly in that play, though it will usually
-make more sense to break up tasks using the ``include:`` directive.  We'll show that a bit later.
+make more sense to break up tasks as described in :doc:`playbooks_reuse`.
 
 .. _action_shorthand:
 
@@ -458,7 +458,7 @@ Let's run a playbook using a parallelism level of 10::
 
     ansible-playbook playbook.yml -f 10
 
-.. _ansible-pull:
+.. _playbook_ansible-pull:
 
 Ansible-Pull
 ````````````

docs/docsite/rst/playbooks_lookups.rst

@@ -458,7 +458,7 @@ Please check https://api.mongodb.org/python/current/api/pymongo/collection.html?
 
 Since there are too many parameters for this lookup method, below is a sample playbook which shows its usage and a nice way to feed the parameters:
 
-.. code-block:: YAML+Jinja
+.. code-block:: yaml
 
     ---
     - hosts: all

docs/docsite/rst/playbooks_loops.rst

@@ -743,7 +743,7 @@ Ansible by default sets the loop variable `item` for each loop, which causes the
 As of Ansible 2.1, the `loop_control` option can be used to specify the name of the variable to be used for the loop::
 
     # main.yml
-    - include: inner.yml
+    - include_tasks: inner.yml
       with_items:
         - 1
         - 2
@@ -807,7 +807,7 @@ Because `loop_control` is not available in Ansible 2.0, when using an include wi
 for `item`::
 
     # main.yml
-    - include: inner.yml
+    - include_tasks: inner.yml
       with_items:
         - 1
         - 2

docs/docsite/rst/playbooks_prompts.rst

@@ -27,8 +27,9 @@ Here is a most basic example::
         - name: "favcolor"
           prompt: "what is your favorite color?"
 
+
 .. note::
-   Prompts for individual ``vars_prompt`` variables will be skipped for any variable that is already defined through the command line ``--extra-vars`` option, or when running from a non-interactive session (such as cron or Ansible Tower). See :ref:`_passing_variables_on_the_command_line` in the /Variables/ chapter.
+    Prompts for individual ``vars_prompt`` variables will be skipped for any variable that is already defined through the command line ``--extra-vars`` option, or when running from a non-interactive session (such as cron or Ansible Tower). See :ref:`passing_variables_on_the_command_line` in the /Variables/ chapter.
 
 If you have a variable that changes infrequently, it might make sense to
 provide a default value that can be overridden.  This can be accomplished using
@@ -77,13 +78,13 @@ You can use any crypt scheme supported by 'Passlib':
 - *sun_md5_crypt* - Sun MD5 Crypt
 - *sha256_crypt* - SHA-256 Crypt
 - *sha512_crypt* - SHA-512 Crypt
-- *apr_md5_crypt* - Apache’s MD5-Crypt variant
-- *phpass* - PHPass’ Portable Hash
+- *apr_md5_crypt* - Apache's MD5-Crypt variant
+- *phpass* - PHPass' Portable Hash
 - *pbkdf2_digest* - Generic PBKDF2 Hashes
-- *cta_pbkdf2_sha1* - Cryptacular’s PBKDF2 hash
-- *dlitz_pbkdf2_sha1* - Dwayne Litzenberger’s PBKDF2 hash
+- *cta_pbkdf2_sha1* - Cryptacular's PBKDF2 hash
+- *dlitz_pbkdf2_sha1* - Dwayne Litzenberger's PBKDF2 hash
 - *scram* - SCRAM Hash
-- *bsd_nthash* - FreeBSD’s MCF-compatible nthash encoding
+- *bsd_nthash* - FreeBSD's MCF-compatible nthash encoding
 
 However, the only parameters accepted are 'salt' or 'salt_size'. You can use your own salt using
 'salt', or have one generated automatically using 'salt_size'. If nothing is specified, a salt

docs/docsite/rst/playbooks_reuse.rst

@@ -9,10 +9,12 @@ Creating Reusable Playbooks
 
 While it is possible to write a playbook in one very large file (and you might start out learning playbooks this way), eventually you'll want to reuse files and start to organize things. In Ansible, there are three ways to do this: includes, imports, and roles.
 
-Includes and imports allow users to break up large playbooks into smaller files, which can be used across multiple parent playbooks or even multiple times within the same Playbook.
+Includes and imports (added in 2.4) allow users to break up large playbooks into smaller files, which can be used across multiple parent playbooks or even multiple times within the same Playbook.
 
 Roles allow more than just tasks to be packaged together and can include variables, handlers, or even modules and other plugins. Unlike includes and imports, roles can also be uploaded and shared via Ansible Galaxy.
 
+.. _dynamic_vs_static:
+
 Dynamic vs. Static
 ``````````````````
 

docs/docsite/rst/playbooks_reuse_includes.rst

@@ -13,17 +13,25 @@ As noted in :doc:`playbooks_reuse`, include and import statements are very simil
 
 Please refer to  :doc:`playbooks_reuse` for documentation concerning the trade-offs one may encounter when using each type.
 
+Also be aware that this behaviour changed in 2.4; prior to that Ansible version only ``include`` was available, and it behaved differently depending on context.
+
+.. versionadded:: 2.4
+
 Importing Playbooks
 ```````````````````
 
 It is possible to include playbooks inside a master playbook. For example::
 
     ---
-    import_playbook: webservers.yml
-    import_playbook: databases.yml
+    - import_playbook: webservers.yml
+    - import_playbook: databases.yml
 
-Each playbook listed will be run in the order they are listed.
+The plays and tasks in each playbook listed will be run in the order they are listed, just as if they had been defined here directly.
 
+Prior to 2.4 only ``include`` was available and worked for both playbooks and tasks as both import and include.
+
+
+.. versionadded:: 2.4
 
 Including and Importing Task Files
 ``````````````````````````````````
@@ -61,7 +69,7 @@ Variables can also be passed to include files using an alternative syntax, which
         - "{{ lookup('file', 'keys/one.pub') }}"
         - "{{ lookup('file', 'keys/two.pub') }}"
 
-Using either syntax, variables passed in can then be used in the included files. These variables will only be available to tasks within the included file. See :doc:`variable_precedence` for more details on variable inheritance and precedence.
+Using either syntax, variables passed in can then be used in the included files. These variables will only be available to tasks within the included file. See :ref:`variable_precedence` for more details on variable inheritance and precedence.
 
 Task include statements can be used at arbitrary depth.
 

docs/docsite/rst/playbooks_reuse_roles.rst

@@ -33,8 +33,8 @@ Roles expect files to be in certain directory names. Roles must include at least
 
 - ``tasks`` - contains the main list of tasks to be executed by the role.
 - ``handlers`` - contains handlers, which may be used by this role or even anywhere outside this role.
-- ``defaults`` - default variables for the role (see :doc:`Variables` for more information).
-- ``vars`` - other variables for the role (see :doc:`Variables` for more information).
+- ``defaults`` - default variables for the role (see :doc:`playbooks_variables` for more information).
+- ``vars`` - other variables for the role (see :doc:`playbooks_variables` for more information).
 - ``files`` - contains files which can be deployed via this role.
 - ``templates`` - contains templates which can be deployed via this role.
 - ``meta`` - defines some meta data for this role. See below for more details.
@@ -42,7 +42,8 @@ Roles expect files to be in certain directory names. Roles must include at least
 Other YAML files may be included in certain directories. For example, it is common practice to have platform-specific tasks included from the ``tasks/main.yml`` file::
 
     # roles/example/tasks/main.yml
-    - import_tasks: redhat.yml
+    - name: added in 2.4, previouslly you used 'include'
+      import_tasks: redhat.yml
       when: ansible_os_platform|lower == 'redhat'
     - import_tasks: debian.yml
       when: ansible_os_platform|lower == 'debian'
@@ -57,7 +58,7 @@ Other YAML files may be included in certain directories. For example, it is comm
         name: "apache2"
         state: present
 
-Roles may also include modules and other plugin types. For more information, please refer to the :doc:`Embedding Modules and Plugins In Roles` section below.
+Roles may also include modules and other plugin types. For more information, please refer to the :ref:`embedding_modules_and_plugins_in_roles` section below.
 
 Using Roles
 ```````````
@@ -113,9 +114,9 @@ As of Ansible 2.4, you can now use roles inline with any other tasks using ``imp
 When roles are defined in the classic manner, they are treated as static imports and processed during playbook parsing.
 
 .. note::
-    The ``include_role`` option was introduced in Ansible 2.3. The usage has changed slightly as of Ansible 2.4 to match the include (dynamic) vs. import (static) usage. See :doc:`Dynamic vs. Static` for more details.
+    The ``include_role`` option was introduced in Ansible 2.3. The usage has changed slightly as of Ansible 2.4 to match the include (dynamic) vs. import (static) usage. See :ref:`dynamic_vs_static` for more details.
 
-The name used for the role can be a simple name (see :doc:`Role Search Path` below), or it can be a fully qualified path::
+The name used for the role can be a simple name (see :ref:`role_search_path` below), or it can be a fully qualified path::
 
     ---
 
@@ -290,6 +291,8 @@ Note that we did not have to use ``allow_duplicates: true`` for ``wheel``, becau
 .. note::
    Variable inheritance and scope are detailed in the :doc:`playbooks_variables`.
 
+.. _embedding_modules_and_plugins_in_roles:
+
 Embedding Modules and Plugins In Roles
 ``````````````````````````````````````
 
@@ -332,6 +335,8 @@ The same mechanism can be used to embed and distribute plugins in a role, using
 
 They can then be used in a template or a jinja template in any role called after 'my_custom_filter'
 
+.. _role_search_path:
+
 Role Search Path
 ````````````````
 

docs/docsite/rst/playbooks_tags.rst

@@ -80,9 +80,14 @@ You may also apply tags to roles::
     roles:
       - { role: webserver, port: 5000, tags: [ 'web', 'foo' ] }
 
-And include statements::
+And import/include statements::
 
-    - include: foo.yml
+    - import_tasks: foo.yml
+      tags: [web,foo]
+
+or::
+
+    - include_tasks: foo.yml
       tags: [web,foo]
 
 All of these apply the specified tags to EACH task inside the play, included

docs/docsite/rst/playbooks_vault.rst

@@ -55,7 +55,7 @@ As of version 2.3, Ansible can now use a vaulted variable that lives in an other
               34623731376664623134383463316265643436343438623266623965636363326136
     other_plain_text: othervalue
 
-To create a vaulted variable, use the :ref:`ansible-vault encrypt_string` command. See :ref:`encrypt_string` for details.
+To create a vaulted variable, use the :ref:`ansible-vault encrypt_string <ansible_vault_encrypt_string>` command. See :ref:`encrypt_string` for details.
 
 This vaulted variable will be decrypted with the supplied vault secret and used as a normal variable. The ``ansible-vault`` command line supports stdin and stdout for encrypting data on the fly, which can be used from your favorite editor to create these vaulted variables; you just have to be sure to add the ``!vault`` tag so both Ansible and YAML are aware of the need to decrypt. The ``|`` is also required, as vault encryption results in a multi-line string.
 

docs/docsite/rst/porting_guide_2.0.rst

@@ -109,7 +109,7 @@ While all items listed here will show a deprecation warning message, they still
 
 * Bare variables in ``with_`` loops should instead use the ``"{ {var }}"`` syntax, which helps eliminate ambiguity.
 * The ansible-galaxy text format requirements file. Users should use the YAML format for requirements instead.
-* Undefined variables within a ``with_`` loop’s list currently do not interrupt the loop, but they do issue a warning; in the future, they will issue an error.
+* Undefined variables within a ``with_`` loop's list currently do not interrupt the loop, but they do issue a warning; in the future, they will issue an error.
 * Using dictionary variables to set all task parameters is unsafe and will be removed in a future version. For example::
 
     - hosts: localhost
@@ -129,8 +129,8 @@ While all items listed here will show a deprecation warning message, they still
 
 * Host patterns should use a comma (,) or colon (:) instead of a semicolon (;) to separate hosts/groups in the pattern.
 * Ranges specified in host patterns should use the [x:y] syntax, instead of [x-y].
-* Playbooks using privilege escalation should always use “become*” options rather than the old su*/sudo* options.
-* The “short form” for vars_prompt is no longer supported.
+* Playbooks using privilege escalation should always use "become*" options rather than the old su*/sudo* options.
+* The "short form" for vars_prompt is no longer supported.
   For example::
 
     vars_prompt:
@@ -138,28 +138,28 @@ While all items listed here will show a deprecation warning message, they still
 
 * Specifying variables at the top level of a task include statement is no longer supported. For example::
 
-    - include: foo.yml
+    - include_tasks: foo.yml
         a: 1
 
 Should now be::
 
-    - include: foo.yml
+    - include_tasks: foo.yml
       vars:
         a: 1
 
 * Setting any_errors_fatal on a task is no longer supported. This should be set at the play level only.
-* Bare variables in the `environment` dictionary (for plays/tasks/etc.) are no longer supported. Variables specified there should use the full variable syntax: ‘{{foo}}’.
+* Bare variables in the `environment` dictionary (for plays/tasks/etc.) are no longer supported. Variables specified there should use the full variable syntax: '{{foo}}'.
 * Tags (or any directive) should no longer be specified with other parameters in a task include. Instead, they should be specified as an option on the task.
   For example::
 
-    - include: foo.yml tags=a,b,c
+    - include_tasks: foo.yml tags=a,b,c
 
   Should be::
 
-    - include: foo.yml
+    - include_tasks: foo.yml
       tags: [a, b, c]
 
-* The first_available_file option on tasks has been deprecated. Users should use the with_first_found option or lookup (‘first_found’, …) plugin.
+* The first_available_file option on tasks has been deprecated. Users should use the with_first_found option or lookup ('first_found', …) plugin.
 
 
 Other caveats

docs/docsite/rst/porting_guide_2.3.rst

@@ -219,28 +219,6 @@ to connect to network devices, network modules now support the use of
 ``ProxyCommand``.
 
 To use ``ProxyCommand`` configure the proxy settings in the Ansible inventory
-file to specify the proxy host.
-
-.. code-block:: ini
-
-    [nxos]
-    nxos01
-    nxos02
-
-    [nxos:vars]
-    ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion01"'
-
-
-With the configuration above, simply build and run the playbook as normal with
-no additional changes necessary.  The network module will now connect to the
-network device by first connecting to the host specified in
-``ansible_ssh_common_args`` which is ``bastion01`` in the above example.
-
-
-.. notes: Using ``ProxyCommand`` with passwords via variables
-
-   It is a feature that SSH doesn't support providing passwords via environment variables.
-   This is done to prevent secrets from leaking out, for example in ``ps`` output.
-
-   We recommend using SSH Keys, and if needed and ssh-agent, rather than passwords, where ever possible.
+file to specify the proxy host via ``ansible_ssh_common_args``.
 
+For details on how to do this see the :ref:`network proxy guide <network_delegate_to_vs_ProxyCommand>`.

docs/docsite/rst/porting_guide_2.4.rst

@@ -0,0 +1,163 @@
+.. _porting_2.4_guide:
+
+*************************
+Ansible 2.4 Porting Guide
+*************************
+
+This section discusses the behavioral changes between Ansible 2.3 and Ansible 2.4.
+
+It is intended to assist in updating your playbooks, plugins and other parts of your Ansible infrastructure so they will work with this version of Ansible.
+
+
+We suggest you read this page along with `Ansible Changelog <https://github.com/ansible/ansible/blob/stable-2.4/CHANGELOG.md#2.4>`_ to understand what updates you may need to make.
+
+This document is part of a collection on porting. The complete list of porting guides can be found at :ref:`porting guides <porting_guides>`.
+
+.. contents:: Topics
+
+Python version
+==============
+
+Ansible will not support Python 2.4 or 2.5 on the target hosts anymore. Going forward, Python 2.6+ will be required on targets, as already is the case on the controller.
+
+
+Inventory
+=========
+
+Inventory has been refactored to be implemented via plugins and now allows for multiple sources. This change is mostly transparent to users.
+
+One exception is the ``inventory_dir``, which is now a host variable; previously it could only have one value so it was set globally.
+This means you can no longer use it early in plays to determine ``hosts:`` or similar keywords.
+This also changes the behaviour of ``add_hosts`` and the implicit localhost; 
+because they no longer automatically inherit the global value, they default to ``None``. See the module documentation for more information.
+
+The ``inventory_file`` remains unchanged, as it was always host specific.
+
+A bug was fixed with the inventory path/directory, which was defaulting to the current working directory. This caused ``group_vars`` and ``host_vars`` to be picked up from the current working directory instead of just adjacent to the playbook or inventory directory when a host list (comma separated host names) was provided as inventory.
+
+Deprecated
+==========
+
+Specifying Inventory sources
+-----------------------------
+
+Use of ``--inventory-file`` on the command line is now deprecated. Use ``--inventory`` or ``-i``.
+The associated ini configuration key, ``hostfile``, and environment variable, :envvar:`ANSIBLE_HOSTS`,
+are also deprecated.  Replace them with the configuration key ``inventory`` and environment variable :envvar:`ANSIBLE_INVENTORY`.
+
+Use of multiple tags
+--------------------
+
+Specifying ``--tags`` (or ``--skip-tags``) multiple times on the command line currently leads to the last one overriding all the previous ones. This behavior is deprecated. In the future, if you specify --tags multiple times the tags will be merged together. From now on, using ``--tags`` multiple times on one command line will emit a deprecation warning. Setting the ``merge_multiple_cli_tags`` option to True in the ``ansible.cfg`` file will enable the new behavior.
+
+In 2.4, the default has change to merge the tags. You can enable the old overwriting behavior via the config option.
+
+In 2.5, multiple ``--tags`` options will be merged with no way to go back to the old behavior.
+
+
+Other caveats
+-------------
+
+No major changes in this version.
+
+Modules
+=======
+
+Major changes in popular modules are detailed here
+
+* The :ref:`win_shell <win_shell>` and :ref:`win_command <win_command>` modules now properly preserve quoted arguments in the command-line. Tasks that attempted to work around the issue by adding extra quotes/escaping may need to be reworked to remove the superfluous escaping. See `Issue 23019 <https://github.com/ansible/ansible/issues/23019>`_ for additional detail.
+
+Modules removed
+---------------
+
+The following modules no longer exist:
+
+* None
+
+Deprecation notices
+-------------------
+
+The following modules will be removed in Ansible 2.8. Please update update your playbooks accordingly.
+
+* :ref:`azure <azure>`, use :ref:`azure_rm_virtualmachine <azure_rm_virtualmachine>`, which uses the new Resource Manager SDK.
+* :ref:`win_msi <win_msi>`, use :ref:`win_package <win_package>` instead
+
+Noteworthy module changes
+-------------------------
+
+* The :ref:`win_get_url <win_get_url>`  module has the dictionary ``win_get_url`` in its results deprecated, its content is now also available directly in the resulting output, like other modules. This dictionary will be removed in Ansible 2.8.
+* The :ref:`win_unzip <win_unzip>` module no longer includes the dictionary ``win_unzip`` in its results; the contents are now included directly in the resulting output, like other modules.
+* The :ref:`win_package <win_package>` module return values ``exit_code`` and ``restart_required`` have been deprecated in favour of ``rc`` and ``reboot_required`` respectively. The deprecated return values will be removed in Ansible 2.6.
+
+
+Plugins
+=======
+
+A new way to configure and document plugins has been introduced.  This does not require changes to existing setups but developers should start adapting to the new infrastructure now. More details will be available in the developer documentation for each plugin type.
+
+Vars plugin changes
+-------------------
+
+There have been many changes to the implementation of vars plugins, but both users and developers should not need to change anything to keep current setups working. Developers should consider changing their plugins take advantage of new features.
+
+The most notable difference to users is that vars plugins now get invoked on demand instead of at inventory build time.  This should make them more efficient for large inventories, especially when using a subset of the hosts.
+
+
+.. note::
+  - This also creates a difference with group/host_vars when using them adjacent to playbooks. Before, the 'first' playbook loaded determined the variables; now the 'current' playbook does. We are looking to fix this soon, since 'all playbooks' in the path should be considered for variable loading.
+  - In 2.4.1 we added a toggle to allow you to control this behaviour, 'top' will be the pre 2.4, 'bottom' will use the current playbook hosting the task and 'all' will use them all from top to bottom.
+
+
+Inventory plugins
+-----------------
+
+Developers should start migrating from hardcoded inventory with dynamic inventory scripts to the new Inventory Plugins. The scripts will still work via the ``script`` inventory plugin but Ansible development efforts will now concentrate on writing plugins rather than enhancing existing scripts.
+
+Both users and developers should look into the new plugins because they are intended to alleviate the need for many of the hacks and workarounds found in the dynamic inventory scripts.
+
+Callback plugins
+----------------
+
+Users:
+
+* Callbacks are now using the new configuration system.  Users should not need to change anything as the old system still works,
+  but you might see a deprecation notice if any callbacks used are not inheriting from the built in classes. Developers need to update them as stated below.
+
+Developers:
+
+* If your callback does not inherit from ``CallbackBase`` (directly or indirectly via another callback), it will still work, but issue a deprecation notice.
+  To avoid this and ensure it works in the future change it to inherit from ``callbackBase`` so it has the new options handling methods and properties.
+  You can also implement the new options handling methods and properties but that won't automatically inherit changes added in the future.  You can look at ``CallbackBase`` itself and/or ``AnsiblePlugin`` for details.
+* Any callbacks inheriting from other callbacks might need to also be updated to contain the same documented options
+  as the parent or the options won't be available.  This is noted in the developer guide.
+
+Networking
+==========
+
+There have been a number of changes to how Networking Modules operate.
+
+Playbooks should still use ``connection: local``.
+
+Persistent Connection
+---------------------
+
+The configuration variables ``connection_retries`` and ``connect_interval`` which were added in Ansible 2.3 are now deprecated. For Ansible 2.4 and later use ``connection_retry_timeout``.
+
+To control timeouts use ``command_timeout`` rather than the previous top level ``timeout`` variable under ``[default]``
+
+See :ref:`Ansible Network debug guide <network_debug_troubleshooting>` for more information.
+
+
+Configuration
+=============
+
+
+The configuration system has had some major changes. Users should be unaffected except for the following:
+
+* All relative paths defined are relative to the `ansible.cfg` file itself. Previously they varied by setting. The new behavior should be more predictable.
+* A new macro ``{{CWD}}`` is available for paths, which will make paths relative to the 'current working directory',
+  this is unsafe but some users really want to rely on this behaviour.
+
+Developers that were working directly with the previous API should revisit their usage as some methods (for example, ``get_config``) were  kept for backwards compatibility but will warn users that the function has been deprecated.
+
+The new configuration has been designed to minimize the need for code changes in core for new plugins.  The plugins just need to document their settings and the configuration system will use the documentation to provide what they need. This is still a work in progress; currently only 'callback' and 'connection' plugins support this.  More  details will be added to the specific plugin developer guides.

docs/docsite/rst/release_and_maintenance.rst

@@ -33,13 +33,15 @@ Release status
 ``````````````
 
 +-----------------+----------------------------+----------------------------------------+
-| Ansible release | Latest version             | Status                                 |
+| Ansible Release | Latest Version             | Status                                 |
 +=================+============================+========================================+
-| devel           | `2.4`_ (unreleased, trunk) | In development                         |
+| devel           | `2.5`_ (unreleased, trunk) | In development                         |
++-----------------+----------------------------+----------------------------------------+
+| 2.4             | `2.4.1`_ (2017-10-25)      | Supported (bug **and** security fixes) |
 +-----------------+----------------------------+----------------------------------------+
 | 2.3             | `2.3.2`_ (2017-08-08)      | Supported (bug **and** security fixes) |
 +-----------------+----------------------------+----------------------------------------+
-| 2.2             | `2.2.3`_ (2017-05-09)      | Supported (**only** security fixes)    |
+| 2.2             | `2.2.3`_ (2017-05-09)      | Supported (**only** security fixes)    |
 +-----------------+----------------------------+----------------------------------------+
 | 2.1             | `2.1.6`_ (2017-06-01)      | Unsupported (end of life)              |
 +-----------------+----------------------------+----------------------------------------+
@@ -47,10 +49,11 @@ Release status
 +-----------------+----------------------------+----------------------------------------+
 | 1.9             | `1.9.6`_ (2016-04-15)      | Unsupported (end of life)              |
 +-----------------+----------------------------+----------------------------------------+
-| <1.9            | n/a                        | Unsupported (end of life)              |
+| <1.9            | n/a                        | Unsupported (end of life)              |
 +-----------------+----------------------------+----------------------------------------+
 
-.. _2.4: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md
+.. _2.5: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md
+.. _2.4.1: https://github.com/ansible/ansible/blob/stable-2.4/CHANGELOG.md
 .. _2.3.2: https://github.com/ansible/ansible/blob/stable-2.3/CHANGELOG.md
 .. _2.2.3: https://github.com/ansible/ansible/blob/stable-2.2/CHANGELOG.md
 .. _2.1.6: https://github.com/ansible/ansible/blob/stable-2.1/CHANGELOG.md

docs/docsite/rst/roadmap/ROADMAP_2_1.rst

@@ -95,11 +95,11 @@ Expand module diff support (already in progress in devel)
 
 Other
 -----
-Things being kicking down the road that we said we’d do
+Things being kicking down the road that we said we'd do
 
 - NOT remerging core with ansible/ansible this release cycle
 
 Community
 ---------
-- Define the process/ETA for reviewing PR’s from community
+- Define the process/ETA for reviewing PR's from community
 - Publish better docs and how-tos for submitting code/features/fixes

docs/docsite/rst/roadmap/ROADMAP_2_2.rst

@@ -19,7 +19,7 @@ Extras split from Core
 ----------------------
 Lead by Jason M and Jimi-c (Targeting 2.2, could move into 2.3).
 
-Targeted towards the 2.2 release or shortly after, we are planning on splitting Extras out of the “Ansible Core” project.  That means that modules that are shipped with Ansible by default are **only** the modules in ansibl-modules-core.  Ansible extras will become a separate project, managed by the community standard.  Over the next few months we’re going to have a lot of work to do on getting all of the modules in the right places for this to work.
+Targeted towards the 2.2 release or shortly after, we are planning on splitting Extras out of the "Ansible Core" project.  That means that modules that are shipped with Ansible by default are **only** the modules in ansibl-modules-core.  Ansible extras will become a separate project, managed by the community standard.  Over the next few months we're going to have a lot of work to do on getting all of the modules in the right places for this to work.
 
 - Create proposal (Jason or Jimi)
 - Review modules for correct location (extras v core)
@@ -40,7 +40,7 @@ AWS
 ---
 Lead by Ryan Brown
 
-- Pagination for all AWS modules (generic pagination exists, but isn’t used everywhere) (bumped to 2.3)
+- Pagination for all AWS modules (generic pagination exists, but isn't used everywhere) (bumped to 2.3)
 - Refactoring ec2.py to be more digestible (bumped to 2.3)
 - Fix inconsistencies with different authentication methods (STS, environment creds, ~/.aws/credentials) (done)
 - AWS Lambda modules (lambda_execute done, others pending)
@@ -85,7 +85,7 @@ Lead by Matt D
 
 - Feature parity
 
-  - PS module API (mirror Python module API where appropriate). Note: We don’t necessarily like the current python module API (AnsibleModule is a huge class with many unrelated utility functions.  Maybe we should redesign both at the same time?) (bumped to 2.3+ due to "moving target" uncertainty)
+  - PS module API (mirror Python module API where appropriate). Note: We don't necessarily like the current python module API (AnsibleModule is a huge class with many unrelated utility functions.  Maybe we should redesign both at the same time?) (bumped to 2.3+ due to "moving target" uncertainty)
   - Environment keyword support (done)
   - win_shell/win_command (done)
   - Async support (done)
@@ -117,7 +117,7 @@ Lead by Nate C, Peter S
 
 Role revamp
 -----------
-- Implement ‘role revamp’ proposal to give users more control on role/task execution (Brian)
+- Implement 'role revamp' proposal to give users more control on role/task execution (Brian)
 
   - **https://github.com/ansible/proposals/blob/master/roles_revamp.md**
 
@@ -125,13 +125,13 @@ Vault
 -----
 Lead by Jtanner, Adrian
 
-- *Extend ‘transparent vault file usage’ to other action plugins other than 'copy'(https://github.com/ansible/ansible/issues/7298)*
+- *Extend 'transparent vault file usage' to other action plugins other than 'copy'(https://github.com/ansible/ansible/issues/7298)*
   **done:** https://github.com/ansible/ansible/pull/16957
-- Add ‘per variable’ vault support (!vault YAML directive, existing PR already) https://github.com/ansible/ansible/issues/13287 https://github.com/ansible/ansible/issues/14721
+- Add 'per variable' vault support (!vault YAML directive, existing PR already) https://github.com/ansible/ansible/issues/13287 https://github.com/ansible/ansible/issues/14721
 - Add vault/unvault filters https://github.com/ansible/ansible/issues/12087 (deferred to 2.3)
 - Add vault support to lookups (likely deferred to 2.3 or until lookup plugins are revamped)
 - Allow for multiple vault secrets https://github.com/ansible/ansible/issues/13243
-- Config option to turn ‘unvaulting’ failures into warnings https://github.com/ansible/ansible/issues/13244
+- Config option to turn 'unvaulting' failures into warnings https://github.com/ansible/ansible/issues/13244
 
 Python3
 -------
@@ -139,7 +139,7 @@ Lead by Toshio
 
 A note here from Jason M: Getting to complete, tested Python 3 is both
 a critical task and one that has so much work and so many moving parts
-that we don’t expect this to be complete by the 2.2 release.  Toshio will
+that we don't expect this to be complete by the 2.2 release.  Toshio will
 lead this overall effort.
 
 - Motivation:
@@ -173,7 +173,7 @@ lead this overall effort.
     - Update: copy of the six library (v1.4.1 for python2.4 compat) and unicode helpers are here (ansible.module_utils._text.{to_bytes,to_text,to_native})
   - A few basic modules ported to python3
 
-    - Stat module best example module since it’s essential.
+    - Stat module best example module since it's essential.
     - Update:
 
       - A handful of modules like stat have been line-by-line ported.  They should work reliably with few python3-specific bugs.  All but three integration tests pass which means that most essential modules are working to some extent on Python3.
@@ -186,7 +186,7 @@ lead this overall effort.
     - lib/ansible/* and all modules now compile under Python-3.5
 
   - Side work to do:
-    - Figure out best ways to run unit-tests on modules.  Start unit-testing modules.  This is going to become important so we don’t regress python3 or python2.4 support in modules  (Going to largely punt on this for 2.2.  Matt Clay is working on building us a testing foundation for the first half of 2.2 development so we’ll re-evaluate towards the middle of the dev cycle).
+    - Figure out best ways to run unit-tests on modules.  Start unit-testing modules.  This is going to become important so we don't regress python3 or python2.4 support in modules  (Going to largely punt on this for 2.2.  Matt Clay is working on building us a testing foundation for the first half of 2.2 development so we'll re-evaluate towards the middle of the dev cycle).
     - More unit tests of module_utils
     - More integration tests.  Currently integration tests are the best way to test ansible modules so we have to rely on those.
 
@@ -202,7 +202,7 @@ Infrastructure Buildout and Changes
 -----------------------------------
 Lead by Matt Clay
 
-Another note from Jason M: A lot of this work is to ease the burden of CI, CI performance, increase our testing coverage and all of that sort of thing.  It’s not necessarily feature work, but it’s \*\*critical\*\* to growing our product and our ability to get community changes in more securely and quickly.
+Another note from Jason M: A lot of this work is to ease the burden of CI, CI performance, increase our testing coverage and all of that sort of thing.  It's not necessarily feature work, but it's \*\*critical\*\* to growing our product and our ability to get community changes in more securely and quickly.
 
 - **CI Performance**
   Reduce time spent waiting on CI for PRs. Combination of optimizing existing Travis setup and offloading work to other services. Will be impacted by available budget.

docs/docsite/rst/roadmap/ROADMAP_2_3.rst

@@ -46,7 +46,7 @@ Lead by nitzmahone
   - Become support **(done/experimental)**
   - Integrated kerberos ticket management (via ansible_user/ansible_password) **(done)**
   - Switch PS input encoding to BOM-less UTF8 **(done)**
-  - Server 2016 support/testing (now RTM’d) **(partial)**
+  - Server 2016 support/testing (now RTM'd) **(partial)**
   - Modularize Windows module_utils (allow N files) **(partial)**
   - Declarative argspec for PS / .NET **(bumped to 2.4)**
   - Kerberos encryption (via notting, pywinrm/requests_kerberos/pykerberos) **(in progress, available in pywinrm post 2.3 release)**
@@ -98,8 +98,8 @@ Python3
 
   - If users report bugs on python3, these should be fixed and will prioritize our work on porting other modules.
 
-- Still have to solve the python3-only and python2-only modules.  Thinking of doing this via metadata.  Will mean we have to use metadata at the module_common level.  Will also mean we don’t support py2-only or py3-only old style python modules.
-- Note: Most of the currently tested ansible features now run.  But there’s still a lot of code that’s untested.
+- Still have to solve the python3-only and python2-only modules.  Thinking of doing this via metadata.  Will mean we have to use metadata at the module_common level.  Will also mean we don't support py2-only or py3-only old style python modules.
+- Note: Most of the currently tested ansible features now run.  But there's still a lot of code that's untested.
 
 Testing and CI
 --------------
@@ -157,7 +157,7 @@ Plugin Loader
 
 ansible-ssh
 -----------
-- Add a ‘ansible-ssh’ convenience and debugging tool (will slip to 2.4)
+- Add a 'ansible-ssh' convenience and debugging tool (will slip to 2.4)
 - Tool to invoke an interactive ssh to a host with the same args/env/config that ansible would.
 - There are at least three external versions
 

docs/docsite/rst/roadmap/ROADMAP_2_4.rst

@@ -56,7 +56,7 @@ Inventory
 
 Facts
 -----
-- Configurable list of ‘fact modules’ for ``gather_facts`` **(done)**
+- Configurable list of 'fact modules' for ``gather_facts`` **(done)**
 - Fact gathering policy finer grained **(done)**
 - Make ``setup.py``/``facts`` more pluggable **(done)**
 - Improve testing of ``setup.py``/``facts.py`` **(done)**
@@ -104,8 +104,8 @@ Globalize Callbacks
 -------------------
 **(pushed out to future release)**
 - Make send_callback available to other code that cannot use it.
-- Would allow for ‘full formatting’ of output (see JSON callback)
-- Fixes static ‘include’ display problem
+- Would allow for 'full formatting' of output (see JSON callback)
+- Fixes static 'include' display problem
 
 Plugins
 -------
@@ -128,13 +128,13 @@ Runtime Check on Modules for Blacklisting
 
 Disambiguate Includes
 ---------------------
-- Create import_x for ‘static includes’ (import_task, import_play, import_role)
+- Create import_x for 'static includes' (import_task, import_playbook, import_role)
 
-  - Any directives are applied to the ‘imported’ tasks
+  - Any directives are applied to the 'imported' tasks
 
-- Create include_x for ‘dynamic includes’ (include_task, include_role)
+- Create include_x for 'dynamic includes' (include_task, include_role)
 
-  - Any directives apply to the ‘include’  itself
+  - Any directives apply to the 'include'  itself
 
 Windows
 -------

docs/docsite/rst/vault.rst

@@ -5,7 +5,7 @@ Ansible Vault
 
 New in Ansible 1.5, "Vault" is a feature of ansible that allows keeping sensitive data such as passwords or keys in encrypted files, rather than as plaintext in your playbooks or roles. These vault files can then be distributed or placed in source control.
 
-To enable this feature, a command line tool, :ref:`ansible-vault` is used to edit files, and a command line flag `--ask-vault-pass` or `--vault-password-file` is used. Alternately, you may specify the location of a password file or command Ansible to always prompt for the password in your ansible.cfg file. These options require no command line flag usage.
+To enable this feature, a command line tool - :ref:`ansible-vault` - is used to edit files, and a command line flag (:option:`--ask-vault-pass <ansible-playbook --ask-vault-pass>` or :option:`--vault-password-file <ansible-playbook --vault-password-file>`) is used. Alternately, you may specify the location of a password file or command Ansible to always prompt for the password in your ansible.cfg file. These options require no command line flag usage.
 
 For best practices advice, refer to :ref:`best_practices_for_variables_and_vaults`.
 
@@ -19,7 +19,7 @@ The vault feature can encrypt any structured data file used by Ansible.  This ca
 Ansible tasks, handlers, and so on are also data so these can be encrypted with vault as well. To hide the names of variables that you're using, you can encrypt the task files in their entirety. However, that might be a little too much and could annoy your coworkers :)
 
 The vault feature can also encrypt arbitrary files, even binary files.  If a vault-encrypted file is
-given as the :ref:`src <src>` argument to the :ref:`copy <copy>`, :ref:`template <template>`,
+given as the 'src' argument to the :ref:`copy <copy>`, :ref:`template <template>`,
 :ref:`unarchive <unarchive>`, :ref:`script <script>` or :ref:`assemble <assemble>` modules, the file will be placed at the destination on the target host decrypted (assuming a valid vault password is supplied when running the play).
 
 As of version 2.3, Ansible also supports encrypting single values inside a YAML file, using the `!vault` tag to let YAML and Ansible know it uses special processing. This feature is covered in more details below.
@@ -48,7 +48,7 @@ The default cipher is AES (which is shared-secret based).
 Editing Encrypted Files
 ```````````````````````
 
-To edit an encrypted file in place, use the :ref:`ansible-vault edit` command.
+To edit an encrypted file in place, use the :ref:`ansible-vault edit <ansible_vault_edit>` command.
 This command will decrypt the file to a temporary file and allow you to edit
 the file, saving it back when done and removing the temporary file:
 
@@ -78,7 +78,7 @@ Encrypting Unencrypted Files
 ````````````````````````````
 
 If you have existing files that you wish to encrypt, use
-the :ref:`ansible-vault encrypt` command.  This command can operate on multiple files at once:
+the :ref:`ansible-vault encrypt <ansible_vault_encrypt>` command.  This command can operate on multiple files at once:
 
 .. code-block:: bash
 
@@ -91,8 +91,8 @@ Decrypting Encrypted Files
 ``````````````````````````
 
 If you have existing files that you no longer want to keep encrypted, you can permanently decrypt
-them by running the :ref:`ansible-vault decrypt` command.  This command will save them unencrypted
-to the disk, so be sure you do not want :ref:`ansible-vault edit` instead:
+them by running the :ref:`ansible-vault decrypt <ansible_vault_decrypt>` command.  This command will save them unencrypted
+to the disk, so be sure you do not want :ref:`ansible-vault edit <ansible_vault_edit>` instead:
 
 .. code-block:: bash
 
@@ -106,7 +106,7 @@ Viewing Encrypted Files
 
 *Available since Ansible 1.8*
 
-If you want to view the contents of an encrypted file without editing it, you can use the :ref:`ansible-vault view` command:
+If you want to view the contents of an encrypted file without editing it, you can use the :ref:`ansible-vault view <ansible_vault_view>` command:
 
 .. code-block:: bash
 
@@ -118,10 +118,11 @@ If you want to view the contents of an encrypted file without editing it, you ca
 Use encrypt_string to create encrypted variables to embed in yaml
 `````````````````````````````````````````````````````````````````
 
-The :ref:`ansible-vault encrypt_string` command will encrypt and format a provided string into a format
+The :ref:`ansible-vault encrypt_string <ansible_vault_encrypt_string>` command will encrypt and format a provided string into a format
 that can be included in :ref:`ansible-playbook` YAML files.
 
 To encrypt a string provided as a cli arg:
+
 .. code-block:: bash
 
     ansible-vault encrypt_string --vault-id a_password_file 'foobar' --name 'the_secret'
@@ -224,7 +225,7 @@ Providing Vault Passwords
 `````````````````````````
 
 Since Ansible 2.4, the recommended way to provide a vault password from the cli is
-to use the :option:`--vault-id` cli option.
+to use the :option:`--vault-id <ansible-playbook --vault-id>` cli option.
 
 For example, to use a password store in the text file :file:`/path/to/my/vault-password-file`:
 
@@ -244,7 +245,7 @@ To get the password from a vault password executable script :file:`my-vault-pass
 
     ansible-playbook --vault-id my-vault-password.py
 
-The value for :option:`--vault-id` can specify the type of vault id (prompt, a file path, etc)
+The value for :option:`--vault-id <ansible-playbook --vault-id>` can specify the type of vault id (prompt, a file path, etc)
 and a label for the vault id ('dev', 'prod', 'cloud', etc)
 
 For example, to use a password file :file:`dev-password` for the vault-id 'dev':
@@ -261,33 +262,26 @@ To prompt for the 'dev' vault id:
 
 *Prior to Ansible 2.4*
 
-To be prompted for a vault password, use the :option:`--ask-vault-pass` cli option:
+To be prompted for a vault password, use the :option:`--ask-vault-pass <ansible-playbook --vault-id>` cli option:
 
 .. code-block:: bash
 
     ansible-playbook --ask-vault-pass site.yml
 
-To specify a vault password in a text file 'dev-password', use the :option:`--vault-password-file` option:
+To specify a vault password in a text file 'dev-password', use the :option:`--vault-password-file <ansible-playbook --vault-password-file>` option:
 
 .. code-block:: bash
 
     ansible-playbook --vault-password-file dev-password site.yml
 
 There is a config option (:ref:`DEFAULT_VAULT_PASSWORD_FILE`) to specify a vault password file to use
-without requiring the :option:`--vault-password-file` cli option.
+without requiring the :option:`--vault-password-file <ansible-playbook --vault-password-file>` cli option.
 
-via config
-  :ref:`ANSIBLE_VAULT_PASSWORD_FILE`
-
-  :ref:`ANSIBLE_VAULT_IDENTITY_LIST`
-
-via env
-   TODO
 
 Multiple vault passwords
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
-Since Ansible 2.4 and later support using multiple vault passwords, :option:`--vault-id` can
+Since Ansible 2.4 and later support using multiple vault passwords, :option:`--vault-id <ansible-playbook --vault-id>` can
 be provided multiple times.
 
 If multiple vault passwords are provided, by default Ansible will attempt to decrypt vault content
@@ -302,7 +296,7 @@ For example, to use a 'dev' password read from a file and to be prompted for the
 In the above case, the 'dev' password will be tried first, then the 'prod' password for cases
 where Ansible doesn't know which vault id is used to encrypt something.
 
-If the vault content was encrypted using a :option:`--vault-id` option, then the label of the
+If the vault content was encrypted using a :option:`--vault-id <ansible-vault --vault-id>` option, then the label of the
 vault id is stored with the vault content. When Ansible knows the right vault-id, it will try
 the matching vault id's secret first before trying the rest of the vault-ids.
 
@@ -317,17 +311,17 @@ use. For example, instead of requiring the cli option on every use, the (:ref:`D
 
     ansible-playbook --vault-id dev@dev-password --vault-id prod@prompt site.yml
 
-The :option:`--vault-id` can be used in lieu of the :option:`--vault-password-file` or :option:`--ask-vault-pass` options,
+The :option:`--vault-id <ansible-playbook --vault-id>` can be used in lieu of the :option:`--vault-password-file <ansible-playbook --vault-password-file>` or :option:`--ask-vault-pass <ansible-playbook --ask-vault-pass>` options,
 or it can be used in combination with them.
 
-When using :ref:`ansible-vault` command that encrypt content (:ref:`ansible-vault encrypt`, :ref:`ansible-vault encrypt_string`, etc)
+When using :ref:`ansible-vault` commands that encrypt content (:ref:`ansible-vault encrypt <ansible_vault_encrypt>`, :ref:`ansible-vault encrypt_string <ansible_vault_encrypt_string>`, etc)
 only one vault-id can be used.
 
 
 
 .. note::
     Prior to Ansible 2.4, only one vault password could be used in each Ansible run. The
-    :option:`--vault-id` option is not support prior to Ansible 2.4.
+    :option:`--vault-id <ansible-playbook --vault-id>` option is not support prior to Ansible 2.4.
 
 
 .. _speeding_up_vault:

docs/docsite/rst_common/ansible_ssh_changes_note.rst

@@ -1,3 +1,3 @@
 .. note::
 
-    Ansible 2.0 has deprecated the “ssh” from ``ansible_ssh_user``, ``ansible_ssh_host``, and ``ansible_ssh_port`` to become ``ansible_user``, ``ansible_host``, and ``ansible_port``. If you are using a version of Ansible prior to 2.0,  you should continue using the older style variables (``ansible_ssh_*``). These shorter variables are ignored, without warning, in older versions of Ansible. 
+    Ansible 2.0 has deprecated the "ssh" from ``ansible_ssh_user``, ``ansible_ssh_host``, and ``ansible_ssh_port`` to become ``ansible_user``, ``ansible_host``, and ``ansible_port``. If you are using a version of Ansible prior to 2.0,  you should continue using the older style variables (``ansible_ssh_*``). These shorter variables are ignored, without warning, in older versions of Ansible.

docs/templates/cli_rst.j2

@@ -0,0 +1,139 @@
+{% set name = cli_name -%}
+{% set name_slug = cli_name -%}
+
+.. _{{name}}:
+
+{% set name_len = name|length + 0-%}
+{{ '=' * name_len }}
+{{name}}
+{{ '=' * name_len }}
+
+
+:strong:`{{short_desc|default('')}}`
+
+
+.. contents::
+   :local:
+   :depth: 2
+
+
+.. program:: {{cli_name}}
+
+Synopsis
+========
+
+.. code-block:: bash
+
+   {{ usage|replace('%prog', cli_name) }}
+
+
+Description
+===========
+
+
+{{ long_desc|default('', True)  }}
+
+{% if options %}
+Common Options
+==============
+
+
+{% for option in options|sort(attribute='options') %}
+
+.. option:: {% for switch in option['options'] %}{{switch}}{% if option['arg'] %} <{{option['arg']}}>{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}
+
+   {{ option['desc'] }}
+{% endfor %}
+{% endif %}
+
+{% if arguments %}
+ARGUMENTS
+=========
+
+.. program:: {{cli_name}}
+
+{% for arg in arguments %}
+.. option:: {{ arg }}
+
+   {{ (arguments[arg]|default(' '))}}
+
+{% endfor %}
+{% endif %}
+
+{% if actions %}
+Actions
+=======
+
+{% for action in actions %}
+
+.. program:: {{cli_name}} {{action}}
+.. _{{cli_name|replace('-','_')}}_{{action}}:
+
+{{ action}}
+{{ '-' * action|length}}
+
+{{ (actions[action]['desc']|default(' '))}}
+
+{% if actions[action]['options'] %}
+
+
+{% for option in actions[action]['options']|sort %}
+.. option:: {% for switch in option['options'] if switch in actions[action]['option_names'] %}{{switch}} {% if option['arg'] %} <{{option['arg']}}>{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}
+
+   {{ (option['desc']) }}
+{% endfor %}
+{% endif %}
+
+{% endfor %}
+.. program:: {{cli_name}}
+{% endif %}
+
+Environment
+===========
+
+The following environment variables may be specified.
+
+{% if inventory %}
+:envvar:`ANSIBLE_INVENTORY`  -- Override the default ansible inventory file
+
+{% endif %}
+{% if library %}
+:envvar:`ANSIBLE_LIBRARY` -- Override the default ansible module library path
+
+{% endif %}
+:envvar:`ANSIBLE_CONFIG` -- Override the default ansible config file
+
+Many more are available for most options in ansible.cfg
+
+
+Files
+=====
+
+{% if inventory %}
+:file:`/etc/ansible/hosts` -- Default inventory file
+
+{% endif %}
+:file:`/etc/ansible/ansible.cfg` -- Config file, used if present
+
+:file:`~/.ansible.cfg` -- User config file, overrides the default config if present
+
+Author
+======
+
+Ansible was originally written by Michael DeHaan.
+
+See the `AUTHORS` file for a complete list of contributors.
+
+
+Copyright
+=========
+
+Copyright © 2017 Red Hat, Inc | Ansible.
+
+Ansible is released under the terms of the GPLv3 License.
+
+See also
+========
+
+{% for other in cli_bin_name_list|sort %}:manpage:`{{other}}(1)`,  {% endfor %}
+

docs/templates/man.j2

@@ -20,8 +20,17 @@ SYNOPSIS
 
 DESCRIPTION
 -----------
-*{{name}}* {{ long_desc|default('', True)|wordwrap }}
+{{ long_desc|default('', True)|wordwrap }}
 
+{% if options %}
+COMMON OPTIONS
+--------------
+{% for option in options|sort(attribute='options') %}
+{% for switch in option['options'] %}*{{switch}}*{% if option['arg'] %} '{{option['arg']}}'{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}::
+
+{{ option['desc'] }}
+{% endfor %}
+{% endif %}
 
 {% if arguments %}
 ARGUMENTS
@@ -38,22 +47,19 @@ ARGUMENTS
 {% if actions %}
 ACTIONS
 -------
-
 {% for action in actions %}
-{{ action }}
+    *{{ action }}*::: {{ (actions[action]['desc']|default(' '))|wordwrap}}
 
-{{ (actions[action]|default(' '))|wordwrap}}
+{% if actions[action]['options'] %}
+{% for option in actions[action]['options']|sort %}
+{% for switch in option['options'] if switch in actions[action]['option_names'] %}*{{switch}}*{% if option['arg'] %} '{{option['arg']}}'{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}::
 
+        {{ (option['desc']) }}
 {% endfor %}
 {% endif %}
-OPTIONS
--------
-
-{% for option in options|sort(attribute='options') %}
-{% for switch in option['options'] %}*{{switch}}* {% if option['arg'] %}'{{option['arg']}}'{% endif %}{% if not loop.last %}, {% endif %}{% endfor %}::
-
-{{ option['desc'] }}
 {% endfor %}
+{% endif %}
+
 
 {% if inventory %}
 INVENTORY
@@ -111,7 +117,7 @@ Ansible is released under the terms of the GPLv3 License.
 SEE ALSO
 --------
 
- {% for other in cli_list|sort %}{% if other != cli %}*ansible{% if other != 'adhoc' %}-{{other}}{% endif %}*(1){% if not loop.last %}, {% endif %}{% endif %}{% endfor %}
+{% for other in cli_list|sort %}{% if other != cli %}*ansible{% if other != 'adhoc' %}-{{other}}{% endif %}*(1){% if not loop.last %}, {% endif %}{% endif %}{% endfor %}
 
 Extensive documentation is available in the documentation site:
 <http://docs.ansible.com>.

docs/templates/playbooks_keywords.rst.j2

@@ -15,8 +15,12 @@ Be aware that this reflects the 'current development branch' and that the keywor
 
 {{ name }}
 {{ '-' * name|length }}
+.. glossary::
+
 {% for attribute in oblist[name]|sort %}
-  - **{{ attribute }}:** {{ oblist[name][attribute] }}
+    {{ attribute }}
+        {{ oblist[name][attribute] }}
+
 {% endfor %}
 
 {% endfor %}

examples/ansible.cfg

@@ -297,10 +297,6 @@
 # only update this setting if you know how this works, otherwise it can break module execution
 #network_group_modules=['eos', 'nxos', 'ios', 'iosxr', 'junos', 'vyos']
 
-# This keeps facts from polluting the main namespace as variables.
-# Setting to True keeps them under the ansible_facts namespace, the default is False
-#restrict_facts_namespace: True
-
 # When enabled, this option allows lookups (via variables like {{lookup('foo')}} or when used as
 # a loop with `with_foo`) to return data that is not marked "unsafe". This means the data may contain
 # jinja2 templating language which will be run through the templating engine.
@@ -315,7 +311,7 @@
 #enable_plugins = host_list, virtualbox, yaml, constructed
 
 # ignore these extensions when parsing a directory as inventory source
-#ignore_extensions = '.pyc', '.pyo', '.swp', '.bak', '~', '.rpm', '.md', '.txt', '~', '.orig', '.ini', '.cfg', '.retry'
+#ignore_extensions = .pyc, .pyo, .swp, .bak, ~, .rpm, .md, .txt, ~, .orig, .ini, .cfg, .retry
 
 # ignore files matching these patterns when parsing a directory as inventory source
 #ignore_patterns=

examples/scripts/uptime.py

@@ -3,7 +3,7 @@
 from collections import namedtuple
 
 from ansible.executor.task_queue_manager import TaskQueueManager
-from ansible.inventory import Inventory
+from ansible.inventory.manager import InventoryManager
 from ansible.parsing.dataloader import DataLoader
 from ansible.playbook.play import Play
 from ansible.plugins.callback import CallbackBase
@@ -33,21 +33,21 @@ def main():
     host_list = ['localhost', 'www.example.com', 'www.google.com']
     Options = namedtuple('Options', ['connection', 'module_path', 'forks', 'remote_user',
                                      'private_key_file', 'ssh_common_args', 'ssh_extra_args', 'sftp_extra_args',
-                                     'scp_extra_args', 'become', 'become_method', 'become_user', 'verbosity', 'check'])
+                                     'scp_extra_args', 'become', 'become_method', 'become_user', 'verbosity', 'check',
+                                     'diff'])
 
     # initialize needed objects
-    variable_manager = VariableManager()
     loader = DataLoader()
     options = Options(connection='smart', module_path='/usr/share/ansible', forks=100,
                       remote_user=None, private_key_file=None, ssh_common_args=None, ssh_extra_args=None,
                       sftp_extra_args=None, scp_extra_args=None, become=None, become_method=None,
-                      become_user=None, verbosity=None, check=False)
+                      become_user=None, verbosity=None, check=False, diff=False)
 
     passwords = dict()
 
     # create inventory and pass to var manager
-    inventory = Inventory(loader=loader, variable_manager=variable_manager, host_list=host_list)
-    variable_manager.set_inventory(inventory)
+    inventory = InventoryManager(loader=loader, sources=','.join(host_list))
+    variable_manager = VariableManager(loader=loader, inventory=inventory)
 
     # create play with tasks
     play_source = dict(

hacking/return_skeleton_generator.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+
+# (c) 2017, Will Thames <will@thames.id.au>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+# return_skeleton_generator.py takes JSON output from a module and
+# and creates a starting point for the RETURNS section of a module.
+# This can be provided as stdin or a file argument
+#
+# The easiest way to obtain the JSON output is to use hacking/test-module
+#
+# You will likely want to adjust this to remove sensitive data or
+# ensure the `returns` value is correct, and to write a useful description
+
+from __future__ import print_function
+from collections import OrderedDict
+import json
+import sys
+import yaml
+
+
+# Allow OrderedDicts to be used when dumping YAML
+# https://stackoverflow.com/a/16782282/3538079
+def represent_ordereddict(dumper, data):
+    value = []
+
+    for item_key, item_value in data.items():
+        node_key = dumper.represent_data(item_key)
+        node_value = dumper.represent_data(item_value)
+
+        value.append((node_key, node_value))
+
+    return yaml.nodes.MappingNode(u'tag:yaml.org,2002:map', value)
+
+
+def get_return_data(key, value):
+    # The OrderedDict here is so that, for complex objects, the
+    # summary data is at the top before the contains information
+    returns_info = {key: OrderedDict()}
+    returns_info[key]['description'] = "FIXME *** add description for %s" % key
+    returns_info[key]['returned'] = "always"
+    if isinstance(value, dict):
+        returns_info[key]['type'] = 'complex'
+        returns_info[key]['contains'] = get_all_items(value)
+    elif isinstance(value, list) and value and isinstance(value[0], dict):
+        returns_info[key]['type'] = 'complex'
+        returns_info[key]['contains'] = get_all_items(value[0])
+    else:
+        returns_info[key]['type'] = type(value).__name__
+        returns_info[key]['sample'] = value
+        # override python unicode type to set to string for docs
+        if returns_info[key]['type'] == 'unicode':
+            returns_info[key]['type'] = 'string'
+    return returns_info
+
+
+def get_all_items(data):
+    items = sorted([get_return_data(key, value) for key, value in data.items()])
+    result = OrderedDict()
+    for item in items:
+        key, value = item.items()[0]
+        result[key] = value
+    return result
+
+
+def main(args):
+    yaml.representer.SafeRepresenter.add_representer(OrderedDict, represent_ordereddict)
+
+    if args:
+        src = open(args[0])
+    else:
+        src = sys.stdin
+
+    data = json.load(src, strict=False)
+    docs = get_all_items(data)
+    if 'invocation' in docs:
+        del(docs['invocation'])
+    print(yaml.safe_dump(docs, default_flow_style=False))
+
+
+if __name__ == '__main__':
+    main(sys.argv[1:])

lib/ansible/cli/__init__.py

@@ -168,6 +168,21 @@ class CLI(with_metaclass(ABCMeta, object)):
         else:
             display.v(u"No config file found; using defaults")
 
+        # warn about deprecated config options
+        for deprecated in C.config.DEPRECATED:
+            name = deprecated[0]
+            why = deprecated[1]['why']
+            if 'alternative' in deprecated[1]:
+                alt = ', use %s instead' % deprecated[1]['alternative']
+            else:
+                alt = ''
+            ver = deprecated[1]['version']
+            display.deprecated("%s option, %s %s" % (name, why, alt), version=ver)
+
+        # warn about typing issues with configuration entries
+        for unable in C.config.UNABLE:
+            display.warning("Unable to set correct type for configuration entry: %s" % unable)
+
     @staticmethod
     def split_vault_id(vault_id):
         # return (before_@, after_@)
@@ -181,7 +196,8 @@ class CLI(with_metaclass(ABCMeta, object)):
 
     @staticmethod
     def build_vault_ids(vault_ids, vault_password_files=None,
-                        ask_vault_pass=None, create_new_password=None):
+                        ask_vault_pass=None, create_new_password=None,
+                        auto_prompt=True):
         vault_password_files = vault_password_files or []
         vault_ids = vault_ids or []
 
@@ -196,7 +212,7 @@ class CLI(with_metaclass(ABCMeta, object)):
 
         # if an action needs an encrypt password (create_new_password=True) and we dont
         # have other secrets setup, then automatically add a password prompt as well.
-        if ask_vault_pass or (create_new_password and not vault_ids):
+        if ask_vault_pass or (auto_prompt and not vault_ids):
             id_slug = u'%s@%s' % (C.DEFAULT_VAULT_IDENTITY, u'prompt_ask_vault_pass')
             vault_ids.append(id_slug)
 
@@ -205,7 +221,8 @@ class CLI(with_metaclass(ABCMeta, object)):
     # TODO: remove the now unused args
     @staticmethod
     def setup_vault_secrets(loader, vault_ids, vault_password_files=None,
-                            ask_vault_pass=None, create_new_password=False):
+                            ask_vault_pass=None, create_new_password=False,
+                            auto_prompt=True):
         # list of tuples
         vault_secrets = []
 
@@ -235,7 +252,8 @@ class CLI(with_metaclass(ABCMeta, object)):
         vault_ids = CLI.build_vault_ids(vault_ids,
                                         vault_password_files,
                                         ask_vault_pass,
-                                        create_new_password)
+                                        create_new_password,
+                                        auto_prompt=auto_prompt)
 
         for vault_id_slug in vault_ids:
             vault_id_name, vault_id_value = CLI.split_vault_id(vault_id_slug)
@@ -374,17 +392,24 @@ class CLI(with_metaclass(ABCMeta, object)):
     @staticmethod
     def unfrack_paths(option, opt, value, parser):
         paths = getattr(parser.values, option.dest)
+        if paths is None:
+            paths = []
+
         if isinstance(value, string_types):
-            paths[:0] = [unfrackpath(x) for x in value.split(os.pathsep)]
+            paths[:0] = [unfrackpath(x) for x in value.split(os.pathsep) if x]
         elif isinstance(value, list):
-            paths[:0] = [unfrackpath(x) for x in value]
+            paths[:0] = [unfrackpath(x) for x in value if x]
         else:
             pass  # FIXME: should we raise options error?
+
         setattr(parser.values, option.dest, paths)
 
     @staticmethod
     def unfrack_path(option, opt, value, parser):
-        setattr(parser.values, option.dest, unfrackpath(value))
+        if value != '-':
+            setattr(parser.values, option.dest, unfrackpath(value))
+        else:
+            setattr(parser.values, option.dest, value)
 
     @staticmethod
     def base_parser(usage="", output_opts=False, runas_opts=False, meta_opts=False, runtask_opts=False, vault_opts=False, module_opts=False,
@@ -408,8 +433,8 @@ class CLI(with_metaclass(ABCMeta, object)):
 
         if module_opts:
             parser.add_option('-M', '--module-path', dest='module_path', default=None,
-                              help="prepend path(s) to module library (default=%s)" % C.DEFAULT_MODULE_PATH,
-                              action="callback", callback=CLI.unfrack_path, type='str')
+                              help="prepend colon-separated path(s) to module library (default=%s)" % C.DEFAULT_MODULE_PATH,
+                              action="callback", callback=CLI.unfrack_paths, type='str')
         if runtask_opts:
             parser.add_option('-e', '--extra-vars', dest="extra_vars", action="append",
                               help="set additional variables as key=value or YAML/JSON, if filename prepend with @", default=[])
@@ -425,9 +450,6 @@ class CLI(with_metaclass(ABCMeta, object)):
                               help="vault password file", action="callback", callback=CLI.unfrack_paths, type='string')
             parser.add_option('--new-vault-password-file', default=[], dest='new_vault_password_files',
                               help="new vault password file for rekey", action="callback", callback=CLI.unfrack_paths, type='string')
-            parser.add_option('--output', default=None, dest='output_file',
-                              help='output file name for encrypt or decrypt; use - for stdout',
-                              action="callback", callback=CLI.unfrack_path, type='string'),
             parser.add_option('--vault-id', default=[], dest='vault_ids', action='append', type='string',
                               help='the vault identity to use')
             parser.add_option('--new-vault-id', default=None, dest='new_vault_id', type='string',
@@ -593,12 +615,10 @@ class CLI(with_metaclass(ABCMeta, object)):
                     self.options.inventory = [self.options.inventory]
 
                 # Ensure full paths when needed
-                self.options.inventory = [unfrackpath(opt) if ',' not in opt else opt for opt in self.options.inventory]
+                self.options.inventory = [unfrackpath(opt, follow=False) if ',' not in opt else opt for opt in self.options.inventory]
 
             else:
-                # set default if it exists
-                if os.path.exists(C.DEFAULT_HOST_LIST):
-                    self.options.inventory = [C.DEFAULT_HOST_LIST]
+                self.options.inventory = C.DEFAULT_HOST_LIST
 
     @staticmethod
     def version(prog):
@@ -763,7 +783,8 @@ class CLI(with_metaclass(ABCMeta, object)):
         vault_secrets = CLI.setup_vault_secrets(loader,
                                                 vault_ids=vault_ids,
                                                 vault_password_files=options.vault_password_files,
-                                                ask_vault_pass=options.ask_vault_pass)
+                                                ask_vault_pass=options.ask_vault_pass,
+                                                auto_prompt=False)
         loader.set_vault_secrets(vault_secrets)
 
         # create the inventory, and filter it based on the subset specified (if any)

lib/ansible/cli/adhoc.py

@@ -138,7 +138,7 @@ class AdHocCLI(CLI):
             raise AnsibleOptionsError(err)
 
         # Avoid modules that don't work with ad-hoc
-        if self.options.module_name in ('include', 'include_role'):
+        if self.options.module_name.startswith(('include', 'import_')):
             raise AnsibleOptionsError("'%s' is not a valid action for ad-hoc commands" % self.options.module_name)
 
         play_ds = self._play_ds(pattern, self.options.seconds, self.options.poll_interval)

lib/ansible/cli/config.py

@@ -44,7 +44,7 @@ except ImportError:
 class ConfigCLI(CLI):
     """ Config command line class """
 
-    VALID_ACTIONS = ("view", "edit", "update", "dump", "list")
+    VALID_ACTIONS = ("view", "dump", "list") # TODO: edit, update, search
 
     def __init__(self, args, callback=None):
 
@@ -56,9 +56,9 @@ class ConfigCLI(CLI):
 
         self.parser = CLI.base_parser(
             usage = "usage: %%prog [%s] [--help] [options] [ansible.cfg]" % "|".join(self.VALID_ACTIONS),
-            epilog = "\nSee '%s <command> --help' for more information on a specific command.\n\n" % os.path.basename(sys.argv[0])
+            epilog = "\nSee '%s <command> --help' for more information on a specific command.\n\n" % os.path.basename(sys.argv[0]),
+            desc="View, edit, and manage ansible configuration.",
         )
-
         self.parser.add_option('-c', '--config', dest='config_file', help="path to configuration file, defaults to first file found in precedence.")
 
         self.set_action()
@@ -69,14 +69,11 @@ class ConfigCLI(CLI):
         if self.action == "dump":
             self.parser.add_option('--only-changed', dest='only_changed', action='store_true',
                                    help="Only show configurations that have changed from the default")
-            self.parser.set_usage("usage: %prog dump [options] [-c ansible.cfg]")
-        elif self.action == "view":
-            self.parser.set_usage("usage: %prog view [options] [-c ansible.cfg] ")
-        elif self.action == "edit":
-            self.parser.set_usage("usage: %prog edit [options] [-c ansible.cfg]")
         elif self.action == "update":
             self.parser.add_option('-s', '--setting', dest='setting', help="config setting, the section defaults to 'defaults'")
             self.parser.set_usage("usage: %prog update [options] [-c ansible.cfg] -s '[section.]setting=value'")
+        elif self.action == "search":
+            self.parser.set_usage("usage: %prog update [options] [-c ansible.cfg] <search term>")
 
         self.options, self.args = self.parser.parse_args()
         display.verbosity = self.options.verbosity

lib/ansible/cli/console.py

@@ -126,9 +126,10 @@ class ConsoleCLI(CLI, cmd.Cmd):
 
     def list_modules(self):
         modules = set()
-        if self.options.module_path is not None:
-            for i in self.options.module_path.split(os.pathsep):
-                module_loader.add_directory(i)
+        if self.options.module_path:
+            for path in self.options.module_path:
+                if path:
+                    module_loader.add_directory(path)
 
         module_paths = module_loader._get_paths()
         for path in module_paths:

lib/ansible/cli/doc.py

@@ -28,9 +28,11 @@ import yaml
 from ansible import constants as C
 from ansible.cli import CLI
 from ansible.errors import AnsibleError, AnsibleOptionsError
+from ansible.module_utils._text import to_native
 from ansible.module_utils.six import string_types
 from ansible.parsing.yaml.dumper import AnsibleDumper
-from ansible.plugins.loader import module_loader, action_loader, lookup_loader, callback_loader, cache_loader, connection_loader, strategy_loader, PluginLoader
+from ansible.plugins.loader import module_loader, action_loader, lookup_loader, callback_loader, cache_loader, \
+    vars_loader, connection_loader, strategy_loader, PluginLoader
 from ansible.utils import plugin_docs
 try:
     from __main__ import display
@@ -53,7 +55,7 @@ class DocCLI(CLI):
     def parse(self):
 
         self.parser = CLI.base_parser(
-            usage='usage: %prog [options] [plugin]',
+            usage='usage: %prog [-l|-s] [options] [-t <plugin type] [plugin]',
             module_opts=True,
             desc="plugin documentation tool",
             epilog="See man pages for Ansible CLI options or website for tutorials https://docs.ansible.com"
@@ -64,13 +66,16 @@ class DocCLI(CLI):
         self.parser.add_option("-s", "--snippet", action="store_true", default=False, dest='show_snippet',
                                help='Show playbook snippet for specified plugin(s)')
         self.parser.add_option("-a", "--all", action="store_true", default=False, dest='all_plugins',
-                               help='Show documentation for all plugins')
+                               help='**For internal testing only** Show documentation for all plugins.')
         self.parser.add_option("-t", "--type", action="store", default='module', dest='type', type='choice',
                                help='Choose which plugin type (defaults to "module")',
-                               choices=['cache', 'callback', 'connection', 'inventory', 'lookup', 'module', 'strategy'])
+                               choices=['cache', 'callback', 'connection', 'inventory', 'lookup', 'module', 'strategy', 'vars'])
 
         super(DocCLI, self).parse()
 
+        if [self.options.all_plugins, self.options.list_dir, self.options.show_snippet].count(True) > 1:
+            raise AnsibleOptionsError("Only one of -l, -s or -a can be used at the same time.")
+
         display.verbosity = self.options.verbosity
 
     def run(self):
@@ -90,15 +95,18 @@ class DocCLI(CLI):
             loader = lookup_loader
         elif plugin_type == 'strategy':
             loader = strategy_loader
+        elif plugin_type == 'vars':
+            loader = vars_loader
         elif plugin_type == 'inventory':
             loader = PluginLoader('InventoryModule', 'ansible.plugins.inventory', 'inventory_plugins', 'inventory_plugins')
         else:
             loader = module_loader
 
         # add to plugin path from command line
-        if self.options.module_path is not None:
-            for i in self.options.module_path.split(os.pathsep):
-                loader.add_directory(i)
+        if self.options.module_path:
+            for path in self.options.module_path:
+                if path:
+                    loader.add_directory(path)
 
         # save only top level paths for errors
         search_paths = DocCLI.print_paths(loader)
@@ -118,6 +126,7 @@ class DocCLI(CLI):
             paths = loader._get_paths()
             for path in paths:
                 self.find_plugins(path, plugin_type)
+            self.args = sorted(set(self.plugin_list))
 
         if len(self.args) == 0:
             raise AnsibleOptionsError("Incorrect options passed")
@@ -128,7 +137,7 @@ class DocCLI(CLI):
 
             try:
                 # if the plugin lives in a non-python file (eg, win_X.ps1), require the corresponding python file for docs
-                filename = loader.find_plugin(plugin, mod_type='.py', ignore_deprecated=True)
+                filename = loader.find_plugin(plugin, mod_type='.py', ignore_deprecated=True, check_aliases=True)
                 if filename is None:
                     display.warning("%s %s not found in:\n%s\n" % (plugin_type, plugin, search_paths))
                     continue
@@ -220,35 +229,38 @@ class DocCLI(CLI):
         deprecated = []
         for plugin in sorted(self.plugin_list):
 
-            # if the module lives in a non-python file (eg, win_X.ps1), require the corresponding python file for docs
-            filename = loader.find_plugin(plugin, mod_type='.py', ignore_deprecated=True)
-
-            if filename is None:
-                continue
-            if filename.endswith(".ps1"):
-                continue
-            if os.path.isdir(filename):
-                continue
-
-            doc = None
             try:
-                doc, plainexamples, returndocs, metadata = plugin_docs.get_docstring(filename)
-            except:
-                display.warning("%s has a documentation formatting error" % plugin)
+                # if the module lives in a non-python file (eg, win_X.ps1), require the corresponding python file for docs
+                filename = loader.find_plugin(plugin, mod_type='.py', ignore_deprecated=True, check_aliases=True)
 
-            if not doc:
-                desc = 'UNDOCUMENTED'
-                display.warning("%s parsing did not produce documentation." % plugin)
-            else:
-                desc = self.tty_ify(doc.get('short_description', '?')).strip()
+                if filename is None:
+                    continue
+                if filename.endswith(".ps1"):
+                    continue
+                if os.path.isdir(filename):
+                    continue
 
-            if len(desc) > linelimit:
-                desc = desc[:linelimit] + '...'
+                doc = None
+                try:
+                    doc, plainexamples, returndocs, metadata = plugin_docs.get_docstring(filename)
+                except:
+                    display.warning("%s has a documentation formatting error" % plugin)
 
-            if plugin.startswith('_'):  # Handle deprecated
-                deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
-            else:
-                text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
+                if not doc or not isinstance(doc, dict):
+                    desc = 'UNDOCUMENTED'
+                    display.warning("%s parsing did not produce documentation." % plugin)
+                else:
+                    desc = self.tty_ify(doc.get('short_description', 'INVALID SHORT DESCRIPTION').strip())
+
+                if len(desc) > linelimit:
+                    desc = desc[:linelimit] + '...'
+
+                if plugin.startswith('_'):  # Handle deprecated
+                    deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
+                else:
+                    text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
+            except Exception as e:
+                raise AnsibleError("Failed reading docs at %s: %s" % (plugin, to_native(e)))
 
         if len(deprecated) > 0:
             text.append("\nDEPRECATED:")
@@ -394,7 +406,7 @@ class DocCLI(CLI):
 
     def get_man_text(self, doc):
 
-        IGNORE = frozenset(['module', 'docuri', 'version_added', 'short_description', 'now_date', 'plainexamples', 'returndocs'])
+        IGNORE = frozenset(['module', 'docuri', 'version_added', 'short_description', 'now_date', 'plainexamples', 'returndocs', self.options.type])
         opt_indent = "        "
         text = []
         pad = display.columns * 0.20
@@ -410,7 +422,12 @@ class DocCLI(CLI):
         text.append("%s\n" % textwrap.fill(CLI.tty_ify(desc), limit, initial_indent=opt_indent, subsequent_indent=opt_indent))
 
         if 'deprecated' in doc and doc['deprecated'] is not None and len(doc['deprecated']) > 0:
-            text.append("DEPRECATED: \n%s\n" % doc.pop('deprecated'))
+            text.append("DEPRECATED: \n")
+            if isinstance(doc['deprecated'], dict):
+                text.append("\tReason: %(why)s\n\tScheduled removal: Ansible %(version)s\n\tAlternatives: %(alternative)s" % doc.pop('deprecated'))
+            else:
+                text.append("%s" % doc.pop('deprecated'))
+            text.append("\n")
 
         try:
             support_block = self.get_support_block(doc)

lib/ansible/cli/pull.py

@@ -67,6 +67,21 @@ class PullCLI(CLI):
                                  "look for a playbook based on the host's fully-qualified domain name,"
                                  'on the host hostname and finally a playbook named *local.yml*.', }
 
+    def _get_inv_cli(self):
+
+        inv_opts = ''
+        if getattr(self.options, 'inventory'):
+            for inv in self.options.inventory:
+                if isinstance(inv, list):
+                    inv_opts += " -i '%s' " % ','.join(inv)
+                elif ',' in inv or os.path.exists(inv):
+                    inv_opts += ' -i %s ' % inv
+
+        if not inv_opts:
+            inv_opts = " -i localhost, "
+
+        return inv_opts
+
     def parse(self):
         ''' create an options parser for bin/ansible '''
 
@@ -158,15 +173,7 @@ class PullCLI(CLI):
 
         # Attempt to use the inventory passed in as an argument
         # It might not yet have been downloaded so use localhost as default
-        inv_opts = ''
-        if getattr(self.options, 'inventory'):
-            for inv in self.options.inventory:
-                if isinstance(inv, list):
-                    inv_opts += " -i '%s' " % ','.join(inv)
-                elif ',' in inv or os.path.exists(inv):
-                    inv_opts += ' -i %s ' % inv
-        else:
-            inv_opts = "-i 'localhost,'"
+        inv_opts = self._get_inv_cli()
 
         # FIXME: enable more repo modules hg/svn?
         if self.options.module_name == 'git':
@@ -231,8 +238,7 @@ class PullCLI(CLI):
         if self.options.vault_password_files:
             for vault_password_file in self.options.vault_password_files:
                 cmd += " --vault-password-file=%s" % vault_password_file
-        if inv_opts:
-            cmd += ' %s' % inv_opts
+
         for ev in self.options.extra_vars:
             cmd += ' -e "%s"' % ev
         if self.options.ask_sudo_pass or self.options.ask_su_pass or self.options.become_ask_pass:
@@ -250,6 +256,11 @@ class PullCLI(CLI):
 
         os.chdir(self.options.dest)
 
+        # redo inventory options as new files might exist now
+        inv_opts = self._get_inv_cli()
+        if inv_opts:
+            cmd += inv_opts
+
         # RUN THE PLAYBOOK COMMAND
         display.debug("running ansible-playbook to do actual work")
         display.debug('EXEC: %s' % cmd)

lib/ansible/cli/vault.py

@@ -66,12 +66,20 @@ class VaultCLI(CLI):
         self.new_encrypt_secret = None
         self.new_encrypt_vault_id = None
 
+        self.can_output = ['encrypt', 'decrypt', 'encrypt_string']
+
         super(VaultCLI, self).__init__(args)
 
     def set_action(self):
 
         super(VaultCLI, self).set_action()
 
+        # add output if needed
+        if self.action in self.can_output:
+            self.parser.add_option('--output', default=None, dest='output_file',
+                                   help='output file name for encrypt or decrypt; use - for stdout',
+                                   action="callback", callback=CLI.unfrack_path, type='string')
+
         # options specific to self.actions
         if self.action == "create":
             self.parser.set_usage("usage: %prog create [options] file_name")
@@ -113,16 +121,12 @@ class VaultCLI(CLI):
 
         display.verbosity = self.options.verbosity
 
-        can_output = ['encrypt', 'decrypt', 'encrypt_string']
-
         if self.options.vault_ids:
             for vault_id in self.options.vault_ids:
                 if u';' in vault_id:
                     raise AnsibleOptionsError("'%s' is not a valid vault id. The character ';' is not allowed in vault ids" % vault_id)
 
-        if self.action not in can_output:
-            if self.options.output_file:
-                raise AnsibleOptionsError("The --output option can be used only with ansible-vault %s" % '/'.join(can_output))
+        if self.action not in self.can_output:
             if len(self.args) == 0:
                 raise AnsibleOptionsError("Vault requires at least one filename as a parameter")
         else:
@@ -138,8 +142,7 @@ class VaultCLI(CLI):
             if '-' in self.args or len(self.args) == 0 or self.options.encrypt_string_stdin_name:
                 self.encrypt_string_read_stdin = True
 
-            # TODO: prompting from stdin and reading from stdin seem
-            #       mutually exclusive, but verify that.
+            # TODO: prompting from stdin and reading from stdin seem mutually exclusive, but verify that.
             if self.options.encrypt_string_prompt and self.encrypt_string_read_stdin:
                 raise AnsibleOptionsError('The --prompt option is not supported if also reading input from stdin')
 
@@ -162,7 +165,7 @@ class VaultCLI(CLI):
 
         # TODO: instead of prompting for these before, we could let VaultEditor
         #       call a callback when it needs it.
-        if self.action in ['decrypt', 'view', 'rekey']:
+        if self.action in ['decrypt', 'view', 'rekey', 'edit']:
             vault_secrets = self.setup_vault_secrets(loader,
                                                      vault_ids=vault_ids,
                                                      vault_password_files=self.options.vault_password_files,
@@ -170,7 +173,7 @@ class VaultCLI(CLI):
             if not vault_secrets:
                 raise AnsibleOptionsError("A vault password is required to use Ansible's Vault")
 
-        if self.action in ['encrypt', 'encrypt_string', 'create', 'edit']:
+        if self.action in ['encrypt', 'encrypt_string', 'create']:
             if len(vault_ids) > 1:
                 raise AnsibleOptionsError("Only one --vault-id can be used for encryption")
 
@@ -181,6 +184,10 @@ class VaultCLI(CLI):
                                          vault_password_files=self.options.vault_password_files,
                                          ask_vault_pass=self.options.ask_vault_pass,
                                          create_new_password=True)
+
+            if len(vault_secrets) > 1:
+                raise AnsibleOptionsError("Only one --vault-id can be used for encryption. This includes passwords from configuration and cli.")
+
             if not vault_secrets:
                 raise AnsibleOptionsError("A vault password is required to use Ansible's Vault")
 

lib/ansible/config/manager.py

@@ -22,11 +22,12 @@ from ansible.parsing.quoting import unquote
 from ansible.utils.path import unfrackpath
 from ansible.utils.path import makedirs_safe
 
-Plugin = namedtuple('Plugin','name type')
-Setting = namedtuple('Setting','name value origin')
+Plugin = namedtuple('Plugin', 'name type')
+Setting = namedtuple('Setting', 'name value origin type')
+
 
 # FIXME: see if we can unify in module_utils with similar function used by argspec
-def ensure_type(value, value_type):
+def ensure_type(value, value_type, origin=None):
     ''' return a configuration variable with casting
     :arg value: The value to ensure correct typing of
     :kwarg value_type: The type of the value.  This can be any of the following strings:
@@ -43,6 +44,11 @@ def ensure_type(value, value_type):
             means colon separated strings.)  Split the value and then expand
             each part for environment variables and tildes.
     '''
+
+    basedir = None
+    if origin and os.path.isabs(origin) and os.path.exists(origin):
+        basedir = origin
+
     if value_type:
         value_type = value_type.lower()
 
@@ -65,18 +71,24 @@ def ensure_type(value, value_type):
                 value = None
 
         elif value_type == 'path':
-            value = resolve_path(value)
+            value = resolve_path(value, basedir=basedir)
 
         elif value_type in ('tmp', 'temppath', 'tmppath'):
-            value = resolve_path(value)
+            value = resolve_path(value, basedir=basedir)
             if not os.path.exists(value):
                 makedirs_safe(value, 0o700)
             prefix = 'ansible-local-%s' % os.getpid()
             value = tempfile.mkdtemp(prefix=prefix, dir=value)
 
+        elif value_type == 'pathspec':
+            if isinstance(value, string_types):
+                value = value.split(os.pathsep)
+            value = [resolve_path(x, basedir=basedir) for x in value]
+
         elif value_type == 'pathlist':
             if isinstance(value, string_types):
-                value = [resolve_path(x) for x in value.split(os.pathsep)]
+                value = value.split(',')
+            value = [resolve_path(x, basedir=basedir) for x in value]
 
         # defaults to string types
         elif isinstance(value, string_types):
@@ -84,13 +96,15 @@ def ensure_type(value, value_type):
 
     return to_text(value, errors='surrogate_or_strict', nonstring='passthru')
 
+
 # FIXME: see if this can live in utils/path
-def resolve_path(path):
+def resolve_path(path, basedir=None):
     ''' resolve relative or 'varaible' paths '''
-    if '{{CWD}}' in path: # allow users to force CWD using 'magic' {{CWD}}
+    if '{{CWD}}' in path:  # allow users to force CWD using 'magic' {{CWD}}
         path = path.replace('{{CWD}}', os.getcwd())
 
-    return unfrackpath(path, follow=False)
+    return unfrackpath(path, follow=False, basedir=basedir)
+
 
 # FIXME: generic file type?
 def get_config_type(cfile):
@@ -107,17 +121,19 @@ def get_config_type(cfile):
 
     return ftype
 
+
 # FIXME: can move to module_utils for use for ini plugins also?
 def get_ini_config_value(p, entry):
     ''' returns the value of last ini entry found '''
     value = None
     if p is not None:
         try:
-            value = p.get(entry.get('section','defaults'), entry.get('key',''), raw=True)
-        except: # FIXME: actually report issues here
+            value = p.get(entry.get('section', 'defaults'), entry.get('key', ''), raw=True)
+        except:  # FIXME: actually report issues here
             pass
     return value
 
+
 def find_ini_config_file():
     ''' Load INI Config File order(first found is used): ENV, CWD, HOME, /etc/ansible '''
     # FIXME: eventually deprecate ini configs
@@ -142,6 +158,7 @@ def find_ini_config_file():
 
     return path
 
+
 class ConfigManager(object):
 
     UNABLE = []
@@ -156,8 +173,7 @@ class ConfigManager(object):
         self._config_file = conf_file
         self.data = ConfigData()
 
-
-        #FIXME: make dynamic? scan for more? make it's own method?
+        # FIXME: make dynamic? scan for more? make it's own method?
         # Create configuration definitions from source
         bconfig_def = to_bytes('%s/base.yml' % os.path.dirname(__file__))
         if os.path.exists(bconfig_def):
@@ -200,11 +216,18 @@ class ConfigManager(object):
             else:
                 raise AnsibleOptionsError("Unsupported configuration file type: %s" % to_native(ftype))
 
-
     def _find_yaml_config_files(self):
         ''' Load YAML Config Files in order, check merge flags, keep origin of settings'''
         pass
 
+    def get_plugin_options(self, plugin_type, name, variables=None):
+
+        options = {}
+        defs = self.get_configuration_definitions(plugin_type, name)
+        for option in defs:
+            options[option] = self.get_config_value(option, plugin_type=plugin_type, plugin_name=name, variables=variables)
+
+        return options
 
     def get_configuration_definitions(self, plugin_type=None, name=None):
         ''' just list the possible settings, either base or for specific plugins or plugin '''
@@ -215,7 +238,7 @@ class ConfigManager(object):
         elif name is None:
             ret = self._plugins.get(plugin_type, {})
         else:
-            ret = {name: self._plugins.get(plugin_type, {}).get(name, {})}
+            ret = self._plugins.get(plugin_type, {}).get(name, {})
 
         return ret
 
@@ -227,12 +250,13 @@ class ConfigManager(object):
         for entry in entry_list:
             name = entry.get('name')
             temp_value = container.get(name, None)
-            if temp_value is not None: # only set if env var is defined
+            if temp_value is not None:  # only set if env var is defined
                 value = temp_value
                 origin = name
 
                 # deal with deprecation of setting source, if used
-                #FIXME: if entry.get('deprecated'):
+                if 'deprecated' in entry:
+                    self.DEPRECATED.append((entry['name'], entry['deprecated']))
 
         return value, origin
 
@@ -273,15 +297,18 @@ class ConfigManager(object):
             if ftype and defs[config].get(ftype):
                 if ftype == 'ini':
                     # load from ini config
-                    try: # FIXME: generaelize _loop_entries to allow for files also
+                    try:  # FIXME: generaelize _loop_entries to allow for files also, most of this code is dupe
                         for ini_entry in defs[config]['ini']:
-                            value = get_ini_config_value(self._parser, ini_entry)
-                            origin = cfile
-                            #FIXME: if ini_entry.get('deprecated'):
+                            temp_value = get_ini_config_value(self._parser, ini_entry)
+                            if temp_value is not None:
+                                value = temp_value
+                                origin = cfile
+                                if 'deprecated' in ini_entry:
+                                    self.DEPRECATED.append(('[%s]%s' % (ini_entry['section'], ini_entry['key']), ini_entry['deprecated']))
                     except Exception as e:
                         sys.stderr.write("Error while loading ini config %s: %s" % (cfile, to_native(e)))
                 elif ftype == 'yaml':
-                    pass # FIXME: implement, also , break down key from defs (. notation???)
+                    pass  # FIXME: implement, also , break down key from defs (. notation???)
                     origin = cfile
 
         '''
@@ -295,37 +322,22 @@ class ConfigManager(object):
         if value is None:
             value = defs[config].get('default')
             origin = 'default'
-            # FIXME: moved eval to constants as this does not have access to previous vars
-            if plugin_type is None and isinstance(value, string_types) and (value.startswith('eval(') and value.endswith(')')):
+            # skip typing as this is a temlated default that will be resolved later in constants, which has needed vars
+            if plugin_type is None and isinstance(value, string_types) and (value.startswith('{{') and value.endswith('}}')):
                 return value, origin
-            #default_value = defs[config].get('default')
-            #if plugin_type is None and isinstance(default_value, string_types) and (default_value.startswith('eval(') and default_value.endswith(')')):
-            #    try:
-            #        eval_string = default_value.replace('eval(', '', 1)[:-1]
-            #        value = eval(eval_string) # FIXME: safe eval?
-            #    except:
-            #        value = default_value
-            #else:
-            #    value = default_value
 
         # ensure correct type
         try:
-            value = ensure_type(value, defs[config].get('type'))
+            value = ensure_type(value, defs[config].get('type'), origin=origin)
         except Exception as e:
             self.UNABLE.append(config)
 
         # deal with deprecation of the setting
-        if defs[config].get('deprecated') and origin != 'default':
+        if 'deprecated' in defs[config] and origin != 'default':
             self.DEPRECATED.append((config, defs[config].get('deprecated')))
 
         return value, origin
 
-    def update_plugin_config(self, plugin_type, name, defs):
-        ''' really: update constants '''
-        # no sense?
-        self.initialize_plugin_configuration_definitions(plugin_type, name, defs)
-        self.update_config_data(defs)
-
     def initialize_plugin_configuration_definitions(self, plugin_type, name, defs):
 
         if plugin_type not in self._plugins:
@@ -346,7 +358,7 @@ class ConfigManager(object):
             raise AnsibleOptionsError("Invalid configuration definition type: %s for %s" % (type(defs), defs))
 
         # update the constant for config file
-        self.data.update_setting(Setting('CONFIG_FILE', configfile, ''))
+        self.data.update_setting(Setting('CONFIG_FILE', configfile, '', 'string'))
 
         origin = None
         # env and config defs can have several entries, ordered in list from lowest to highest precedence
@@ -358,12 +370,4 @@ class ConfigManager(object):
             value, origin = self.get_config_value_and_origin(config, configfile)
 
             # set the constant
-            self.data.update_setting(Setting(config, value, origin))
-
-        # FIXME: find better way to do this by passing back to where display is available
-        if self.UNABLE:
-            sys.stderr.write("Unable to set correct type for:\n\t%s\n" % '\n\t'.join(self.UNABLE))
-        if self.DEPRECATED:
-            for k, reason in self.DEPRECATED:
-                sys.stderr.write("[DEPRECATED] %(k)s: %(why)s. It will be removed in %(version)s. As alternative use one of [%(alternatives)s]\n"
-                                 % dict(k=k, **reason))
+            self.data.update_setting(Setting(config, value, origin, defs[config].get('type', 'string')))

lib/ansible/constants.py

@@ -6,14 +6,17 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import os  # used to set lang
+import os  # used to set lang and for backwards compat get_config
 
+from ast import literal_eval
+from jinja2 import Template
 from string import ascii_letters, digits
 
 from ansible.module_utils._text import to_text
 from ansible.module_utils.parsing.convert_bool import boolean, BOOLEANS_TRUE
 from ansible.module_utils.six import string_types
-from ansible.config.manager import ConfigManager
+from ansible.config.manager import ConfigManager, ensure_type, get_ini_config_value
+
 
 def _deprecated(msg):
     ''' display is not guaranteed here, nor it being the full class, but try anyways, fallback to sys.stderr.write '''
@@ -24,36 +27,39 @@ def _deprecated(msg):
         import sys
         sys.stderr.write('[DEPRECATED] %s, to be removed in 2.8' % msg)
 
+
 def mk_boolean(value):
     ''' moved to module_utils'''
     _deprecated('ansible.constants.mk_boolean() is deprecated.  Use ansible.module_utils.parsing.convert_bool.boolean() instead')
     return boolean(value, strict=False)
 
+
 def get_config(parser, section, key, env_var, default_value, value_type=None, expand_relative_paths=False):
     ''' kept for backwarsd compatibility, but deprecated '''
     _deprecated('ansible.constants.get_config() is deprecated. There is new config API, see porting docs.')
 
-    import os
-
     value = None
     # small reconstruction of the old code env/ini/default
     value = os.environ.get(env_var, None)
     if value is None:
         try:
-            value = config.get_ini_config(parser, [{'key': key, 'section': section}])
+            value = get_ini_config_value(parser, {'key': key, 'section': section})
         except:
             pass
     if value is None:
         value = default_value
-    try:
-        value = config.ensure_type(value, value_type)
-    except:
-        pass
+
+    value = ensure_type(value, value_type)
 
     return value
 
+
+def set_constant(name, value, export=vars()):
+    ''' sets constants and returns resolved options dict '''
+    export[name] = value
+
+
 ### CONSTANTS ### yes, actual ones
-BLACKLIST_EXTS = ('.pyc', '.pyo', '.swp', '.bak', '~', '.rpm', '.md', '.txt')
 BECOME_METHODS = ['sudo', 'su', 'pbrun', 'pfexec', 'doas', 'dzdo', 'ksu', 'runas', 'pmrun']
 BECOME_ERROR_STRINGS = {
     'sudo': 'Sorry, try again.',
@@ -75,7 +81,9 @@ BECOME_MISSING_STRINGS = {
     'ksu': 'No password given',
     'pmrun': ''
 }  # FIXME: deal with i18n
+BLACKLIST_EXTS = ('.pyc', '.pyo', '.swp', '.bak', '~', '.rpm', '.md', '.txt')
 BOOL_TRUE = BOOLEANS_TRUE
+CONTROLER_LANG = os.getenv('LANG', 'en_US.UTF-8')
 DEFAULT_BECOME_PASS = None
 DEFAULT_PASSWORD_CHARS = to_text(ascii_letters + digits + ".,:-_", errors='strict')  # characters included in auto-generated passwords
 DEFAULT_SUDO_PASS = None
@@ -98,15 +106,19 @@ config = ConfigManager()
 # Generate constants from config
 for setting in config.data.get_settings():
 
-    # FIXME: find better way to do in manager class and/or ensure types
-    if isinstance(setting.value, string_types) and (setting.value.startswith('eval(') and setting.value.endswith(')')):
+    value = setting.value
+    if setting.origin == 'default' and \
+       isinstance(setting.value, string_types) and \
+       (setting.value.startswith('{{') and setting.value.endswith('}}')):
         try:
-            eval_string = setting.value.replace('eval(', '', 1)[:-1]
-            vars()[setting.name] = eval(eval_string) # FIXME: safe eval?
-            continue
+            t = Template(setting.value)
+            value = t.render(vars())
+            try:
+                value = literal_eval(value)
+            except ValueError:
+                pass  # not a python data structure
         except:
-            pass
-
-    vars()[setting.name] = setting.value
-
+            pass # not templatable
+        value = ensure_type(value, setting.name)
 
+    set_constant(setting.name, value)

lib/ansible/executor/task_executor.py

@@ -414,6 +414,8 @@ class TaskExecutor:
             # We also add "magic" variables back into the variables dict to make sure
             # a certain subset of variables exist.
             self._play_context.update_vars(variables)
+
+            # FIXME: update connection/shell plugin options
         except AnsibleError as e:
             # save the error, which we'll raise later if we don't end up
             # skipping this task during the conditional evaluation step
@@ -474,18 +476,11 @@ class TaskExecutor:
                 not getattr(self._connection, 'connected', False) or
                 self._play_context.remote_addr != self._connection._play_context.remote_addr):
             self._connection = self._get_connection(variables=variables, templar=templar)
-            hostvars = variables.get('hostvars', None)
             # only template the vars if the connection actually implements set_host_overrides
             # NB: this is expensive, and should be removed once connection-specific vars are being handled by play_context
             sho_impl = getattr(type(self._connection), 'set_host_overrides', None)
-            if hostvars and sho_impl and sho_impl != ConnectionBase.set_host_overrides:
-                try:
-                    target_hostvars = hostvars.get(self._host.name)
-                except:
-                    # FIXME: this should catch the j2undefined error here
-                    #        specifically instead of all exceptions
-                    target_hostvars = dict()
-                self._connection.set_host_overrides(host=self._host, hostvars=target_hostvars)
+            if sho_impl and sho_impl != ConnectionBase.set_host_overrides:
+                self._connection.set_host_overrides(self._host, variables, templar)
         else:
             # if connection is reused, its _play_context is no longer valid and needs
             # to be replaced with the one templated above, in case other data changed
@@ -731,11 +726,11 @@ class TaskExecutor:
             conn_type = self._play_context.connection
 
         connection = self._shared_loader_obj.connection_loader.get(conn_type, self._play_context, self._new_stdin)
-        self._play_context.set_options_from_plugin(connection)
-
         if not connection:
             raise AnsibleError("the connection plugin '%s' was not found" % conn_type)
 
+        self._play_context.set_options_from_plugin(connection)
+
         if self._play_context.accelerate:
             # accelerate is deprecated as of 2.1...
             display.deprecated('Accelerated mode is deprecated. Consider using SSH with ControlPersist and pipelining enabled instead', version='2.6')

lib/ansible/executor/task_queue_manager.py

@@ -84,9 +84,9 @@ class TaskQueueManager:
         self._start_at_done = False
 
         # make sure any module paths (if specified) are added to the module_loader
-        if isinstance(options.module_path, list):
+        if options.module_path:
             for path in options.module_path:
-                if path is not None:
+                if path:
                     module_loader.add_directory(path)
 
         # a special flag to help us exit cleanly
@@ -176,6 +176,12 @@ class TaskQueueManager:
                 raise AnsibleError("Invalid callback for stdout specified: %s" % self._stdout_callback)
             else:
                 self._stdout_callback = callback_loader.get(self._stdout_callback)
+                try:
+                    self._stdout_callback.set_options(C.config.get_plugin_options('callback', self._stdout_callback._load_name))
+                except AttributeError:
+                    display.deprecated("%s stdout callback, does not support setting 'options', it will work for now, "
+                                       " but this will be required in the future and should be updated,"
+                                       " see the 2.4 porting guide for details." % self._stdout_callback._load_name, version="2.9")
                 stdout_callback_loaded = True
         else:
             raise AnsibleError("callback must be an instance of CallbackBase or the name of a callback plugin")
@@ -198,7 +204,14 @@ class TaskQueueManager:
                         C.DEFAULT_CALLBACK_WHITELIST is None or callback_name not in C.DEFAULT_CALLBACK_WHITELIST)):
                     continue
 
-            self._callback_plugins.append(callback_plugin())
+            callback_obj = callback_plugin()
+            try:
+                callback_obj .set_options(C.config.get_plugin_options('callback', callback_plugin._load_name))
+            except AttributeError:
+                    display.deprecated("%s callback, does not support setting 'options', it will work for now, "
+                                       " but this will be required in the future and should be updated, "
+                                       " see the 2.4 porting guide for details." % self._stdout_callback._load_name, version="2.9")
+            self._callback_plugins.append(callback_obj)
 
         self._callbacks_loaded = True
 
@@ -366,4 +379,4 @@ class TaskQueueManager:
                     display.warning(u"Failure using method (%s) in callback plugin (%s): %s" % (to_text(method_name), to_text(callback_plugin), to_text(e)))
                     from traceback import format_tb
                     from sys import exc_info
-                    display.debug('Callback Exception: \n' + ' '.join(format_tb(exc_info()[2])))
+                    display.vvv('Callback Exception: \n' + ' '.join(format_tb(exc_info()[2])))

lib/ansible/inventory/data.py

@@ -19,7 +19,6 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import os
 import sys
 
 from ansible import constants as C
@@ -187,7 +186,7 @@ class InventoryData(object):
             h = Host(host, port)
             self.hosts[host] = h
             if self.current_source:  # set to 'first source' in which host was encountered
-                self.set_variable(host, 'inventory_file', os.path.basename(self.current_source))
+                self.set_variable(host, 'inventory_file', self.current_source)
                 self.set_variable(host, 'inventory_dir', basedir(self.current_source))
             else:
                 self.set_variable(host, 'inventory_file', None)
@@ -204,7 +203,7 @@ class InventoryData(object):
         else:
             h = self.hosts[host]
 
-        if g and h not in g.get_hosts():
+        if g:
             g.add_host(h)
             self._groups_dict_cache = {}
             display.debug("Added host %s to group %s" % (host, group))

lib/ansible/inventory/group.py

@@ -18,7 +18,6 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 from ansible.errors import AnsibleError
-from ansible.utils.vars import combine_vars
 
 
 class Group:
@@ -31,6 +30,7 @@ class Group:
         self.depth = 0
         self.name = name
         self.hosts = []
+        self._hosts = None
         self.vars = {}
         self.child_groups = []
         self.parent_groups = []
@@ -54,11 +54,14 @@ class Group:
         for parent in self.parent_groups:
             parent_groups.append(parent.serialize())
 
+        self._hosts = None
+
         result = dict(
             name=self.name,
             vars=self.vars.copy(),
             parent_groups=parent_groups,
             depth=self.depth,
+            hosts=self.hosts,
         )
 
         return result
@@ -68,6 +71,8 @@ class Group:
         self.name = data.get('name')
         self.vars = data.get('vars', dict())
         self.depth = data.get('depth', 0)
+        self.hosts = data.get('hosts', [])
+        self._hosts = None
 
         parent_groups = data.get('parent_groups', [])
         for parent_data in parent_groups:
@@ -75,6 +80,12 @@ class Group:
             g.deserialize(parent_data)
             self.parent_groups.append(g)
 
+    @property
+    def host_names(self):
+        if self._hosts is None:
+            self._hosts = set(self.hosts)
+        return self._hosts
+
     def get_name(self):
         return self.name
 
@@ -112,17 +123,19 @@ class Group:
             raise AnsibleError("The group named '%s' has a recursive dependency loop." % self.name)
 
     def add_host(self, host):
-        if host in self.hosts:
-            return
-        self.hosts.append(host)
-        host.add_group(self)
-        self.clear_hosts_cache()
+        if host.name not in self.host_names:
+            self.hosts.append(host)
+            self._hosts.add(host.name)
+            host.add_group(self)
+            self.clear_hosts_cache()
 
     def remove_host(self, host):
 
-        self.hosts.remove(host)
-        host.remove_group(self)
-        self.clear_hosts_cache()
+        if host.name in self.host_names:
+            self.hosts.remove(host)
+            self._hosts.remove(host.name)
+            host.remove_group(self)
+            self.clear_hosts_cache()
 
     def set_variable(self, key, value):
 

lib/ansible/inventory/host.py

@@ -143,7 +143,7 @@ class Host:
         results['inventory_hostname_short'] = self.name.split('.')[0]
         results['group_names'] = sorted([g.name for g in self.get_groups() if g.name != 'all'])
 
-        return combine_vars(self.vars, results)
+        return results
 
     def get_vars(self):
         return combine_vars(self.vars, self.get_magic_vars())

lib/ansible/inventory/manager.py

@@ -168,6 +168,7 @@ class InventoryManager(object):
         return self._inventory.get_groups_dict()
 
     def reconcile_inventory(self):
+        self.clear_caches()
         return self._inventory.reconcile_inventory()
 
     def get_host(self, hostname):
@@ -176,7 +177,7 @@ class InventoryManager(object):
     def _setup_inventory_plugins(self):
         ''' sets up loaded inventory plugins for usage '''
 
-        inventory_loader = PluginLoader('InventoryModule', 'ansible.plugins.inventory', 'inventory_plugins', 'inventory_plugins')
+        inventory_loader = PluginLoader('InventoryModule', 'ansible.plugins.inventory', C.DEFAULT_INVENTORY_PLUGIN_PATH, 'inventory_plugins')
         display.vvvv('setting up inventory plugins')
 
         for name in C.INVENTORY_ENABLED:
@@ -189,7 +190,7 @@ class InventoryManager(object):
         if not self._inventory_plugins:
             raise AnsibleError("No inventory plugins available to generate inventory, make sure you have at least one whitelisted.")
 
-    def parse_sources(self, cache=True):
+    def parse_sources(self, cache=False):
         ''' iterate over inventory sources and parse each one to populate it'''
 
         self._setup_inventory_plugins()
@@ -213,7 +214,7 @@ class InventoryManager(object):
 
         self._inventory_plugins = []
 
-    def parse_source(self, source, cache=True):
+    def parse_source(self, source, cache=False):
         ''' Generate or update inventory for the source provided '''
 
         parsed = False
@@ -268,7 +269,7 @@ class InventoryManager(object):
                 if not parsed and failures:
                     # only if no plugin processed files should we show errors.
                     if C.INVENTORY_UNPARSED_IS_FAILED:
-                        msg = "Could not parse inventory source %s with availabel plugins:\n" % source
+                        msg = "Could not parse inventory source %s with available plugins:\n" % source
                         for fail in failures:
                             msg += 'Plugin %s failed: %s\n' % (fail['plugin'], to_native(fail['exc']))
                             if display.verbosity >= 3:

lib/ansible/module_utils/aci.py

@@ -733,7 +733,7 @@ class ACIModule(object):
     def post_config(self):
         """
         This method is used to handle the logic when the modules state is equal to present. The method only pushes a change if
-        the object has differences than what exists on the APIC, and if check_mode is Fasle. A successful change will mark the
+        the object has differences than what exists on the APIC, and if check_mode is False. A successful change will mark the
         module as changed.
         """
         if not self.result['config']:

lib/ansible/module_utils/asa.py

@@ -62,7 +62,7 @@ def check_args(module):
     provider = module.params['provider'] or {}
 
     for key in asa_argument_spec:
-        if key not in ['provider', 'authorize'] and module.params[key]:
+        if key not in ['context', 'passwords', 'provider', 'authorize'] and module.params[key]:
             module.warn('argument %s has been deprecated and will be removed in a future version' % key)
 
     if provider:

lib/ansible/module_utils/azure_rm_common.py

@@ -149,7 +149,7 @@ class AzureRMModuleBase(object):
     def __init__(self, derived_arg_spec, bypass_checks=False, no_log=False,
                  check_invalid_arguments=True, mutually_exclusive=None, required_together=None,
                  required_one_of=None, add_file_common_args=False, supports_check_mode=False,
-                 required_if=None, supports_tags=True, facts_module=False):
+                 required_if=None, supports_tags=True, facts_module=False, skip_exec=False):
 
         merged_arg_spec = dict()
         merged_arg_spec.update(AZURE_COMMON_ARGS)
@@ -252,8 +252,9 @@ class AzureRMModuleBase(object):
         if self.module.params.get('tags'):
             self.validate_tags(self.module.params['tags'])
 
-        res = self.exec_module(**self.module.params)
-        self.module.exit_json(**res)
+        if not skip_exec:
+            res = self.exec_module(**self.module.params)
+            self.module.exit_json(**res)
 
     def check_client_version(self, client_name, client_version, expected_version):
         # Ensure Azure modules are at least 2.0.0rc5.

lib/ansible/module_utils/basic.py

@@ -1133,9 +1133,9 @@ class AnsibleModule(object):
                 return True
             try:
                 os.lchown(b_path, uid, -1)
-            except OSError:
+            except (IOError, OSError) as e:
                 path = to_text(b_path)
-                self.fail_json(path=path, msg='chown failed')
+                self.fail_json(path=path, msg='chown failed: %s' % (to_text(e)))
             changed = True
         return changed
 
@@ -2533,6 +2533,7 @@ class AnsibleModule(object):
         # sadly there are some situations where we cannot ensure atomicity, but only if
         # the user insists and we get the appropriate error we update the file unsafely
         try:
+            out_dest = in_src = None
             try:
                 out_dest = open(dest, 'wb')
                 in_src = open(src, 'rb')
@@ -2601,6 +2602,7 @@ class AnsibleModule(object):
             strings on python3, use encoding=None to turn decoding to text off.
         '''
 
+        shell = False
         if isinstance(args, list):
             if use_unsafe_shell:
                 args = " ".join([shlex_quote(x) for x in args])
@@ -2608,27 +2610,17 @@ class AnsibleModule(object):
         elif isinstance(args, (binary_type, text_type)) and use_unsafe_shell:
             shell = True
         elif isinstance(args, (binary_type, text_type)):
-            if not use_unsafe_shell:
-                # On python2.6 and below, shlex has problems with text type
-                # On python3, shlex needs a text type.
-                if PY2:
-                    args = to_bytes(args, errors='surrogate_or_strict')
-                elif PY3:
-                    args = to_text(args, errors='surrogateescape')
-                args = shlex.split(args)
+            # On python2.6 and below, shlex has problems with text type
+            # On python3, shlex needs a text type.
+            if PY2:
+                args = to_bytes(args, errors='surrogate_or_strict')
+            elif PY3:
+                args = to_text(args, errors='surrogateescape')
+            args = shlex.split(args)
         else:
             msg = "Argument 'args' to run_command must be list or string"
             self.fail_json(rc=257, cmd=args, msg=msg)
 
-        shell = False
-        if use_unsafe_shell:
-            if executable is None:
-                executable = os.environ.get('SHELL')
-            if executable:
-                args = [executable, '-c', args]
-            else:
-                shell = True
-
         prompt_re = None
         if prompt_regex:
             if isinstance(prompt_regex, text_type):

lib/ansible/module_utils/ce.py

@@ -61,7 +61,7 @@ ce_provider_spec = {
     'use_ssl': dict(type='bool'),
     'validate_certs': dict(type='bool'),
     'timeout': dict(type='int'),
-    'transport': dict(choices=['cli']),
+    'transport': dict(default='cli', choices=['cli']),
 }
 ce_argument_spec = {
     'provider': dict(type='dict', options=ce_provider_spec),

lib/ansible/module_utils/cnos.py

@@ -2178,7 +2178,8 @@ def checkVlanNameNotAssigned(
     retVal = waitForDeviceResponse(command, prompt, timeout, obj)
     if(retVal.find(vlanName) != -1):
         return "Nok"
-    return retVal
+    else:
+        return "ok"
 # EOM
 
 
@@ -3158,10 +3159,13 @@ def waitForDeviceResponse(command, prompt, timeout, obj):
             if(gotit != -1):
                 flag = True
         except:
-            if prompt != "(yes/no)?":
-                retVal = retVal + "\n Error-101"
-            else:
+            # debugOutput(prompt)
+            if prompt == "(yes/no)?":
                 retVal = retVal
+            elif prompt == "Password:":
+                retVal = retVal
+            else:
+                retVal = retVal + "\n Error-101"
             flag = True
     return retVal
 # EOM
@@ -3175,7 +3179,7 @@ def checkOutputForError(output):
         index = output.lower().find("invalid")
         startIndex = index + 8
         if(index == -1):
-            index = output.lower().find("Cannot be enabled in L2 Interface")
+            index = output.lower().find("cannot be enabled in l2 interface")
             startIndex = index + 34
             if(index == -1):
                 index = output.lower().find("incorrect")

lib/ansible/module_utils/crypto.py

@@ -109,6 +109,12 @@ keyUsageLong = {
 }
 
 extendedKeyUsageLong = {
+    "anyExtendedKeyUsage": "Any Extended Key Usage",
+    "ipsecEndSystem": "IPSec End System",
+    "ipsecTunnel": "IPSec Tunnel",
+    "ipsecUser": "IPSec User",
+    "msSGC": "Microsoft Server Gated Crypto",
+    "nsSGC": "Netscape Server Gated Crypto",
     "serverAuth": "TLS Web Server Authentication",
     "clientAuth": "TLS Web Client Authentication",
     "codeSigning": "Code Signing",

lib/ansible/module_utils/dellos6.py

@@ -159,7 +159,7 @@ def os6_parse(lines, indent=None, comment_tokens=None):
         re.compile(r'support-assist.*$'),
         re.compile(r'template.*$'),
         re.compile(r'address-family.*$'),
-        re.compile(r'spanning-tree mst.*$'),
+        re.compile(r'spanning-tree mst configuration.*$'),
         re.compile(r'logging.*$'),
         re.compile(r'(radius-server|tacacs-server) host.*$')]
 
@@ -205,7 +205,7 @@ def os6_parse(lines, indent=None, comment_tokens=None):
                     if parent:
                         cfg._parents.extend(parent)
                     parent = list()
-                    config.append(cfg)
+                config.append(cfg)
             # handle sublevel children
             elif parent_match is False and len(parent) > 0:
                 if not children:
@@ -231,7 +231,11 @@ class Dellos6NetworkConfig(NetworkConfig):
         for item in self.items:
             if str(item) == "exit":
                 for diff_item in diff:
-                    if item._parents == diff_item._parents:
+                    if diff_item._parents:
+                        if item._parents == diff_item._parents:
+                            diff.append(item)
+                            break
+                    else:
                         diff.append(item)
                         break
             elif item not in other:

lib/ansible/module_utils/ec2.py

@@ -28,7 +28,6 @@
 
 import os
 import re
-from time import sleep
 
 from ansible.module_utils._text import to_native
 from ansible.module_utils.cloud import CloudRetry
@@ -437,7 +436,7 @@ def ansible_dict_to_boto3_filter_list(filters_dict):
     return filters_list
 
 
-def boto3_tag_list_to_ansible_dict(tags_list, tag_name_key_name='Key', tag_value_key_name='Value'):
+def boto3_tag_list_to_ansible_dict(tags_list, tag_name_key_name=None, tag_value_key_name=None):
 
     """ Convert a boto3 list of resource tags to a flat dict of key:value pairs
     Args:
@@ -460,12 +459,17 @@ def boto3_tag_list_to_ansible_dict(tags_list, tag_name_key_name='Key', tag_value
         }
     """
 
-    tags_dict = {}
-    for tag in tags_list:
-        if tag_name_key_name in tag:
-            tags_dict[tag[tag_name_key_name]] = tag[tag_value_key_name]
+    if tag_name_key_name and tag_value_key_name:
+        tag_candidates = {tag_name_key_name: tag_value_key_name}
+    else:
+        tag_candidates = {'key': 'value', 'Key': 'Value'}
 
-    return tags_dict
+    if not tags_list:
+        return {}
+    for k, v in tag_candidates.items():
+        if k in tags_list[0] and v in tags_list[0]:
+            return dict((tag[k], tag[v]) for tag in tags_list)
+    raise ValueError("Couldn't find tag key (candidates %s) in tag list %s" % (str(tag_candidates), str(tags_list)))
 
 
 def ansible_dict_to_boto3_tag_list(tags_dict, tag_name_key_name='Key', tag_value_key_name='Value'):

lib/ansible/module_utils/eos.py

@@ -53,7 +53,7 @@ eos_provider_spec = {
     'validate_certs': dict(type='bool'),
     'timeout': dict(type='int'),
 
-    'transport': dict(choices=['cli', 'eapi'])
+    'transport': dict(default='cli', choices=['cli', 'eapi'])
 }
 eos_argument_spec = {
     'provider': dict(type='dict', options=eos_provider_spec),
@@ -79,7 +79,7 @@ def check_args(module, warnings):
                     module.params[key]):
                 warnings.append('argument %s has been deprecated and will be removed in a future version' % key)
         else:
-            if key not in ['provider', 'authorize'] and module.params[key]:
+            if key not in ['provider', 'transport', 'authorize'] and module.params[key]:
                 warnings.append('argument %s has been deprecated and will be removed in a future version' % key)
 
     # set argument's default value if not provided in input

lib/ansible/module_utils/facts/collector.py

@@ -203,6 +203,28 @@ def build_fact_id_to_collector_map(collectors_for_platform):
     return fact_id_to_collector_map, aliases_map
 
 
+def select_collector_classes(collector_names, all_fact_subsets, all_collector_classes):
+    # TODO: can be a set()
+    seen_collector_classes = []
+
+    selected_collector_classes = []
+
+    for candidate_collector_class in all_collector_classes:
+        candidate_collector_name = candidate_collector_class.name
+
+        if candidate_collector_name not in collector_names:
+            continue
+
+        collector_classes = all_fact_subsets.get(candidate_collector_name, [])
+
+        for collector_class in collector_classes:
+            if collector_class not in seen_collector_classes:
+                selected_collector_classes.append(collector_class)
+                seen_collector_classes.append(collector_class)
+
+    return selected_collector_classes
+
+
 def collector_classes_from_gather_subset(all_collector_classes=None,
                                          valid_subsets=None,
                                          minimal_gather_subset=None,
@@ -248,19 +270,8 @@ def collector_classes_from_gather_subset(all_collector_classes=None,
                                           aliases_map=aliases_map,
                                           platform_info=platform_info)
 
-    # TODO: can be a set()
-    seen_collector_classes = []
-
-    selected_collector_classes = []
-
-    for collector_name in collector_names:
-        collector_classes = all_fact_subsets.get(collector_name, [])
-
-        # TODO? log/warn if we dont find an implementation of a fact_id?
-
-        for collector_class in collector_classes:
-            if collector_class not in seen_collector_classes:
-                selected_collector_classes.append(collector_class)
-                seen_collector_classes.append(collector_class)
+    selected_collector_classes = select_collector_classes(collector_names,
+                                                          all_fact_subsets,
+                                                          all_collector_classes)
 
     return selected_collector_classes

lib/ansible/module_utils/facts/default_collectors.py

@@ -32,6 +32,7 @@ from ansible.module_utils.facts.system.fips import FipsFactCollector
 from ansible.module_utils.facts.system.local import LocalFactCollector
 from ansible.module_utils.facts.system.lsb import LSBFactCollector
 from ansible.module_utils.facts.system.pkg_mgr import PkgMgrFactCollector
+from ansible.module_utils.facts.system.pkg_mgr import OpenBSDPkgMgrFactCollector
 from ansible.module_utils.facts.system.platform import PlatformFactCollector
 from ansible.module_utils.facts.system.python import PythonFactCollector
 from ansible.module_utils.facts.system.selinux import SelinuxFactCollector
@@ -72,58 +73,81 @@ from ansible.module_utils.facts.virtual.netbsd import NetBSDVirtualCollector
 from ansible.module_utils.facts.virtual.openbsd import OpenBSDVirtualCollector
 from ansible.module_utils.facts.virtual.sunos import SunOSVirtualCollector
 
+# these should always be first due to most other facts depending on them
+_base = [
+    PlatformFactCollector,
+    DistributionFactCollector,
+    LSBFactCollector
+]
+
+# These restrict what is possible in others
+_restrictive = [
+    SelinuxFactCollector,
+    ApparmorFactCollector,
+    FipsFactCollector
+]
+
+# general info, not required but probably useful for other facts
+_general = [
+    PythonFactCollector,
+    SystemCapabilitiesFactCollector,
+    PkgMgrFactCollector,
+    OpenBSDPkgMgrFactCollector,
+    ServiceMgrFactCollector,
+    CmdLineFactCollector,
+    DateTimeFactCollector,
+    EnvFactCollector,
+    SshPubKeyFactCollector,
+    UserFactCollector
+]
+
+# virtual, this might also limit hardware/networking
+_virtual = [
+    VirtualCollector,
+    DragonFlyVirtualCollector,
+    FreeBSDVirtualCollector,
+    LinuxVirtualCollector,
+    OpenBSDVirtualCollector,
+    NetBSDVirtualCollector,
+    SunOSVirtualCollector,
+    HPUXVirtualCollector
+]
+
+_hardware = [
+    HardwareCollector,
+    AIXHardwareCollector,
+    DarwinHardwareCollector,
+    DragonFlyHardwareCollector,
+    FreeBSDHardwareCollector,
+    HPUXHardwareCollector,
+    HurdHardwareCollector,
+    LinuxHardwareCollector,
+    NetBSDHardwareCollector,
+    OpenBSDHardwareCollector,
+    SunOSHardwareCollector
+]
+
+_network = [
+    DnsFactCollector,
+    NetworkCollector,
+    AIXNetworkCollector,
+    DarwinNetworkCollector,
+    DragonFlyNetworkCollector,
+    FreeBSDNetworkCollector,
+    HPUXNetworkCollector,
+    HurdNetworkCollector,
+    LinuxNetworkCollector,
+    NetBSDNetworkCollector,
+    OpenBSDNetworkCollector,
+    SunOSNetworkCollector
+]
+
+# other fact sources
+_extra_facts = [
+    LocalFactCollector,
+    FacterFactCollector,
+    OhaiFactCollector
+]
+
 # TODO: make config driven
-collectors = [ApparmorFactCollector,
-              CmdLineFactCollector,
-              DateTimeFactCollector,
-              DistributionFactCollector,
-              DnsFactCollector,
-              EnvFactCollector,
-              FipsFactCollector,
-
-              HardwareCollector,
-              AIXHardwareCollector,
-              DarwinHardwareCollector,
-              DragonFlyHardwareCollector,
-              FreeBSDHardwareCollector,
-              HPUXHardwareCollector,
-              HurdHardwareCollector,
-              LinuxHardwareCollector,
-              NetBSDHardwareCollector,
-              OpenBSDHardwareCollector,
-              SunOSHardwareCollector,
-              LocalFactCollector,
-              LSBFactCollector,
-
-              NetworkCollector,
-              AIXNetworkCollector,
-              DarwinNetworkCollector,
-              DragonFlyNetworkCollector,
-              FreeBSDNetworkCollector,
-              HPUXNetworkCollector,
-              HurdNetworkCollector,
-              LinuxNetworkCollector,
-              NetBSDNetworkCollector,
-              OpenBSDNetworkCollector,
-              SunOSNetworkCollector,
-
-              PkgMgrFactCollector,
-              PlatformFactCollector,
-              PythonFactCollector,
-              SelinuxFactCollector,
-              ServiceMgrFactCollector,
-              SshPubKeyFactCollector,
-              SystemCapabilitiesFactCollector,
-              UserFactCollector,
-
-              VirtualCollector,
-              DragonFlyVirtualCollector,
-              FreeBSDVirtualCollector,
-              LinuxVirtualCollector,
-              OpenBSDVirtualCollector,
-              NetBSDVirtualCollector,
-              SunOSVirtualCollector,
-              HPUXVirtualCollector,
-
-              FacterFactCollector,
-              OhaiFactCollector]
+collectors = _base + _restrictive + _general + _virtual + _hardware + _network + _extra_facts

lib/ansible/module_utils/facts/hardware/linux.py

@@ -193,6 +193,7 @@ class LinuxHardware(Hardware):
 
             # model name is for Intel arch, Processor (mind the uppercase P)
             # works for some ARM devices, like the Sheevaplug.
+            # 'ncpus active' is SPARC attribute
             if key in ['model name', 'Processor', 'vendor_id', 'cpu', 'Vendor', 'processor']:
                 if 'processor' not in cpu_facts:
                     cpu_facts['processor'] = []
@@ -216,6 +217,8 @@ class LinuxHardware(Hardware):
                 cores[coreid] = int(data[1].strip())
             elif key == '# processors':
                 cpu_facts['processor_cores'] = int(data[1].strip())
+            elif key == 'ncpus active':
+                i = int(data[1].strip())
 
         # Skip for platforms without vendor_id/model_name in cpuinfo (e.g ppc64le)
         if vendor_id_occurrence > 0:
@@ -671,7 +674,7 @@ class LinuxHardware(Hardware):
             if vgs_path:
                 rc, vg_lines, err = self.module.run_command('%s %s' % (vgs_path, lvm_util_options))
                 for vg_line in vg_lines.splitlines():
-                    items = vg_line.split(',')
+                    items = vg_line.strip().split(',')
                     vgs[items[0]] = {'size_g': items[-2],
                                      'free_g': items[-1],
                                      'num_lvs': items[2],
@@ -684,7 +687,7 @@ class LinuxHardware(Hardware):
             if lvs_path:
                 rc, lv_lines, err = self.module.run_command('%s %s' % (lvs_path, lvm_util_options))
                 for lv_line in lv_lines.splitlines():
-                    items = lv_line.split(',')
+                    items = lv_line.strip().split(',')
                     lvs[items[0]] = {'size_g': items[3], 'vg': items[1]}
 
             pvs_path = self.module.get_bin_path('pvs')
@@ -693,7 +696,7 @@ class LinuxHardware(Hardware):
             if pvs_path:
                 rc, pv_lines, err = self.module.run_command('%s %s' % (pvs_path, lvm_util_options))
                 for pv_line in pv_lines.splitlines():
-                    items = pv_line.split(',')
+                    items = pv_line.strip().split(',')
                     pvs[self._find_mapper_device_name(items[0])] = {
                         'size_g': items[4],
                         'free_g': items[5],