Commit Graph

356 Commits

Author SHA1 Message Date
80561ccef0 Improve sanity test handling in CI after migration. 2020-02-07 21:29:39 -08:00
a0242b2173 Improve unit test handling in CI after migration. (#67217)
* Improve unit test handling in CI after migration.

* Fix undefined var in bash 4.3 and earlier.

See https://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n878
2020-02-07 21:28:46 -08:00
5e68bb3d93 Add code coverage target analysis to ansible-test. (#67141)
* Refactor coverage file enumeration.
* Relocate sanitize_filename function.
* Support sets when writing JSON files.
* Generalize setting of info_stderr mode.
* Split out coverage path checking.
* Split out collection regex logic.
* Improve sanitize_filename type hints and docs.
* Clean up coverage erase command.
* Fix docs and type hints for initialize_coverage.
* Update type hints on CoverageConfig.
* Split out logic for finding modules.
* Split out arc enumeration.
* Split out powershell coverage enumeration.
* Raise verbosity level of empty coverage warnings.
* Add code coverage target analysis to ansible-test.
2020-02-05 22:16:15 -08:00
4fd2dce7f3 Testing: Add support for AIX platform (#65802) 2020-01-29 15:56:51 -08:00
0971a0e7bc Consolidate remote Shippable scripts. 2020-01-28 17:48:54 -08:00
5c1fe78685 Move CI coverage stub to sanity/5. (#66827) 2020-01-27 16:19:49 -08:00
452ade007e Clean up Shippable scripts in tools directory. 2020-01-07 14:38:15 -08:00
78e476eb20 Increase unit test timeout in CI. 2019-11-01 17:16:07 -07:00
d2562cde1c Temporarily increase CI timeout. 2019-11-01 10:10:02 -07:00
ada02f1966 Limit Shippable matrix check to ansible repo. 2019-10-01 09:53:19 -07:00
cc758ea574 Revert "shippable: temporary bump the timeout for Azure jobs"
The Azure timeout should not accure anymore since
173d47d1f4.

This reverts commit aaa8835311.
2019-09-09 18:17:25 -07:00
c9e7a2122d Increase unit test timeout to account for long image download times (#61717) 2019-09-03 12:14:55 -04:00
aaa8835311 shippable: temporary bump the timeout for Azure jobs 2019-08-30 18:47:51 -04:00
e4e5005640 Fix ansible-test coverage path handling. (#61528)
* Fix ansible-test coverage path handling.
* Split CI unit tests into two groups.
2019-08-30 00:18:49 -07:00
e5b24a68cc Add additional group for sanity tests
Put validate-modules in its own group
2019-08-29 00:21:07 -07:00
20d9c34862 Fix CI test result placement. 2019-08-28 09:58:00 -07:00
f5d829392a Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
2019-08-27 23:40:06 -07:00
faaa669764 PowerShell - Added coverage collector (#59009)
* Added coverage collection for PowerShell - ci_complete ci_coverage

* uncomment out coverage uploader call

* Generate XML for PowerShell coverage

* Use whitelist to exclude coverage run on non content plugins

* Remove uneeded ignore entry

* Try to reduce diff in cover.py

* Fix up coverage report package - ci_complete ci_coverage
2019-08-28 07:03:23 +10:00
4d424d0830 Only run package-data sanity once in CI. 2019-08-20 14:09:40 -07:00
3a3727d200 Remove Python 2.6 from Windows tests (#60235) 2019-08-08 07:32:09 +10:00
b446ca100f Prepare ansible-test code for relocation. (#60110)
* Fix imports in cli.py.

* Fix imports in executor.py.

* Remove old test/runner/ansible-test entry point.

Use the official bin/ansible-test entry point instead, which has been around since Ansible 2.5.

* Use bin/ansible-test on Shippable.
2019-08-05 14:56:38 -07:00
31c7a61f9d Clean up old ansible-test references and code. (#60103)
* Remove obsolete ansible-test retry.py.
* Use ansible-test from bin/ intead of test/runner/.
2019-08-05 13:29:50 -07:00
d3da8e4a5b Replace old shippable.py with new check_matrix.py. (#60022)
This new script does not depend on ansible-test and provides much more robust job matrix testing.

It is also run on every job in the matrix now, to detect issues with jobs being re-run after matrix changes are made.
2019-08-05 11:27:37 -07:00
f944bd2358 Update ansible-test to properly skip unit tests.
Unit tests will no longer run on "remote only" Python versions (2.6) for tests which are not "remote" (modules and module_utils).
2019-07-31 00:27:07 -07:00
95f4282c42 Verify package data in setup.py installs all files (#59537)
* Add sanity test to ensure all non-py files are installed

* Fix mode and regex

* Fix role skel inventory package_data

* Add docs

* Update package_data for inventory files

* Address pylint concerns

* Another tweak to package_data

* Address review feedback

* Change index to 1

* add to ansible-only.txt
2019-07-24 13:58:13 -05:00
13bbfacb04 Show ansible-test --explain errors in CI. 2019-07-17 15:34:00 -07:00
3f2b766d10 Add future and metaclass boilerplate to test code.
Continue to ignore:

- test/integration/
- test/legacy/
- test/units/
2019-07-12 09:18:24 -07:00
79685a4a2d Use Python 3.7 to run ansible-test coverage in CI. (#57555)
Generating XML for coverage reports on Python 3.7 uses far less
memory than on Python 2.7. This change should avoid the MemoryError
we've been seeing in the nightly coverage runs.
2019-06-07 15:48:17 -07:00
b1e6ad9754 Limit Python 2.6 units to modules & module_utils. (#57377) 2019-06-04 15:15:36 -07:00
85ae8f5258 Add Hetzner Cloud to CI integration tests. 2019-04-09 15:41:50 -07:00
c846501829 Increase unit test timeout for --coverage. 2019-04-05 09:43:42 -07:00
2a8252ceb9 Slightly increase non-coverage unit test timeout. 2019-03-21 09:30:43 -07:00
8fa294a33a Fix ansible-test coverage stub memory usage. (#53652)
Split coverage stub files into multiple parts based on the number
of lines in files included in each stub. This will keep memory usage
down to avoid hanging on Shippable.
2019-03-11 15:14:09 -07:00
887ab35656 Add support for ansible-test --coverage-check. (#53573) 2019-03-11 15:13:09 -07:00
9546761407 Fix coverage stub generation in CI. 2019-03-08 16:29:14 -08:00
a8d829d9c3 Add timeout support to ansible-test. (#53302)
* Add timeout support to ansible-test.
* Fix ansible-test tar filename filter bug.
* Update timeouts used on Shippable.
* Kill subprocesses when parent process terminates.
* Require explicit use of env --show option.
2019-03-05 11:58:13 -08:00
b2f354ce29 Ignore errors during container cleanup in CI. 2019-02-21 21:20:48 -08:00
a1fec259c1 Reduce console noise at start of CI session. (#52387)
* Reduce console noise at start of CI session.
* Use allowed shebang.
2019-02-18 13:41:45 -08:00
6a82827ddd Add Python 3.8 to ansible-test and CI. 2019-02-14 23:24:24 -08:00
d2a9b16f7e Make Shippable timing script more resilient.
This will permit the script to pass through content which could
not be previously decoded or encoded. This could occur when running
some tests on macOS using file paths with non-ASCII characters.
2019-01-17 00:03:38 -08:00
721a28202e Revert Debug: Size of coverage file #50380 (#50945) 2019-01-15 20:36:12 +00:00
31c1bdd6a8 Fix run.py --branch argument.
The API docs state that both `branch` and `branchName` are valid,
but only `branchName` appears to work.
2019-01-07 13:56:35 -08:00
e6ced9c73f Debug: Size of coverage file (#50380) 2018-12-29 13:01:35 +00:00
7ec82cdfc9 Revert TEMP HACK: Upload all to codecov (#50279)
Codecov.io have made some improvements to results processing.
Upload all the coverage data to see is codecov can process this.

Reverts https://github.com/ansible/ansible/pull/47515
2018-12-23 09:42:29 +00:00
01833b6fb1 Add env command to ansible-test and run in CI. (#50176)
* Add `env` command to ansible-test and run in CI.
* Avoid unnecessary docker pull.
2018-12-20 22:08:57 -08:00
9979a32e5c Fix passing of env vars to Shippable. 2018-11-14 10:50:31 -08:00
fb2e838a42 TEMP HACK: Avoid overloading codecov.io (#47515)
Without this patch we are overloading codecov.io by uploading 90+ tests.

As a workaround limit uploading to only "Group 1"

Will be removed/updated based on codecov.io's support team
2018-10-23 17:32:34 +01:00
6a51b5ed26 Use 4 nodes for sanity tests in CI. 2018-10-16 09:55:26 -07:00
2761fe8272 Revert "Use 4 nodes for sanity tests in CI."
This reverts commit b887333029.
2018-10-15 12:11:17 -07:00
b887333029 Use 4 nodes for sanity tests in CI. 2018-10-15 12:03:43 -07:00