I have recently released the Perl distribution Test::Timer 2.09, the last release I blogged about was 2.00 – a lot has happened in regard to stabilisation. Attempts at making some minor improvements resulted in tests failing and a long road to get things stable again.
2.09 is a culmination of a lot of releases aiming at getting stability for the tests run by CPAN-testers. I think I have succeeded as you can read from the test reports, with 361 passes and 1 fail (at the time of writing)
So lets revisit the changes and releases:
2.01 2017-06-12 Bug fix release, update recommended - Fixed bug where execution/time would be reported as 0 (#13)
A bug introduced in 2.00, this happens, see issue #13
2.02 2017-06-30 Maintenance release, update recommended - Correction to documentation - Improvements to alarm signal handling and other internal parts - Addressed issue #15 meaning thresholds are now included in the assertions
Improvements to the test assertions, documentation and signal handling, see issue 15. This was based on a bug report from a user, so I was most happy to fix this. I do not think my distribution has many users, so I have to cater to the ones providing me with feedback and using my small open source contribution.
2.03 2017-07-01 Maintenance release, update not required - Minor clean up in code and tests
minor clean up to code, removed a lot of the Perl versions from the Travis integration, it seems a bit overkill with so much testing and it takes a lot of time, so I decided on only: 5.10, 5.20, 5.22. and 5.24 – next step will be to exchange 5.22 and 5.25 for 5.26
2.04 2017-10-15 Maintenance release, update not required - Minor improvements to Test::Timer::TimeoutException, some obsoleted code could be removed - Generalising test asserting, since CPAN testers are sometime constrained on resources, making it impossible to predict the actual timeout value Example: http://www.cpantesters.org/cpan/report/2561e32c-9efa-11e7-bc90-bbe42ddde1fb - Correction of spelling mistake in PR #16 from Gregor Herrmann
Removed some more code, which was of no use to the actual implementation. I sometimes observe some failing tests with CPAN tests, which I suspect are due to high loads on the smoker machines, since I am not always able to reproduce the fails. I received a PR from a Debian maintainer, see issue #16. I can only say that I am happy to support other open source contributors putting in the effort and taking the time to distribute my work.
2.05 2017-11-12 Maintenance release, update not required - Addressed issue #11 adding experimental graphical support elements to the documentation
Added some graphical assistance, I have for long pondered about this. You can see it in the documentation as ASCII or on the homepage for the distribution as actual images.
2.06 2017-11-14 Maintenance release, update not required - Added cancellation of alarm, based on advice from Erik Johansen - Implemented own sleep, based on select, this might address possible issues with sleep implementations
Still boxing the issue with constrained environments I mailed my local Perl user group and I talked to one of my colleagues about some of the issues I was observing. Apparently it is not easy to identify whether a system is under heavy load. My colleague did advice me to handle the alarm more appropriately, it sounded reasonable, it did not fix the issue, but it did feel more right to add this code. At the same time I implemented a my own sleep method, so I could easily exchange the implementation if need arose. Somebody hinted to me that the sleep function could be problematic on some operating systems, so I exchanged it for select.
2.07 2017-11-18 Maintenance release, update not required - Addressing issue #17, the tests are now more liberal, so when executed on smokers, CI environments and similar, load will not influence the test results. The requirement for Test::Tester has been updated and a patch required by this distribution has been included
Out of desperation I decided to make the tests more liberal and yes it did give me more passes with CPAN-testers. This change did not feel right, but I know I could correct it again, but I needed to see the feedback from CPAN-testers, but I knew I was treating the diagnosis not the root cause of the problem, see issue #17. I am using Test::Tester and older but really nice module. In order to implement the changes I required I pushed a patch upstream and it got accepted, so at least I had some nice syntactic sugar for implementing the more liberal test assertions.
2.08 2017-11-20 Maintenance release, update not required - Addressing reports on failing tests from CPAN testers
This release was even more steps in the wrong direction, ignoring the timeout test assertions by treating them as normal tests failures even though the situation would not be the same. When you implement unit-tests and you have the opportunity to be strict and make tight and correction assertions, do so. Never the less more passes.
2.09 2017-11-24 Maintenance release, update not required - Attempting to address issues with tests on Windows REF: http://www.cpantesters.org/distro/T/Test-Timer.html?grade=3&perlmat=2&patches=2&oncpan=2&distmat=2&perlver=ALL&osname=ALL&version=2.08 - Reinstated sleep over select in the test suite - Changed some test parameters was made a bit less relaxed attempting to decrease the execution time for the test suite - Removed loose match in regular, it should be possible to anticipate the timeout - Removed redundant tests, trying to cut down execution time for the test suite
With release 2.09 I decided to make a real effort to kick the test suite back into shape. With focus and effort I was able to pull it through and 2.09 passes almost all tests but one. I exchanged select for sleep and it proved to be a good decision.
So now I am stuck with this test failure report (excerpt):
Output from 'C:\Strawberry240\perl\bin\perl.exe ./Build test': t/00-compile.t ............ ok # Failed test at t/_benchmark.t line 21. # Looks like you failed 1 test of 3. t/_benchmark.t ............ Dubious, test returned 1 (wstat 256, 0x100) Failed 1/3 subtests t/author-critic.t ......... skipped: these tests are for testing by the author t/author-pod-coverage.t ... skipped: these tests are for testing by the author t/author-pod-syntax.t ..... skipped: these tests are for testing by the author t/release-cpan-changes.t .. skipped: these tests are for release candidate testing t/release-kwalitee.t ...... skipped: these tests are for release candidate testing t/release-meta-json.t ..... skipped: these tests are for release candidate testing # Failed test 'subtest 'time_between, failing test' of 'Failing test of time_between' compare ok' # at t/test-tester.t line 54. # got: '1' # expected: '0' # Failed test 'subtest 'time_between, failing test' of 'Failing test of time_between' compare diag' # at t/test-tester.t line 54. # '' # doesn't match '(?^:Test ran \d+ seconds and did not execute within specified interval 1 - 2 seconds)' # Looks like you failed 2 tests of 77. t/test-tester.t ........... Dubious, test returned 2 (wstat 512, 0x200) Failed 2/77 subtests t/time_alert.t ............ ok Test Summary Report t/_benchmark.t (Wstat: 256 Tests: 3 Failed: 1) Failed test: 2 Non-zero exit status: 1 t/test-tester.t (Wstat: 512 Tests: 77 Failed: 2) Failed tests: 39, 42 Non-zero exit status: 2 Files=10, Tests=84, 31 wallclock secs ( 0.07 usr + 0.13 sys = 0.20 CPU) Result: FAIL Failed 2/10 test programs. 3/84 subtests failed. In the context of all of the other reports succeeding it does not make much sense and it fails in a place I have not observed a failure in before - perhaps a bad smoker, any how I need to investigate.
Until next timely release – take care