Hello,
I found an issue when running uClibc test "regex/testregex". I launched tests using "uclibcng-testrunner.sh" script. And the script printed that "testregex" had been passed successfully. But when I opened "testregex.out" file I found, that many "testregex" sub-tests failed. This behavior is caused because test returns 0 exit code even when sub- tests fail.
As a result we don't know was the "testregex" execution successful or not though it is marked as "PASSED".
The problem was found when running tests on ARC HS38 with Linux 4.6.1. It also was reproduced on ARMv5 with Linux 4.5.3.
Hi Vlad, Vlad Zakharov wrote,
Hello,
I found an issue when running uClibc test "regex/testregex". I launched tests using "uclibcng-testrunner.sh" script. And the script printed that "testregex" had been passed successfully. But when I opened "testregex.out" file I found, that many "testregex" sub-tests failed. This behavior is caused because test returns 0 exit code even when sub- tests fail.
As a result we don't know was the "testregex" execution successful or not though it is marked as "PASSED".
The problem was found when running tests on ARC HS38 with Linux 4.6.1. It also was reproduced on ARMv5 with Linux 4.5.3.
Thanks for the report. Do you have any preferred way how we should fix the issue? Should we fail if any sub-test fail?
best regards Waldemar
Hi Waldemar,
On Thu, 2016-06-09 at 23:22 +0200, Waldemar Brodkorb wrote:
Hi Vlad, Vlad Zakharov wrote,
Hello,
I found an issue when running uClibc test "regex/testregex". I launched tests using "uclibcng-testrunner.sh" script. And the script printed that "testregex" had been passed successfully. But when I opened "testregex.out" file I found, that many "testregex" sub-tests failed. This behavior is caused because test returns 0 exit code even when sub- tests fail.
As a result we don't know was the "testregex" execution successful or not though it is marked as "PASSED".
The problem was found when running tests on ARC HS38 with Linux 4.6.1. It also was reproduced on ARMv5 with Linux 4.5.3.
Thanks for the report. Do you have any preferred way how we should fix the issue? Should we fail if any sub-test fail?
Let me answer on behalf of Vlad while he's off today.
Indeed I think it worth returning <> 0 value on any subtest failure. Otherwise IMHO results will be a bit confusing.
-Alexey
Hi Alexey, Alexey Brodkin wrote,
Hi Waldemar,
On Thu, 2016-06-09 at 23:22 +0200, Waldemar Brodkorb wrote:
Hi Vlad, Vlad Zakharov wrote,
Hello,
I found an issue when running uClibc test "regex/testregex". I launched tests using "uclibcng-testrunner.sh" script. And the script printed that "testregex" had been passed successfully. But when I opened "testregex.out" file I found, that many "testregex" sub-tests failed. This behavior is caused because test returns 0 exit code even when sub- tests fail.
As a result we don't know was the "testregex" execution successful or not though it is marked as "PASSED".
The problem was found when running tests on ARC HS38 with Linux 4.6.1. It also was reproduced on ARMv5 with Linux 4.5.3.
Thanks for the report. Do you have any preferred way how we should fix the issue? Should we fail if any sub-test fail?
Let me answer on behalf of Vlad while he's off today.
Indeed I think it worth returning <> 0 value on any subtest failure. Otherwise IMHO results will be a bit confusing.
Do you have a patch suggestion for this?
best regards Waldemar
Hi Waldemar,
On Wed, 2016-06-15 at 20:37 +0200, Waldemar Brodkorb wrote:
Hi Alexey, Alexey Brodkin wrote,
Hi Waldemar,
On Thu, 2016-06-09 at 23:22 +0200, Waldemar Brodkorb wrote:
Hi Vlad, Vlad Zakharov wrote,
Hello,
I found an issue when running uClibc test "regex/testregex". I launched tests using "uclibcng-testrunner.sh" script. And the script printed that "testregex" had been passed successfully. But when I opened "testregex.out" file I found, that many "testregex" sub-tests failed. This behavior is caused because test returns 0 exit code even when sub- tests fail.
As a result we don't know was the "testregex" execution successful or not though it is marked as "PASSED".
The problem was found when running tests on ARC HS38 with Linux 4.6.1. It also was reproduced on ARMv5 with Linux 4.5.3.
Thanks for the report. Do you have any preferred way how we should fix the issue? Should we fail if any sub-test fail?
Let me answer on behalf of Vlad while he's off today.
Indeed I think it worth returning <> 0 value on any subtest failure. Otherwise IMHO results will be a bit confusing.
Do you have a patch suggestion for this?
Vlad is already looking at this. Stay tuned and patches will arrive soon I hope :)
-Alexey
-
Alexey Brodkin -
Vlad Zakharov -
Waldemar Brodkorb