Reclassify failures of tests listed in flaky_tests_to_ignore as "ignored"
instead of retrying them. Ignored tests still run and their output is
reported, but they don't affect the exit code. The ci.sh --exclude lists
for these tests are removed so they run normally.
Signed-off-by: Andrew Leech <andrew.leech@planet-innovation.com>
This is convenient when trying to figure out the correct values
for --include/--exclude/--test-dirs/... arguments.
Signed-off-by: stijn <stijn@ignitron.net>
Prior to this change, if a test crashed a board to the point that it was no
longer responsive but the serial connection was still alive and able to
read/write (eg due to an infinite loop with ctrl-C unavailable), it would
take (30 + 10 * number-of-remaining-tests) seconds to finish the test run.
That can be 10 minutes or more.
The change here aborts the test run if the target does not respond 4 times
to when trying to enter the raw REPL at the start of a test. Now, it only
takes a maximum of (30 + 4 * 5) = 50 seconds to detect a crashed state and
abort the entire test run.
Can test this by adding the following to a test:
import time, micropython
micropython.kbd_intr(-1)
while 1:
time.sleep(1)
For example, add the above to the end of `tests/extmod/random_extra.py` and
test via:
$ ./run-tests.py -t a0 extmod/[r-z]*.py
$ ./run-natmodtests.py -t a0 extmod/[r-z]*.py
Previously the `run-tests.py` run would take around 16 minutes. Now it
takes under a minute.
Signed-off-by: Damien George <damien@micropython.org>
All test runners (except `run-multitests.py`) will now abort with error
code 2 if the target is a serial port and could not be detected, ie
`enter_raw_repl` failed at the detection stage.
Signed-off-by: Damien George <damien@micropython.org>
The test runners have evolved over time and become more and more complex.
In particular `tests/run-tests.py` is rather large now. The test runners
also duplicate some functionality amongst themselves.
As a start to improving this situation, this commit factors out the helper
functions from `run-tests.py` into a new `test_utils.py` file, and uses
that new module in all test runners.
There should be no functional change here.
Signed-off-by: Damien George <damien@micropython.org>
