Reclassify failures of tests listed in flaky_tests_to_ignore as "ignored"
instead of retrying them. Ignored tests still run and their output is
reported, but they don't affect the exit code. The ci.sh --exclude lists
for these tests are removed so they run normally.
Signed-off-by: Andrew Leech <andrew.leech@planet-innovation.com>
Includes corresponding .exp files because this feature is only available in
Python 3.14+.
Tests for `!a` conversion specifier and space after `!` are not included
because they are not supported by MicroPython.
Signed-off-by: Koudai Aono <koxudaxi@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support for t-strings by leveraging the existing f-string
parser in the lexer. It includes:
- t-string parsing in `py/lexer.c`
- new built-in `__template__()` function to construct t-string objects
- new built-in `Template` and `Interpolation` classes which implement all
the functionality from PEP 750
- new built-in `string` module with `templatelib` sub-module, which
contains the classes `Template` and `Interpolation`
The way the t-string parser works is that an input t-string like:
t"hello {name:5}"
is converted character-by-character by the lexer/tokenizer to:
__template__(("hello ", "",), name, "name", None, "5")
For reference, if it were an f-string it would be converted to:
"hello {:5}".format(name)
Some properties of this implementation:
- it's enabled by default at the full feature level,
MICROPY_CONFIG_ROM_LEVEL_AT_LEAST_FULL_FEATURES
- when enabled on a Cortex-M bare-metal port it costs about +3000 bytes
- there are no limits on the size or complexity of t-strings, and it allows
arbitrary levels of nesting of f-strings and t-strings (up to the memory
available to the compiler)
- the 'a' (ascii) conversion specifier is not supported (MicroPython does
not have the built-in `ascii` function)
- space after conversion specifier, eg t"{x!r :10}", is not supported
- arguments to `__template__` and `Interpolation` are not fully validated
(it's not necessary, it won't crash if the wrong arguments are passed in)
Otherwise the implementation here matches CPython.
Signed-off-by: Damien George <damien@micropython.org>
Because it requires a different configuration of the pins (in `setUp`).
Eg on mimxrt pins used for a `machine.Counter` cannot be read.
Signed-off-by: Damien George <damien@micropython.org>
Because it requires a different configuration of the pins (in `setUp`).
Eg on mimxrt pins used for an Encoder cannot be read.
Signed-off-by: Damien George <damien@micropython.org>
As per CPython behaviour, `b"".hex()` should return an empty str object
(not an empty bytes object).
Fixes issue #18807.
Signed-off-by: Damien George <damien@micropython.org>
As of CPython 3.7 `bytes.fromhex()` skips all ASCII whitespace. And as of
CPython 3.8 `bytes.hex()` supports the optional separator argument. So
this test no longer needs a .exp file.
Signed-off-by: Damien George <damien@micropython.org>
Fixes rp2 issue where socket.getaddrinfo() could block indefinitely if an
interface goes down and still has a DNS server configured, as the LWIP
timer stops running and can't time out the DNS query.
Adds a regression test under multi_wlan that times out on rp2 without this
fix.
Fixes issue #18797.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Occasionally, it's useful to be able to compare MicroPython's performance
figures to CPython's. This change adds the ability to run the internalbench
test runner with `--test-instance=cpython` in order to execute the same
test routines against CPython and produce a benchmark performance report in
the same format as MicroPython.
Signed-off-by: Anson Mansfield <amansfield@mantaro.com>
Previously, mpycert.der was the Intermediate certificate which is regularly
re-issued by Letsencrypt.
Also changes ssl_cert.py to load the cert data from the same file as
test_sslcontext_client.py, so the DER string doesn't have to be pasted
into the source.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Fixes fatal crash if serial port access returns an error (for example: port
is native USB-CDC and the host hard faults during the test run). Instead of
crashing, have the runner mark this as a test run error and continue.
It's not certain the next test will run successfully, but this provides the
context of output showing what was happening when the communication error
occurred. Without this change, that output is lost when the fatal exception
terminates the runner process.
This work was funded through GitHub Sponsors.
Signed-off-by: Angus Gratton <angus@redyak.com.au>
Test file paths which get passed to the run_tests function can be
absolute or relative and with or without leading slash in the latter
case, depending on the arguments to run-tests.py, but since that path
is used to:
- display which tests run
- record which tests ran in the results.json
- craft the filename for the .exp/.out file for failed tests
it is desirable to always use the same file path irregardless of
how the user passed the path.
In practice this means that all forms of running our own tests like:
>python ./run-tests.py -i extmod
>python ./run-tests.py -d extmod
>python ./run-tests.py -d ./extmod
>python ./run-tests.py -d ../tests/extmod
>python ./run-tests.py -d /full/path/to/tests/extmod
will now consistently all display the tests like
pass extmod/time_time_ns.py
FAIL extmod/some_failing_test.py
and produce output files like
results/extmod_some_failing_test.py.exp
results/extmod_some_failing_test.py.out
instead of displaying/using the exact path as passed.
For external tests, meaning not in the tests/ directory, we also want
to be consistent so there the choice was made to always use absolute
paths.
Signed-off-by: stijn <stijn@ignitron.net>
Scan the --test-dirs argument for the main tests directory being
passed and if so do the same thing as if running from within that
main test directory.
In practice this makes the following (which used to counterintuitively
try and fail to run the .py files in the tests/ directory itself)
>python micropython/tests/run-tests.py -d micropython/tests
do the same thing as
>cd micropython/tests
>python ./run-tests.py
which is logical and convenient.
Signed-off-by: stijn <stijn@ignitron.net>
Test file paths which get passed to the run_tests function can be
absolute or relative and with or without leading slash in the latter
case, depending on the arguments to run-tests.py, but the skip_tests
list with tests to skip only contains relative paths so using simple
string equality comparison easily leads to false negatives.
Compare the full absolute path instead such that it doesn't matter
anymore in which form the tests are passed. Note:
- use realpath to resolve symlinks plus make the comparison case
insensitive on windows
- the test_file passed to run_one_test is not altered by this commit,
such that when the user inputs relative paths the tests are also
still displayed with relative paths
- likewise the test_file_abspath is not modified because functions
like run_micropython rely on it having forward slashes
In practice this means that it used to be so that the only forms
of running tests for which the skip_tests lists actually worked were:
>python ./run-tests.py
>python ./run-tests.py -d extmod
whereas it now works consistently so also for these invocations,
which in the end all point to the exact same path:
>python ./run-tests.py -d ./extmod
>python ./run-tests.py -d ../tests/extmod
>python ./run-tests.py -d /full/path/to/tests/extmod
These examples used to not skip any of the tests in the extmod/
directory thereby leading to test failures.
Signed-off-by: stijn <stijn@ignitron.net>
This is convenient when trying to figure out the correct values
for --include/--exclude/--test-dirs/... arguments.
Signed-off-by: stijn <stijn@ignitron.net>
This commit introduces a test that should check whether viper load or
store operations won't clobber either the buffer address or the index
value being used.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit introduces support for writing inline assembler code
snippets when targeting Xtensa cores that use register windows (eg.
the whole ESP32 family).
Opcodes support is still limited to what the ESP8266 supports (as in,
LX3 cores opcodes), however each LX core version is guaranteed to
support all previous versions' opcodes as well. The ESP32 does not have
the inline assembler enabled by default, following the existing
expectations when it comes to firmware footprint.
Since now emitted functions may have one of two possible exit sequences,
the L32I test had to be fixed. It would return the word containing the
L32I opcode itself, but the upper 8 bits of the word came from the
following opcode - which can change depending on the exit code sequence.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This is a follow up to 6436f8b93f that
catches more cases of a failed raw REPL.
If the target device is broken in a certain way then it can have a serial
write error instead of just not returning any data. In that case the error
raised by `pyboard.py` is "could not enter raw repl: Write timeout", which
is slightly different to "could not enter raw repl" (the latter is raised
when a serial read fails to return the correct data).
The patch here accounts for all cases of a failed raw REPL by using
`str.startswith()` instead of a string equality.
This can be tested on RPI_PICO_W and RPI_PICO2_W which currently crash in
the specific way needed to trigger the write timeout when running the
`tests/extmod/socket_badconstructor.py` test. In particular adding the
following code to the end of a test (eg `tests/extmod/random_extra.py`)
will trigger the issue:
import socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, None)
except TypeError:
pass
With the fix here, the `run-tests.py` and `run-natmodtests.py` will abort
early if that code is added (prior to the fix they would continue to run
all tests and take a long time).
Signed-off-by: Damien George <damien@micropython.org>
Prior to this fix, `mp_import_all()` assumed that its argument was exactly
a native module instance. That would lead to a crash if something else was
passed in, eg a user class via a custom `__import__` implementation or by
writing to `sys.modules`.
MicroPython already supports injecting non-module objects into the import
machinery, so it makes sense to round out that implementation by supporting
`from x import *` where `x` is a non-module object.
Fixes issue #18639.
Signed-off-by: Damien George <damien@micropython.org>
It turns out that it's relatively simple to support nested f-strings, which
is what this commit implements.
The way the MicroPython f-string parser works at the moment is:
1. it extracts the f-string arguments (things in curly braces) into a
temporary buffer (a vstr)
2. once the f-string ends (reaches its closing quote) the lexer switches to
tokenizing the temporary buffer
3. once the buffer is empty it switches back to the stream.
The temporary buffer can easily hold f-strings itself (ie nested f-strings)
and they can be re-parsed by the lexer using the same algorithm. The only
thing stopping that from working is that the temporary buffer can't be
reused for the nested f-string because it's currently being parsed.
This commit fixes that by adding a second temporary buffer, which is the
"injection" buffer. That allows arbitrary number of nestings with a simple
modification to the original algorithm:
1. when an f-string is encountered the string is parsed and its arguments
are extracted into `fstring_args`
2. when the f-string finishes, `fstring_args` is inserted into the current
position in `inject_chrs` (which is the start of that buffer if no
injection is ongoing)
3. `fstring_args` is now cleared and ready for any further f-strings
(nested or not)
4. the lexer switches to `inject_chrs` if it's not already reading from it
5. if an f-string appeared inside the f-string then it is in `inject_chrs`
and can be processed as before, extracting its arguments into
`fstring_args`, which can then be inserted again into `inject_chrs`
6. once `inject_chrs` is exhausted (meaning that all levels of f-strings
have been fully processed) the lexer switched back to tokenizing the
stream.
Amazingly, this scheme supports arbitrary numbers of nestings of f-strings
using the same quote style.
This adds some code size and a bit more memory usage for the lexer. In
particular for a single (non-nested) f-string it now makes an extra copy of
the `fstring_args` data, when copying it across to `inject_chrs`.
Otherwise, memory use only goes up with the complexity of nested f-strings.
Signed-off-by: Damien George <damien@micropython.org>
The null byte cannot exist in source code (per CPython), so use it to
indicate the end of the input stream (instead of `(mp_uint_t)-1`). This
allows the cache chars (chr0/1/2 and their saved versions) to be 8-bit
bytes, making it clear that they are not `unichar` values. It also saves a
bit of memory in the `mp_lexer_t` data structure. (And in a future commit
allows the saved cache chars to be eliminated entirely by storing them in
a vstr instead.)
In order to keep code size down, the frequently used `chr0` is still of
type `uint32_t`. Having it 32-bit means that machine instructions to load
it are smaller (it adds about +80 bytes to Thumb code if `chr0` is changed
to `uint8_t`).
Also add tests for invalid bytes in the input stream to make sure there are
no regressions in this regard.
Signed-off-by: Damien George <damien@micropython.org>
Prior to this change, if a test crashed a board to the point that it was no
longer responsive but the serial connection was still alive and able to
read/write (eg due to an infinite loop with ctrl-C unavailable), it would
take (30 + 10 * number-of-remaining-tests) seconds to finish the test run.
That can be 10 minutes or more.
The change here aborts the test run if the target does not respond 4 times
to when trying to enter the raw REPL at the start of a test. Now, it only
takes a maximum of (30 + 4 * 5) = 50 seconds to detect a crashed state and
abort the entire test run.
Can test this by adding the following to a test:
import time, micropython
micropython.kbd_intr(-1)
while 1:
time.sleep(1)
For example, add the above to the end of `tests/extmod/random_extra.py` and
test via:
$ ./run-tests.py -t a0 extmod/[r-z]*.py
$ ./run-natmodtests.py -t a0 extmod/[r-z]*.py
Previously the `run-tests.py` run would take around 16 minutes. Now it
takes under a minute.
Signed-off-by: Damien George <damien@micropython.org>
All test runners (except `run-multitests.py`) will now abort with error
code 2 if the target is a serial port and could not be detected, ie
`enter_raw_repl` failed at the detection stage.
Signed-off-by: Damien George <damien@micropython.org>
The tests runners usually exit with error code 1 when something fails. But
it's useful to distinguish between "a test failed" and "the test runner
failed".
To that end, `run-tests.py` now returns error code 2 when unittest is not
installed but is needed. And `run-natmodtests.py` now returns error code 2
when it cannot automatically detect the target architecture.
Signed-off-by: Damien George <damien@micropython.org>
This commit converts `tests/extmod/machine_spi_rate.py` to use the target
wiring mechanism, moving the port-specefic definition of the SPI instances
that are being tested out of the `tests/extmod/machine_spi_rate.py` file
and into the target wiring code. As part of this, the test is changed to
always use the default pins for the given SPI instance.
This simplifies the test, makes it easier to add support for new
ports/boards, and allows reusing the `spi_standalone_args_list` definition
in other tests (if needed one day).
Signed-off-by: Damien George <damien@micropython.org>
The `getrandbits()` function is guaranteed to exist on all ports that have
the `random` module (but `randrange()` is not).
Signed-off-by: Damien George <damien@micropython.org>
The axTLS implementation of the tls module only has a basic set of
features. In particular it doesn't support the CERT_REQUIRED constant nor
DTLS, nor can it load the `ec_key.der` key when acting as a server. So
skip tests that require these features, which ends up being all the ssl/tls
tests.
Signed-off-by: Damien George <damien@micropython.org>
The test runners have evolved over time and become more and more complex.
In particular `tests/run-tests.py` is rather large now. The test runners
also duplicate some functionality amongst themselves.
As a start to improving this situation, this commit factors out the helper
functions from `run-tests.py` into a new `test_utils.py` file, and uses
that new module in all test runners.
There should be no functional change here.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds support to the `marshal` module to be able to dump
functions that have child functions. For example:
import marshal
def f():
def child():
return 1
return child
marshal.dumps(f.__code__)
It also covers the case of marshalling functions that use list
comprehensions, because a list comprehension uses a child function.
This is made possible by the newly enhanced
`mp_raw_code_save_fun_to_bytes()` that can now handle nested functions.
Unmarshalling via `marshal.loads()` already supports nested functions
because it uses the standard `mp_raw_code_load_mem()` function which is
used to import mpy files (and hence can handle all possibilities).
Signed-off-by: Damien George <damien@micropython.org>
If a function has children then the code object returned from __code__ must
contain an `mp_raw_code_t` (actually `mp_raw_code_truncated_t` is enough)
that points to the child table.
Signed-off-by: Damien George <damien@micropython.org>
These changes test a few more things related to TCP RST:
- add a second iteration to drain incoming data after TCP RST
- read data after closing socket
Signed-off-by: Damien George <damien@micropython.org>
This test can now run correctly with the native emitter because the
`thread()` function will run the scheduler during the call to
`time.sleep_ms(1)`.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This variant of `schedule.py` explicitly calls a zero sleep. The existing
variant is kept to ensure the scheduler is called between bytecodes.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This test was ignored since 2020, which hid a new bug in the test for the
native emitter added in 2024. Hopefully other changes to the test will
make it more reliable.
Signed-off-by: Jeff Epler <jepler@unpythonic.net>
This commit fixes a test exercising complex numbers creation, working
around a deprecation introduced in CPython 3.14.
Creating complex numbers using a complex number as the real part in the
constructor is deprecated in CPython 3.14, but that construction method
is still supported by MicroPython and covered by tests. To work around
this, the specific constructor is extracted into its own test, providing
an expected output file recorded using CPython 3.11.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit makes tests exercising certain math functions' limit work
when using CPython 3.14 to validate the tests' output.
CPython 3.14 introduced more descriptive messages when math domain error
conditions are encountered rather than a single generic message. This
breaks the tests in question as MicroPython uses a single error message
when reporting these conditions (both to closely follow CPython and to
save firmware space).
The math domain tests now look for an error pattern that is compatible
with both CPython 3.14 and previous versions, converting messages in the
newer format into the previous one. This makes the tests' behaviour
under MicroPython comparable with CPython for the foreseeable future.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit fixes tests that will always fail when using CPython 3.14
to get a known-good output to compare MicroPython's behaviour against.
Starting from CPython 3.14, statements inside a `finally` block that
will prevent the execution flow to reach the block's last statement will
raise a SyntaxWarning. That text would end up in the comparison data
and thus make known-good tests fail.
Given that those tests explicitly exercise flow control interruptions
in finally blocks, there is no real workaround that can be applied to
the tests themselves. Therefore those tests will now check
MicroPython's behaviour against expected output files recorded from
the tests' output with CPython 3.11.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This commit adds the ability to compile native modules for the RV64
platform, using "rv64imc" as its architecture name (eg.
"make ARCH=rv64imc" should build a RV64 natmod).
The rest of 64-bits relocations needed to build a native module are now
implemented, and all sample native modules build without errors or
warnings. The same Picolibc caveats on RV32 also apply on RV64, thus
the documentation was updated accordingly.
RV64 native modules are also built as part of the CI process, but not
yet executed as the QEMU port is not yet able to load and run them.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
This change lets the natmod test runner report status information on
session end if a prelude script file is chosen.
The script serialises its input data as part of the end of session
reporting data, but since the prelude file is not a serialisable object
serialisation would fail (it's a file handle as far as the argument
container structure is aware).
Now the file is explicitly open by the script rather than relying on
argparse's file handle argument class wrapper.
Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
