411 lines
16 KiB
Plaintext
411 lines
16 KiB
Plaintext
|
LTP Test Writing Guidelines
|
|||
|
|
===========================
|
|||
|
|
|
|||
|
|
This document describes LTP guidelines and is intended for anybody who want to
|
|||
|
|
write or modify a LTP testcase. It's not a definitive guide and it's not, by
|
|||
|
|
any means, a substitute for common sense.
|
|||
|
|
|
|||
|
|
NOTE: See also
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API],
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API],
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/LTP-Library-API-Writing-Guidelines[LTP Library API Writing Guidelines].
|
|||
|
|
|
|||
|
|
Rules and recommendations which are "machine checkable" should be
|
|||
|
|
tagged with an ID like +LTP-XXX+. There will be a corresponding entry
|
|||
|
|
in
|
|||
|
|
https://github.com/linux-test-project/ltp/tree/master/doc/rules.tsv[doc/rules.tsv]. When
|
|||
|
|
you run 'make check' or 'make check-test' it will display these IDs as
|
|||
|
|
a reference.
|
|||
|
|
|
|||
|
|
1. Guide to clean and understandable code
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
For testcases it's required that the source code is as easy to follow as
|
|||
|
|
possible. When a test starts to fail the failure has to be analyzed, clean
|
|||
|
|
test codebase makes this task much easier and quicker.
|
|||
|
|
|
|||
|
|
Here are some hints on how to write clean and understandable code, a few of
|
|||
|
|
these points are further discussed below:
|
|||
|
|
|
|||
|
|
* First of all *Keep things simple*
|
|||
|
|
|
|||
|
|
* Keep function and variable names short but descriptive
|
|||
|
|
|
|||
|
|
* Keep functions reasonably short and focused on a single task
|
|||
|
|
|
|||
|
|
* Do not overcomment
|
|||
|
|
|
|||
|
|
* Be consistent
|
|||
|
|
|
|||
|
|
* Avoid deep nesting
|
|||
|
|
|
|||
|
|
* DRY
|
|||
|
|
|
|||
|
|
1.1 Keep things simple
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
For all it's worth keep the testcases simple or better as simple as possible.
|
|||
|
|
|
|||
|
|
The kernel and libc are tricky beasts and the complexity imposed by their
|
|||
|
|
interfaces is quite high. Concentrate on the interface you want to test and
|
|||
|
|
follow the UNIX philosophy.
|
|||
|
|
|
|||
|
|
It's a good idea to make the test as self-contained as possible too, ideally
|
|||
|
|
tests should not depend on tools or libraries that are not widely available.
|
|||
|
|
|
|||
|
|
Do not reinvent the wheel!
|
|||
|
|
|
|||
|
|
* Use LTP standard interface
|
|||
|
|
|
|||
|
|
* Do not add custom PASS/FAIL reporting functions
|
|||
|
|
|
|||
|
|
* Do not write Makefiles from scratch, use LTP build system instead
|
|||
|
|
|
|||
|
|
* Etc.
|
|||
|
|
|
|||
|
|
1.2 Keep functions and variable names short
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
Choosing a good name for an API functions or even variables is a difficult
|
|||
|
|
task do not underestimate it.
|
|||
|
|
|
|||
|
|
There are a couple of customary names for different things that help people to
|
|||
|
|
understand code, for example:
|
|||
|
|
|
|||
|
|
* For loop variables are usually named with a single letter 'i', 'j', ...
|
|||
|
|
|
|||
|
|
* File descriptors 'fd' or 'fd_foo'.
|
|||
|
|
|
|||
|
|
* Number of bytes stored in file are usually named as 'size' or 'len'
|
|||
|
|
|
|||
|
|
* Etc.
|
|||
|
|
|
|||
|
|
1.3 Do not overcomment
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
Comments can sometimes save you day but they can easily do more harm than
|
|||
|
|
good. There has been several cases where comments and actual implementation
|
|||
|
|
drifted slowly apart which yielded into API misuses and hard to find bugs.
|
|||
|
|
Remember there is only one thing worse than no documentation, wrong
|
|||
|
|
documentation.
|
|||
|
|
|
|||
|
|
Ideally everybody should write code that is obvious, which unfortunately isn't
|
|||
|
|
always possible. If there is a code that requires to be commented keep it
|
|||
|
|
short and to the point. These comments should explain *why* and not *how*
|
|||
|
|
things are done.
|
|||
|
|
|
|||
|
|
Never ever comment the obvious.
|
|||
|
|
|
|||
|
|
In case of LTP testcases it's customary to add an asciidoc formatted comment
|
|||
|
|
paragraph with highlevel test description at the beginning of the file right
|
|||
|
|
under the GPL SPDX header. This helps other people to understand the overall
|
|||
|
|
goal of the test before they dive into the technical details. It's also
|
|||
|
|
exported into generated documentation hence it should mostly explain what is
|
|||
|
|
tested.
|
|||
|
|
|
|||
|
|
1.4 DRY (Code duplication)
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
Copy & paste is a good servant but very poor master. If you are about to copy a
|
|||
|
|
large part of the code from one testcase to another, think what would happen if
|
|||
|
|
you find bug in the code that has been copied all around the tree. What about
|
|||
|
|
moving it to a library instead?
|
|||
|
|
|
|||
|
|
The same goes for short but complicated parts, whenever you are about to copy &
|
|||
|
|
paste a syscall wrapper that packs arguments accordingly to machine
|
|||
|
|
architecture or similarly complicated code, put it into a header instead.
|
|||
|
|
|
|||
|
|
2 Coding style
|
|||
|
|
~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
2.1 C coding style
|
|||
|
|
^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
LTP adopted Linux kernel coding style:
|
|||
|
|
https://www.kernel.org/doc/html/latest/process/coding-style.html
|
|||
|
|
|
|||
|
|
If you aren't familiar with its rules please read it, it's a well written
|
|||
|
|
introduction.
|
|||
|
|
|
|||
|
|
Run `make check` in the test's directory and/or use `make check-$TCID`,
|
|||
|
|
it uses (among other checks) our vendored version of
|
|||
|
|
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/scripts/checkpatch.pl[checkpatch.pl]
|
|||
|
|
script from kernel git tree.
|
|||
|
|
|
|||
|
|
NOTE: If `make check` does not report any problems, the code still may be wrong
|
|||
|
|
as all tools used for checking only look for common mistakes.
|
|||
|
|
|
|||
|
|
2.1.1 LTP-004: Test executable symbols are marked static
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
Test executables should not export symbols unnecessarily. This means
|
|||
|
|
that all top-level variables and functions should be marked with the
|
|||
|
|
static keyword. The only visible symbols should be those included from
|
|||
|
|
shared object files.
|
|||
|
|
|
|||
|
|
2.2 Shell coding style
|
|||
|
|
^^^^^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
When writing testcases in shell write in *portable shell* only, it's a good
|
|||
|
|
idea to try to run the test using alternative shell (alternative to bash, for
|
|||
|
|
example dash) too.
|
|||
|
|
|
|||
|
|
*Portable shell* means Shell Command Language as defined by POSIX with an
|
|||
|
|
exception of few widely used extensions, namely 'local' keyword used inside of
|
|||
|
|
functions and '-o' and '-a' test parameters (that are marked as obsolete in
|
|||
|
|
POSIX).
|
|||
|
|
|
|||
|
|
You can either try to run the testcases on Debian which has '/bin/sh' pointing
|
|||
|
|
to 'dash' by default or install 'dash' on your favorite distribution and use
|
|||
|
|
it to run the tests. If your distribution lacks 'dash' package you can always
|
|||
|
|
compile it from http://gondor.apana.org.au/~herbert/dash/files/[source].
|
|||
|
|
|
|||
|
|
Run `make check` in the test's directory and/or use `make check-$TCID.sh`,
|
|||
|
|
it uses (among other checks) our vendored version of
|
|||
|
|
https://salsa.debian.org/debian/devscripts/raw/master/scripts/checkbashisms.pl[checkbashism.pl]
|
|||
|
|
from Debian, that is used to check for non-portable shell code.
|
|||
|
|
|
|||
|
|
NOTE: If `make check` does not report any problems, the code still may be wrong
|
|||
|
|
as `checkbashisms.pl` used for checking only looks for common mistakes.
|
|||
|
|
|
|||
|
|
Here are some common sense style rules for shell
|
|||
|
|
|
|||
|
|
* Keep lines under 80 chars
|
|||
|
|
|
|||
|
|
* Use tabs for indentation
|
|||
|
|
|
|||
|
|
* Keep things simple, avoid unnecessary subshells
|
|||
|
|
|
|||
|
|
* Don't do confusing things (i.e. don't name your functions like common shell
|
|||
|
|
commands, etc.)
|
|||
|
|
|
|||
|
|
* Quote variables
|
|||
|
|
|
|||
|
|
* Be consistent
|
|||
|
|
|
|||
|
|
3 Backwards compatibility
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
LTP test should be as backward compatible as possible. Think of an enterprise
|
|||
|
|
distributions with long term support (more than five years since the initial
|
|||
|
|
release) or of an embedded platform that needs to use several years old
|
|||
|
|
toolchain supplied by the manufacturer.
|
|||
|
|
|
|||
|
|
Therefore LTP test for more current features should be able to cope with older
|
|||
|
|
systems. It should at least compile fine and if it's not appropriate for the
|
|||
|
|
configuration it should return 'TCONF'.
|
|||
|
|
|
|||
|
|
There are several types of checks we use:
|
|||
|
|
|
|||
|
|
The *configure script* is usually used to detect availability of a function
|
|||
|
|
declarations in system headers. It's used to disable tests at compile time or
|
|||
|
|
to enable fallback definitions.
|
|||
|
|
|
|||
|
|
Checking the *errno* value is another type of runtime check. Most of the
|
|||
|
|
syscalls returns either 'EINVAL' or 'ENOSYS' when syscall was not implemented
|
|||
|
|
or was disabled upon kernel compilation.
|
|||
|
|
|
|||
|
|
LTP has kernel version detection that can be used to disable tests at runtime,
|
|||
|
|
unfortunately kernel version does not always corresponds to a well defined
|
|||
|
|
feature set as distributions tend to backport hundreds of patches while the
|
|||
|
|
kernel version stays the same. Use with caution.
|
|||
|
|
|
|||
|
|
Lately we added kernel '.config' parser, a test can define a boolean
|
|||
|
|
expression of kernel config variables that has to be satisfied in order for a
|
|||
|
|
test to run. This is mostly used for kernel namespaces at the moment.
|
|||
|
|
|
|||
|
|
Sometimes it also makes sense to define a few macros instead of creating
|
|||
|
|
configure test. One example is Linux specific POSIX clock ids in
|
|||
|
|
'include/lapi/posix_clocks.h'.
|
|||
|
|
|
|||
|
|
3.1 Dealing with messed up legacy code
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
LTP still contains a lot of old and messy code and we are cleaning it up as
|
|||
|
|
fast as we can but despite the decade of efforts there is still a lot. If you
|
|||
|
|
start modifying old or a messy testcase and your changes are more complicated
|
|||
|
|
than simple typo fixes you should convert the test into a new library first.
|
|||
|
|
|
|||
|
|
It's also much easier to review the changes if you split them into a smaller
|
|||
|
|
logical groups. The same goes for moving files. If you need a rename or move
|
|||
|
|
file do it in a separate patch.
|
|||
|
|
|
|||
|
|
4 License
|
|||
|
|
~~~~~~~~~
|
|||
|
|
|
|||
|
|
Code contributed to LTP should be licensed under GPLv2+ (GNU GPL version 2 or
|
|||
|
|
any later version).
|
|||
|
|
|
|||
|
|
Use `SPDX-License-Identifier: GPL-2.0-or-later`
|
|||
|
|
|
|||
|
|
5 LTP Structure
|
|||
|
|
~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
The structure of LTP is quite simple. Each test is a binary written either in
|
|||
|
|
portable shell or C. The test gets a configuration via environment variables
|
|||
|
|
and/or command line parameters, it prints additional information into the
|
|||
|
|
stdout and reports overall success/failure via the exit value.
|
|||
|
|
|
|||
|
|
Tests are generally placed under the 'testcases/' directory. Everything that
|
|||
|
|
is a syscall or (slightly confusingly) libc syscall wrapper goes under
|
|||
|
|
'testcases/kernel/syscalls/'.
|
|||
|
|
|
|||
|
|
Then there is 'testcases/open_posix_testsuite/' which is a well maintained fork
|
|||
|
|
of the upstream project that has been dead since 2005 and also a number of
|
|||
|
|
directories with tests for more specific features.
|
|||
|
|
|
|||
|
|
5.1 Runtest Files
|
|||
|
|
^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
The list of tests to be executed is stored in runtest files under the
|
|||
|
|
'runtest/' directory. The default set of runtest files to be executed is
|
|||
|
|
stored in 'scenario_groups/default'. When you add a test you should add
|
|||
|
|
corresponding entries into some runtest file(s) as well.
|
|||
|
|
|
|||
|
|
For syscall tests (these placed under 'testcases/kernel/syscalls/') use
|
|||
|
|
'runtest/syscalls' file, for kernel related tests for memory management we
|
|||
|
|
have 'runtest/mm', etc.
|
|||
|
|
|
|||
|
|
IMPORTANT: The runtest files should have one entry per a test. Creating a
|
|||
|
|
wrapper that runs all your tests and adding it as a single test
|
|||
|
|
into runtest file is strongly discouraged.
|
|||
|
|
|
|||
|
|
5.2 Datafiles
|
|||
|
|
^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
If your test needs datafiles to work, these should be put into a subdirectory
|
|||
|
|
named 'datafiles' and installed into the 'testcases/data/$TCID' directory (to
|
|||
|
|
do that you have to add 'INSTALL_DIR := testcases/data/TCID' into the
|
|||
|
|
'datafiles/Makefile').
|
|||
|
|
|
|||
|
|
You can obtain path to datafiles via $TST_DATAROOT provided by test.sh
|
|||
|
|
'$TST_DATAROOT/...'
|
|||
|
|
or via C function 'tst_dataroot()' provided by libltp:
|
|||
|
|
|
|||
|
|
[source,c]
|
|||
|
|
-------------------------------------------------------------------------------
|
|||
|
|
const char *dataroot = tst_dataroot();
|
|||
|
|
-------------------------------------------------------------------------------
|
|||
|
|
|
|||
|
|
Datafiles can also be accessed as '$LTPROOT/testcases/data/$TCID/...',
|
|||
|
|
but '$TST_DATAROOT' and 'tst_dataroot()' are preferred as these can be used
|
|||
|
|
when running testcases directly in git tree as well as from install
|
|||
|
|
location.
|
|||
|
|
|
|||
|
|
The path is constructed according to these rules:
|
|||
|
|
|
|||
|
|
1. if '$LTPROOT' is set, return '$LTPROOT/testcases/data/$TCID'
|
|||
|
|
2. else if 'tst_tmpdir()' was called return '$STARTWD/datafiles'
|
|||
|
|
(where '$STARTWD' is initial working directory as recorded by 'tst_tmpdir()')
|
|||
|
|
3. else return '$CWD/datafiles'
|
|||
|
|
|
|||
|
|
See 'testcases/commands/file/' for example.
|
|||
|
|
|
|||
|
|
5.3 Subexecutables
|
|||
|
|
^^^^^^^^^^^^^^^^^^
|
|||
|
|
|
|||
|
|
If your test needs to execute a binary, place it in the same directory as the
|
|||
|
|
testcase and name the file starting with '${test_binary_name}_'. Once the
|
|||
|
|
test is executed by the framework, the path to the directory with all LTP
|
|||
|
|
binaries is added to the '$PATH' and you can execute it just by its name.
|
|||
|
|
|
|||
|
|
TIP: If you need to execute such test from the LTP tree, you can add path to
|
|||
|
|
current directory to '$PATH' manually with: 'PATH="$PATH:$PWD" ./foo01'.
|
|||
|
|
|
|||
|
|
6 Test Contribution Checklist
|
|||
|
|
------------------------------
|
|||
|
|
|
|||
|
|
NOTE: See also
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/Maintainer-Patch-Review-Checklist[Maintainer Patch Review Checklist].
|
|||
|
|
|
|||
|
|
1. Test compiles and runs fine (check with `-i 10` too)
|
|||
|
|
2. `make check` does not emit any warnings for the test you are working on
|
|||
|
|
(hint: run it in the test's directory and/or use `make check-$TCID`)
|
|||
|
|
3. The runtest entries are in place
|
|||
|
|
4. Test binaries are added into corresponding '.gitignore' files
|
|||
|
|
5. Patches apply over the latest git
|
|||
|
|
|
|||
|
|
6.1 About .gitignore files
|
|||
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
There are numerous '.gitignore' files in the LTP tree. Usually there is a
|
|||
|
|
'.gitignore' file per a group of tests. The reason for this setup is simple.
|
|||
|
|
It's easier to maintain a '.gitignore' file per directory with tests, rather
|
|||
|
|
than having single file in the project root directory. This way, we don't have
|
|||
|
|
to update all the gitignore files when moving directories, and they get deleted
|
|||
|
|
automatically when a directory with tests is removed.
|
|||
|
|
|
|||
|
|
7 Testing pre-release kernel features
|
|||
|
|
-------------------------------------
|
|||
|
|
|
|||
|
|
Tests for features not yet in a mainline kernel release are accepted. However
|
|||
|
|
they must only be added to the +staging+ runtest file. Once a feature is part
|
|||
|
|
of the stable kernel ABI the associated test must be moved out of staging.
|
|||
|
|
|
|||
|
|
This is primarily to help test kernel RCs by avoiding the need to download
|
|||
|
|
separate LTP patchsets.
|
|||
|
|
|
|||
|
|
8 LTP C And Shell Test API Comparison
|
|||
|
|
-------------------------------------
|
|||
|
|
|
|||
|
|
Comparison of
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/C-Test-API[C Test API] and
|
|||
|
|
https://github.com/linux-test-project/ltp/wiki/Shell-Test-API[Shell Test API].
|
|||
|
|
|
|||
|
|
[options="header"]
|
|||
|
|
|================================================================================
|
|||
|
|
| C API ('struct tst_test' members) | shell API ('$TST_*' environment variables)
|
|||
|
|
| '.all_filesystems' | 'TST_ALL_FILESYSTEMS'
|
|||
|
|
| '.bufs' | –
|
|||
|
|
| '.caps' | –
|
|||
|
|
| '.child_needs_reinit' | not applicable
|
|||
|
|
| '.cleanup' | 'TST_CLEANUP'
|
|||
|
|
| '.dev_extra_opts' | 'TST_DEV_EXTRA_OPTS'
|
|||
|
|
| '.dev_fs_opts' | 'TST_DEV_FS_OPTS'
|
|||
|
|
| '.dev_fs_type' | 'TST_FS_TYPE'
|
|||
|
|
| '.dev_min_size' | not applicable
|
|||
|
|
| '.format_device' | 'TST_FORMAT_DEVICE'
|
|||
|
|
| '.max_runtime' | –
|
|||
|
|
| '.min_cpus' | not applicable
|
|||
|
|
| '.min_kver' | 'TST_MIN_KVER'
|
|||
|
|
| '.min_mem_avail' | not applicable
|
|||
|
|
| '.mnt_flags' | 'TST_MNT_PARAMS'
|
|||
|
|
| '.mntpoint', '.mnt_data' | 'TST_MNTPOINT'
|
|||
|
|
| '.mount_device' | 'TST_MOUNT_DEVICE'
|
|||
|
|
| '.needs_cgroup_ctrls' | –
|
|||
|
|
| '.needs_checkpoints' | 'TST_NEEDS_CHECKPOINTS'
|
|||
|
|
| '.needs_cmds' | 'TST_NEEDS_CMDS'
|
|||
|
|
| '.needs_devfs' | –
|
|||
|
|
| '.needs_device' | 'TST_NEEDS_DEVICE'
|
|||
|
|
| '.needs_drivers' | 'TST_NEEDS_DRIVERS'
|
|||
|
|
| '.needs_kconfigs' | 'TST_NEEDS_KCONFIGS'
|
|||
|
|
| '.needs_overlay' |
|
|||
|
|
| '.needs_rofs' | –
|
|||
|
|
| '.needs_root' | 'TST_NEEDS_ROOT'
|
|||
|
|
| '.needs_tmpdir' | 'TST_NEEDS_TMPDIR'
|
|||
|
|
| '.options' | 'TST_PARSE_ARGS', 'TST_OPTS'
|
|||
|
|
| '.resource_files' | –
|
|||
|
|
| '.restore_wallclock' | not applicable
|
|||
|
|
| '.sample' | –
|
|||
|
|
| '.save_restore' | –
|
|||
|
|
| '.scall' | not applicable
|
|||
|
|
| '.setup' | 'TST_SETUP'
|
|||
|
|
| '.skip_filesystems' | 'TST_SKIP_FILESYSTEMS'
|
|||
|
|
| '.skip_in_compat' | –
|
|||
|
|
| '.skip_in_lockdown' | –
|
|||
|
|
| '.supported_archs' | not applicable
|
|||
|
|
| '.tags' | –
|
|||
|
|
| '.taint_check' | –
|
|||
|
|
| '.tcnt' | 'TST_CNT'
|
|||
|
|
| '.tconf_msg' | not applicable
|
|||
|
|
| '.test', '.test_all' | 'TST_TESTFUNC'
|
|||
|
|
| '.test_variants' | –
|
|||
|
|
| '.timeout' | 'TST_TIMEOUT'
|
|||
|
|
| '.tst_hugepage' | not applicable
|
|||
|
|
| .format_device | 'TST_DEVICE'
|
|||
|
|
| not applicable | 'TST_NEEDS_KCONFIGS_IFS'
|
|||
|
|
| not applicable | 'TST_NEEDS_MODULE'
|
|||
|
|
| not applicable | 'TST_POS_ARGS'
|
|||
|
|
| not applicable | 'TST_USAGE'
|
|||
|
|
|================================================================================
|