147 lines
5.2 KiB
Markdown
Executable file
147 lines
5.2 KiB
Markdown
Executable file
# Assembly Tests
|
|
|
|
The Benchmark library provides a number of functions whose primary
|
|
purpose in to affect assembly generation, including `DoNotOptimize`
|
|
and `ClobberMemory`. In addition there are other functions,
|
|
such as `KeepRunning`, for which generating good assembly is paramount.
|
|
|
|
For these functions it's important to have tests that verify the
|
|
correctness and quality of the implementation. This requires testing
|
|
the code generated by the compiler.
|
|
|
|
This document describes how the Benchmark library tests compiler output,
|
|
as well as how to properly write new tests.
|
|
|
|
|
|
## Anatomy of a Test
|
|
|
|
Writing a test has two steps:
|
|
|
|
* Write the code you want to generate assembly for.
|
|
* Add `// CHECK` lines to match against the verified assembly.
|
|
|
|
Example:
|
|
```c++
|
|
|
|
// CHECK-LABEL: test_add:
|
|
extern "C" int test_add() {
|
|
extern int ExternInt;
|
|
return ExternInt + 1;
|
|
|
|
// CHECK: movl ExternInt(%rip), %eax
|
|
// CHECK: addl %eax
|
|
// CHECK: ret
|
|
}
|
|
|
|
```
|
|
|
|
#### LLVM Filecheck
|
|
|
|
[LLVM's Filecheck](https://llvm.org/docs/CommandGuide/FileCheck.html)
|
|
is used to test the generated assembly against the `// CHECK` lines
|
|
specified in the tests source file. Please see the documentation
|
|
linked above for information on how to write `CHECK` directives.
|
|
|
|
#### Tips and Tricks:
|
|
|
|
* Tests should match the minimal amount of output required to establish
|
|
correctness. `CHECK` directives don't have to match on the exact next line
|
|
after the previous match, so tests should omit checks for unimportant
|
|
bits of assembly. ([`CHECK-NEXT`](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-next-directive)
|
|
can be used to ensure a match occurs exactly after the previous match).
|
|
|
|
* The tests are compiled with `-O3 -g0`. So we're only testing the
|
|
optimized output.
|
|
|
|
* The assembly output is further cleaned up using `tools/strip_asm.py`.
|
|
This removes comments, assembler directives, and unused labels before
|
|
the test is run.
|
|
|
|
* The generated and stripped assembly file for a test is output under
|
|
`<build-directory>/test/<test-name>.s`
|
|
|
|
* Filecheck supports using [`CHECK` prefixes](https://llvm.org/docs/CommandGuide/FileCheck.html#cmdoption-check-prefixes)
|
|
to specify lines that should only match in certain situations.
|
|
The Benchmark tests use `CHECK-CLANG` and `CHECK-GNU` for lines that
|
|
are only expected to match Clang or GCC's output respectively. Normal
|
|
`CHECK` lines match against all compilers. (Note: `CHECK-NOT` and
|
|
`CHECK-LABEL` are NOT prefixes. They are versions of non-prefixed
|
|
`CHECK` lines)
|
|
|
|
* Use `extern "C"` to disable name mangling for specific functions. This
|
|
makes them easier to name in the `CHECK` lines.
|
|
|
|
|
|
## Problems Writing Portable Tests
|
|
|
|
Writing tests which check the code generated by a compiler are
|
|
inherently non-portable. Different compilers and even different compiler
|
|
versions may generate entirely different code. The Benchmark tests
|
|
must tolerate this.
|
|
|
|
LLVM Filecheck provides a number of mechanisms to help write
|
|
"more portable" tests; including [matching using regular expressions](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-pattern-matching-syntax),
|
|
allowing the creation of [named variables](https://llvm.org/docs/CommandGuide/FileCheck.html#filecheck-variables)
|
|
for later matching, and [checking non-sequential matches](https://llvm.org/docs/CommandGuide/FileCheck.html#the-check-dag-directive).
|
|
|
|
#### Capturing Variables
|
|
|
|
For example, say GCC stores a variable in a register but Clang stores
|
|
it in memory. To write a test that tolerates both cases we "capture"
|
|
the destination of the store, and then use the captured expression
|
|
to write the remainder of the test.
|
|
|
|
```c++
|
|
// CHECK-LABEL: test_div_no_op_into_shr:
|
|
extern "C" void test_div_no_op_into_shr(int value) {
|
|
int divisor = 2;
|
|
benchmark::DoNotOptimize(divisor); // hide the value from the optimizer
|
|
return value / divisor;
|
|
|
|
// CHECK: movl $2, [[DEST:.*]]
|
|
// CHECK: idivl [[DEST]]
|
|
// CHECK: ret
|
|
}
|
|
```
|
|
|
|
#### Using Regular Expressions to Match Differing Output
|
|
|
|
Often tests require testing assembly lines which may subtly differ
|
|
between compilers or compiler versions. A common example of this
|
|
is matching stack frame addresses. In this case regular expressions
|
|
can be used to match the differing bits of output. For example:
|
|
|
|
```c++
|
|
int ExternInt;
|
|
struct Point { int x, y, z; };
|
|
|
|
// CHECK-LABEL: test_store_point:
|
|
extern "C" void test_store_point() {
|
|
Point p{ExternInt, ExternInt, ExternInt};
|
|
benchmark::DoNotOptimize(p);
|
|
|
|
// CHECK: movl ExternInt(%rip), %eax
|
|
// CHECK: movl %eax, -{{[0-9]+}}(%rsp)
|
|
// CHECK: movl %eax, -{{[0-9]+}}(%rsp)
|
|
// CHECK: movl %eax, -{{[0-9]+}}(%rsp)
|
|
// CHECK: ret
|
|
}
|
|
```
|
|
|
|
## Current Requirements and Limitations
|
|
|
|
The tests require Filecheck to be installed along the `PATH` of the
|
|
build machine. Otherwise the tests will be disabled.
|
|
|
|
Additionally, as mentioned in the previous section, codegen tests are
|
|
inherently non-portable. Currently the tests are limited to:
|
|
|
|
* x86_64 targets.
|
|
* Compiled with GCC or Clang
|
|
|
|
Further work could be done, at least on a limited basis, to extend the
|
|
tests to other architectures and compilers (using `CHECK` prefixes).
|
|
|
|
Furthermore, the tests fail for builds which specify additional flags
|
|
that modify code generation, including `--coverage` or `-fsanitize=`.
|
|
|