A while ago I ordered two Beagle boards to play with test and evaluate Qt on embedded hardware for KDAB, my part-time employer.

Hardware
The Beagle Board [let's call it BB] is a small single-board computer with an ARM CPU; keyboard, mouse, disk an…

Regarding workarounds, I am well aware that -fno-tree-vectorize will disable the broken optimisation. However, having vectorisation with such severe bugs enabled by default at -O3 is not acceptable in my opinion.

The “preview release” of which you speak was nothing other than an evaluation download of the commercial package, and it is correct that I did not immediately find any errors in that release. As I said in the email you quoted above, my test case for the other bug passed, and FFmpeg worked correctly in a few simple tests. I have since added the ability to run the full FFmpeg regression test suite on the target board, which enabled me to find the latest bug.

As for testing, the number of tests, even if inflated into the millions (a number I simply do not believe), is irrelevant. What matters is coverage. In particular, every feature should be exercised at least once during testing. Specifically, if the right-shifting every element in an array is recognised as vectorisable, this should be tested. It is evident that this did not happen.

To the remainder of your game with test suite names and numbers, I can think of no response more appropriate than the words of the brilliant Dijkstra: Testing shows the presence, not the absence of bugs.

Thank you for continuing to exercise Sourcery G++ with FFMPEG.

Of course, we’re disappointed that you found a defect in the latest
release.

(In the future, it would be helpful if you could post any defects to
our public mailing list here:

http://www.codesourcery.com/archives/arm-gnu/maillist.html

so that we are sure to see them. It’s very helpful to have both
preprocessed source code and command-line options with defect
reports.)

We have been able to reproduce this bug, though not with the source
code you posted. It looks like the case that triggers the defect uses
“unsigned char” rather than “unsigned short” as in your posting. With
that change, and the options “-mcpu=cortex-a8 -mfpu=neon -O3
-mfloat-abi=softfp”, we do see the bug.

You can work around this bug by using “-fno-tree-vectorize” or “-O2″,
both of which will disable automatic vectorization. You can use those
options just on the affected kernels in FFMPEG, while still getting
the benefits of vectorization elsewhere.

As you know, we made available to you a preview version of this latest
release specifically so that you could test FFMPEG with it. You
reported that:

> I’ve done some simple testing, and I’m not finding any bugs this
> time. The test case from before works, and FFmpeg builds and runs
> correctly.

We’ve verified that the same bug that you reported here was indeed
present in that preview release. It sounds like you’ve broadened your
test coverage since then. Obviously, we’re sympathetic to the problem
of testing large, complex software packages; it’s very difficult to
cover all possible cases. We’re continually expanding our test
coverage as well.

We run hundreds of thousands (or millions, depending how you count) of
test cases on every release, both on real hardware (including
Cortex-A8) and in simulation. Those include the DejaGNU regression
testsuites that are part of the FSF sources for the tools, as well as
(depending on the exact release), testsuites and benchmarks like
Plum-Hall, Dhrystone, EEMBC, and SPEC.

We maintain a database of failures in these testsuites, and analyze
new failures to determine their origin. We also do manual testing of
each release in order to provide a “sanity check”. The pass rates in
our testing are generally 99.99% or higher, with the remainder of the
failures explained by vagaries of the test environment.

The vectorization testsuite in DejaGNU does test the compiler’s
ability to vectorize a variety of loops, and, further, that the
vectorized loops run correctly. We know that on NEON our releases
vectorize the same set of loops from these test cases that are
vectorized on other platforms with similar vector capabilities. We
also know that the vectorized loops run correctly on Cortex-A8.
Unfortunately, the testsuites that we have do not cover the test case
which you posted here. Therefore, when we fix this defect, we will
add a new test case to the DejaGNU testsuite, ensuring that the defect
does not reoccur in future releases.

Unfortunately, the GNU toolchain does have defects. The tools
community (in which CodeSourcery is an active participant) attempts to
correct as many defects as possible — both in the context of
volunteer and commercial development — but, inevitably, every release
does have defects. A quick search of the public GCC bug database
indicates 236 open wrong-code bugs; these are all cases where GCC is
known to generate incorrect code on a valid input program, as in your
test case here.

We will continue to do our best to improve the quality of GCC in
general, and our releases in particular. We have entered the defect
you report into our internal defect tracker and expect that it will be
corrected in a future release.

Best regards,

–
Mark Mitchell
CodeSourcery

I wonder how the various buggy tests you find behave in qemu which has so many bugs in its NEON implementation.