Testing a PL/I Compiler Using Precomputation-based Program Generation

In automated compiler testing, the focus typically lies in uncovering bugs caused by optimisations performed by the compiler. However, there is a class of compilers where little to no optimisations are performed: those for migration of legacy software. Therefore, it is not clear to what extent such legacy compilers would benefit from automated compiler testing. We investigated this in the context of the Raincode legacy compiler for PL/I, an industrial compiler targeting the. NET platform. We designed and implemented a framework for automated PL/I compiler testing through precomputation-based program generation and ran it on two versions of the Raincode PL/I compiler: an older with known bugs and the latest release. On the older version, our framework generated around 127.000 programs and found five bugs, two of which were previously unknown to us. For the latest compiler release, after 180 hours of tests and more than 718.000 generated programs, the framework did not reveal any bugs.