PCP Portability and QA

|  | Home

# Portability Guidelines for PCP QA

Ken McDonell <kenj@kenj.id.au>

September 2018

## Background

In this document the generic term "component" will be used to describe a piece of functionality in Performance Co-Pilot (PCP) that could be any of the following:

- a library
- a method or module or class in a library
- an application
- a PMDA
- a service or API
- an object in the file system

Obviously the functional goals and feature focus of PCP has been to build tools that help understand hard performance problems that are seen in some the the world's most complex systems. Over more than two decades, the delivery of components to address the feature requirements has been build on engineering processes and approaches with a technical focus on:

- Portability

  From the outset we were concerned with making PCP work successfully across a wide variety of environments.

  The earliest PCP architectural deliberations began at Pyramid Technology in 1993, but by this time I had already been working on portable software for more than 15 years. Then the PCP incubator moved to SGI, and in the very early days of Linux, Mark Goodwin undertook an "IRIX to Linux" port as a skunk works project within SGI Engineering. A little later, SGI's Clustered File System (CXFS) had PCP components for IRIX, Windows, HPUX, AIX, MacOS and Linux, across a variety of CPU architectures, C compilers and operating systems.

  Although many of these have passed away, PCP survives and is now actively supported on all Linux variants (most importantly the Fedora/RPM, OpenSuSE/RPM and Debian/dpkg platforms and their respective derivatives), the BSD family, OpenIndiana (nee Sun Solaris), Mac OS X and Windows.

  So there is a very rich history of engineering for portability in the PCP DNA, and an expectation that new PCP components will be "robust" (see below for a definition of "robust") across all of the supported platforms and environments.

- Testability

  The PCP project has never had a dedicated QA team, so by necessity the QA model that was adopted (and at times enforced) was one where if an engineer added a feature or fixed a bug, there was an expectation that there would be additional QA coverage for the associated changes. Although this approach suffers from tunnel-vision and common assumptions between development and testing, it does have the advantage that testing and QA is a communal responsibility.

  This effort grew into the very large QA infrastructure that lives below the qa directory in the source tree and is shipped in the pcp-testsuite package. This represents a significant engineering effort in its own right as the table below shows:

  | below the src dir | below the qa dir
  --- | --- | ---
  C or C++ | 320,000 | 37,000
  shell | 23,000 | 85,000
  perl | 18,000 | 700
  python | 17,000 | 200
  Lines of Code

  The QA test suite comprises close to 1,100 test scripts. The scripts are used by individual developers to test code changes and check for regressions. In addition the entire suite of scripts is run regularly across the 30+ machines in the Melbourne QA Farm.

## Robustness

Robustness simply means that every PCP application and service either works correctly or detects that the environment will not support correct execution and is either omitted from the build or omitted from the packaging or reports a warning and exits in an orderly fashion.

Some examples may help illustrate this philosophy.

- The platform PMDAs (linux, windows, freebsd, openbsd, netbsd, opensolaris, aix, ...) are only compiled and packaged if the build is being run for the corresponding target platform.
- Optional headers are checked for using the AC_CHECK_HEADERS() macro in configure.ac and cpp conditionals and macros are used to guard the inclusion of these headers, e.g. <strings.h> in src/dbpmda/src/util.c.
- In the Apple operating systems there used to be a rpc.server.nqnfs.leases metric but this has been deprecated and is no longer available. The corresponding darwin PMDA uses PMAPI protocols to return the type of this metric as PM_TYPE_NOSUPPORT which all PCP clients are able to handle appropriately (either ignore the metric or report an explanatory warning). See src/pmdas/darwin/pmda.c
- Every PCP QA test script completes with a status of pass, fail or notrun. The latter is used to indicate that the test either cannot be run because some required component is missing or the test is not appropriate for the current platform. See the tests for the Python OrderedDict module and the pmrep application in qa/035.

## Source Code and Compile Time Portability

Mostly what's been done here is common and good engineering practice. For example using configure, conditionals in the GNUmakefiles and assorted sed/awk rewriting scripts to ensure the code will compile cleanly on all platforms. Compiler warnings are enabled and builds are expected to complete without warnings. And in the source code we demand thorough error checking and error handling on all system and library calls.

We've extended the normal concept of macros to include a set of globally defined names that are are used for path and directory names, search paths, application names and application options and arguments. These are defined in src/include/pcp.conf.in, bound to values at build time by configure and installed in /etc/pcp.conf. These can then be used in shell scripts and applications in C, C++, Perl, Python have run-time access to these via pmGetConfig() or pmGetOptionalConfig(), see for example src/pmie/src/pmie.c.

Even file pathname separators (/ for the sane world, \ elsewhere) have been abstracted out and pmPathSeparator() is used to construct pathnames from directory components.

At a higher level we don't even try to build code if it is not intended for the target platform.

## Packaging Portability

At packaging time we use conditionals to include only those components that can be built and are expected to work for the target platform.

This extends to wrapping some of the prerequisites in conditionals if the prerequisite piece may not be present or may have a different name.

For Debian packaging this means debian/control is build from debian/control.main and the ?{foo} elements of the Build-Depends and Depends clauses are conditionally expanded by the debian/fixcontrol.main script during the build.

For RPM packaging this means using configure to perform macro substitution to create build/rpm/pcp.spec from build/rpm/pcp.spec.in and using %if within the spec file to selectively include packages and adjust the BuildRequires and Requires clauses.

## QA Portability

There are purpose-designed QA applications in the qa/src directory and the source code for these applications should follow all of the same guidelines for portability and build resilience as those that apply for the applications and libraries that ship with the main part of PCP. The only possible exception is that error handling can be a little more relaxed for the QA applications as they are used in a more controlled manner.

### Use /etc/pcp.conf

All QA test scripts will have the variables set in /etc/pcp.conf placed in the environment using $PCP_DIR/etc/pcp.env which is called from common.rc (with $PCP_DIR typically unset). And all QA tests source common.rc, although the sequence is a little convoluted (and irrelevant for this discussion).

The important thing is that the following environment variables become available to improve the portability of QA test scripts and the output filtering (see below) they must perform.

- A QA script should never use a literal pathname to a PCP installed file or directory because there will be a $PCP_ variable that can be used. For example:
  `$ cd $PCP_PMDAS_DIR/mypmda`
  is always correct, whereas
  `$ cd /var/lib/pcp/pmdas/mypmda`
  may not work on some platforms.
- The current platform is available via $PCP_PLATFORM. The common values are linux (of any flavour), freebsd, netbsd, openbsd, darwin (Mac OS X), solaris (really OpenIndiana) and mingw (Windows). This can be used to make high-level decisions about the behavior of a QA test, see the Not run and Alternate .out files sections below.
- If a QA tests depends on a particular version or versions of PCP, then $PCP_VERSION can be tested.
- Because there may be more than one version of a (non-PCP) command installed or because we need special command line arguments to make a (non-PCP) command behave in a standard manner, the following are available:
  - $PCP_AWK_PROG Posix-compliant awk
  - $PCP_SORT_PROG Unix-like sort
  - $PCP_MAKE_PROG the preferred gmake
  - $PCP_CC_PROG the preferred gcc
  - $PCP_ECHO_PROG the standard echo and importantly the prefix ($PCP_ECHO_N) and the suffix ($PCP_ECHO_C) to allow output without a trailing newline. These are important because
    `$ echo -n "Prompt? "`
    is not portable, nor is
    `$ echo "Prompt? \c"`
    but this portable and correct, albeit ugly!
    `$ $PCP_ECHO_PROG $PCP_ECHO_N "Prompt? $PCP_ECHO_C"`
  - $PCP_PS_PROG the preferred ps and more importantly when $PCP_PS_PROG is run with $PCP_PS_ALL_FLAGS as the argument we get the fields and field-order that a QA test can rely upon.
  - $PCP_PYTHON_PROG the version of python that the installed PCP components are expecting (critical for version 2 vs version 3 differences).

### Use the predefined variables

PCP QA scripts follow a standard template when created with the qa/new script (which is the recommended way to create new QA test scripts) and then the following local shell variables are available:

- $seq: the current test sequence number
- $tmp: a private and unique path that maybe used to create temporary files or a temporary directory all of which will be removed when the script ends. For example:
  `$ ... >$tmp.out 2>$tmp.err`
  `$ mkdir $tmp`
- $sudo: run the sudo command with arguments that make it work the same everywhere
- $DSO_SUFFIX: if you need to refer to a shared library by name, avoid the literal ".so" suffix; instead use .$DSO_SUFFIX that will become .so or .dll or .dylib or ... and will work across all platforms.

### check-vm is your friend, not the enemy

There is a a very large set of applications and packages outside of PCP that are required to build PCP from source and/or run PCP QA. The script qa/admin/check-vm tries to capture all of these requirements. check-vm should be used as follows:

- Before starting a new QA cycle, running this script with a -p argument will identify any packages that you should consider installing. Without the -p you'll see more details of what's needed and why.
- If you are a developer and add a new component, please make sure that any additional dependencies for you new component have been captured in qa/admin/check-vm.

### Avoid exotic commands and non-standard command arguments

Our QA tests scripts are run with sh not bash and on some platforms these are really different!

Things like an == operand used with test (aka [) as in
`$ if [ "$foo" == "bar" ] ...`
will not work with the Bourne shell. Instead, use the = operator, e.g.
`$ sh`
`$ x=1`
`$ [ "$x" = 1 ] && echo yippee`
`yippee`
`$ [ "$x" == 1 ] && echo boo`
`sh: 3: [: 1: unexpected operator`

Also and any use of the bash [[ operator
`$ ... [[ ... ]]`
is going to blow up when presented to a real Bourne shell. In most cases test (aka [) can be used for a straight forward rewrite.

Even less portable is any use of the bash $((...)) construct for in-built arithmetic. For these cases, you'll need equivalent logic using expr, e.g.
instead of
`x=$(($x+1))`
use
`x=`expr $x + 1``

Another recurring one is the -i argument for sed; this is not standard and not supported everywhere so just do not use it. The alternative:
`$ sed ... <somefile >$tmp.tmp; mv $tmp.tmp somefile`
works everywhere for most of the cases in QA. If cross-filesystem linking and a lame mv are in play then the following is even more portable:
`$ sed ... <somefile >$tmp.tmp; cp $tmp.tmp somefile; rm $tmp.tmp`
If permissions are in play, then you may need:
`$ sed ... <somefile >$tmp.tmp; $sudo cp $tmp.tmp somefile; rm $tmp.tmp`

Do not use seq as it is not portable. For example, this is not portable:
`for i in $(seq 4); do ...; done`
but it can be rewritten as follows and this will work everywhere:
`i=1; while [ $i -le 4 ]; do ...; i=`expr $i + 1`; done`

### Output filtering

Each QA test script must produce a standard output stream that is deterministic. The qa/check script that is responsible for running each QA test script (say NNNN) captures the standard output from NNNN and if this matches the expected result in NNNN.out then then test is deemed to have passed, otherwise the standard output from NNNN is saved in NNNN.out.bad and the test is deemed to have failed.

So it is critical that output of NNNN is deterministic across all platforms and timezones.

Applications used in a QA test script will potentially produce output that is irrelevant to the success or otherwise of the test. The most obvious example is the current date from an application run on the test system. Consequently, most QA test scripts include one or more _filter() functions that are responsible to translating the raw output from the applications run in the test into the deterministic output.

These filtering functions in turn often use a set of standard functions in qa/common.filter that may be applied to the output of common PCP commands and operations.

There are way too many filtering cases to describe them all here, but the list below is illustrative of the range of techniques that may be needed. And each QA test script may need employ more than one of the techniques to produce deterministic output for "correct" execution across all platforms.

- For many operations involving starting or stopping PCP daemons and for common output formats from PCP commands there are existing filtering functions in qa/common.filter that may be used, for example:
  - _filter_pcp_start() filters the output from running the pcp "init" script that starts pmcd and pmlogger. May also be used for the separate pmcd and pmlogger "init" scripts that start just pmcd and pmlogger respectively.
  - _filter_pcp_stop() filters the output from running the pcp "init" script that stops pmcd and pmlogger. May also be used for the separate pmcd and pmlogger "init" scripts that stop just pmcd and pmlogger
  - _filter_pmie_start() filters the output from running the pmie "init" script that starts pmie.
  - _filter_pmda_install() filters the output from installing a new PMDA.
  - _filter_pmcd_log() filters the log file from pmcd.
  - _filter_pmlogger_log() filters the log file from pmlogger.
  - _filter_pmdumplog_log() filters the log file from pmlogdump.
  - _filter_pmdumptext_log() filters the log file from pmdumptext.
  - _filter_optional_pmdas() removes lines containing the names of executable PMDAs that may, or may not, be installed.
  - _filter_top_pmns() removes lines containing the just the top-level PMNS names for PMDAs that may, or may not, be installed.
  - _filter_pmie_log() filters the log file from pmie.
  - _filter_dumpresult() filters the diagnostic output from pmDumpResult() (translate timestamp to a literal TIMESTAMP, remove data addresses, replace metric values by literals like NUMBER, HEXNUMBER, STRING or AGGREGATE).
  - _filter_dbg() strips timestamps from some of the diagnostics generated by the -D command line option to PCP commands.
- Local date/time filtering. Typically the current date and time is not critical to the success of the test, so replacing the actual date and/or time by a constant is common filtering operation. See for example DATE in _filter() in qa/866, or TIMESTAMP in _filter_pmdumplog() in qa/common.filter
- Another approach to ensuring deterministic output for tests that use PCP archives is to use the -z command line argument to most PCP commands so that any date and time reporting uses the timezone of the archive rather than the timezone of the machine where the test is being run. See for example qa/1072.
- Remote host name, IP addresses, process ids (pids) and port numbers are all examples of data that is likely to vary across the platforms where the QA test scripts are run and so are candidates for filtering. qa/100 includes several examples of this type of filtering.
- For some QA tests scripts the "correct" information is confined to one or more fields or columns. If the other fields and columns are not deterministic, then vertical filtering may be appropriate. For an example using cut see qa/951 while qa/957 shows how vertical filtering can be achieved using awk.
- Similarly, sometimes the information that is critical to the outcome of a QA test may be in only some of the output lines while the other lines are not constant across platforms. Horizontal filtering may be pattern-based (e.g. using grep or sed) or ordinal line number based (e.g. using sed or awk to remove or keep lines N-M) or FSA-based (e.g. using awk to skip lines until the first line of interest is found, then output the following lines until the last line of interest is found).
- Gratuitous white space differences for the same command across different platforms is another common scenario. See for example od output filtering in qa/079, or wc output filtering in qa/1059.
- For some tests, a value may be deemed "correct" if it is in an expected range. In these cases filtering is required to map the actual value onto a constant literal value. For an example, see the _range() function in qa/178.

To the extent that filtering removes or rewrites what would otherwise be the output of a QA test script, there maybe some difficulty in triaging test failures if you have only the expected NNNN.out and the observed NNNN.out.bad files. Most QA test scripts will emit unfiltered output and diagnostics to aid triage and these are written to a NNNN.full file which is retained for inspection after the test has been run.

### Not run

If QA test script NNNN decides it cannot or should not be run on the current platform, it should create a NNNN.notrun file. Optionally NNNN.notrun contains the reason the test has not been run.

The convenience function _notrun() defined in qa/common.check may be used to create the file, write the reason to the file and exit.

Some common reasons for a test to be "not run" are:

- the test is platform-specific and makes no sense on other platforms
- an optional PMDA that is required for the QA test script is not installed
- a command or application or service that is required for the QA test script is not installed or cannot be found
- a Python or Perl module that is required for the QA test script is not installed
- the QA test script requires a kernel, library, application or service at some minimum release or version level

### PMNS ordering

If a QA test script uses an application that traverses the PMNS for a non-leaf metric name (usually from the command line or a configuration file) then depending on the PMDA(s) involved, there is a chance that the names in the PMNS may be processed in different orders on different platforms.

The simplest solution is to enumerate the PMNS nodes of interest in the QA script using pminfo, sort the list and then have the application being tested operate on the leaf nodes of the PMNS one at a time. qa/647 illustrates an example of this approach.

### Instance ordering

Some PMDAs maintain their instance domains in a manner that may present instances in non-deterministic order, so while all instances may be present on all platforms, the sequence of the instances within a instance domain or in a pmResult is not the same everywhere.

The QA helper application $here/src/sortinst (source in qa/src/sortinst.c may be use to re-order the reported instances so the sequence is deterministic. See qa/1180 for an example that uses $here/src/sortinst to re-order pminfo -f output.

### Sort ordering

Unfortunately sort does not produce the same sorted order on all platforms by default. We need to take control of LC_COLLATE and the decision has been made to standardize on POSIX as the collating sequence for PCP QA.

$LC_COLLATE is set and exported into the environment in common.rc, but this is a relatively recent change so LC_COLLATE=POSIX is liberally sprinkled throughout the QA test suite ahead of running sort.

### Filesystem directory order

Some PMDAs use readdir() or similar routines to scan a directory contents to find metrics and/or instances. This practice is certain to expose platform differences as the order of directory entries is unlikely to be deterministic.

Judicious use of sort will be required. See qa/496 for a typical example of how this should be done.

### Alternate .out files

For some PMDAs and/or some QA test scripts, no amount of clever engineering will hide the fact that a "correct" test execution on one platform may produce different output to "correct" test execution on another platform. In these cases the simplest choice is to not have a single NNNN.out file, but rather have a set of them and the QA test script chooses the most appropriate one and links this to NNNN.out when it starts.

Alternate NNNN.out files may be required in the following types of cases:

- PMNS differences (especially for the kernel PMDA), e.g qa/012
- Base service differences, e.g. IPv6 may, or may not, be present for qa/365
- Hashing differences
- Different word size may mean different underlying metric data type differences, e.g. in qa/926
- Endianness differences make some dump outputs completely different, e.g. in qa/422