moiji-mobile

The GNU autoconf ships with a not well known piece of software. It is called GNU autotest and we will focus about it in this blog post.

GNU autotest is a very simple framework/test runner. One needs to define a testsuite and this testsuite will launch test applications and record the exit code, stdout and stderr of the test application. It can diff the output with expected one and fail if it is not matching. Like any of the GNU autotools a log file is kept about the execution of each test. This tool can be nicely integrated with automake’s make check and make distcheck. This will execute the testsuite and in case of a test failure fail the build.

The way we use it is also quite simple as well. We create a simple application inside the test/testname directory and most of the time just capture the output on stdout. Currently no unit-testing framework is used, instead a simple application is built that is mostly using OSMO_ASSERT to assert the expectations. In case of a failure the application will abort and print a backtrace. This means that in case of a failure the stdout will not not be as expected and the exit code will be wrong as well and the testcase will be marked as FAILED.

The following will go through the details of enabling autotest in a project.

Enabling GNU autotest

Integrating with the automake

 # The `:;' works around a Bash 3.2 bug when the output is not writeable.  
 $(srcdir)/package.m4: $(top_srcdir)/configure.ac  
  :;{   
         echo '# Signature of the current package.' &&   
         echo 'm4_define([AT_PACKAGE_NAME],' &&   
         echo ' [$(PACKAGE_NAME)])' &&;   
         echo 'm4_define([AT_PACKAGE_TARNAME],' &&   
         echo ' [$(PACKAGE_TARNAME)])' &&   
         echo 'm4_define([AT_PACKAGE_VERSION],' &&   
         echo ' [$(PACKAGE_VERSION)])' &&   
         echo 'm4_define([AT_PACKAGE_STRING],' &&   
         echo ' [$(PACKAGE_STRING)])' &&   
         echo 'm4_define([AT_PACKAGE_BUGREPORT],' &&   
         echo ' [$(PACKAGE_BUGREPORT)])';   
         echo 'm4_define([AT_PACKAGE_URL],' &&   
         echo ' [$(PACKAGE_URL)])';   
        } &>'$(srcdir)/package.m4'  
 EXTRA_DIST = testsuite.at $(srcdir)/package.m4 $(TESTSUITE)  
 TESTSUITE = $(srcdir)/testsuite  
 DISTCLEANFILES = atconfig  
 check-local: atconfig $(TESTSUITE)  
  $(SHELL) '$(TESTSUITE)' $(TESTSUITEFLAGS)  
 installcheck-local: atconfig $(TESTSUITE)  
  $(SHELL) '$(TESTSUITE)' AUTOTEST_PATH='$(bindir)'   
  $(TESTSUITEFLAGS)  
 clean-local:  
  test ! -f '$(TESTSUITE)' ||   
  $(SHELL) '$(TESTSUITE)' --clean  
 AUTOM4TE = $(SHELL) $(top_srcdir)/missing --run autom4te  
 AUTOTEST = $(AUTOM4TE) --language=autotest  
 $(TESTSUITE): $(srcdir)/testsuite.at $(srcdir)/package.m4  
  $(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at  
  mv $@.tmp $@  

 AT_INIT  
 AT_BANNER([Regression tests.])  
 AT_SETUP([gsm0408])  
 AT_KEYWORDS([gsm0408])  
 cat $abs_srcdir/gsm0408/gsm0408_test.ok > expout  
 AT_CHECK([$abs_top_builddir/tests/gsm0408/gsm0408_test], [], [expout], [ignore])  
 AT_CLEANUP  

Executing a testsuite and dealing with failure

 $ make testsuite  
 make: `testsuite' is up to date.  
 $ ./testsuite  
 ## ---------------------------------- ##  
 ## openbsc 0.13.0.60-1249 test suite. ##  
 ## ---------------------------------- ##  
 Regression tests.  
  1: gsm0408                     ok  
  2: db                       ok  
  3: channel                     ok  
  4: mgcp                      ok  
  5: gprs                      ok  
  6: bsc-nat                     ok  
  7: bsc-nat-trie                  ok  
  8: si                       ok  
  9: abis                      ok  
 ## ------------- ##  
 ## Test results. ##  
 ## ------------- ##  
 All 9 tests were successful.  

  ...  
  2: db                       FAILED (testsuite.at:13)  
 ...  
 ## ------------- ##  
 ## Test results. ##  
 ## ------------- ##  
 ERROR: All 9 tests were run,  
 1 failed unexpectedly.  
 ## -------------------------- ##  
 ## testsuite.log was created. ##  
 ## -------------------------- ##  
 Please send `tests/testsuite.log' and all information you think might help:  
   To: 
   Subject: [openbsc 0.13.0.60-1249] testsuite: 2 failed  
 You may investigate any problem if you feel able to do so, in which  
 case the test suite provides a good starting point. Its output may  
 be found below `tests/testsuite.dir'.