Analyze cellular problems using Quectel modules

Analyze cellular problems using Quectel modules

Introduction

Previously I have written about connectivity options for IoT devices and today I assume that a cellular technology (e.g. names like GSM, 3G, UMTS, LTE, 4G) has been chosen. Unless you are a big vendor you will end up using a module (instead of a chipset) and either you are curious what the module is doing behind its AT command interface or you are trying to understand a real problem. The following is going to help you or at least be entertaining.

The xgoldmon project was a first to provide air interface traces and logging to the general public but it was limited to Infineon baseband (and some Gemalto devices), needed special commands to enable and didn’t include all messages all the time.

In the last months I have intensively worked with modules of a vendor called Quectel. They are using Qualcomm chipsets and have built the GSM/UMTS Quectel UC20 and the GSM/UMTS/LTE Quectel EC20 modules. They are available as a variant to solder but for speeding up development they provide them as miniPCI express as well. I ended up putting them into a PCengines APU2, soldered an additional SIM card holder for the second SIM card, placed U.FL to SMA connectors and put it into one of their standard cases. While the UC20 and EC20 are pretty similar the software is not the same and some basic features are missing from the EC20, e.g. the SIM ToolKit support. The easiest way to acquire these modules in Europe seems to be through the above links.

The extremely nice feature is that both modules export Qualcomm’s bi-directional DIAG debug interface by USB (without having to activate it through an undocumented AT command). It is a framed protocol with a simple checksum at the end of a frame and many general (e.g. logging and how regions are described) types of frames are known and used in projects like ModemManager to extract additional information. Some parts that include things like Tx-power are not well understood yet.

I have made a very simple utility available on github that will enable logging and then convert radio messages to the Osmocom GSMTAP protocol and send it to a remote host using UDP or write it to a pcap file. The result can be analyzed using wireshark.

Set-up

You will need a new enough Linux kernel (e.g. >= Linux 4.4) to have the modems be recognized and initialized properly. This will create four ttyUSB serial devices, a /dev/cdc-wdmX and a wwanX interface. The later two can be used to have data as a normal network interface instead of launching pppd. In short these modules are super convenient to add connectivity to a product.

apu2_quectel_uc20_ec20
PCengines APU2 with Quectel EC20 and Quectel UC20

Building

The repository includes a shell script to build some dependencies and the main utility. You will need to install autoconf, automake, libtool, pkg-config, libtallocmake, gcc on your Linux distribution.

[shell]
git clone git://github.com/moiji-mobile/diag-parser
cd diag-parser
./build/build_local.sh
[/shell]

Running

Assuming that your modem has exposed the DIAG debug interface on /dev/ttyUSB0 and you have your wireshark running on a system with the internal IPv4 address of 10.23.42.7 you can run the following command.

[shell]
./diag-parser -g 10.23.42.7 -i /dev/ttyUSB0
[/shell]

Exploring

Analyzing UMTS with wireshark. The below shows a UMTS capture taken with the Quectel module. It allows you to see the radio messages used to register to the network, when sending a SMS and when placing calls.

diag-parse_rantrace
Wireshark dissecting UMTS
Know your tools – mudflap

Know your tools – mudflap

I am currently implementing GSM ARFCN range encoding and I do this by writing the algorithm and a test application. Somehow my test application ended in a segmentation fault after all tests ran. The first thing I did was to use gdb on my application:

$ gdb ./si_test
(gdb) r
...
Program received signal SIGSEGV, Segmentation fault.
0x00000043 in ?? ()
(gdb) bt
#0  0x00000043 in ?? ()
#1  0x00000036 in ?? ()
#2  0x00000040 in ?? ()
#3  0x00000046 in ?? ()
#4  0x00000009 in ?? ()
#5  0xb7ff6821 in ?? () from /lib/ld-linux.so.2

The application crashed somewhere in glibc on the way to the exit. The next thing I used was valgrind but it didn’t report any invalid memory access so I had to resort to todays tool. It is called mudflap and part of GCC for a long time. Let me show you an example and then discuss how valgrind fails and how mudflap can help.

int main(int argc, char **argv) {
  int data[23];
  data[24] = 0;
  return 0;
}

The above code obviously writes out of the array bounds. But why can’t valgrind detect it? Well we are writing somewhere to the stack and this stack has been properly allocated. valgrind can’t know that &data[24] is not part of the memory to be used by data.

mudflap comes to the rescue here. It can be enabled by using -fmudflap and linking to -lmudflap this will make GCC emit extra code to check all array/pointer accesses. This way GCC will track all allocated objects and verify the access to memory before doing it. For my code I got the following violation.

mudflap violation 1 (check/write): time=1350374148.685656 ptr=0xbfd9617c size=4
pc=0xb75e1c1e location=`si_test.c:97:14 (range_enc_arfcns)'
      /usr/lib/i386-linux-gnu/libmudflap.so.0(__mf_check+0x3e) [0xb75e1c1e]
      ./si_test() [0x8049ab5]
      ./si_test() [0x80496f6]
Nearby object 1: checked region begins 29B after and ends 32B after
mudflap object 0x845eba0: name=`si_test.c:313:6 (main) ws'
I am presented with the filename, line and function that caused the violation, then I also get a backtrace, the kind of violation and on top of that mudflaps informs me which objects are close to the address I allocated. So in this case I was writing to ws outside of the bounds.