Memory profiling on GNU systems

Memory profiling on GNU systems

This is a small guide on how to observe memory allocations of a process. When carrying out a change it is not only of interest if all test cases still pass, if the benchmarks are faster but it is also important to figure out if there was a change in storage (stack and RAM) requirement.

If you are using the GNU libc it is likely you have a /lib/libmemusage.so installed on your system. This library can be preloaded using LD_PRELOAD and will intercept calls to malloc,free,realloc and various other calls. In short it will trace memory allocations for you. The limitation of that tool is that it will not tell you how much memory the kernel actually mapped, anything about memory fragmentation, etc.

To use libmemusage all you have to do is to prepend MEMUSAGE_OUTPUT=mytrace and LD_PRELOAD=/lib/libmemusage.so to your application. This will instruct the library to write out a trace to the mytrace file.

This trace file can be converted to a graph using the memusagestat utility. It is not installed by most GNU distributions and can be either build from the glibc sources or from the QtWebKit performance measurement utilities. Using memusagestat -o output.png mytrace an image with memory allocations and stack usage like the one at the end of this post will be created. The redline is the heap usage, the green one is the stack usage of the application. The x-scale is the number of allocations.

NOTES: As of today the graph creation with the x-Axis being the time is broken as the generated trace file has some issues, I’m looking into the problem but it will take some more days.

memusage on qtwebkit

Standing on the shoulders of giants – Fixing OpenOCD for the Beagleboard

Standing on the shoulders of giants – Fixing OpenOCD for the Beagleboard

Ever since fixing a little bug in the breakpoint code for the s3c2442 I’m subscribed to the OpenOCD mailinglist. One mail catched my eye, the mail was from a former Openmoko colleague Matt Hsu. He is working with most of the Taiwan Openmoko Team at the newly founded 0xlab and he was trying OpenOCD on the OMAP3 beagleboard.

Yesterday I decided to go to their lab to hack with Matt on the Cortex-A8 support in OpenOCD. After downloading the datasheets, we started to poke the code and tried to get a picture of how things are supposed to work. We have fixed various candidates for failures, like not waiting for the previous instruction to be finished when executing, not waiting for data to be there before reading.

Our symptoms were that after typing “halt” a lot of “invalid mode” messages got printed. After some tracing we ended at the execution of one instruction. The Cortex-A8 supports a mode were an ARM instruction can be written to a register and this instruction will be executed when being in debug mode. Now for a debugger the instruction is copying registers and other data to a special channel that can be read from the debugger. After looking further every register dumped liked this had the value 0x0…After this we were sure to have found the place were things go wrong. All we needed to find out why this instruction is not properly executed.

After studying the datasheet some more, going through the bits of DSCR and DCCR, and looking at the value of DSCR we have, it was clear that we just need to enable execution of instructions from the ITR (Instruction Transfer Register).

After this debugging session I still have little knowledge about ADI and the related things and this is why Free Software is so great, I can spend one afternoon, and can make a big change, and I can do it because I build on support from previous developers!

People should have stayed with Typewriters… or why DHL sucks

People should have stayed with Typewriters… or why DHL sucks

Todays epic fail of DHL. When shipping a package you will have to leave a phone number to DHL. My dear reader, don’t worry, this phone number will never be used to call the receiver of the package in case of an error.

So today I was checking the website where the package is and I’m supposed to call the DHL customer service. And in all glory of the internet and HTML they are not able to embed a link to the customer service package, in case there is no useful link on the tracking page.

Using my geek power I made a google query for “MyCountry dhl customer service”. After loading of a java applet, and then selecting my country again, I’m on a page to show me the phone number of my local customer service. Oh my dear reader, if you think the odysee stops here you are wrong… The number on this page triggers a voice message that I can call this other number, this other number gives me another number, and this others number number gives me a number to a private person…

DHL… how hard can it be to have a working and DHL Customer Service phone number…

Embedded syntax highlighted sourcecode into OpenOffice.org Write

Embedded syntax highlighted sourcecode into OpenOffice.org Write

The other day I was in need to embed some sourcecode into an OpenOffice.org Document. Normally I’m using LaTex and then can use the lgrind tool to prepare my sourcecode and use lgrindfile{file} to embed it. Well but I’m somehow required to use OpenOffice.org Writer to generate a MS Office .doc file.

After some googling the GNU Source-highlight project came to rescue. I can syntax highlight my source code and patches, then I can use a HTML renderer to render these pages, select and copy the text and finally paste it into OpenOffice.org writer.

It works great!

WebKit on non mainstream platforms

WebKit on non mainstream platforms

We have seen some interest in MIPS on the mailinglist and little in the bugtracker but no contribution from these folks. So the other week I decided to attempt to run some of our regression tests on a MIPS platform. The only problem is that I don’t have any MIPS hardware… can you change that?

I had this idea of implementing enough of MacroAssembler to run/compile yarr with the RegexJIT.cpp. In the absence of MIPS hardware I used qemu-mipsel useremulation, enabled yarr/RegexInterpreter.cpp and ran into an infinite loop. I have fixed the bug and landed the patch upstream and now I’m able to complete run-javascriptcore-tests using yarr. The next step will be to work on the MacroAssembler and learn about MIPS assembly…. maybe I progress a bit more next weekend.

Searching for hardware donations

Searching for hardware donations

In the last couple of weeks I have increased my involvement with OpenEmbedded again. I have worked on ARM/x86, keeping uclibc building, getting more things build with uclibc, worked on Qt, started merging patches, specially the MIPS side of things have catched my eye. So everything is fine… if there wouldn’t be a little twist.

I don’t have any ARM or MIPSEL targets at home. For my work on Openembedded and WebKit I’m searching for one MIPS and one ARM board, with network connectivitiy and working oprofile support. Anyone has a board spare and is able to ship it to Taiwan? Having a board would actually allow me to run the code I write for ARM and MIPS.

thanks

OpenEmbedded gains ability to create Qt Embedded Linux SDK

OpenEmbedded gains ability to create Qt Embedded Linux SDK

Generating a Qt Embedded Linux SDK is as easy as typing bitbake meta-toolchain-qte. The generated SDK will contains the cross compiler, depedencies needed to recompile Qt Embedded Linux and Qt Emebedded Linux itself.

Building Qt applications for the software is as easy as sourcing a script and invoking qmake2. More documentation can be found in a new section of the OpenEmbedded User manaul.

Using SDKs produced by OpenEmbedded can greatly improve the productivity of development teams as they can start developing their applications right away and in contrast to classic SDKs if a new dependency is needed it can be easily installed using the opkg-target utility.

Fun on the tech-bsd mailinglist

Fun on the tech-bsd mailinglist

For various reasons I’m subscribed to the tech bsd mailinglist and this mail made me smile. Let me quote some parts of it.

“I don’t know who you are, but you do nothing. What do you do?
I don’t see your name on any the source code.
The source code does define the behaviour. Exactly. Perhaps the
source code is wrong, but it *EXACTLY DEFINES THE BEHAVIOUR*.
So you shut up, loser. Just go away.”
Theo de Raadt

Tale of a day/morning

Tale of a day/morning

My debug build of libQtWebKit finished and I thought the horror of linking a huge library is over and that I could start to debug. Turned out that gdb was segfaulting when launching my application. Eeeek., google… well googling for gdb and crash is not really finding gdb things. Okay so I was using cvs to get the latest version of gdb, compiling… *crash*. It seemed to crash in demangling, so just in case I was updating binutils and rebuilding gdb. same crash. Hmmm, let us try a debug build… hmmm. Okay one of these 300 char symbols, using the libiberty/testsuite it worked fine… hmmm. using valgrind… using it again and reading the output… oh stack overflow… Turns out alloca is a nice api… there is no error checking and no way to get a breakpoint set… even with -fno-builtin…I was kicking out alloca on my stracktrace from the crash and made it go away, filed a bug and attached a patch, let us see how long this takes to end up in a version. Fixed my QtWebKit within 30 seconds after seeing the stack trace…

I spent the other half of the day/morning with copying the Q.713 data structure into a headerfile and sketching out my header file for a SCCP lite GSM A-Interface implementation. I hope I can publish this soon. Before starting to implement this I will go back to do some WebKit performance analysis…

And with all the GNU bashing going on, I wonder if GNOME 3.0 will be called NOME 3.0… This reminds me of Goethe and the “Die Geister, die ich ruf werd ich nicht mehr los”.

Fun reading landline telephony specs

Fun reading landline telephony specs

For various reasons I have started reading the GSM 08.0x chapter. It is mostly about communication between a Base Station System (BSS) and a Mobile Switching Center (MSC). The protocol used is actually based on the SCCP protocol from landline telephony networks (SS #7). The most funny part is the way these specs are written. Most sentences in GSM 08.06 end in “this is of national concern”, or in the Q.71x specification you see classes of reserved fields. One for international use and one for national use where the national telephone monopoly can set the bits they want…. Just by reading the specs you figure out that international telephony is about monopolies communicating to each other, that is so different to the TCP RFCs…