On getting Free Support

On getting Free Support

Most of the things I know about Software and Hardware I have from reading books, looking at sourcecode but most importantly people willing to answer my questions on IRC and giving me a direction I could look for answers.

Now it seems to be my part to give back and help others to gain knowledge, but some things have changed. Free Software has made it to the mainstream, so besides hackers that want to understand things, we do have paid programmers that don’t want to understand but still need to make it work.

So the other day I was finding me in a IRC query. His target board was a Freescale i.MX27 ADS and his mission was to make it boot from NAND. He was using the OpenEmbedded “mx27ads” machine and had successfully build a kernel and rootfs. Now the problem is that Freescale is not particulary liked in the Free and Open Source Software Community and that Freescale prefers to be on an isolated island. After about 8 hours of spending my time on this, i decided to get back to paid work and carry on.

If you are searching for support on mailinglists, irc channels and irc queries be prepared to think, giving Free Support means that you will be helped to understand the problem and have to pick a solution yourself. If you don’t like that, don’t want to think, don’t have the time to think, you should consider getting someone from the irc channel as consultant.

So here is a list of things that work when paying a consultant but not when you are paid to do your job and you need someone else to do it for you:

  1. Pasting your log somewhere and then ask what is wrong with it. In case of compile failures with GNU make you have to search for ‘***’ in the error log, in case of an early failure of a bitbake run it tells you what software to install, in case of configure failures read the config.log, in case of other failures search the log for ‘Error’, ‘rror’, ‘Failure’, ‘ailure’. And most importantly think before you post it.

    At OpenEmbedded we check for Software being installed on the system. This includes checking for GIT, CVS, SVN and other projects needed to bootstrap. If you don’t have it installed OpenEmbedded will tell you “Error… you don’t have installed:” and the output will be finished by “You will have to install….”. If you paste such an error and ask what the problem is you are really embarassing yourself, your parents, your teachers and your country. The Error message tells you which binaries it searched for and couldn’t find, and it points you to common package names for them. I don’t think it can be any easier for a Software Engineer.

  2. Knowing your hardware. Remember, you have the hardware in front of you, you have it connected somewhere, you have something compiled on your system. So if you are asked about the system specification you will have to point to a PDF, website that names the SoC, the used flash chip (NAND, NOR, erase size, name of it), the SDRAM used and whatever else is used on the system. If you paste a FAQ of something like Buildroot you have clearly failed. Know the stuff that is on your desk, if you don’t nobody can help you.
  3. Pasting unrelated information. When you are asked to paste the output of something. Do not just write that oneline by hand. In many cases we humans apply interpretation to things. When you are asking for help it is an indication that you are not able to interpret the result in a way that lead to a result. Paste the full log, it is saving everyone a lot of time.
  4. Listen/Read to what people tell you. If you have a kernel without NAND support, but want to boot from NAND. You will have to enable NAND and the MXC NAND driver in your kernel. You will have to compile the kernel again, and you will need to boot that kernel. Now we are all humans and have done mistakes before.

    One of the most common mistakes is to not change the config, or not to boot the kernel you have built.

    For the config, check .config after you have built if it is the one you expected it to be, in case of OpenEmbedded copy the .config back to the defconfig (recipes/linux/${PN}-${PV}/${MACHINE}/defconfig). You should copy it back to have a backup in the case you are rebuilding the kernel or such.

    For the second thing it is rather easy. The kernel does contain the TIME it was built and it contains the number of times you have built it from the same directory. So if you rebuild the kernel the TIME and the NUMBER will go up. The kernel does print this information at bootup. If you are asked to check that, don’t paste the log, but check it for yourself. Last but not least, if the NUMBER did not go up or the TIME did not change you have either not rebuilt it, or not from the same directory… And if you didn’t rebuild from the same directory then you are likely to not use the right config…

  5. Don’t do crazy things while people spend their time to help you. Do not remove your build directories and just start over. No, even after your next rebuild the kernel will lack NAND support. That is because it is not enabled in the defconfig for your machine, and the build process is deterministic… You have someone on the other side that decided he wants to help you, if you are not focused, why should he?

And for all of you, that have read until the end. If you decide to seek help in an open forum. First do your homework by having some idea about the problem, have information ready (hardware spec, build logs, whatever), be prepared to think, formulate a hypothesis and try it.

With Free Software you are in the fortunate situation that you can talk to the guys who build the stuff you are using, all you need to do is to be focused on receiving help and be prepared to think.

Looking back to 2009

Looking back to 2009

The second year as part time freelancer has passed.

Looking back the most significant things are:

  • Signing the contribution agreement for gdb and glibc with the Free Software Foundation and trying to contribute to both projects. So picking future work will always have to be compatible with this.
  • Hacked on OpenBSC. At first just simple stuff like a telnet interface, paging and later doing paid work for On Waves to add SCCP over IP, GSM 08.08 and other things for “toy” integration of OpenBSC into a real network.
  • Mid this year I asked Nokia if they have work for me in Asia, later I started focusing on QtWebKit performance. Allowing me to improve QtWebKit and Qt (which will benefit a lot more users), but also to look into various tools like OProfile, memprof, memusagestat and just know netfilter queue’s… more on this later.
  • I have done my usual things on OpenEmbedded, working on landing patches through the patchwork queue, finally redoing the Bitbake parser and working on the Qt recipes.
  • I didn’t manage to make a Linux Kernel contribution. I wanted to write a i2c driver for a fm radio chip but I fried my hardware with a broken power supply, my MIPS patches are not yet done. So if you know of any Kernel work where stuff can be released/upstreamed please let me know!
GSM AMR (Speech Version 3) with OpenBSC

GSM AMR (Speech Version 3) with OpenBSC

This week I had to make parts of OpenBSC work with TCH/H and use AMR. This work is needed for On Waves and when I say parts I mean the strict BSC subset of OpenBSC (in contrast to make the MSC code we have work as well).

The first part is to make TCH/H work and that was easy as LaF0rge did almost everything to make it work. You have to change the OpenBSC configuration to use TCH/H instead of TCH/F for the given timeslots. The next thing was to make channel assignment work. The Mobile Station (MS) comes on the Random Access Channel (RACH) and is asking for a channel and gives a random number (so it can identify the response). Now depending on a global indicator (NECI) the MS will ask for different channels.

So the next step was to add a NECI configuration to our VTY configuration code and then change the code that decodes the channel request to know about the NECI and pick the right channel. On top of that a small hack to assign a TCH/H in case of a MS requesting “any” channel as part of paging.

Now that TCH/H should work one has to focus on the speech. GSM 08.08 and GSM 04.08 have different enums for speech. GSM 08.08 is differenting speech version 1,2,3 for full and half rate totalling in six different values, for GSM 04.08 there is a TCH mode that includes speech version 1,2,3, various data modes and signalling (but no differentation full/half rate channel). After getting this right and selecting speech version 3 it still didn’t work. It turned out that one has to fillout the optional Multirate Configuration when one is using speech version 3. This multi rate configuration needs to be present in the GSM 04.08 RR Assignment Command, Modify Channel but also in the RSL messages for Modify Request and Channel Assignment.

After this AMR on a TCH/H should work (when the BTS is supporting it too). The next step for someone else is to make the MSC code in OpenBSC work with TCH/H and other audio codecs. This would require to stop to ask for a TCH/F, change the channel requested decoding again..

Visiting On Waves in Iceland

Visiting On Waves in Iceland

Currently I’m sitting in the nice offices of On Waves and when not trying to convince the embassy of India to give me a visa I’m working on OpenBSC. For this week I try to make call handling with the MSC rock solid.
So far I have fixed some bugs, added features to OpenBSC, enabled A5/1 encryption, started using TCH/H, started using AMR, fixed bringup on nanoBTS coldstart and now I’m working on the MGCP to verify that I can hear audio for my calls.

Painting on ARM

Painting on ARM

I’m currently work on making QtWebKit faster on ARM (hopefully later MIPS hardware) and in my current sprint I’m focused on the painting speed. Thanks to Samuel Rødal my work is more easy than before. He added a new paintengine and graphicssystem that allows to trace the painting done with QPainter and then later replay that. Some of you might feel reminded of Carl Worth’s post that mostly did the same for cairo.

How to make painting faster? The Setup

  1. Record a paint trace of your favorite app with tst_cycler -graphicssystem trace, do the rendering and on exit the trace will be generated
  2. Use qttracereplay to replay the trace on your hardware (I had some issues on my target hardware though)
  3. Use OProfile to look where the time is spent and do something about it…
  4. Change code go back to qttracereplay..

What did I do so far?
Most samples are recorded in the comp_func_SourceOver routine. With some searching in the MMX optimized routines and talking to the rasterman I’m doing the following things to improve things on the const_alpha=255 path. In the qttracereplay I go from about 17.4 fps to around 26 fps on my beagleboard with Qt Embedded Linux on the plain OMAP3 fb but I still need to do a more careful visual inspection of the result.

  • Handle alpha=0x00 on the source special by not doing anything
  • Handle alpha=0xff on the source special by simply copying it to the dest
  • Unroll the above block eight times interleaved with preloads…

I will have to clean all this up, merge it with the symbian optimized copies (which sometimes require armv6 or later)… I will probably look at BYTE_MUL now and see if I can make it faster without taking a armv6 or later instruction… or honestly first understand how the current BYTE_MUL is working…

memprof 0.6.2 release

memprof 0.6.2 release

Today I have released memprof 0.6.2. The most prominent change is merging a merge of raster’s timegraph for memory allocations and fixing various stability bugs introduced post 0.6.0. The code is currently located on gitorious and the release tarball is here and the shortlog can be seen below:

Cristi Magherusan (2):
some other minor changes, mostly guint -> gsize’s
fixed a typo, bug #51556 in the gnome bugzilla

Holger Hans Peter Freyther (10):
mi-perfctr.c: Remove the O_CREAT (from the openSUSE buildservice)
memprof.glade: Open and save the file
Provide a GtkFileChooseButton to select the executable.
merge rasterman’s extra window
.gitignore: Ignore generated files
process_find_line: Clarify who is owning the returned pointer
detailwin.c: Fix possible crash when opening the maps file fails
process_locate_symbol: Make sure a valid string is always returned
add_leaf_to_tree: Avoid running into a crash
memprof release 0.6.2

Stefan Schmidt (2):
configure.in: Use AM_SILENT_RULES if available
stack-frame: Introduce macros for stack pointer regs and use them.

Tomasz Mon (2):
configure.in: Search for bfd.h provided by binutils development package
Integrate the detailwin into the main GtkNotebook

William Pitcock (1):
use elf_demangle() in more places

memprof-0.6.2

Memory profiling on GNU systems

Memory profiling on GNU systems

This is a small guide on how to observe memory allocations of a process. When carrying out a change it is not only of interest if all test cases still pass, if the benchmarks are faster but it is also important to figure out if there was a change in storage (stack and RAM) requirement.

If you are using the GNU libc it is likely you have a /lib/libmemusage.so installed on your system. This library can be preloaded using LD_PRELOAD and will intercept calls to malloc,free,realloc and various other calls. In short it will trace memory allocations for you. The limitation of that tool is that it will not tell you how much memory the kernel actually mapped, anything about memory fragmentation, etc.

To use libmemusage all you have to do is to prepend MEMUSAGE_OUTPUT=mytrace and LD_PRELOAD=/lib/libmemusage.so to your application. This will instruct the library to write out a trace to the mytrace file.

This trace file can be converted to a graph using the memusagestat utility. It is not installed by most GNU distributions and can be either build from the glibc sources or from the QtWebKit performance measurement utilities. Using memusagestat -o output.png mytrace an image with memory allocations and stack usage like the one at the end of this post will be created. The redline is the heap usage, the green one is the stack usage of the application. The x-scale is the number of allocations.

NOTES: As of today the graph creation with the x-Axis being the time is broken as the generated trace file has some issues, I’m looking into the problem but it will take some more days.

memusage on qtwebkit

Standing on the shoulders of giants – Fixing OpenOCD for the Beagleboard

Standing on the shoulders of giants – Fixing OpenOCD for the Beagleboard

Ever since fixing a little bug in the breakpoint code for the s3c2442 I’m subscribed to the OpenOCD mailinglist. One mail catched my eye, the mail was from a former Openmoko colleague Matt Hsu. He is working with most of the Taiwan Openmoko Team at the newly founded 0xlab and he was trying OpenOCD on the OMAP3 beagleboard.

Yesterday I decided to go to their lab to hack with Matt on the Cortex-A8 support in OpenOCD. After downloading the datasheets, we started to poke the code and tried to get a picture of how things are supposed to work. We have fixed various candidates for failures, like not waiting for the previous instruction to be finished when executing, not waiting for data to be there before reading.

Our symptoms were that after typing “halt” a lot of “invalid mode” messages got printed. After some tracing we ended at the execution of one instruction. The Cortex-A8 supports a mode were an ARM instruction can be written to a register and this instruction will be executed when being in debug mode. Now for a debugger the instruction is copying registers and other data to a special channel that can be read from the debugger. After looking further every register dumped liked this had the value 0x0…After this we were sure to have found the place were things go wrong. All we needed to find out why this instruction is not properly executed.

After studying the datasheet some more, going through the bits of DSCR and DCCR, and looking at the value of DSCR we have, it was clear that we just need to enable execution of instructions from the ITR (Instruction Transfer Register).

After this debugging session I still have little knowledge about ADI and the related things and this is why Free Software is so great, I can spend one afternoon, and can make a big change, and I can do it because I build on support from previous developers!

OpenEmbedded gains ability to create Qt Embedded Linux SDK

OpenEmbedded gains ability to create Qt Embedded Linux SDK

Generating a Qt Embedded Linux SDK is as easy as typing bitbake meta-toolchain-qte. The generated SDK will contains the cross compiler, depedencies needed to recompile Qt Embedded Linux and Qt Emebedded Linux itself.

Building Qt applications for the software is as easy as sourcing a script and invoking qmake2. More documentation can be found in a new section of the OpenEmbedded User manaul.

Using SDKs produced by OpenEmbedded can greatly improve the productivity of development teams as they can start developing their applications right away and in contrast to classic SDKs if a new dependency is needed it can be easily installed using the opkg-target utility.

Tale of a day/morning

Tale of a day/morning

My debug build of libQtWebKit finished and I thought the horror of linking a huge library is over and that I could start to debug. Turned out that gdb was segfaulting when launching my application. Eeeek., google… well googling for gdb and crash is not really finding gdb things. Okay so I was using cvs to get the latest version of gdb, compiling… *crash*. It seemed to crash in demangling, so just in case I was updating binutils and rebuilding gdb. same crash. Hmmm, let us try a debug build… hmmm. Okay one of these 300 char symbols, using the libiberty/testsuite it worked fine… hmmm. using valgrind… using it again and reading the output… oh stack overflow… Turns out alloca is a nice api… there is no error checking and no way to get a breakpoint set… even with -fno-builtin…I was kicking out alloca on my stracktrace from the crash and made it go away, filed a bug and attached a patch, let us see how long this takes to end up in a version. Fixed my QtWebKit within 30 seconds after seeing the stack trace…

I spent the other half of the day/morning with copying the Q.713 data structure into a headerfile and sketching out my header file for a SCCP lite GSM A-Interface implementation. I hope I can publish this soon. Before starting to implement this I will go back to do some WebKit performance analysis…

And with all the GNU bashing going on, I wonder if GNOME 3.0 will be called NOME 3.0… This reminds me of Goethe and the “Die Geister, die ich ruf werd ich nicht mehr los”.