Finally trying the OpenSUSE build service

Finally trying the OpenSUSE build service

For the upcoming SIMtrace hardware (e.g. see the communication between SIM and Phone) I want to provide a patched Wireshark and packages for our host utilities to make the installation (and upgrading) very easy.

Being a happy Ubuntu user I started with launchpad, trying to create my own wireshark package. It was very easy to get source of the wireshark package, add a patch, upload it to my ppa. Then I spent some hours figuring out that launchpad does not allow 3.0 (git) git packages (it simply rejects the upload without a proper error message). My biggest issue is to figure how with one upload I could end up with packages for multiple series. E.g. simply copying the package to another series does not work due version numbers (and the need to be able to easily upgrade). So how do people provide to multiple series?

The next thing I tried was the OpenSUSE Build Service. I know that Cornelius is working on it for many years but so far I have always failed using it via the Web Interface. This time I started with an installation of OpenSUSE and used the osc application. The documentation is very sparse (there are many commands and some inconsistencies in naming) and I decided to start with the easy task of modifying an existing package. I have picked wireshark again (it was a bit difficult to find the authorative copy of it), added my patch, uploaded, and I ended up with SuSE packages. The next task was to build packages for the other RPM based distributions and as expected just adding a repository for Fedora didn’t work. But here comes the power of this system. First of all the interface shows me that I have a build failure or some missing dependencies and using osc build DISTRO ARCH I can start a local build and take a look at things. awesome! And the build farm of OpenSUSE is fast as well!

The biggest problem I have with OBS is finding documentation, or google not showing it to me. E.g. the cross distribution howto always fails to come up in searches but is very valuable when trying to make your package work on different distributions.

One thing that impressed me a lot was adding a new package. I used osc mkpac placed my sourcecode there and then I had to write my first ever RPM spec file from scratch. So I typed vim libosomocore.spec and somehow I had a skeleton of the spec file present. This is very nice!

OsmoST SIP (SIP in Smalltalk)

OsmoST SIP (SIP in Smalltalk)

Intro

The last couple of days, to some degree weeks I was implementing a SIP stack for GNU Smalltalk to be used in my Smalltalk GSM project. The first time I encountered SIP was around 2001 when we were excited to run a SIP Phone on our Linux powered iPAQs (Linphone on Opie).

For Smalltalk I started with looking at SipStack by Frank Shearar (who was very responsive to questions regarding his code and SIP in general). The main problem was that his stack was incomplete and as usual it is difficult to pick things up without knowing SIP and not knowing where the code was heading.

Grammar

I began with mapping the BNF SIP Grammar to rules for the PetitParser parsing framework. I have shortened some rules (e.g. because the generic rule is a catch-all one and I don’t need the specialized form yet).

Complexity

The next big task was to deal with the concepts of a SIP Dialog, a SIP Session, a SIP Transaction and getting the relationship of these entities right. E.g. to place a call one creates an unconfirmed Dialog, prepares the INVITE Transaction and passes it to the transport layer. During this transaction one can get a reply that leads to a confirmation of the dialog. In fact one can end up with multiple confirmed dialogs (called forking, the target phones start ringing in multiple places and can be picked up multiple times as well). To make matters worse some answers need to be sent sent within the same transaction, some open a new one. My code seems to work now but there are some things I don’t do properly (e.g. routing). I have documented this and if I have a need for those I will address them.

Processes

In the last couple of years I was mostly dealing with a single select IO model, coming to Smalltalk makes me use Processes to read and write from sockets. My basic model is to have a RX Process to use Socket>>#next and a TX process that will do SharedQueue>>#next and then Socket>>#nextPutAll. But with having multiple processes one is required to do proper locking, e.g. have a simple locking hierarchy (first pick the lock of the CallAgent, then the lock of a Dialog/Session). This all works nicely until I reached the point that I have two sub systems talking to each other.

For setting up a call I might have one leg coming from GSM and use the MGCP Protocol, the other leg might be SIP. To hang-up I will need to inform the other leg of the call about it, but this means I will need to have a locking hierarchy that first takes all locks of the left leg of a call, then all locks of the right one which in turns means me I need to dispatch messages without any locks being held. Now besides having RX from maybe a couple of different Processes, I also deal with timeouts that will come from different Processes as well.

Central dispatch

To put an end to this (and the complications in the code) I created a simple queue that will BlockClosure>>#value a block from a central dispatch process, this way I have serialized everything and can mostly ignore locking. I hope that Smalltalk VMs will have a nice M:N Process model in the future and that my code will evolve into having different dispatchers running on these N native processes.

Using the code

st> PackageLoader fileInPackage: #OsmoSIP.
st> transport :=SIPUdpTransport startOn: ‘172.16.254.55’ port: 5061.
st> useragent := SIPUserAgent createOn: transport.
st> transport start. “Starts the RX/TX process”

st> call := SIPCall fromUser: ‘sip:1000@on-waves.com’
host: ‘172.16.1.72’ port: 5060 to: ‘sip:9198@172.16.1.72’ on: useragent.
st> call createCall: ‘SomeSDP file’.
st> call cancel. “E.g. before it is connected”
st> call hangup. “E.g. when the call is connected.”
st> call terminate. “End the call somehow..”

Cross compiling Qt and dealing with teams

Cross compiling Qt and dealing with teams

Let me share some notes on how Embedded Linux programming can be, if you decide the war/pain is over. We do this by looking at the desktop build and then look at the tons of stuff that you need for embedded programming.

Desktop

$ pkg-config –cflags libpng12
…
$ qmake && make
…

Embedded
$ . /usr/local/angstrom/arm/environment
$ pkg-config –cflags libpng12..
…
$ qmake-qt4 && make

Where is the difference?
There is little, you just use a cross compiler. The GNU/Linux Toolchain is prepared for cross compilation, pkg-config works with cross compilation.

What if I want different options
You can decide to use different -mtune and -march values, or you can easily regenerate the SDK targeting a different CPU architecture or just the instruction set level of it.

How is this possible
One creates a level of abstraction. The question is why do you want to suffer and don’t use the abstraction known from the desktop? The above is given to you by the Yocto/OpenEmbedded and details can be found in the OpenEmbedded Manual. Read here for more details on creating a Qt Toolchain/SDK and using it.

But I want to recompile Qt
Maybe you want to change the qconfig.h, or you have a bug in Qt where you want to test things more interactively. You could either do this in the build directory of Qt of Yocto/OpenEmbedded but then again the created SDK contains all the dependencies for building Qt. All that is missing is the generic qmake spec (that takes its values from the environment) that would need to be copied into mkspecs.

But my application needs libpainful-to-build
The SDK is deployed from header files of packages, you can install more -dev packages into your toolchain. If you use any of armv4t, armv5te, armv6, armv6-nofp, armv7, armv7a, … chances are high that there is already a build for your machine…

So why is it so painful?
I really don’t know, tell me.

Going to the Qt Contributors’ Summit

Going to the Qt Contributors’ Summit

I will leave Reykjavik around midnight, arrive in the early morning in Berlin and then try to get to the venue. This means I will most likely miss the keynotes but should be there by lunch time.

Besides being interested in technical topics like the networking stack, signals for Qt5, I think I will try to focus on the license part of the contribution model. With recent events like Lodys going after individual developers I wonder if there is something Nokia can do to protect individual developers that acted in good faith.

Calligra Flow

Calligra Flow

Currently I am mostly working on (FreeSoftware) GSM networks, adding software probes, monitors, filtering and patching of certain messages. With any complex system this is spanning across multiple applications, multiple hosts.

Because we collectively know that a picture is worth a thousand words my technical documentation should have a picture, showing the computers, their IPs, some details of how the messages flow and such. Doing this kind of work with Free Software is still a lot harder than it should be.

Currently I am getting the job done with dia, the Cisco stencils are great, many other parts (printing, exporting, scaling, text editing) are not as great and this is where Calligra Flow enters the picture. I compiled a version some months ago and there were almost no stencils, printing was broken…

Yesterday I compiled a newer version of Calligra and there is some very nice progress with Flow, printing does work, saving did work too (no crashes), there are a tons stencils. So thanks a lot for the progress and I will try again in some months.

Collection of WebKit ports

Collection of WebKit ports

WebKit is a very successfull project. It is that in many ways. The code produced seems to very fast, the code is nice to work on, the people are great, the partys involved collaborate with each other in the interest of the project. The project is also very successfull in the mobile/smartphone space. All the major smartphone platforms but Windows7 are using WebKit. This all looks great, a big success but there is one thing that stands out.

From all the smartphone platforms no one has fully upstreamed their port. There might be many reasons for that and I think the most commonly heard reason is the time needed to get it upstreamed. It is specially difficult in a field that is moving as fast as the mobile industry. And then again there is absolutely no legal obligation to work upstream.

For most of today I collected the ports I am aware of, put them into one git repository, maybe find the point where they were branched, rebase their changes. The goal is to make it more easy to find interesting things and move them back to upstream. One can find the combined git tree with the tags here. I started with WebOS, moved to iOS, then to Bada and stopped at Android as I would have to pick the sourcecode for each android release for each phone from each vendor. I think I will just be happy with the Android git tree for now. At this point I would like to share some of my observations in the order I did the import.

Palm

Palm’s release process is manual. In the last two releases they call the file .tgz but forgot to gzip it, in 2.0.0 the tarball name was in camel case. The thing that is very nice about Palm is that they provide their base and their changes (patch) separately. From looking at the 2.1.0 release it looks that for the Desktop version they want to implement Complex Font rendering. Earlier versions (maybe it is still the case) lack the support for animated GIF.

iOS

Apple’s release process seems to be very structured. The source can be downloaded here. What I think is to note is that the release tarball contains some implementations of WebCore only as .o file and Apple has stopped releasing the WebKit sourcecode beginning with iOS 4.3.0.

Bada

This port is probably not known by many. The release process seems to be manual as well, the name of directories changed a lot between the releases, they come with a WML Script engine and they do ship something they should not ship.

I really hope that this combined tree is useful for porters that want to see the tricks used in the various ports and don’t want to spend the time looking for each port separately.

Random notes on building Android

Random notes on building Android

This is mostly a note to myself. The easy way to build a profilable image for Android (e.g. Gingerbread). I am applying the patch from the bottom to build with framepointers. I am build an unstripped image, I use the environment script and then launch the emulator. With the default engineering mode opcontrol and oprofiled will be installed only the events will be missing from the image.

$ make TARGET_STRIP_MODULE=false
$ . ./build/envsetup.sh
$ help
$ setpaths
$ ./out/host/linux-x86/bin/emulator

diff --git a/core/combo/TARGET_linux-arm.mk b/core/combo/TARGET_linux-arm.mk
index ae1997c..66e55fd 100644
--- a/core/combo/TARGET_linux-arm.mk
+++ b/core/combo/TARGET_linux-arm.mk
@@ -56,7 +56,7 @@ TARGET_LD := $(TARGET_TOOLS_PREFIX)ld$(HOST_EXECUTABLE_SUFFIX)
 TARGET_NO_UNDEFINED_LDFLAGS := -Wl,--no-undefined
 
 TARGET_arm_CFLAGS :=    -O2 
-                        -fomit-frame-pointer 
+                        -fno-omit-frame-pointer 
                         -fstrict-aliasing    
                         -funswitch-loops     
                         -finline-limit=300
@@ -68,7 +68,7 @@ TARGET_arm_CFLAGS :=    -O2 
 ifeq ($(ARCH_ARM_HAVE_THUMB_SUPPORT),true)
 TARGET_thumb_CFLAGS :=  -mthumb 
                         -Os 
-                        -fomit-frame-pointer 
+                        -fno-omit-frame-pointer 
                         -fno-strict-aliasing 
                         -finline-limit=64
 else
Switching back to Ubuntu/Debian from Fedora

Switching back to Ubuntu/Debian from Fedora

My initial reason to try Fedora Linux was that a lot of good things (glibc, SELinux, systemtap, gdb) are mostly developed by Red Hat and put into Fedora. So it felt natural to try Fedora Linux to see the state of things. Now after more than a year with Fedora I am back to Ubuntu/Debian. For some simple reasons:

yum is ultra slow. When I try to search a package it decides to update the package database over the network, it is slow to start, it is slow to carry out simple operations. The other is that the various databases on the system are quite big and take a lot of space.

My upgrade to Fedora15 was hosed. Not so much to complain about as the distribution is not meant to be released the next week or such. So I was experiencing integration issues (e.g. setroubleshootd causing SELinux denies which will start the settroubleshootd), systemd not logging to /var/log/messages.

With Fedora15 I also had the opportunity to try GNOME 3.0 and in many ways it is promising, I don’t think it is ready for end users yet though.

The biggest issue with Ubuntu right now is to get SELinux up and running, their policy modules are quite outdated.

Playing with a MSC over the A-link

Playing with a MSC over the A-link

One of the benefits of having a dynamic language like Smalltalk is that it becomes easier to experiment. For one of my current projects I am dealing with some simple USSD message handling and want to see how much load my application and the backend system can handle. Using the Smalltalk TestPhone this becomes quite easy.

The first thing I do is to make up an Identity. This includes the IMSI and the AuKey (e.g. COMPv1 as found in libosmocore) and the next is to establish the A-link to the MSC.

> phoneConfig := PhoneConfig new.
> phoneConfig imsi: ‘XXXX…’ auKeyv1: ‘AAAAAAA’.
> (server := IPAConfig new addr: ‘A.B.C.D’ port: 5000) connect serve.

The next part is to do ‘procedures’ (SCCP connections with a specific goal). Right now there is an implementation of the LU Procedure, the beginning of Call Control (no way to hang up or such) and and sending some bits of USSD (processUnstructuredData Request). To warm up I can execute a simple USSD query. The below code will wait until the operation has completed, it could also print out if it was considered a success or a failure.

> (server doUSSD: phoneConfig nr: ‘**1234#’) execute

Now to produce load I can repeat this, or create multiple processes. In the below code I am going to create four processes that each will do the query 100 times.

> 4 timesRepeat: [:time |
[
100 timesRepeat: [:each | (server doUSSD: phoneConfig nr: ‘**1234#’) execute]
] fork.

Another thing missing is to add a timeout on each of these operations, e.g. if the MSC does not answer fast enough or such but I will blog about transactions and dealing with time-outs in another blog.

Tsunamies and other things

Tsunamies and other things

This is just some random thoughts after the news from Japan, hiking the Taroko National Park in Taiwan, driving with the stop train on the Coast Line in Taiwan, studying chinese, and being away from Computers for a day or two.

My idea for the tsunami would be something like an instant balloon. In Taiwan every floor has an emergency exit through the window and a device to use to safely get to the ground, there is also the famous lantern festival were hundreds of them fly to the sky. So in my idea groups could board an emergency balloon and escape the water, and some minutes later land on the more calm water.

Angelique’s idea was having a transparent Tsunami Ball like structure. This way one would float on the water and the structure would be strong enough to withhold the enormous pressure the moving water has.

On something unrelated I have been playing a bit with Android, a very badish game idea, and XMLVM. So it appears that JavaScript is somehow getting the new low-level virtual machine language.