QtVirtualKeyboard on Wayland

QtVirtualKeyboard on Wayland

For the last couple of years my focus was on the Osmocom project to bring Free Software to the world of telecommunication. With a group of enthusiasts we have implemented the components necessary to run a complete network using Free Software. The Rhizomatica project is using the software to connecting people that were left behind. Our tools enabled high impact security research leading, leading to improvements to privacy and security for all of us….

But during the last months I had the opportunity to return to C++ and Qt work and it feels like coming home to the world of ARM powered hardware. When I left, the transition from consumer electronics (e.g. settop boxes) to automative (e.g. IVI) began and it seems it successfully completed! On Friday I explored a regression in OpenSSL and today I had the pleasure to understand input method handling of wayland a little bit better.

I wanted to see if I can use wayland and run QtVirtualKeyboard only in the Compositor. I couldn’t find answers in the documentation and started to read the code. Once I understood how it should work, I found a simple example in QtWayland. Isn’t Qt wonderful?

As I have gone down the rabbit hole let me try to share some of my reading. At the core is the unstable (API subject to change) text-input protocol. Upstream of wayland-protocols is still at v1 of the protocol but QtWayland includes a copy of v2 and has rather complete support for the compositor and client (thanks KDAB!).

Compositor/Server

To make it work the compositor needs to signal support for the “zwp_text_input_manager_v2” interface. In QML/Quick this needs to be done by adding the following inside the WaylandCompositor component:

...
TextInputManager {
}
...

This will instantiate a QWaylandTextInputManager and call ::initialize on it. Then auto-generated code will use wl_global_create to register the interface in the global registry. Clients will request a text_input and the manager will create one or use a QWaylandTextInput and then send events down to the client. E.g. if there is a soft keyboard set a breakpoint in QWaylandTextInput::sendKeyEvent in the compositor and see from where it is coming and how it will send the event to a client.

Client

Once the client starts the QWaylandIntegration will create a QWaylandDisplay and then query/initialize the registry. The answer will call QWaylandDisplay::registry_global for every registered interface. For the the zwp_text_input_manager_v2 interface the integration will initialize a QtWayland::zwp_text_input_manager_v2 and for each input device a QtWaylandClient::QWaylandTextInput (different namespace) will be created.

Communication

Now Compositor and Client will communicate through the wayland connection (no dbus). With Qt APIs the client will notice which element has been focused and will request the input method to be shown (QtWaylandClient::QWaylandInputContext::showInputPanel) and the Compositor  will forward this to the QInputMethod (see QWaylandTextInputPrivate::zwp_text_input_v2_show_input_panel)

Putting it together

In the end this is rather simple.. create a TextInputManager, use the InputPanel of the QtVirtualKeyboard and start your compositor with QT_IM_MODULE=qtvirtualkeyboard. Done!

 

DirectFB contribution to the Qt Project

DirectFB contribution to the Qt Project

The Qt project was launched today, I got my 15 DirectFB patches merged, got some first experience with Gerrit, created an account for the Qt wiki, fixed some documentation, so all in all I think it is the great start of the Qt Project we have been waiting for! So thank you very much for all you involved with it!

Now to something completely else, somehow I like to see the parts that are not great yet. But most importantly it is a great opportunity for everyone to get involved with the Qt project and improve things. So here is my short wishlist for the Qt project.

  • Single account for the Bugtracker, Wiki, other services.

  • Read-Only access to gerrit.

  • Public CI based on Jenkins, right now build failures will still point to internal Nokia servers. I assume KDE can help a bit with the Jenkins setup.

  • Make it possible for non-mainstream QPA backends and platforms to be part of the CI System, if proven stable be considered core builders.

  • The wiki being part of the Qt project should be part of the Qt Project, the license should probably be made compatible with the license of the Qt documentation, to allow copying from one to the other.

Once again, thank you Nokia, thanks everyone involved!

Calligra Flow

Calligra Flow

Currently I am mostly working on (FreeSoftware) GSM networks, adding software probes, monitors, filtering and patching of certain messages. With any complex system this is spanning across multiple applications, multiple hosts.

Because we collectively know that a picture is worth a thousand words my technical documentation should have a picture, showing the computers, their IPs, some details of how the messages flow and such. Doing this kind of work with Free Software is still a lot harder than it should be.

Currently I am getting the job done with dia, the Cisco stencils are great, many other parts (printing, exporting, scaling, text editing) are not as great and this is where Calligra Flow enters the picture. I compiled a version some months ago and there were almost no stencils, printing was broken…

Yesterday I compiled a newer version of Calligra and there is some very nice progress with Flow, printing does work, saving did work too (no crashes), there are a tons stencils. So thanks a lot for the progress and I will try again in some months.

How to make the GNU Smalltalk Interpreter slower

How to make the GNU Smalltalk Interpreter slower

This is another post about a modern Linux based performance measurement utility. It is called perf, it is included in the Linux kernel sources and it entered the kernel in v2.6.31-rc1. In many ways it is obsoleting OProfile, in fact for many architectures oprofile is just a wrapper around the perf support in the kernel. perf comes with a few nice application. perf top provides a statistics about which symbols in user and in kernel space are called, perf record to record an application or to start an application to record it and then perf report to browse this report with a very simple CLI utility. There are tools to bundle the record and the application in an archive, a diff utility.

For the last year I was playing a lot with GNU Smalltalk and someone posted the results of a very simplistic VM benchmark ran across many different Smalltalk implementations. In one of the benchmarks GNU Smalltalk is scoring last among the interpreters and I wanted to understand why it is slower. In many cases the JavaScriptCore interpreter is a lot like the GNU Smalltalk one, a simple direct-threaded bytecode interpreter, uses computed goto (even is compiled with -fno-gcse as indicated by the online help, not that it changed something for JSC), heavily inlined many functions.

There are also some differences, the GNU Smalltalk implementation is a lot older and in C. The first notable is that it is a Stack Machine and not register based, there are global pointers for the SP and the IP. Some magic to make sure that in the hot loop the IP/SP is ‘local’ in a register, depending on the available registers also keep the current argument in one, the interpreter definition is in a special file format but mostly similar to how Interepreter::privateExecute is looking like. The global state mostly comes from the fact that it needs to support switching processes and there might be some event during the run that requires access to the IP to store it to resume the old process. But in general the implementation is already optimized and there is little low hanging fruits and most experiments result in a slow down.

The two important things are again: Having a stable benchmark, having a tool to help to know where to look for things. In my case the important tools are perf stat, perf record, perf report and perf annotate. I have put a copy of the output to the end of this blog post. The stat utility provides one with number of instructions executed, branches, branch misses (e.g. badly predicted), L1/L2 cache hits and cache misses.

The stable benchmark helps me to judge if a change is good, bad or neutral for performance within the margin of error of the test. E.g. if I attempt to reduce the code size the instructions executed should decrease, if I start putting __builtin_expect.. into my code the number of branch misses should go down as well. The other useful utility is to the perf report that allows one to browse the recorded data, this can help to identify the methods one wants to start to optimize, it allows to annotate these functions inside the simple TUI interface, but does not support searching in it.

Because the codebase is already highly optimized any of my attempts should either decrease the code size (and the pressure on the i-cache), the data size (d-cache), remove stores or loads from memory (e.g. reorder instructions), fix branch predictions. The sad truth is that most of my changes were either slow downs or neutral to the performance and it is really important to undo these changes and not have false pride (unless it was also a code cleanup or such).

So after about 14 hours of toying with it the speed ups I have managed to make come from inlining a method to unwind a context (callframe), reordering some compares on the GC path and disabling the __builtin_expect branch hints as they were mostly wrong (something the kernel people found to be true in 2010 as well). I will just try harder, or try to work on the optimizer or attempt something more radical…

$ perf stat gst -f Bench.st
219037433 bytecodes/sec; 6025895 sends/sec

Performance counter stats for ‘gst -f Bench.st’:

17280.101683 task-clock-msecs # 0.969 CPUs
2076 context-switches # 0.000 M/sec
123 CPU-migrations # 0.000 M/sec
3925 page-faults # 0.000 M/sec
22215005506 cycles # 1285.583 M/sec (scaled from 70.02%)
40593277297 instructions # 1.827 IPC (scaled from 80.00%)
5063469832 branches # 293.023 M/sec (scaled from 79.98%)
70691940 branch-misses # 1.396 % (scaled from 79.98%)
27844326 cache-references # 1.611 M/sec (scaled from 20.02%)
134229 cache-misses # 0.008 M/sec (scaled from 20.03%)

17.838888599 seconds time elapsed

PS: The perf support probably works best on Intel based platforms and the biggest other problem is that perf annotate has some issues when the code is included from other c files.

Using systemtap userspace tracing…

Using systemtap userspace tracing…

At the 27C3 we were running a GSM network and during the preparation I noticed a strange performance problem coming from the database library we are using running. I filled our database with some dummy data and created a file with the queries we normally run and executed time cat queries | sqlite3 file as a mini benchmark. I also hacked this code into our main routine and ran it with time as well. For some reason the code running through the database library was five times slower.

I was a bit puzzled and I decided to use systemtap to explore this to build a hypothesis and to also have the tools to answer the hypothesis. I wanted to find out if if it is slow because our database library is doing some heavy work in the implementation, or because we execute a lot more queries behind the back. I was creating the below probe:

probe process(“/usr/lib/libsqlite3.so.0.8.6”).function(“sqlite3_get_table”)
{
a = user_string($zSql);
printf(“sqlite3_get_table called ‘%s’n”, a);
}

This probe will be executed whenever the sqlite3_get_table function of the mentioned library will be called. The $zSql is a variable passed to the sqlite3_get_table function and contains the query to be executed. I am converting the pointer to a local variable and then can print it. Using this simple probe helped me to see which queries were executed by the database library and helped me to do an easy optimisation.

In general it could be very useful to build a set of probes (I think one calls set a tapset) that check for API misusage, e.g. calling functions with certain parameters where something else might be better. E.g. in Glib use truncate instead of assigning “” to the GString, or check for calls to QString::fromUtf16 coming from Qt code itself. On second thought this might be better as a GCC plugin, or both.

Back from the DevDays

Back from the DevDays

Like many others I was at the DevDays. I happened to be in Munich for GSM related things but managed to attend half a day on Monday and half on Tuesday. I was very happy to reconnect with friends from Australia, Oslo, Brazil and the US and I am always impressed by the team that the Qt unit has recruited and still manages to recruit. Creating awesome technology is just the consequence of that and as written in earlier posts, Qt Quick is great and the Qt Creator integration is getting good as well.

While still having your attention. Is there any interest in low-level GSM stuff in the KDE Community? The OsmocomBB project is close to make phone calls in a commercial GSM network and we could use a simple GUI running on a Linux host OpenBSC on the other hand is the swiss-army-knife of a GSM network and ideal for tests. E.g. testing emergency calls, security research, generating stuff that is difficult on a commercial network, do prototypes…
And on a private note… I am back to Berlin…
Tips and Tricks for QML

Tips and Tricks for QML

I would like to share some more information about how I resolve my QML issues. In my last post I got a friendly reply of hiding states and such information of a Component inside an Item of that component. This was a very good hint.

My application is making heavy use of delegates for the ListView, PathView, Repeater and sometimes qmlviewer stops displaying content with a warning: “QDeclarativeComponent: Component is not ready”. In contrast to the many other places this error message is not very helpful. In all of my cases this error came from a syntax error (unbalanced {}), using a duplicate property or having fun with lists. So check your model (if it is static) and your delegate file for syntax errors, e.g. load them in the qmlviewer and see if the error message is changing.
IIRC writing something like Rectangle { Item {} Item{} } is just a short form for writing Rectangle { data: [ Item {}, Item {} ] }. Now with the shortened way one does not need to use “,” at all, with the real list way, one may not have a “,” after the last item. It is good that the syntax checker is so strong, it just happens to conflict with my C99 usage.
The application requires some kind of table display and in general ListView/PathView do not allow such models. My not so unique idea was to have a ListModel and then another ListModel hanging off as an property. So the first approach might look like the one below.
import Qt 4.7
ListModel {
ListElement {
moreData: ListModel {}
}
}
But it is not supported by QML and you will get a nice error telling you that this is not possible. The workaround is to use the simple [] way to create a model that is working on the views.
import Qt 4.7
ListModel {
ListElement {
moreData: [
ListElement {}
]
}
}
With the above I can use a Repeater/ListView inside a Component/Delegate with a model of this row and I have a small table view. With my current approach I have to use a Flickable and two Repeaters increasing my memory usage but that is acceptable for now.
QML and dealing with states

QML and dealing with states

In the last days I have resumed my QML work (I had a small break to work on the MGCP GW code of OpenBSC to fix some real world issues) and there is one kind of issue I tend to run in and I wonder how others are solving it. Let us imagine we have a QML Component for a Button. The Button itself can hold a text (property alias text: buttonLabel.text) and the button has three states (enabled, focused, pressed) that depend on the MouseArea that inside the button as well. Actually this approach is directly coming from the many nice examples and demos provided by Nokia.

Now the problem is… I’m using the Button in many places and depending on some other external state the label of the Button should change and I keep writing things like this:
UI.Button {
id: text_button
text: ‘My Text’
MouseArea {
anchors.fill: parent
onClicked: { console.log(‘clicked’); }
}
states: [
State {
name: ‘some-state’
PropertyChanges {
target: text_button;
text: ‘Other text’; }
}
/* more states… */
]
}
And then I am going to wonder why things don’t work. The first issue is that my own MouseArea will receive the mouse click and the button will not work… but that is easily fixed. Do not add a custom MouseArea and have a clicked signal inside the button component. The second issue is with the states.. the above code is breaking the focus/pressed logic.

The way I am dealing with this kind of problem is to move the state into a parent and control the text from there. What is the proper way of solving this problem? Creating multiple buttons and control the visibility/opacity from outside? Duplicate the component states inside the custom states (cross product of my states and the component states)?
I know that talking about errors is bad as this will make people remember the wrong solution but I hope that other people stepping into these kind of problems will remember this as a possible problem..
Qt Quick vs. Android XML Layouts

Qt Quick vs. Android XML Layouts

In the last month and next couple of weeks we were/are developing a similiar application for Android, iPhone and the Desktop (using Qt and Quick) with native technologies. I will focus on Android and Qt as both are Open Source and we were developing them in a similiar fashion with one designer and one developer interacting with each other.

In both cases the application has a couple of views mainly consisting of a ListView of some sort and I will explain how this has been realized for Android and then with Qt Quick and provide some summary of what has worked well, what Qt Quick could learn from that. I have not worked with Qt Quick or Android prior to this project so I might have used the technology in the wrong way, and at your option you can either hold this against me or the technology.
We have started the project by doing the Android application. The model for my data is based on the abstract class called android.widget.BaseAdapter and I needed to implement a couple of functions. The first is the answer how many items I have, return an Object at the given position, an identifier for a position and finally a function that creates a View.
This function is called getView(int position, View convertView, ViewGroup parent) and the common usage is to check if convertView can be used to display the item at position and then modify (e.g. set the iccon, text, hide things on that view) otherwise we were using the LayoutInflater to create a View from one of the pre-defined layouts and set the items.
In Android one can describe Layouts in a XML Document, or use a still very basic Layout Designer (that will mess with the indenting of the XML), one unique feature is one can have different layouts for different properties, e.g. one can create a layout that will be only displayed if the application is in Landscape, uses Docomo as Operator and more. This is a very powerful system and we have only used it to have different layouts in portrait/landscape so far. The next important thing is one can ask the LayoutInflater to create the View for a given identifier, the next part is to find the items inside that view. Each View has a method called findViewById that is searching the children for that id.
One common pattern is to not call findViewById everytime one needs to access a widget but create a ViewHolder class with references to the Views and use View.setTag(Object tag) on the top View to set that ViewHolder on the View.
So how did the Designer and Developer interaction work here? First of all the Designer was responsible to create the View for the various parts of the UI and I was responsible for creating that BaseAdapter derived class, implement the getView method and set the right text/icon/visibility on the items of the layout based on the item that is to be displayed.
How did that work? In general it worked great, I could create a UI that worked, the Designer could make it look nice, the biggest problem (besides the basic UI editor, fighting with the RelativeLayout class) is that during redesigning a layout id’s were dropped from the Widgets and I had to assist the Designer to identify the missing IDs and add them back to the Design to keep things working and display the right thing again.
The Quick project is still work in progress and a lot like the Android project, the Developer creates the code (or states and interaction) and the Designer makes it look good. Our ListView’s are backed with classes based on QAbstractListModel (using setItemRoles to export the roles as QString) and the designer is forced to edit the files by hand right now.
The thing that really works better with Qt Quick is that the delegate (equivalent to the getView in Android) is in the hand of the Designer. This means that the Developer only needs to export the right strings for the roles from the model and the designer can use that to create the UI.
From my point of view the Qt Quick approach is really superior to Android’s XML Layout system, with Qt Quick we could get started faster by having a full click dummy (create the ListModel and fill it with dummy data), allowing the designer to fill the UI with content from the first day of the project, and by having the delegate in the Designers hand we have no “losing” id’s in the diff of an unformatted XML Stream. The only thing that has no proper equivalen in QML/Quick is basing the state on external properties, or with other words, I have not seen any documentation for some basic system/device properties.
First Steps with Qt’s Quick

First Steps with Qt’s Quick

I am very excited by the Qt Quick technology and I have finally found a reason to use it and gain some experience with it and want to report from my first hours of exploring it. The first time I have seen Declarative UI was with Enlightenment and the Edje framework. I liked it so much that I registered Epicenter five years ago. The plan was to create a handheld framework using declarative UI.

Now Qt Quick is not just a copy cat of Edje but adds some nice improvements over it. The first one is the usage of Javascript instead of the C like language, better integration with Qt’s Object Model (properties) and libraries/plugins. From my observation at Openmoko, our developers kept a set of C preprocessor macros around to do coming things with edje with Quick it seems to be better as one can import libraries and such.
The most common mistake I have made so far is dealing with the width/heigh of an Item. In my case I created an Item, placed a MouseArea (to deal with user input) of the size of the parent (anchors.fill: arent) in it and then also add some Text (as sibling). Now it appears possible that the Text is bigger than the parent item. For performance reasons no clipping is done by default so it renders just fine, just clicking doesn’t work. Which is bringing me to my debug trick…
I place a Rectangle { anchors.fill: parent; color: ‘blue’ } inside my item and then I actually see where it is. Another nice thing would be an Xray view showing the border of each item, their id but only in black and white. My solutions for this problem so far (from my first hours of using it) is to either use a Row/Column which seems to get the widh/heigt updated based on its children, or in some cases place a width/height inside the Item itself.
This is bringing me to the biggest issue with the qmlviewer and also an idea on how to resolve it… In the last couple of month’s I started to contribute to GNU Smalltalk and looking more into Self, Smalltalk-80 and such. The clever Xerox PARC people invented Model-View-Controller as part of Smalltalk-80, Qt adopted some parts of it with Qt4. In the meantime something called Morphic emerged. Morphich is a direct-manipulation User Interface. This means one is creating the UI inside a running UI by composing it from simple objects (just like Quick). In contrast to it one can inspect each element and interact with it, change it at runtime without restarting. This allows faster changes, and easier debugging in case something goes wrong. E.g. it easily answers the question of what is the width of that?
So for the immediate future I would like to see something like the WebKit inspector emerge for QML. This would allow to inspect the scene graph, change it from the console, has some simple hit testing to inspect one element, has the JavaScript Debugger and Profiler available, the timeline… and I am pretty sure to not be the first one to want it.