moiji-mobile

I encourage to read this article on how to design a office space for software teams (or any kind of engineer). There is scientific evidence showing the correlation between a good office space and productivity and his main points are:

1. Private offices with doors that close were absolutely required and not open to negotiation.
2. Programmers need lots of power outlets. They should be able to plug new gizmos in at desk height without crawling on the floor.
3. We need to be able to rewire any data lines (phone, LAN, cable TV, alarms, etc.) easily without opening any walls, ever.
4. It should be possible to do pair programming.
5. When you’re working with a monitor all day, you need to rest your eyes by looking at something far away, so monitors should not be up against walls.
6. The office should be a hang out: a pleasant place to spend time. If you’re meeting your friends for dinner after work you should want to meet at the office

a great read…

There is one thing in OE that is pretty time consuming. If you try to get some variables right that influence the entire system (SRCREV, DISTRO, MACHINE…) you will find yourself running bitbake over and over again and each run will take several minutes of your time. There are two solutions to this problem and both are very good.

• Have less data. If you have less data, less data need to be parsed. Apparently poky is doing that with their meta module. You can also see this with jhbuild of GNOME where you have to enable modulesets to have more data. For OE we should consider splitting up things that are orthogonal to each other. GUI and console networking tools? For Distro- and Autobuilders this will not make a difference, for the avergae joe it might. I’m not sure we should split OE into several independent modules just for parsing speed
• Parse faster. This is what I will talk about now

I’m bothered by the Bitbake parser for several years now. And all these years I had a paid job/main contractor and was doing work for them and never finished my work on the various approaches based on Marc Singers lexer/grammar. As of now this obstacle is gone, I have plenty of time (wanna change that?), and the last two days I was working on the Bitbake parser.

The current approach was to base on Marc Singers flex and lemon work (fixed to really parse everything), try to hook it into python, try to hook it into the rest of Bitbake. At some point this always stalled because it is very hard to verify that the new parser is doing things properly. And it is quite frustrating as well. lemon/flex is pretty fast in figuring out the structure but we have quite some python code in our metadata which needs to be executed and so far this has not been optimized. While one is able to lex and analysis the grammar below a second, it will take quite some time to execute the python code. Anyway, all of my previous attempts stalled at some point, mostly when trying to verify…
So this time I just ignored how horrible our current regexp based scanner is and decided to turn it step by step into a parser that creates a syntax tree/list and then evaluates this list/tree into the bb.data dictionary.
The first commits attempted to move the actual data handling out of the line based regexp handling into a new python module. Afterwards I turned all these methods into methods creating a Node for the syntax tree and immediately evaluating it to match the current behavior. Finally I was able to change that to evaluate the tree to a bb.data at the end of the parsing. So I have successfully (git bisectable) converted the current scanner into something producing the AST and then evaluating it. When parsing the OE metadata certain files like *.inc or *.bbclass will get parsed over and over again. With the above change we can scan these files once, keep the syntax tree around and then just evaluate again.

I ended up with something like 27 patches against Bitbake, plenty of baby steps, each with high confidence that there a no regressions and this leads to turning down the parsing time from 3m9.573s to 2m35.994s on my rusty macbook.

There is some more work ahead to improve this situation, move away from the regexp to PLY, attempt multithreaded parsing, attempt to write a peep-hole optimizer(???), look at the data module again…quite some time is spent in the cache too…

In case people didn’t notice a small information. According to this video MontaVista has looked into OpenEmbedded and the bitbake task executor. Apparently they liked the idea of cooking a customized rootfs by combining a set of recipes and have adopted this strategy and the tools. They have added extra value by having an easy to use installer, a source mirror and pre-built packages to speed up the engineer.
It is not clear how much and if they have used recipes and classes from OpenEmbedded but in any case we have adopted the MIT License because we want to have the widest use possible. I’m really happy to see this happening and thanks for everyone working on this.

I’m seriously amused by the recent announcement, I had to go through the irc log and laugh badly. There is one joke our lecturer made in the software engineering class and I would have never assumed that people in the real world would say something like this… oh well… today they did. It made my day.
So what uneducated people like to put into a product requirement is a sentence like “The software should be easy to use.”. And you should wonder why someone should design a software that is not easy to use… But today I found out that ofono’s API is supposed to be easy to use. All welcome a world where APIs are meant to be easy to used, finally someone is putting an end to all the APIs that are created to not be used. I know this takes a lot of changing on all our sides.

So I’m still in Taipei and my productivity on the 9th floor is not as good as I would like it to have it… So we mostly have an empty 8th floor, with silence, no artificial light, and very good air… I assumed I could just walk down, sit in one of the empty (conference) rooms and be productive… it worked… until.. oh well… there was an urgent meeting, suprisingly in the room I was sitting on… and suprisingly the other eight empty (conference) rooms with plenty of chairs and tables too couldn’t be used… and the meeting had to be held in the very same room… so I got kicked out and decided to leave the entire building *happy or not*.

After Christmas I returned to Berlin to be at the BCC for the 25th C3 in time. I was honored to talk to John Gilmore and the German Openmoko population was also present and we enjoyed talking, reading and hacking away on GSM. From the many things I planned to do I have only finished some but as usual I will work on these round robin… But it is time to leave the black hole and return to work.