Bitbake parser performance

There is one thing in OE that is pretty time consuming. If you try to get some variables right that influence the entire system (SRCREV, DISTRO, MACHINE...) you will find yourself running bitbake over and over again and each run will take several minutes of your time. There are two solutions to this problem and both are very good.

• Have less data. If you have less data, less data need to be parsed. Apparently poky is doing that with their meta module. You can also see this with jhbuild of GNOME where you have to enable modulesets to have more data. For OE we should consider splitting up things that are orthogonal to each other. GUI and console networking tools? For Distro- and Autobuilders this will not make a difference, for the avergae joe it might. I'm not sure we should split OE into several independent modules just for parsing speed


• Parse faster. This is what I will talk about now

I'm bothered by the Bitbake parser for several years now. And all these years I had a paid job/main contractor and was doing work for them and never finished my work on the various approaches based on Marc Singers lexer/grammar. As of now this obstacle is gone, I have plenty of time (wanna change that?), and the last two days I was working on the Bitbake parser.

The current approach was to base on Marc Singers flex and lemon work (fixed to really parse everything), try to hook it into python, try to hook it into the rest of Bitbake. At some point this always stalled because it is very hard to verify that the new parser is doing things properly. And it is quite frustrating as well. lemon/flex is pretty fast in figuring out the structure but we have quite some python code in our metadata which needs to be executed and so far this has not been optimized. While one is able to lex and analysis the grammar below a second, it will take quite some time to execute the python code. Anyway, all of my previous attempts stalled at some point, mostly when trying to verify...
So this time I just ignored how horrible our current regexp based scanner is and decided to turn it step by step into a parser that creates a syntax tree/list and then evaluates this list/tree into the bb.data dictionary.
The first commits attempted to move the actual data handling out of the line based regexp handling into a new python module. Afterwards I turned all these methods into methods creating a Node for the syntax tree and immediately evaluating it to match the current behavior. Finally I was able to change that to evaluate the tree to a bb.data at the end of the parsing. So I have successfully (git bisectable) converted the current scanner into something producing the AST and then evaluating it. When parsing the OE metadata certain files like *.inc or *.bbclass will get parsed over and over again. With the above change we can scan these files once, keep the syntax tree around and then just evaluate again.

I ended up with something like 27 patches against Bitbake, plenty of baby steps, each with high confidence that there a no regressions and this leads to turning down the parsing time from 3m9.573s to 2m35.994s on my rusty macbook.

There is some more work ahead to improve this situation, move away from the regexp to PLY, attempt multithreaded parsing, attempt to write a peep-hole optimizer(???), look at the data module again...quite some time is spent in the cache too...

MontaVista using Bitbake for MontaVista Linux version 6

In case people didn't notice a small information. According to this video MontaVista has looked into OpenEmbedded and the bitbake task executor. Apparently they liked the idea of cooking a customized rootfs by combining a set of recipes and have adopted this strategy and the tools. They have added extra value by having an easy to use installer, a source mirror and pre-built packages to speed up the engineer.
It is not clear how much and if they have used recipes and classes from OpenEmbedded but in any case we have adopted the MIT License because we want to have the widest use possible. I'm really happy to see this happening and thanks for everyone working on this.

Amused by ofono.foo

I'm seriously amused by the recent announcement, I had to go through the irc log and laugh badly. There is one joke our lecturer made in the software engineering class and I would have never assumed that people in the real world would say something like this... oh well... today they did. It made my day.
So what uneducated people like to put into a product requirement is a sentence like "The software should be easy to use.". And you should wonder why someone should design a software that is not easy to use... But today I found out that ofono's API is supposed to be easy to use. All welcome a world where APIs are meant to be easy to used, finally someone is putting an end to all the APIs that are created to not be used. I know this takes a lot of changing on all our sides.

Scrolling in GTK+ and WebKit/GTK+

• Model/View GtkAdjustment and GtkScrollbar

  
  The GtkAdjustment is probably best described as a model for scrolling. It has several properties, e.g. the current position (value), the lower and upper possibilities, and the size increments. The GtkScrollbar is operating on top of a GtkAdjustment. It is responsible for taking user events and painting. When scrolling it will look at the lower and upper properties, it will update the value in case of something is happening.
  




• WebCore::Scrollbar in WebCore

  
  The Scrollbar is created by the Scrollbar::createNativeScrollbar "factory". For many platforms the painting/theming and behaviour is entriely done within that class. For GTK+ we will use a GtkScrollbar, this widget happens to not have its own GdkWindow which makes painting a bit more easy, we will just forward the original expose event and let it draw as well (from ScrollbarGtk::paint). WebCore::Scrollbar in WebCore are used in two places. One prominent one is the WebCore::ScrollView which is the base class of the WebCore::FrameView and will be used to enable people to scroll on their content in horizontal and vertical direction, the other big user are scrollable div's (we have a manual test in WebCore/manual-tests/gtk/ to test positioning of these scrollbars).
  




• Scrolling on GtkWidget

  
  The scrolling starts quite innocently with the description of gtk_widget_set_scroll_adjustments. In case of WebKitWebView in WebKit/GTK+ we do want to support scrolling so we need to return TRUE... but how. If you take a look at the class structure of GtkWidget there is a GObject signal identifier you will have to set with the signal you have created. This is done in the WebKitWebView class init and we will remember and use the GtkAdjustment, e.g. set by a GtkScrolledWindow, in the WebCore::ScrollView (base of WebCore::FrameView class). The usage of GtkAdjustment allows to have different means of scrolling, e.g. by fingers on a touchscreen, or by a wheel... the representation of the scroll concept will change, the implementation not...
  




• The problem of mainFrame and subframes

  
  So far we will most of the times only have external GtkAdjustment set on the mainFrame that is embedded in something like a GtkScrolledWindow but there are pages that will create a frameset and you will have subframes that require scrolling (my test case here is Google Images). What we are ending up with is having a WebCore::FrameView with GtkAdjustments set from the WebKitWebView and some WebCore::FrameView without GtkAdjustments set at all. On cases without a GtkAdjustment, there is no one that will place a GtkScrollbar (and resize the WebKitWebView to be next to it), so the WebCore::ScrollView will create a ScrollbarGtk to handle this job. This creates the siutation that one WebCore::FrameView class will or will not have a Scrollbar (and manage its size...).
  




• The current solution

  
  When we have a GtkAdjustment set in the WebCore::ScrollView do not think about scrollbars, the need of them, the positioning of them at all. Simply update the properties including the current value, the visibleWidth and contentWidth. The upside is we have a GtkWidget that is working like a GtkWidget and can be embedded into a GtkScrolledWindow or a MokoFingerScroll (back in the better days). The downside of this include that we have more platform specific code and that we will not send onscroll events... due not going through ScollbarClient::valueChanged....
  




• The new solution

  
  Wrap the GtkAdjustment we get set into a ScrollbarGtk but do not create a GtkScrollbar. This should create a WebCore::Widget with WebCore::Widget::platformWidget() returning zero and this should set the width() and height() of this Scrollbar to zero meaning that the ScrollView calculation (e.g. updating visibleWidth/visibleHeight) is not negatively influenced, we can kill some more #ifdef PLATFORM(GTK) from WebCore::ScrollView and that we properly send onscroll events as all scrolling is going through the WebCore::Scrollbar code path, like with every other port. For having navigation working we must be sure to reset the GtkAdjustment, due the nature of the FrameView this is best done when creating the ScrollBar... The progress of this can be tracked in bug #25646.
  

FrameView and navigation

Visiting a site can have certain changes on a FrameView that will not be undone when leaving it. The most prominent example is a fixed background (by CSS) which will disable using blit on scrolling. There might be different things as well. Another one would be the PageCache and navigation to pages in it. The result is that for every page navigation you will create a new FrameView. For most/all ports this is done in FrameLoaderClient::transitionToCommittedForNewPage(). There is even a convience method shared between most ports in the form of Frame::createView.


From an API point of view this means that there is one WebKitWebView (in terms of Gtk+) and this WebKitWebView has one WebCore::Page and this page has exactly one WebCore::Frame the so called mainFrame but throughout the lifetime of WebKitWebView there will be multiple instances of FrameView's setup on the WebCore::Frame mainFrame. This also means that there are times were multiple WebCore::FrameView instances are around but only one should be active at a time.


For WebKit/GTK+ we will need to make sure that all the state of a WebKitWebView is transferred/set to the WebCore::FrameView when it gets created and ever change to a WebKitWebView gets forwarded to it. The most prominet state information are colors and GtkAdjustment (another posting). I think Gustavo is also about to introduce Scrollbar policies soon.

Embedding a GtkWidget into a HTML page

In the last few days I have suprisingly (actually not, it comes natural when resigning from a job) more time to do stuff that I really want to do. For the last months my webkit involvement has been mostly been reviewing patches created by other people but now I have done something I wanted to add for quite a long time...

I think one of the benefits of having a native web renderer widget in the toolkit is the integration that is possible. You can embed it into other native widgets and you should be able to embed native widgets into the web renderer. I have been working on the second part and the result can be seein in this bug. The attached example is putting the Gtk Scribble example into every plugin.

In the arora web browser this feature is used to implement ClickToFlash and I plan to still this idea too.