Mercurial 3.7 and 3.8

The Mercurial project continues its fast pace of innovation in version control. Both major releases this year (3.7 and 3.8) have very important new features that promise to improve user experience to a large degree.

Mercurial 3.7

Mercurial 3.7 had a major focus on performance. This is — to a large degree — due to large users like Facebook and Mozilla working on both performance and scalability.
The first major feature is one that has been in development for several years and changes the internals of how Mercurial stores its history. Originally, Mercurial stored history by comparing each changeset to the ‘previous’ changeset. In repositories with a very linear history (where you only have one branch, or where you rebase a lot instead of merging), this worked efficiently. Multiple active branches in parallel did pose a problem: rather than storing small deltas, deltas between entire branches would be stored.
Mercurial 3.7 solves this issue by automatically using ‘generaldelta‘ (which allows deltas to parent revisions, rather than ‘previous’ revisions) for new changesets.
It’s possible to convert an old repository by making a clone:

hg clone -U --config format.generaldelta=1 --pull originalrepo generaldeltarepo


A second important feature is a new serverside extension called ‘clonebundles’. By default, making a clone results in the server creating a zipped ‘bundle’, which is then transferred to the client. For small repositories, this works perfectly fine. For large repositories, creating such a bundle on the fly for every clone has quite some performance impact. The clonebundles extension allows a server operator to generate a daily/weekly bundle, place it on an HTTP server and point the client to that bundle. This results in less CPU usage on the server and faster clones for the client. Details can be found in the help page:

hg help clonebundles


Mercurial 3.7 has quite a few additional improvements. A list can be found on the website. Some highlights:

  • Interactive commit (hg commit -i) and amend (hg commit --amend) can now be used together.
  • ‘hg histedit’ now has a default range of commits it works on. This means you no longer have to specify any arguments for this command. By default, the first changeset used for histedit is “the first ancestor of the current changeset that is draft and does not have any descendants that are merges”.
  • The ‘hg revert’  command generates ‘*.orig’ files. It’s now possible to store these in a custom location.
  • hg clone --uncompressed is now 3-4 times faster on Windows. Details about why cloning was slower on Windows in the first place, can be found in a blog post from Gregory Szorc, who implemented the improvements.

Mercurial 3.8

Mercurial 3.8 again had a number of improvements focused on performance, along with focuses on user experience.

A first major feature is the fsmonitor extension (previously known as ‘hgwatchman’). This extension was created by Facebook developers to improve performance and has now been integrated as a base Mercurial extension.

Operations like ‘hg status’, ‘hg diff’, ‘hg commit’ require seeing what files in the repository have been changed. In a normal situation, this requires going over every file in the repository to check for changes. Fsmonitor uses the ‘watchman’ service to be notified when files have changed. Watchman in turn uses platform-specific API’s like ‘inotify’ or ‘FSevents’ to be notified by the operating system every time a repository file has changed.

Using fsmonitor, ‘hg status’, ‘hg diff’ and others only have to check files that have actually changed, rather than going over the entire repository. The result: all of the disk-intensive operations become a lot faster.


Another important improvement is the introduction of the experimental ‘automv’ extension. Normally, people move files around in their repositories using ‘hg mv’ or ‘hg cp’ (or aliases ‘hg move’/’hg rename’ and ‘hg copy’). However, it’s quite easy to forget about these command and use a regular move, especially when using an IDE. The ‘automv’ extension tries to detect similar files upon commit and marks these as moves/copies.


The newly integrated ‘chg’ client offers an alternative way to run Mercurial commands.
One of the issues Mercurial has with regard to speed of commands is that it’s implemented in Python. This is usually not a limiting factor, but starting the Python interpreter does add some overhead.

Chg resolves this using a client implemented in C and a server implemented in Python (the Mercurial command server).  Rather than starting a Python process for every Mercurial command, calling ‘chg’ starts a simple C application which communicates with the command server for every command.


Finally, there are a lot of additional improvements regarding user experience. Again, a list is available on the website. Some important highlights are:

  • ‘hg update’ no longer updates to the tip-most changeset on any topological head, but to the tip-most changeset on the same topological head. This behaviour is a lot more intuitive.
  • Mercurial now gives more hints on continuing interrupted commands.
  • Rebase chooses the default destination the same way merge does.
  • Graphical logging now displays an improved style.
  • The ‘[templatealias]’ section allows users to specify keywords and functions.

As usual, the latest Mercurial website can be found on the official website.

A simple ‘Game of Life’ implementation

I’ve been working on my Javascript skills recently. It’s a useful skill to have in general, it helps with my Mozilla patches and writing something for the web is an ‘quick feedback’ activity. One thing I’ve been doing in the past few days is write a simple Game Of Life implementation.

I started with the logic, creating a Grid ‘class’ to hold the state of the cells. As you can see, this is done by assigning functions to the prototype of the Grid. I’m not really used to this way of working, and I guess major benefits or downsides will show up only in more advanced development.

QUnit allowed me to write some tests for the logic of the Grid class. It’s somewhat strange to see the tests embedded in an HTML page (though some Mozilla tests have this as well). On the other hand, this makes it easy to run the tests in different browsers.

Finally, I used HTML5 Canvas for the user interface. I have to say this API is incredibly easy to use. For my particular use case, all I had to do was draw rectangles, and doing so only required a few lines of code.

This is all you need to start using canvas:

var canvas = document.getElementsByTagName('canvas')[0];
if(!canvas.getContext) {
var ctx = canvas.getContext('2d');

Basically, you get a ‘<canvas>’ area and request its context.

Drawing is equally simple:

ctx.fillStyle = "rgb(0,0,0)";
ctx.fillRect(x * 10, y * 10, 10, 10);

One can simply set the drawing colour and start drawing. ‘fillRect’ creates rectangles, but ‘lineTo’, ‘arc’, ‘fillText’ and others are available as well.

There are a few design choices I want to clarify:

  • Using a ‘Grid’ approach: I know a grid is not the most efficient way to store the ‘Game of Life’ data. However, it defines a simple area in which calculations are done. Besides improving my Javascript skills, an additional goal is investigating technologies like ZeroMQ and Node.js. I want to look into creating a distributed Game of Life-implementation. The boundaries of the grid will be the locations where data exchange is needed.
    In fact, making the implementation distributed is part of a ‘work group’ at Sioux, which has ‘spreading knowledge about “The Cloud”‘ as its primary goal.
  • Using ECMAScript 5: I wanted the code to work on as many Javascript implementations as possible.

Finally, let’s have a look at the result…

Megaman at Comic-Con

The following story is based on a writing prompt on Reddit: A super-villain attacks Comic Con, assuming it’s a summit of the worlds most powerful super-heroes plotting against him.

John and Henry went to Comic Con every year. It became a tradition for them to watch the cosplayers walk by, dressed up in their home-made costumes. As the hours passed and the collection of beer in front of them grew, their comments became louder and somewhat obnoxious. The other visitors having a drink glared at them, hoping they’d leave soon.

“Wooh, sweety, nice Lara Croft costume. Care to show us those pixels?” The girl strode past them, clearly annoyed, but too shy to shame them for their remark.

John and Henry didn’t notice, they were too busy commenting on their next victim.

“Hey, buddy, impressive get-up! You look just like that supervillain dude… Gipsyman!”

The man froze. He slowly turned his head while clenching his fists.

“Who said that?”

Wonderwoman pointed to John and Henry. Henry smiled, waving his beer at the man. John had been drinking at a slower pace. He wasn’t as inebriated as Henry and was still able to look slightly worried about a big fellow angrily staring at them.

“Hey, man, what’s wrong? your Gipsyman costume is really nicely done.”

The man exploded with anger. “How dare you mock me like this?! IT’S GIGAMAN! I should crush you like a bug for your insults!”

John started to get scared. “This guy looks like he really plans to kick our ass, what now?”, he thought.

Henry however, was too far gone to be scared of anything. “That’s not gonna work, Gigaman, buddy! We’re wearing our Ironman costumes, you’ll never be able to beat us!”

Gigaman hesitated. “The ugly guy has a point”, he thought. “I’ve had my ass kicked by Ironman and other heroes more times than I can count. How should I handle this?”

Meanwhile, John finally managed to put his hazy mind to good use and remarked “Hey, Gigaman, we didn’t mean to offend you. How about we buy you a beer? Come join us!”

Gigaman looked startled. “J-join you? But I’m a supervillain, how can I join you?” John smiled. “Don’t worry, man. Enough beer and it won’t matter if you’re a hero or a villain.” “Interesting”, Gigaman thought to himself. “Perhaps I should try this so-called… beer, see what it does.” He sat down with John and Henry.

Two hours later, a girl dressed as Lara Croft walked by.

“Wooh, sweety, nice Lara Croft costume. Call me if you want to handle something bigger than those guns you’re carrying!”

“Wooh! Nice one, Gipsyman!”

“WHAT?” Gigaman turned towards Henry once more, fuming. Henry smiled. “Guess I owe you another beer!”

Thou shalt resize thy strings!

I messed up.

I wrote a little string-join utility for a piece of software, but for some reason, it was behaving strangely. If I joined together two specific strings, my utility returned the second string, instead of joining them together! Well, isn’t that strange…

To start, here’s (a simplified) version of the algorithm:

WARNING! There is nothing wrong with the code below! I started looking at this code, but the eventual issue turns out to be somewhere else. Don’t waste your time looking at this code for way too long!

std::string join(std::initializer_list<std::string> paths) {
    std::string result;
    bool first = true;
    for(std::string path: paths) {
        if(!first || ((path.size() > 0) && (path[0] != '/'))) {
        if(first) {
            first = false;
    return result;

I know, it’s not very special, and it probably doesn’t handle all corner cases correctly. In fact, in a way, I already made my first mistake here: I didn’t use Boost::Filesystem. However, that’s something for a different topic.

My algorithm was working perfectly for most string combinations, but at one point, instead of getting first_string/second_string, I received /second_string as output. However, I couldn’t see any error in my code that could lead to such an issue.

Unittests to the rescue

Okay, perhaps a unittest can clear up the strange behaviour. I’ll reproduce the error in a unittest and I’ll be sure to find and resolve the issue! I used some Google Test magic, and voila:

TEST(Filesystem, join) {
    EXPECT_EQ(join({"foo", "bar"}), "/foo/bar");
    EXPECT_EQ(join({"first_string", "second_string"}), "/first_string/second_string");

Hm, that’s strange… My unittest is passing! That means the algorithm is behaving correctly for this case. So why is my actual application failing?

Debugging intermezzo

At this point, you might be wondering why I don’t just run the application in a debugger or under Valgrind, as these tools might help pointing out the issue. Well, you’re absolutely correct. The reason I didn’t jump straight to a debugger is because my application uses capabilities. It’s somewhat annoying to try to retain those capabilities while running under a debugger.

In other words: yes, I considered a debugger or memory checker, but I wanted to try a few other things first, to avoid going through the hassle.

Up and At ‘Em

We’ve established that the piece of code above seems to work fine, so perhaps the issue is somewhere else?
One of the things I do in my code is start up a second executable (if needed). In other to do this, I check the path of the current application, extract the directory, and then use the above ‘join’ to add the filename of the second executable.

The code to extract the directory looks somewhat like this:

std::string getExec() {
    static const unsigned BUF_SIZE = 1024;
    char buf[BUF_SIZE];
    ssize_t res = readlink("/proc/self/exe", buf, BUF_SIZE - 1);
    if(res < 0) {
        LOG_ERROR("Failed to read link: %s", strerror(errno));
        return "";
    buf[res] = '\0';
    std::string s(buf);
    return s;

std::string getExecDirectory() {
    std::string execPath = getExec();
    std::string execDir;
    std::string::size_type pos = execPath.find_last_of("/");
    if(pos == std::string::npos) {
        return "";
    std::copy(execPath.begin(), execPath.begin() + pos, execDir.begin());
    return execDir;

I start by using readlink to check where the current process is located. /proc/self/exe always points to that location.
Next, the result is stored in 'execPath'. The location of the last '/' is stored in 'pos', and used to only copy the first part of the string into execDir.
Looks fine, right?

At first, yes. However, when using 'join' later on, it turns out '.size()' returns 0 for the directory containing the executable! That's when it dawned on me. If you look at the above 'getExecDirectory', I don't actually allocate any memory for execDir. As a result, std::copy is copying to an empty string. By sheer luck, the application does not crash, and it's possible to use the string contents, as long as you don't ask silly questions (like: what's the size of this string?).

Resolving this issue is as simple as adding a single line right before std::copy:


Silly mistake? Definitely.
Easy to miss? Well, I like to think so...

In fact, while the resize mentioned above fixes the issue, it's possible to go with a much cleaner solution (thanks to passwordisRACIST on Reddit), which is also pretty obvious... Just use std::string::substr!

std::string getExecDirectory() {
    std::string execPath = getExec();
    std::string::size_type pos = execPath.find_last_of("/");
    if(pos == std::string::npos) {
        return "";
    return execPath.substr(0, pos);

Arriba! A new solution for build speed improvement

I’ve been working on a solution for improving build speed for quite a while now, during my free time.

Currently, its main focus is caching: it can cache compilation, but it can cache other things (for example scripts or tests) as well.

I want to continue developing this if there is enough demand. Interested? Have a look at

What is your main requirement when it comes to build speed? What do you want to speed up, and why? How important is it for you?

A C++ questionnaire on build speed — the results are in!


About a week ago, I posted a questionnaire on Reddit on C++ and build speed. I wanted to know how much of a problem build speed was to C++ developers, and also see what other issues C++ developers encounter.
Additionally, I’m working on an application (still in the prototype phase, I don’t plan on publishing it any time soon) to improve build speed, and I wanted to see if people would be interested.

I had a total of 250 responses (that’s the maximum amount of free answers per month allowed on SurveyMoz). Since I left pretty much all of the responses as ‘optional’, not all results will add up to 250.

How are you involved in C++ development?

To start, let’s see what kind of developers we have on Reddit. Possible answers were ‘Professional software development’, ‘Projects in spare time’ and ‘Other’.

I analyzed ‘Other’ to see if I’d need to add any additional categories of developers. Turns out most of the people in ‘Other’ are students, so I added that category.
I also added a category ‘Professional and spare time’ to show the overlap.


As you can see, there’s a lot of overlap here. A lot of developers have side projects, work on open source, … Somewhat expected, I guess 🙂

What do you see as your biggest waste of time while developing in C++?

This question is the reason it took so long to analyze the data (well, that and ‘real life’). I hoped it would give a good overview of pain points in C++ development. I had to do a lot of cleanup and merging of results, and below are the top answers (five or more people mentioning the issue).

  • Build time (both linking and compiling)
  • Header files and the include system
  • Unclear compiler warnings
  • Lack of good libraries
  • Coworkers
  • Debugging
  • Bad tools or lack of tools (IDE’s, refactoring tools, compilers, static analysis, …)
  • Boilerplate code
  • Build reliability
  • Legacy code
  • Bad third-party libraries

I also got a few responses stating that I shouldn’t have asked this question, since the title of the questionnaire already pointed people towards build speed. That’s absolutely correct! In that sense, it’s indeed skewed towards build times. However, it shows a nice overview of other possible issues, so I think the question was still useful.

Which of the following tools or techniques do you use to improve build speed?

Lots of different tools are available to speed up builds. It turns out that most people stick to using only a few to improve build speed. Most people only need parallel builds, language techniques (like forward declarations, pimpl, …) and possibly precompiled headers to improve build speed.
Tools built specifically to improve build speed are used, but far less than I anticipated.
Additionally, ‘adding more hardware’ was an option I forgot 🙂 Specifically mentioned there were SSD’s.


I also tried to correlate the people who mentioned build times with the tools used, to see if specific tools were used more on problematic builds. However, I didn’t see a correlation. Perhaps some people with those tools no longer have problematic builds, while others can only make their build times ‘less horrible’?

What are the development platforms you often use?

Who wins the big operating system war of 2014 amongst C++ developers?

*drum roll*


Besides these, some operating systems mentioned (but only once or twice) were:

  • VxWorks
  • ETS PharLap
  • QNX
  • INtime
  • Game consoles
  • Android
  • iOS
  • Bare metal development

Someone also mentioned I should not talk about ‘Linux’, but instead about ‘GNU/Linux’ or ‘GNU + Linux’.
To be honest, I don’t really care what the exact name is, I’ll leave that to Wikipedia.

Are build speeds a significant issue for you and why?

This question is one that made me very curious. I had no idea if I was one of the only people with build speed issues, or if it was a common problem.

Also, I got to use a nice colourful pie chart!


A lot of people do find long builds to be a serious issue. Some of the comments mention:

  • Turnaround time for development is terrible because of long builds
  • Incremental builds (due to dependency issues in their build system) fail too often, causing people to use full builds, which take way too long
  • Several developers have issues due to long link times, which are unavoidable during both full and incremental builds

On the other hand, quite a few comments mention simply doing other tasks while a build is ongoing.

How much (if anything) would you be willing to pay for a solution if it significantly speeds up your builds?

The most common response is simple: “I don’t want to pay!”

Additionally, plenty of people worry about caveats when using build acceleration tools, since they can have downsides (issues when compiling, build reliability, …).

People that do want to pay, vary a lot on price point. Now, I should mention beforehand: I don’t think all responses will be reliable, as quite a few were accompanied by ‘maybe’, or ‘I might pay’, …

Also, it’s a lot easier to say ‘I would pay X’ than to actually pay it. Meaning: some people who say they would pay, would probably back out if given the chance to pay for an actual application.

One last caveat: I simply went with the numbers without conversion (some gave euros, some dollars, some pounds). Additionally, some people gave ‘a couple of hundred’ as response, which is also a bit hard to quantify.

Anyways, here’s a nice chart to finish off this post:


Thanks again to everyone who participated in the questionnaire!

If you want, you can subscribe for updates on my build speed improvement application: