Adding Mercurial support to Gitlab

2018-09-18

Good news everyone!

We are taking on the largest pain-point in the Mercurial community: integration with newest collaborative development platforms. Using the Mercurial version control system locally is pleasant, but when it comes to start collaborating, things get bumpy.

For that reason, we have experimented with adding Mercurial to the Gitlab platform.

Yes, you read right, we are making it possible to use our favorite version control system with the trendiest collaborative development platform.

We have built a prototype, that we and a few others now use internally. Now that we have proven feasibility, we are in contact with Gitlab to work on adding Mercurial support upstream. So far, we have received warm feedback. It cheers us up to push forward our efforts to offer the best development workflow and developer experience for all Mercurial users.

Why Work On A Collaborative Development Platform

Version control is a critical part of the development workflow, but it is only a part of it. It needs to integrate with other tools in order to provide a complete workflow, including access management, code review, continuous integration, continuous deployment, ticket tracking, etc.

Over the years, these tools grew more and more powerful, becoming a larger part of the developer experience. Collaborative development platforms (forges) supporting Mercurial exist but they now tend to lag behind the solutions existing around another version control tool: Git.

Fortunately, a lot of what makes current collaborative development platforms so appealing is only loosely related to the underlying version control system. Which makes it relatively easy to add support for an alternative one.

That lack of support in the best collaborative development platforms out there has been affecting Mercurial users for several years. We have decided to contribute to a solution. At Octobus, our core expertise is version control so the best course of action for us is to collaborate with an existing platform for the rest of the development tooling.

Why Pick Gitlab?

Gitlab has been one of the fastest growing platforms in the past couple of years. It has a strong feature set praised by many. We have seen all kinds of users switching from Mercurial to Git just to be able to use it. So the quality and the reputation of the product itself made us look into it.

Moreover, Gitlab has an open source version "Gitlab CE", provided under an MIT license. They have a healthy community with significant external open source contribution. It fits with our own open source philosophy and allows us to directly hack on the product core to add support for Mercurial.

Also, other open source communities, Gnome and Debian, picked Gitlab as their tool. Having such serious actors of the free software world trust Gitlab was important in our choice.

Why Mercurial?

You might be wondering: why pick Mercurial in 2018?.

Mercurial and Git are similar by many aspects. They were created at the same time, by people from the same project, using the same sources of inspiration and for the same use-cases. Yet they have different implementations and made different choices that have a significant impact on the user experience. Having multiple tools offering similar services is beneficial for innovation. Since their creations, the two projects have been reusing each other's good ideas.

Overall, we believe Mercurial has a more extensible codebase allowing for more innovation. It has been used by large companies for several years, giving it pretty good scalability properties. We find it has a clearer user experience for both simple and advanced use-cases. Furthermore, the Mercurial development is very much alive, meaning that things will keep getting better. For example, we are getting real narrow cloning as a first class feature, allowing developers to combine the performance of using a small repository and the benefits of having a larger mono-repository. The Changeset Evolution feature is also unlocking new collaboration possibilities for team members.

However, we realized that all Mercurial powerful features needed an appropriate ecosystem to achieve their full potential. That is why it has become critical for Mercurial to be supported by state-of-the-art collaborative development platforms. In return, new features from Mercurial will help shape the workflow that those platforms will offer in the future.

Why us?

At Octobus, we have been working on Mercurial for many years, getting involved in its community. Part of that involvement is talking to many users with diverse workflow, structure and backgrounds. We have been hearing more and more complaints about the lack of a good collaborative development platform for Mercurial. In the past two years, those grew at an alarming rate. We are convinced that is the biggest challenge that Mercurial is facing today. So, as much as we enjoy making the core of Mercurial faster and better, we decided we had to do something to address that tooling issues.

What now?

In addition to discussing with Gitlab how to get the changes needed for Mercurial support upstreamed, we are now improving our prototype, cleaning up the code and making it more capable. Our main goal is to provide people with a first class Mercurial experience with Gitlab.

If you are interested in testing the prototype or in helping us achieve that goal, please contact us at contact@octobus.net, or join us on #octobus-hg IRC channel on Freenode .

(And if you are a developer interested in the source control version world, we are hiring.)

New Evolve extension release: version 8.2.1

2018-09-14

We pushed a new release for the evolve extension: 8.2.1

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

This version fixes issues around the obshashrange caches (enabled by default since 8.2.0).

Thanks to all people who reported issues. Special Thank to Gerald Squelart from Mozilla who submitted a patch along his bug report ☺

Full changelog:

Evolve (8.2.1)

  • obshashrange: issue the "long warmup time" message only once
  • obshashrange: reduce impact of cache invalidation from many new obsmarkers
  • obshashrange: properly silence permission error related to caches

New Evolve extension release: version 8.2.0

2018-09-03

We pushed a new release for the evolve extension: 8.2.0

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

This version introduces user interface adjustment and turn the obshashrange based discovery on by default. Thanks to Pulkit Goyal and Dan Villiom Podlaski Christiansen for helping with those questions.

The topic extension received improvement too.

Full changelog:

Evolve (8.2.0)

  • prune: rename --biject flag to --pair (old flag is kept as an alias)
  • pick: rename the grab command to pick (to avoid ambiguity with graft)
  • discovery: enable obshashrange based discovery by default

topic (0.11.0)

  • revset: topic("patterns") now handle standard patterns ("re:", etc)
  • revset: topic(REVS) matches revisions with same topic as REVS
  • topic: using s# alias instead of t# and b# alias (compat with old form is preserved)

Status of Mercurial's Changeset Evolution

2018-09-03

TL;DR; All the questions that we needed to clarify for Changeset Evolution to complete have been answered. We are now working on having a final implementation enabled by default in Mercurial core.

In the past couple of years, multiple important milestones have been reached for the Mercurial's Changeset Evolution project. Changeset Evolution allows for simple, safe and powerful exchange of draft changesets that can be rewritten by a distributed team. (The concept is also useful locally, but most innovative part are the distributed ability). Those progresses are the result of a combined effort of a dozen people, thanks to all of them.

In particular, we have been working on final algorithms for two of the central concepts of changeset evolution: exchanging markers and automatic resolution of instabilities. They now just need a final implementation in Core.

In addition, the last uncharted territory, the restoration of older evolutions of changesets, have been explored. We discovered that we have to track more data regarding changeset folding to perform this task well. This will trigger what we expect to be the last update in tracked data and marker format.

There are many improvements made apart from the ones we just highlighted, and we'll go over them later in this post. However, the highlighted ones are important milestones. All the major algorithms questions are now cleared, we know how each complex issues will be handled and what associated data storage we need. Removing unknowns on those questions was a priority because they can have a high impact on how we design user interfaces and on-disk data formats. Two things that are complicated to change afterwards.

Now that all those areas have been explored, we have a clear sight of the route to Evolve completion and can focus on actions directly moving toward this goal. We need to polish and upstream the concept and code that are still inside the Evolve extension. Some of that code is already in a good shape to go upstream.

The road is clear, but there is a lot to carry along, any kind of help is welcome!

Summary of work done since 2017

In general, all areas received many fixes, improvement, and polish. Some important areas made significant progress:

Vocabulary changes

Evolution provides powerful features, capable to handle complex issues. We must do all we can to smooth this complexity, in particular when it comes to naming new concepts. To address multiple feedback regarding the naming scheme, a group of people dedicated a significant amount of time to come up with a new naming scheme for most of the changeset evolution concept. The resulting scheme was then enforced in Core Mercurial, the Evolve extensions and Tortoisehg:

  • Instability replaces Troubles
  • Unstable replaces Troubled
  • Predecessor replaces Precursor
  • Orphan replaces Unstable
  • Content-divergent replaces Divergent
  • Phase-Divergent replaces Bumped

Check the wiki page associated with the renaming discussion for details.

In addition, we made some update to command and flag names:

  • The hg uncommit command is merged into hg amend as hg amend --extract
  • The unloved hg prune --biject flag has been renamed to hg prune --pair

Obsolescence History

To make distributed history edition manageable by the users involved, it is important to make it easy for someone to understand what happened outside of its local repository.

Changeset evolution tracks the change to all changesets that have been rewritten. This information is valuable for many different facets, for example for displaying what happened to a changeset, or automatically stabilize it. We made multiple improvements to make sure the information tracked was more informative and better put to use. Previously the obsolescence information was mostly used to detect and resolve instabilities. Obsmarkers now store more data:

  • the effect of an obsolescence marker (message change, patch change, etc),
  • the operation that created it (amend, rebase, ...),
  • possibly record a user-defined note.

And that data get used in more places:

  • New hg obslog command that displays the evolution of a changeset,
  • Display successors and predecessors in hgweb when appropriate,
  • Point at latest successors of obsolete changeset in hg log,
  • Point to latest successors of obsolete working copy and accessed hidden changesets.

This greatly improved users ability to understand evolution that they, and their team members, have performed. Making Changeset Evolution more accessible to people unfamiliar with it.

This area is in a very good shape, there are multiple possible improvements, but overall we have the feature we need to provide a reasonable experience. A good share of them have been implemented in Core already, the rest needs to be upstreamed. See the "What next?" section for possible future improvements in this area.

History Rewriting Commands

Smooth distributed history rewriting requires a good set of commands to rewrite history. They need to be easy to use and to record the user intend well so that we can leverage it.

Changesets evolutions allow more flexible history editing but also rely on the user using the right command for the right operation to build a useful evolution history. A quality evolution history provides the user with the best experience. To achieve this goal we need a clear, compact and powerful set of command to edit history. Progresses were made on existing commands and new commands:

  • hg fold got flags to clarify its two modes --from and --exact, (We are not entirely happy with the result yet)
  • hg uncommit is now available as hg amend --extract. This reduces the number of commands and closes the debate about the uncommit name,
  • hg amend gains a new --patch flag to directly edit changesets diff,
  • The hg grab alias got upgraded to a full hg pick command (with proper --abort/--continue support). The command is a mix of graft and rebase. The command can be used to reorganize a stack of changesets.

An important milestone is the introduction of a rewind command. It is a command to restore a stack of changeset to a predecessors state. The command is at an early stage and will need some time to mature, however, it is already useful and part of some people main workflow.

Implementing the rewind command was part of our exploration of the evolution problem space. And, indeed, we gathered very important insights. The hg rewind command can automatically find and restore predecessors of other changesets. To do so, it walks the evolution graph, but it does it in the opposite direction than the hg evolve command. It turns out that, in the same way we had to store special information for split, we will need extra information to properly differentiate fold from divergences resolutions. We currently do not have this information.

This is a good example of data requirement we had to discover early in order to shape a functional changeset evolution in the end.

Obsolescence-markers exchange

Changeset Evolution core purpose it to unlock history edition in a distributed setting. To achieve this goal, the exchange of obsolescence information between repositories is critical.

The question of which obsolescence markers should be exchanged during push and pull have been solved a while ago. Further testing in diverse environment setups have confirmed this logic is correct.

However, at the start of 2017, an important issue remained: how to discover which markers are missing on the other side? Without an efficient way to detect them, we could not provide an efficient synchronization of the obsolescence data.

Fortunately, we developed an algorithm and protocol that can perform an efficient discovery for obsolescence markers. In our daily usage on the Mercurial repository, this saves us the extra minute we were spending on obsmarkers discovery using the previous method.

The data structure used by this algorithm scale well with repository size, an important point that qualifies this solution for all usages.

This new method got tested in various settings and it will be used by default in the next version of the Evolve extension.

In addition one data structure we developed for this algorithm, "stablerange", will likely be useful to help with exchanges and caching of other data. It could be used for pull bundle caching decisions.

The main goal of the current implementation was to validate the approach and the scaling property of our algorithm. Its performance and cache storage implementation is not great and this will have to be reworked when upstreamed.

Workflow and stack

Something important connected to evolution is the clear definition of the group of changesets related to the current user work. Such clarity is a great help to provide good behavior and information to the users regarding possible instability in their current work. We made many progress in this area in the past couple of year.

In particular:

  • Automatic instability resolution, using hg evolve, restrict itself to the current topic by default. Avoiding selection of unrelated changeset during stabilization, something that has been confusing users.
  • The next and prev commands are also restricting themselves to the current topic, making their movement more predictable and useful.
  • The hg stack command allows for a quick view of the work in progress, including listing changeset in a semantic order, same as if they were all stabilized.

This work also offers a simple, yet powerful, workflow for feature branching in Mercurial. Having topic tightly linked to phases make them a good tools to enforce a healthy phase movement practice in a project. Since Changeset evolution is also tightly coupled with phases, healthy phase movements is important here. Here are the progresses made:

  • New server repository mode: changeset with topic stay draft of push, other get published. This is especially useful as it bridges the gap between publishing and non-publishing repository. Using this mode by default will preserve backward compatibility with current Mercurial behavior regarding phase movement while allowing the user to opt in from the client side for exchanging drafts when they want to.

  • New push flag --publish: to publish selected changeset on push.

There have been various improvements on topic usability. Notably:

  • clarification of topic activation, state, and movement on push/pull and publish,
  • ability to force new draft commit to have a topic (or automatically assign a random one),
  • improved topic discoverability with hg topic --age,
  • Introduction of a s0 label referring to the parent of the topic root
  • update to s0 will keep the topic active, making insertion of new changeset at the start of the topic simpler
  • hg prev can go down to s0

We are getting good feedback from people using this workflow. We would like to upstream all these improvements. Some of them are not really attached to topic (eg: hg push --publish) but we feels like topic is overall a good branching solution for Mercurial and would like to see it more of it upstream.

Instability Management

This is a critical part of Evolution and we made many important progress in this area. Exchanging draft changesets can bring "instabilities" in one's repository. We can use various strategy to reduce the odds for it to happen, however, given the distributed nature of Mercurial, we cannot guarantee it won't.

Because instability can happen, we need a good automatic resolution of it. We are now is a pretty good shape. The hg evolve command can now keep tracks of multi-step operations unlocking important features:

  • Good handling of evolution interrupted by merge conflict, --abort, --continue and --stop flags works as expected. (for both hg evolve and hg next)
  • Automatic stabilization of situation involving and orphan merge,
  • Automatic stabilization of phase-divergence in most complex case,
  • Automatic stabilization of content-divergence in most complex case.

So we now have all the instability types well under control. There are still some corner case involving split, fold or merge that remain to be properly handled. However, the core tool to handle them now exists.

With all these improvements to hg evolve and hg next, we will be able to update the default behavior of this two commands to something final, and upstream them (more about this in the next section).

What next?

This section focus on concrete actions to bring Changeset Evolution to completion. It is intended for people with some knowledge of Mercurial internals.

Summary

The most important things to clear up right now is that need for better tracking of fold. Fixing it right might eventually affect disk storage and various algorithms. Getting this out of the way as soon as possible is important.

In the same move, we need to mature the hg rewind command. Easy undo is a critical item if we are to hand changeset evolution to all users.

In parallel, there are multiple areas in the Evolve extension that are ready to be cleaned up and upstreamed (obsmarkers discovery, cache, internal rewriting toolkit, …).

We should also resume the upstreaming of the improvement regarding stacks management and publishing workflow contained in the topic extension. At first, this area might seem unrelated to enable changeset evolution into Core, in practice the clarification in working set and publishing workflows greatly reduce complexity around local and distributed work on draft changesets. So time spent improving these areas is well balanced by the one we do not have to spend solving more complex user experience issue elsewhere.

Finally, there are a couple of areas we that are mostly done but need to focus to be wrapped up. This is mainly about the history rewriting command and automatic instabilities resolution. In these cases, there are small actions to take before freezing the user experience in Core. Of course, more improvement will be made to them once into Core. However such improvement does not seem necessary to enable evolution by default.

Internals

Fold tracking

We need better tracking of fold operation. This is necessary to provide a fully functional hg rewind command. This command is important because users need a simple way to undo mistakes.

We need to update the marker creation API and to store this information. Such change might impact the on-disk storage format and more. The way we currently store split have a couple of quirks (eg: when only some of the split successors are exchanged). The current ideas for storing fold data could also be applied to split, handling these cases better than with the current split encoding. Tracking splits and folds is a quite interresting issues, we'll get back to it in an independant blog post.

These possible low-level changes makes the tracking of fold a priority target. Upgrading up-disk storage of existing users is never simple and it might also affect multiple algorithms that will have to cope with a different split encoding.

History rewriting toolkits

To power its history rewriting commands and its UI, Evolve has a full toolkit for building history rewriting commands. This toolkit is ready to be upstreamed and would be a good first step toward upstreaming the commands, free of any question related to user experience.

Evolution history

All the current work done around better evolution history is either already upstream, or ready to be upstreamed. We should keep that effort going.

There is one extra feature that users have been requesting, a clear transaction log to keep trace of what exactly happens from each past operations. This would be especially useful on push and pull as they may introduce many changes from other people in one repository. This would fit the journal extension, making it tracking a wider amount of data. This is not a road blocker, but it would be nice to have.

Commands

The Evolve extension provides multiples commands, most of them are close to be "done" enough for our goal of Evolve enabled by default in Core.

Rewind command

The one command that needs serious attention to mature is the rewind command. Users can "rewind" evolution of stack of changeset using this command. The first version of this command revealed we needed to record more data about changeset folding. Once this data is available we should have all the pieces to build a final user experience for this command. Having this command is very important for the Changeset Evolution experience. To trust the tool, users need to be confident they can undo their mistake.

History rewriting command

Over the year history rewriting commands in the Evolve extension evolved into a pretty solid base. As mentioned in a previous subsection, the internal toolkit powering these functions could be upstreamed now. The commands themselves need various adjustments to ensure consistency and solve some long lasting questions (those we'll cover in the second part of this blog post). All these adjustments are small and should be easy to perform. That last round of polish is best done in Evolve extension to offer it the wider testing possible before we settle it down.

Here are examples of small adjustment we need to do:

  • The hg amend --extract need to expose a flag similar to hg uncommit --rev before can fully drop the older command,
  • Currently hg fold requires either one of --from or --exact. We should pick one to be the default. --exact has a bit more mercurial developer supporters, but also requires the user to learn some revsets which is bad. If we have access to the shorter in-stack reference from topic (s1, s2, etc), the hg fold s1+s2 form might be good enough. Provided we give a proper example of how to use this in the documentation.
  • The hg split comment is inconsistent with other similar commands (amend, revert, uncommit, etc). It does not accept filename and is interactive by default. We should align its behavior with the other commands.

Of course, this command-set will keep getting larger improvements. For example, ideas have been floating around about making it easier to control changeset order in the stack via new commands flags. However, those improvements are not on the critical path to enable changeset evolution in Core. So we don't plan to spend time on them until then.

Automatic evolution

There has been a long-standing debate: should we automatically evolve descendants of rewritten changesets". Automatically evolving orphans provide a smoother experience to users in many cases but come with multiple drawbacks:

  • It can trigger merge earlier than the user wants, forcing them to switch content from the changeset they were currently rewriting.
  • When rewriting all changesets in a stack, automatically evolving all descendant means the creation of an O(len(stack)²) changesets. N² complexity scale very badly and this can have a very dire impact on users repository and overall experience. So we tend to focus users into gradual evolution instead. This issue is referred as "obs markers explosion".

However, there has been a large demand for automatic evolution at least in some cases. Some command can guarantee they won't introduce conflict, lifting related concern. We could even consider extending that logic to all command as long as no conflict is detected.

The N² obsmarkers explosion is trickier because all command could possible triggers it. Math does not really make compromises, N² is unsustainable for most values of N. However, we still have room for improved experience. We could imagine allowing auto evolution if the number of descendants is small (maybe 4 or 5), or try to detect repeated rewrite in the stack. Once we detect a problematic pattern or stack size, we could skip auto evolution and redirect the user toward a helpful documentation page.

The hg stack command provided by the topic extensions provide a clear view of the current work in progress, even when orphans are involved. It is a good tool to reduce the user surge to run hg evolve --all prematurely.

Instability prevention

A good way to avoid exposing users to the complexity inherent to instability is to avoid creating it in the first place. Core mercurial already have some mechanism that history rewriting commands can use to check if their planned rewrite is valid. However, we need to make sure it is used by all history rewriting operation and that it catches all the instability types we want to prevent.

Obsmarker Discovery

All the logic we use for obsmarkers discovery is sound and ready to be upstreamed. However, the code will have to be rewritten as it gets in. The current implementation evolved from a multi-step experiment, and use too many complicated indirections and is not very effective in general. Some of the cache storage we use is also problematic and will have to be re-implemented.

In practice, most of the data we cache are not volatile. They are an inherent property of changeset, so we could imagine storing these value directly into a "changelog v3" format instead of keeping an independent cache structure.

I expect the discovery to keep being improved over time, however, these improvement can come later and are not in the critical path for enabling evolution in Core.

The work needed to upstream this is well defined but significant. Finding other use cases for the stablerange data might get more people interested in making it happens.

The caches used for obsmarkers discovery shares a lot of code with other performance related caches that should move upstream too.

Stack and workflow

The topic extensions provide various features that contribute to providing a smooth Changeset Evolution experience. While not strictly necessary these features simplify the task of making changeset evolution accessible to all users.

Stack definition

One of the core features of topic is the clear definition of the current working set: the stack. As each changeset explicitly carries the topic information, there is no room for ambiguity. Related changesets that a user is actively working on are not always linear. Either the user action or the instability brought in by distributed rewriting can spread them on multiple topological branches. The clear stack definition from topic handle these situations, making it an ideal candidates.

This stack definition simplifies operation around evolution. For example, restricting most operations within the current stack make things much more predictable: hg prev and hg next ignores other unrelated changesets; hg evolve only select items in the stack. Having a predictable outcome for these commands is important for users to trust them. Without clear stack boundary, the behavior of there command becomes either more limited or more complicated to explain.

The stack defines a limited number of changeset relevant to the current situation. A small number allows for better UI. For example, we can provide a hg stack command that displays orphan changeset as if they were already evolved. This provides a preview of the final structure of their stack, even if some of it is still orphan. This is a powerful tool to make changeset evolution accessible to all kind of users.

The limited amount of changeset make it possible to bring back incremental numbers to refer to a changeset. With topic the first item in the stack is nicknamed "s1", the second "s2". This is very useful to refer to changeset without having to copy paste obscure hash around. The numbering is also preserved across rewrite, making them useful for a longer time. Changeset Evolution empower commit centric workflow and "s#" alias make it easier to reference individual commits.

In practice, other ways to define resilient stacks that provide this benefits. For example, a stack definition based on phases and named-branch would do. Named-branches are a strong fit for long-lived branches, but they have life cycle limitations. Topic feels like a better solution for feature branching.

Exchange and Publishing workflow

Another interesting aspect of topic is how their life cycle is tied to phases.

Changeset Evolution can be used on the non-public part of history, and for sanity reason, this part should stay fairly limited. Nobody wants to see a six-month-old changeset rewritten, creating tons of orphan changesets in the process. A core property of topic is to fade away when they are published. This means that accepting a topic into the main branch requires to publish it. Having a workflow step that explicitly involves publishing make sure the set of non-public changeset remains reasonable.

Topic also solves another pain point with regarding phases. The main purpose of Changeset evolution is to unlock distributed collaboration on draft changesets. However, the current default for any Mercurial server is to publish changeset on push … so long for exchanging drafts. It is possible to configure the server to no longer publish on push but this has multiple drawbacks. First, this is a server-side config, usually more complicated to setup. Second, it means the phase cycle has to be manually though about and handled for all changesets. Finally, the change affects all users, you can't have a small group of advanced users playing with advanced feature without impacting other users. A common workaround is to have two repositories, a publishing, and a non-publishing one. This makes things more complicated than we would like.

Because topic is a new concept that people have to opt-in, we have more flexibility there. We could have a new default mode for servers where changesets are published on push as usual unless they have a topic. This way, users can opt-in draft changeset exchange without server side configuration and without impacting the other users. And since exchanging draft requires them to have a topic, they won't interfere with the usual branch resolution of users not interested in the topic. This offer a smooth path to draft changesets exchange without breaking backward compatibility. As a bonus that scheme force draft changeset to explicitly belong to a feature branches.

To deal with phases cycle, the topic extensions provides various options:

  • A new hg push --publish flag to push that make a push publishing even when pushing to a non-publishing server
  • A config option to have server behave as described above (non-publishing for topic only).

The topics also comes with small workflow improvement, like intuitive rebase destination, making it easier to use. One of the key improvement is the ability to push a new topic to a server without --force. Removing the cumbersome need to use this dangerous flag.

Upstreaming

The concept explained above are in a good overall shape, and we got good feedback on them. So, what would it take to add support for all this stack and workflow related concept upstream?

Some of them are clear and good workflow improvement not strongly related to topic:

  • The hg push --publish option proved very useful,
  • The experimental config option to limit a repository to 1 head per name should learn about closed heads and graduate from experimental.

Some of the "stack" related logic is also independent of topic and could be first implemented around named branches.

  • "s#" alias for changeset in a stack,
  • Feature from hg stack (eg: evolution aware order),
  • Constraint to the range of hg evolve, hg next and hg prev.

Finally, there are interesting pieces directly related to topic:

  • The topic manipulation commands and lifecycle,
  • The new publishing mode for servers,
  • The ability to push new topic without --force.

General performance and cleanup work will be needed. However implementing these specific features directly in Core will be significantly simpler than from the extension.

Instability Management

Commands changes and upstreaming

There are some incoming change to hg next: The clear definition of a working set to move within and the improved recovery when conflict occurs during evolution means we no longer need to hide hg next triggered evolution behind a --evolve flag. It should become the default.

And related change to hg evolve: step by step evolution is an important feature. However, since hg next will be able to fill this role, is make sense for hg evolve to evolve all changesets in the current stack by default. In addition, matching rebase behavior of preserving the working copy parent seems in order.

These changes will happen in the Evolve extension. However, after they happen, both hg evolve and hg next should be ready to be upstreamed provided we have access to a clear definition of the "current working set" of revisions (as topic provides).

We already have a couple of ways to trigger automatic instability resolution (eg: next, evolve, …). However, there could be another good vector to expose it to the user hg evolve. The command is already dealing with working copy change and merge conflict. Adding a hg update --evolve flag could make sense and offer a simple user experience for some of the simpler case. Lowering the barrier of entry is always useful.

Instability resolution:

Even if hg evolve can now handle the majority of instability cases, some remains to be handled. The unhandled case usually involves a mix of phase-divergence and content-divergence or some merges, split or fold. We need to hunt down and handle these remaining cases.

It is probably simpler to tackle the last corner case of automatic stabilization from the Evolve extension. So that we can have a wider set of users to test them more quickly. Besides that, the whole logic is in good shape and ready to be upstreamed. We can probably start to upstream some of the core bits sooner (eg: upstreaming hg next would make a good excuse to upstream the orphan resolution). An alternative approach would be to upstreaming the current instabilities resolution logic in a way where the Evolve extension can monkey patch fixes for people using an older version. This might be more work overall.

User Documentation

There have been inline documentation and tutorials written alongside evolve development. However, most of it does not contain the latest commands and workflow. Help is welcome to refresh them.

Conclusion

I'm happy with the progress made in the past years. As many, I wished we could have done more, faster, but I'm very excited to have a clear view on project completion now. By project completion, I mean mercurial Core to contain a good enough subset of Changeset Evolution so it could be enabled by default.

At Octobus, we do our best to bring this to completion and mobilize people and resource to reach this goal. Next, our own effort will focus on getting the fundations concept in Core: rewind command, automated resolution of instabilities and efficient obsmarkers exchange. In parallel, we'll take care of the supporting concept necessary to safely enable Changeset Evolution to all users (obs-history viewing, supporting command, stack and workflow, …).

Of course, this effort is not just performed by people at Octobus or people working closely with us. Other Mercurial contributors are also working on their own to complete this project. What they will exactly do and in what orders is something for them to decide.

All sort of help is welcome. For multiple years now we have helped the project moved in many ways: through direct contributions of course, but also by training new people to contribute to the concept and finally by efficiently gathering and spending money to make the concept move forward, reaching out, funding, and steering other members of our open source community to make the project move forward. Reach out to us if you want to contribute time or money to see Changeset Evolution enabled by default in Mercurial.

Discussions around the Changeset Evolution concept usually happens on the #hg-evolve IRC channel on freenode. If you are using the Evolve extension, do not forget to subscribe to the user list, Evolve-testers@mercurial-scm.org.

New Evolve extension release: version 8.1.2

2018-08-28

We pushed a new bugfix release for the Mercurial's evolve extension: 8.1.2

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

This release fixes concurrency robustness issues around the sqlite based storage used by the experimental version of the obshashrange discovery caches.

With now more experience of running python sqlite in very concurrent situation, it does not seems to handle the job too well. Since it is only used for cache, the problematic case can be silently ignored when they occur.

Thanks to the people who have been actively testing the obshashrange discovery. This help detect these situations.

In addition, this release invalidate previous versions of the final cache layer that was affected by issues fixed in 8.1.1. This will ensure everybody has a valid cache content without requiring manual intervention (beside the update). This cache will be recomputed once (and is not the slowest to compute).

Version changelog

evolve (8.1.2)

Bug fixes

  • obshashrange: improved robustness of the cache under heavy load
  • obshashrange: force recomputation of the final obshash related cache (to make sure people benefit from the 8.1.1 fixes)

New Evolve extension release: version 8.1.1

2018-08-21

We pushed a new bugfix release for the Mercurial's evolve extension: 8.1.1

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

This release contains fixes for hg next, the obshashrange cache and other areas.

The obshashrange protocol is used to discovery missing markers during exchanges exchanging. It might get enabled by default in the next version of the evolve extension. We recommend trying it now by adding the following to your config:

[experimental]
obshashrange = yes

If you were already using it, we recommend clearing your caches after upgrading to this release, their consistency might have been impacted by (now fixed) bugs.

hg rm .hg/cache/evoext*

Version changelog

evolve (8.1.1)

Bug fixes

  • evolve: properly set second parent during conflict (issue5927)
  • next: delete the evolvestate after aborting interrupted next --evolve
  • next: fix topic restriction when passing --evolve
  • prune: improve documentation
  • clone: fix possible crash when using clone bundle and forcing cache warming
  • obshashrange: fix speed and consistency issues during cache invalidation
  • obshashrange: properly persist all caches involved in obshashrange discovery

New Evolve extension release: version 8.1.0

2018-08-04

We pushed a new release for the Mercurial's evolve extension: 8.1.0

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

This release contains many improvements to the hg evolve command. It now handles content-divergence and phase-divergence much better. In addition, a first version of the hg rewind command has been pushed. This command let you restore stack of changesets to a previous state. See the Changelog for others changes

This release support Mercurial version from 4.3 to the latest 4.7, tagged a couple of days ago. Support for Mercurial 4.3 might be dropped in the next version.

Version changelog

evolve (8.1.0)

New features and improvements

  • evolve: improve multiple aspects of content-divergence automatic resolution
    • branch changes handling,
    • parent changes handling,
    • description changes handling,
    • divergent stack handling,
    • improved resume, stop and abort of divergent resolution
  • evolve: improve orphan resolution when combined with divergence (issue5946)
  • evolve: improved automatic resolution of phase-divergence
  • rewind: first limited version of rewind command to restore stack of commit to a precusors state (check command help for detail and limitation)
  • evolve: add a --update and --no-update flags to evolve. It control final working copy parent
  • obslog: new --filternonlocal flag to hide change unknown locally
  • evolve: new help section dedicated to resuming operation interrupted by merge conflit, hg help evolve.interrupted.
  • evolve: show unfinished state information in hg status -v (issue5886)

Bug fixes

  • evolve: move bookmarks also when updating to successors (issue5923)
  • amend: abort --patch by saving an empty file (issue5925)

Compatibility changes

  • compatibility with mercurial 4.7

topic (0.10.0)

  • display a hint when a topic becomes empty
  • compatibility with mercurial 4.7

Performance benchmarking 101

2018-07-30

We all want top-quality software without any bug. But if there is something as frustrating than buggy software, it's slow software. Who never raged against software that seemed frozen? Who never reloaded a browser tab only to make the interesting make a brief appearance before the page reload?

So how to fix software slowness? Why is it hard to improve? Because before fixing a problem, we need to measure it. And measuring performance is a complex task by itself.

At Octobus, a lot of our work focus on Mercurial, a command-line tool. You invoke it dozens or thousands of times per day creating, rewriting and exchanging commits. The two facets that are interesting for us are the time for commands to finish (independently from the human interaction time) and more low-level functions benchmarking.

Measuring performance

Measuring performance is hard. Why is it hard? Because it's not a single boolean value (fast or not), it's a discrete value with a lot of influences from the environment. Here is a non-exhaustive list of things that can impact performance measurement:

  • Other running processes.
  • Hardware (CPU/Disk/Memory) performance.
  • CRON jobs.
  • AV scans.
  • CPU context switching.
  • Temperature throttling.
  • OS tuning.
  • Network traffic.
  • Cosmic radiations.
  • Noise (no kidding: https://www.youtube.com/watch?v=tDacjrSCeq4).
  • etc...

The "solution" is to try reducing the impact of these variables on the measurement. The other "solution" is to run the performance test not once but several times, potentially hundreds or thousands of times. This will give us a better representation of the performance of what we test.

One thing also to consider, there is little value to compare values obtained on different machines. What we can and will do is measure the performance of a single action on the same machine before and after applying a patch to see its impact on performances.

Performance tools

Luckily for us, performance tools already exist. But what are the required performance tools features?

  • Track a remote repository and associate measurement with each commit.
  • Build the environment for the selected commit.
  • Run the performance test suite for the built commit.
  • Save the results. As we run performance tests several times, it's useful to store more than 1 value (the median of the results for example) in order to be able to quantify the quality of the run.
  • Generate a report, in HTML it would be good.

For our use-case, we chose ASV (https://github.com/airspeed-velocity/asv) which stands for airspeed velocity. The documentation can be found here: https://asv.readthedocs.io/en/stable/. Our repository with the config and the performance tests are here: https://bitbucket.org/octobus/bighgperf.

Why ASV

ASV is handling a lot for us:

  • Allow tracking a mercurial repository, better as we are benchmarking mercurial itself.
  • The building part is already tailored for Python projects.
  • Run all the test suite or only a subset of tests for the built commit.
  • Store the results in JSON file, one directory per machine.
  • Create an easy to host HTML report.

The bonus features of ASV are nice:

  • ASV allows passing a complex set of commits to tests (for example all tagged commits on a specific branch).
  • It can also take only a subset of passed commits, you can ask it to take only 10 commits in the whole repository history. Very useful to quickly get a glance at the performance evolution of a project or release cycle.
  • Auto-detection of regressions between commits or range of commits. Like operation A is 1.5x slower between commit N and N+1. Or operation B is 1.8x slower between tag X and X+1.
  • Automatic bisection of regression. In case of a regression in a range of commits, ASV has the tooling to bisect which is the commits that introduced the regression, very useful to pinpoint the responsible commit.

In the next blog posts about ASV we will show you how to get started with ASV to start benchmarking your code.

Further reading

We made a presentation, in French, on the subject: https://octobus.net/presentations/perf_test.html.

Our friend Victor Stinner, a Python Core-Developer, also makes very good blog posts about the subject: https://vstinner.readthedocs.io/benchmark.html.

New Evolve extension release: version 8.0.1

2018-06-11

We pushed a new release for the Mercurial's evolve extension: 8.0.1

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

Version changelog

evolve (8.0.1)

  • compatibility with mercurial 4.6.1,
  • next-prev: respect commands.update.check config option (issue5808),
  • next-prev: fix evolve --abort on conflicts (issue5897),
  • obslog: fix breakage when commit has no description,
  • amend: use context manager for locks (issue5887),
  • evolve: fix detection of interactive shell.

topic (9.0.1)

  • topic: fix documentation formatting.

Mercurial Mini-Sprint - May 2018

2018-05-28

From Monday 21th to may 25th, around 12 people gathered in the Logilab office to discuss version control and hack on Mercurial. A big thanks to Logilab for hosting us. We covered multiple topics:

Mercurial 4.6

  • Fixing Mercurial 4.6 compatibility for the following extensions:
  • hgsubversion,
  • hg-git (and preparation of the associated release),
  • confman,
  • preparing a Windows installer for tortoise-hg 4.6

hgweb

  • First version of a Readme extension that displays the lasted content of a project README on hgweb landing page,
  • Various usability improvements for hg serve (automatic browser opening, printing URL, etc),
  • Displaying obsolescence log information in hgweb,
  • Ability to view and exchange hidden changeset through hgweb,
  • Discussion how to facilitate theme development and maintenance.

Performance tracking

  • Upstreaming of some of our work on ASV,
  • Work on benchmarking clone and stream clone time,
  • Discussion with a Git core developer about similar efforts in the git community.

Evolution and feature branches

  • New flag for obslog to filter out changesets that are unknown locally,
  • Improvement of core ability to do in memory merge and in memory diffing,
  • New message about local phase movement on pull,
  • Improvement of message around empty topics,
  • Implementation of an "internal phase" for changesets that are a byproduct of Mercurial command (eg: evolve, shelve).
  • Discussion about feature branch and workflow.

Mercurial hosting

  • Installation of an experimental heptapod instance (gitlab + Mercurial) for further testing,
  • Mercurial compatibility for the Gitlab CI runners.

Other

  • Experiment with tortoise-hg styling to get a mode closer to what hgview does,
  • Implementing strip-less shelve using the new internal phase,
  • Demo of an experimental test orchestrator atop the Mercurial test runner,
  • Refactoring of context.py to better facilitate in-memory merges.

Thanks to Anton Shestakov, Arthur Lutz, Aurelien Campeas, Boris Feld, Christian Couder, David Douard, Denis Laxalde, Nicolas Spanti, Paul Morelle, Phillipe Pépiot, Pierre-Yves David and Sean Farley for being part of this event. Extra thanks go to Pulkit Goyal for remotely helping with the shelve code.

New Evolve extension release: version 8.0.0

2018-04-25

We pushed a new release for the Mercurial's evolve extension: 8.0.0

This extension extends core features around history rewriting and draft changesets sharing.

As usual, the release is available on pypi and upgrade is recommended.

The release drops support for Mercurial 4.1 and 4.2 and adds support for Mercurial 4.6 to be released in a couple of days. Support for some deprecated flag or template has been dropped too.

Version changelog

evolve (8.0.0)

New feature

  • evolve: a new --abort flag which aborts an interrupted evolve resolving orphans,
  • hg evolve now return 0 if there is nothing to evolve,
  • amend: a new --patch flag to make changes to current changeset by editing patch,

Bug fixes

  • evolve: fixed some memory leak issue,
  • evolve: prevent some crash with merge and split (issue5833 and issue5832),
  • evolve: improved support for solving phase-divergence situation,
  • evolve: improved support for solving orphan situation,
  • obs-discovery: added unit to various progress bars,
  • evolve: record "operation" for command where it was missing,

Compatibility changes

(for both evolve version 8.0.0 and topic version 0.9.0)

  • compatibility with Mercurial 4.6,
  • drop support for Mercurial 4.1 and 4.2,
  • --obsolete and --old-obsolete flags for hg graft are dropped,
  • templatekw: remove obsfatedata templatekw. Individuals fields are available in core as single template functions,
  • topic: restraining name to letter, '-', '_' and '.'

New Evolve extension release: version 7.2.0

2018-01-16

We pushed a new release of evolve and topic: 7.2.0.

As usual, the release is available on pypi and upgrade is recommended.

This version comes with the usual various bug fixes and improvement. It also includes a large rework of the experimental obsolescence markers discovery protocol we call "obshashrange". The newer protocol is faster to compute and cache. It was able to handle all the repositories it has been tested on (up to around 1 millions revisions). The experimental is still turned off by default, use experimental.obshashrange=yes to use it.

Version changelog

evolve (7.2.0)

  • evolve: changes to the on-disk format for interrupted evolve
  • evolve: --continue now properly preserve phase (issue5720)
  • evolve: --continue now properly reports merges as evolve
  • commit: suggest using topic on new heads
  • uncommit: --revert flag added to clean the wdir after uncommit
  • obslog: add color support to content-diff output with --patch
  • fix hg prev behavior on obsolete changesets
  • no longer issues "obsolete working copy" message during no-op
  • use the new instabilities names from mercurial 4.4+ (in hg evolve --list and other messages)

New algorithm for obshashrange discovery:

The new algorithm is faster, simpler to cache and with better complexity. It is able to handle repository of any size (naive python implementation is a bit slow). Support for the previous experimental approach has been dropped, please update both clients and servers. This new discovery method is disabled by default. Use experimental.obshashrang=yes on both client and server.

topic (0.7.0)

  • fix compatibility with Mercurial-4.3
  • new template keyword topic to get changesets topic

New Evolve extension release: version 7.0.1

2017-11-14

We just pushed small bug fix release for Evolve: 7.0.1.

As usual, the release is available on pypi and upgrade is recommended.

Version changelog

evolve (7.0.1)

  • obsdiscovery: extend the config option to disable discovery to server-side (it was previously only honored on the client side),
  • server: avoid exposing 'abort' to evolution enabled client talking to server with the extension bu obsolescence marker exchange disabled.

topic (0.5.1)

  • fix new-heads check when pushing new topic with --publish.

New ConfigExpress extension relase: 0.3.0

2017-11-03

The Config Express extension allows for clients to share more details with the server and the server to push recommended config to the client.

Version 0.3.0 has just been published. It brings various improvements and bug fixes:

  • improvement to the extension documentation,
  • Mercurial-4.4 compatibility,
  • fixes configuration recommendations on windows,
  • allows client to send the list of enabled extensions and their version.

New Evolve extension release: version 7.0.0

2017-11-02

We just pushed a release for Evolve: 7.0.0.

As usual, the release is available on pypi and upgrade is recommended.

This release does not bring new features to evolve but drop compatibility for older Mercurial versions (4.0 and below) and drop multiple old and deprecated protocols. This allowed us to clean up the code and should help the situation where Evolve is selectively turned on only some repository server side.

On the other hand, Topic got multiple new workflow related options to experiments with.

Version changelog

evolve (7.0.0)

  • drop compatibility with Mercurial 3.8, 3.9 and 4.0,
  • drop support for old and deprecated method to exchange obsmarkers (now requires bundle2 enabled clients),
  • forbid usage of the old pushkey based protocol to exchange obsmarkers,
  • evolve: rename --contentdivergent flag to --content-divergent,
  • evolve: rename --phasedivergent flag to --phase-divergent.

topic (0.5.0)

  • add an experimental flag to enforce one head per name policy, (off by default, see hg help -e topic for details),
  • add an experimental flag to have changesets without topic published on push, (off by default, see 'hg help -e topic' for details),
  • add a --publish flag to hg push (4.4+ only).

New Evolve extension release: version 6.8.0

2017-10-23

We just pushed a new release for evolution: 6.8.0.

As usual, the release is available on pypi and upgrade is recommended.

The main thing about this release is compatibility with the Mercurial 4.4-rc. Multiple bits of evolve have been uploaded into core recently and evolution now relies on them when available.

Be advised that we are planning to drop compatibility for multiple older Mercurial versions and deprecated older experimental protocols in the next version. Our current plan is to support Mercurial 4.1 and above in the next release (7.0.0).

Version changelog

evolve (6.8.0)

  • compatibility with Mercurial 4.4 (use upstream implementation for obsfate and effect-flags starting hg 4.4+).
  • pager: pager support to obslog and evolve --list.

topic(0.4.0)

  • topic: fix handling of bookmarks and phases while changing topics. (Mercurial 4.2 and above only)
  • topic: fix 'topic-mode' behavior when amending.
  • pager: pager support to topics and stack.

New evolve extension release: version 6.7.0

2017-09-28

We just pushed a new release for evolution: 6.7.0.

As usual, the release is available on pypi and upgrade is recommended.

The most notable new features are the --interactive flag support for hg amend --extract (also known as hg uncommit) and a command to help migration from bookmark to topic (hg debugconvertbookmark).

It also contains multiple small improvements to both evolve and topic. In particular regarding documentation following the evolve mini-sprint in August.

We would like to thank all the people who helped with this version of evolve during the August minisprint and the Pycon-fr sprint last week. Here is the contributor list for this release:

  • Aurélien Campéas
  • Boris Feld
  • Denis Laxalde
  • FUJIWARA Katsunori
  • Philippe Pépiot
  • Pierre-Yves David
  • Pulkit Goyal
  • Ryan McElroy

Version changelog

evolve (6.7.0)

  • compatibility with change in future 4.4 at this release date,
  • documentation: improvement to content, wording and graphs,
  • obslog: improved templatability,
  • obslog/log: improve verb used to describe and evolution,
  • pstatus/pdiff: update to full command. They now appears in the help,
  • uncommit: add a --interactive option (4.3+ only).

topic (0.3.0)

  • push: add a --topic option to mirror --bookmark and --branch,
  • stack: improve display of interleaved topic,
  • stack: improve display of merge commit,
  • topic: add a new 'debugconvertbookmark' commands (4.3+ only), It helps migrating from bookmark feature branch to topic feature branch,
  • topic: --age flag also shows the user who last touched the topic,
  • topic: be more informative about topic activation and deactivation,
  • topic: gain a --current flag,
  • topic: small clarification and cleanup on various output.

Evolve documentation mini-sprint

2017-09-17

A few weeks ago we organized a mini-sprint on Evolve to improve the project documentation, smoothen some rough-edges and makes the project more newcomers friendly.

The sprint has gathered several folks:

  • Boris Feld (me),
  • Pierre-Yves David (Octobus Founder),
  • Pulkit Goyal invited by Octobus,
  • Denis Laxalde (from Logilab),
  • Philippe Pépiot (from Logilab),
  • Christophe de Vienne (from orus.io) who made a nice debrief,
  • Aurélien campéas (from Pythonian),
  • Ryan McElroy from (from Facebook).

The sprint was hosted by Logilab in Paris for two days. Thanks a lot!

The sprint was focused on three main goals:

Improve the documentation content

A big goal of the sprint was to improve the evolve documentation. Pierre-Yves, Philippe Pepiot and Ryan McElroy worked in this subject to:

  • Update installation instructions to use pip, it should facilitate the extension installation for newcomers.
  • Reorder and lighten the landing page.
  • Various fixes of typos, cases consistency and general rewording.
  • Missing and outdated content is now listed in a page, providing raw pointers. This should help reader to be aware of what they are missing and investigate on their own.
  • Content from the inline documentation is now directly reused in the online sphinx documentation (eg: hg help amend).

The online documentation now reflect these changes.

Improve the documentation tooling

Multiple thing that was desperately missing from evolve tutorials were history graphs. We wanted to have graphs like this one:

graphviz

We improved one of our extension, mercurial-docgraph, which can generate a graphviz img based on a Mercurial repository and a revset.

Boris Feld and Christophe de Vienne worked to:

  • Make it a proper Mercurial extension.
  • Add command line arguments to choose whether to output a png image or the graphviz dot content on stdout.
  • Also add support for --sphinx-directive option that output a sphinx graphviz extension compatible directive on stdout.
  • Work on grouping and aligning nodes.
  • Polish the theme used.

The evolve tutorials are now updated with nice graphs.

Usability bugs

Pulkit Goyal worked on:

  • Add the --current option to hg topic to helps setting the current topic to several changesets at the time.
  • Restrict topic names to classic rules for tags and branches (no integer only name, no reserved words...).
  • Add a new debugconvertbookmark command to convert bookmarks to topics.
  • Improving various wording and rendering of various topic related commands.
  • add a --interactive support for hg amend --extract (also known as uncommit).

Bugs cleaning

  • Fix a bug where obsolete tag could sneak back into the tag file (issue5539).
  • Improve behavior of rebase when obsolete and unstable revision are present (issue5300) (still in progress).

Thanks again to all attendees for the progress made and to Logilab for hosting. We are looking forward to the next mini-sprint.

Mercurial 4.3 release

2017-08-02

Mercurial 4.3 has been released and it's a pretty big release, you can find the release note here or below where we will highlight some changes:

Notable changes

  • Support for Python 2.6 has been dropped, so if you are still using it, stay on version 4.2 or update to Python 2.7.
  • Bundles now store phase information, you can now safely share draft and public changesets with your colleagues using bundles.
  • Two security issues have been fixed related to ssh repository URL and symlinks, for all details, see below.
  • The share extension now properly shares relevant caches between shares.
  • Server can now allow more concurrent pushes as long as they do not affect the same heads.

Multiple interesting experiments are going on:

  • Status is gaining a 'terse' mode. In this mode, the status of files is aggregated by directory.
  • A new option to improve manifest storage on repository with many branches has been added.
  • The sparse extensions have been integrated into the core distribution as experimental.

There have been a couple of update on the changeset evolution side:

  • Strip extension now removes relevant obsmarkers. So if you want to get rid of an amend, strip the successor and relevant obs-markers will be stripped too. The backup bundle will contain the stripped obsmarkers so they can be restored if necessary.
  • pushs and pulls now informs the user when changesets get obsoleted.

A handful of significant internal reworks happened in this release:

  • The support for Python 3 has been improved, every release brings us closer to be fully Python 3 compatible.
  • Transactions are now storing precise data about the changes it contains. This allows for accurate reports and more advanced extensions logic.
  • The rebase extensions internals were reworked to allow more advance rebase scenario with multiple destinations at the same time.
  • Config options can now be formally registered. This will unlock nice feature in the future like config validation and automatic documentation generation.

Security issue

The 4.3 release contains a fix for two CVE: CVE-2017-1000115 and CVE-2017-1000116. Upgrade as soon as possible. See the mailing-list for more information about the vulnerabilities.

Please be aware that the 4.3 release drop support for Python 2.7. Due to the vulnerabilities, a 4.2.3 version has been released with the security fixes. Upgrade to 4.2.3 if you are still using Python 2.6. You can download the release here.

We had to backport the security patches for Mercurial 4.1.3 for some of our customers. Due to the severity of the vulnerabilities, we made the patches available for everyone at the following address: https://bitbucket.org/octobus/mercurial-backport/branch/backport-4.1.

User-visible changes

Packaging

We did the necessary work to generate Linux wheels for the 4.3 release. For people who don't know what a wheel is, it's basically a compiled packaged of Mercurial installable by pip. We built wheels for Linux:

  • No need for a compiler
  • Declaratives metadata
  • Faster installation

On my laptop, the installation of Mercurial through pip went from 13 seconds with the default package to 1 second only.

If you want to test them, here are the instructions but be aware that we provide them as an experiment, you are using them at your own risk:

Run pip install -U --index-url=https://packagecloud.io/octobus/Mercurial/pypi/simple mercurial to update your mercurial installation with the 4.3 release of mercurial. Be warned that they are only Linux wheels, Windows wheels are available on Pypi and should be selected automatically. Mac OS X wheels are not available yet.

Behavior changes

There is a couple of backward incompatible changes, be sure to take a look at them! We compile a non-exhaustive list of the most notable ones:

  • The backup bundle name has changed for most operations.
  • The clonebundle hook arguments have changed.
  • hg update now show the commit it updated to in case of multiple heads.

Sharing caches

The share extension helps working with big repositories. It can share the 'store' (ie history) portion of a repository between multiple working copy. One thing was still painful, the various caches were not shared so each repository has to compute the phase cache, tag cache, bookmark cache, etc... which takes a significant amount of time per repository.

But that was before and with Mercurial 4.3 caches are now correctly shared, improving the performance when working with big repositories.

Concurrent pushes

When, pushing, a client first exchanges data with the server to discover the minimal information that actually needs to be pushed. Then it prepares a bundle containing these information and push it. It is possible for another client to update the repository between the discovery phase and the actual push phase. Voiding the validation that the client might have made before sending the data. Mercurial has a mechanism to detect such situation and abort the raced push.

This race detection can get in the way for repository with a lot of incoming traffic. So we introduced a new server.concurrent-push-mode option to control this behavior. The server config can be setup to accept pushed prepared concurrently if they affect unrelated heads.

    [server]
    concurrent-push-mode = related

Change in the Experimental realms

Manifest size

We have been investigating a sudden increase in the manifest size for some of our clients. The issue comes from a heuristic responsible of triggering new full snapshots. That heuristic fails badly when the number of concurrent branches becomes too large.

During the 4.3 cycle, we have upstreamed a new experimental option named 'maxdeltachainspan'. With this option set to 0, we have seen a reduction of manifest size in the order of 70x in our clients. This impacts push, pull and day-to-day commands durations. The manifest size exposition triggers when the number of parallel branch growth. So the more branches and merges you have, the bigger the win should be.

If you want to test it against your repositories:

  1. Make a new local clone
  2. Update the config to contains

        [format]
        aggressivemergedeltas = yes
        [experimental]
        maxdeltachainspan = 0 # no limit
    

  3. Run hg debugupgraderepo --optimize redeltaall --run --config format.aggressivemergedeltas=yes --config experimental.maxdeltachainspan=0 (this will likely take a long time).

Terse status

Pulkit Goyal started to integrate the terse status feature in the core hg status commands. When this flag is passed, directory summary will be used whenever possible. For example:

    $ hg status
    ? foo/x
    ? foo/y

Becomes:

    $ hg status
    ? foo/

[Experimental] - Sparse extension

The sparse extension is now shipped as an experimental extension. It is helpful with big mono-repositories when you only want a subset of your repository files to be checked out.

All the information can be found on the sparse extension documentation accessible with: hg help sparse.

Contribution statistics

To conclude, we compiled contribution in number of changesets from various contribution sources. Changesets are not an absolute metric, but it helps to get an idea of where contributions come from. Contributors from the same company have been grouped for clarity.

361 @octobus.net
291 @google.com
220 @fb.com
197 yuya@tcha.org
127 @mozilla.com
104 7895pulkit@gmail.com
84 matt_harbison@yahoo.com
61 foozy@lares.dti.ne.jp
18 sean@farley.io
12 kbullock@ringworld.org
6 wbruna@softwareexpress.com.br
6 @logilab.com
5 @nokia.com
4 demelier.david@gmail.com
4 danek.duvall@oracle.com
4 av6@dwimlabs.net
3 rishabhmadan96@gmail.com
3 marutosijp2@gmail.com
2 @unity.com
2 steve@borho.org
2 bamccaig@gmail.com
1 mtietze@gmx.com
1 cryo@cyanite.org
1 blacktrash@gmx.net
1 andrew.zwicky@gmail.com