New Evolve extension release: version 7.2.0

2018-01-16

We pushed a new release of evolve and topic: 7.2.0.

As usual, the release is available on pypi and upgrade is recommended.

This version comes with the usual various bug fixes and improvement. It also includes a large rework of the experimental obsolescence markers discovery protocol we call "obshashrange". The newer protocol is faster to compute and cache. It was able to handle all the repositories it has been tested on (up to around 1 millions revisions). The experimental is still turned off by default, use experimental.obshashrange=yes to use it.

Version changelog

evolve (7.2.0)

  • evolve: changes to the on-disk format for interrupted evolve
  • evolve: --continue now properly preserve phase (issue5720)
  • evolve: --continue now properly reports merges as evolve
  • commit: suggest using topic on new heads
  • uncommit: --revert flag added to clean the wdir after uncommit
  • obslog: add color support to content-diff output with --patch
  • fix hg prev behavior on obsolete changesets
  • no longer issues "obsolete working copy" message during no-op
  • use the new instabilities names from mercurial 4.4+ (in hg evolve --list and other messages)

New algorithm for obshashrange discovery:

The new algorithm is faster, simpler to cache and with better complexity. It is able to handle repository of any size (naive python implementation is a bit slow). Support for the previous experimental approach has been dropped, please update both clients and servers. This new discovery method is disabled by default. Use experimental.obshashrang=yes on both client and server.

topic (0.7.0)

  • fix compatibility with Mercurial-4.3
  • new template keyword topic to get changesets topic

New Evolve extension release: version 7.0.1

2017-11-14

We just pushed small bug fix release for Evolve: 7.0.1.

As usual, the release is available on pypi and upgrade is recommended.

Version changelog

evolve (7.0.1)

  • obsdiscovery: extend the config option to disable discovery to server-side (it was previously only honored on the client side),
  • server: avoid exposing 'abort' to evolution enabled client talking to server with the extension bu obsolescence marker exchange disabled.

topic (0.5.1)

  • fix new-heads check when pushing new topic with --publish.

New ConfigExpress extension relase: 0.3.0

2017-11-03

The Config Express extension allows for clients to share more details with the server and the server to push recommended config to the client.

Version 0.3.0 has just been published. It brings various improvements and bug fixes:

  • improvement to the extension documentation,
  • Mercurial-4.4 compatibility,
  • fixes configuration recommendations on windows,
  • allows client to send the list of enabled extensions and their version.

New Evolve extension release: version 7.0.0

2017-11-02

We just pushed a release for Evolve: 7.0.0.

As usual, the release is available on pypi and upgrade is recommended.

This release does not bring new features to evolve but drop compatibility for older Mercurial versions (4.0 and below) and drop multiple old and deprecated protocols. This allowed us to clean up the code and should help the situation where Evolve is selectively turned on only some repository server side.

On the other hand, Topic got multiple new workflow related options to experiments with.

Version changelog

evolve (7.0.0)

  • drop compatibility with Mercurial 3.8, 3.9 and 4.0,
  • drop support for old and deprecated method to exchange obsmarkers (now requires bundle2 enabled clients),
  • forbid usage of the old pushkey based protocol to exchange obsmarkers,
  • evolve: rename --contentdivergent flag to --content-divergent,
  • evolve: rename --phasedivergent flag to --phase-divergent.

topic (0.5.0)

  • add an experimental flag to enforce one head per name policy, (off by default, see hg help -e topic for details),
  • add an experimental flag to have changesets without topic published on push, (off by default, see 'hg help -e topic' for details),
  • add a --publish flag to hg push (4.4+ only).

New Evolve extension release: version 6.8.0

2017-10-23

We just pushed a new release for evolution: 6.8.0.

As usual, the release is available on pypi and upgrade is recommended.

The main thing about this release is compatibility with the Mercurial 4.4-rc. Multiple bits of evolve have been uploaded into core recently and evolution now relies on them when available.

Be advised that we are planning to drop compatibility for multiple older Mercurial versions and deprecated older experimental protocols in the next version. Our current plan is to support Mercurial 4.1 and above in the next release (7.0.0).

Version changelog

evolve (6.8.0)

  • compatibility with Mercurial 4.4 (use upstream implementation for obsfate and effect-flags starting hg 4.4+).
  • pager: pager support to obslog and evolve --list.

topic(0.4.0)

  • topic: fix handling of bookmarks and phases while changing topics. (Mercurial 4.2 and above only)
  • topic: fix 'topic-mode' behavior when amending.
  • pager: pager support to topics and stack.

New evolve extension release: version 6.7.0

2017-09-28

We just pushed a new release for evolution: 6.7.0.

As usual, the release is available on pypi and upgrade is recommended.

The most notable new features are the --interactive flag support for hg amend --extract (also known as hg uncommit) and a command to help migration from bookmark to topic (hg debugconvertbookmark).

It also contains multiple small improvements to both evolve and topic. In particular regarding documentation following the evolve mini-sprint in August.

We would like to thank all the people who helped with this version of evolve during the August minisprint and the Pycon-fr sprint last week. Here is the contributor list for this release:

  • Aurélien Campéas
  • Boris Feld
  • Denis Laxalde
  • FUJIWARA Katsunori
  • Philippe Pépiot
  • Pierre-Yves David
  • Pulkit Goyal
  • Ryan McElroy

Version changelog

evolve (6.7.0)

  • compatibility with change in future 4.4 at this release date,
  • documentation: improvement to content, wording and graphs,
  • obslog: improved templatability,
  • obslog/log: improve verb used to describe and evolution,
  • pstatus/pdiff: update to full command. They now appears in the help,
  • uncommit: add a --interactive option (4.3+ only).

topic (0.3.0)

  • push: add a --topic option to mirror --bookmark and --branch,
  • stack: improve display of interleaved topic,
  • stack: improve display of merge commit,
  • topic: add a new 'debugconvertbookmark' commands (4.3+ only), It helps migrating from bookmark feature branch to topic feature branch,
  • topic: --age flag also shows the user who last touched the topic,
  • topic: be more informative about topic activation and deactivation,
  • topic: gain a --current flag,
  • topic: small clarification and cleanup on various output.

Evolve documentation mini-sprint

2017-09-17

A few weeks ago we organized a mini-sprint on Evolve to improve the project documentation, smoothen some rough-edges and makes the project more newcomers friendly.

The sprint has gathered several folks:

  • Boris Feld (me),
  • Pierre-Yves David (Octobus Founder),
  • Pulkit Goyal invited by Octobus,
  • Denis Laxalde (from Logilab),
  • Philippe Pépiot (from Logilab),
  • Christophe de Vienne (from orus.io) who made a nice debrief,
  • Aurélien campéas (from Pythonian),
  • Ryan McElroy from (from Facebook).

The sprint was hosted by Logilab in Paris for two days. Thanks a lot!

The sprint was focused on three main goals:

Improve the documentation content

A big goal of the sprint was to improve the evolve documentation. Pierre-Yves, Philippe Pepiot and Ryan McElroy worked in this subject to:

  • Update installation instructions to use pip, it should facilitate the extension installation for newcomers.
  • Reorder and lighten the landing page.
  • Various fixes of typos, cases consistency and general rewording.
  • Missing and outdated content is now listed in a page, providing raw pointers. This should help reader to be aware of what they are missing and investigate on their own.
  • Content from the inline documentation is now directly reused in the online sphinx documentation (eg: hg help amend).

The online documentation now reflect these changes.

Improve the documentation tooling

Multiple thing that was desperately missing from evolve tutorials were history graphs. We wanted to have graphs like this one:

graphviz

We improved one of our extension, mercurial-docgraph, which can generate a graphviz img based on a Mercurial repository and a revset.

Boris Feld and Christophe de Vienne worked to:

  • Make it a proper Mercurial extension.
  • Add command line arguments to choose whether to output a png image or the graphviz dot content on stdout.
  • Also add support for --sphinx-directive option that output a sphinx graphviz extension compatible directive on stdout.
  • Work on grouping and aligning nodes.
  • Polish the theme used.

The evolve tutorials are now updated with nice graphs.

Usability bugs

Pulkit Goyal worked on:

  • Add the --current option to hg topic to helps setting the current topic to several changesets at the time.
  • Restrict topic names to classic rules for tags and branches (no integer only name, no reserved words...).
  • Add a new debugconvertbookmark command to convert bookmarks to topics.
  • Improving various wording and rendering of various topic related commands.
  • add a --interactive support for hg amend --extract (also known as uncommit).

Bugs cleaning

  • Fix a bug where obsolete tag could sneak back into the tag file (issue5539).
  • Improve behavior of rebase when obsolete and unstable revision are present (issue5300) (still in progress).

Thanks again to all attendees for the progress made and to Logilab for hosting. We are looking forward to the next mini-sprint.

Mercurial 4.3 release

2017-08-02

Mercurial 4.3 has been released and it's a pretty big release, you can find the release note here or below where we will highlight some changes:

Notable changes

  • Support for Python 2.6 has been dropped, so if you are still using it, stay on version 4.2 or update to Python 2.7.
  • Bundles now store phase information, you can now safely share draft and public changesets with your colleagues using bundles.
  • Two security issues have been fixed related to ssh repository URL and symlinks, for all details, see below.
  • The share extension now properly shares relevant caches between shares.
  • Server can now allow more concurrent pushes as long as they do not affect the same heads.

Multiple interesting experiments are going on:

  • Status is gaining a 'terse' mode. In this mode, the status of files is aggregated by directory.
  • A new option to improve manifest storage on repository with many branches has been added.
  • The sparse extensions have been integrated into the core distribution as experimental.

There have been a couple of update on the changeset evolution side:

  • Strip extension now removes relevant obsmarkers. So if you want to get rid of an amend, strip the successor and relevant obs-markers will be stripped too. The backup bundle will contain the stripped obsmarkers so they can be restored if necessary.
  • pushs and pulls now informs the user when changesets get obsoleted.

A handful of significant internal reworks happened in this release:

  • The support for Python 3 has been improved, every release brings us closer to be fully Python 3 compatible.
  • Transactions are now storing precise data about the changes it contains. This allows for accurate reports and more advanced extensions logic.
  • The rebase extensions internals were reworked to allow more advance rebase scenario with multiple destinations at the same time.
  • Config options can now be formally registered. This will unlock nice feature in the future like config validation and automatic documentation generation.

Security issue

The 4.3 release contains a fix for two CVE: CVE-2017-1000115 and CVE-2017-1000116. Upgrade as soon as possible. See the mailing-list for more information about the vulnerabilities.

Please be aware that the 4.3 release drop support for Python 2.7. Due to the vulnerabilities, a 4.2.3 version has been released with the security fixes. Upgrade to 4.2.3 if you are still using Python 2.6. You can download the release here.

We had to backport the security patches for Mercurial 4.1.3 for some of our customers. Due to the severity of the vulnerabilities, we made the patches available for everyone at the following address: https://bitbucket.org/octobus/mercurial-backport/branch/backport-4.1.

User-visible changes

Packaging

We did the necessary work to generate Linux wheels for the 4.3 release. For people who don't know what a wheel is, it's basically a compiled packaged of Mercurial installable by pip. We built wheels for Linux:

  • No need for a compiler
  • Declaratives metadata
  • Faster installation

On my laptop, the installation of Mercurial through pip went from 13 seconds with the default package to 1 second only.

If you want to test them, here are the instructions but be aware that we provide them as an experiment, you are using them at your own risk:

Run pip install -U --index-url=https://packagecloud.io/octobus/Mercurial/pypi/simple mercurial to update your mercurial installation with the 4.3 release of mercurial. Be warned that they are only Linux wheels, Windows wheels are available on Pypi and should be selected automatically. Mac OS X wheels are not available yet.

Behavior changes

There is a couple of backward incompatible changes, be sure to take a look at them! We compile a non-exhaustive list of the most notable ones:

  • The backup bundle name has changed for most operations.
  • The clonebundle hook arguments have changed.
  • hg update now show the commit it updated to in case of multiple heads.

Sharing caches

The share extension helps working with big repositories. It can share the 'store' (ie history) portion of a repository between multiple working copy. One thing was still painful, the various caches were not shared so each repository has to compute the phase cache, tag cache, bookmark cache, etc... which takes a significant amount of time per repository.

But that was before and with Mercurial 4.3 caches are now correctly shared, improving the performance when working with big repositories.

Concurrent pushes

When, pushing, a client first exchanges data with the server to discover the minimal information that actually needs to be pushed. Then it prepares a bundle containing these information and push it. It is possible for another client to update the repository between the discovery phase and the actual push phase. Voiding the validation that the client might have made before sending the data. Mercurial has a mechanism to detect such situation and abort the raced push.

This race detection can get in the way for repository with a lot of incoming traffic. So we introduced a new server.concurrent-push-mode option to control this behavior. The server config can be setup to accept pushed prepared concurrently if they affect unrelated heads.

    [server]
    concurrent-push-mode = related

Change in the Experimental realms

Manifest size

We have been investigating a sudden increase in the manifest size for some of our clients. The issue comes from a heuristic responsible of triggering new full snapshots. That heuristic fails badly when the number of concurrent branches becomes too large.

During the 4.3 cycle, we have upstreamed a new experimental option named 'maxdeltachainspan'. With this option set to 0, we have seen a reduction of manifest size in the order of 70x in our clients. This impacts push, pull and day-to-day commands durations. The manifest size exposition triggers when the number of parallel branch growth. So the more branches and merges you have, the bigger the win should be.

If you want to test it against your repositories:

  1. Make a new local clone
  2. Update the config to contains

        [format]
        aggressivemergedeltas = yes
        [experimental]
        maxdeltachainspan = 0 # no limit
    

  3. Run hg debugupgraderepo --optimize redeltaall --run --config format.aggressivemergedeltas=yes --config experimental.maxdeltachainspan=0 (this will likely take a long time).

Terse status

Pulkit Goyal started to integrate the terse status feature in the core hg status commands. When this flag is passed, directory summary will be used whenever possible. For example:

    $ hg status
    ? foo/x
    ? foo/y

Becomes:

    $ hg status
    ? foo/

[Experimental] - Sparse extension

The sparse extension is now shipped as an experimental extension. It is helpful with big mono-repositories when you only want a subset of your repository files to be checked out.

All the information can be found on the sparse extension documentation accessible with: hg help sparse.

Contribution statistics

To conclude, we compiled contribution in number of changesets from various contribution sources. Changesets are not an absolute metric, but it helps to get an idea of where contributions come from. Contributors from the same company have been grouped for clarity.

361 @octobus.net
291 @google.com
220 @fb.com
197 yuya@tcha.org
127 @mozilla.com
104 7895pulkit@gmail.com
84 matt_harbison@yahoo.com
61 foozy@lares.dti.ne.jp
18 sean@farley.io
12 kbullock@ringworld.org
6 wbruna@softwareexpress.com.br
6 @logilab.com
5 @nokia.com
4 demelier.david@gmail.com
4 danek.duvall@oracle.com
4 av6@dwimlabs.net
3 rishabhmadan96@gmail.com
3 marutosijp2@gmail.com
2 @unity.com
2 steve@borho.org
2 bamccaig@gmail.com
1 mtietze@gmx.com
1 cryo@cyanite.org
1 blacktrash@gmx.net
1 andrew.zwicky@gmail.com