Getting the Vim source with Mercurial

Tip 1653 Printable Monobook Previous Next

created May 14, 2010 · complexity basic · author Tonymec · version 7.0

This tip describes how to get the Vim source from the new Mercurial repository. It assumes that you know how to compile Vim once you have the source. Examples are for Linux but it shouldn't be hard to adapt them to whatever OS you're running on (provided of course that there exists a version of Mercurial that runs on your OS). Also, the examples are written for GNU make: in case of doubt, try replacing make by gmake everywhere.

Also take a look at the Mercurial instructions from vim.org.

First time only: Creating your repository

Make sure that Mercurial and Python are installed on your system

For example, check there is a program named hg and another named python in your PATH. If you haven't got Mercurial, install it, either (if available) by installing the Mercurial package from your distro, or else from the Mercurial site. As of May 2010 the Python version required by the Mercurial package is Python 2.4, but it seems to work fine with later versions like Python 2.6.2.

If you're uncomfortable with a command-line interface (or simply prefer a GUI), there are options available such as TortoiseHg to simplify common tasks with an easy-to-use GUI front-end. On Windows, the installer even includes Python and a command-line hg tool, so that you get everything you need in one package.

Install a copy of Bram's latest source, runtime and ancillary files

1. Set up a directory head "for building" and cd to it, e.g. (on Linux)

mkdir -p ~/.build/hg
cd ~/.build/hg

2. Clone Bram's repository (this may take some time, depending on the speed of your CPU and Internet connection):

(date && hg clone https://bitbucket.org/vim-mirror/vim vim) 2>&1 |tee hg-vim.log

The above is slightly more complicated than absolutely necessary because it can be useful to keep a running log. The important part starts with hg and ends immediately before the right parenthesis. Similarly for other hg commands below.

Simple case: Getting new patches

This applies if all of the following are true:

You have no local changes to the Vim source, ancillary and runtime files (including the Makefile; you can easily compile a custom Vim with no Makefile changes, as detailed in the compiling "how to" pages below)
You compile only one Vim and therefore don't use a shadow directory
You don't build a cscope database inside the Vim source tree

In this case, all you need to check for (and, if necessary, get) any changes to the Vim source is:

cd ~/.build/hg/vim
(date && hg pull -u) 2>&1 |tee -a ../hg-vim.log

Complex case: You have local changes and/or compile several Vim versions

Enable the fetch extension

Add the following to the .hg/hgrc file in your repository (it was created by the clone process, with Bram's repository as the default remote source):

[extensions]
hgext.fetch =

No need to add anything after the equal sign, this extension comes packed with Mercurial (but disabled by default)

Patch your `.hgignore` file

It is found at the top level of your repository. This step is unnecessary if you have neither additional help files nor additional Vim versions and you don't add a cscope database inside the Vim source tree. Copy the following patch and apply it with patch -p1 < hgignore.diff from the top directory where the file resides.

diff -r 2bd29808d1f6 -r a5e628a08c4e .hgignore
--- a/.hgignore Fri May 14 18:56:38 2010 +0200
+++ b/.hgignore Fri May 14 20:05:50 2010 +0200
@@ -30,12 +30,25 @@ src/auto/pathdef.c
 *.res
 *.RES
 src/pathdef.c
 src/Obj*/pathdef.c
 gvimext.dll
 gvimext.lib
 
 # All platforms
+runtime/doc/tags
 *.rej
 *.orig
 *.mo
 *~
+
+# shadow directories
+# the directory names could be anything but we restrict them
+# to shadow (the default) or shadow-*
+src/shadow
+src/shadow-*
+# src/runtime and src/pixmaps are softlinks needed for proper 'make install'
+# when in a shadow directory
+src/runtime
+src/pixmaps
+# avoid tracking cscope.out even if built here
+src/cscope.out

Then commit this change:

(date && hg commit -m 'Ignore shadow directories and help tag changes') 2>&1 |tee -a ../hg-vim.log

Create shadow directories

If you want to compile more than one version of Vim, create shadow directories to avoid conflicts between them. We create them with names starting with shadow- so .hgignore (after applying the above patch) will see them. Here is an example:

cd ~/.build/hg/vim/src
(date && SHADOWDIR='shadow-huge' make -e shadow) 2>&1 |tee -a ../../hg-vim.log
(date && SHADOWDIR='shadow-tiny' make -e shadow) 2>&1 |tee -a ../../hg-vim.log

The -e command-line switch is necessary so that make won't override our SHADOWDIR names with the default which is just shadow

Apply your local changes and commit them

You do this by editing the sources in the src directory. The case where you want to compile different versions of Vim with not only different configuration options but even different changes to the sources is not covered by the present tip, and neither is the case when your changes could conflict with Bram's (the changes I have consist of added sections in out-of-the-way parts of Makefile and feature.h plus Bill McCarthy's extra float functions).

Also, check that your "nonstandard" patches are mentioned in src/version.c so that they will appear in the :version output of your home-compiled Vim: here is the relevant part of mine, normally you would add one line (with a string followed by a comma) per extra patch with a /**/ comment between each of them:

static char *(extra_patches[]) =
{   /* Add your patch description below this line */
/**/
#ifdef FEAT_FLOAT
    "Extra float functions (Bill McCarthy)",
#endif
/**/
    NULL
};

Then, commit your local changes with

(date && hg commit -m 'Local source changes') 2>&1 |tee -a ../hg-vim.log

Get any new official patches and merge them with your local ones

cd ~/.build/hg/vim
(date && hg fetch --switch-parent) 2>&1 |tee -a ../hg-vim.log

The --switch-parent switch places the local directory (where changes are few and far between) as the "first" parent of any resulting merge, so that the local changes won't be removed and added back every time: this reduces the number of source files which make will see as "modified".

Before compiling, you may want to check any files listed as "merged" (possibly with the help of the hg diff function) to make sure that the changes are what you would expect.

Comments

Known bug: Logging and interactive merge

A problem which I haven't solved about the above method of keeping a running log, is that is doesn't mesh well with Mercurial's "interactive merge tool": if the interactive merge asks a question, you don't see it, and you will have to blind-type a guessed response.

In my experience, questions are about runtime/doc/tags being deleted locally, reintroduced on the remote repository, and the question is "Use (c)changed version or leave (d)eleted?" -- in that case I reply d and the fetch goes on. I get this on the first non-null fetch after a "make install".

If you hit Ctrl-C to break the run, you won't see what Mercurial has output so far: it is not in the log; and your local repository is left in an uncertain state: once I did this, but then "hg status" replied nothing and "hg fetch" gave an error saying local files needed update -C or merge. Finally "hg merge" (with no logging) gave me the c/d question again so I could bring back my local repo to a "sane" state by replying d then manually committing the merge (with a -m message inspired by the "description" on the remote head, as obtained with "hg heads").

— Tonymec 04:39, May 18, 2010 (UTC)

Workaround: First, use hg -v incoming (see below; -v means "verbose"). No need to log this since it makes no change to the repository. If it tells you "No changes found", no need to do anything more at the moment. Otherwise, it will tell you which files will be changed if you pull. Then if runtime/doc/tags is included, you know that fetch may stop until you type d<Enter>

— If there are many new changesets, and you want to exceptionally use (just this once) vim as a pager, repeat the command like this (all on one line):

hg -v --config 'extensions.hgext.pager=' --config 'pager.pager=view -' --config 'pager.attend=incoming' incoming

where the three --config arguments mean that this single command will be run as if your hgrc included the following (in addition tothe other lines already there):

[extensions]
hgext.pager=
[pager]
pager=view -
attend=incoming

i.e. enable the pager extension, use "view -" i.e. Vim read-only from stdin, as a pager, and enable the pager for the hg incoming command. — Tonymec 21:42, July 7, 2010 (UTC)

I finally found how to force Python programs (including Mercurial, which is written in Python) to always use unbuffered I/O on stdin, stdout and stderr. Simple: set the environment variable PYTHONUNBUFFERED to some nonempty string.

Example (for the bash shell on Unix-like systems):

cd ~/.build/hg/vim
PYTHONUNBUFFERED='unbuffered' (date && hg fetch --switch-parent) 2>&1 |tee -a ../hg-vim.log

CAUTION: It might be unwise to do this in your permanent environment, since unbuffering means more frequent and smaller transfers, thus reducing throughput. You may want to do it only when logging via |tee

— Tonymec (talk) 02:56, July 16, 2012 (UTC)

Useful commands to "get information"

hg -v help
hg -v help |less

what are the basic hg commands, and what do they do?

hg -v help command

(where command is a Mercurial command) what are the possible arguments to that command (and what do they mean)?
(-v or --verbose is necessary to get the full text of the concerned help section)

hg status

are there local files waiting for a disposition (commit possibly preceded by add and/or forget)? The reply is one filename per line, preceded by A (added to the list of tracked files), M (merged), D (deleted i.e. specifically not tracked), ? (new file, neither "tracked" nor "not tracked" yet).

hg heads

what are all the "tree leaves" ? (commits which are not a parent to a later commit)

hg log -l N

(where N is a number): show the N latest commits, with relative number and short hash, branch name if any, tags if any, parent(s) if not, or not only, the immediately older commit, and description (commit message)

hg log -l N -f

same as above, but only for the current tip and its parents and ancestors.

hg incoming

tells you (in the same format as hg log) which changesets (if any) are waiting for you to pull them.

hg diff -r xxxxxxxxxxxx [-r yyyyyyyyyyyy] [file ...]

where xxxxxxxxxxxx (and yyyyyyyyyyyy if present) are revision IDs: Produce a diff of the file(s) named, as they changed between the named revisions. With only one revision: compare it to the current tip. With no filenames: give all differences between these revisions.

Notes:

Contrary to what is said in the manpage, with no revision there is no output.
I find it useful to use |view - as a pager for the output of this command. Then a simple :saveas patchname.diff will, if you want to, create the patch file after you review it.
To always use view as pager for the more "bulky" output-producing Mercurial commands, add the following to your ~/.hgrc file (or, on Windows, your %USERPROFILE%\Mercurial.ini):

[extensions]
hgext.pager=
[pager]
pager = view -
attend = annotate, cat, diff, export, glog, log, qdiff

— Tonymec 09:43, May 24, 2010 (UTC)

Changing branches

Changing branches, or going to a "tagged" revision, are two of the functions of the hg update command (which has up as an alias) — in general, its function is to "update" the files in the current repository (the "working copy" as Mercurial names it) to any revision you specify. So:

To go to the Vim 7.2 source

hg up vim72

To go back to the latest sources

hg up default

To go to a specific patchlevel

(only for 7.3.142 or later)

hg up v7-3-155

Getting the Vim source with Mercurial

Contents

First time only: Creating your repository

Make sure that Mercurial and Python are installed on your system

Install a copy of Bram's latest source, runtime and ancillary files

Simple case: Getting new patches

Complex case: You have local changes and/or compile several Vim versions

Enable the fetch extension

Patch your `.hgignore` file

Create shadow directories

Apply your local changes and commit them

Get any new official patches and merge them with your local ones

See also

Manpages

The Mercurial online help

In this wiki

External links

Comments

Known bug: Logging and interactive merge

Useful commands to "get information"

Changing branches

To go to the Vim 7.2 source

To go back to the latest sources

To go to a specific patchlevel

Fan Feed

Getting the Vim source with Mercurial

First time only: Creating your repository

Make sure that Mercurial and Python are installed on your system

Install a copy of Bram's latest source, runtime and ancillary files

Simple case: Getting new patches

Complex case: You have local changes and/or compile several Vim versions

Enable the fetch extension

Patch your .hgignore file

Create shadow directories

Apply your local changes and commit them

Get any new official patches and merge them with your local ones

See also

Manpages

The Mercurial online help

In this wiki

External links

Comments

Known bug: Logging and interactive merge

Useful commands to "get information"

Changing branches

To go to the Vim 7.2 source

To go back to the latest sources

To go to a specific patchlevel

Fan Feed

Patch your `.hgignore` file