Editing Getting the Vim source with Mercurial

{{TipNew
|id=1653
|previous=1652
|next=1654
|created=May 14, 2010
|complexity=basic
|author=Tonymec
|version=7.0
|subpage=/201005
|category1=Building Vim
|category2=
}}
This tip describes how to get the Vim source from the new Mercurial repository. It assumes that you know how to compile Vim once you have the source. Examples are for Linux but it shouldn't be hard to adapt them to whatever OS you're running on (provided of course that there exists a version of Mercurial that runs on your OS). Also, the examples are written for GNU make: in case of doubt, try replacing <code>make</code> by <code>gmake</code> everywhere.

Also take a look at the [http://www.vim.org/mercurial.php Mercurial instructions from vim.org].

==First time only: Creating your repository==
===Make sure that Mercurial and Python are installed on your system===
For example, check there is a program named <code>hg</code> and another named <code>python</code> in your PATH. If you haven't got Mercurial, install it, either (if available) by installing the <code>Mercurial</code> package from your distro, or else from the [http://mercurial.selenic.com/ Mercurial site]. As of May 2010 the Python version required by the Mercurial package is Python 2.4, but it seems to work fine with later versions like Python 2.6.2.

If you're uncomfortable with a command-line interface (or simply prefer a GUI), there are options available such as [http://tortoisehg.bitbucket.org/ TortoiseHg] to simplify common tasks with an easy-to-use GUI front-end. On Windows, the installer even includes Python and a command-line hg tool, so that you get everything you need in one package.

===Install a copy of Bram's latest source, runtime and ancillary files===
1. '''Set up a directory head''' "for building" and cd to it, e.g. (on Linux)
<pre>
mkdir -p ~/.build/hg
cd ~/.build/hg
</pre>
2. '''Clone Bram's repository'''
(this may take some time, depending on the speed of your CPU and Internet connection):
<pre>
(date && hg clone http://hg.256bit.org/vim vim) 2>&1 |tee hg-vim.log
</pre>
The above is slightly more complicated than absolutely necessary because it can be useful to keep a running log. The important part starts with <code>hg</code> and ends immediately before the right parenthesis. Similarly for other hg commands below.

==Simple case: Getting new patches==
This applies if '''all''' of the following are true:
*You have no local changes to the Vim source, ancillary and runtime files ''(including the Makefile; you can easily compile a custom Vim with no Makefile changes, as detailed in the [[#External links|compiling "how to" pages]] below)''
*You compile only one Vim and therefore don't use a shadow directory
*You don't build a cscope database inside the Vim source tree
In this case, all you need to check for (and, if necessary, get) any changes to the Vim source is:
<pre>
cd ~/.build/hg/vim
(date && hg pull -u) 2>&1 |tee -a ../hg-vim.log
</pre>

==Complex case: You have local changes and/or compile several Vim versions==
===Enable the fetch extension===
Add the following to the .hg/hgrc file in your repository (it was created by the clone process, with Bram's repository as the default remote source):
<pre>
[extensions]
hgext.fetch =
</pre>
No need to add anything after the equal sign, this extension comes packed with Mercurial (but disabled by default)

===Patch your <code>.hgignore</code> file===
It is found at the top level of your repository. This step is unnecessary if you have '''neither''' additional help files nor additional Vim versions '''and''' you don't add a cscope database inside the Vim source tree.
Copy the following patch and apply it with <code>patch -p1 < hgignore.diff</code> from the top directory where the file resides.
<pre>
diff -r 2bd29808d1f6 -r a5e628a08c4e .hgignore
--- a/.hgignore Fri May 14 18:56:38 2010 +0200
+++ b/.hgignore Fri May 14 20:05:50 2010 +0200
@@ -30,12 +30,25 @@ src/auto/pathdef.c
 *.res
 *.RES
 src/pathdef.c
 src/Obj*/pathdef.c
 gvimext.dll
 gvimext.lib
&nbsp;
 # All platforms
+runtime/doc/tags
 *.rej
 *.orig
 *.mo
 *~
+
+# shadow directories
+# the directory names could be anything but we restrict them
+# to shadow (the default) or shadow-*
+src/shadow
+src/shadow-*
+# src/runtime and src/pixmaps are softlinks needed for proper 'make install'
+# when in a shadow directory
+src/runtime
+src/pixmaps
+# avoid tracking cscope.out even if built here
+src/cscope.out
</pre>

Then commit this change:
<pre>
(date && hg commit -m 'Ignore shadow directories and help tag changes') 2>&1 |tee -a ../hg-vim.log
</pre>

===Create shadow directories===
If you want to compile more than one version of Vim, create shadow directories to avoid conflicts between them. We create them with names starting with <code>shadow-</code> so <code>.hgignore</code> (after applying the above patch) will see them. Here is an example:
<pre>
cd ~/.build/hg/vim/src
(date && SHADOWDIR='shadow-huge' make -e shadow) 2>&1 |tee -a ../../hg-vim.log
(date && SHADOWDIR='shadow-tiny' make -e shadow) 2>&1 |tee -a ../../hg-vim.log
</pre>
The -e command-line switch is necessary so that make won't override our <code>SHADOWDIR</code> names with the default which is just <code>shadow</code>

===Apply your local changes and commit them===
You do this by editing the sources in the src directory. The case where you want to compile different versions of Vim with not only different configuration options but even different changes to the sources is not covered by the present tip, and neither is the case when your changes could conflict with Bram's (the changes I have consist of added sections in out-of-the-way parts of <code>Makefile</code> and <code>feature.h</code> plus Bill McCarthy's extra float functions).

Also, check that your "nonstandard" patches (if any) are mentioned in <code>src/version.c</code> so that they will appear in the <code>:version</code> output of your home-compiled Vim: here is the relevant part of mine, normally you would add one line (with a string followed by a comma) per extra patch with a <code>/**/</code> comment between each of them:
<pre>
static char *(extra_patches[]) =
{   /* Add your patch description below this line */
/**/
#ifdef FEAT_FLOAT
    "Extra float functions (Bill McCarthy)",
#endif
/**/
    NULL
};
</pre>

Then, commit your local changes with
<pre>
(date && hg commit -m 'Local source changes') 2>&1 |tee -a ../hg-vim.log
</pre>

===Get any new official patches and merge them with your local ones===
<pre>
cd ~/.build/hg/vim
(date && hg fetch --switch-parent) 2>&1 |tee -a ../hg-vim.log
</pre>
The <code>--switch-parent</code> switch places the local directory (where changes are few and far between) as the "first" parent of any resulting merge, so that the local changes won't be removed and added back every time: this reduces the number of source files which make will see as "modified".

Before compiling, you may want to check any files listed as "merged" (possibly with the help of the <code>hg diff</code> function) to make sure that the changes are what you would expect.

==See also==
===Manpages===
*<code>man hg</code>
*<code>man hgrc</code>
*<code>man hgignore</code>

===The Mercurial online help===
In my experience, second only to the Vim help itself. It includes the whole contents of the [[#Manpages|manpages]], and more.
*<code>hg help</code>
*<code>hg help ''subject''</code>
where <code>''subject''</code> is a Mercurial command ''(e.g. <code>pull</code>)'' or other help subject ''(e.g. <code>extensions</code>).''
: '''Note:''' The full text of the help is only obtained with <code>-v</code> or <code>--verbose</code>, or when the <code>[ui]</code> section of your <code>hgrc</code> (or, on Windows, <code>Mercurial.ini</code>) contains the line <code>verbose=1</code>

===In this wiki===
*[[:Category:Building Vim]]

===External links===
*[http://mercurial.selenic.com/ The Mercurial site]
*[http://mercurial.selenic.com/guide/ The Mercurial Guide]
*[http://selenic.com/mailman/listinfo/mercurial The Mercurial mailing list]
*[http://users.skynet.be/antoine.mechelynck/vim/compile.htm Compiling Vim on Windows]
*[http://users.skynet.be/antoine.mechelynck/vim/compunix.htm Compiling Vim on Unix/Linux]

==Comments==
===Known bug: Logging and interactive merge===
A problem which I haven't solved about the above method of keeping a running log, is that is doesn't mesh well with Mercurial's "interactive merge tool": if the interactive merge asks a question, you don't see it, and you will have to blind-type a guessed response.

In my experience, questions are about runtime/doc/tags being deleted locally, reintroduced on the remote repository, and the question is "Use (c)changed version or leave (d)eleted?" -- in that case I reply d and the fetch goes on. I get this on the first non-null fetch after a "make install".

If you hit Ctrl-C to break the run, you won't see what Mercurial has output so far: it is not in the log; and your local repository is left in an uncertain state: once I did this, but then "hg status" replied nothing and "hg fetch" gave an error saying local files needed update -C or merge. Finally "hg merge" (with no logging) gave me the c/d question again so I could bring back my local repo to a "sane" state by replying d then manually committing the merge (with a -m message inspired by the "description" on the remote head, as obtained with "hg heads").
:— [[User:Tonymec|Tonymec]] 04:39, May 18, 2010 (UTC)
:'''Workaround:''' First, use <code>hg -v incoming</code> (see below; <code>-v</code> means "verbose"). No need to log this since it makes no change to the repository. If it tells you "No changes found", no need to do anything more at the moment. Otherwise, it will tell you which files will be changed if you pull. Then if <code>runtime/doc/tags</code> is included, you know that fetch may stop until you type <code>d&lt;Enter&gt;</code>
:— If there are many new changesets, and you want to exceptionally use (just this once) vim as a pager, repeat the command like this (all on one line):
::<code>hg -v --config 'extensions.hgext.pager=' --config 'pager.pager=view -' --config 'pager.attend=incoming' incoming</code>
:where the three <code>--config</code> arguments mean that '''this single command''' will be run as if your hgrc included the following (in addition tothe other lines already there):
<pre>[extensions]
hgext.pager=
[pager]
pager=view -
attend=incoming</pre>
:i.e. enable the <code>pager</code> extension, use "'''<code>view -</code>'''" i.e. Vim read-only from stdin, as a pager, and enable the pager for the <code>hg incoming</code> command. — [[User:Tonymec|Tonymec]] 21:42, July 7, 2010 (UTC)
----
I finally found how to force Python programs (including Mercurial, which is written in Python) to always use '''unbuffered''' I/O on stdin, stdout and stderr. Simple: '''set the environment variable <tt>PYTHONUNBUFFERED</tt> to some nonempty string.'''

Example (for the <tt>bash</tt> shell on Unix-like systems):
<pre>
cd ~/.build/hg/vim
PYTHONUNBUFFERED='unbuffered' (date && hg fetch --switch-parent) 2>&1 |tee -a ../hg-vim.log
</pre>

<div style="background-color: #FFFFCC; border: 2px dashed grey; padding: 1ex">'''CAUTION:''' It might be unwise to do this in your permanent environment, since unbuffering means more frequent and smaller transfers, thus reducing throughput. You may want to do it '''only''' when logging via <tt>|tee</tt></div>
— [[User:Tonymec|Tonymec]] ([[User talk:Tonymec|talk]]) 02:56, July 16, 2012 (UTC)

===Useful commands to "get information"===
*<code>hg help</code>
*<code>hg help |less</code>
::what are the basic hg commands, and what do they do?
*<code>hg help <u>command</u></code>
::(where <u>command</u> is a Mercurial command) what are the possible arguments to that command (and what do they mean)?
*<code>hg status</code>
::are there local files waiting for a disposition (<code>commit</code> possibly preceded by <code>add</code> and/or <code>forget</code>)? The reply is one filename per line, preceded by A (added to the list of tracked files), M (merged), D (deleted i.e. specifically not tracked), ? (new file, neither "tracked" nor "not tracked" yet).
*<code>hg heads</code>
::what are all the "tree leaves" ? (commits which are not a parent to a later commit)
*<code>hg log -l <u>N</u></code>
::(where <u>N</u> is a number): show the <u>N</u> latest commits, with relative number and short hash, branch name if any, tags if any, parent(s) if not, or not only, the immediately older commit, and description (commit message)
*<code>hg log -l <u>N</u> -f</code>
::same as above, but only for the current tip and its parents and ancestors.
*<code>hg incoming</code>
::tells you (in the same format as <code>hg log</code>) which changesets (if any) are waiting for you to pull them.
*<code>hg diff -r ''xxxxxxxxxxxx'' [-r ''yyyyyyyyyyyy''] [<u>file</u> ...]</code>
::where ''xxxxxxxxxxxx'' (and ''yyyyyyyyyyyy'' if present) are revision IDs: Produce a diff of the file(s) named, as they changed between the named revisions. With only one revision: compare it to the current tip. With no filenames: give <u>all</u> differences between these revisions.
::'''Notes:'''
:# Contrary to what is said in the manpage, with no revision there is no output.
:# I find it useful to use <span style="background-color: #CFC"><code>|view -</code></span> as a pager for the output of this command. Then a simple <code>:saveas <u>patchname</u>.diff</code> will, if you want to, create the patch file after you review it.
:# To always use <code>view</code> as pager for the more "bulky" output-producing Mercurial commands, add the following to your <code>~/.hgrc</code> file (or, on Windows, your <code>%USERPROFILE%\Mercurial.ini</code>):
<pre>
[extensions]
hgext.pager=
[pager]
pager = view -
attend = annotate, cat, diff, export, glog, log, qdiff
</pre>
— [[User:Tonymec|Tonymec]] 09:43, May 24, 2010 (UTC)

===Changing branches===
Changing branches, or going to a "tagged" revision, are two of the functions of the <code>hg update</code> command (which has <code>up</code> as an alias) — in general, its function is to "update" the files in the current repository (the "working copy" as Mercurial names it) to any revision you specify. So:
====To go to the Vim 7.2 source====
<pre>hg up vim72</pre>

====To go back to the latest sources====
<pre>hg up default</pre>

====To go to a specific patchlevel====
(only for 7.3.142 or later)
<pre>hg up v7-3-155</pre>