Wikia

Vim Tips Wiki

JohnBeckett/Help template and script

Talk0
1,610pages on
this wiki

< User:JohnBeckett

IntroductionEdit

The template guidelines list some of the templates used in the Vim Tips wiki, including the help template.

Our fearless leader bastl imported the tips from the vim.org tips site in June 2007, and wrote the original Help template and script.

Template:Help creates wikitext that links to a CGI script. That script generates the URL to redirect the user's browser to the correct page at the online Vim documentation site.

The script runs on the Sourceforge server for the vimplugin project maintained by bastl – http://vimplugin.sourceforge.net/

The online documentation is on the Sourceforge server for the vimdoc project maintained by Dan Sharp – http://vimdoc.sourceforge.net/

It turns out that a lot of tricks are required to make Template:Help create valid wikitext, and to have the CGI script process Vim help topics in a manner similar to how Vim looks up a help item.

Example

If you want to comment or ask a question, please edit my talk page and add a new section.

Required filesEdit

Two Python scripts were created (see below):

  • help.py (renamed to help on the Sourceforge server)
  • maketagsidx.py

Get the Vim tags file from the vim70/doc directory (must be from the Vim version that was used to create the documentation on vimdoc.sourceforge.net).

The maketagsidx.py script reads file tags and creates file tagsidx. File tagsidx is a text file containing the Vim help tags, rearranged for fast lookup.

ProcedureEdit

At a workstation with Python installed:

  • Make a temp directory.
  • Copy the vim70/doc/tags file into the directory.
  • Copy maketagsidx.py into the directory (on Linux, you also need to make it executable).
  • Run maketagsidx.py. It reads file tags and creates file tagsidx.
  • Copy files tagsidx and help.py to the cgi-bin directory on the vimplugin Sourceforge server.
  • Use a shell in the cgi-bin directory on the Sourceforge server to execute:
mv help.py help
chmod 755 help
chmod 644 tagsidx

Files tags and maketagsidx.py are not needed again.

Script help.pyEdit

#!/usr/bin/python
'''
 Handle CGI query to redirect client browser to wanted Vim Help topic.
 John Beckett 2007/06/30

 Releases:
 2007/07/09 jb  First release (from code by bastl).
 2007/07/16 jb  Replace '*' with 'star'.
 2007/07/18 jb  Replace '"' with 'quote' and '|' with 'bar'.

 Example:
 To find online docs for Vim ":help word", client browser issues GET:
    http://vimplugin.sourceforge.net/cgi-bin/help?tag=word
 That invokes this server-side script to lookup "word".
 Output from script causes a redirect telling the client to go to:
    http://vimdoc.sourceforge.net/htmldoc/motion.html#word
'''

import sys
import cgi

BASEURL = "http://vimdoc.sourceforge.net/htmldoc"
TFILE = "tagsidx"
# Following must agree with the script that made TFILE.
# See that script for documentation.
NRHDR = 100
OFFSET_BODY = 2800

RESPONSE = """\
Content-type: text/html

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
 <title>%s</title>
</head><body>
 <h1>%s</h1>
 <p><big>%s</big></p>
</body></html>
"""

# Following is:
#  0 for normal operation (CGI script doing redirects)
#  1 for test (CGI script with html output; no redirect)
#  2 for command-line test (not CGI; text output)
testmode = 0

def Show(title, msg):
    if testmode == 2:
        print "%s - %s" % (title, msg)
    else:
        print RESPONSE % (title, title, msg)

def Quit(msg):
    Show("Error", msg)
    sys.exit(1)

def Redirect(url):
    ''' Tell server to issue a Client Redirect Response (RFC 3875). '''
    # Note 1: We don't use cgi.escape(url) because it breaks some tags for
    # some browsers. It's not clear whether some of the weird punctuation
    # in Vim's help topics will cause trouble when used here.
    if testmode == 0:
        print "Location: %s\n\n" % url  # note 1
    elif testmode == 1:
        Show("Response", "Location: %s" % cgi.escape(url))
    else:
        print "Location: %s" % url

def GetOffset(fin, taglwr):
    ''' Return file offset at or before where tag should be.
        This iterates over the header of the input file.
    '''
    last = OFFSET_BODY
    nrlines = 0
    for line in fin:
        key,offset = line.split()
        if taglwr < key:
            break
        last = int(offset,16)
        nrlines += 1
        if nrlines >= NRHDR:
            break
    return last

def GetMatchingTags(fin, tag):
    ''' Return [key1,file1,key2,file2...] matching tag, or None.
        This iterates over a small section of the body of the input file.
    '''
    taglwr = tag.lower()
    fin.seek(GetOffset(fin, taglwr))
    prefix = taglwr + '\t'  # 'x' > '\t' where x is any printable char
    for line in fin:
        test = cmp(prefix, line[:len(prefix)].lower())
        if test == 0:
            return line.split()
        if test < 0:
            break
    return None

def GetTagAndFile(tag):
    ''' Return [ExactTag,HelpFile] for best match to tag, or None. '''
    try:
        fin = open(TFILE)
    except:
        Quit("Could not open tags file.")
    try:
        kf = GetMatchingTags(fin, tag)
        ok = True
    except:
        ok = False
    fin.close()
    if not ok:
        Quit("Error while reading or seeking in tags file (corrupt?).")
    if kf is not None:
        for i in range(0,len(kf),2):
            if tag == kf[i]:
                return kf[i:i+2]
        return kf[:2]
    return None

def Process(tag):
    ''' Lookup tag and output redirect or error message.
        Can call this from another script to test results.
    '''
    kf = GetTagAndFile(tag)
    if kf is None:
        Quit("Tag <tt>%s</tt> not found in Vim Help." % cgi.escape(tag))
    elif len(kf) == 2:
        Redirect("%s/%s.html#%s" % (BASEURL, kf[1], kf[0]))
    else:
        Quit("Row in tags file has bad number of fields.")

def ProcessEncoded(wikitag):
    ''' Process a tag which was wiki urlencoded by the wiki server,
        then decoded like unquote_plus() here (done by cgi.parse()).
        The wiki converts '<' to '<' and '>' to '>' before urlencode.
        Vim help tags never have ' ', so wikitag should not have any spaces.
        If this script is used with a manually-entered URL, any '+' will
        have been changed to ' '. We change them back (so '+' is '+', not ' ').
        We also do these conversions that Vim does for all valid help tags:
           '*' --> 'star', '"' --> 'quote', '|' --> 'bar'
    '''
    t = wikitag.replace('<', '<').replace('>', '>').replace(' ', '+')
    t = t.replace('*', 'star').replace('"', 'quote').replace('|', 'bar')
    Process(t)

def main():
    tag = cgi.parse().get('tag')
    if tag:
        ProcessEncoded(tag[0])
    else:
        Quit("Need 'tag=WordToLookup' in the query URL.")

if __name__ == "__main__":
    try:
        main()
    except SystemExit, n:
        sys.exit(n)
    except:
        t,v = sys.exc_info()[:2]
        t = cgi.escape(str(t))
        v = cgi.escape(str(v))
        Quit("Exception %s: %s" % (t,v))

Script maketagsidx.pyEdit

#!/usr/bin/python
'''
 Make text file 'tagsidx' containing indexed Vim Help tags.
 John Beckett 2007/06/30

 Releases:
 2007/07/09 jb  First release.
 2007/07/16 jb  Replace 'index' with 'vimindex'.

 A CGI script uses our output file to translate a query for a help tag
 to a redirection to the correct Vim documentation page and anchor.
 Input is the file 'tags' copied from the Vim doc directory.

 Output is a text file with '\n' line terminators.
 The file consists of a fixed-size header followed by a body.
 Header is the first NRHDR lines.
 Padding may follow the header (ignored).
 Body starts at OFFSET_BODY in the file.

 Each header line is of form:
    keylwr~hexoffset#
 where
    keylwr    = tag (Vim Help keyword) in lowercase
    ~         = tab ('\t')
    hexoffset = file offset (string of hex digits, no '0x')
    #         = newline ('\n')
 Lines in the header and body are sorted to ascending alphabetical order,
 ignoring case (header lines are all lowercase, while the body lines are
 the exact case, but sorted as if key1 were lowercase).

 Example (two consecutive lines from header):
    word~1600#
    writing~1700#

 This specifies that the line matching 'word' is at offset 0x1600 in
 the file (0 = BOF), and the line for 'writing' is at offset 0x1700.

 In the body, each line is of form:
    key1~file1~key2~file2...#
 where
    file1 is the file for tag key1 (exact case)
    file2 is the file for tag key2 (exact case)
    file3 is the file for tag key3 (exact case)
    etc
 All the keys on one line (key1, key2, ...) are different,
 but are equal when compared ignoring case.
 A line will always have fields key1~file1 (each non-empty).
 key2~file2 and following are only present if required
 (example: keys "ze" and "zE" would be on one line).

 Example:
    word~first~Word~second~WORD~third
 For this artificial example:
 - There is a tag spelt exactly "word" in file 'first'.
 - There is a tag spelt exactly "Word" in file 'second'.
 - There is a tag spelt exactly "WORD" in file 'third'.
 - No other help tags are equal to "word" when ignoring case.
 - The keys could be in any order.
 - If the tag being looked up does not exactly match any key,
   the result is file1 (looking up "WOrd" would give "first").
 - Any file extension (".txt" or ".txt.gz" or ".whatever") is omitted.
'''

import sys

INFILE = "tags"
OUTFILE = "tagsidx"
# Following must agree with the script that uses OUTFILE.
NRHDR = 100  # exactly this many lines in header
OFFSET_BODY = 2800  # offset of first byte in body

def StripFile(file):
    ''' Return bare file name after stripping any '.extension'. '''
    pos = file.find('.')
    if pos < 0:
        return file;
    if pos == 0:
        print "Fatal Error: File name '%s' starts with '.'." % file
        sys.exit();
    return file[:pos]

def FixFileName(file):
    file = StripFile(file)
    if file == 'index':
        file = 'vimindex'   # Vim index.txt is vimindex.html (help.txt is index.html)
    return file

def AddTag(dict, tag, file):
    tlwr = tag.lower()
    if tlwr in dict:
        dict[tlwr].append(tag)
    else:
        dict[tlwr] = [tag]
    dict[tlwr].append(FixFileName(file))

def ReadTags(ifile, tags):
    try:
        fin = open(ifile)
    except:
        print "Error: Could not open tags file '%s'." % ifile
        sys.exit()
    try:
        nrlines = 0
        for line in fin:
            nrlines += 1
            fields = line.split()
            if len(fields) != 3:
                print "Error (ignored): Number fields = %d" % len(fields)
            AddTag(tags, fields[0], fields[1])
    finally:
        fin.close()
    print "Read %d lines from tags file '%s'." % (nrlines, ifile)

def MakeStrings(dict, header, body):
    allhdr = []
    offset = OFFSET_BODY
    for k in sorted(dict.iterkeys()):
        allhdr.append([k, offset])
        s = '\t'.join(dict[k]) + '\n'
        body.append(s)
        offset += len(s)
    step = len(allhdr) / (NRHDR+1)
    sample = step
    for i in range(NRHDR):
        if sample < len(allhdr):
            row = allhdr[sample]
        else:
            row = ["~~~~", offset]
        header.append("%s\t%x\n" % tuple(row))
        sample += step

def WritePadding(fout, off1, off2):
    ''' Pad header to exact size with dummy lines of text. '''
    for blob in (50,20,5,1):
        pad = "~"*(blob-1) + "\n"
        while off1+blob <= off2:
            fout.write(pad)
            off1 += blob

def WriteOutput(ofile, header, body):
    try:
        fout = open(ofile, "wb")
    except:
        print "Error: Could not create output file '%s'." % ofile
        sys.exit()
    try:
        fout.writelines(header)
        WritePadding(fout, fout.tell(), OFFSET_BODY)
        pos = fout.tell()
        if pos != OFFSET_BODY:
            print "Fatal Error: Header is %d bytes (should be %d)."\
                   % (pos,OFFSET_BODY)
            sys.exit();
        fout.writelines(body)
        print "Wrote tags index to file '%s'." % ofile
    finally:
        fout.close()

def main():
    tags = {}
    header = []
    body = []
    ReadTags(INFILE, tags)
    MakeStrings(tags, header, body)
    WriteOutput(OUTFILE, header, body)

if __name__ == "__main__":
    main()

Around Wikia's network

Random Wiki