Changes: Creating your own syntax files

Revision as of 19:12, 1 May 2008

Proposed tip Please edit this page to improve it, or add your comments below (do not use the discussion page).

Please use new tips to discuss whether this page should be a permanent tip, or whether it should be merged to an existing tip.

created April 29, 2008 · complexity intermediate · author Clearmoments · version 7.0

Introduction

Since there isn't a lot of material on creating your own syntax files in Vim, I've decided to write a tutorial on the process. Currently, this tutorial only shows how to make Vim interpret the syntax of files by their file extension. Another tutorial may extend this to show how to interpret the syntax of files by their contents.

Example: Celestia star catalogs

For this tutorial I am creating a syntax file for Celestia star catalogs. Celestia is a great program for anyone who likes astronomy and space, but I digress. All we need to know for this tutorial is that a star catalog lists a star name along with its positional information, distance and attributes (color, radius, mass, brightness). An example entry in a star catalog file (.stc) can be:

600000 "My Star"
{
  RA 24.406489
  Dec -9.404052
  SpectralType "Q"
  Mass 1.09
  AbsMag 1.29
  Distance 124.729260
}

As you can see it consists of a number, a string, and a block ({...}), with some keywords within that block ("RA", "Dec", etc). Comments are marked out by a "#" like in shell scripts or conf files. In fact, the syntax looks a lot like a conf file. There can be multiple entries like this with a number (the HIP number), the string, and the block containing the attributes. Celestia gets more complicated than this because you can have multiple stars going around a barycenter, etc, but we are not going to get fancy. We'll cover only stars.

Syntax files

Get your directory straight

Syntax files are editor scripts, just like everythin else, and are generally located in your personal $HOME/.vim/syntax directory. If that directory does not exist you can create it. There should also be a system-wide syntax file directory. It is preferable to use the personal syntax directory over the system-wide one since it is less likely to be overwritten by a newer Vim installation. A good reason to use the system-wide directory is if more than one user requires use of a syntax file. You can see these in Vim with the command:

:set rtp?

The system-wide and user syntax directories are OS and system dependent.

Unix

/usr/share/vim/vimxx/syntax

Where xx is the vim version. You can verify its location using the previous editor command, set :rtp?.

Windows

%PROGRAMFILES%\vim\vim71\syntax
%PROGRAMFILES%\vim\vim71\cream\syntax

Under Windows you can create a $HOME environment variable manually. From Change the color scheme:

On Windows, the $HOME environment variable can be set by going to your desktop, right click on "My Computer", click the "Advanced" tab, select "Environment Variables". If HOME is not in the list of variables, create a new variable named HOME and point it to the location of your vimrc.

Mac OS

I am going to assume that Mac OS is the same as the Unix. See Unix.

The system-wide syntax directory is going to be one of the entries separated by commas with. The first value I know is your Vim personal syntax directory. You may want to study those syntax files and see how they are put together before playing around with it.

Build a syntax file

First, create a new file in Vim, and add the following contents:

" Vim syntax file
" Language: Celestia Star Catalogs
" Maintainer: Kevin Lauder
" Latest Revision: 26 April 2008

if exists("b:current_syntax")
  finish
endif

Vim comments start with a quote. So I am following the convention of the built-in syntax files, and making a little comment flower box. The test if exists("b:current_syntax") ... checks whether an earlier file has defined a syntax already. If so, the script exits with finish.

Keyword, match & region elements

There are three major syntax elements, and commands to describe those elements. In order to syntax highlight, we must be able to describe what to highlight. Here is an example of what they look like:

" Keywords
syn keyword syntaxElementKeyword keyword1 keyword2 nextgroup=syntaxElement2

" Matches
syn match syntaxElementMatch 'regexp' contains=syntaxElement1 nextgroup=syntaxElement2 skipwhite

" Regions
syn region syntaxElementRegion start='x' end='y'

Keywords

Keywords are simple. Take for example the programming language BASIC. In BASIC there are several keywords like PRINT, OPEN and IF. Let's say you would like the editor to recognize them. You can define them like so:

syn keyword basicLanguageKeywords PRINT OPEN IF

For now we are not going to worry about nextgroup=.

The editor will now recognize the keywords PRINT, OPEN and IF as syntax elements of type basicLanguageKeywords. You can add more on the same line, or add another line with the same type (basicLanguageKeywords). In other words if I wanted to add the keywords DO, WHILE and WEND to my list, I could add to the previous line like this:

syn keyword basicLanguageKeywords PRINT OPEN IF DO WHILE WEND

Or, I could add another line, like this:

syn keyword basicLanguageKeywords PRINT OPEN IF
syn keyword basicLanguageKeywords DO WHILE WEND

Let's use a more relevant example. Take our star catalog entry from above

600000 "My Star"
{
  RA 24.406489
  Dec -9.404052
  SpectralType "Q"
  Mass 1.09
  AbsMag 1.29
  Distance 124.729260
}

We can group the following keywords as part of a syntax element called celstcStarBlockCmd by adding the following to our syntax file.

syn keyword celstcStarBlockCmd RA Dec SpectralType Mass Distance AbsMag

The editor will now recognize them. Maybe that's enough for your purposes, but I wanted to make things a little more interesting with matches following the keywords, like those numbers and string values. How do we get those to be recognized?

Matches (and addendum to keywords)

All this keyword stuff logically leads to matches. Take the above example once again. After the keywords ("RA", "Dec", "AbsMag" etc) there are numbers. Let's say we want Vim to know that following a certain keyword there is going to be some set of characters to follow (defined as a regular expression maybe).

This is where matches come in; along with an additional caveat to using keywords, the nextgroup and skipwhite arguments as seen above.

syn match celstcNumber '\d\+'
syn keyword celstcStarBlockCmd RA Dec Mass Distance AbsMag nextgroup=celstcNumber skipwhite

Now as you can see the match was given a regular expression \d\+ meaning to match one or more (\+) digits 0-9 (\d). The keyword syntax element celstcStarBlockCmd has been modified slightly because following the SpectralType keyword is not a number but a string. Hence it has been excluded from the list of keywords for now. Later in this article, we will address that problem by creating another regular expression to match strings and apply it to that keyword.

Notice the nextgroup argument. We are telling the editor to expect a celstcNumber after the keyword. So that's the first pattern the editor will attempt to match after finding one of those keywords.

The skipwhite argument simply tells the editor to expect some whitespace (tabs, spaces etc) between the keyword and the number.

You may have noticed a slight problem with my implementation. It will only match numbers like 19938, 93, and 0. It won't match decimals 3.91881 or negative numbers (-9).

How do we fix that? With more interesting regular expressions of course! This was taken from one of the existing Vim syntax files and modified slightly to work for our needs.

" Regular int like number with - + or nothing in front
syn match celstcNumber '\d\+'
syn match celstcNumber '[-+]\d\+'

" Floating point number with decimal no E or e (+,-)
syn match celstcNumber '\d\+\.\d*'
syn match celstcNumber '[-+]\d\+\.\d*'

" Floating point like number with E and no decimal point (+,-)
syn match celstcNumber '[-+]\=\d[[:digit:]]*[eE][\-+]\=\d\+'
syn match celstcNumber '\d[[:digit:]]*[eE][\-+]\=\d\+'

" Floating point like number with E and decimal point (+,-)
syn match celstcNumber '[-+]\=\d[[:digit:]]*\.\d*[eE][\-+]\=\d\+'
syn match celstcNumber '\d[[:digit:]]*\.\d*[eE][\-+]\=\d\+'

Notice how we can keep creating more lines of syn match celstcNumber 'a pattern' to match all those patterns as one syntax element type (in this case celstcNumber.)

Regions

But we have another challenge that waits us. Let's look at the star catalog entry again.

600000 "My Star"
{
  RA 24.406489
  Dec -9.404052
  SpectralType "Q"
  Mass 1.09
  AbsMag 1.29
  Distance 124.729260
}

The first number is outside of the brackets, and that's not really a number, that's an HIP catalog entry (more like a star's ID number rather than a value with physical meaning like mass or distance). The real numbers are the arguments to the keywords (like "RA" and "Dec"). It would be nice if we could have the editor match those differently than regular numbers. But, since celstcNumbers and HIPs consist of the digits 0-9 they conflict with one another. How can we fix that discrepancy?

Note that numbers with values exist only within brackets. Outside of the brackets it is an ID number rather than a value. We have to add another argument to the keyword and match definition blocks, and introduce another type of syntax element: a region.

First, we have to let the editor know that the aforementioned keywords only exist within brackets. Second, we specify that celstcNumber syntax elements only exist within brackets. This is the concept of a region.

syn region celstcDescBlock start="{" end="}" fold transparent

There we go, and we threw in some arguments for kicks. The fold argument means that Vim can increase the fold count inside brackets so you can press Ctrl+<F9> to expand and contract the code. The transparent is the important keyword here. It tells the editor to continue to apply matches and keywords to what's inside the region. Otherwise the region would not be colorized properly.

We must add another additional argument to finish off everything.

syn region celstcDescBlock start="{" end="}" fold transparent contains=celstcNumber,celstcStarBlockCmd

The contains argument tells the editor which syntax elements this region will contain. In this case keywords and numbers. But we have strings too, right? So let's implement the required syntax elements since we know all about keywords, matches and regions now. In addition we pickup another argument along the way, contained.

Let's define comments as a syntax element and see how contained works.

syn keyword celstcTodo contained TODO FIXME XXX NOTE
syn match celstcComment "#.*$" contains=celstcTodo

Comments start with a "#" and run until the end of line. So that's a simple regular expression '#.*$'. Starts with a "#" and match all characters until the end of a line.

contained simply tells the editor that the keyword is only valid when contained by another syntax element, in this case a celstcTodo is only treated as a separate syntax element when contained by celstcComment.

So let's redefine everything from before, and implement all the required grammar, and bring this puppy together.

Bringing it together

syn keyword celstcTodo contained TODO FIXME XXX NOTE
syn match celstcComment "#.*$" contains=celstcTodo

"----------------------------------------------------------------
" Celestia Star Catalog Numbers
"----------------------------------------------------------------

" Regular int like number with - + or nothing in front
syn match celstcNumber '\d\+' contained display
syn match celstcNumber '[-+]\d\+' contained display

" Floating point number with decimal no E or e (+,-)
syn match celstcNumber '\d\+\.\d*' contained display
syn match celstcNumber '[-+]\d\+\.\d*' contained display

" Floating point like number with E and no decimal point (+,-)
syn match celstcNumber '[-+]\=\d[[:digit:]]*[eE][\-+]\=\d\+' contained display
syn match celstcNumber '\d[[:digit:]]*[eE][\-+]\=\d\+' contained display

" Floating point like number with E and decimal point (+,-)
syn match celstcNumber '[-+]\=\d[[:digit:]]*\.\d*[eE][\-+]\=\d\+' contained display
syn match celstcNumber '\d[[:digit:]]*\.\d*[eE][\-+]\=\d\+' contained display

syn region celstcString start='"' end='"' contained
syn region celstcDescString start='"' end='"'

syn match celstcHip '\d\{1,6}' nextgroup=celstcString
syn region celstcDescBlock start="{" end="}" fold transparent contains=ALLBUT,celstcHip,celstcDescString

syn keyword celstcStarBlockCmd RA Dec Distance AbsMag nextgroup=celstcNumber
syn keyword celstcStarBlockCmd SpectralType nextgroup=celstcString

Telling Vim how to highlight + final touches

You've reached the easiest part, just take the syntax element names you've used and use the hi def link command to tell Vim how to highlight.

Set the b:current_syntax variable to a name. I called it "celstc". You'll need that name to modify the Un/Commentify (<F6>/Shift+<F6>) script in Cream for example to block comment-out lines with your new file types.

let b:current_syntax = "celstc"

hi def link celstcTodo               Todo
hi def link celstcComment            Comment
hi def link celstcStarBlockCmd       Statement
hi def link celstcHip                Type
hi def link celstcString             Constant
hi def link celstcDescString         PreProc
hi def link celstcNumber             Constant

The hi def link command has different types of highlighting options that we needn't consider. The ones used here are:

Todo: used for the todo comments (ones that have "TODO: blah blah" in them)
Comment: indicates a code comment
Statement: a code statement like a for loop or something
Type: a user defined type generally
PreProc: a pre-processor (like a C #include <stdio.h>)
Constant: like a string or number in code

These of course are guidelines. For our purposes we really don't have statements or pre-processor commands, since this is really more like a conf file or like HTML. As you can see I set the celstcStarBlockCmds to use the Statement highlighting. As for the celstcHip I set that to a Type (seems like a close match to meaning).

You can view more options in Vim help: :help syntax

Save your work

Finally save your file as celstc.vim (that's what I called it), in your Vim syntax directory (preferably your local one $HOME/.vim/syntax).

Setup your filetype.vim file

Okay, now we have to make sure Vim knows how to interpret your file. First if $HOME/.vim/filetype.vim exists then you are going to find the part of the file that looks like this:

augroup filetypedetect
au! BufRead,BufNewFile *.stc setfiletype celstc
augroup END

Comments

TO DO

How about replacing celstcStarBlockCmd with a simpler name?
Check out that Windows $HOME stuff. The "proper" procedure for setting $HOME on Windows involves setting the home folder on the Profile tab of the user's properties in lusrmgr.msc (Windows 2000 or later). We need a tip on this; then just link to it.
Perhaps should remove the embedded tab characters. I'm inclined to just indent the code with our two-space convention (I use four spaces for my work, but two seems to work well on the wiki).
When I get a chance, I'll think about what to recommend for where syntax files should be placed. I suspect that the following statement is a little too dogmatic: "Syntax files ... are located in your personal $HOME/.vim/syntax directory".
I don't think you need the $ in the regex '#.*$'.
On Windows, note that %PROGRAMFILES% includes the drive letter, so X:\%PROGRAMFILES%\... should really be %PROGRAMFILES%\...

Also mention using the $HOME/vimfiles/syntax (or $HOME/.vim/syntax) directory for syntax files that not everyone on the system will need. Less likely to be "blown away" by accident, and it won't need to be copied every version of Vim that comes out.

--Fritzophrenic 13:15, 29 April 2008 (UTC)

Thanks for the great recommendations. I just edited the page, I am very pleased about the reaction; frankly I thought it wasn't going to be as good. Feel free to make all the changes in the world. This is no longer my work, it belongs to the community, though I intend to make further changes. Clearmoments 00:29, 30 April 2008 (UTC)

I am debating adding the EnhancedCommentify section as a separate article or to leave it as part of the article. Please comment.

Clearmoments 02:45, 30 April 2008 (UTC)

Definitely EnhancedCommentify should be in another tip (it just clutters up this tip, and it is really a somewhat different topic). However, perhaps do that later (in a couple of days when the text here is nearly finished). I don't have time to consider the issues at the moment, but I think you are recommending that people edit a system file. Not a good idea (people should update Vim versions from time to time, and that will wipe out any customisations). There generally are other techniques (put something in an after directory, or define some variable mentioned in the plugin documentation), but I'll need to think over what is actually going on in this case before commenting further.

A bit of thought should be given to a suitable title for any new EnhancedCommentify tip. That's another reason for making a suggestion first, waiting for comments, then doing it.

I've added more todo items for someone to consider. There's always more to do! --JohnBeckett 11:15, 30 April 2008 (UTC)

How long will the pastebin links be valid? Perhaps you should upload the files instead, or put them somewhere in your user page. --Fritzophrenic 15:06, 30 April 2008 (UTC)

I agree that the pastebin links should be removed. They are not doing anything bad in this case (apart from the worry that they will disappear and give annoying broken links), however we have to be fairly brutal about discouraging people from posting links because they are a maintenance nightmare, usually aren't very helpful for someone browsing the Vim Tips wiki, and are just too tempting for those who want to promote some favourite web site. Therefore, we want to minimise the number of external links to non-Vim specialty sites so they don't set a precedent.

If you want to display this info, I suggest you put it on a subpage of your user page. To do this, you would go to User:Clearmoments and edit it. Put some wikitext like the following:

Here are some sample syntax files [[Creating_your_own_syntax_files|discussed in my tip]]:
*An actual .stc syntax file [[/celstarcat|celstarcat.vim]]
*An actual .ssc syntax file [[/celssc|celssc.vim ]]

Click "Show preview". Check that it reads how you want it. Do not save yet. In the preview, do a Control-Click (or Shift-Click or whatever works) on the first link. That should open a new tab or window with a page called "celstarcat" under your user page (Clearmoments/celstarcat). Since it doesn't exist yet, you will see an edit window. You can now enter the contents like this:

This is a sample syntax file discussed in [[Creating your own syntax files]].

==celstarcat.vim==
<pre>
    (paste contents of file here)
</pre>

If you have any questions, please ask. Later, we will clean up all these comments. --JohnBeckett 00:04, 1 May 2008 (UTC)

The information you provided was extremely helpful; thank you! I had no idea you could add to your user page in that fashion; had that been the case I would have opted for that (that was precisely the feature I was looking for.) I removed the pastebin links, and used subpages as you suggested.

Shall I remove the links to the Celestia, and Hippacros as well?

Clearmoments 01:12, 1 May 2008 (UTC)

Good – I think the user subpages worked out well (although a little less attractive than the pastebin site ... can't have everything).

The tip has a link for "Celestia" and another for "HIP catalog entry". I think they should be kept. However, they probably are sufficient, so I would be inclined to remove the entire "Further reading" section. It's interesting stuff, but I imagine the other links (and Google) would satisfy anyone who wants more, and we probably should not have precedents for (say) a tip on using Vim to handle yachting information, with links to a bunch of sailing sites.

I just did the following:

Removed "[[Link title]]" from the tip and the todo (if there was some reason for it, let me know).
Removed the "This new tip is not quite complete" todo item (although naturally anyone is welcome to develop the tip further).

Later (when the tip is processed with the other April new tips), I will delete pretty well all these comments.

I still think you should consider the following todo item:

How about replacing celstcStarBlockCmd with a simpler name?

I see that you've made the name shorter, but someone new to syntax files (and with no knowledge of Celestia) will be perplexed by the name. It has too many components for comprehension in a tutorial. Keep the correct name on your user subpage, but I suggest simplifying it here. Perhaps omit the "StarBlock". Or omit "celstc" and "Block", or change "celstc" to "cel". Does the "Cmd" signify anything? I suspect that if you did a global substitute for a simpler name, then pasted it in here, then clicked 'Show preview' you would see a much more readable tip. Also, you would want to shorten a couple of the other names ... I guess it could be too much hassle, so maybe forget it. Anyway, the tip is great as it is. --JohnBeckett 08:38, 1 May 2008 (UTC)