Talk: Working with CSV files

==Name of tip==
I'm wondering about the title "Navigate large CSV files more easily". It's not just large CSV files where this is handy, and I generally try to avoid weasel words like "more easily".

What do you think about renaming it to "Working with CSV files"? Any other ideas? --[[User:JohnBeckett|JohnBeckett]] 02:00, 13 July 2008 (UTC)

:I would support changing the name to "Working with CSV Files". I think it captures the features of this tip better. Also it leaves room for future improvements. -- lpr612

::I have renamed it. --[[User:JohnBeckett|JohnBeckett]] 12:02, 19 July 2008 (UTC)

==Featured tip==
Congratulations folks! The tip is magnificent, although if the script gets much longer it will need to be made a plugin on vim.org (I'm resisting the tempatation to make it handle tab delimiters, which wouldn't be too hard now we have GetExpr()).

I'm going to feature this on the [[Main Page]], probably [[Vim_Tips_Wiki_talk:Featured_Tip|in August]]. --[[User:JohnBeckett|JohnBeckett]] 02:00, 13 July 2008 (UTC)

----
Yep, this tip has progressed very well, and it now has a quality that would warrant uploading to vim.org. The big disadvantage of vim.org is that only a single owner is able to upload new versions, and if that owner goes away, nobody can contribute any more. Here on the Wiki, everyone can collaborate on a script, and there doesn't need to be a single owner. How should this be handled? Maybe there could be one vim.org login ''VimTipsUser'', with password and rules and guidelines posted somewhere here in the Wiki, that can then be used by anyone from the Wiki community to upload new versions to vim.org?! (Assuming the vim.org owners are okay with posting such an open login to their site, and the potential for spam and vandalism.)

What do you think? Have other tips from here already made the move to a vim.org script (without a single owner/adopter)? -- [[User:Inkarkat|Inkarkat]] 11:04, 13 July 2008 (UTC)

----
That's an attractive idea. I vaguely recall (but have forgotten the details) that in recent months there was a tip that someone developed into a script on vim.org. In the past, that has happened several times. However, as you point out, the script then suffers bit rot because usually the author moves on to other things and no maintenance occurs.

We have no system for collaborative access to vim.org. At the moment, I'm inclined to leave things the way they are until we get a case where someone really wants to proceed. We could have a public name/password, or we could have a page explaining that there is a well-known account, and if someone wants it they can email [list of 3 or 4 users here]. For email, I would be inclined to NOT publish any email addresses (even if obfuscated). Instead, rely on the Wikia procedure whereby there is an "email this user" link (that requires the sender to have a Wikia account, and the recipient to have supplied an email address; the sender does not see that address unless the recipient replies).

Also, I'm somewhat selfishly wanting to keep this tip here, at least for the moment. In general, I wouldn't want to have that long script on a tip ''and'' a maintained plugin on vim.org. So, if you ''really'' want to make a plugin with yourself as the initial maintainer, let's sort out the details. Otherwise, let's wait until this tip has developed even further.

Another option: It is possible to upload binary files to the wiki. So, we could upload a zip (or text file, or whatever), and provide a link for a download. This procedure uses the "Image" feature of the wiki (whereby it is possible to upload images). I'm not sure what Wikia thinks of this, but I could easily find out (I don't think they would mind), and I have noticed it elsewhere (we haven't done it). --[[User:JohnBeckett|JohnBeckett]] 00:28, 14 July 2008 (UTC)

==Talk pages on this wiki==
Just so everyone knows, we normally try to avoid using talk pages to comment on tips. Instead, if you see something that should be noted in a tip (an improvement, a defect, a question about the tip), please just add to the Comments section at the bottom (use a line consisting of only <tt>----</tt> to give a horizontal rule to separate each comment). We find that generally is easier for editors and readers (if there may be a problem in a tip, a reader should see the comment raising the problem, without needing to examine the talk page as well).

Note also that here we don't keep comments (we just want tips without too many distractions). As soon as a comment is dealt with, it should be deleted (although if a newcomer has a comment, I would generally keep the comment for at least a week, with some sort of reply to indicate why the comment was not applicable). Likewise, unless this discussion keeps going, I'll probably recommend deletion of this talk page in a couple of months (if there's nothing of long-term value remaining on it).

This is the first time that we have had a real reason for holding a discussion on the talk page of a tip, so go wild with any suggestions! --[[User:JohnBeckett|JohnBeckett]] 02:00, 13 July 2008 (UTC)

==Highlighting within status line==
Question: Is it possible to highlight the column header on the status line to be the same color as the highlighted column? It is some small convenience, I think it would be nice to have although what we have now is pretty good already.
:Yes, it's possible (after all). Good (though tiny) point! I've added it to the code; in the status line, the column heading is now highlighted the same way as the current column. -- [[User:Inkarkat|Inkarkat]] 20:32, 20 July 2008 (UTC)
:: I just noticed the feature, while using the code. It makes using the code just a bit more pleasant. Thanks! -- [[User:Lpb612|Lpb612]] 19:02, 23 July 2008 (UTC)

==Sorting by column==
When I get some quality time, I might try and better understand lpb612's amazing regexs. I suspect that some minor tweaks would give a pattern that you could use so <tt>:sort</tt> would sort based on the fields in a column! --[[User:JohnBeckett|JohnBeckett]] 12:02, 19 July 2008 (UTC)

Please see the main page. There is a new section explaining the regex. -- [[User:Lpb612|Lpb612]] 06:06, 20 July 2008 (UTC)

This feature has been added. See the main page. I used a function instead of a simple :sort call, because sort call with the "skip" feature (ie. sort /pattern/) can probably do correct sorting in most cases, by skip the first n-1 columns, but then the text in all remaining columns are ordered, instead of just the n-th column.

A "delete column" feature has also been added.
--[[User:Lpb612|Lpb612]] 20:40, 20 July 2008 (UTC)

'''Suggestion'''<br />
Look at the form of the entries in the column and convert to numbers if that's what they look like (this will improve the sort when sorting tables of data). Something along the lines of:
<pre>
" If the entry looks like a hex string, a number or
" a float, convert to a number representation.
function! s:ChangeToNumberIfApplicable(entry)
  if match(a:entry, '^0x\x\+$') != -1
    let result = str2nr(a:entry, 16)
  elseif match(a:entry, '^[+-]\?\d\+\.\d\+$') != -1
    let result = str2float(a:entry)
  elseif match(a:entry, '^[+-]\?\d\+$') != -1
    let result = str2nr(a:entry)
  else
    let result = a:entry
  endif
  return result
endfunc

" Compare two lines based on the highlighted column.
function! s:CompareLines(line1, line2)
  let col1 = matchstr(a:line1, s:GetExpr(b:csv_column_sort))
  let col2 = matchstr(a:line2, s:GetExpr(b:csv_column_sort))
  let comp1 = s:ChangeToNumberIfApplicable(col1)
  let comp2 = s:ChangeToNumberIfApplicable(col2)
  return comp1 > comp2
endfunction
</pre>

I'm sure there's a better way to do this (and it probably has undefined results if columns have multiple types present), but it's a start. --[[User:Abudden|Al]] 13:30, 1 August 2008 (UTC)
:This is nice, but more than "<tt>return comp1 > comp2</tt>" is needed. Also, the automatic type detection makes me feel a little nervous. It might be better if it were done once, before the start. Anyway, I have preferred another approach in an update I've put in the tip.
:Please see [[#Sorting floating point numbers|below]] for an explanation. --[[User:JohnBeckett|JohnBeckett]] 00:45, 10 October 2008 (UTC)

==Search within column (original)==
I have added a new feature: search within column. See the introduction before the script.

[[User:Lpb612|Lpb612]] 03:19, 18 July 2008 (UTC)

:More magnificent magic! --[[User:JohnBeckett|JohnBeckett]] 09:09, 19 July 2008 (UTC)

----
Hi lpb612: Please check following. '''[Update: These comments are obsolete -- see next section.]'''

I notice that <tt>:SC 2=john</tt> works very nicely, but it doesn't highlight the hits (I use <tt>:set hlsearch</tt>). The following line used instead of the current line seems to fix this issue. I used 'normal' (not 'normal!') so that the mapping for 'n' is used.
<pre>
command! -nargs=1 SC execute <SID>SearchCol("<args>") | normal n
</pre>

--[[User:JohnBeckett|JohnBeckett]] 12:25, 19 July 2008 (UTC)

----
I've corrected the scope of the command to <tt>command! -buffer</tt>. -- [[User:Inkarkat|Inkarkat]] 20:34, 20 July 2008 (UTC)

==Search within column (new)==
I have implemented a new version of csv.vim:
*Expands <tt>undo_ftplugin</tt> so more changes are undone, including removal of the autocommands. That means <tt>set ft=</tt> really does turn off highlighting.
*Uses 'normal n' in the <tt>SC</tt> command so the first search highlights hits (if <tt>'hlsearch'</tt> is set).
*Uses <tt>matchadd()</tt> rather than <tt>:match</tt> so a priority can be assigned to the highlight. That allows search hits to be highlighted in the current column (and leaves the <tt>:match</tt> command available for the user).

My code needs checking (there are a couple of things I'm not sure about)! Please fix any issues you find, or if it's ok let me know here.

[<b>RESPONSE</b>] I checked the new version with matchadd. It works where matchadd is provided (new version of vim?). But it seems that not all VIM 7.1 has matchadd. Another solution, as you (JohnBeckett) mentioned is the following: use "2match Keyword" instead of "match Keyword" in the s:Highlight function, and add <pre>execute 'match Search /\c' . @/ . '/'</pre> at the end of SearchCol function. This way, the SC search has a higher priority than "Keyword" match. Thanks JohnBeckett for both pointing out the problem, and also providing (two!) solutions. Update: Now I have added both methods to the code so that if matchadd is missing there is no warning message, and still highlighting search is correctly done--- [[User:Lpb612|Lpb612]] 06:13, 20 July 2008 (UTC)

I guess everyone knows that you can use the 'history' tab on the wiki to see old versions. If you click the date for a particular version, that version if opened. If wanted, you can edit the old version and copy out all the wikitext to a local file. That's a good way to compare two versions, although the wiki difference tool on the history page is surprisingly good and is often all that you need. Be careful not to save an old version unless you want to wipe out changes since that time. --[[User:JohnBeckett|JohnBeckett]] 04:21, 20 July 2008 (UTC)

----

[<b>DISCUSSION</b>]I think the <pre>nnoremap <silent> <buffer> n n</pre> should not be removed. Otherwise, there will be a "Press Enter" message for a long regular expression. Try a CSV file with many columns (e.g, 5 or 6) and try to search in the last few columns, you should be able to notice the problem. The trick is the <silent> part, which suppresses the message. -- [[Special:Contributions/12.202.152.182|12.202.152.182]] 00:36, 21 July 2008 (UTC)

:I agree with lpb612, and have restored the mappings for n and N. BTW, if you use a secure computer (which only you access), you can select the "remember me on this computer" option when you log on. I think you then only need to log on once a month, and it won't log you off after a timeout.
:Thanks Inkarkat for fixing undo_ftplugin and more. --[[User:JohnBeckett|JohnBeckett]] 06:07, 21 July 2008 (UTC)

:Sorry, I cannot reproduce the "Press Enter" message for a long regular expression. The VIM 7.1 default 'n' command seems to automagically shorten a long regexp in order to avoid this. Do you have any plugin / custom mapping for 'n' which could interfere here? (Try with <tt>gvim -N -u NONE; filetype plugin on; syntax on; e test.csv; SC ...</tt>) -- [[User:Inkarkat|Inkarkat]] 09:24, 21 July 2008 (UTC)

::OK, I think I found where the problem was. I had "set shm=at" which is supposed to make message shorter. But that has the strange side effect of the Press-Enter message. [[User:Lpb612|Lpb612]] 18:10, 21 July 2008 (UTC)

==Code queries==
Breathtaking work lpb612 and Inkarkat! I couldn't resist tweaking the code a bit, but I haven't changed what it does. Some questions on the "05:30, 21 July 2008" version follow. --[[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)

:By the way, the wiki way of responding is like this (start your response with a ':' to indent your reply).
:If you want a line break (new para), also start the new line with ':'. Add your signature <tt><nowiki>~~~~</nowiki></tt> on the same line (not on a new line). [[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)

::And here is a reply to the response (this one uses '::' for more indent). [[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)
::: Thanks for the help. I am learning to use wiki. [[User:Lpb612|Lpb612]] 17:17, 21 July 2008 (UTC)
----
Function SearchCol() finishes with:
 execute 'match Search /\c' . @/ . '/'

I don't understand the point of this. If the user has 'hlsearch' then the match is redundant? If the user has 'nohlsearch' then they do not want all hits highlighted? Also, if you do a search, then use ':noh' or ':set ft=' to switch highlighting off, the :match is still active (i.e. highlighting is ''not'' switched off). Note also that a search like <tt>:SC a/b</tt> (embedded slash) gives an error with the match. So, I think this line should be deleted. --[[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)
:Even if hlsearch is on, the 2match still "masks" the search. So the priority seems to be like this (low to high): hlsearch,...,2match,match. [[User:Lpb612|Lpb612]] 17:17, 21 July 2008 (UTC)
:Maybe we can check for hls before executing match Search? [[User:Lpb612|Lpb612]] 17:52, 21 July 2008 (UTC)
::A test suggests that whichever of 'match' and '2match' is used last wins. On Vim 7.1.40 or later, enter ':echo getmatches()' to see the matches. They are listed in the order that the commands were entered, and it appears that the last in the list is applied last, and therefore "wins".
:::To expand on this, the :match family acts just like matchadd() with the default priority of 10. Search highlighting with hlsearch has priority 0. Highest priority wins, though as John has discovered, last defined will win within the same priority. --[[User:Fritzophrenic|Fritzophrenic]] 12:41, 22 July 2008 (UTC)
::::'''Update''' I've spoken with Bram about this (because <tt>:help :3match</tt> contradicts the above). What I and Fritzophrenic describe above is a bug introduced in Vim 7.1.40 (I've suggested a patch to fix it). What is supposed to happen (as in help) is that <tt>:match</tt> always beats <tt>:2match</tt>, which always beats <tt>:3match</tt>. Prior to 7.1.40, that's what happened. --[[User:JohnBeckett|JohnBeckett]] 11:43, 24 July 2008 (UTC)

::Are you saying that the purpose of the 'match Search...' is for people who have a version older than 7.1.40? If so, perhaps some further ugly 'if &hlsearch && !exists("*matchadd")' code could be used if you really need this statement for older Vim. In general, I would advise against adding complexity to add a feature for old versions. However, a lot of people will have versions older than 7.1.40 so I guess some complexity would be justified, but only if the feature is really needed.
:::I am also against making code ugly unless necessary. Removing the line would be fine with me. For "older vim" users, they will just not be able to see the highlighted search when the search is in the current column (not big deal). [[User:Lpb612|Lpb612]] 04:35, 23 July 2008 (UTC)

::I have commented out the line pending your thoughts because the command gives problems for my newer Vim: I don't need it, and I can't easily remove its highlighting (I map Space to :noh to remove search highlighting; pressing Space does not remove the 'match Search...'). --[[User:JohnBeckett|JohnBeckett]] 10:48, 22 July 2008 (UTC)

:::Following is the commented-out line (was last line in SearchCol()):
:::<tt>"-- execute 'match Search /\c' . @/ . '/'</tt>
:::--[[User:JohnBeckett|JohnBeckett]] 06:01, 29 July 2008 (UTC)

----
Function SortCol() has line:
  execute setline(2, sort(getline(2, '$'), function("s:CompareLines")))

Should replace "execute" with "call"? --[[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)
:I don't know the difference between execute and call (this is my first vim script). So I just tried the first one that worked. Please feel free to change it. In fact, after reading the help manual, I still don't quite get it. It seems that execute is like the eval in bash? [[User:Lpb612|Lpb612]] 17:17, 21 July 2008 (UTC)
::Use 'call' to invoke a function (it's a syntax limitation that 'call' is required). Use 'execute' to execute an expression. --[[User:JohnBeckett|JohnBeckett]] 10:48, 22 July 2008 (UTC)
----
Sometimes the code displays a message like this:
 echohl WarningMsg
 echo "column number out of range."
 echohl NONE
and other times like this:
 echoerr "column number out of range"

Is there a reason for the difference? We could almost justify having a function to display a warning message. --[[User:JohnBeckett|JohnBeckett]] 08:19, 21 July 2008 (UTC)

:There should be no difference. echoerr should be changed. I will change it. [[User:Lpb612|Lpb612]] 17:17, 21 July 2008 (UTC)

====Version "09:12, 22 July 2008"====
In this new version of csv.vim, I have:
*Removed the n and N mappings (Inkarkat was correct: they are not needed; I had jumped to some conclusions).
:I think they are needed if the use happens to have set shortmess variable (at least if set shm=at) -- [[User:Lpb612|Lpb612]] 20:43, 22 July 2008 (UTC)
::If shortmess is set like that, the user doesn't want truncation of such messages, and will deal with the "Hit ENTER" (e.g. by reducing its likelihood via a larger 'cmdheight'). csv.vim should be consistent with the default behavior of the '/' and 'n' commands. -- [[User:Inkarkat|Inkarkat]] 21:21, 22 July 2008 (UTC)
*Commented out "<tt>execute 'match Search ...'</tt>" (see my comments on SearchCol() above).
*Tweaked new Warn() for consistency.
*Removed trailing space from message "No comma-separated columns were detected. ". If it is wanted, consider having the new Warn() function add it to each message (however, Vim does not add spaces around warnings).
--[[User:JohnBeckett|JohnBeckett]] 10:48, 22 July 2008 (UTC)

====Version "03:09, 24 July 2008"====
In DeleteCol() I replaced '<tt>let b:csv_column=b:csv_max_col</tt>' with '<tt>call s:HighlightPrevCol()</tt>'.

Previously, using <tt>:DC</tt> on the last column left no column highlighted, yet another <tt>:DC</tt> would delete another column.

Problem (not critical): The nice PrintColInfo message is not appearing (instead, get result of the global substitute command). I inserted redraw + redrawstatus and one or two other things, but could not make the PrintColInfo message appear.

Another issue is that if you <tt>:DC</tt> then <tt>u</tt> to undo, csv_max_col is too small. Need a way to rescan (without too much complexity!). You can rescan by saving the buffer and using <tt>:e</tt>.
:How about put a 'hook' (line) in Highlight that <pre>call GetNumCols()</pre> each time a highlight is changed? -- [[User:Lpb612|Lpb612]] 17:04, 24 July 2008 (UTC)

I used '<tt>echohl Type</tt>' (not '<tt>echohl Keyword</tt>') in PrintColInfo to match the attractive highlighting of the header row.

In a week, I will be featuring this tip on the [[Main Page]]. Therefore I think I should move the "Wish List" to the talk page (which I intend to keep, although I might delete some obsolete comments). --[[User:JohnBeckett|JohnBeckett]] 03:17, 24 July 2008 (UTC)

==Automatic Downloading the Code==
As the code gets longer, it is inconvenient to cut and paste code from wiki to the hard disk. The following shell script can be used to download and save the code on the main page. It depends on wget, lynx, and awk.
<pre>
#!/bin/bash

# parameters
output="$HOME/.vim/ftplugin/csv.vim"
title="Working_with_CSV_files"

#Thanks to JB for the "raw" link tip.
link="http://vim.wikia.com/index.php?action=raw&title=$title"

# margins for the code
begin="Filetype plugin for editing CSV files"
end="^augroup END"

# back up old file
mv $output ${output}_old

# get the new file
wget -O - -q $link | awk "/$begin/,/$end/{print;next}" > $output
</pre>
I save it in a file called sync667. So every time after I run sync667, I get a fresh copy of the code from wiki to my .vim/ftplugin/ dir.
--[[User:Lpb612|Lpb612]] 16:46, 24 July 2008 (UTC)

----
Hi lpb612, Nice, but you're using the old title! The tip has been renamed (the page with the old title is now a redirect). Also, you can download the wikitext rather than muck around with the html:
<pre>
wget -O 667.wik "http://vim.wikia.com/index.php?action=raw&title=Working_with_CSV_files"
</pre>
--[[User:JohnBeckett|JohnBeckett]] 04:18, 25 July 2008 (UTC)
:Thanks for the tip. There is just no end to improvements! -- [[Special:Contributions/12.202.152.182|12.202.152.182]] 05:06, 25 July 2008 (UTC)
::Hi lpb -- matter of fact, I'm thinking of uploading a tweaked version of csv.vim very soon. It's not a substantive change, just looks more elegant to me.
::Suggestion: Use the '''Show preview''' button! Then you will notice things like the fact that wikia didn't think you were logged on. I think I mentioned that you can choose an option to be remembered when you log on (you can visit wikia next day, and still be logged on even after shutting down). --[[User:JohnBeckett|JohnBeckett]] 11:18, 25 July 2008 (UTC)

==Wish List / Ideas==
The header line (line 1) can be differently highlighted. Does anybody know how this can be done? I cannot think of a way of doing it. --[[User:Lpb612|Lpb612]] 19:05, 23 July 2008 (UTC)
:How about this? (The main trick is the <tt>\%1l</tt> regexp item, which matches line 1; you know the rest of the regexp :-) Strictly, syntax highlighting shouldn't be done in the ftplugin, but in a separate syntax/csv.vim, though. -- [[User:Inkarkat|Inkarkat]] 20:37, 23 July 2008 (UTC)
<pre>
syntax match csvHeading /\%1l\%(\%("\zs\%([^"]\|""\)*\ze"\)\|\%(\zs[^,"]*\ze\)\)/
highlight def link csvHeading Type
</pre>
::That is neat. I have added a new section (see above). -- [[User:Lpb612|Lpb612]] 22:19, 23 July 2008 (UTC)

A future unified interface could be the following:
<pre>

The regular expressions needed to define the syntax. All dots are literal.
Columns are numbered from 1. zero means before the first.

col=0|^|\d|$|.   # 0-th, 0-th, \d-th, last, current
cols=((col|col-col),)*(col|col-col) # a list of column, or column ranges
rows=cols

flags=[c|C][a|d]  # flag used in sorting
sortcol=column[flags]*
sortcols=(sortcol.)*sortcol

Future commands can be as follows. All commands are two-letter, starting
with c or r.  The space after the two letter command should be optional.

:cd cols?
  # delete columns cols or delete the current column if empty
  # change directory if argument are not columns

:c/ (cols=)?pattern/?
  # search forward in columns cols (use current if empty), for pattern

:c? (cols=)?pattern/?
  # search backward in columns cols (use current if empty), for pattern

:cx cols
  # rearrange (eXchange) the columns specified, so that the order after
  # rearrangement is as specified. Example: after
  # :cx 6,1,2-4
  # columns 1-6 should become columns 6,1,5,2-4

:cm cols col?
  # move the columns cols to col (use current if empty)
  # equivalent to :cx col cols

:ca col?
  # append an empty column at col (use current if empty)

:ci col?
  # insert an empty column at col (use current if empty)

:cy cols?
  # yank (copy) the columns cols (use current if empty)
  # Note: in implementation, only remember the cols for future paste

:cp col?
  # paste (append) the previous yanked columns to col (use current if empty)
  # clear the yanked list

:cP col?
  # Paste (insert) the previous yanked columns to col (use current if empty)
  # clear the yanked list

:cs sortcols?
  # sort rows by the columns specified, in that order. c|C denotes
  # case sensitivity, a|d denotes direction (ascending/decending)
  # sort current column by default
  # Note: use string concatenation for implementation
  # Note: should we use :rs instead?

:rh rows?
  # highlight the given rows (use current if emtpy)

:rH
  # turn of row highlighting
</pre>

==Sorting floating point numbers==
The original script called <tt>sort()</tt> with a compare function. However, that function tested only for <tt>></tt>. A sort compare function needs to return negative/zero/positive for <tt>< == ></tt> (the docs actually specify -1/0/1).

I decided to use the built-in <tt>:sort</tt> command instead as it supports more options and has much better performance. However, <tt>:sort n ...</tt> sorts integers only, and regards any field that does not start with an integer as zero. Therefore, the script supports an <tt>f</tt> option that calls <tt>sort()</tt> using a compare function that handles floats and integers.

The sort code calls <tt>sort()</tt> if the <tt>f</tt> option is used (to sort floats), and uses the <tt>:sort</tt> command otherwise (for better performance and more options). The sort code works on a range of lines (default is whole file), and accepts <tt>!</tt> to sort in descending order.

<tt>str2float()</tt> converts string fields to floating point:
*It correctly converts numbers like 12, -1.2, 1e2, .5, 0.0.
*It ignores trailing junk ("1.5abc" is 1.5).
*It returns 0 for an invalid number ("abc1.5" is 0).

Rather than use a regex to restrict what a valid float should look like, the code calls <tt>str2float()</tt>. If the result is 0, a simple check is done to see if the field really is zero. If not, the field must be invalid, and the field is returned as a string. The <tt>CompareLines()</tt> function compares floats with floats and strings with strings. When comparing a float with a string, the string always comes first.

When comparing strings, if <tt>:set ic?</tt> shows that <tt>'ignorecase'</tt> is set, then a string like "abc" will be regarded as equal to "ABC", and "abc" will sort before "XYZ".

Please add any comments or problem reports below. --[[User:JohnBeckett|JohnBeckett]] 11:45, 11 October 2008 (UTC)
----

==Delimiter==
I have enhanced the script with a command to specify the delimiter, and a global variable to set the default delimiter (see the "Features" documentation above script).

The delimiter has to be a single character because many of the regular expressions include things like '<tt>[^,]</tt>' (search for ''not'' the delimiter). However, you can use a multibyte character as the delimiter. For example, assuming UTF-8 or other suitable encoding, you can press Ctrl-k and type <tt>.M</tt> in the command line (<tt>:Delimiter ·</tt>) to specify a middle dot as the delimiter. The new <tt>CharLen()</tt> function uses a trick from Vim help to determine the number of characters in a multibyte string.

It is possible to use '<tt>/</tt>' as the delimiter (<tt>:Delimiter /</tt>). Therefore, I have called <tt>escape()</tt> a couple of times to put a backslash in front of any slash (there will only be a slash if the delimiter is a slash). That is necessary to make commands like <tt>:s/xxx/yyy/</tt> work with an embedded slash.

Please add any comments or problem reports below. --[[User:JohnBeckett|JohnBeckett]] 02:58, 14 October 2008 (UTC)
----

==Getting rid of control characters==
I know a similar topic is posted elsewhere on the wiki, but I wanted to tell anyone looking at the CSV script and running into errors with MacVim or Vim that this [http://www.devdaily.com/blog/post/linux-unix/shell-script-error-bad-interpreter-no-such-file/ DevDaily blog post] solved my issues when copy and pasting the sync script above.

Following the procedure after copying and pasting the CSV script code works as well to alleviate any errors on my machine. YMMV. --[[Special:Contributions/129.174.97.34|129.174.97.34]] 17:58, September 17, 2009 (UTC) Jeffrey Horn
:Thanks for info. I had a very quick look and it looks like it's just saying (badly) to do <tt>:%s/\r//g</tt> to remove extraneous ^M characters. Surely that would only be needed if you copied a script on (say) a Windows system, then tried to use the script on a different platform? Is there anything about this particular tip which requires this info? [[User:JohnBeckett|JohnBeckett]] 21:23, September 17, 2009 (UTC)

==Multi-line fields==
Note that the script does not work with multi-lines fields.

==Update February 2010==
Following two suggestions by [[User:Sathishvj|Sathishvj]], I have updated the script:
*Change <tt>GetStr()</tt> so it works with delimiters that have special meanings (like '<tt>~</tt>' when using the pattern for searching, and '<tt>&</tt>' when using substitute() to replace '<tt>,</tt>' with '<tt>&</tt>').
*Add a <tt>CopyColumn()</tt> function with <tt>:CC</tt> user command to copy a column.
I also did some minor refactoring. Given that the user can enter their own search pattern to search a column, I found it too tricky to use <tt>\V</tt> to turn each regex into "very nomagic", so some fairly heavy kludges were required to work with the special characters.

More testing is needed. Please add any comments here, or in the "Comments" section at the bottom of the tip. [[User:JohnBeckett|JohnBeckett]] 03:20, February 21, 2010 (UTC)

== Only first column highlighted ==

I am relatively new to VIM scripting, please apologize if I made a blatant mistake...

I have followed the installation instructions closely and have configured the csv.vim on both VIm 7.2 for Windows (gVIm) and for Mac OS. In both cases I have downloaded the most recent executable available from vim.org, just to make sure I hadn't miss something important.

In both cases I got consistent behaviour: opening a comma delimited csv file with header the first column will be correctly highlighted, many commands like 0, $ behave as expected, but cursor movements (h, j, k, l) do NOT move to columns. They continue to behave as in "regular" VIm.

I need to mention that VIm in both cases has recognized the filetype as csv.

Any hint?

Thank you for this incredible piece of code, anyway.

--[[User:LucaM|LucaM]] 13:03, February 24, 2010 (UTC)

: Just wanted to check, are you using Shift+h?  And similary Shift+j, shift+k, shift+l, etc. and not h, j, k, l?

: --[[User:sathishvj|sathishvj]]

:: Thank you! That was the ISSUE!!
::
:: --[[User:LucaM|LucaM]]
@@ Line 113: / Line 113: @@
 I guess everyone knows that you can use the 'history' tab on the wiki to see old versions. If you click the date for a particular version, that version if opened. If wanted, you can edit the old version and copy out all the wikitext to a local file. That's a good way to compare two versions, although the wiki difference tool on the history page is surprisingly good and is often all that you need. Be careful not to save an old version unless you want to wipe out changes since that time. --[[User:JohnBeckett|JohnBeckett]] 04:21, 20 July 2008 (UTC)
-[<b>RESPONSE</b>] The <tt>b:undo_ftplugin</tt> didn't execute anything after the <tt>:call matchdelete()</tt> on plain VIM 7.0, because (I think) the | binds stronger than the <tt>:sil!</tt>, and so if <tt>matchdelete()</tt> fails, the following commands aren't executed, and the mappings aren't undone. I fixed that, and also added the <tt>2match none</tt> undo for plain VIM 7.0. I also removed the <tt>nnoremap <buffer>n n</tt>, which is a no-op, isn't it?! At least, highlighting also works for me without this. -- [[User:Inkarkat|Inkarkat]] 20:29, 20 July 2008 (UTC)
 ----