Uniq - Removing duplicate lines

Duplicate tip

This tip is very similar to the following:

• VimTip1292

These tips need to be merged – see the merge guidelines.

Tip 648 Printable Monobook Previous Next

created February 1, 2004 · complexity intermediate · author Michael Geddes · version 6.0

---

In Vim 7.0 and later, the following command will sort all lines and remove duplicates:

:sort u

If you need more control, here are some alternatives.

There are two versions, the first leaves only the last line, the second leaves only the first line.

g/^\(.*\)$\n\1$/d
g/\%(^\1$\n\)\@<=\(.*\)$/d

Breakdown of the second version:

g//d <-- Delete the lines matching the regexp
\@<= <-- If the bit following matches, make sure the bit preceding this symbol directly precedes the match
\(.*\)$ <-- Match the line into subst register 1
\%( ) <--- Group without placing in a subst register.
^\1$\n <--- Match subst register 1 followed by end of line and the new line between the 2 lines

In this simple format (matching the whole line), it's not going to make much difference, but it will start to matter if you want to do stuff like match the first word only

This does a uniq on the first word in the line, and deletes all but the first line:

g/\%(^\1\>.*$\n\)\@<=\(\k\+\).*$/d

See also

• VimTip1148 for a way to apply uniq on elements from a VimL List (and not text lines)

Comments

Here are some more Vim-native ways for removing duplicate lines. This time they don't have to be adjacent. Line order is preserved.

This one can be a bit slow.

:nno \d1 :g/^/m0<CR>:g/^\(.*\)\n\_.*\%(^\1$\)/d<CR>:g/^/m0<CR>

This is faster. Uses mark l.

:nno \d2 :g/^/kl\|if search('^'.escape(getline('.'),'\.*[]^$/').'$','bW')\|'ld<CR>

---