Wikia

Vim Tips Wiki

Changes: Uniq - Removing duplicate lines

Edit

Back to page

m (Reverted edits by 117.205.88.28 (talk | block) to last version by Bohrshaw)
Line 1: Line 1:
Brief/Short Notes (All Subject Areas)
+
{{TipImported
  +
|id=648
  +
|previous=647
  +
|next=649
  +
|created=2004
  +
|complexity=intermediate
  +
|author=Michael Geddes
  +
|version=7.0
  +
|rating=18/9
  +
|category1=
  +
|category2=
  +
}}
  +
The following command will sort all lines and remove duplicates (keeping unique lines):
  +
<pre>
  +
:sort u
  +
</pre>
   
Write short notes of the following:
+
If you need more control, here are some alternatives.
   
1. Press council
+
There are two versions, the first leaves only the last line, the second leaves only the first line. (Use \zs for speed reason.)
2. News features services
+
<pre>
3. PIB
+
g/^\(.*\)$\n\1$/d
4. Nonaligned news pool
+
g/\%(^\1$\n\)\@<=\(.*\)$/d
5. Defamation and libel
+
</pre>
6. North-South Dialogue
+
7. Structural Adjustment Programme
+
Breakdown of the second version:
8. Balance of Payments
+
<pre>
9. Intellectual Property Rights
+
g//d <-- Delete the lines matching the regexp
10. Feedback
+
\@<= <-- If the bit following matches, make sure the bit preceding this symbol directly precedes the match
11. Mass Audience
+
\(.*\)$ <-- Match the line into subst register 1
12. Role of opinion leader in communication
+
\%( ) <--- Group without placing in a subst register.
13. Traditional Media
+
^\1$\n <--- Match subst register 1 followed by end of line and the new line between the 2 lines
14. Rural Newspaper
+
</pre>
15. Brand commission
+
16. G-7 nations
+
In this simple format (matching the whole line), it's not going to make much difference, but it will start to matter if you want to do stuff like match the first word only.
17. Communication Process
+
18. Harare Summit of NAM countries
+
This does a uniq on the first word in the line, and deletes all but the first line:
19. International News Flows
+
<pre>
20. Press Council
+
g/\%(^\1\>.*$\n\)\@<=\(\k\+\).*$/d
21. News Feature Service
+
</pre>
22. Non-aligned News pool
+
23. Libertarian Theory of Press
+
==See also==
24. News pool
+
*[[VimTip1148|Unique sorting]] script to 'sort unique' a List (not text lines)
25. Cable TV
+
*[http://code.google.com/p/lh-vim/source/browse/system-tools/trunk/plugin/system_utils.vim system_utils.vim] command to remove duplicate lines in a range (uses <code>g//d</code> method)
26. Press Commission
+
*[[VimTip1166|Sort lines]] how to sort lines
27. Yellow Journalism
+
28. Wire Services
+
==Comments==
29. Style Book
+
Here are some more Vim-native ways for removing duplicate lines. This time they don't have to be adjacent. Line order is preserved.
30. Reporting Beats
+
31. Language Press
+
This one can be a bit slow. And the pattern would match a single empty line which would also be deleted. The part ":g/^m0<CR>" at beginning and end of the command maybe optional.
32. News Feature Agencies
+
<pre>
33. Developmental News
+
:nno \d1 :g/^/m0<CR>:g/^\(.*\)\n\_.*\%(^\1$\)/d<CR>:g/^/m0<CR>
34. Communication gaps
+
</pre>
35. Media and social change
+
36. Press information Bureau
+
This is faster. Uses mark <code>l</code>.
37. Distortions by Western media sources
+
<pre>
38. News feature agencies
+
:nno \d2 :g/^/kl\|if search('^'.escape(getline('.'),'\.*[]^$/').'$','bW')\|'ld<CR>
39. Law of defamation
+
</pre>
40. Press in a traditional communist society
+
41. News exchange programmes
+
----
42. Concept of development
+
Following uses a substitute to delete all repeated lines (leaving only the first line, while deleting following duplicate lines). This is a variation on the <code>g//d</code> method.
43. Parliamentary reporting
+
<pre>
44. Accreditation of journalists
+
%s/^\(.*\)\(\n\1\)\+$/\1/
45. Prototype production
+
</pre>
46. Satellite broadcasting
+
47. ESP of India
+
----
48. CBS of India
 
49. Feedback Mechanism
 
50. Barriers to communication
 
51. McLuhan’s Understanding Media
 
52. Communication and Culture
 
53. Propaganda
 
54. Noise
 
55. Stages of Economic Growth
 
56. Dominant Paradigm
 
57. Dance’s Helical Model
 
58. Effective Communication
 
59. Barriers of Communication
 
60. Verbal Communication
 
61. Two step flow of Communication
 
62. Effective Communication
 
63. Semantic Noise
 
64. Berger’s Uncertainty Theory
 
65. Hot and Cool Media
 
66. The Influence of culture on communication
 
67. The concept of feedback
 
68. Intro
 
69. Editing symbols
 
70. News values
 
71. First lead story
 
72. News feature
 
73. Features
 
74. An editorial
 
75. Private TV channels
 
76. Piece to camera
 
77. News bulletin
 
78. New Radio channels.
 
79. Basic shots
 
80. P to C & VO
 
81. TV Lighting
 
82. TV Studio Production
 
83. MCD
 
84. Auto strike in the Capital
 
85. Use of Subways in the capital
 
86. Public Transport
 
87. Brand Width
 
88. Information Super highway
 
89. HDTV
 
90. LAN
 
91. Law
 
92. Norms
 
93. Convention
 
94. Ethics
 
95. Defamation
 
96. Intellectual Property Right
 
97. Contempt of Legislature
 
98. Right to Information
 
99. Gulf News
 
100. International Herald Tribune
 
101. Toronto Star
 
102. Intellectual Property Rights
 
103. Press Registration Act
 
104. Prasar Bharti Act
 
105. Press and Human values
 
106. Pressures on the press
 
107. Media Ethics
 
108. Annie Besant
 
109. Sadanand
 
110. Pothen Joseph
 
111. K Rama Rao
 
112. Founder Editor of "UDANT MARTAND"
 
113. Bhartendu Harish Chandra
 
114. Hindustan Times
 
115. Aryavart
 
116. Aajtak
 
117. UNI
 
118. Al-Hilal
 
119. PTI
 
120. Pratap
 
121. Times of India
 
122. How would you cover a scientific institution?
 
123. Write a brief note on Indian Agricultural Research Institute.
 
124. Automobiles are the first sign of affluence. Critically examine it.
 
125. What are the key factors for evaluating a new car?
 
126. Consolidated Fund
 
127. Contingency Fund
 
128. Revenue Budget
 
129. Capital Budget
 
130. GDP
 
131. On-line learning
 
132. History of computers
 
133. Lithographic printing process
 
134. Wireless technology
 
135. Development Communication
 
136. Copyright Law
 
137. Working Journalist Act
 
138. Growth of Electronic Media
 
139. Opinion leaders
 
140. Language press
 
141. International news flow
 
142. Press Council of India
 
143. Registration Act
 
144. Parliamentary Privileges
 
145. Communication support for developing
 
146. Vicious circle of poverty.
 
147. Use of Media for development
 
148. Human Rights and development
 
149. Print Media
 
150. Traditional Media
 
151. Electronic Media
 
152. Public Relations
 
153. Editing
 
154. Radio news
 
155. Press and Human values
 
156. Yellow Journalism
 
157. Pressures on the press
 
158. Defamation
 
159. Media Ethics
 
160. Consolidated Fund
 
161. Contingency Fund
 
162. Revenue Budget
 
163. Capital Budget
 
164. GDP
 
165. Official Secrets Act
 
166. Democracy and press laws
 
167. Right to Information
 
168. Broadcast Bill
 
169. Bengal Gazette
 
170. Samvaad Kaumadi
 
171. Navjeevan
 
172. 1836
 
173. 1872
 
174. 1881
 
175. 8 columns
 
176. 6 columns
 
177. 2 columns
 
178. Kesari
 
179. Samvaad Kaumadi
 
180. Navjeevan
 
181. Print Media
 
182. Traditional Media
 
183. Electronic Media
 
184. Rural Economy
 
185. Weaker Sections of Society
 
186. Environment
 
187. 1889
 
188. 1905
 
189. 1923
 
190. Navbharat Times
 
191. Dainik Jagran
 
192. Rajasthan Patrika
 
193. 1991
 
194. 1995
 
195. 1999
 
196. Press India Bench
 
197. Parliamentary Information Branch
 
198. Press Information Bureau
 
199. www.indiatimes.com
 
200. www.india.com www.india-today.corn
 
201. First page
 
202. Second Page
 
203. Third page
 
204. Public Relations
 
205. Editing
 
206. Radio news
 
207. Portable TV
 
208. Portable Computer Telecommunication System
 
209. India Today Group
 
210. Times of India Group
 
211. Enaudu Group
 
212. PTI
 
213. ANT
 
214. UNI
 
215. Varanasi
 
216. Agra
 
217. Calcutta
 
218. Cultural Society
 
219. Public Relations Society
 
220. Broadcasting Corporation
 
221. Offset
 
222. Online
 
223. More than mobile phones
 
224. More than color TVs
 
225. Less than B/W TVs.
 
226. Defamation
 
227. Media ethics
 
228. News flow
 
229. Pressures on press
 
230. Account Management
 
231. Creative
 
232. Media
 
233. Positioning
 
234. Socio-Economic classification
 
235. Television Rating Points (TRPs)
 
236. Brand Equity Qualitative Research.
 
237. Feedback Mechanism
 
238. Barriers to Communication
 
239. McLuhan's Understanding Media
 
240. Communication and Culture
 
241. Propaganda
 
242. Noise
 
243. Stages of Economic Growth
 
244. Dominant Paradigm
 
245. Dance's Helical Model
 
246. James Silk Buckingham
 
247. Al-Hilal
 
248. Sadanand
 
249. Times of India
 
250. Press Council of India
 
251. Right to Information
 
252. Yellow journalism
 
253. Vernacular Press Act 1878
 
254. Curb the paparazzi
 
255. Make newspapers responsible
 
256. Safeguard the freedom of journalists
 
257. Ensure a free and vibrant press to defend democratic values.
 
258. Comprehensive layout
 
259. Type legibility
 
260. Halftone illustration
 
261. Graphics
 
262. Language press
 
263. Information Super Highway
 
264. Development Communication
 
265. Communication Research
 
266. Positioning
 
267. Socio-Economic classification
 
268. Television Rating Points (TR Ps)
 
269. Brand Equity
 
270. Qualitative Research.
 
271. "Publics" in PR
 
272. PR and Laws
 
273. Internal Communication Media.
 
274. AIDS
 
275. Communal Harmony
 
276. Pulse Polio Campaign
 
277. Violence against women
 
278. Style-book
 
279. Cross-heads
 
280. Five Ws and One H
 
281. Portable TV
 
282. Telecommunication System
 
283. Portable Computer
 
284. India Today Group
 
285. Times of India Group
 
286. Enaudu Group
 
287. PTI
 
288. ANI
 
289. UNI
 
290. Varanasi
 
291. Agra
 
292. Calcutta
 
293. Cultural Society
 
294. Broadcasting Corporation
 
295. Public Relations Society
 
296. Offset
 
297. Online Laser
 
298. More than mobile phones
 
299. Less than B/W TVs.
 
300. More than color TVs
 
301. Any English language daily born during the first half of the 19th Century.
 
302. The first news agency of Mumbai owned by an Indian editor.
 
303. The first Hindi newspaper
 
304. Bhartendu Harish Chandra
 
305. Act
 
306. Ordinance
 
307. Statute
 
308. Constitution
 
309. Copy Right Act
 
310. Working Journalist Act
 
311. Press Commission
 
312. Contempt of Court
 
313. Principles of Organization
 
314. Delegation & Decentralization
 
315. Staffing
 
316. Concept of Management as an art and/or a science.
 
317. Role of Print Media
 
318. Identification of Letterpress Printing Process
 
319. Screen Printing Process
 
320. Evolution of Print Media.
 
321. Trade Fairs as PR Exercise
 
322. House Journals
 
323. 10-point guidelines for Government Media Units
 
324. Social Impact of Advertising
 
325. Automobiles are the first sign of affluence. Critically examine it.
 
326. What are the key factors for evaluating a new car?
 
327. Consolidated Fund
 
328. Contingency Fund
 
329. Revenue Budget
 
330. Capital Budget
 
331. GDP
 
332. Style-book
 
333. Cross-heads
 
334. Five Ws and One H
 
335. Feature Agencies
 
336. Box Items
 
337. News Sources
 
338. Newspaper Language
 
339. News Values
 
340. The Front Page
 
341. Inverted Pyramid Style
 
342. Photo Editing
 
343. Style-book
 
344. Cross-heads
 
345. Five Ws and One H
 
346. Feature Agencies
 
347. Box Items
 
348. News Sources
 
349. Newspaper Language
 
350. Research process
 
351. Data collection methods
 
352. Stratified sampling
 
353. Questionnaire design
 
354. Planning a field based survey
 
355. Sign as the basic unipolar of message
 
356. "A photograph is worth a thousand words'
 
357. Morality and electronic media
 
358. Media Ethics
 
359. Editing Symbols
 
360. News Sources
 
361. Newspaper language
 
362. Privileges of Parliament : Cautions in reporting Parliament
 
363. Trial by the Press
 
364. News worthiness
 
365. Scoop
 
366. Defamation
 
367. Media ethics
 
368. News flow
 
369. Pressures on press
 
370. Page make-up
 
371. DTP Software
 
372. Johann Gutenberg
 
373. Primary colors for printing
 
374. The feedback mechanism
 
375. Opinion leaders
 
376. The role of audiences
 
377. Emerging information scenario
 
378. Barriers to communication
 
379. James Augustus Hicky
 
380. James Silk Buckingham
 
381. Amrit Bazar Patrika
 
382. Robert Knight Annie Besant
 
383. Essar SMS
 
384. Timex Alto ("The hottest little car in town")
 
385. Provogue (Fardeen Khan)
 
386. Clinic All Clear ("Dho Dhala”)
 
387. Whirlpool refrigerators ("Ice, Ice, baby”)
 
388. Role of Print Media
 
389. Identification of Letterpress Printing Process
 
390. Screen Printing Process
 
391. Evolution of Print Media
 
392. Mass Communication
 
393. Communication is a two-way process
 
394. Social Responsibility theory of communication
 
395. Shannon and Weaver model of communication
 
396. Developmental Communication
 
397. Language press
 
398. Information Super Highway
 
399. Press Council of India
 
400. Development Communication
 
401. Communication Research
 
402. Emerging information scenario
 
403. Two-step flow of communication
 
404. Interpersonal communication
 
405. Sampling design
 
406. Trade Fairs as PR Exercise
 
407. House Journals
 
408. 10-point guidelines for Government Media Units
 
409. Social Impact of Advertising.
 
410. Amrit Bazar Patrika
 
411. B.G. Horniman
 
412. Times of India
 
413. Mahatma Gandhi's views about Journalism
 
414. Account Management
 
415. Creative
 
416. Media
 
417. Positioning
 
418. Socio-Economic classification
 
419. Television Rating Points (TRPs)
 
420. Brand Equity
 
421. Qualitative Research.
 
422. "Publics" in PR
 
423. PR and Laws
 
424. Internal Communication Media.
 
425. Prototype production
 
426. Satellite broadcasting
 
427. ESP of India
 
428. CBS of India.
 
429. Basic shots
 
430. P to C & VO
 
431. TV Lighting
 
432. TV studio production
 
433. HDTV
 
434. Writing for the Internet Digital and Analog Technologies
 
435. Web based news
 
436. AIDS
 
437. Communal Harmony
 
438. Pulse Polio Campaign
 
439. Violence against women
 
440. Trade Fairs as PR Exercise.
 
441. House Journals
 
442. 10-point guidelines for Government Media Units
 
443. Social Impact of Advertising
 
444. Communication support for development
 
445. Vicious circle of poverty
 
446. Use of media for development
 
447. Human Rights and development
 
448. Act
 
449. Ordinance
 
450. Statute Constitution
 
451. Press and Euoks Registration Act
 
452. Working Journalists Act
 
453. Social Responsibility of Press
 
454. Contempt of Court
 
455. Conventions
 
456. Right to Information
 
457. Yellow journalism
 
458. Vernacular Press Act 1878
 
459. Verbal and Non-verbal Communication
 
460. Visual Communication
 
461. Culture and Communication Media
 
462. Official Secrets Act
 
463. Democracy and press laws
 
464. Right to Information
 
465. Broadcast Bill
 
466. Press and Hainan values
 
467. Yellow Journalism
 
468. Pressures on the press
 
469. Defamation - • e. Media Ethics
 
470. MCD Poll campaign
 
471. Auto strike in the Capital
 
472. Brand width
 
473. HDTV
 
474. Use of Subways in the Capital
 
475. Public Transport
 
476. Trace the development of cable TV in India.
 
477. Information Super highway d) LAN
 
478. James Buckingham
 
479. Commercial TV
 
480. Mahatma Gandhi as a Communicator
 
481. Maulana Azad as a journalist
 
482. Public Service Broadcasting
 
483. Press Council Act, 1978
 
484. Copy Right Act 1957 (Amendmntie Act to curb/prevent Video Piracy)
 
485. Working Journalists Act.
 
486. Comprehensive layout
 
487. Type legibility
 
488. Halftone illustration
 
489. Graphics
 
490. t-test
 
491. z-test
 
492. Covariance and correlation
 
493. SPSS statistical package
 
494. Types of mean
 
495. Defamation
 
496. Prasar Bharati Act
 
497. Official Secrets Act
 
498. Media Ethics.
 
499. Law
 
500. Norms
 
501. Convention
 
502. Ethics
 
503. Legibility of type composition
 
504. Pica scale
 
505. Display type
 
506. Centre spread of a magazine
 
507. Comprehensive layout
 
508. Type legibility
 
509. Half-tone illustration
 
510. Graphic
 

Revision as of 11:51, October 4, 2013

Tip 648 Printable Monobook Previous Next

created 2004 · complexity intermediate · author Michael Geddes · version 7.0


The following command will sort all lines and remove duplicates (keeping unique lines):

:sort u

If you need more control, here are some alternatives.

There are two versions, the first leaves only the last line, the second leaves only the first line. (Use \zs for speed reason.)

g/^\(.*\)$\n\1$/d
g/\%(^\1$\n\)\@<=\(.*\)$/d

Breakdown of the second version:

g//d <-- Delete the lines matching the regexp
\@<= <-- If the bit following matches, make sure the bit preceding this symbol directly precedes the match
\(.*\)$ <-- Match the line into subst register 1
\%( ) <--- Group without placing in a subst register.
^\1$\n <--- Match subst register 1 followed by end of line and the new line between the 2 lines

In this simple format (matching the whole line), it's not going to make much difference, but it will start to matter if you want to do stuff like match the first word only.

This does a uniq on the first word in the line, and deletes all but the first line:

g/\%(^\1\>.*$\n\)\@<=\(\k\+\).*$/d

See also

Comments

Here are some more Vim-native ways for removing duplicate lines. This time they don't have to be adjacent. Line order is preserved.

This one can be a bit slow. And the pattern would match a single empty line which would also be deleted. The part ":g/^m0<CR>" at beginning and end of the command maybe optional.

:nno \d1 :g/^/m0<CR>:g/^\(.*\)\n\_.*\%(^\1$\)/d<CR>:g/^/m0<CR>

This is faster. Uses mark l.

:nno \d2 :g/^/kl\|if search('^'.escape(getline('.'),'\.*[]^$/').'$','bW')\|'ld<CR>

Following uses a substitute to delete all repeated lines (leaving only the first line, while deleting following duplicate lines). This is a variation on the g//d method.

%s/^\(.*\)\(\n\1\)\+$/\1/

Around Wikia's network

Random Wiki