Changes: Auto-detect number of cores for parallel build

Latest revision as of 08:22, 14 July 2012

Proposed tip Please edit this page to improve it, or add your comments below (do not use the discussion page).

Please use new tips to discuss whether this page should be a permanent tip, or whether it should be merged to an existing tip.

created April 13, 2012 · complexity basic · author Knue · version 7.0

Vim's :make command defaults to invoke make. This tip shows how, on Linux, you can autodetect the number of processors you are running in order to use make -jN instead. N is a number that is one more than the number of processors on the principle that there should always be a job waiting to be run.

Simple[]

Put the following in your vimrc:

if filereadable('/proc/cpuinfo')
  let &makeprg = 'make -j'.(system('grep -c ^processor /proc/cpuinfo')+1)
endif

The above command sets the 'makeprg' option using :let to avoid the escaping that would be required if the normal :set command were used. Using :let also allows an expression to be assigned to the option. Vim concatenates make -j and N where N is the number of occurrences of "processor" at the start of a line (^) in file /proc/cpuinfo, plus 1. The output from system() includes a trailing newline, but Vim ignores that when converting the string to a number before adding 1.

Alternative[]

Might use this alternative formulation instead of the above, and possibly expand to handle other techniques?

The following sets 'makeprg' to make if there is one processor, or to make -jN if multiple processors are detected (N is a number that is one more than the number of processors).

function! SetMakeprg()
  if !empty($NUMBER_OF_PROCESSORS)
    " this works on Windows and provides a convenient override mechanism otherwise
    let n = $NUMBER_OF_PROCESSORS + 0
  elseif filereadable('/proc/cpuinfo')
    " this works on most Linux systems
    let n = system('grep -c ^processor /proc/cpuinfo') + 0
  elseif executable('/usr/sbin/psrinfo')
    " this works on Solaris
    let n = system('/usr/sbin/psrinfo -p')
  else
    " default to single process if we can't figure it out automatically
    let n = 1
  endif
  let &makeprg = 'make' . (n > 1 ? (' -j'.(n + 1)) : '')
endfunction
call SetMakeprg()

Use set makeprg? to view the current setting.

Comments[]

I have updated the tip with some ideas by Christian Brabandt on the vim_use mailing list. JohnBeckett 10:18, April 16, 2012 (UTC)

Related tips[]

I wonder if one of the above might be a good place to merge this tip? Might need to rename the old tip and clean it up. JohnBeckett 10:16, April 15, 2012 (UTC)

No, I think this task is specific and separate enough that it deserves its own tip. Especially if we add other systems (see below). --Fritzophrenic 16:41, April 16, 2012 (UTC)

OK, although I will have a closer look at the others later and see if they have useful content (I've only identified they exist now).

Any thoughts on the "Alternative" above? I'm pretty sure NUMBER_OF_PROCESSORS is only used on Windows, but it seems harmless to use it if it exists on other systems. Is there anything in the "Windows" section below that should be added here? If not, let's remove the link so I won't wonder whether to look at it later. JohnBeckett 23:12, April 17, 2012 (UTC)

I think the alternative is good, assuming it works. I've added the Solaris tool to the mix and added comments, so I've deleted the links I provided earlier. --Fritzophrenic 21:47, May 7, 2012 (UTC)

@@ Line 1: / Line 1: @@
+{{TipProposed
-Vim's <code>make</code> falls back to just invoke <code>make</code>.
+|id=0
-With this trick you can autodetect the number of cores you are running. Usually it's a good idea to add '1' to this number and pass this number to <code>make</code> using the <code>-j</code> switch.. Simply add
+|previous=0
+|next=0
+|created=April 13, 2012
+|complexity=basic
+|author=Knue
+|version=7.0
+|subpage=/201204
+|category1=Building Vim
+|category2=
+}}
+Vim's <code>:make</code> command defaults to invoke <code>make</code>. This tip shows how, on Linux, you can autodetect the number of processors you are running in order to use <code>make -jN</code> instead. <code>N</code> is a number that is one more than the number of processors on the principle that there should always be a job waiting to be run.
+==Simple==
+Put the following in your [[vimrc]]:
 <pre>
+if filereadable('/proc/cpuinfo')
-set makeprg=make\ -j`cat\ /proc/cpuinfo\ \\\|\ grep\ processor\ \\\|\ echo\ \\`wc\ -l\\`\ +\ 1\ \\\|\ bc`
+  let &makeprg = 'make -j'.(system('grep -c ^processor /proc/cpuinfo')+1)
+endif
 </pre>
-to your <code>~/vimrc</code>.
+The above command sets the <code>'makeprg'</code> option using <code>:let</code> to avoid the escaping that would be required if the normal <code>:set</code> command were used. Using <code>:let</code> also allows an expression to be assigned to the option. Vim concatenates <code>make -j</code> and <code>N</code> where <code>N</code> is the number of occurrences of "<code>processor</code>" at the start of a line (<code>^</code>) in file <code>/proc/cpuinfo</code>, plus 1. The output from <code>system()</code> includes a trailing newline, but Vim ignores that when converting the string to a number before adding 1.
+==Alternative==
+:''Might use this alternative formulation instead of the above, and possibly expand to handle other techniques?''
+The following sets <code>'makeprg'</code> to <code>make</code> if there is one processor, or to <code>make -jN</code> if multiple processors are detected (<code>N</code> is a number that is one more than the number of processors).
+<pre>
+function! SetMakeprg()
+  if !empty($NUMBER_OF_PROCESSORS)
+    " this works on Windows and provides a convenient override mechanism otherwise
+    let n = $NUMBER_OF_PROCESSORS + 0
+  elseif filereadable('/proc/cpuinfo')
+    " this works on most Linux systems
+    let n = system('grep -c ^processor /proc/cpuinfo') + 0
+  elseif executable('/usr/sbin/psrinfo')
+    " this works on Solaris
+    let n = system('/usr/sbin/psrinfo -p')
+  else
+    " default to single process if we can't figure it out automatically
+    let n = 1
+  endif
+  let &makeprg = 'make' . (n > 1 ? (' -j'.(n + 1)) : '')
+endfunction
+call SetMakeprg()
+</pre>
+Use <code>set makeprg?</code> to view the current setting.
+==Comments==
+I have updated the tip with some ideas by Christian Brabandt on the vim_use mailing list. [[User:JohnBeckett|JohnBeckett]] 10:18, April 16, 2012 (UTC)
+===Related tips===
+*[[VimTip210|210 Make-compile current buffer]]
+*[[VimTip264|264 Map function keys to compile and run your code]]
+*[[VimTip476|476 Errorformat and makeprg]]
+I wonder if one of the above might be a good place to merge this tip? Might need to rename the old tip and clean it up. [[User:JohnBeckett|JohnBeckett]] 10:16, April 15, 2012 (UTC)
+:No, I think this task is specific and separate enough that it deserves its own tip. Especially if we add other systems (see below). --[[User:Fritzophrenic|Fritzophrenic]] 16:41, April 16, 2012 (UTC)
+::OK, although I will have a closer look at the others later and see if they have useful content (I've only identified they exist now).
+::Any thoughts on the "Alternative" above? I'm pretty sure NUMBER_OF_PROCESSORS is only used on Windows, but it seems harmless to use it if it exists on other systems. Is there anything in the "Windows" section below that should be added here? If not, let's remove the link so I won't wonder whether to look at it later. [[User:JohnBeckett|JohnBeckett]] 23:12, April 17, 2012 (UTC)
+:::I think the alternative is good, assuming it works. I've added the Solaris tool to the mix and added comments, so I've deleted the links I provided earlier. --[[User:Fritzophrenic|Fritzophrenic]] 21:47, May 7, 2012 (UTC)