[Home]The Isolcpus Boot Parameter And GRUB2

LinuxCNCKnowledgeBase | RecentChanges | PageIndex | Preferences | LinuxCNC.org

1. SMP
2. Latency Testing with 3.4-9-rtai-686-pae
2.1. Latency test results for isolcpus=
2.2. Latency test results with idle=poll vs isolcpus=x
3. Latency Testing with 3.14 RTAI

1. SMP

This wiki page addresses the increasingly common problem of running LinuxCNC on computers with multiple cpus (aka cores). This situation is often called Symmetric Multiprocessing (SMP).

Other pages in this wiki describe how, for many multi-cpu/multi-core computers (but not necessarily all; a trial-and-error approach is best), the realtime performance of LinuxCNC significantly improves if the linux-rtai kernel is invoked with the boot parameter isolcpus=n, where n is the last cpu (starting with 0, so a dual-core machine would use isolcpus=1).

For computers using classic GRUB as the boot manager, this is accomplished by editing the file /boot/grub/menu.lst to include the boot parameter "isolcpus=1" (for example) at the end of appropriate kernel lines.
eg.
The below entries were used to isolate cores 1,2,3 on a quad core processor, leaving core 0 doing all the work

title EMC2 - rtai kernel isolcpu Ubuntu 10.04 (on /dev/sdb7)
uuid 42d5e8ad-507d-41ea-8ad6-868e87be2ed6
kernel /boot/vmlinuz-2.6.32-122-rtai root=UUID=42d5e8ad-507d-41ea-8ad6-868e87be2ed6 ro isolcpus=1,2,3
initrd /boot/initrd.img-2.6.32-122-rtai
quiet

For LinuxCNC installations based on Ubuntu 9.10 (Karmic Karma) and above, however, the default boot manager is no longer classic GRUB but instead GRUB2 (confusingly, this refers to GRUB v1.90 and above). The situation is more complicated with GRUB2 because of the manner in which various configuration files dynamically contribute to the file /boot/grub/grub.cfg, which replaces menu.lst. In particular, the most transparent attempt to accomplish the same end is to edit the appropriate "isolcpus=1" parameter into the key string GRUB_CMDLINE_LINUX_DEFAULT in the file /etc/default/grub, but this results in all linux kernel boot options being so marked, even if one has installed multiple linux kernels, some of which one wishes to use with all available cpus (for example, for CAD work).

This situation has been discussed at length on the linuxcnc-users email list in late March/early April 2011. The following is the solution Kent Reed proposed. There are other possible solutions, and Kent recommends reading the email archive.

As shipped, Ubuntu distributions using GRUB2 include a set of configuration script files in the directory /etc/grub.d/ with names 00_header, 05_debian_theme, 10_linux, etc. These files are processed in numerical order as a component of GRUB2 creates the file grub.cfg. Kent's solution is to add to this directory a new file 07_rtai that scans for linux-rtai kernels and for them only creates boot menu entries including the "isolcpus=1" parameter. Kent uploaded a copy of this file at [[07_rtai]].

This file has been updated to add support for Ubuntu 12.04, LinuxCNC 2.6.x, as it now looks for kernels ending not only in "rtai", but also "rtai-686-pae".

Assuming one has downloaded this file to one's home directory, the following commands will implement it.

<username>@<hostname>$ sudo cp 07_rtai /etc/grub.d
<username>@<hostname>$ sudo chmod +x /etc/grub.d/07_rtai
<username>@<hostname>$ sudo update-grub

At this point, one should find that the file /boot/grub/grub.cfg contains menu entries for the linux-rtai kernels with the boot parameter "isolcpus=1" set at the end of the kernel line. If the default file 10_linux is present and executable, the same linux-rtai kernels as well as all other installed linux kernels will also occur in following menu entries without this additional parameter present.

Kent Reed created the file 07_rtai by editing a copy of the distributed file 10_linux. Following Kent's introductory block of comments, all changes to the logic of 10_linux have been marked in 07_rtai by comment lines beginning #KAR:.

Two points to note are

1. The way GRUB2 works, Kent was not free to define his own keys, such as GRUB_CMDLINE_LINUX_RTAI and ISOLCPUS, in the file /etc/default/grub and expect them to be passed into 07_rtai as it is being created. This means one may need to edit two different files to modify what my work does. For example, if one wishes to change the boot menu entries so all linux kernels boot with no splash, one must edit the key GRUB_CMDLINE_LINUX_DEFAULT in the file /etc/default/grub. On the other hand, if one wishes to change the number of the highest cpu, then one must edit the string 'ISOCPUS="isolcpus=1"' defined early in the file /etc/grub.d/07_rtai. Kent Says "I'm truly sorry for this, but I can only blame the GRUB2 developers."

2. The default configuration of GRUB2 results in an immediate boot of the first linux kernel in the boot menu entries. With the addition of the file 07_rtai to a default Ubuntu distribution, this will be the latest version linux-rtai kernel GRUB2 found when creating the file /boot/grub/grub.cfg. During boot up, one can hold down the Shift-key while GRUB2 is starting to display the boot menu display, or one can edit the /etc/default/grub file to change GRUB2 behavior. Please see the GRUB2 manual or the appropriate Ubuntu forums.

This page was originally created by Kent Reed (aka cncdreamer) on 20110409. Updated for 12.04 pae kernels by John A. Stewart (aka Cnc-For-Model-Engineers) on 20140813.

2. Latency Testing with 3.4-9-rtai-686-pae

Wheezy LXDE, 3.4-9-rtai-686-pae, version 3.4.55-3 linuxcnc, Gigabyte F2A85X-UP4, A10 6800K 4-core APU

Other kernel parameters used for all tests:

radeon.modeset=0

UEFI Settings:

CPU Frequency 4.1GHz

Disable:

Core Performance Boost, Turbo CPB, Cool&Quiet, SVM Mode, C6 Mode, APM

kernel parametersMax Jitter(ns) 1mS thread latency-test25uS thread latency-testrtai kern/latency
radeon.modeset=0 (only) ~171,000 ~36,000 ~42,000

2.1. Latency test results for isolcpus=

Same hardware and UEFI settings as above.

isolcpus=Max Jitter(ns) 1mS thread latency-test25uS thread latency-testrtai kern/latency
0 ~75,000 ~30,000 ~4,300
1 ~98,000 ~30,000 ~32,000
2 ~13,000 ~7,000 ~50,000
3 ~160,000 ~32,000 ~32,000
0,1 ~55,000 ~45,000 ~4,100
2,3 ~10,500 ~4,500 ~32,000
1,2,3 ~10,800 ~5,300 ~32,000
0,1,2 ~10,500 ~5,400 ~2,400

2.2. Latency test results with idle=poll vs isolcpus=x

Same hardware and UEFI settings as above.

kernel parametersMax Jitter(ns) 1mS thread latency-test25uS thread latency-testrtai kern/latency
idle=poll ~200,000 ~38,000 ~42,000

The advice is to isolate both CPUs in the pair for a 4 core APU.

The recommended kernel parameter for 3.4-9-rtai-686-pae is:

isolcpus=2,3

We also confirmed that the latency-test and the rtai kern/latency test select cpu cores differently.

You want it to use cpu 0 if there is only cpu 0; cpu 1 if there are cpu 0 and cpu 1; and cpu 2 if there are cpu 0,1,2 or more cores.

The recommended kernel parameter for a 2 core APU is:

isolcpus=1

To save the kernel parameter in Wheezy so it boots with it every time edit:

/etc/default/grub

Then run:

update-grub

3. Latency Testing with 3.14 RTAI

There is a new scheduler in RTAI for 3.10 and 3.14. It no longer requires any isolcpus settings.

3.14 RTAI is in the testing stage at the time of this writing (September 2014). It should be in the official Linuxcnc release in the near future.


LinuxCNCKnowledgeBase | RecentChanges | PageIndex | Preferences | LinuxCNC.org
This page is read-only. Follow the BasicSteps to edit pages. | View other revisions
Last edited September 11, 2014 9:28 am by Bari (diff)
Search:
Published under a Creative Commons License