[Home]FixingSMIIssues

LinuxCNCKnowledgeBase | RecentChanges | PageIndex | Preferences | LinuxCNC.org

Fixing Realtime problems caused by SMI on Ubuntu

1. Intro
2. Turning off SMI may be dangerous
3. Typical SMI interrupt
4. Configuring SMI on Hardy or Lucid (Ubuntu 8.04 or 10.04, any LiveCD installation newer than about 2008 (emc2-2.3.0))
5. configuring SMI.ko on Dapper or any other installation where it is not present by default
6. Changing smi-module.c
7. Compiling the new module
8. Installing the new module
9. Making EMC2 use the module
10. Alternate Approach: smictrl

See also: RealTime, The Isolcpus Boot Parameter And GRUB2

IMPORTANT ==========

This is not something you can just try if your latency is high.

You must establish that you have an Intel chipset which is amongst those affected AND that you are getting periodic regular spikes (32 or 64 seconds apart is common)

Update 20141008: I have a Dell precision laptop with an Intel 6 series/C200 chipset that was not included in the list below and which had periodic but not entirely regular latency spikes up to 200us. These came at intervals ranging from roughly 10 to 50 seconds. I was able to successfully eliminate them using the smictrl alternate approach documented below.

The chipsets in the 2.6.32-122-rtai realtime module are

82801AA_0
82801AB_0
82801BA_0
82801BA_10
82801E_0
82801CA_0
82801CA_12
82801DB_0
82801DB_12
82801EB_0
ICH6_0
ICH6_1
ICH6_2
ICH7_0
ICH7_1
ICH8_4

(Also ICH10_1 has been introduced in the later versions of magma, but to use this you would have to be comfortable with compiling your own module)

Run lspci -vv from a terminal to determine your chipset, the revision number in brackets is normally the match to the _N portion of the list above, eg ICH6 (revision 1) = ICH6_1

1. Intro

If you have a fairly new motherboard with Intel chipset (e.g. Intel 82845 845 (Brookdale) is a very common one), it most likely has SMI (System Maintenance Interrupt). SMI is a real-time killer and to avoid latencies of several 100 microseconds up to a few milliseconds, you need to disable them.

This issue is known by the RTAI people, as described in [[README.SMI]]

This document describes the needed steps to turn off SMI.

2. Turning off SMI may be dangerous

Disabling SMI can cause some computers to overheat and may damage those computers. Intel developer documentation states that running their modern processors with SMI disabled is running their processors outside of thermal spec, just like running the CPU with inadequate cooling is. Turn off SMI at your own risk.

3. Typical SMI interrupt

Typical SMI interrupt, seen while running the testsuite/kern/latency example of RTAI 3.3:
RTD|      -1234|      -1345|        583|       2206|       9884|          0
RTD|      -1209|      -1345|        575|       2098|       9884|          0
RTD|      -1206|      -1345|        591|     236277|     236277|          2
RTD|      -1261|      -1345|        609|       5581|     236277|          2
RTD|      -1274|      -1345|        579|       2468|     236277|          2
The SMI interrupt usually is very periodic, so if you let the latency test run for a while, you'll see it repeats. It has been reported that one value for the period is exactly 64 seconds.

RTAI comes with a kernel module, which disables all or certain SMI interrupt. This is included in the Hardy and Lucid EMC2 builds, but you will need to compile it yourself or download the binary for Dapper (Does anyone still use EMC2 on Dapper?)

4. Configuring SMI on Hardy or Lucid (Ubuntu 8.04 or 10.04, any LiveCD? installation newer than about 2008 (emc2-2.3.0))

To make EMC actually load the module at startup, you need to edit a couple lines in /etc/linuxcnc/rtapi.conf (Or /etc/emc2/rtapi.conf on versions < 2.5). With a Run-in-place compiled version you will need to find a version in /usr/local/etc/{linuxcnc / emc2} instead. You can work out which one is being used by renaming them to see which one stops LinuxCNC starting.)

rtapi.conf
#List of realtime kernel modules to be loaded
if [ "3" = "3" ] ; then \ 
    MODULES="adeos rtai_hal rtai_ksched rtai_fifos rtai_shm rtai_sem rtai_math rtai_smi"
    RTAI=3
    MODPATH_adeos=
    MODPATH_rtai_smi=/usr/realtime-$(uname -r)/modules/rtai_smi.ko
    MODPATH_rtai_hal=/usr/realtime-$(uname -r)/modules/rtai_hal.ko
    MODPATH_rtai_ksched=/usr/realtime-$(uname -r)/modules/rtai_ksched.ko
    MODPATH_rtai_fifos=/usr/realtime-$(uname -r)/modules/rtai_fifos.ko
    MODPATH_rtai_shm=/usr/realtime-$(uname -r)/modules/rtai_shm.ko
    MODPATH_rtai_sem=/usr/realtime-$(uname -r)/modules/rtai_sem.ko
    MODPATH_rtai_math=/usr/realtime-$(uname -r)/modules/rtai_math.ko

You need to add 'rtai_smi' into the list of modules, and add the extra MODPATH line.

When next you start EMC2, rtai_smi should be loaded:

$ lsmod | grep rtai
rtai_smi                2444  0
rtai_math              30468  0
rtai_sem               16640  1 rtapi
rtai_shm                9092  1 rtapi
rtai_fifos             26060  1 rtapi
rtai_sched             77848  4 rtapi,rtai_sem,rtai_shm,rtai_fifos
rtai_hal               32464  5 rtapi,rtai_sem,rtai_shm,rtai_fifos,rtai_sched
$

The above works for installed emc2 - for 'run in place' it is a bit different.

Copy the rtai_smi.ko to the correct location shown above

You need to edit the /your/rip/directory/scripts/rtapi.conf.in and make it look like

   
if [ "@RTAI@" = "3" ] ; then \     
    MODULES="adeos rtai_hal rtai_ksched rtai_fifos rtai_shm rtai_sem rtai_math rtai_smi"
    RTAI=@RTAI@
    MODPATH_adeos=@MODPATH_adeos@
    MODPATH_rtai_smi=/usr/realtime-2.6.15-magma/modules/rtai_smi.ko
    MODPATH_rtai_hal=@MODPATH_rtai_hal@

Then go into the /your/rip/directory/src
Make

It should now all work.

5. configuring SMI.ko on Dapper or any other installation where it is not present by default

You have to set the SMI options before you compile RTAI, or compile the SMI disabling module individually as described further.
You can either follow steps 6 and 7 and build a new module, or grab the binary (for Dapper only) from [[here]].

6. Changing smi-module.c

First you need to get the sources for the dapper version of RTAI:

to get the sources for the dapper version you need to
apt-get source rtai-3.3

Find the smi-module.c inside rtai/base/arch/i386/calibration, and change the following line:

#define CONFIG_RTAI_HW_SMI_ALL 1

Once that is changed, you can compile the module and replace the version already installed.

7. Compiling the new module

Unless you want to use the existing build process (which means you need to put the rtai sources in /usr/src/modules/rtai-3.3/ and use make-kpkg to build the modules) use the following Makefile:

Makefile
UNAME := $(shell uname -r)
KERNEL26 := 2.6
KERNELVERSION := $(findstring $(KERNEL26),$(UNAME))

obj-m	:= rtai_smi.o
rtai_smi-objs := smi-module.o

INCLUDE	:= -I/usr/include/asm/mach-default/
KDIR	:= /lib/modules/$(shell uname -r)/build
PWD		:= $(shell pwd)

all::
	$(MAKE) -C $(KDIR) $(INCLUDE) SUBDIRS=$(PWD) modules

Simply run make, and the new module should be built. (Alternative: grab the binary (for Ubuntu 6.06-Dapper Drake only) from [[here]]).

8. Installing the new module

Installing means simply copying the new binary module over the old version:

 sudo cp rtai_smi.ko /usr/realtime-2.6.15-magma/modules/rtai_smi.ko
(change 2.6.15-magma if your system uses a different kernel version)

9. Making EMC2 use the module

Follow the instructions for Hardy / Lucid in Step 4 above.

10. Alternate Approach: smictrl

I have a Dell precision laptop with an Intel 6 series/C200 chipset that was not recognized by rtai_smi.ko (20140922), and which had periodic but not entirely regular latency spikes up to 200us. These came at intervals ranging from roughly 10 to 50 seconds, see screenshot below.

I was eventually able to elimate these spikes with a usermode tool called [[smictrl]] developed by Jan Kiszka that directly reads/writes the SMI_EN register. Unlike rtai_smi.ko, this code is not limited to a specific set of chipsets. According to Jan's changelog: "All modern Intel chipsets have the SMI control register in their LPC at the same address. Make use of this pattern to avoid having to maintain a list of chipsets." Thus, if you have a fairly modern machine with an intel chipset not recognized by rtai_smi.ko, and you have large intermittent latency spikes like those shown above, you may want to try this.

In the process I [[added some code]] to optionally print out the individual SMI_EN bits with names, some options to set and clear individual bits, a dry run option, and an option to alternately operate on the "alternate GPIO SMI_EN" register. On my machine that register seems to be related to latency spikes that occur specifically when I flip the wireless disable switch on the chassis. Also, on my machine the GBL_SMI_EN (global SMI enable) bit in the SMI_EN register seems to be locked, so I cannot globally disable SMI, but experimentally I found that by disabling all the individual enable bits in SMI_EN and the "alternate GPIO SMI_EN" registers I was able to eliminate these large latency spikes. The fan still seems to operate and automatically regulates its own speed (probably it has a dedicated microcontroller for that). The wireless disable switch and the "fn+" keyboard special functions do not seem to operate with SMI disabled in this way, but other than that I have not found any other deficiencies. Wifi does work.


LinuxCNCKnowledgeBase | RecentChanges | PageIndex | Preferences | LinuxCNC.org
This page is read-only. Follow the BasicSteps to edit pages. | View other revisions
Last edited October 8, 2014 10:39 am by Martyvona (diff)
Search:
Published under a Creative Commons License