EMC Documentation Wiki: TroubleShooting

Troubleshooting

1. emc2 doesn't run

Error messages like this:

....
Realtime system did not load
....
Debug file information:
insmod: can't read
'/usr/realtime-2.6.24-19-generic/modules/rtai_hal.ko': No such file or directory

This error means the wrong kernel has been booted (in this case 2.6.24-19-generic). Always make sure you have a rtai patched kernel selected at boottime (2.6.24-16-rtai for hardy, 2.6.15-magma for dapper).

2. Checking the RealTime subsystem

EMC uses a RealTime operating system in order to ensure precise timing of i/o signals and trajectory calculations. Error messages like one of the following may indicate that the realtime operating system is unreachable:

Can't write to /dev/rtai_shm - aborting
RTAPI: ERROR: could not open shared memory
ERROR: Could not load 'rtapi'
insmod: error inserting '/lib/modules/2.6.12.6-magma/rtai/rtai_up.ko': 
-1 Operation not permitted

A script is provided with emc2 to load and unload all the realtime modules, and is much quicker for troubleshooting than starting and stopping all of emc2. To use it, type:

 /etc/init.d/realtime

2.1. RT device files

There are several device files required for Linux applications such as emc2 to talk to the RealTime OS. These device files have a reputation for getting deleted by udev. Make sure that these files exist:

ubuntu:$ ls -l /dev/rt*
lrwxrwxrwx  1 root root 8 2006-05-23 23:34 /dev/rtai_shm -> RTAI_SHM
crw-rw----  1 root audio 10, 135 2006-05-14 16:12 /dev/rtc

ubuntu:$ ls -l /dev/RTAI_SHM 
crw-rw-rw-  1 root root 10, 254 2006-05-23 23:34 /dev/RTAI_SHM

Or, your system may look something like this:

fedora]$ ls -l /dev/rt*
crw-rw-rw-  1 root root  10, 254 Jul 18  2005 /dev/rtai_shm
crw-r--r--  1 root root  10, 135 Jun  4 16:21 /dev/rtc
crw-rw-rw-  1 root root 150,   0 Jul 18  2005 /dev/rtf0
crw-rw-rw-  1 root root 150,   1 Jul 18  2005 /dev/rtf1
crw-rw-rw-  1 root root 150,   2 Jul 18  2005 /dev/rtf2
crw-rw-rw-  1 root root 150,   3 Jul 18  2005 /dev/rtf3
crw-rw-rw-  1 root root 150,   4 Jul 18  2005 /dev/rtf4
crw-rw-rw-  1 root root 150,   5 Jul 18  2005 /dev/rtf5
crw-rw-rw-  1 root root 150,   6 Jul 18  2005 /dev/rtf6
crw-rw-rw-  1 root root 150,   7 Jul 18  2005 /dev/rtf7
crw-rw-rw-  1 root root 150,   8 Jul 18  2005 /dev/rtf8
crw-rw-rw-  1 root root 150,   9 Jul 18  2005 /dev/rtf9

If these files do not exist, you can create them, for example: sudo mknod /dev/RTAI_SHM c 10, 254;

3. Unexpected realtime delay; check dmesg for details

Starting in EMC2 version 2.0.4, the number of CPU cycles between invocations of the real-time motion thread is tracked. If you see this message, it usually indicates that some element of your hardware is incompatible with the real-time software. See the next section for more information about possible causes and remedies.

3.1. RTAI Latency test

To specifically test the operation of the RealTime kernel, use the kernel latency test supplied with RTAI.
DO NOT TRY TO RUN EMC2 WHILE THE TEST IS RUNNING
On Ubuntu Dapper this is accomplished by doing:

sudo mkdir /dev/rtf; sudo mknod /dev/rtf/3 c 150 3;
sudo mknod /dev/rtf3 c 150 3; 
cd /usr/realtime*/testsuite/kern/latency; ./run

and then you should see something like this:

ubuntu:/usr/realtime-2.6.12-magma/testsuite/kern/latency$ ./run 
*
*
* Type ^C to stop this application.
*
*

## RTAI latency calibration tool ##
# period = 100000 (ns) 
# avrgtime = 1 (s)
# do not use the FPU
# start the timer
# timer_mode is oneshot

RTAI Testsuite - KERNEL latency (all data in nanoseconds)
RTH|    lat min|    ovl min|    lat avg|    lat max|    ovl max|   overruns
RTD|      -1571|      -1571|       1622|       8446|       8446|          0
RTD|      -1558|      -1571|       1607|       7704|       8446|          0
RTD|      -1568|      -1571|       1640|       7359|       8446|          0
RTD|      -1568|      -1571|       1653|       7594|       8446|          0
RTD|      -1568|      -1571|       1640|      10636|      10636|          0
RTD|      -1568|      -1571|       1640|      10636|      10636|          0

There should be no overruns, and "lat max" should probably be below your BASE_PERIOD setting.

Some issues with hardware that cause the RT kernel to not work correctly:

3.1.1. On-board video

Disable it and plug in an AGP or PCI video card.

Avoid anything nvidia. Old matrox (millenium, G400, G450 era) work great. Some older ATI work great, not so sure about recent stuff.

3.1.2. APM and ACPI bios settings

Turn everything off that you can. Any power saving, anything related to suspending, cpu frequency scaling, etc. Enabling ACPI could be the only way to get access to local APIC timer (much lower latency than PIC), as new mainboards do not come with legacy support of MPTABLEs. Disable C1E power-saving feature in BIOS (could save about ~10-15ms on recent CPUs), this feature is activated regardless of ACPI or APM, thus needs to be disabled independently.

3.1.3. NVidia and ATI graphics cards

Your machine should have smooth movement while opening and closing windows on linux. If not there may be an issue with the RT kernel, or the way it works together with your setup. First try using the open-source driver (e.g., "nv" instead of "nvidia") and then try using the unaccelerated driver ("vesa").

3.1.4. Also take note of hardware that isn't plugged in all the time..

There have been issues with usb key chain drives not playing well. The latest emc does latency tests on the fly which will help in finding uncooperative hardware.

3.1.5. Some Intel boards have issues with the SMI (System Management Interrupt).

The way to address this is described further.

3.1.6. PC speaker module (pcspkr)

PC speaker module causes some problems for some systems. To prevent to load pcspkr in Debian Etch (and maybe in Ubuntu):

 su -
 echo install pcspkr /bin/true >/etc/modprobe.d/rtai
 rmmod pcspkr

3.1.7. Additional potential [[sources]].

4. Parallel port no longer works in EMC 2.0.1 or later (hal_parport: Device or resource busy)

Those who are not using the official emc2 packages for Ubuntu may encounter this error when starting EMC:

 insmod: error inserting '/home/jepler/src/emc2/rtlib/hal_parport.ko': -1 Device or resource busy

If you encounter this problem, you must make sure that the Linux kernel module parport_pc is not loaded at boot time. On Ubuntu systems, you can do this by creating a file in /etc/modprobe.d/ with the one line

 install parport_pc /bin/true

The official packages put this line in the file /etc/modprobe.d/emc2. Different Linux distributions may have a different method to stop parport_pc from loading.

5. Parallel port no longer works in EMC 2.0.1 or later (emc starts but motors don't turn)

Some users have reported that in EMC 2.0.1, EMC appears to start, but no signals ever appear on the parallel port (the motors don't turn).

To fix this problem, upgrade to EMC 2.0.3 or newer. Then, add the command

 loadrt probe_parport

before the line loading hal_parport in your hal file.

In EMC 2.0.1, the Linux parallel port driver must be completely disabled for EMC's hal_parport driver to load. However, as we learned after the release of 2.0.1, this disabled certain parallel ports that are "PNP" (plug and play) devices. The new probe_parport realtime module performs the probing for one type of PNP port. If probe_parport doesn't allow your card to work, please contact the EMC developers.

6. Stepper motors lose steps

See TweakingSoftwareStepGeneration.

7. Mesa 5i20 FPGA firmware and/or driver won't load

First, you gotta have a 5i20 board ;-)

Assuming that's not the issue, one user reported that his board wasn't enabled by the BIOS at boot time. Running "lspci -v" revealed this (info for other devices has been snipped):

0000:00:0e.0 Bridge: PLX Technology, Inc. PCI <-> IOBus Bridge Hot Swap
	Subsystem: PLX Technology, Inc.: Unknown device 3131
	Flags: medium devsel, IRQ 9
	Memory at feddfc00 (32-bit, non-prefetchable) [disabled] [size=128]
	I/O ports at fc00 [disabled] [size=128]
	I/O ports at f400 [disabled] [size=256]
	I/O ports at f800 [disabled] [size=256]
	Memory at fede0000 (32-bit, non-prefetchable) [disabled] [size=64K]
	Memory at fedf0000 (32-bit, non-prefetchable) [disabled] [size=64K]
	Capabilities: <available only to root>

The important item is "[disabled]" in the "I/O ports" and "Memory" lines. The user was able to solve the problem by setting "Plug-N-Play OS" in the BIOS setup screen to "NO" and rebooting. Googling finds that "NO" is probably the correct answer to that BIOS question anyway, see http://tldp.org/HOWTO/Plug-and-Play-HOWTO-3.html

8. Printing to parallel-port printers does not work

By default, the emc2 package for Ubuntu disables the linux parport driver entirely, because the linux parport driver interferes with emc hardware drivers that use the parallel port.

To reenable the linux parport driver and disable emc hardware drivers that use the parallel port, remove the line "install parport_pc /bin/true" from the file /etc/modprobe.d/emc2, and reboot or otherwise cause the parport-related linux kernel modules to be loaded.

If your system has multiple parallel ports, you can also use some for linux and the others for emc. In this case, remove the "install" line as above and add an "options parport_pc" line to /etc/modprobe.d/emc2 which gives the correct I/O addres(es) for the ports to be used by Linux. For the format of the options line, see the [linux kernel documentation].

9. Testing Parallel Port Outputs

Not sure if a parallel port output is working on the hardware level. Turn off power to the steppers or unplug the motors first. From the Hal Configuration Screen drill down to Pins/parport/0 now you see all the pins for your parallel port. You must unlink the pin to manually turn it on and off if it is linked. In the Test HAL command: unlinkp parport.0.pin-nn-out Where nn is 01 02 etc. Do this for each pin you want to test. The unlinkp is unlink pin. Now to toggle each pin on and off use the command: setp parport.0.pin-nn-out 1 The setp is set pin. The 1 turns it on a 0 turns it off.

10. Via Unichrome Display Issue with Hardy Install

If you see part of the Axis tool path display missing, just plain black, and it looks like you can slide the tool path behind it, you are probably seeing a bug in the new via display driver. To fix it make a backup copy of /etc/X11/xorg.conf named xorg.conf.bak using a terminal command like this.

 sudo cp /etc/X11/xorg.conf /etc/X11/xorg.conf.bak

Then open xorg.conf using this terminal command

 sudo gedit /etc/X11/xorg.conf

and look down until you find the following lines.

     Section "Device
            Identifier   "Configured Video Device"

Add the following line immediately below this.

            Driver "vesa"

Save the file and restart the X system by holding down the keyboard keys <control> and <alt> and tapping <Backspace>. You'll have to log in again when X restarts.

If the display does not come up properly and you are dropped into text mode you should be able to log in using your name and password and issue the following commands.

 sudo rm /etc/X11/xorg.conf
 sudo cp /etc/X11/xorg.conf.bak /etc/X11/xorg.conf

You may be able to restart the X server using this command.

 sudo gdm

If not you'll have to reboot and try the edit again. As always YMMV.

11. USB thumb drive fat corrupted

 I fixed 'unexpected realtime delays' caused by a USB thumb drive
 the error was repeatable during 3 reboots
 dmesg showed no problems
 but messages reported that when the thumb was mounted , there were errors detected and suggested running e2fsck
 now this thumb had 2 partitions, one was ext2 , the other was vfat
 so i ran e2fsck on the ext2, then inspected (didnt change) the vfat paritition
 i used fsck.vfat to look, just didnt let it change things
 (reason, some old memory of 'always let the native OS fix drive errors )
 then i let Vista check & repair the vfat partition
 now, i can mount and unmount the device, read & write to it, play music off it, read pdfs off it
 and no UDR's (no unexpected realtime delays)
 tom3p 11jan2010