itrackutil 0.1.3 available

After a recent update in Ubuntu 10.10 and 10.04 itrackutil.py exits with the the error message “USBError: Resource Busy” before the user is able to download GPS data from the device.
This version fixes the problem.

Developer information:

A previously unrelated kernel module (cdc_acm) suddenly started to grab the USB device.  cdc_acm  which usually supports USB modems now creates the device /dev/ttyACM0 as soon as the GPS mouse is plugged in.

If you read /dev/ttyACM0 you get the real-time GPS data in NEMA format.  This is an unusual use case. The normal data connection is over Bluetooth.

However, the creation of this device file blocks all other USB access to the GPS mouse; in this case the direct USB communication which itrackutil.py uses to access the data stored within the unit's memory.

Fortunately there is an USB API call for this situation: detachKernelDriver.
This function does, what it says it does.  You have to call it after opening the device. Root privileges are not necessary.

The call will fail, if there is no kernel driver to detach. You have to catch this exceptions:

... determine device, interface, and configuration ...

try:
    handle = device.open()
    handle.detachKernelDriver(interface)
except usb.USBError, err:
    if str(err).find('could not detach kernel driver from interface') >= 0:
        pass
    else:
        raise usb.USBError,err       # any other USB error

handle.setConfiguration(configuration)


The companion API call attachKernelDriver is not available via PyUSB 0.1. But this is only a minor problem, because as soon as you unplug the unit and reconnect it, a new USB device file is created (with the kernel driver attached).

WinTV-HVR-1900 under Ubuntu 10.04 and 10.10

A few weeks ago I bought a Hauppauge WinTV-HVR-1900 (USB id 2040:7300) which I wanted to use with a (32 bit) Ubuntu 10.04 and 10.10 system.
 
Quick installation summary:

• Does it work out of the box: No
• Does it work at all: Yes
• If you are able to update to Ubuntu 10.10, do it.

This post only describes how to get the basic functionality working (i.e. display/record a signal on the composite video input).
 
The HVR 1900 is a small box which is connected via a USB2 port (USB1 is not supported for bandwidth reasons). There are inputs for composite video (e.g. VCR or set-top box) an antenna input and a S-VHS input. The device comes with a UK power supply (with a UK mains plug) and a bulky mains plug adapter for continental Europe.
 
In order to get the digitizer running, this device needs firmware.  Ubuntu comes with a selection of firmware images for various Hauppauge systems, but none was suitable for this device.
 
The firmware is included on the Windows driver disk you get with the device, but which files do you need?  Fortunately there is a perl script available that scans the CD and extracts the files you need based on a bunch of MD5 sums stored in that script.  Perl is installed by default on a Ubuntu system, so slide in the CD, open a terminal and enter
 
perl fwextract.pl /media/XXXXXXXXXX
 
(where XXXXX is the CD title)
 
In my case the following files were found:

• v4l-cx2341x-enc.fw
• v4l-cx25840.fw
• v4l-pvrusb2-29xxx-01.fw
• v4l-pvrusb2-73xxx-01.fw

 
For the next steps you need root privileges:
 

• Change the ownership of those files to root (for security reasons)
  sudo chown root:root *fw 
• Copy the extracted files to /lib/firmware
  sudo cp *fw /lib/firmware
  

 If you're running Ubuntu 10.10, that's all you have to do.
 
Keep an eye on the system log when you now plug the digitizer into the USB port.
tail -f /var/log/syslog
 
Among other messages, it should confirm that the firmware was uploaded successfully.
 
...
cx25840 5-0044: firmware: requesting v4l-cx25840.fw
cx25840 5-0044: loaded v4l-cx25840.fw firmware (16382 bytes)
...
 
Utilities for controlling the device from the command line can be found in the package ivtv-utils (from the Ubuntu repo).
 
v4l2-ctl --set-input 1
 switches to the composite video input
 
v4l2-ctl --set-ctrl=brightness=128,hue=0,contrast=68,volume=54000
 sets basic parameters for the digitizer.
 
Call v4l2-ctl without parameters for more help, and you will definitively want to try the switch -L
 
Recording is as simple as:
cp /dev/video0 video.mp2
 
This works most of the time.  Approx. 5% of the recordings contain a distorted audio stream. This distortion is present for the whole length of the recording and usually ends the next time the device /dev/video0 is opened. 
 
If the audio is ok at the beginning, it stays that way.  This looks like an initialization problem when the device is opened.  I haven't found a fix, yet.
 
Now to the more difficult part:

Getting the device to work under Ubuntu 10.04.
 
As mentioned before, many Hauppauge devices need firmware, which is uploaded to the unit when you plug it into the USB port.  Older hardware only needed 8192 bytes of firmware.  The firmware for this device however is 16382 bytes long (see the above firmware upload message from the log).  The device driver controlling the HVR-1900 (pvrusb2) that comes with kernel 2.6.32 and earlier is only capable of transferring 8192 bytes. And Ubuntu 4.10 LTS uses... 2.6.32.
 
Newer versions of the pvrusb2 driver can also upload the larger firmware.  For older kernels (like the one used in Ubuntu 4.10), you have to compile the updated driver yourself.
 
Compiling a kernel is usually a simple task, because the kernel source code already contains all dependencies. But this time, there were complications.
 
You need:

•  the kernel source
•  the tools to compile the kernel
  
•  and the source of the updated driver

The easiest way to get the Ubuntu Linux source is by installing a package named "linux-source". It will store the source code as an archive in /usr/src/linux-source-2.6.32.tar.bz2 
 
You have to unpack it - make sure that you have plenty of disk space available:
 
cd /usr/src
tar xvfj linux-source-2.6.32.tar.bz2
 
Then run the following commands
 
cd linux-source-2.6.32
make oldconfig
make prepare
 
This will "regenerate" the .config file used by your distribution's kernel.  This file is needed during the kernel compilation.
 
Now we have to download the source code of the current pvrusb2 driver which can be found here. Unpack it and copy the content of the directory driver to /usr/src/linux-source-2.6.32/drivers/media/video/pvrusb2/ overriding the current files with the same name.
 
(Please note: the pvrusb2 documentation is describing a different approach, that did not work for me (modprobe rejected the module))
 
The next step would be:
 
make modules
 
But due to a totally unrelated bug  the compilation will fail, while trying to compile the "omnibook" files.
 
Download the patch for this bug from here and apply it
 
cd /usr/src
patch -p0 _name_of_the_path_file_
 
Now it's time to compile the modules:
 
cd /usr/src/linux-source-2.6.32
make modules
 
This step is very time consuming. If you have a multi core processor use the -j# option (where # is the number of cores you have).
 
Copy the new module from
/usr/src/linux-source-2.6.32/drivers/media/video/pvrusb2/pvrusb2.ko
to
/lib/modules/`uname -r`/kernel/drivers/media/video/pvrusb2/pvrusb2.ko
 
(where `uname -r` (backticks!) will be replaced by the name of your current kernel)
 
Keep in mind that you have to repeat that process after each kernel update.
 
After the next reboot the new module should be active. If you can't wait, unload the old one and load the new module manually:
 
rmmod pvrusb2
modprobe pvrusb2
 
Again, check /var/log/syslog for any problems.
 
Links:

• http://www.isely.net/pvrusb2
• http://www.isely.net/pipermail/pvrusb2/
• https://help.ubuntu.com/community/Kernel/Compile
• http://www.isely.net/downloads/fwextract.pl

Integrating a re-branded Huawai UMTS modem into Ubuntu 10.04

The Huawai E160 is a popular UMTS stick and works with Ubuntu Lucid Lynx out-of-the-box.

It is also commonly used by European phone companies in their mobile internet starter packs.  Unfortunately these sticks are usually sim-locked to the phone company and some identify themselves with a USB id unknown to current Ubuntu setup.

When you plug the stick into the computer, all you see is a small flash drive icon containing the Windows and sometimes Macintosh drivers.  On these systems the drivers switches the stick from mass storage device mode into modem mode after the initial phase.

The Linux utility for this task is called modem-modeswitch, and can be found in /lib/udev.  The actual task is to automate the execution of this utility when the UMTS stick is plugged in.

The UMTS stick in this example is/was sold by the Belgian phone company Mobistar.  When you query the USB IDs:

sudo lsusb

it is identified as

12d1:1446 Huawei Technologies Co., Ltd. 


The two numbers at the beginning of the line are the vendor ID and product ID of the stick in the current mode.

To check if the modem switch works as expected, call the utility manually:

sudo /lib/udev/modem-modeswitch -v 0x12d1 -p 0x1446 -t option-zerocd

Wait a few seconds and check the USB IDs again.  The entry should now read:

12d1:1003 Huawei Technologies Co., Ltd. E220 HSDPA Modem / E270 HSDPA/HSUPA Modem


If nothing happens, David Santinoli suggests in a related document to unmount the device containing the drivers before attempting the switch.

You can automate the process by creating the file 62-huawai.rules in /lib/udev/rules.d:

ATTRS{idVendor}=="12d1", ATTRS{idProduct}=="1446", RUN+="modem-modeswitch -v 0x%s{idVendor} -p 0x%s{idProduct} -t option-zerocd"

(The file name is arbitrary, as long as it starts with “62-” and ends with “.rules”.) The rule itself is based on similar entries in 61-option-modem-modeswitch.rules.

Next time you boot your computer, the rule becomes active, and the modem should be recognized automatically.

Changing USB device permissions in Ubuntu Karmic

If you plug an USB device into a Linux box that uses a current distro, udev dynamically creates an entry in the /dev directory. You can list all currently available USB units with the lsusb command:

$ lsusb

Bus 002 Device 003: ID 03eb:6125 Atmel Corp. 

Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The only "real" device here is my XAiOX GPS logger (first entry). The output shows that it has the vendor id (hex) 03eb, the product id (hex) 6125.

The device file created by udev is: /dev/bus/usb/002/003
(Yes, these are the bus and device numbers shown in the lsusb output).

The default permissions are rather restrictive:

$ ls -l /dev/bus/usb/002

insgesamt 0

crw-rw-r-- 1 root root 189, 128 2009-12-23 22:14 001

crw-rw-r-- 1 root root 189, 128 2009-12-23 22:19 003

Only root can read and write to the USB device. Anybody else only has read access.

In order to retrieve the stored GPS info from my GPS logger (e.g. by using itrackutil), I have to instruct the device to send the data. In order to send this command I need the write privileges.

I could start itrackutil as root, but from a security standpoint that is not what I want.

This problem usually arises only, if the USB device is not (yet) managed by the system, because part of that managing process is... changing the permissions.

The solution I'm suggesting here works with Ubuntu Karmic.

The first step is to create a group named usbusers (or any other name), then make yourself member of that group and instruct udev to set the device group id to usbusers and set the permissions accordingly.

On the command line, creating a group and adding the user (e.g. mike) is quite simple:

sudo groupadd usbusers

sudo adduser mike usbusers

If you pefer the Gnome GUI, you find the appropriate program under "System - Administration - User and groups":

After unlocking the panel ("Click to make changes"), click onto "Manage Groups".

"Add Group".

Make enter "usbusers" as group name, make sure that the group id suggested by the system is not zero. It's usually in the 1000+ range, enable the "Group Members" and click "Ok".

As a last step you have to create a new udev rule:

SUBSYSTEM=="usb", ENV{DEVTYPE}=="usb_device", MODE="0664", GROUP="usbusers"

Save it under the following name: /etc/udev/rules.d/45-xaiox.rules

Then reboot.

Two remarks regarding the filename:

• in order to be executed, the filename must end with .rules (all other files are ignored)
• the 45 ensures that the file is executed before the corresponding "old" rule found in /lib/udev/rules.d/50-udev-default.rules

If you now plug-in your usb device, the permissions look like this:

$ ls -l /dev/bus/usb/002/total 0crw-rw-r-- 1 root usbusers 189, 128 2009-12-25 20:20 001crw-rw-r-- 1 root usbusers 189, 133 2009-12-25 22:22 006

And being a member of usbusers, you have read and write access.

itrackutil 0.1 veröffentlicht

Seit einigen Wochen bin ich Besitzer eines XAiOX iTrackU GPS Loggers.

Das Gerät verwendet den SirfIII Chipsatz, besitzt Bluetooth und kann bis zu 250.000 Wegpunkte speichern, das sind, wenn man einen Punkt pro Sekunde aufzeichnet, fast drei Tage.

So lange hält der eingebaute Akku zwar nicht, aber die - laut Prospekt - immerhin 17 Stunden sind reichlich bemessen.

Außerdem hat er eine Sprachansage, zur Quittierung verschiedener Betriebszustände (Bluetooth ein/aus, Satellitensuche, Fix).

Die Sprachansage ist wohl eher dem (bis auf die Firmware) baugleichen Gerät mit dem Namen TrapScout zu verdanken, das als "Speed Camera Detector" per USB eine Datenbank mit Koordinaten programmiert bekommt und dann an den "Points of Interest" die Ansage "Sie fahren zu schnell." ausgibt.

Für OSM-Tracker ist natürlich die erste Variante interessanter.

Und zunächst sah es so aus, als hätte ich nach längerer Zeit ein Gerät gefunden, für das es unter Linux keine Software gab. Aber auch hier war mir jemand zuvorgekommen. Harmut Schimmel hatte bereits vor einiger Zeit ein paar Perl-Skripte geschrieben, mit denen das Gerät von der Kommandozeile aus programmiert und ausgelesen werden konnte und diese auf seiner Site veröffentlicht.

Ein paar offene Fragen gab es dann aber doch noch. Die Ergebnisse von USB Sniffer und ein wenig Knobelei habe ich dokumentiert und sie sind in meine Utility itrackutil, das auf SourceForge in der Version 0.1 veröffentlicht worden ist, eingeflossen.

Viele Stellen in den vom Logger gesendeten Daten sind noch unklar, und können nur durch Vergleich mit anderen XAiOX iTrackU mit SirfIII Chipsatz vielleicht weiter entschlüsselt werden.

Außerdem berichtet Hartmut, dass weitere Geräte "kompatibel" sind. Tester mit diesen Geräten suche ich auch:

• XAiOX iTrackU mit Nemerix Chipsatz
• Gisteq PhotoTrackr
• Gisteq PhotoTrackr Lite (DPL700)

Das Programm läuft zurzeit unter Linux (Ubuntu Karmic), stellt aber von den weiteren Abhängigkeiten keine allzu großen Hürden auf.

Und obwohl diese Pakete auch unter Windows verfügbar sind, haben erste Versuche gezeigt, dass die zur USB-Kommunikation eingesetzte Bibliothek libusb dann doch nicht 100%ig ihrem Linux-Pendant entspricht.

USB Sniffing unter Linux

USB Sniffing – also das „Belauschen“ der USB-Kommunikation zwischen Computer und Endgerät – wird gerne eingesetzt, wenn für ein Gerät zwar ein Windows-Treiber aber kein Linux-Treiber vorhanden ist. Man versucht dabei herauszubekommen, welche Aktionen in der Windows-Anwendung welche USB-Kommunikation nach sich zieht, um dies dann in einer Linux-Anwendung nachzubilden.

Hierzu wird normalerweise ein Treiber im Windows-System installiert, der die Daten abfängt und protokolliert, bevor Sie an die USB-Schnittstelle weitergeleitet werden.

Auch unter Linux gibt es eine derartige Einrichtung. Sie nennt sich:

usbmon

Das Aktivieren ist unter Ubuntu Jaunty denkbar einfach:

sudo mount -t debugfs none_debugs /sys/kernel/debug

Unter Hardy ist usbmon noch ein separates Kernelmodul, das zusätzlich mit

sudo modprobe usbmon

eingebunden werden muss.

Danach erscheinen unter /sys/kernel/debug/usbmon diverse Dateien. Zum Beispiel:

$ls /sys/kernel/debug/usbmon

 0s 0t 0u 1s 1t 1u 2s 2t 2u 3s 3t 3u

Die Zahl im Namen bezeichnet die Nummer des USB-Busses (hier 1, 2 und 3). Der virtuelle „Bus 0“ liefert die Events aller Busse. Der Buchstabe hinter der Zahl (s, t, u) beschreibt, was ausgegeben werden soll:

u – der mitgeschnittene Datenverkehr
s – Statusmeldungen
t – eine Teilmenge von „u“ (dieser Eintrag wird in absehbarer Zeit nicht mehr unterstützt).

Was bleibt, ist diese Datei auszugeben und/oder in eine Datei umzuleiten, z.B.:

cat /sys/kernel/debug/usbmon/0u > usbmon.txt

Wireshark

Ganz ohne dieses Modul kommt Wireshark aus. Mit „Capture – Interfaces“ kann auch ein USB-Bus ausgewählt werden. Hierzu sind natürlich root-Rechte notwendig.

Wireshark stellt die USB-Kommunikation in ähnlicher Weise wie sonst den Netzwerkverkehr dar. Außerdem werden die diversen Konstanten und Strukturen, wie bei Wireshark üblich, schon dekodiert und mit Klartext ausgegeben.

Was bringt's?

Virtuelle Maschinen (z.B. VirtualBox, Vmware) haben unterdessen die Möglichkeit, USB-Geräte im virtualisierten System laufen zu lassen, die über das Gast-System auf die USB-Schnittstelle zugreifen.

Kombiniert man diese beiden Möglichkeiten, so lassen sich z.B. Geräte in einer virtualisierten Windows-Umgebung betreiben, und „von außen“ belauschen.