How-to abcde from an audio CD image

abcde is a command line CD encoder.  It rips CDs and encodes it in MP3, OGG or other formats.

By default it reads from your CD drive.  If you only have an image of your CD you are out of luck. You can either first burn the image on let abcde work on the CD, or use the only other format abcde currently accepts as input: a flac file with embedded cue sheet.

There are two common images files for audio CDs: cue/bin and toc/bin

The bin file contains the digital representation of the audio, whereas the remaining file describes where the tracks start and end.

cue/bin

You can create the flac file from cue/bin files using the following command:

flac --best --force-raw --sample-rate=44100 --channels=2 --bps=16 --endian=big --sign=signed --cuesheet=image.cue image.bin -o image.flac

Check the compression ratio after flac has finished.  A ratio of approx. 0.99 usually indicates that the byte order is reversed.  This may happen if the image was created on a Mac. If you play back the flac file, you will hear mostly noise. In this case change the byte order to

--endian=little

Normal compression ratios are in the range of 0.6 to 0.7.

You can then convert the flac file to single tagged MP3 files using:

abcde -d image.flac

toc/bin

flac can't process TOC files. You have to convert them into the CUE format. Fortunately there is an app for that: cueconvert from the cuetools package:

cueconvert image.toc image.cue

I had to find out that cueconvert will abort with a syntax error when I used toc files created by Brasero.

In this case load the toc file into a text editor and remove the CD_TEXT { … } block, if present.

And while you're at it, delete all lines containing ISRC codes, as flac does not like these.

After the conversion use the resulting CUE file as described earlier.

Replay Gain on Sansa Clip+ with mp3gain

If you play MP3 songs from different sources in shuffle mode on a MP3 player, their difference in loudness become apparent and you are constantly keep adjusting the volume.

In order to prevent that, Replay Gain can be used. With this function the MP3 player uses loudness information in the header of a MP3 file to adjust the volume automatically.

However, the loudness information is not present by default. You need software to analyse the content and store those values in the MP3 header. Replay Gain does not change the underlying (music) data, thus avoiding loss in quality by decoding, processing and re-encoding.

A popular utility in the Linux world to calculate these values and tagging the file is mp3gain.

I used it regularly, but I was still adjusting the volume on my Sansa Clip+ ...

What I didn't know was, that by default mp3gain stores the information in the APEv2 tag which the Sansa Clip does not read. In order to be used by the Sansa Clip's Replay Gain function, it must be stored in the ID3v2 tag.

mp3gain has an option to do exactly that: -s i

I'm now using the following command to analyse and tag my MP3 files:

mp3gain -p -s i *

Where the option -p also preserves the original file date.

Lift and AutoComplete

The Lift-Web-Framework allows you easy access to JQuery's AutoComplete widget. If you have used Google, you might agree that the drop down list which is populated with entries depending on your current input is a nice thing to have.

Here is how you integrate it into your Lift application.

First you have to add a dependency to your project (in this case Lift 2.2, using Scala 2.8.0):

If you use SBT in ./project/build/Project.scala

override def libraryDependencies = Set(
...
"net.liftweb" % "lift-widgets_2.8.0" % “2.2” % "compile->default",

In a Maven project this should translate to (in pom.xls - not tested, I'm using SBT):

<dependency>
  <groupid>net.liftweb</groupid>
  <artifactid>lift-widgets</artifactid>
  <version>2.2-scala280</version>
</dependency>

The first step in your application is the initialisation, by adding the following code – usually into into Boot.scala

import _root_.net.liftweb.widgets.autocomplete._

class Boot {
def boot {
…
AutoComplete.init

In your template, the widget is represented like any other input field with a tag of your choice. This example uses

Here the snippet code to bind it:

Helpers.bind("dt", in,
...
"autocomplete" -> AutoComplete(startValue,
buildQuery _,
value => processResult(value)
),
...

where:
startValue is a string for the initial value of the AutoComplete text field.
buildQuery is a function to populate the drop down list.
processResult is a function called when the form is submitted, containing the value of the widget (or not, see bug report below).

Here is a simple example of processResult

def processResult(s : String) = {
println("%s".format(s))
}

The interesting part is the buildQuery function. It has the following signature:

def buildQuery(current: String, limit: Int): Seq[String] = {
...
}

where
current is the string typed into the autoComplete text field.
limit is an integer to limit the number of Strings returned by the function

A valid but not very useful implementation would be (from http://demo.liftweb.net/ajax)

def buildQuery(current: String, limit: Int): Seq[String] = {
(1 to limit).map(n => current+""+n)
}

Most application however will query a database. Lets assume that Mydatabase has a field called name (e.g. with the names of employees). A query might look like this:

def buildQuery(current: String, limit: Int): Seq[String] = {
Mydatabase.findAll(
Like(Mydatabase.name,(current + "%")),
OrderBy(Mydatabase.name,Ascending),
MaxRows(limit)
).map( _.name.is)
}

This will findAll names in Mydatabase which start with the currently entered string, sorts them, and limit the number of results.

That's all you need.

Bug or a feature?

The current implementation of the AutoComplete functionality (Lift 2.2) only allows the selection from a drop down list as a valid input. This sounds reasonable, but it also prevents you from clearing the input after you have made a selection. This can lead to the following situation:

At a shopping site, the user selects an article via an autocomplete field. Then he changes his mind and clears the input field using the delete key. When he clicks on the submit button the previous selected article is sent to the server, even if the screen shows an empty text field.

With the help of Sergey Andreev, I have developed a small patch which has also been published in Lift ticket 892. Even though it is only 8 lines long, it is unlikely to make it into Lift due to Intellectual Property policy consideration by DPP.

The patched version is available on GitHub.

If you copy it into your project, you might want to change the package name (first and last three lines) and replace

import _root_.net.liftweb.widgets.autocomplete._

with your package name.

Keep in mind that the returned result may contain strings (including an empty string) that might be invalid.

itrackutil 0.1.3 available

After a recent update in Ubuntu 10.10 and 10.04 itrackutil.py exits with the the error message “USBError: Resource Busy” before the user is able to download GPS data from the device.
This version fixes the problem.

Developer information:

A previously unrelated kernel module (cdc_acm) suddenly started to grab the USB device.  cdc_acm  which usually supports USB modems now creates the device /dev/ttyACM0 as soon as the GPS mouse is plugged in.

If you read /dev/ttyACM0 you get the real-time GPS data in NEMA format.  This is an unusual use case. The normal data connection is over Bluetooth.

However, the creation of this device file blocks all other USB access to the GPS mouse; in this case the direct USB communication which itrackutil.py uses to access the data stored within the unit's memory.

Fortunately there is an USB API call for this situation: detachKernelDriver.
This function does, what it says it does.  You have to call it after opening the device. Root privileges are not necessary.

The call will fail, if there is no kernel driver to detach. You have to catch this exceptions:

... determine device, interface, and configuration ...

try:
    handle = device.open()
    handle.detachKernelDriver(interface)
except usb.USBError, err:
    if str(err).find('could not detach kernel driver from interface') >= 0:
        pass
    else:
        raise usb.USBError,err       # any other USB error

handle.setConfiguration(configuration)


The companion API call attachKernelDriver is not available via PyUSB 0.1. But this is only a minor problem, because as soon as you unplug the unit and reconnect it, a new USB device file is created (with the kernel driver attached).

WinTV-HVR-1900 under Ubuntu 10.04 and 10.10

A few weeks ago I bought a Hauppauge WinTV-HVR-1900 (USB id 2040:7300) which I wanted to use with a (32 bit) Ubuntu 10.04 and 10.10 system.
 
Quick installation summary:

• Does it work out of the box: No
• Does it work at all: Yes
• If you are able to update to Ubuntu 10.10, do it.

This post only describes how to get the basic functionality working (i.e. display/record a signal on the composite video input).
 
The HVR 1900 is a small box which is connected via a USB2 port (USB1 is not supported for bandwidth reasons). There are inputs for composite video (e.g. VCR or set-top box) an antenna input and a S-VHS input. The device comes with a UK power supply (with a UK mains plug) and a bulky mains plug adapter for continental Europe.
 
In order to get the digitizer running, this device needs firmware.  Ubuntu comes with a selection of firmware images for various Hauppauge systems, but none was suitable for this device.
 
The firmware is included on the Windows driver disk you get with the device, but which files do you need?  Fortunately there is a perl script available that scans the CD and extracts the files you need based on a bunch of MD5 sums stored in that script.  Perl is installed by default on a Ubuntu system, so slide in the CD, open a terminal and enter
 
perl fwextract.pl /media/XXXXXXXXXX
 
(where XXXXX is the CD title)
 
In my case the following files were found:

• v4l-cx2341x-enc.fw
• v4l-cx25840.fw
• v4l-pvrusb2-29xxx-01.fw
• v4l-pvrusb2-73xxx-01.fw

 
For the next steps you need root privileges:
 

• Change the ownership of those files to root (for security reasons)
  sudo chown root:root *fw 
• Copy the extracted files to /lib/firmware
  sudo cp *fw /lib/firmware
  

 If you're running Ubuntu 10.10, that's all you have to do.
 
Keep an eye on the system log when you now plug the digitizer into the USB port.
tail -f /var/log/syslog
 
Among other messages, it should confirm that the firmware was uploaded successfully.
 
...
cx25840 5-0044: firmware: requesting v4l-cx25840.fw
cx25840 5-0044: loaded v4l-cx25840.fw firmware (16382 bytes)
...
 
Utilities for controlling the device from the command line can be found in the package ivtv-utils (from the Ubuntu repo).
 
v4l2-ctl --set-input 1
 switches to the composite video input
 
v4l2-ctl --set-ctrl=brightness=128,hue=0,contrast=68,volume=54000
 sets basic parameters for the digitizer.
 
Call v4l2-ctl without parameters for more help, and you will definitively want to try the switch -L
 
Recording is as simple as:
cp /dev/video0 video.mp2
 
This works most of the time.  Approx. 5% of the recordings contain a distorted audio stream. This distortion is present for the whole length of the recording and usually ends the next time the device /dev/video0 is opened. 
 
If the audio is ok at the beginning, it stays that way.  This looks like an initialization problem when the device is opened.  I haven't found a fix, yet.
 
Now to the more difficult part:

Getting the device to work under Ubuntu 10.04.
 
As mentioned before, many Hauppauge devices need firmware, which is uploaded to the unit when you plug it into the USB port.  Older hardware only needed 8192 bytes of firmware.  The firmware for this device however is 16382 bytes long (see the above firmware upload message from the log).  The device driver controlling the HVR-1900 (pvrusb2) that comes with kernel 2.6.32 and earlier is only capable of transferring 8192 bytes. And Ubuntu 4.10 LTS uses... 2.6.32.
 
Newer versions of the pvrusb2 driver can also upload the larger firmware.  For older kernels (like the one used in Ubuntu 4.10), you have to compile the updated driver yourself.
 
Compiling a kernel is usually a simple task, because the kernel source code already contains all dependencies. But this time, there were complications.
 
You need:

•  the kernel source
•  the tools to compile the kernel
  
•  and the source of the updated driver

The easiest way to get the Ubuntu Linux source is by installing a package named "linux-source". It will store the source code as an archive in /usr/src/linux-source-2.6.32.tar.bz2 
 
You have to unpack it - make sure that you have plenty of disk space available:
 
cd /usr/src
tar xvfj linux-source-2.6.32.tar.bz2
 
Then run the following commands
 
cd linux-source-2.6.32
make oldconfig
make prepare
 
This will "regenerate" the .config file used by your distribution's kernel.  This file is needed during the kernel compilation.
 
Now we have to download the source code of the current pvrusb2 driver which can be found here. Unpack it and copy the content of the directory driver to /usr/src/linux-source-2.6.32/drivers/media/video/pvrusb2/ overriding the current files with the same name.
 
(Please note: the pvrusb2 documentation is describing a different approach, that did not work for me (modprobe rejected the module))
 
The next step would be:
 
make modules
 
But due to a totally unrelated bug  the compilation will fail, while trying to compile the "omnibook" files.
 
Download the patch for this bug from here and apply it
 
cd /usr/src
patch -p0 _name_of_the_path_file_
 
Now it's time to compile the modules:
 
cd /usr/src/linux-source-2.6.32
make modules
 
This step is very time consuming. If you have a multi core processor use the -j# option (where # is the number of cores you have).
 
Copy the new module from
/usr/src/linux-source-2.6.32/drivers/media/video/pvrusb2/pvrusb2.ko
to
/lib/modules/`uname -r`/kernel/drivers/media/video/pvrusb2/pvrusb2.ko
 
(where `uname -r` (backticks!) will be replaced by the name of your current kernel)
 
Keep in mind that you have to repeat that process after each kernel update.
 
After the next reboot the new module should be active. If you can't wait, unload the old one and load the new module manually:
 
rmmod pvrusb2
modprobe pvrusb2
 
Again, check /var/log/syslog for any problems.
 
Links:

• http://www.isely.net/pvrusb2
• http://www.isely.net/pipermail/pvrusb2/
• https://help.ubuntu.com/community/Kernel/Compile
• http://www.isely.net/downloads/fwextract.pl

ColorCube on a 64 bit machine

After a recent interview with Quentin Bolsee, the lead developer of the game ColorCube, I  downloaded the demo version of his game.  ColorCube is a simple nice puzzle game that can keep you occupied for hours.

But I had to find out that the program did not start on my main machine.

The command

ldd Launcher

quickly revealed that it a 32 bit application, and various libraries were missing on my 64 bit system.

The package ia32-libs which provide 32 libraries for 64 bit systems does understandably not contain all libraries that may be installed on a 32 bit system.  A simple work-around is to copy these files from a 32 bit set-up (/usr/lib) to the 64 bit system (/usr/lib32).

Don't forget to set the file owner to root on the target system., and keep in mind that those libraries will not be updated.

In order go get ColorCube running, I needed to copy the following libraries:

libHalf.so.6
libIlmImf.so.6
libIlmThread.so.6
libspeex.so.1
libtheora.so.0
libIex.so.6
libImath.so.6
libraw1394.so.11
libusb-1.0.so.0

Happy gaming...

Lift + Scala, Installation + First contact

I have installed Lift and Scala on a Linux computer running the current Ubuntu version 10.04. Depending on your distribution, the commands may be different.

What you need is:

• maven2 - a Java software project management and comprehension tool
• jetty - a Java servlet engine
• Java
  

Note that this is not the only possible configuration. There are other project management tools like SBT and other servlet engines, e.g. Tomcat. However, the documentation I've seen mostly describes maven and jetty.

Installation of Scala

Please note: The following step will install Scala. It also lets you run Scala commands interactively from the command line. This is a very good learning tool, especially for beginners.

This is a modified version of the instructions given by the London Scala User Group Dojo:

• Warning: Do not use Scala from the Ubuntu (10.04) repositories. The repo contains an older 2.7 variant - we want 2.8!
• Create a folder that will hold one or more versions of Scala:
  sudo mkdir /opt/scala
• Change that folders ownership to your username:
  sudo chown -R username.username /opt/scala
• From http://www.scala-lang.org/downloads download Scala final for Unix, Mac OS X, Cygwin. This is a gziped tar archive file.
• Open the dowloaded file (Archive manager - file-roller) and extract to /opt/scala/
• Create a sybolic link (future proof the install):
  ln -s /opt/scala/scala-2.8.0.final /opt/scala/current
• Define environment variable SCALA_HOME by editing ~/.bashrc and adding the line:
  export SCALA_HOME=/opt/scala/current
• Add the Scala executables to your path by editing ~/.bashrc and adding the line:
  export PATH=$PATH:$SCALA_HOME/bin
• Open a new terminal or load the changes in your ~/.bashrc file using the command:
  source ~/.bashrc
• Test scala is working using the following command:
  scala -version or scalac -version
  You should see the following as output:
  Scala code runner version 2.8.0.final -- Copyright 2002-2010, LAMP/EPFL
  Scala compiler version 2.8.0.final -- Copyright 2002-2010, LAMP/EPFL
  

The version of Scala used for compiling the web app is stored in the maven repos and is at the moment (Juli 2010) at version 2.8.0.RC7. It will be downloaded and executed during the build process.

Installation of Java

Ubuntu comes with OpenJDK as standard. You might want to download the official Java from Sun... I mean Oracle:

• Enable the partner repositories in Synaptic (or uncomment them in /etc/apt/sources.list) and reload the indices:
  sudo apt-get update
• Download Java
  sudo apt-get install sun-java6-jdk sun-java6-plugin
• Switch to the Oracle version:
  sudo update-alternatives --config java
  sudo update-alternatives --config javac
  

Installation of maven2 and jetty

sudo apt-get install maven2 jetty

Your first lift application:

Go into an empty directory an issue the following maven2 command:

mvn archetype:generate \
-DarchetypeGroupId=net.liftweb \
-DarchetypeArtifactId=lift-archetype-blank \
-DarchetypeVersion=2.0-scala280-SNAPSHOT \
-DarchetypeRepository=http://scala-tools.org/repo-snapshots \
-DremoteRepositories=http://scala-tools.org/repo-snapshots \
-DgroupId=com.example  \
-DartifactId=lift.test \
-Dversion=0.1 \
-DscalaVersion=2.8.0.RC7 

An archetype is a project template. lift-archetype-blank is a simple example Lift application, which is downloaded from http://scala-tools.org/repo-snapshots. You can browse this repository at http://scala-tools.org/repo-snapshots/net/liftweb
If you use http://scala-tools.org/repo-releases instead you get older more stable builds. repo-snapshots contains the nightly builds.

• The artifactId (lift.test) is the name of the top level directory for this project (your choice).
• The groupId is mapped (Java style: com.example becomes com/example) to the ./src/main/scala directory (also your choice).
• version refers to the version of YOUR software.
• scalaVersion of Scala in the maven repo.
  

After the download the project tree looks like this

This might look scary at first. But there are only 10 files.

./pom.xml
./src/main/scala/bootstrap/liftweb/Boot.scala
./src/main/scala/com/example/snippet/HelloWorld.scala
./src/main/webapp/index.html
./src/main/webapp/templates-hidden/default.html

./src/main/webapp/WEB-INF/web.xml
./src/packageLinkDefs.properties
./src/test/scala/com/example/AppTest.scala
./src/test/scala/LiftConsole.scala
./src/test/scala/RunWebApp.scala 

I'm not sure yet about the last group. My guess is they are used for unit testing and packaging of the WAR files for later deployment.

As for the first group:

• pom.xml:
  This is the maven configuration file.
  It contains (among others) the URL of the repositories and the dependencies which have to be resolved during the build process. You will rarely edit this file. However, if you need modules that are not within the “standard” scope (e.g. a connector to MySQL servers), you have to edit this file in order to get them drawn into your project.
• Boot.scala (in ./src/main/scala/bootstrap/liftweb/):
  This file contains code that is run once when the application is started.
  It is used to set-up menus, database connections, etc. You will have to edit this Scala code frequently when you want to use and configure these features.
• index.html (in ./src/main/webapp/):
  The default template
  This is the template that is evaluated when you access it via the application server without any special file name (e.g. http://localhost:8080/). You will most likely create other templates.
  This particular template refers back to
  
• default.html (in ./src/main/webapp/templates-hidden/default.html).
  This is also a template, but one that you will rarely change. It defines the “frame” in which your application runs, i.e. header, footer, menu bar, the html HEAD section (page title, meta tags, CSS definitions, etc.).
  Not all of its content is static. This file may contain Lift tags, most commonly the ones for displaying the menu, to change the page title, and to display feedback to the user (error messages, notices, warnings).
  You may create and use more of these “frame” templates, but in most cases you only need one.
• HelloWorld.scala (in ./src/main/scala/com/example/snippet/):
  a so called snippet.
  These files contain the Scala code which fills the in the templates (e.g. index.html). This is where most of your coding will happen. You will create more than one snippet file.
  

Some folders are currently empty:
model, view, comet (in ./src/main/scala/com/example/).

In the example code I have seen so far, Scala code that defines persistent data structures (e.g. for a database) is stored here.

Bringing your application to life

In order to start the application change into the top directory (lift.web) and type

mvn jetty:run

After maven has finished downloading all dependencies and building the application, you can access the web application in your browser via http://localhost:8080