Learnosity Logo
Learnosity Banner Image

Remote collection of vmstat log files

We've been doing a lot of load testing recently and are planning on doing a lot more, so we've developed a simple little script to make the collection of the vmstat (performance statistics) a bit quicker.

multivmstat is a php command line script which makes it easy to collect the statistics. Download the script

To run it you specify a list of servers to check:

./multivmstat server1,server2,server3 5

This will run vmstat on each of the servers with a 5 second interval between samples.

It will create the following files:

server1-vmstat.log
server2-vmstat.log
server3-vmstat.log

Note: You mush have ssh access to each machine - ideally with a certificate so that no interactive authentication is required.

Download the script

Ubuntu Server upgrade howto

One of the nice things about Ubuntu is how easy it makes it to upgrade between versions - typically start up update-manager and it will prompt you if there is a new version available and handle all the details of the upgrade for you. I've done numerous upgrades this way and never had any issues.

However, I have a number of ubuntu servers which I manage and previously I have upgraded them by changing the /etc/apt/sources.list and then running "sudo apt-get dist-upgrade"

Today I discovered that there is an automated script to handle it - which I presume is what powers the gui version behind the scenes.

You need to make sure you have update-manager-core installed:

sudo apt-get install update-manager-core

Then you simply run do-release-upgrade

sudo do-release-upgrade

I've just upgraded two different servers from Feisty as it has recently stopped being supported. I upgraded them to Gutsy and then to Hardy which is a Long Term Supported version. The whole process took about an hour of which it took about 5 minutes of my attention to answer a few simple questions.

Who said Linux is hard :-) Cheers, Mark

Choppy sound on Twinkle Softphone on Ubuntu Linux - Solved

I just reinstalled my system with the latest Ubuntu 8.10 x64 and one of the apps I use regularly for testing and developing Asterisk applications is Twinkle, which is a great VOIP softphone.

It works great, but when I tried it in my new install the sound was all choppy, i.e. I would get every half of each second of the audio.

A bit of fiddling about with the settings and I got it solved. Posting it here so when I forget about it and run into the same problem again I'll know how to fix it.

The solutions is as simple as:

  • From the menu select Edit -> System Settings.
  • Select Audio on the left
  • Change the audio devices to use the most approriate device that is NOT "ALSA: default: Default Device" - in my case it was "ALSA: plughw:0,0: HDA Intel (AD193x Analog)"

I do like the very friendly naming scheme for devices :-) Hope it helps, Mark

Loading MySQL timezone info on Linux and Macs

MySQL has some really useful functions that make timezone support a bit easier as described in a previous blog post..

However, you need to do a tiny bit of setup work before you can use the functions. You need to load the timezone info from your system into the mysql db.

Thankfully, the MySQL Developers have included a very handy script that will do just that from a zoneinfo files which are used on Linux and on Mac's.

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql

Obviously change the mysql params or add a -p if you have a password or are connecting to a remote server.

Check out the MySQL developer site for full details on MySQL timezone support

Cheers, Mark

Change timezone on Ubuntu server (CLI only)

It should be simple to do this and it is. The hardest part was finding the correct way to do it, so for future reference:

Using the command line, you can use dpkg-reconfigure tzdata.

sudo dpkg-reconfigure tzdata

Follow the instructions to reconfigure the timezone. This was taken from the ubuntu wiki article

Cheers, Mark

Custom Language files for Asterisk on Ubuntu

There are a number of steps needed to get custom language files working in a nice fashion for asterisk on ubuntu.

For this example I'm going to replace the standard digits of 0 to 10 with files that I've had recorded.

Create files in the correct format:

Starting with files in the format: 0.wav, 1.wav .... 10.wav which are 44.1kHz 16 bit PCM mono wav files.

We can convert them to alaw, ulaw and gsm formats which reduces asterisk transcoding with the following script: (requires sox to be installed)

#!/bin/sh
#
# Script to batch convert wav files to gsm,alaw and ulaw formats
# to help reduce overhead for asterisk server
# Run it in a directory of wav files.

for a in *.wav; do
   echo "Processing $a";
   baseName=`echo $a|sed -e s/.wav//`;
   #Convert to 8bit wav first at 8000Hz - this can take a while
   sox "$a" -r 8000 -c1 "$baseName_temp.wav" resample -ql;

   echo "...additional formats";
   #Convert to gsm
   sox "$baseName_temp.wav" "$baseName.gsm";

   #convert to alaw - note using .alaw extension as asterisk likes that
   sox "$baseName_temp.wav" -t al "$baseName.alaw";
   #convert to ulaw - note using .ulaw extension as asterisk likes that
   sox "$baseName_temp.wav" -t ul "$baseName.ulaw";

   #clean up the .wav files
   rm "$baseName_temp.wav"
done

Move the files to the correct place on the server:

I'm going to use the name 'learnosity' for the language as these are the files that we've recorded. So we need to create the following directory on the asterisk machine:

/usr/share/asterisk/sounds/digits/learnosity

If you wanted to do more audio files, you can create learnosity directories for the other audio files.

Once this is done, copy the alaw, ulaw and gsm files in but leave the original wav files out.

Tell asterisk to use these files:

The easiest way to use these language files is to set the channel language for the sections you want to use it:

Here's an example from extensions.conf:

exten => 4,1,Answer()
exten => 4,n,SayDigits(123456789)
exten => 4,n,Set(CHANNEL(language)=learnosity)
exten => 4,n,SayDigits(123456789)
exten => 4,n,Hangup()

If you dial extension 4 you get the digits 1 to 9 in the default language and if you've done everything correctly you will get your newly recorded digits the second time.

Cheers, Mark

Howto refresh /dev/disk/by-uuid on Ubuntu

I was recently setting up a Ubuntu server and was partitioning it after it had been installed.

There was lots of free space on the drives as the root partition was only using a small portion of the disk. After running fdisk to partition it and mkfs.ext3 on the partitions to format them I couldn't see them in /dev/disk/by-uuid.

A quick google presented the solution:

sudo udevtrigger

According to the man page this makes udev request the kernel device uevents, which in essence makes it read the disk info again and show it all up so you can mount it happily.

Cheers, Mark

CF Sandbox Security Tricks and Tips

I've been working on getting CF Sandbox security working. It's trickier than I first thought so here's how to do it for future reference:

Enable security Manager

Instructions from Steven Erat's blog:
Stop ColdFusion.
Locate the jvm.config file in jrun_root/bin.
Back up the file.
Open the file in a text editor.
Add the following lines to the java.args section:
-Djava.security.manager
"-Djava.security.policy=[cf_webapp_root]/WEB-INF/cfusion/lib/coldfusion.policy"
"-Djava.security.auth.policy=[cf_webapp_root]/WEB-INF/cfusion/lib/neo_jaas.policy"

NOTE the example from the adobe site has the quotes in the wrong place. Note you also need to change [cf_webapp_root] to match the location on your machine.

Datasources

After I enabled Datasource security I began to receive the following error, a bit of digging reminded me that the MySQL connector tries to do some autoconfiguration for coldfusion to optimise it.

Message: Can't find configuration template named 'coldFusion'
Type: java.sql.SQLException

It was failing to load the configuration file which is inside the mysql connector jar file /com/mysql/jdbc/configs/coldFusion.properties

I haven't gotten to the bottom of why it couldn't be loaded but adding the following to the datasource query string fixed it up:

autoConfigureForColdFusion=false

I would suggest adding some of the settings from this file as paramaters in your datasource settings as per previous post

Note: This was using the following:

  • ColdFusion 7.0.2 Cumulative Hotfix 1 Multiserver install
  • MySQL connector/J 5.0.8

Hope it helps. Cheers, Mark

Extracting from sound from Flash (aka NellyMoser)

A current project that we are working on requires us to be able to record sound via a flash plugin. Initially I thought it would be nice and easy as I've seen demos of both video and audio broadcasting - however, the one big problem is that the current flash client only allows you to record video to a netstream, you can't get any access to it in the flash player.

So you need something like Flash Media Server or Red5 to record it.

However, once you record it to the server it is an flv and the audio codec is stored as a NellyMoser encoded audio portion. This codec is not supported by many applications and after a quick google found the nelly2pcm project on google code.

This will convert a flv sound file to a raw pcm file - which you can then do useful stuff with. So here's how to do it on a Ubuntu machine.

$ tar -xjvf nelly2pcm.tar.bz2
$ cd nelly2pcm
$ make
cc -Wall -c -o nelly2pcm.o nelly2pcm.c
cc -Wall -c -o nelly.o nelly.c
cc -Wall -c -o nelly_tables.o nelly_tables.c
cc -Wall nelly2pcm.o nelly.o nelly_tables.o -lm -o nelly2pcm
You should now have a nelly2pcm executable file in this directory.

You can run it as follows:

$ ./nelly2pcm test.flv > test.raw
mono Nellymoser stream with 16-bit samples at 44kHz

This will create a raw sound file with no headers and output the line above which you'll need for the next part.

To play this file back you can use sox (apt-get install sox)

$ play -r 44100 -c 1 -2 -s test.raw

If you've got all the options correct then this will play the file back. Once you got it correct you can then use sox to create a wav file which is essentially the same except that it has a header which contains all the settings (eg bitrate etc)

$ sox -r 44100 -c 1 -2 -s test.raw test.wav

From a wav you can convert to whatever you like. I'm looking forward to the Flash Player 10 release as this messing will no longer be necessary as it will support encoding with the free Speex codec.

Cheers, Mark

Update:

In the 12 hours since I wrote this post it appears that a DMCA takedown notice has been served on the nelly2pcm site and google have taken it down. I've no idea if it's related to this post or not and no details are available yet on the chillingeffects site but hopefully it will be updated shortly. As I mentioned earlier, I can't wait for flash 10 with the speex codec so we don't have to use Nellymoser, well done Adobe for including it.

Convert Mac Line endings to Linux

I was cleaning up some source code yesterday and wanted to print it all out, however the Mac line endings (we use Mac's and Ubuntu mostly) were messing up the printout on my Ubuntu machine.

A quick google later and I found the solution, a program called 'tr', posting here for next time:

tr "\r" "\n" < filewithmaclineendings.txt > filewithunixlineendings.txt

Obviously you could convert from Unix to Mac by reversing it.

tr "\n" "\r" < filewithunixlineendings.txt > filewithmaclineendings.txt

Cheers, Mark

More Entries