We've been doing a lot of load testing recently and are planning on doing a lot more, so we've developed a simple little script to make the collection of the vmstat (performance statistics) a bit quicker.

multivmstat is a php command line script which makes it easy to collect the statistics. Download the script

To run it you specify a list of servers to check:

This will run vmstat on each of the servers with a 5 second interval between samples.

It will create the following files:

Note: You mush have ssh access to each machine - ideally with a certificate so that no interactive authentication is required.

Download the script

One of the nice things about Ubuntu is how easy it makes it to upgrade between versions - typically start up update-manager and it will prompt you if there is a new version available and handle all the details of the upgrade for you. I've done numerous upgrades this way and never had any issues.

However, I have a number of ubuntu servers which I manage and previously I have upgraded them by changing the /etc/apt/sources.list and then running "sudo apt-get dist-upgrade"

Today I discovered that there is an automated script to handle it - which I presume is what powers the gui version behind the scenes.

You need to make sure you have update-manager-core installed:

Then you simply run do-release-upgrade

I've just upgraded two different servers from Feisty as it has recently stopped being supported. I upgraded them to Gutsy and then to Hardy which is a Long Term Supported version. The whole process took about an hour of which it took about 5 minutes of my attention to answer a few simple questions.

Who said Linux is hard :-) Cheers, Mark

It should be simple to do this and it is. The hardest part was finding the correct way to do it, so for future reference:

Using the command line, you can use dpkg-reconfigure tzdata.

Follow the instructions to reconfigure the timezone. This was taken from the ubuntu wiki article

Cheers, Mark

I was cleaning up some source code yesterday and wanted to print it all out, however the Mac line endings (we use Mac's and Ubuntu mostly) were messing up the printout on my Ubuntu machine.

A quick google later and I found the solution, a program called 'tr', posting here for next time:

Obviously you could convert from Unix to Mac by reversing it.

Cheers, Mark

The Problem

We've been working on an application which needed significant integration with an external system which is done via web services.

During out load testing we came across a major issue with the web services when the application was under load which caused the ColdFusion application server to crash very badly.

The problem was caused by the fact that the web service calls typically took about 1 second to complete whereas pages that didn't need to use a web-service completed in about 100ms.

When we put the application under load if the number of requests that needed web-services ran at a sustained high rate then very soon all the running requests would be doing web service calls. This meant that all 25 java threads were getting swallowed up the "long running" web-service calls.

This caused all the other threads to queue up and very shortly the application fell over in a heap.

The Solution

Initially we looked a using cflock tags to handle it but this would essentially serialize all web service requests and only one web-service thread would run at any one time. This meant that the application would not be able to handle the required load.

After a bit of a brainstorming session we came up with the idea to develop a semaphore type object which would limit the amount of threads that could get tied up with long running web-service requests.

Ideally it works like this:

• A web-service request comes in
• It requests a thread from the Semphore object
• If it gets one it runs the Web Service and then releases the thread

However, if the system is busy it works like this:

• A web-service request comes in
• It requests a thread from the Semphore object
• It doesn't get a thread so doesn't run the web service and quickly returns and lets the user know the app is busy

Usage

In order to use the semaphore you need to set it up in a shared scope - typically application or server scope. We use it in our webservice wrapper CFC which is stored in the application scope.

Here is a code example of how this looks in practice:

Note: One of the issues that we came across during the implementation of this was that when long running requests were caught by the long running request timeout the releaseThread function would not get called as the webservice code was timing out and it was getting killed before it hit the releaseThread line. To work around this we implemented an internal garbage collection mechanism to ensure that we didn't leak threads, or when we did we could recover from it.

This has allowed us to manage the number thread by limiting the number of long running requests so as the server gets busy it will now reject web service calls before it runs out of memory.

You can get the full code for the Semaphore object from our Opensource CF Library

We've been performing some load testing on a new website we've developed and our helpful sysadmin noticed lots of queries happening on the DB that looked like this:

We weren't intentionally doing these queries but they were coming from somewhere. A bit of detective work via google found a very enlightening article about this problem.

A quick read of this explained:

It turns out ColdFusion was asking Connector/J for the metadata on every field, which in turn triggered a SHOW FULL COLUMNS query for every varchar and text column returned.

It also went on to say that is had been fixed as of Connector/J 5.07.

On reading the release notes it mentions that the

Driver detects when it is running in a ColdFusion MX server (tested with version 7), and uses the configuration bundle coldFusion, which sets useDynamicCharsetInfo to false (see previous entry), and sets useLocalSessionState and autoReconnect to true.

However, from my testing it wasn't doing it as we were still seeing the problem. However, we are running the MultiServer version of ColdFusion which sits atop Jrun4.

So I added the parameter directly to the querystring:

So it now looks like:

This has stopped the problem, and given a significant performance gain. On one of the sites I was load testing it gave a 12% throughput increase and on another one which had less queries it gave a 4% increase.

Hope it helps.

Cheers, Mark

While chatting with my friendly local sysadmin the other day about monitoring multiple log files across our cluster of server he pointed me towards multitail.

With a little bit messing about I managed to get it doing exactly what I needed.

I created the following little script which connects to 4 remote servers and monitors the log files (file names and server names have been changed to protect the innocent).

Now with a single command I can monitor the log files across the entire cluster.

Happy days. Cheers, Mark

I've just completed setting up a new ColdFusion server on Ubuntu Server (I know it's not a supported config) and ran into the following problem when I went to the admin page:

A bit of googling turned up this Technote from Adobe which mentioned the same problem on RedHat Linux.

The redhat solution is also documented on Talking Tree but it doesn't cover Ubuntu.

For Ubuntu it's very simple - you need to install 3 libraries and you are good to go - one single line.

This will install a dozen or so packages - then restart ColdFusion and you have a shiny working install.

Cheers, Mark Lynch

One of the things that Ubuntu (gutsy 7.10) and probably most versions of linux) are currently lacking is out of the box support for power saving and hence suffers from poor battery life when unplugged.

My HP nc8430 also runs particularly hot by default - but I've spent a bit of time hunting down tips for reducing the power usage and heat produced from the laptop to get a longer battery life. I'm sure there are a lot more ways that power can be saved and I suspect that future versions of Ubuntu will catch up and produce the same level of battery life or even longer than windows.

Here is what I am using so far:

Set powerplay mode on ATI fglrx driver

If anyone has equivalent code for other video cards please let me know and I'll add it here.

Disable 'hal' from polling your cdrom

From powertop: 'hal' is the component that auto-opens a window if you plug in a CD but disables SATA power saving from kicking in.

Disable bluetooth if you are not using it

Remove the usb 1.1 kernel driver if not using it

Apparently the usb 1.1 driver does frequent polling and prevents the processor from staying in low power states for any length of time.

Increase the VM dirty writeback time

From powertop: This wakes the disk up less frequenty for background VM activity.

Turn off second processor

This is a very extreme measure and I haven't measured how much extra battery life it gives - but it is a bit cool to be able to turn it on and off if you have a dual core or dual CPU machine.

and to turn it back on use the following:

How cool is that?

Let me know if you come across any more tips. Cheers, Mark

Updated 17/Mar/07: Fixed commands based on feedback from Neil. Thanks Neil.

If you've just put you Adobe AIR package on you server and you get something that looks like the picture below then you need to add a mime type so that apache can tell your browser what type of file it is.

The reason that apache doesn't already know about it is that the Adobe AIR format has only just been released.

To fix up a Ubuntu server all you need to do is add the following line to the /etc/mime.types file:

Then restart apache with:

All done. I'm sure that the mime type will be included going forward so this won't be a problem.