Tech Blog

Web Developer/Software Engineers - Sydney

Posted At : December 14, 2011 4:50 AM

Learnosity are looking for a Web Developer/Software Engineers to join our growing team.

Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents.

We are looking for multiple roles from junior to mid level range.

Tomcat: Out of memory - permgen

Posted At : August 4, 2011 6:56 AM

I ran into a permgen out of memory issue with one of our applications running on Tomcat/Railo and did a bit of digging around.

The main answer on the web is: Increase the size of you permgen by adding the following:

-XX:MaxPermSize=128m

However, to me this just delays the problem, particularly if you are using dynamic languages which load a lot of classes eg Railo.

So a bit of further digging found these options:

-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

These 3 options allow the permgen memory to be garbage collected. I tried to find a good reference link for them but couldn't.

The final thing that I discovered was the jmap tool - which is very helpful for understanding the memory usage.

You run it as follows and it gives a great summary of the memory usage of your running jvm.

$ sudo jmap -heap 18068

Attaching to process ID 18068, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
markl@davoip:/opt/tomcat/bin$ sudo jmap -heap 18068
[sudo] password for markl:
Attaching to process ID 18068, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 530579456 (506.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 16777216 (16.0MB)
MaxPermSize = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 114294784 (109.0MB)
used = 54227688 (51.715553283691406MB)
free = 60067096 (57.284446716308594MB)
47.445461728157255% used
From Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
To Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
PS Old Generation
capacity = 353763328 (337.375MB)
used = 317909912 (303.1825180053711MB)
free = 35853416 (34.192481994628906MB)
89.86514057217371% used
PS Perm Generation
capacity = 46399488 (44.25MB)
used = 44117200 (42.07344055175781MB)
free = 2282288 (2.1765594482421875MB)
95.08122158589336% used

Cheers, Mark

Using AWS SimpleDB with Railo and Tomcat

Posted At : July 26, 2011 3:36 AM 2 Comments

I've been working with Amazon Webservices (AWS) and needed to connect to it from some of our railo servers.

Getting Setup

Here is some documentation for reference and to help others:

1. Get the SDK from Amazon

2. Get the HTTPComponents Client

  • You will also need get the HTTPComponents Client from apache as it depends on this. http://hc.apache.org/downloads.cgi I downloaded the 4.1.1 release and this worked a treat
  • Extract the 6 jars and drop them in your tomcat/lib folder.

3. Restart Tomcat and write some code

Comparison of techniques

I played with a few different techniques of using the AWS with tomcat and found the following:

1. CFHTTP direct API calls

This was based on a modified version of the AWS CF Console code. This has the benefit of being simple (none of the jar files were needed) and it took approx 900ms to do a query from Australia to US East AWS. From our US datacenter this took less than 30ms). However, this method had the significant drawback or requiring lots of code to be written to handle each function. Not good.

2. Java AWS SDK from Railo

I then did the steps above and used the AWS SDK from a test page using the following code:
<cfset awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
<cfset sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(awscreds)>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

This worked well, but each request took on average of 1500ms from AU to US East AWS. This was disappointing, so I tried caching in the the application scope as below.

3. Java AWS SDK cached in App Scope

<cfif NOT structKeyExists(application,'inited')>
   <cfset application.awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
   <cfset application.sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(application.awscreds)>
   <cfset application.inited = true>
</cfif>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

The first request took ~ 1500ms but then subsequent requests were about 300ms, which was 1/3 of the time of the raw CF code. With the added benefit of having full access to all the API's, this is definitely the way we'll be using it going forward.

Cheers, Mark

References

Monitor java memory usage

Posted At : February 17, 2011 11:24 AM

A quickie on how to monitor java memory usage for tomcat. Note you need to have the JDK installed

sudo jstat -gcutil `sudo jps -lv | grep tomcat | awk '{print $1}'` 10000

Change the "tomcat" to any other unique identifier to pick a different process.

Thanks to Maximdim for his post on this.

The JStat reference has the low down on what all the codes mean.

Learnosity are looking for Junior to Mid level Web Application Developers x2

Posted At : October 20, 2010 4:36 AM

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level. Our service:
  • Makes it practical for students to practice Oral and Aural skills
  • Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them
  • Can be used for homework assignments or “High Stakes Assessments”

Learnosity are creating the next generation of language and assessment technology for use in schools and education worldwide. The current product portfolio includes:

  • High availability web based systems
  • Cross platform software (Windows, Mac and Linux)
  • iPhone/Android native applications
  • Telephony/VOIP and SMS applications

We need someone who can:

  • Use Javascript or Actionscript to create great user interfaces
  • Develop highly scalable web applications using ColdFusion, PHP or similar
  • Cut code with the best in the world

You will also need to be:

  • keen to continue learning new technologies
  • able to have a conversation with non technical people
  • enthusiastic and ready to push the boundaries

You'll need:

  • 1-3 years of programming experience
  • Experience in at least one Client side language (Actionscript or Javascript/jQuery)
  • Experience in at least one Server side language (eg PHP, Java, ColdFusion, etc)
  • Understanding of Object Oriented design
  • Understanding of XHTML and CSS

It would be good if you have:

  • A degree in Computer Science, Engineering or similar.
  • been working with open source tools
  • Experience with Adobe AIR or Flex
  • been playing around with iPhone/Android applications
  • experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD. Salary commensurate with experience.

If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark@learnosity.com - no agencies please.

Upcoming book review - Tomcat 6 Developer's Guide

Posted At : December 21, 2009 10:12 AM

I've just received a copy of the Tomcat 6 Developer's Guide from packt publishing to review.

It's nice timing as I've been working with Tomcat 6 a bit lately and in the new year plan to move some of our production systems over to running Railo on top of Tomcat.

After the extremely busy year Learnosity has had I'm looking forward to reading a few books over the break and coming back in the New Year with lots more ideas and technology to implement.

"Show Full Columns" problem with CFMX and MySQL solved

Posted At : June 1, 2008 11:01 AM

We've been performing some load testing on a new website we've developed and our helpful sysadmin noticed lots of queries happening on the DB that looked like this:

SHOW FULL COLUMNS FROM `dbname`.`tablename`

We weren't intentionally doing these queries but they were coming from somewhere. A bit of detective work via google found a very enlightening article about this problem.

A quick read of this explained:

It turns out ColdFusion was asking Connector/J for the metadata on every field, which in turn triggered a SHOW FULL COLUMNS query for every varchar and text column returned.

It also went on to say that is had been fixed as of Connector/J 5.07.

On reading the release notes it mentions that the

Driver detects when it is running in a ColdFusion MX server (tested with version 7), and uses the configuration bundle coldFusion, which sets useDynamicCharsetInfo to false (see previous entry), and sets useLocalSessionState and autoReconnect to true.

However, from my testing it wasn't doing it as we were still seeing the problem. However, we are running the MultiServer version of ColdFusion which sits atop Jrun4.

So I added the parameter directly to the querystring:

&useDynamicCharsetInfo=false
So it now looks like:

This has stopped the problem, and given a significant performance gain. On one of the sites I was load testing it gave a 12% throughput increase and on another one which had less queries it gave a 4% increase.

Hope it helps.

Cheers, Mark

Job Vacancy Sydney AU - Graduate or Junior OO Programmer

Posted At : May 8, 2008 2:17 PM

We're looking for a Computer Science Graduate or similar with strong Object Oriented programming skills and the ability to grasp cutting edge technologies quickly.

If you are looking to work with the latest technogies and are a motivated self starter with a positive "can do" attitude this is the job for you. The ability to take problems and deliver solutions is a must.

There will be significant on the job learning as we are always pushing the boundaries and using the latest and greatest technologies. We are currently working on projects using Actionscript/Flex, ColdFusion, PHP, and Javascript, and are leveraging technologies such as VOIP, SMS, Instant Messenger (Jabber XMPP) to deliver cutting edge solutions.

Suitable applicants will have a Bachelors Degree in Computer Science or Engineering, or similar tertiary qualification. Remuneration commensurate with experience.

Responsibilities would include:

  • Developing new functionality for Asterisk and VOIP applications
  • Creating Rich Internet Applications with Adobe Flex
  • Creation of Desktop applications with Adobe AIR
  • Design and development of new Web applications

Must haves:

  • A robust foundation in Object Oriented programming
  • Demonstrated experience in at least one OO language
  • Motivation to learn and push the boundaries
  • Understanding of XHTML and CSS

Preferable:

  • Experience with PHP, ColdFusion, Farcry CMS, XML, Web Standards etc
  • Competency using linux-based tools (SSH, bash, etc)
  • Familiar with using source control tools (Subversion, etc)
  • Familiarity with Actionscript or Flash

This role may be full time, part-time or on a contract basis depending on the candidates skills and experience. You will be working in a casual workplace with flexible hours in the centre of Sydney.

If this sounds like the job for you email your resume to mark@lynchconsulting.com.au if interested - no agencies please.

Getting Flex 4.0 up and running on Ubuntu

Posted At : May 8, 2008 10:41 AM

I came across a blog entry today from Mike Morearty about the flex 4 source tree.

It also mentioned the swfdump tool which looked interesting

I downloaded it and went to the bin directory to try swddump but it didn't work straight away complaining about a missing jar file.

I thought this would mean a lot of pain to get it all compiling but I was very surprised. Here were the steps to get it working on Ubuntu 8.04 Hardy Heron.

Step 1

Get the code from the repository

This will check out the code to a directory called flex4 under your current directory. I'm assuming you have Subversion installed but if you don't you'll need to run this first.
sudo apt-get install subversion

Get the tools to compile flex

sudo apt-get install ant ant-optional sun-java6-jdk
That wasn't hard now was it?

Compile

Go into the correct directory and call "ant" which will use the build.xml to build it all.
cd flex4
ant

Some couple of minutes later you should get the following message

BUILD SUCCESSFUL
Total time: 2 minutes 41 seconds

Now you have a build of Flex4. Next step is to figure out how to configure Flex Builder to use the new compiler.

NOTE: This is not a finished version of Flex4 yet. Just the work in progress.

Cheers, Mark

MD5 Checksum for Images (Binaries) in CF

Posted At : January 14, 2008 10:00 AM 8 Comments

I've been working on a piece of an application which is needs to verify that the images received have not been modified or tampered by using MD5 hashes.

Note: If you want to skip my ramble and get to the solution scroll to the bottom for the example code and download.

I thought it would be nice and easy to do it with the CF Hash function and on first attempt it appeared to be:

<cffile action="read" file="#getDirectoryFromPath(getCurrentTemplatePath())#firework.jpg" variable="myTextFile">
Hash: <cfoutput>#hash(myTextFile)#</cfoutput>
Gives:
Hash: 2A3643821420D5349665D2231842FD89

However, the hashes get created on by CF but get verified by an Flex application so I was using the as3corelib's MD5 function and couldn't get the hashes to match.

So I decided to try another MD5 implementation to see which of my functions was wrong - unfortunately it turned out that both of them were wrong but for two different reasons. I'll cover the AS3 problem in a different post.

The output from md5sum was as follows:

$ md5sum firework.jpg
2346b6ab017de31688ee35949612db07 firework.jpg

This didn't match what I had generated. A bit more head scratching and I realised that the problem was that I was trying to do an md5 hash on a binary file but the CF Hash() only takes a "string".

After loading the file with "readbinary" the hash function didn't work (no surprise) and a bit of digging around led me to use some Java functionality to handle the generation of the Hash.

<cffile action="readbinary" file="#getDirectoryFromPath(getCurrentTemplatePath())#firework.jpg" variable="myBinaryFile">
<cfset var md5 = createObject("java","java.security.MessageDigest").getInstance("MD5")>
<cfset md5.update(myBinaryFile,0,len(myBinaryFile))>
<cfset checksumByteArray = md5.digest()>

This leaves us with a ByteArray digest of the file - which we need to convert to a familiar hex encoded MD5 Hash.

<cfloop from="1" to="#len(checksumByteArray)#" index="i">
   <cfset hexCouplet = formatBaseN(bitAND(checksumByteArray[i],255),16)>
   <!--- Pad with 0's --->
   <cfif len(hexCouplet) EQ 1>
      <cfset hexCouplet = "0#hexCouplet#">
   </cfif>
   <cfset checkSumHex = "#checkSumHex##hexCouplet#">
</cfloop>
Binary Hash: <cfoutput>#checkSumHex#</cfoutput>
This gives the following checksum:
Binary Hash: 2346b6ab017de31688ee35949612db07

Bingo - all working nicely now. I've packaged it all up to make my life easier I've packaged it up into a nice CFC - Crypto.cfc.

Solution and Download

Here's the example code and a running demo of it.

The Crypto CFC is available from the subversion repository.

Here is the code to use it:

<cffile action="readbinary" file="#getDirectoryFromPath(getCurrentTemplatePath())#firework.jpg" variable="myBinaryFile">
<cfset md5 = createObject("component","au.com.lynchconsulting.cfc.utility.Crypto").hashBinary(myBinaryFile)>

<h3>MD5 Sum calculated from hashBinary()</h3>
Hash Binary: <cfoutput>#md5#</cfoutput>

Hope it helps. Cheers, Mark