Tech Blog

Web Developer/Software Engineers - Sydney

Posted At : December 14, 2011 4:50 AM

Learnosity are looking for a Web Developer/Software Engineers to join our growing team.

Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents.

We are looking for multiple roles from junior to mid level range.

Tomcat: Out of memory - permgen

Posted At : August 4, 2011 6:56 AM

I ran into a permgen out of memory issue with one of our applications running on Tomcat/Railo and did a bit of digging around.

The main answer on the web is: Increase the size of you permgen by adding the following:

-XX:MaxPermSize=128m

However, to me this just delays the problem, particularly if you are using dynamic languages which load a lot of classes eg Railo.

So a bit of further digging found these options:

-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

These 3 options allow the permgen memory to be garbage collected. I tried to find a good reference link for them but couldn't.

The final thing that I discovered was the jmap tool - which is very helpful for understanding the memory usage.

You run it as follows and it gives a great summary of the memory usage of your running jvm.

$ sudo jmap -heap 18068

Attaching to process ID 18068, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
markl@davoip:/opt/tomcat/bin$ sudo jmap -heap 18068
[sudo] password for markl:
Attaching to process ID 18068, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 530579456 (506.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 16777216 (16.0MB)
MaxPermSize = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 114294784 (109.0MB)
used = 54227688 (51.715553283691406MB)
free = 60067096 (57.284446716308594MB)
47.445461728157255% used
From Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
To Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
PS Old Generation
capacity = 353763328 (337.375MB)
used = 317909912 (303.1825180053711MB)
free = 35853416 (34.192481994628906MB)
89.86514057217371% used
PS Perm Generation
capacity = 46399488 (44.25MB)
used = 44117200 (42.07344055175781MB)
free = 2282288 (2.1765594482421875MB)
95.08122158589336% used

Cheers, Mark

Using AWS SimpleDB with Railo and Tomcat

Posted At : July 26, 2011 3:36 AM 2 Comments

I've been working with Amazon Webservices (AWS) and needed to connect to it from some of our railo servers.

Getting Setup

Here is some documentation for reference and to help others:

1. Get the SDK from Amazon

2. Get the HTTPComponents Client

  • You will also need get the HTTPComponents Client from apache as it depends on this. http://hc.apache.org/downloads.cgi I downloaded the 4.1.1 release and this worked a treat
  • Extract the 6 jars and drop them in your tomcat/lib folder.

3. Restart Tomcat and write some code

Comparison of techniques

I played with a few different techniques of using the AWS with tomcat and found the following:

1. CFHTTP direct API calls

This was based on a modified version of the AWS CF Console code. This has the benefit of being simple (none of the jar files were needed) and it took approx 900ms to do a query from Australia to US East AWS. From our US datacenter this took less than 30ms). However, this method had the significant drawback or requiring lots of code to be written to handle each function. Not good.

2. Java AWS SDK from Railo

I then did the steps above and used the AWS SDK from a test page using the following code:
<cfset awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
<cfset sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(awscreds)>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

This worked well, but each request took on average of 1500ms from AU to US East AWS. This was disappointing, so I tried caching in the the application scope as below.

3. Java AWS SDK cached in App Scope

<cfif NOT structKeyExists(application,'inited')>
   <cfset application.awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
   <cfset application.sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(application.awscreds)>
   <cfset application.inited = true>
</cfif>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

The first request took ~ 1500ms but then subsequent requests were about 300ms, which was 1/3 of the time of the raw CF code. With the added benefit of having full access to all the API's, this is definitely the way we'll be using it going forward.

Cheers, Mark

References

Learnosity are looking for Junior to Mid level Web Application Developers x2

Posted At : October 20, 2010 4:36 AM

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level. Our service:
  • Makes it practical for students to practice Oral and Aural skills
  • Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them
  • Can be used for homework assignments or “High Stakes Assessments”

Learnosity are creating the next generation of language and assessment technology for use in schools and education worldwide. The current product portfolio includes:

  • High availability web based systems
  • Cross platform software (Windows, Mac and Linux)
  • iPhone/Android native applications
  • Telephony/VOIP and SMS applications

We need someone who can:

  • Use Javascript or Actionscript to create great user interfaces
  • Develop highly scalable web applications using ColdFusion, PHP or similar
  • Cut code with the best in the world

You will also need to be:

  • keen to continue learning new technologies
  • able to have a conversation with non technical people
  • enthusiastic and ready to push the boundaries

You'll need:

  • 1-3 years of programming experience
  • Experience in at least one Client side language (Actionscript or Javascript/jQuery)
  • Experience in at least one Server side language (eg PHP, Java, ColdFusion, etc)
  • Understanding of Object Oriented design
  • Understanding of XHTML and CSS

It would be good if you have:

  • A degree in Computer Science, Engineering or similar.
  • been working with open source tools
  • Experience with Adobe AIR or Flex
  • been playing around with iPhone/Android applications
  • experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD. Salary commensurate with experience.

If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark@learnosity.com - no agencies please.

ServerStats mentioned on CFHour

Posted At : June 29, 2010 3:34 AM

Just came across a mention of a little project that I did back in 2006 in the most recent CFHour podcast - Show #58 - Monitoring, Debugging, and Guests. (Mention is at approx 53 minutes).

The ServerStats code wraps up a few undocumented methods for CF to make it easy to get information on the number of sessions active on a server as well as some basic information on memory usage.

Thanks to Charlie Arehart for mentioning it - keep up the good work.

I must have a look at updating it to support Railo when I get a moment.

Cheers, Mark

proxy_http vs proxy_ajp benchmark

Posted At : March 18, 2010 1:20 AM 2 Comments

After I posted a previous blog entry about configuring railo & tomcat with apache and mod_proxy_http, Paul Kukiel and Gary Gilbert suggested that I should be using mod_proxy_ajp.

This has been something I've been looking at, but haven't found a compelling reason for one over the other.

Proxy AJP is claimed to be faster as it is a "Wire protocol" but I couldn't find any benchmarks around this.

So I decided to do a very quick and dirty benchmark to satisfy my curiosity. This is not a scientific process, I just ran a simple railo testpage on the same machine with 50 threads of jmeter requests hitting it.

First I enabled proxy_http and ran it four times, then enabled proxy_ajp and repeated. The config is below:

# Proxy HTTP config
<IfModule mod_proxy_http.c>
   <Proxy *>
   Order deny,allow
   Allow from all
   </Proxy>
   ProxyPassMatch ^/(.*\.cfm)$ http://testsite.railo:8080/$1
   ProxyPassReverse / http://testsite.railo:8080/
</IfModule>

# Proxy AJP config
<IfModule mod_proxy_ajp.c>
   <Proxy *>
   Order deny,allow
   Allow from all
   </Proxy>
   ProxyPassMatch ^/(.*\.cfm)$ ajp://testsite.railo:8009/$1
   ProxyPassReverse / ajp://testsite.railo:8009/
</IfModule>

Results:

RunHTTP Requests/secAJP Requests/sec
Run 1206.9181.4
Run 2203.9143.6
Run 3194.6189.2
Run 4204.6191.4
Average202.5176.4

The results showed that the proxy_http module was faster - i.e. more requests per second could be pushed through.

I'm putting this down to the fact that proxy_ajp has to convert the http request into it's binary format, while proxy_http really just has to pass it along.

In different scenarios and network configurations the results may be different, but for now I'm going to stick with the http proxy.

Proxy AJP has one other benefit, in that is passes along some extra flags such as whether the request is https or not, but for our purposes we don't need this.

Cheers, Mark

Railo on Tomcat revisited - mod_proxy

Posted At : March 12, 2010 12:01 AM 5 Comments

Updated: Changed the linking between railo and tomcat to use shared.loader.

I've been doing some more work on configuring railo to work flexibly in the numerous different environments we work in, and also making it simpler to set up.

To that end I investigated the use of mod_proxy for linking it to apache instead of mod_jk.

Advantages of this approach are:

  • Simple - communications are in plain http
  • Flexible - Load balancing can be easily added at the apache layer
  • Simple - No compiling mod_jk

Here are the basic install instructions for Railo/Tomcat/Apache on Ubuntu.

Download & Install Tomcat

Download tomcat and extract content:

tar xvzf apache-tomcat-6.0.26.tar.gz

Move Tomcat to a more appropriate place:

sudo mv apache-tomcat-6.0.26 /opt/tomcat

Download Railo

Download Railo custom version jars file

Extract and move into Tomcat lib directory:

tar zxvf railo-3.1.2.001-jars.tar.gz
sudo mv railo-3.1.2.001-jars /opt/railo

Make Tomcat load the railo jars by editing catalina.properties to change the shared loader path:

shared.loader=/opt/railo/*.jar

Make Tomcat and Railo work together by modifying the web config file:

sudo nano -w /opt/tomcat/conf/web.xml

add the following inside the <web-app> element:

<servlet>
<servlet-name>CFMLServlet</servlet-name>
<servlet-class>railo.loader.servlet.CFMLServlet</servlet-class>
<init-param>
<param-name>configuration</param-name>
<param-value>{web-root-directory}/WEB-INF/railo/</param-value>
<description>Configuraton directory</description>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfm</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfml</url-pattern>
</servlet-mapping>
<servlet-mapping>
<servlet-name>CFMLServlet</servlet-name>
<url-pattern>*.cfc</url-pattern>
</servlet-mapping>

add the following inside <welcome-file-list> element:

<welcome-file>index.cfm</welcome-file>
<welcome-file>index.cfml</welcome-file>

Start up tomcat:

/opt/tomcat/bin/startup.sh
Once this is done you should be able to access the railo admin by going to the following URL:

Back to Tomcat

To test our Railo installation, let's create a test site by adding a new virtual host in both Tomcat and Apache. We do this by modifying Tomcat server.xml file (/opt/tomcat/conf/server.xml )
<Host name="testsite.railo" appBase="webapps">
<Context path="" docBase="/vhosts/testsite.railo/www"/>
</Host>

Linking with Apache via Mod Proxy

Ensure the modules proxy and proxy_http are enabled. On Ubuntu this is done as follows:

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo /etc/init.d/apache2 restart

Create vhost

Now we need to create a virtual host entry in Apache as well:
<VirtualHost *:80>
DocumentRoot /vhosts/testsite.railo/www
ServerName testsite.railo
DirectoryIndex index.cfm
   #Proxy .cfm requests to railo
   <IfModule mod_proxy.c>
      <Proxy *>
      Order deny,allow
      Allow from all
      </Proxy>
      ProxyPassMatch ^/(.*\.cfm)$ http://testsite.railo:8080/$1
      ProxyPassReverse / http://testsite.railo:8080/
   </IfModule>

   #Deny access to admin except for local/portforwarded clients
   <Location /railo-context/>
      Order deny,allow
      Deny from all
      Allow from 127.0.0.1
   </Location>
</VirtualHost>

This tells apache to forward all requests for CFM files to the railo instance.

Finally restart apache and railo and you should be good to go.

sudo /opt/tomcat/bin/shutdown.sh
sudo /opt/tomcat/bin/startup.sh
sudo /etc/init.d/apache2 restart

Upcoming book review - Tomcat 6 Developer's Guide

Posted At : December 21, 2009 10:12 AM

I've just received a copy of the Tomcat 6 Developer's Guide from packt publishing to review.

It's nice timing as I've been working with Tomcat 6 a bit lately and in the new year plan to move some of our production systems over to running Railo on top of Tomcat.

After the extremely busy year Learnosity has had I'm looking forward to reading a few books over the break and coming back in the New Year with lots more ideas and technology to implement.

Learnosity are looking for a Web Ninja at a Mid to Senior level

Posted At : December 4, 2009 5:03 AM

4 December 2009, Learnosity are looking for a Web Ninja at a Mid to Senior level.

About Learnosity

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level.

Our service:

  • Makes it practical for students to practice Oral and Aural skills
  • Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them
  • Can be used for homework assignments or “High Stakes Assessments”

We are continuing to grow our core development team and we need another great developer to help us keep up with demand.

We need someone who can:

  • Use Javascript or Actionscript to create great user interfaces
  • Develop highly scalable web applications
  • Cut code with the best in the world

You will also need to be:

  • energetic with a butt kicking attitude
  • ready to create cutting edge web 2.0 apps
  • keen to continue learning new technologies
  • able to have a conversation with non technical people

You'll need:

  • 3 or more years of programming experience
  • Expert in at least on Client side language (Actionscript or AJAX)
  • Expert in at least one Server side language (eg PHP, Java, ColdFusion, etc)
  • Understanding of Object Oriented design
  • Understanding of XHTML and CSS

It would be good if you have:

  • A degree in Computer Science, Engineering or similar.
  • been working with open source tools
  • been playing around with iPhone/Android applications
  • experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD.

If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark@learnosity.com - no agencies please.

Chinese characters not working cfdocument for PDF exports - fixed

Posted At : October 26, 2009 5:38 AM

I've just spent quite a few hours grappling with an annoying PDF export issue where the font's were not displaying correctly. This is on a CFMX 7 (7.0.2) on Ubuntu Linux system.

All the DB storage and application was using UTF8 throughout, but the PDF exports were showing nothing when they should have been showing Chinese characters.

To fix this up you need to do the following:

1. Need fonts installed on system:

sudo apt-get install ttf-arphic-bsmi00lp ttf-arphic-gbsn00lp

2. Need fonts installed in CFAdmin

  • Go to CFAdmin and Font Management
  • Select directory /usr/share/fonts/truetype/arphic
  • Select "Add" and you should see 2 new fonts added.

3: CFdocument Hotfix for CF7.0.2

The CFDocument hotfix for 7.0.2 provides some essential fixes and makes this work - Follow the instructions to install: http://kb2.adobe.com/cps/402/kb402093.html

Configure cffont.properties file

The final step is to set the cffont.properties file so that it will check additional font sets for any characters it doesn't know about - i.e. Chinese characters. Each line in the file is like a CSS font-family rule, i.e. it will start at the left and look for the correct font to render the text. I appended the two extra font names to ensure they are checked, so the file looks like this:

defaultbasefont=
dialog=Arial, Helvetica, AR PL SungtiL GB,AR PL Mingti2L Big5
dialog.bold=Arial Bold, Helvetica-Bold, AR PL SungtiL GB,AR PL Mingti2L Big5
dialog.italic=Arial Italic, Helvetica-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialog.bolditalic=Arial Bold Italic, Helvetica-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput=Courier New, Courier, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.bold=Courier New Bold, Courier-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.italic=Courier New Italic, Courier-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
dialoginput.bolditalic=Courier New Bold Italic, Courier-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
serif=Times New Roman, Times-Roman, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.bold=Times New Roman Bold, Times-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.italic=Times New Roman Italic, Times-Italic, AR PL SungtiL GB, AR PL Mingti2L Big5
serif.bolditalic=Times New Roman Bold Italic, Times-BoldItalic, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif=Arial, Helvetica, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.bold=Arial Bold, Helvetica-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.italic=Arial Italic, Helvetica-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
sansserif.bolditalic=Arial Bold Italic, Helvetica-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced=Courier New, Courier, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.bold=Courier New Bold, Courier-Bold, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.italic=Courier New Italic, Courier-Oblique, AR PL SungtiL GB, AR PL Mingti2L Big5
monospaced.bolditalic=Courier New Bold Italic, Courier-BoldOblique, AR PL SungtiL GB, AR PL Mingti2L Big5

The cffont.properties file is located in /opt/jrun4/servers/{instance name}/cfusion.ear/cfusion.war/WEB-INF/cfusion/lib on the jrun multiserver version.

5. Restart CF

Once you restart CF you documents should be coming out with their Chinese fonts intact.

I'm sure there are additional fonts to add to get it working natively with all the other asian languages, so I'll add to this as and when I need/discover them. Please feel free to post a comment if you find any more font's that should be added to the list.