Tech Blog

Using Mac Homebrew to install ffmpeg

Posted At : January 31, 2012 11:40 PM 3 Comments

Super quick one - came across this issue again today and had to search for the solution, so as note to self:

To install ffmpeg via homebrew you need to specify which compiler to use on 10.7 at least:

brew install ffmpeg --use-clang

Cheers, Mark

Web Developer/Software Engineers - Sydney

Posted At : December 14, 2011 4:50 AM

Learnosity are looking for a Web Developer/Software Engineers to join our growing team.

Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents.

We are looking for multiple roles from junior to mid level range.

Frontend Web Application Developers - Sydney

Posted At : December 14, 2011 4:41 AM

Learnosity are looking for Frontend Web Application Developers

Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents.

We are looking for multiple roles from junior to mid level range.

DevOps/Linux Engineer in Sydney, Australia

Posted At : December 13, 2011 5:50 AM

Learnosity are looking for a DevOps/Linux Engineer to join our growing team.

Note: this role has now been filled. Thanks to all who applied.

Learnosity is an award winning company which develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year to users on 6 continents.

Generate Railo DSN encrypted passwords

Posted At : September 7, 2011 7:19 AM

When dealing with scripted deployments to production environments it's often very worthwhile to be able to remove any dependencies on the GUI/Web interface and just edit the config files directly.

For a recent project we wanted to use Railo encrypted DSN's passwords in the railo-server.xml.

The only issue was that the encryption of the passwords was not documented as to how it works. A bit of digging around in the source code showed me how it was done and I've now made a simple page that will generate these encrypted passwords.

Benefits: - Can use encrypted passwords - Don't need to be able to see the production DB when creating the config file

So I can now generate encrypted passwords for railo without hooking my laptop railo instance up to the production db's. Nice.

Generating PDFs and ePubs from an AsciiDoc ... on a Mac

Posted At : August 11, 2011 9:41 AM 2 Comments


I've been updaging our existing documentation which was written in AsciiDoc and also creating new documentation recently.

Our normal process is to update the source text file, commit to our SVN and the run the documentation build script from the deployment server to create the HTML and PDF versions of the document.

This works fine from our server but to speed things up while I was writing the documents and make sure I had the formatting correct I thought I'd install AsciiDoc on my laptop and 'build' the document locally before committing to SVN, building and checking.

Installing AsciiDoc is simple enough using MacPorts:

$ sudo port install asciidoc

Creating a html version of your AsciiDoc is simple at this point:

$ asciidoc /home/ajdyka/sample.txt

That will output a file called sample.html in the same directory as the source file. No worries ...

To create a PDF or an ePub, it's recommended that you use the a2x command instead of asciidoc though.

According to the AsciiDoc website "a2x is A toolchain manager for AsciiDoc (converts Asciidoc text files to other file formats)"

So I gave this a try:

$ a2x -fpdf -dbook -L sample.txt
$ a2x -fepub -dbook -L sample.txt

It generated the following error though!

a2x: ERROR: xmllint --nonet --noout --valid sample.xml returned non-zero exit status 4

It took a bit of Googling (sp?) but I found the core of my solution in this post:Fixing the ePub problem with Docbook-XSL/A2X/Asciidoc

The only 'tricky' part was finding my catalogue.xml, which ended up being in /opt/local/share/xsl/docbook-xsl ... and I modified the contents to be:

<?xml version="1.0" encoding="utf-8"?>
<catalog xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog">
<!-- XML Catalog file for DocBook XSL Stylesheets v1.76.1 -->
<rewriteURI uriStartString="/usr/local/docbook-xsl-1.76.1/epub" rewritePrefix="./"/>
<rewriteSystem systemIdStartString="/usr/local/docbook-xsl-1.76.1/epub" rewritePrefix="./"/>
</catalog>
* the new path is where I extracted the docbook-xsl-1.76.1 archive

Once I had done that, everything generated fine :)

The next thing I need to work on is getting the default XSLT template cleaned up :)

HTH

A.J.

Tomcat: Out of memory - permgen

Posted At : August 4, 2011 6:56 AM

I ran into a permgen out of memory issue with one of our applications running on Tomcat/Railo and did a bit of digging around.

The main answer on the web is: Increase the size of you permgen by adding the following:

-XX:MaxPermSize=128m

However, to me this just delays the problem, particularly if you are using dynamic languages which load a lot of classes eg Railo.

So a bit of further digging found these options:

-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

These 3 options allow the permgen memory to be garbage collected. I tried to find a good reference link for them but couldn't.

The final thing that I discovered was the jmap tool - which is very helpful for understanding the memory usage.

You run it as follows and it gives a great summary of the memory usage of your running jvm.

$ sudo jmap -heap 18068

Attaching to process ID 18068, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
markl@davoip:/opt/tomcat/bin$ sudo jmap -heap 18068
[sudo] password for markl:
Attaching to process ID 18068, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 530579456 (506.0MB)
NewSize = 1048576 (1.0MB)
MaxNewSize = 4294901760 (4095.9375MB)
OldSize = 4194304 (4.0MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 16777216 (16.0MB)
MaxPermSize = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
capacity = 114294784 (109.0MB)
used = 54227688 (51.715553283691406MB)
free = 60067096 (57.284446716308594MB)
47.445461728157255% used
From Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
To Space:
capacity = 31260672 (29.8125MB)
used = 0 (0.0MB)
free = 31260672 (29.8125MB)
0.0% used
PS Old Generation
capacity = 353763328 (337.375MB)
used = 317909912 (303.1825180053711MB)
free = 35853416 (34.192481994628906MB)
89.86514057217371% used
PS Perm Generation
capacity = 46399488 (44.25MB)
used = 44117200 (42.07344055175781MB)
free = 2282288 (2.1765594482421875MB)
95.08122158589336% used

Cheers, Mark

Using AWS SimpleDB with Railo and Tomcat

Posted At : July 26, 2011 3:36 AM 2 Comments

I've been working with Amazon Webservices (AWS) and needed to connect to it from some of our railo servers.

Getting Setup

Here is some documentation for reference and to help others:

1. Get the SDK from Amazon

2. Get the HTTPComponents Client

  • You will also need get the HTTPComponents Client from apache as it depends on this. http://hc.apache.org/downloads.cgi I downloaded the 4.1.1 release and this worked a treat
  • Extract the 6 jars and drop them in your tomcat/lib folder.

3. Restart Tomcat and write some code

Comparison of techniques

I played with a few different techniques of using the AWS with tomcat and found the following:

1. CFHTTP direct API calls

This was based on a modified version of the AWS CF Console code. This has the benefit of being simple (none of the jar files were needed) and it took approx 900ms to do a query from Australia to US East AWS. From our US datacenter this took less than 30ms). However, this method had the significant drawback or requiring lots of code to be written to handle each function. Not good.

2. Java AWS SDK from Railo

I then did the steps above and used the AWS SDK from a test page using the following code:
<cfset awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
<cfset sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(awscreds)>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

This worked well, but each request took on average of 1500ms from AU to US East AWS. This was disappointing, so I tried caching in the the application scope as below.

3. Java AWS SDK cached in App Scope

<cfif NOT structKeyExists(application,'inited')>
   <cfset application.awscreds = createObject("java","com.amazonaws.auth.BasicAWSCredentials").init(accessKeyId,secretAccessKey)>
   <cfset application.sdb = createObject("java","com.amazonaws.services.simpledb.AmazonSimpleDBClient").init(application.awscreds)>
   <cfset application.inited = true>
</cfif>
<cfset selectReq = createObject("java","com.amazonaws.services.simpledb.model.SelectRequest").init("select * from #domainName# where testid = '10000000'")>

The first request took ~ 1500ms but then subsequent requests were about 300ms, which was 1/3 of the time of the raw CF code. With the added benefit of having full access to all the API's, this is definitely the way we'll be using it going forward.

Cheers, Mark

References

Learnosity Job Vacancy - Web Application Developer (Junior to Mid Level)

Posted At : April 4, 2011 6:32 AM

Learnosity develop cutting edge tools for teachers and educators.  Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level.

Our service:

  • Makes it practical for students to practice Oral and Aural skills.
  • Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them.
  • Can be used for homework assignments or “High Stakes Assessments”.

 

Learnosity are creating the next generation of language and assessment technology for use in schools and education worldwide.  The current product portfolio includes:

  • High availability web based systems.
  • Cross platform software platforms (Windows, Mac and Linux).
  • iPhone/Android native applications.
  • VOIP and SMS applications.

 

We need someone who can:

  • Use Javascript or Actionscript to create great user interfaces.
  • Develop highly scalable web applications using ColdFusion, PHP or similar.
  • Cut code with the best in the world.

 

You will also need to be:

  • keen to continue learning new technologies.
  • able to have a conversation with non technical people.

 

You'll need:

  • 1-3 years of programming experience.
  • Expert in at least one Client side language (Actionscript or Javascript/jQuery).
  • Experience in at least one Server side language (eg PHP, Java, ColdFusion, etc).
  • Understanding of Object Oriented design.
  • Understanding of XHTML and CSS.

 

It would be good if you have:

  • a degree in Computer Science, Engineering or similar.
  • been working with open source tools.
  • been playing around with iPhone/Android applications.
  • experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP.

 

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD.   Salary commensurate with experience.

If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark.lynch[at]learnosity.com - no agencies please. 

Internet Explorer cannot download file

Posted At : February 28, 2011 11:51 PM

I had a report recently from a client that they couldn't download a file from one of our servers using Internet Explorer 7:

The message they got was:

Internet Explorer cannot download [filename] from [sitename]

Internet Explorer was not able to open this Internet site. The requested site is either unavailable or cannot be found. Please try again later.

I knew that the site and file were both available and a bit of hunting around I was reminded that IE used to have issues with Gzip encoding in the past - but I presumed that it was a thing of the past.

I also noticed that when downloading the file Firefox would not show a progress bar but would instead show a "indeterminate progress" bar.

Looking at the headers the it turned out that there was no Content-Length header but there was a header as follows:

Transfer-Encoding: chunked
The transfer encoding chunked allows the server so start sending the file before if has completely compressed it - but was not playing nice with IE7.

To fix this I added some more exclusions to my mod_gzip rules to not zip some particular file types.

#Compress eveything that is not already well compressed
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \
   \.(?:gif|jpe?g|png|mp3|air|exe|zip)$ no-gzip dont-vary

Now its working a treat. It would also be possible to do this only for IE - but most of the files we use on this are already well compressed to gzip compression doesn't help too much anyway.

Cheers, Mark