Learnosity Tech Blog

Using Mac Homebrew to install ffmpeg

Tue, 31 Jan 2012 23:40:00 -0000

Super quick one - came across this issue again today and had to search for the solution, so as note to self: To install ffmpeg via homebrew you need to specify which compiler to use on 10.7 at least:


brew install ffmpeg --use-clang

Cheers, Mark

Web Developer/Software Engineers - Sydney

Wed, 14 Dec 2011 04:50:00 -0000

Learnosity are looking for a Web Developer/Software Engineers to join our growing team. Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents. We are looking for multiple roles from junior to mid level range. [More]

Frontend Web Application Developers - Sydney

Wed, 14 Dec 2011 04:41:00 -0000

Learnosity are looking for Frontend Web Application Developers Learnosity develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year, to users on 6 continents. We are looking for multiple roles from junior to mid level range. [More]

DevOps/Linux Engineer in Sydney, Australia

Tue, 13 Dec 2011 05:50:00 -0000

Learnosity are looking for a DevOps/Linux Engineer to join our growing team.

Note: this role has now been filled. Thanks to all who applied.

Learnosity is an award winning company which develops cutting edge tools for language learning and is used by the leading educational publishers and assessment companies globally. We deliver millions of assessments every year to users on 6 continents.

[More]

Generate Railo DSN encrypted passwords

Wed, 07 Sep 2011 06:19:00 -0000

When dealing with scripted deployments to production environments it's often very worthwhile to be able to remove any dependencies on the GUI/Web interface and just edit the config files directly. For a recent project we wanted to use Railo encrypted DSN's passwords in the railo-server.xml. The only issue was that the encryption of the passwords was not documented as to how it works. A bit of digging around in the source code showed me how it was done and I've now made a simple page that will generate these encrypted passwords. Benefits: - Can use encrypted passwords - Don't need to be able to see the production DB when creating the config file So I can now generate encrypted passwords for railo without hooking my laptop railo instance up to the production db's. Nice.

Generating PDFs and ePubs from an AsciiDoc ... on a Mac

Thu, 11 Aug 2011 08:41:00 -0000

I've been updaging our existing documentation which was written in AsciiDoc and also creating new documentation recently. Our normal process is to update the source text file, commit to our SVN and the run the documentation build script from the deployment server to create the HTML and PDF versions of the document. This works fine from our server but to speed things up while I was writing the documents and make sure I had the formatting correct I thought I'd install AsciiDoc on my laptop and 'build' the document locally before committing to SVN, building and checking. Installing AsciiDoc is simple enough using MacPorts:


$ sudo port install asciidoc

Creating a html version of your AsciiDoc is simple at this point:


$ asciidoc /home/ajdyka/sample.txt

That will output a file called sample.html in the same directory as the source file. No worries ... To create a PDF or an ePub, it's recommended that you use the a2x command instead of asciidoc though. According to the AsciiDoc website "a2x is A toolchain manager for AsciiDoc (converts Asciidoc text files to other file formats)" So I gave this a try:


$ a2x -fpdf -dbook -L sample.txt
$ a2x -fepub -dbook -L sample.txt

It generated the following error though!


a2x: ERROR: xmllint --nonet --noout --valid sample.xml returned non-zero exit status 4

It took a bit of Googling (sp?) but I found the core of my solution in this post:Fixing the ePub problem with Docbook-XSL/A2X/Asciidoc The only 'tricky' part was finding my catalogue.xml, which ended up being in /opt/local/share/xsl/docbook-xsl ... and I modified the contents to be:

* the new path is where I extracted the docbook-xsl-1.76.1 archive Once I had done that, everything generated fine :) The next thing I need to work on is getting the default XSLT template cleaned up :) HTH A.J.

Tomcat: Out of memory - permgen

Thu, 04 Aug 2011 05:56:00 -0000

I ran into a permgen out of memory issue with one of our applications running on Tomcat/Railo and did a bit of digging around. The main answer on the web is: Increase the size of you permgen by adding the following:


 -XX:MaxPermSize=128m

However, to me this just delays the problem, particularly if you are using dynamic languages which load a lot of classes eg Railo. So a bit of further digging found these options:


-XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

These 3 options allow the permgen memory to be garbage collected. I tried to find a good reference link for them but couldn't. The final thing that I discovered was the jmap tool - which is very helpful for understanding the memory usage. You run it as follows and it gives a great summary of the memory usage of your running jvm.


$ sudo jmap -heap 18068

Attaching to process ID 18068, please wait...
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process
markl@davoip:/opt/tomcat/bin$ sudo jmap -heap 18068
[sudo] password for markl: 
Attaching to process ID 18068, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 530579456 (506.0MB)
   NewSize          = 1048576 (1.0MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 4194304 (4.0MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 16777216 (16.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 114294784 (109.0MB)
   used     = 54227688 (51.715553283691406MB)
   free     = 60067096 (57.284446716308594MB)
   47.445461728157255% used
From Space:
   capacity = 31260672 (29.8125MB)
   used     = 0 (0.0MB)
   free     = 31260672 (29.8125MB)
   0.0% used
To Space:
   capacity = 31260672 (29.8125MB)
   used     = 0 (0.0MB)
   free     = 31260672 (29.8125MB)
   0.0% used
PS Old Generation
   capacity = 353763328 (337.375MB)
   used     = 317909912 (303.1825180053711MB)
   free     = 35853416 (34.192481994628906MB)
   89.86514057217371% used
PS Perm Generation
   capacity = 46399488 (44.25MB)
   used     = 44117200 (42.07344055175781MB)
   free     = 2282288 (2.1765594482421875MB)
   95.08122158589336% used

Cheers, Mark

Using AWS SimpleDB with Railo and Tomcat

Tue, 26 Jul 2011 02:36:00 -0000

I've been working with Amazon Webservices (AWS) and needed to connect to it from some of our railo servers.

Getting Setup

Here is some documentation for reference and to help others:

1. Get the SDK from Amazon

I got version 1.2.4 from here: http://aws.amazon.com/sdkforjava/
Extract the lib/aws-java-sdk-1.2.4jar file and put it in your tomcat/lib folder

2. Get the HTTPComponents Client

You will also need get the HTTPComponents Client from apache as it depends on this. http://hc.apache.org/downloads.cgi I downloaded the 4.1.1 release and this worked a treat
Extract the 6 jars and drop them in your tomcat/lib folder.

3. Restart Tomcat and write some code

Comparison of techniques

I played with a few different techniques of using the AWS with tomcat and found the following:

1. CFHTTP direct API calls

This was based on a modified version of the AWS CF Console code. This has the benefit of being simple (none of the jar files were needed) and it took approx 900ms to do a query from Australia to US East AWS. From our US datacenter this took less than 30ms). However, this method had the significant drawback or requiring lots of code to be written to handle each function. Not good.

2. Java AWS SDK from Railo

I then did the steps above and used the AWS SDK from a test page using the following code:

This worked well, but each request took on average of 1500ms from AU to US East AWS. This was disappointing, so I tried caching in the the application scope as below.

3. Java AWS SDK cached in App Scope

The first request took ~ 1500ms but then subsequent requests were about 300ms, which was 1/3 of the time of the raw CF code. With the added benefit of having full access to all the API's, this is definitely the way we'll be using it going forward. Cheers, Mark

References

Learnosity Job Vacancy - Web Application Developer (Junior to Mid Level)

Mon, 04 Apr 2011 05:32:00 -0000

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level.

Our service:

Makes it practical for students to practice Oral and Aural skills.
Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them.
Can be used for homework assignments or “High Stakes Assessments”.

Learnosity are creating the next generation of language and assessment technology for use in schools and education worldwide. The current product portfolio includes:

High availability web based systems.
Cross platform software platforms (Windows, Mac and Linux).
iPhone/Android native applications.
VOIP and SMS applications.

We need someone who can:

Use Javascript or Actionscript to create great user interfaces.
Develop highly scalable web applications using ColdFusion, PHP or similar.
Cut code with the best in the world.

You will also need to be:

keen to continue learning new technologies.
able to have a conversation with non technical people.

You'll need:

1-3 years of programming experience.
Expert in at least one Client side language (Actionscript or Javascript/jQuery).
Experience in at least one Server side language (eg PHP, Java, ColdFusion, etc).
Understanding of Object Oriented design.
Understanding of XHTML and CSS.

It would be good if you have:

a degree in Computer Science, Engineering or similar.
been working with open source tools.
been playing around with iPhone/Android applications.
experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP.

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD. Salary commensurate with experience.

If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark.lynch[at]learnosity.com - no agencies please.

Internet Explorer cannot download file

Mon, 28 Feb 2011 23:51:00 -0000

I had a report recently from a client that they couldn't download a file from one of our servers using Internet Explorer 7: The message they got was:


Internet Explorer cannot download [filename] from [sitename]

Internet Explorer was not able to open this Internet site.  The requested site is either unavailable or cannot be found.  Please try again later.

I knew that the site and file were both available and a bit of hunting around I was reminded that IE used to have issues with Gzip encoding in the past - but I presumed that it was a thing of the past. I also noticed that when downloading the file Firefox would not show a progress bar but would instead show a "indeterminate progress" bar. Looking at the headers the it turned out that there was no Content-Length header but there was a header as follows:


Transfer-Encoding: chunked

The transfer encoding chunked allows the server so start sending the file before if has completely compressed it - but was not playing nice with IE7. To fix this I added some more exclusions to my mod_gzip rules to not zip some particular file types.


#Compress eveything that is not already well compressed
SetOutputFilter DEFLATE
SetEnvIfNoCase Request_URI \
	\.(?:gif|jpe?g|png|mp3|air|exe|zip)$ no-gzip dont-vary

Now its working a treat. It would also be possible to do this only for IE - but most of the files we use on this are already well compressed to gzip compression doesn't help too much anyway. Cheers, Mark

Monitor java memory usage

Thu, 17 Feb 2011 11:24:00 -0000

A quickie on how to monitor java memory usage for tomcat. Note you need to have the JDK installed


sudo jstat -gcutil `sudo jps -lv | grep tomcat | awk '{print $1}'` 10000

Change the "tomcat" to any other unique identifier to pick a different process. Thanks to Maximdim for his post on this. The JStat reference has the low down on what all the codes mean.

Block and IP Address with iptables

Thu, 20 Jan 2011 10:44:00 -0000

I had a nasty hacker trying to get access to one of my Asterisk machines tonight so a quick reminder for myself on how to block a single IP address using iptables: Block an IP:


iptables -I INPUT -s 111.222.333.444 -j DROP

Unblock an IP (if you typed the wrong address)


#substitute -D for -I
iptables -D INPUT -s 111.222.333.444 -j DROP

Show current rules:


iptables -n -L

Thanks to good folks at www.pbxer.com for the details on this:Asterisk Security: Use iptables to Block the Bad Guys

Convert FLV to MP4 with ffmpeg Howto

Wed, 29 Dec 2010 05:13:00 -0000

I recently needed to convert a lot of FLV files which were H264 encoded to MP4 so that they would play nice on my Mac Mini. Initial attempts using ffmpeg were making it re-encode the entire video which would take ages and result in a larger file or worse quality. A bit of googling and reading the man pages later I discovered


-vcodec copy -acodec copy

This tells ffmpeg to copy the video and audio without re-encoding. So insted of this


ffmpeg -i input.flv output.mp4

The Solution


ffmpeg -i input.flv -vcodec copy -acodec copy output.mp4

Further complications

This worked a treat for all except one file - which gave the following error.


[NULL @ 0x9b6b9f0]error, non monotone timestamps 37464141 >= 37463126
av_interleaved_write_frame(): Error while opening file

By using the "-an" and "-vn" flags to skip the video and audio encoding in turn I narrowed it down to a problem in the audio codec timestream. To try to get ffmpeg to fix the problem I got it to reencode the audio but copy the video codec with the following:


ffmpeg -i input.flv -vcodec copy -acodec mp2 -ar 44100 -ab 128k  output.mp4

Worked a treat. I love ffmpeg :-)

Update

As usually happens, by the time I finished writing the post I spotted another enhancement - using the libfaac codec (I had been trying to use it as aac which was failing). Final code to fix it using aac with audio quality quite high 200 (range is 0-255)


ffmpeg -i input.flv -vcodec copy -acodec libfaac -aq 200  output.mp4

Learnosity are looking for Junior to Mid level Web Application Developers x2

Wed, 20 Oct 2010 03:36:00 -0000

Learnosity develop cutting edge tools for teachers and educators. Our flagship product Learnosity Voice uses the telephone to enable language students and teachers to interact on a one to one level. Our service:

Makes it practical for students to practice Oral and Aural skills
Is efficient and effective for teachers, as they can listen to each student individually at a time to suit them
Can be used for homework assignments or “High Stakes Assessments”

Learnosity are creating the next generation of language and assessment technology for use in schools and education worldwide. The current product portfolio includes:

High availability web based systems
Cross platform software (Windows, Mac and Linux)
iPhone/Android native applications
Telephony/VOIP and SMS applications

We need someone who can:

Use Javascript or Actionscript to create great user interfaces
Develop highly scalable web applications using ColdFusion, PHP or similar
Cut code with the best in the world

You will also need to be:

keen to continue learning new technologies
able to have a conversation with non technical people
enthusiastic and ready to push the boundaries

You'll need:

1-3 years of programming experience
Experience in at least one Client side language (Actionscript or Javascript/jQuery)
Experience in at least one Server side language (eg PHP, Java, ColdFusion, etc)
Understanding of Object Oriented design
Understanding of XHTML and CSS

It would be good if you have:

A degree in Computer Science, Engineering or similar.
been working with open source tools
Experience with Adobe AIR or Flex
been playing around with iPhone/Android applications
experience with some of Linux/VOIP/SIP/Asterisk/Jabber/XMPP

This is a full time role and you will be working in a casual workplace with flexible hours in the Sydney CBD. Salary commensurate with experience. If this sounds like the job for you, email a covering letter explaining why you'll be great and your resume to mark@learnosity.com - no agencies please.

Remove svn directories using find command on linux

Wed, 29 Sep 2010 06:17:00 -0000

A quick tip on how to remove .svn directories if you accidentally copy them using a folder copy.


find . -name '.svn' -exec rm -rf {} \;

If you copy code from on svn codebase to another then you can often have hidden .svn directories copied also, which will cause svn operations to fail. Running the command above will fix it up by finding them all and deleting them.