I decided to spend a few hours packaging up Weston’s code as a re-usable library and here it is – https://github.com/hiddentao/google-tts. It’s small (< 1KB minified and gzipped), supports 42 languages, and plays back the audio using the HTML5 Audio tag (just as Weston’s example does).

Note: Audio playback in the browser will only work when running the script locally as Google’s server only returns audio if you can prevent the browser from sending the Referrer HTTP Header to their server. If you want to add background playback to your online site perhaps SoundManager will do the trick (I haven’t tested this).

To get started with Weber install it using npm:

$ npm install -g weber

Then goto the root of your project folder and create a skeleton Weber config file with:

$ weber init

At this point you will have a file called weber.json which looks something like this:

{
    "/" : "./public_assets",

    "/css/app.css" : [
        "./css/reset.css",
        "./css/main.styl"
    ],

    "/css/test.css" : {
        "minify": false,
        "input" : [
            "./css/test"
        ]
    },

    "/js/app.js" : {
        "build" : "./app.js",
        "input" : {
            "dependency" : [ "es5-shimify" ],
            "lib" : [ "./lib/jquery.js" ],
            "module": [ "./coffee" ]
        }
    },

    "/js/test.js" : {
        "minify": false,
        "input": [
            "./test/testbase.js",
            "./test/test.coffee"
        ]
    }
}

The above configuration is put there by Weber just to show you what configuration options are available.

You can now run Weber in server mode by typing:

$ weber server

Weber is now running a simple HTTP server accessible at http://localhost:9294. This mode allows you to quickly test out and rapidly iterate with your scripts, styles and templates.

The first key-value pair in the config file above is mandatory and tells Weber which folder to use as the document root when running in server mode. In this case Weber will expect an index.html file inside the ./public_assets folder to serve up when the browser navigates to the address above. By the way, Weber can be told to listen on a different port by adding a port: <portnum> key in the config file above.

The remaining key-value pairs in the config file tell Weber what to do when the browser visits the relative URLs /css/app.css, /css/test.css, /js/app.js and /js/test.js respectively. When this happens Weber dynamically builds and outputs these files using the information provided in the config file. Here is what it does for each file:

• /css/app.css – concatenates reset.css with main.styl and then minifies the final result.
• /css/test.css – concatenates all CSS and Stylus files in ./css/test.
• /js/apps.js – concatenates the node module es5-shimify with ./lib/jquery.js with all the modularized versions of all the code in ./coffee and then minifies the final result.
• /js/test.js – concatenates modularized versions of ./test/testbase.js and ./test/test.coffee.

Two points to note:

• Weber automatically compiles coffee, stylus and other files into their JS and CSS equivalents when needed.
• To modularize means to wrap the code such that it thinks of itself and behaves like a CommonJS module, i.e. require, exports, module are available.

The example folder in the Weber source code contains a fully working example app with the above configuration file.

Once you’re done testing in server mode you can tell Weber to build the actual static output files configured by doing:

$ weber build

Weber automatically decided where to place the built output file based on the config. For example, if we have:

{
    "/" : "./public_assets",
    "/js/app.js" : [ "./coffee" ]
}

Then the output file for /js/app.js will be at ./public_assets/js/app.js. But we can override this path by using the build key as follows:

{
    "/" : "./public_assets",
    "/js/app.js" : {
        "build" : [ "./built_app.js" ]
        "input" : [ "./coffee" ]
    }
}

Now Weber will build the output file at ./built_app.js. Weber can also watch the input files and folders for changes and automatically rebuild the output files when necessary if you start it in watch mode:

$ weber watch

Weber works quite well for now but there are some limitations. It doesn’t support all the available dialects and languages for input sources (e.g. SASS, LESS, Mustache) so these need adding. It would also be nice to be able to write pluggable input and output formats for Weber so that you could process your own custom format. And finally Weber needs some tests!

You can fork it at: https://github.com/hiddentao/weber

You can see the result at: zhongwen.co.uk. The source is code on github. It’s built using Spine, jQuery Mobile, Coffeescript and Weber.

Here are some notes on the technical aspects:

Stroke input recognition

For the character recognition I was able to find an existing Javascript demo of stroke input. I grabbed this code, cleaned it up and optimized and got it working in a canvas on my mobile, only to find that the character recognition algorithm was particularly weak. It calculates the angle and length of every stroke you make in proportion to the overall character size and then matches this information to a database of character strokes. The problem is that if there is even a slight different in stroke order the matching will fail to find the right character. It uses the Shortstraw algorithm for finding corners – this algorithm doesn’t tend to do too well for curved lines.

After much testing I decided to disable canvas stroke input for now and instead provide Pinyin input as well as the ability to input characters directly (in case you have a Chinese keyboard input method available for your device, which I do .

Note: Thinking about the stroke input recognition, a smarter algorithm would compare the changes in angle and length between consecutive strokes rather than the individual stroke measurements themselves. A harder problem is solving for stroke mismatches. If the user inputs a curved line which gets interpreted as a straight line and yet the actual dictionary character stroke sees it as two closely connected straight lines, how do yo match them up in a consistently, repeatable manner? I might get around to these problems later on. For now you can see the code I’ve got by looking at the canvas_stroke_input branch in the Git repository.

Data strings

For now I’ve hard-coded a whole bunch of sentences and their English translations in the data module, categorizing them by study unit. In future it would be good to implement true sentence builders, i.e. algorithms which pick a subject, object, action, etc. and construct an appropriate sentence. Such randomization will be a better test for the user.

The dictionary module

A core part of the system is the dict module. This contains a list of characters along with their matching pinyin representations (one character may have multiple pinyin representations) and also contains methods for looking up character by pinyin.

There is also a Sentence object. This takes as input a string of characters and then allows you to see whether they match another string of characters to. The matching algorithm is careful enough to avoid punctuation marks (because different users may input them differently) and also returns a list of mismatched characters. To understandl exactly how it works you can look at the nodeunit test for this module.

Contributions

I’ll happily accept any contributions to making this site better, and in particular any help on the stroke input recognition system. Please feel free to fork the github repo.

LNUG February 2012 – Ramesh Nair from Forward Technology on Vimeo.

Update: I recently gave a talk on Squel at the London Node.js User Group meetup.

For a requirement like this, you don’t really want to be performing such a long-running task as part of the standard Django request -> response cycle since 1) the Django server is single-threaded, and 2) even if you’re using WSGI to run your Django app behind Apache or a similar server you’re not really going to want one of your Apache worker processes being tied up with a single client request for long periods of time as this would adversely impact other incoming client requests. Thus, any such long-running task should be performed in a process which then informs the main Django app once it’s done. Celery is a readily-available such system (a task-queue to be precise) which enables this and it is easy to integrate into Django using django-celery.

Setting up Django Celery

Setting up Django Celery has already been documented elsewhere so I’ll simply list the settings I used to get things working (Note: I’m assuming that you’re running a Debian-based Linux system). First of all I installed RabbitMQ to use the message queue system:

$ sudo apt-get install rabbitmq-server

Then I added a vhost and username and password for my Django app to RabbitMQ:

$ sudo rabbitmqctl add_user myapp myapp
$ sudo rabbitmqctl add_vhost myapp
$ sudo rabbitmqctl set_permissions -p myapp myapp ".*" ".*" ".*"

Then in my celeryconfig.py I set the following:

BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672   # default RabbitMQ listening port
BROKER_USER = "myapp"
BROKER_PASSWORD = "myapp"
BROKER_VHOST = "myapp"
CELERY_BACKEND = "amqp"  # telling Celery to report the results back to RabbitMQ
CELERY_RESULT_DBURI = ""

To test that my setup was correct I ran:

$ ./manage.py celeryd -l INFO

[2012-01-27 12:29:01,344: WARNING/MainProcess]  
/home/ram/dev/myapp/virtualenv/lib/python2.7/site-packages/djcelery/loaders.py:86: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn("Using settings.DEBUG leads to a memory leak, never "
[2012-01-27 12:29:01,344: WARNING/MainProcess]  

 -------------- celery@RamLaptop2 v2.4.6
---- **** -----
--- * ***  * -- [Configuration]
-- * - **** ---   . broker:      amqp://guest@localhost:5672//
- ** ----------   . loader:      djcelery.loaders.DjangoLoader
- ** ----------   . logfile:     [stderr]@INFO
- ** ----------   . concurrency: 8
- ** ----------   . events:      OFF
- *** --- * ---   . beat:        OFF
-- ******* ----
--- ***** ----- [Queues]
 --------------   . celery:      exchange:celery (direct) binding:celery
                  
[Tasks]
  . REPORT_CREATE

[2012-01-27 12:29:01,399: INFO/PoolWorker-1] child process calling self.run()
[2012-01-27 12:29:01,401: INFO/PoolWorker-2] child process calling self.run()
[2012-01-27 12:29:01,403: INFO/PoolWorker-3] child process calling self.run()
[2012-01-27 12:29:01,405: INFO/PoolWorker-4] child process calling self.run()
[2012-01-27 12:29:01,406: INFO/PoolWorker-5] child process calling self.run()
[2012-01-27 12:29:01,408: INFO/PoolWorker-6] child process calling self.run()
[2012-01-27 12:29:01,409: INFO/PoolWorker-7] child process calling self.run()
[2012-01-27 12:29:01,410: INFO/PoolWorker-8] child process calling self.run()
[2012-01-27 12:29:01,411: WARNING/MainProcess] celery@RamLaptop2 has started.

At this point if you’re not familiar with writing Celery tasks then check out their tutorial on how to write Celery tasks for use by Celery daemon workers started above.

Deploying to production using Supervisord

In the production environment we need a reliable way of running the Celery daemon processes. Enter Supervisord. Essentially it’s a processes which in turn launches other processes you tell it to launch, and then monitors those child processes, restarting them if they die, etc. Here is what I did to set it up:

$ sudo easy_install supervisor

I created /etc/supervisord to hold all the configuration info:

$ sudo mkdir /etc/supervisord
[/code lang="bash"]

I then edited `/etc/supervisord/supervisord.conf`:

1
[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/var/log/supervisord/main.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
childlogdir=/var/log/supervisord            ; ('AUTO' child log dir, default $TEMP)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[include]
files = /etc/supervisord/conf.d/*.conf

I created a folder called conf.d inside /etc/supervisord and created a file called myapp-celery.conf inside that:

[program:myapp-celery]
command=/home/ram/dev/myapp/virtualenv/bin/python /home/ram/dev/myapp/manage.py celeryd --loglevel=INFO
environment=PYTHONPATH=/home/ram/dev/myapp
directory=/home/ram/dev/myapp
user=www-data
numprocs=1
stdout_logfile=/var/log/celeryd/myapp.log
stderr_logfile=/var/log/celeryd/myapp.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs = 600
priority=998

Note: I'm running my Django app inside a virtual environment, which is why I specify the path to the Python interpreter in the above file.

I created the /var/log/supervisord and /var/log/celeryd log folders and setup the appropriate permissions on them to enable logging.

To run Supervisord I did the following:

$ supervisord -c /etc/supervisord/supervisord.conf

I checked that my Celery workers were active:

$ ps aux | grep celeryd
...
www-data  26655  0.5  0.3 210000 36248 ?        Sl   12:49   0:02 /home/ram/dev/myapp/virtualenv/bin/python /home/ram/dev/myapp/manage.py celeryd --loglevel=INFO
www-data  26656  0.5  0.3 210012 36232 ?        Sl   12:49   0:02 /home/ram/dev/myapp/virtualenv/bin/python /home/ram/dev/myapp/manage.py celeryd --loglevel=INFO
www-data  26671  0.0  0.2 103364 33012 ?        S    12:49   0:00 /home/ram/dev/myapp/virtualenv/bin/python /home/ram/dev/myapp/manage.py celeryd --loglevel=INFO
...

To shut it down I did:

$ supervisorctl -c /etc/supervisord/supervisord.conf shutdown

Monitoring Supervisord

Supervisord will monitor our Celery workers and ensure they stay alive, but how will we ensure that Supervisord itself stays alive? Enter Monit. Monit does essentially the same thing as Supervisord except that it monitors child processes through pid files and other checks (e.g. checking that a web page loads in order to verify that Apache is running) and doesn't directly own them (unlike Supervisord). Monit also installs itself as an init.d script which gets launched at server boot time. It comes with a web interface (with optional authentication) which lets you easily start, stop and restart any services it is monitoring along with providing memory and CPU usage statistics.

I installed Monit:

$ sudo apt-get install monit

I edited /etc/monit/monitrc and uncommented the lines relating to the web interface:

set httpd port 2812 and
#     use address localhost  # only accept connection from localhost
#     allow localhost        # allow localhost to connect to the server and
#     allow admin:monit      # require user 'admin' with password 'monit'
     allow @monit           # allow users of group 'monit' to connect (rw)
#     allow @users readonly  # allow users of group 'users' to connect readonly

Note: I configured it above such that it accepts authenticated connections from anywhere, whereby only local Linux users who belongs to the monit group can gain access..

Then I created /etc/monit/conf.d/supervisord.monit:

 check process supervisord with pidfile /tmp/supervisord.pid
   group supervisord
   start program = "/usr/local/bin/supervisord -c /etc/supervisord/supervisord.conf"
   stop  program = "/usr/local/bin/supervisorctl -c /etc/supervisord/supervisord.conf shutdown"
   if 5 restarts within 5 cycles then timeout

Restart Monit:

$ sudo /etc/init.d/monit restart

All done! I then created a monit Linux group and added myself to it. From then on I was able to visit http://localhost:2812 to see the status of the supervisord process and control it.

Full documentation along with in-browser examples are available at squeljs.org. To get started with using Squel in your node.js program use npm install squel.

Enjoy

Update: I’ve created a Prezi for Squel which gives you an overview!

Update: I recently gave a talk on Squel at the London Node.js User Group meetup.

I came up with this little snippet to ensure that cake exited abnormally (with error code 1) if any of the tests failed:

run_tests = (callback) ->
    options = [
        'test/expression.coffee'
        'test/select.coffee'
        'test/update.coffee'
        'test/delete.coffee'
        'test/insert.coffee'
        '--spec'
    ]
    vows = spawn "#{binpath}/vows", options
    output = ""
    data_handler = (data) ->
        output =+ data if data
        stream_data_handler data
    vows.stdout.on 'data', data_handler
    vows.stderr.on 'data', data_handler
    vows.on 'exit', (status) ->
        if 0 isnt status or (output and -1 isnt output.indexOf("✗ Broken"))
            return process.exit(1)
        callback?()

Essentially it looks for the ✗ Broken string in the vows output – this string only gets output if one or more vows were ‘broken’. Now Travis also picks up on test failures. There might be a more elegant way to do the above – if you know give me a shout. Otherwise I hope somebody else finds this useful.

Although the Django test server is single-threaded (and thus all requests execute in the same thread) it does not maintain a global application state which is accessible to each request, as far as I’m aware. However, because it is single-threaded we can simulate a globally accessible application state through the use of static class member variables.

My snippet below does just this. It uses request-response middleware which kicks in during the response phase to add all SQL queries executed as part of the current request to a static class member list variable. Accessing the data requires calling a separate URL (/profiling) which will then output all the SQL queries so far ‘collected’.

Further improvements to this may include establishing a persistent socket connection to the server through which SQL query strings are received as and when they get executed. And instead of displaying the list of queries on their own page we could inject them into the Debug toolbar, if it’s enabled. In fact, enhancing the Debug toolbar itself to pick up queries executed as part of AJAX requests would be the best implementation yet.

The snippet (see http://djangosnippets.org/snippets/2632/):

# file: settings.py

MIDDLEWARE_CLASSES = (
    'django.middleware.common.CommonMiddleware',
    'yourapp.middleware.SqlProfilingMiddleware',
    ...
)



# file: yourapp/midddlware.py

from django.db import connection
import time

class SqlProfilingMiddleware(object):
    Queries = []

    def process_request(self, request):
        return None
    def process_view(self, request, view_func, view_args, view_kwargs):
        return None
    def process_template_response(self, request, response):
        self._add_sql_queries(request)
        return response
    def process_response(self, request, response):
        self._add_sql_queries(request)
        return response
    def process_exception(self, request, exception):
        return None

    def _add_sql_queries(self, request):
        for q in connection.queries:
            q["time"] = time.time() + float(q["time"])
            SqlProfilingMiddleware.Queries.insert(0, q)
            # add request info as a separator
        SqlProfilingMiddleware.Queries.insert(0, {"time": time.time(), "path" : request.path})



# file: yourapp/views.py

from yourapp.middleware import SqlProfilingMiddleware

def profiling(request):
    return render_to_response("profiling.html", {"queries": SqlProfilingMiddleware.Queries})



# file: urls.py

urlpatterns = patterns('',
  (r'^profiling/$', 'yourapp.views.profiling'),
)

var User = sequelize.define('user', {
  name: Sequelize.STRING
},{
  underscored : true
})

var Profile = sequelize.define('profile', {
  user_id: {
    type: Sequelize.INTEGER,
    primaryKey: true
  },
  dob: Sequelize.DATE
},{
  underscored : true
})

var Profile2 = sequelize.define('profile2', {
  dob: Sequelize.DATE
},{
  underscored : true
})

User.hasOne(Profile)
User.hasOne(Profile2)

/* 
At this point Profile.user_id will act as both primary and foreign key, whereas Profile2.user_id and Profile.id will have been auto-added by Sequelize and will act as foreign key and primary key respectively.
*/

Date.js alternatives

Date.js has been causing me a number of problems due to its nature of extending/modifying the built-in Date object. It really messed up my use of cookie sessions in Express, specifically to do with setting the cookie expiry dates. I still haven’t figured out exactly why cookies weren’t getting set properly but I suspect that it’s some internal change to Date made by date.js which was causing the problem. I then also had problems with it on the browser side of things.

Looking for an alternative I’m happy to have found XDate, a more light-weight and better-designed date library than date.js. It wraps the default Date object whilst still exposing an identical API as well as providing its own. The nice thing about XDate is that its custom API methods are almost identical to those provided by date.js so switching one out for the other didn’t require any massive code changes.

XDate was very recently created and it’s on Github too so I expect it will only get better with time. There isn’t a node module for it yet, as far as I’m aware. Then again, I’ve just had a look and found Moment, another nice-looking date library available through NPM. I might give that a whirl for my server side.