screeley.com

Django Daemon Command Extension

June7

Recently I started to move Cubby Scott away from a cron and towards a queue. It's hard to be real time when you wake up a cron job once every 3 minutes. Lame. I'm also in the process of adding screenshots and content retrieval. Both take a good amount of time to process. The queue part was easy after reading Rabbits and Warrens and Working with Python and RabbitMQ. The problem came when I started working on the consumer, no one ever talks about the consumer. Well I'm going to give the consumer some love.

The consumer should be a daemon, but what's the best way to do that? It would be nice if I could just use Django's built in management functions rather than having one off scripts. i.e:

python manage.py linkconsumer

I would run that once when I started up the server and be good to go. It turned out to be pretty easy with python-daemon. I threw together a quick class to handle it and you can get a copy of the DaemonCommand here. All it really does is create an interface for a daemon context and open it. When subclassing the DaemonCommand instead of calling handle use handle_daemon.

Carrot is the open source project that ties the two together, but was a little too complex. So for the purpose of keeping it simple stupid, I used Nathan Borror's Flopsy. Dead simple way to communicate with a queue.

If we put that together we accomplish our goal in 20 lines of code.

from daemonextension import DaemonCommand
from django.conf import settings
import os

class Command(DaemonCommand):
    #Declare Daemon std.
    stdout = os.path.join(settings.DIRNAME, "log/cubbyscott.out")
    stderr = os.path.join(settings.DIRNAME, "log/cubbyscott.err")
    pidfile = os.path.join(settings.DIRNAME, "pid/cb_link.err")
    
    def handle_daemon(self, *args, **options):
        from flopsy import Connection, Consumer
        consumer = Consumer(connection=Connection())
        consumer.declare(queue='links', 
                         exchange='cubbyscott', 
                         routing_key='importer', auto_delete=False)
        
        def message_callback(message):
            print 'Recieved: ' + message.body
            consumer.channel.basic_ack(message.delivery_tag)
        
        consumer.register(message_callback)
        
        consumer.wait()

Comments

Have you seen celery? http://pypi.python.org/pypi/celery

What do you think is complex in carrot? it's basically the same interface.

at 3:22a.m. June 10, 2009
Ask Solem

@Ask. celery looks awesome. I'll be playing with that today.

My issue with carrot was declaring queues and the lack of documentation. I don't have a lot of patience when it comes to that.

celery looks like it solves both those problems. Send me an email, would like to talk to you about it. sean@screeley.com

at 11:15a.m. June 10, 2009
Sean Creeley

@sean I don't know if you've followed carrot recently, but after Nathan's blog post we've started documenting carrot, and it's really come a long way. See: http://ask.github.com/carrot/index.html Any comments/critique on the documentation is more than welcome.

carrot was originally just a part of celery, but seems to be useful on its own. Of course, py-amqplib is really what binds all this together, but I feel we need a common, tested, library, solving common messaging tasks as well.

I'll send you an e-mail.

at 1:23p.m. June 10, 2009
Ask Solem

screeley.com

Django Daemon Command Extension

Comments

Topics

Tags

Archives

Ruminations