Celery Worker wide configuration

Celery is a distributed task execution environment for Python. While the emphasis is on distributed in this software, the concept of having┬áworkers allows for settings beyond the individual task. While the first rule of optimisation is “don’t”, sharing database connections┬áis a low hanging fruit in most┬ácases. And this can be configured per worker with Celery provided signals. To create a database connection for individual worker instances, leverage these signals to create the connection when the worker starts.

This can be achieved leveraging the worker_process_init signal, and the corresponding worker_process_shutdown signal to clean up when the worker shuts down.

The code should obviously be picked up at worker start, hence the tasks.py file will be a good location to keep these settings.

Example tasks.py:

from celery.signals import worker_process_init
from celery.signals import worker_process_shutdown

app = Celery('tasks', broker=CELERY_BROKER_URL)
db = None

def init_worker(**kwargs):
  global db
  log.debug('Initializing database connection for worker.')
  db = sqlite3.connect("urls.sqlite")

def shutdown_worker(**kwargs):
  global db
  if db:
    log.debug('Closing database connectionn for worker.')

The example above opens a connection to a sqlite3 database, which in itself has other issues, but is only meant as an example. This connection is established for each individual worker at startup.


Box-256 is a browser game Bildschirmfoto 2016-09-04 um 18.38.57
where you need to solve
small tasks, e.g. let a program draw a square, in your browser. Through writing assebly. Since I wrote quite a bit assembly throughout my career, I thought this is interesting. Still, I failed at level one. Mostly because of impatience.

Source: BOX-256 – Tiny game about writing assembly to pass the graphics tests.

M├╝nchner Webwoche

Die M├╝nchner WebWoche des Isarnetz findet dieses Jahr vom 13.Juni.15 bis zum 21.Juni.15 statt. Erste Veranstaltungen sind im zugeh├Ârigen Kalender zu finden.

Anstehende Events ┬╗ M├╝nchner Webwoche.