Celery Worker wide configuration

Celery is a distributed task execution environment for Python. While the emphasis is on distributed in this software, the concept of having workers allows for settings beyond the individual task. While the first rule of optimisation is “don’t”, sharing database connections is a low hanging fruit in most cases. And this can be configured per worker with Celery provided signals. To create a database connection for individual worker instances, leverage these signals to create the connection when the worker starts.

This can be achieved leveraging the worker_process_init signal, and the corresponding worker_process_shutdown signal to clean up when the worker shuts down.

The code should obviously be picked up at worker start, hence the tasks.py file will be a good location to keep these settings.

Example tasks.py:

from celery.signals import worker_process_init
from celery.signals import worker_process_shutdown

app = Celery('tasks', broker=CELERY_BROKER_URL)
db = None

@worker_process_init.connect
def init_worker(**kwargs):
  global db
  log.debug('Initializing database connection for worker.')
  db = sqlite3.connect("urls.sqlite")

@worker_process_shutdown.connect
def shutdown_worker(**kwargs):
  global db
  if db:
    log.debug('Closing database connectionn for worker.')
    db.close()

The example above opens a connection to a sqlite3 database, which in itself has other issues, but is only meant as an example. This connection is established for each individual worker at startup.

BOX-256

Box-256 is a browser game Bildschirmfoto 2016-09-04 um 18.38.57
where you need to solve
small tasks, e.g. let a program draw a square, in your browser. Through writing assebly. Since I wrote quite a bit assembly throughout my career, I thought this is interesting. Still, I failed at level one. Mostly because of impatience.

Source: BOX-256 – Tiny game about writing assembly to pass the graphics tests.

Münchner Webwoche

Die Münchner WebWoche des Isarnetz findet dieses Jahr vom 13.Juni.15 bis zum 21.Juni.15 statt. Erste Veranstaltungen sind im zugehörigen Kalender zu finden.

Anstehende Events » Münchner Webwoche.

via

Why monitoring is hard

(and why your vendor will only sell you tools, not solutions)

https://www.flickr.com/photos/nnova/2923863893/in/photolist-5snySr-jeSGCg-eXcWqz-geQjsH-anmVGs-qedFb-nDbeu-5cyqbZ-5Utz4K-dmQRo-niwwp2-ngu4D5-3FkH1H-9M5SRv-61eVuz-5aWZm4-7EGNmd-iQPUJ-h69bVu-gGLPDU-i5yXU-cNtg63-6VbuRr-gGLP4f-3eNSrQ-4J8zy8-ynoqx-5prCky-cCU7a-6dCwzA-7GiFib-8Ya1fz-4bbgRN-7ddyJ-6zQC9V-btU3pv-6zz5rH-gGMbdw-bnZkqK-8idtba-62z3jS-ejFXkL-8F4KAM-a6uDj-bg2uJH-dxoB9N-jbjCu-gGM5RD-gGLMjy-5Dzd6b

Intro

Monitoring infrastructure in a meaningful way is important to any IT operations, yet it is hard to realize. Many vendors adress this problem and promise a silver bullet.

Continue reading “Why monitoring is hard”

Music Timeline.

Musikpopularität
Musik Timeline

Google hat gestern die Music Timeline vorgestellt. Das Projekt gibt äusserst spannende Einblicke was wann wie populär war. Sehr schön ist an den Daten und dem User-Interface, dass man quasi

Nur Country, kein Western
Nur Country, kein Western

“hineinzoomen” kann, also jede Musikrichtung genau anschauen kann. Man klickt durch alle Sub-Genres und von dort zu den Interpreten. All das hat Google für die letzten 60 Jahre aufbearbeitet.

 

 

via Laughing Squid.