Category: Internet & Cloud

Release: Kubernetes 1.12

Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability – Kubernetes Blog

Author: The 1.12 Release TeamWe’re pleased to announce the delivery of Kubernetes 1.12, our third release of 2018!Today’s release continues to focus on internal improvements and graduating features to stable in Kubernetes. This newest version graduates key features such as security and Azure. Notable additions in this release include two highly-anticipated features graduating to general availability: Kubelet TLS Bootstrap and Support for Azure Virtual Machine Scale Sets (VMSS).

Source: Kubernetes 1.12: Kubelet TLS Bootstrap and Azure Virtual Machine Scale Sets (VMSS) Move to General Availability – Kubernetes

The terrifying, hidden reality of Ridiculously Complicated Algorithms

Leseempfehlung: Ein Journalist spricht mit einem anonymen Big Data Engineer/Analyst über die Komplexität von Algorithmen. Wie erschreckend die Abhängigkeit von undurchschaubaren Komponenten geworden ist gegenüber dem Einfluss den Maschinen damit auf unser Leben haben.

Man kann das auch als Laie verstehen, wie ich meine, selbst mein Verständnis von Big Data reicht nur so weit als das als realistisch einzuschätzen.

‘I’ll lose my job if anyone knows about this.”There was a long silence which I didn’t dare to break. I had begged to make this meeting happen. And now the person I had long been trying to meet leaned towards me. “Someone is going to go through your book line by line,” he said, “to try to work out who I am.”He’d been a talented researcher, an academic, until his friend started a small technology company. He had joined the company and helped it to grow. It eventually became so big that the company had been acquired by one of the tech giants. And so, then, was he.He was now paid a fortune to help design the algorithms that were central to what the tech giant did. And he had signed solemn legal documents prohibiting him from speaking to me, or to anyone, about his work. But as the…

Source: The terrifying, hidden reality of Ridiculously Complicated Algorithms

Photos: Graffiti targeting Amazon and Jeff Bezos illustrates animosity toward tech giant in Seattle – GeekWire

Gegen Amazon wird hierzulande ja immer wieder mal gestreikt. In der Mutterstadt des Konzerns scheint der Protest mittlerweile entschieden tiefer in der Gesellschaft angekommen zu sein. Thematisiert werden dabei nicht alleine die Arbeitsbedingungen, sondern auch die Steuermoral des Internet-Versandhändlers.

From Capitol Hill to Crown Hill, graffiti targeting the tech giant and the CEO is showing up on sidewalks, the sides of buildings, light poles, bike racks, bridges and even Amazon delivery lockers.

Source: Photos: Graffiti targeting Amazon and Jeff Bezos illustrates animosity toward tech giant in Seattle – GeekWire

Corporate Open Source

One concept that is under active discussion for the past decade or so but constantly being misunderstood. Open Source is often taken as a label for software downloaded from the internet, packages free of charge, components under a particular license filed under “Creative”. Often enough it’s misused for lower quality software, which reality has proven wrong by 2017, not to mention the issue with intellectual property.

However, there are many much more aspects to the concept, that add substantial value to any software centrist product organisation. And in times of digitalisation and digital transformation, software will move into the value chain of many organisation that don’t anticipate it yet. Whenever a customer offering is complex and / or service based, transparency and documentation are often key to a satisfactory result and efficient processes.

Open source may not be the one single bullet for any organization, but the concept will help becoming more transparent and efficient.

Single Source of Truth

While SharePoint is a powerful tool with many opportunities to improve processes, many organisations use it to maintain a file server. Which has information about any other effort, therefore creating a large spread between the tangible product and the then theoretical documentation. Not to mention the version horror everybody experienced at least once, trying to ask a few people for the latest version.

Reversing this process through Wiki or even Version Control Repositories allows to keep only one version, that is automatically the latest. Software will take care of all versioning, that would go in filenames_v01_final.docx otherwise.

Transparency

Adding together the product with documentation allows quick reference, pointing back and forth between customer facing and engineering. While this may sound terrible technical, the nasty guts of any product can still be ignored by those who don’t need to see it. However, for those requiring insight, they don’t have to go through a process to see it. Or even have to talk to a colleague first and ask. Oh, and the colleague will be on vacation anyway.

Opening the product internals will remove any barrier to productive work and allow employees for quick insight. Obviously, some may argue an open repository may lead to uncontrollable product results, but that’s actually a different point. Write access or merge credentials are not required for anybody without responsibility.

Applicable Metrics

To shape it all up, the management world has plenty of nice metrics that can be applied to measure the whole thing. Not all of them express quality by themselves, but applied consciously these can carry a product far.

Documentation Coverage is something that will serve as a great basis for the point I’m trying to make here. With closed projects, or engineering only code, it’s often difficult to understand whether a product, feature or bug is actually just badly documented or the colleague just doesn’t want to help.

With a metric to measure percentage of a product being documented, at a minimum the amount of available documentation can be measured. And with the product internals being transparent, any reader can – at least theoretically – see whether the documentation correctly corresponds.

Conclusion

While being a strong supporter of open source software in general, I’m not trying to make a point for open sourcing anything outside an organisation. However, transparency will help any organisation improve the offering and processes. And the concept of open source will help achieve this transparency. It has a hurdle to overcome, in particular management will have to overcome their fear of software and technology to adopt this concept, but the step is worth taking on the way to digital transformation.

Change.

Yesterday, a software engineer, also new to the organization, roughly told me the following. The way the organisation plans projects is so different to what he is used to as a software engineer. Planning projects with a horizon of 12 or even 24 months is something he says he just cannot wrap his head around.

While this is very common and necessary in the hardware industry, it is indeed something terribly alienating software people. Software is typically treated as a living product, that takes tiny changes at a time, it is more governed towards a direction to take than having the one exact goal it has to hit by a specific date.

These very fundamental goals both mindsets follow make it difficult for change to happen. While the software engineer above obviously has a point to make, he cannot reach the people he needs to reach, because both sides are just too far apart.

At the same time, I don’t yet have an answer to the problem, but the problem itself became so obvious when this colleague told me he just doesn’t know what to say. The digital world does not yet have a common language, not to mention a common way to think about approaching problems, and unless this hurdle is taken, change will only happen slowly.

Celery Worker wide configuration

Celery is a distributed task execution environment for Python. While the emphasis is on distributed in this software, the concept of having workers allows for settings beyond the individual task. While the first rule of optimisation is “don’t”, sharing database connections is a low hanging fruit in most cases. And this can be configured per worker with Celery provided signals. To create a database connection for individual worker instances, leverage these signals to create the connection when the worker starts.

This can be achieved leveraging the worker_process_init signal, and the corresponding worker_process_shutdown signal to clean up when the worker shuts down.

The code should obviously be picked up at worker start, hence the tasks.py file will be a good location to keep these settings.

Example tasks.py:

from celery.signals import worker_process_init
from celery.signals import worker_process_shutdown

app = Celery('tasks', broker=CELERY_BROKER_URL)
db = None

@worker_process_init.connect
def init_worker(**kwargs):
  global db
  log.debug('Initializing database connection for worker.')
  db = sqlite3.connect("urls.sqlite")

@worker_process_shutdown.connect
def shutdown_worker(**kwargs):
  global db
  if db:
    log.debug('Closing database connectionn for worker.')
    db.close()

The example above opens a connection to a sqlite3 database, which in itself has other issues, but is only meant as an example. This connection is established for each individual worker at startup.