Random   •   Archives   •   RSS   •   About   •   Contact > Tag: DevOps

Ubuntu 22.04 Letsencrypt Docker Hints

letsencrypt certbot is now installable via snap (the deb apt repository is no longer maintained).

alternatively you can use certbot via docker if you plan to use the certonly mode.

I did run into some issues & I will document my workarounds here:


for domain …
Continue Reading

Vertically Scaling GitLab Server For As Cheap As Possible

Today's essay acts as a power-up love story for the underdog.

A living document & quickstart for:

  • bootstrappers
  • small business under 99 employees
  • solo ops or devs
  • entrepreneurs
  • hackers
  • tinkerers

You also deserve a quick start to a GitLab Server power house!

Regardless of my intended audience, this strategy should scale …

Continue Reading

GNUnet GNS Nameserver Operator Notes, Quickstart, & Cheatsheet

We GROW with Truth, Freedom, Love.

"You have an ally".

We scroll through the official GNUnet handbook, everything is covered in details, sections have examples. Seems verbose at first glance but also familiar .

Freedom of speech is under attack.

I AM not scared, we understand the technology we have and …

Continue Reading

Copying files between cloud object stores like S3 GCP and Spaces using Boto3

tl;dr if you just want something like aws s3 cp cli, try gsutil rsync.

At work one key item of team's sprint is properly utilizing and securing Google Cloud Platform (GCP).

For one of my projects, I'm learning GCP's Object Store called Google Cloud Storage (GCS).

I have prior …

Continue Reading

I tried to install GitLab on TrueNAS and failed

Ok, so a month ago I changed employment and the new company uses GitLab exclusively for centralized code version control system.

This is my first time using GitLab and my first projects was integrating a dou/cloudmapper with a GitLab runner on a schedule.

After a couple weeks I've learned …

Continue Reading

How to fallback to an older FreeNAS version on update failure

try to log in as root locally, I needed to press ctrl-alt-f7 to get to a log in prompt.

Once logged in as root, I used the beadm command.

For example, beadm list and beadm activate <old-version>.

Finally reboot.

I followed this process and then got my FreeNAS server to …

Continue Reading

Pre-signed GET and POST for Digital Ocean Spaces

A pre-signed request grants a semi-trusted user temporary access to a private resource.

Let's unpack that statement ...

Pre-signed means, we bless a specific action on a specific private resource for a short duration of time.

Semi-trusted means, we have authenticated the user, but we don't trust them to have full …

Continue Reading

AWS nvme to block mapping

Recently at work I transitioned our fleet from Ubuntu 14.04 LTS to Ubuntu 18.04 LTS. During the process I noticed an issue with our newer generation AWS EC2 "nitro" based instance types (specifically c5.2xlarge).

AWS was presenting my root block device as /dev/nvme1n1 and my data …

Continue Reading

Running DynamoDB Local service container on CircleCI 2.0

tl;dr use a custom entrypoint in your CircleCI 2.0 config to limit Java memory to 1G.

The new CircleCI 2.0 docker configuration supports a "primary image" (listed first) which runs all the "steps" as well as zero or many "service images" (listed subsequently). The "service images", although …

Continue Reading

webwords: a minimal viable web app with docker in as many languages as possible

The companion webwords git repo lives here.

This project shows how to code the same minimal web app called webwords in as many different programming languages as possible. It also provides guides for building and running webwords as a docker image.

Continue Reading

Selenium grid on Kubernetes

This post continues from where we left off on the Minikube guide. If you do not already have a Kubernetes cluster, you should read that first.

Selenium Grid allows you to build a cluster of Selenium nodes. Today we will create a Selenium cluster with 1 hub and 4 nodes …

Continue Reading

Nginx throw HTTP 503 maintenance JSON for all requests

I found this technique after stumbling on Aaron Parecki's blog. You can read his post here:

Lets pretend you have an API and you need to turn on maintenance for a major change. All your …

Continue Reading

Build RPM or DEB packages for Node.js using Jenkins and FPM

This blog post assumes you already have:

  • a Jenkins master and none or many build servers
  • FPM installed on the build servers
  • Node.js installed on the build servers

I add jenkins-build.sh in the root of the Node.js code repo:

# example usage: JOB_NAME=example-api BUILD_NUMBER=101 bash jenkins-build …
Continue Reading

Register Super Powers with Pyramid add_request_method

The Pyramid web application framework uses a request object to hold state regarding an inbound HTTP connection. A view must accept a request object as the first argument which makes it always available to our views and templates.

This behavior rocks, but Pyramid makes it even better by allowing us …

Continue Reading

My first Systemd Service Script and Override

At work we mostly run Centos and I have some NodeJS services to deploy. I feel most familiar with Ubuntu / Upstart so this post serves as my notes on systemd.

In this contrived example, we define a service for our taco-api application. The taco-api source code lives in /opt/taco-api …

Continue Reading

Set static hostname for RHEL Centos 7 on AWS

This took me about 2 hours to figure out, hopefully it saves you time.






preserve_hostname: true
Continue Reading

If I swallow lots of air will I be lighter?

After dinner tonight, Carter, my four year old asked me, "If I swallow lots of air will I be lighter?". I thought about the question for a moment and then told him it depends on what surrounds you.

If you are surrounded by:
  • air and you swallow lots of air …
Continue Reading

Output all instance identifiers of an AWS VPC to JSON

At work today I needed an easy way to collect private IP addresses of every instance in one of our production VPCs.

I ended up adding a tool to https://botoform.com to perform this task.

bf --profile <aws_profile> dump <vpc_name_tag> instances --output-format json

For example:

bf --profile customer3 dump …
Continue Reading

Migrating from WordPress to Pelican

Five hints to save time during your migration from WordPress.

Continue Reading

Boto3 get main route table

Library work around.

Continue Reading

List all installed package names in Python

You only need a two lines of code to access package names and versions.

Continue Reading

Filtering AWS resources with Boto3

References to take you from filtering novice to expert.

Continue Reading

Working with botocore's ~/.aws/config

Don't reinvent the wheel, use Botocores Config facilities for work with AWS.

Continue Reading

A Python script which searches for available interpreters

Continue Reading

Migrating libvirt KVM guest to SmartOS KVM guest

Stop worrying and replace your Linux hypervisor with SmartOS.

Continue Reading

Setting region programmatically in Boto3

Continue Reading

Securely publish Jenkins build artifacts on Salt Master

Your project deserves an asset pipeline.

Continue Reading

Set postgres user password on PostgreSQL SmartOS Zone

Continue Reading

Risk, Process, and Balance

The operations of a company will have intrinsic risk. Risk occurs each time we decide to take an action or an inaction. This means that anything we choose to do, or not do, has associated risk.

An organization which has an unhealthy aversion to risk has a much higher chance …

Continue Reading

autofs /net automount stopped working

So autofs randomly stopped working on one of my Ubuntu hosts (this issue has been found on Arch as well so its most likely a change upstream). I found this error in the logs:

attempting to mount entry /net/freenas.example.net
get_exports: lookup(hosts): exports lookup failed for freenas …
Continue Reading

Custom Rundeck HipChat notification templates

Today I built a GUI and workflow around Ansible using Rundeck. Tonight I started diving into sending HipChat notifications and after a bit of research, I managed to create a custom notification template for each Rundeck project.

Modify your project's configuration file, on Ubuntu it was in /var/rundeck/projects …

Continue Reading

Build release pipelines on S3 with s3p

This weekend I finished my first sprint on s3p which is a Python library and CLI application that manages release pipelines on AWS S3. I put a lot of effort into the readme.rst file, so look there for usage and examples.

The main purpose of s3p is to use …

Continue Reading

Dealing with pagination in Python

So I'm working with an API (AWS ElastiCache) that offers mandatory pagination of results. I need to get all results, so I took some time to work out this logic.

def combine_results(function, key, marker=0, **kwargs):
    """deal with manditory pagination of AWS result descriptions"""
    results = []
    while marker != None:
        result …
Continue Reading

Turn python dict into a key=value string and back again

I'm currently refactoring a script that tags AWS resources and I came up with this one liner to generate pretty output. It basically turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2. Here is the code:

', '.join(['='.join(key_value) for key_value in {'a':'1','b':'2'}.items() ])

Oh, and …

Continue Reading

Migrating MongoDB from Ubuntu to SmartOS

First, I installed the mongodb 14.2.0 (uuid a5775e36-2a02-11e4-942a-67ae7a242985) dataset:

imgadm avail | grep mongo
imgadm import a5775e36-2a02-11e4-942a-67ae7a242985

Next, I launched a new zone with this image.

Then I grabbed the uuid of the zone (211b992b-a448-40b4-94c9-00fa82615cec) and I connected into the zone

zlogin 211b992b-a448-40b4-94c9-00fa82615cec

The zone automatically creates a username …

Continue Reading

Set Root Password SmartOS Percona MySQL Zone

I used project-fifo to launch the percona (14.2.0) MySQL dataset. I couldn't get into the MySQL instance so I reached out on IRC. Johngrasty, a friendly guy in the #smartos IRC channel, provided a command to display the randomly generated MySQL password emitted to the zone-init log:

cat …
Continue Reading

Heka, World2!

This article expands on my “Hello World” for Heka blog post. Check that one out first if you are new to Heka.

In this guide we introduce using Heka over the network by utilizing two Hekad processes on localhost. For discussion purposes we name one of the Hekad processes "sender …

Continue Reading

Mailpile Salt States for Ubuntu or Debian

I wrote these Salt States to install Mailpile on an Ubuntu host. Fun fact, it took me 20 minutes to write these states and they worked the first time I ran them. Disclaimer - I used a throw away server and wasn't concerned that buckets of packages were installed to the …

Continue Reading

IRC Bot (Foxbot) runs canned remote executions using Salt Stack

I extended my IRC Bot Foxbot today to allow it to run canned remote executions on behalf of users in an IRC channel. This is only a prototype or proof-of-concept. Be very careful not to allow users to inject their own commands. Foxbot must be running on the Salt Master …

Continue Reading

Filter Salt Stack Return Data Output

Sometimes you only want to see what has changed, and that is OK.

Create a file like this:



from json import loads
from json import dumps

import fileinput

stdin_lines = [line for line in fileinput.input()]

ret = loads(''.join(stdin_lines))

for minion_id, data in ret.items …
Continue Reading

Replace the Nagios Scheduler and NRPE with Salt Stack

Note: I will update this post as I progress.

So the idea is to use Salt Stack's remote execution to communicate with all nodes and run the Nagios checks and collect the return output instead of using the NRPE client/service protocol. This reduces the number of agents running on …

Continue Reading

Configuration Management and the Golden Image

When operations first became a thing, system administrators stood up servers using a base image from their favourite distribution. Things were done manually. Some administrators created their own distros, some wrote customised shell scripts to be run once-and-only-once to provision software and settings. This method worked, but it was slow …

Continue Reading

Automatic Backups

My tools for creating automatic backups for various systems

Continue Reading


tar-back is a backup utility to tar and gzip target filesystems.
It supports a custom retention, filter exclusions, and backup directory.

I use tar-back in combination with cron to perform regular backups of all localhost filesystems into /archive/fs. I then have a central long term storage server that collects …

Continue Reading


mysql-backis a backup utility script to dump (backup) and gzip every MySQL database on a host.

I use mysql-back in combination with cron to perform regular database dumps of MySQL servers to the /archive/db partition on localhost. I then have a central long term storage server that collects …

Continue Reading

The Three Deployment Management Strategies

There are three deployment management strategies that could be used to maintain a system. Each has pros and cons which I outline in this document.

run once

A proceedure that is run once and only once to setup a system's configuration values and settings. A semaphore or flag generally blocks …

Continue Reading

Heka, World!

This post serves as a "Hello World" for the data collection and processing software called Heka. Heka is written in Go and was open sourced by Mozilla, the same fabulous group that brings us Firefox!

I intend to use Heka to replace Logstash agents by sending logs directly to ElasticSearch …

Continue Reading


A while back I explained how to Create your own fleet of servers with Digital Ocean and salt-cloud. Today I will extend that post and show how I deployed a test environment for Sensu, an open source monitoring framework.

Before I test out new infrastructure software, I always attempt to …

Continue Reading

Backup all virtual machines on a SmartOS hypervisor with smart-back.sh

This post will explain how to create a cronjob to backup of every virtual machine on a SmartOS hypervisor.

Create the following bash script in /opt/smart-back.sh:


# Backup all virtual machines on a SmartOS hypervisor
# Author:  russell@ballestrini.net
# Website: https://russell.ballestrini.net/

# Backup directory …
Continue Reading

Simplify deployments with Upstart and uWSGI

As you know from my previous post, I recently deleted LinkPeek.com and after struggling to get it back online, I vowed to start utilizing configuration management. During this exercise, I noticed that the architecture I use in production seems overly complicated.

The current production deployment stack:

  • Nginx listen on …
Continue Reading

Postfix Salt State Formula

Continue Reading

Control a MongoDB collection in configuration management

This post explains how to use configuration management (Salt Stack) to completely control a MongoDB collection. In our example we want to control a store's collection of plans.

First we create a JSON representation of the collection.


  "_id" : { "$oid" : "4ef8b9e2be329f491d98f74b" },
  "cost" : 20, "description" : "development",
  "name" : "good", "count …
Continue Reading

Configuration Management vs Remote Execution

It's about time to learn the difference.

Continue Reading

Understanding Salt Stack user and group management

This state will create a user:

    - present

This state will create a user and a group. This also makes the user part of the group, and handles creating the group first:

    - present
    - present
    - groups:
      - russell
    - require:
      - group: russell

This state handles user and group generation …

Continue Reading

Create your own fleet of servers with Digital Ocean and salt-cloud

Have you heard about Digital Ocean? They offer a polished user interface, KVM guests with SSD storage, and an API to interact with a cloud of hypervisors. API integration got you down? Don't worry, salt-cloud has already integrated Digital Ocean among it's list of providers! The rest of this post …

Continue Reading

Add a custom header to all Salt managed files using pillar and jinja templates

Salt-stack (salt) provides a solution for centralized configuration management and remote execution. One of the most basic things Salt provides is the ability to manage the contents of a file or a directory of files. Using Salt we can dictate the state of our minions and as a result we …

Continue Reading

Honey! I just DELETED LinkPeek.com

During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need …

Continue Reading

virt-back's Domfetcher class returns doms from libvirt API

Continue Reading

Prevent a certain program from running too long in bash

Update - I opensourced this script here: bash kira

I came up this this script to kill certain programs after they run for too long. This works like similar to a timeout. Warning this script is pretty harsh and kills the program.


for …
Continue Reading

Always attempt to scale vertically first

I spent the weekend fretting because one of my servers was basically being DOS'd by paying customers. During the outage I started thinking about the best way to scale and how I could make the code-base more efficient.

Linux top reported high load, in the 20's. Eventually I figured out …

Continue Reading

High load on web server after updating from Ubuntu 10.04 to Ubuntu 12.04 LTS

Charts that show the load difference.

Continue Reading

How to rescue logs and config from a failed Citrix NetScaler App Gateway

Today our production Citrix NetScaler broke. The box wouldn't boot and our only backup copy of the config was on the NetScaler itself.

Being the only Unix guy around I attempted to help out the admins working the outage. I SSH'd into the development NetScaler and noticed it runs on …

Continue Reading

Trouble mounting filesystem on KVM guest after reboot

Just found this out the hard way...

It looks like the attachment of /KVMROOT/guest-dev-app.img on guest-dev did not persist when the KVM host rebooted for patching.

As it appears the virsh attach-disk command works a lot like the mount command.

In order to have a disk attachment persist …

Continue Reading

Zenoss or Nagios monitoring of HTTPS using client certificate authentication

I recently needed to monitor an HTTPS API for response time and availability. At first I planned to just use the Nagios check_http command.

After gathering more requirements I learned that the API was protected by client certificate authentication. After some research I quickly found that no solution existed to …

Continue Reading

Webmaster tools alerted issue turned out Pylon session files flooded inodes

Continue Reading

A system administrators guide to installing and maintaining multiple python environments

Some operating systems depend on a specific version of python to function properly. For example, Yum on Redhat Enterprise Linux 5 (RHEL5) depends on python 2.4.3. This version of python lacks support from many utilities …

Continue Reading

virt-back: restoring from backups

In a perfect world we should create backups but never need them. Although this statement holds truth, creating guest backups provides many more benefits.

The most common reasons system administrators restore from a virt-back guest backup:

  • recovering from data corruption
  • recovering deleted files
  • recovering from a virus infection
  • recovering from …
Continue Reading

virt-back: a python libvirt backup utility for kvm xen virtualbox

backup your virtual maching guests.

Continue Reading
© Russell Ballestrini.