Chronicles of a washed up systems administrator.
In the basement, M. Bison (mbision.foxhop.net.), a T430 with a cracked screen runs quietly with his lid closed, acting as a SmartOS hypervisor to 3 Solaris derived zones and 4 KVM ubuntu guests.
One of these guests is the oldest of …
Unlimited computer storage capacity is a common science fiction trope and fun thought experiment.
What would be possible if we could potentially store near infinite data at your house in a normal sized desktop computer?
Even better using today's tech, what ideas could we dream up?
This post will explore …
This issue blocked me for about two days and prevented me from collaborating with another engineer at gumyum co-operative video game studio.
The other engineer's environment was Windows and he was able to interact with collab and our shared project owned by our shared org. My environment was Fedora 30 …
try to log in as root locally, I needed to press ctrl-alt-f7 to get to a log in prompt.
Once logged in as root, I used the beadm command.
Try beadm list and activate beadm activate <old-version>.
Finally reboot.
I followed this process and then got my FreeNAS server to …
Russell, why are you installing Unity? I thought you were an open source developer and homegrown video game engine builder?!
The TL;DR we are starting a video game co-operative called gumyum. The basic idea is to build a large video game company owned completely by members. There will be …
A pre-signed request grants a semi-trusted user temporary access to a private resource.
Let's unpack that statement ...
Pre-signed means, we bless a specific action on a specific private resource for a short duration of time.
Semi-trusted means, we have authenticated the user, but we don't trust them to have full …
Disclosure: This post contains affiliate links. See full disclosure page here.
Exactly one year ago I fufilled a longtime childhood dream when I invested in roof top solar on my house in Eastern Connecticut (Zone 6b).
Over the past 12 months, I have religiously tracked my families energy consumption using …
Recently at work I transitioned our fleet from Ubuntu 14.04 LTS to Ubuntu 18.04 LTS. During the process I noticed an issue with our newer generation AWS EC2 "nitro" based instance types (specifically c5.2xlarge).
AWS was presenting my root block device as /dev/nvme1n1 and my data …
Each holiday season I find myself drawn to a side passion of mine.
While some build model trains and others create Christmas light shows with synchronized music you'll find me on my sofa where I build, explore, and tinker on my own video game engine.
Maybe the sound of wrapping …
Five months ago during my preparation and ramp up to gardening season, I started thinking about what it would be like to have a partner, a side kick, a secretary to keep me honest and on task.
Somebody who could review old journal notes from previous years and give me …
tl;dr use a custom entrypoint in your CircleCI 2.0 config to limit Java memory to 1G.
The new CircleCI 2.0 docker configuration supports a "primary image" (listed first) which runs all the "steps" as well as zero or many "service images" (listed subsequently). The "service images", although …
So I've struggled for a while with the best way to properly setup a console script for my SQLAlchemy Pyramid apps.
I use the Pyramid Cookiecutter Alchemy to setup my projects and as such, I do not have a global and thus importable DBSession object. Instead my database session is …
For a long time I have put off DKIM signing email sent from my web services because I couldn't wrap my head around all the indirection Postfix requires to make it work.
Honestly, I put it off for over 5 years...
Today a thought sprung into my head:
"Could I …
Ever since I was an 8 year old boy I have wanted solar. I remember reading about the environment and alternative energy sources in a monthly "socal studies" flyer my school subscribed our classroom to. I questioned, even at my young age, why the world wasn't actively switching over to …
"I work from home" — a phrase I have uttered hundreds of times and is often met with instant amazement, envy, or jokes about pants. My goal for this book is to teach you the hacks I have learned in my career to help you land your dream job. If you …
The companion webwords git repo lives here.
This project shows how to code the same minimal web app called webwords in as many different programming languages as possible. It also provides guides for building and running webwords as a docker image.
Contents
I'm running a beta for Remarkbox and here is my biggest take away:
Always collect information from potential customers as soon as possible. Catch customers while they pursue a solution to their problems.
What do I mean?
Well for starters, don't just ask for an email address, have them fill …
Nearly 6 years ago I launched the beta for LinkPeek, a web page screenshot service. After all this time, I'm writing to share something new I've been passionately working on called Remarkbox.
I started working on Remarkbox back in 2014 to solve a problem I had after moving my Wordpress …
This post continues from where we left off on the Minikube guide. If you do not already have a Kubernetes cluster, you should read that first.
Selenium Grid allows you to build a cluster of Selenium nodes. Today we will create a Selenium cluster with 1 hub and 4 nodes …
Minikube allows you to run a self contained, single node, Kubernetes cluster on your workstation. Once installed and configured, you may use kubectl to interact with it, just like a production Kubernetes cluster.
Contents
I found this technique after stumbling on Aaron Parecki's blog. You can read his post here:
Lets pretend you have an API and you need to turn on maintenance for a major change. All your …
A web page does not need to look the same on every browser or device. We cannot control the capabilities of a user's browser or device. As web designers, we have the duty to give the viewer the best experience possible. A user will come with what they have and …
This blog post assumes you already have:
I add jenkins-build.sh in the root of the Node.js code repo:
# example usage: JOB_NAME=example-api BUILD_NUMBER=101 bash jenkins-build …The Pyramid web application framework uses a request object to hold state regarding an inbound HTTP connection. A view must accept a request object as the first argument which makes it always available to our views and templates.
This behavior rocks, but Pyramid makes it even better by allowing us …
Do you have a Pyramid application which authenticates users and uses a signed cookie as a session? Do you want to build a microservice using another framework and allow it to use the same cookie and session? Me too!
First we will review a bit of Pyramid code which describes …
At work we mostly run Centos and I have some NodeJS services to deploy. I feel most familiar with Ubuntu / Upstart so this post serves as my notes on systemd.
In this contrived example, we define a service for our taco-api application. The taco-api source code lives in /opt/taco-api …
This took me about 2 hours to figure out, hopefully it saves you time.
/etc/sysconfig/network:
HOSTNAME=desired-hostname.example.com
/etc/hostname:
desired-hostname.example.com
/etc/cloud/cloud.cfg:
preserve_hostname: true
After dinner tonight, Carter, my four year old asked me, "If I swallow lots of air will I be lighter?". I thought about the question for a moment and then told him it depends on what surrounds you.
At work today I needed an easy way to collect private IP addresses of every instance in one of our production VPCs.
I ended up adding a tool to https://botoform.com to perform this task.
bf --profile <aws_profile> dump <vpc_name_tag> instances --output-format json
For example:
bf --profile customer3 dump …The right way to persist resolver options across many Linux distributions.
Tricks to reduce Bind9's large memory footprint on Joyent Triton.
Five hints to save time during your migration from WordPress.
Library work around.
You only need a two lines of code to access package names and versions.
References to take you from filtering novice to expert.
Don't reinvent the wheel, use Botocores Config facilities for work with AWS.
My brutal yet simple work around.
Stop worrying and replace your Linux hypervisor with SmartOS.
Your project deserves an asset pipeline.
The operations of a company will have intrinsic risk. Risk occurs each time we decide to take an action or an inaction. This means that anything we choose to do, or not do, has associated risk.
An organization which has an unhealthy aversion to risk has a much higher chance …
So autofs randomly stopped working on one of my Ubuntu hosts (this issue has been found on Arch as well so its most likely a change upstream). I found this error in the logs:
attempting to mount entry /net/freenas.example.net get_exports: lookup(hosts): exports lookup failed for freenas …
Today I built a GUI and workflow around Ansible using Rundeck. Tonight I started diving into sending HipChat notifications and after a bit of research, I managed to create a custom notification template for each Rundeck project.
Modify your project's configuration file, on Ubuntu it was in /var/rundeck/projects …
This weekend I finished my first sprint on s3p which is a Python library and CLI application that manages release pipelines on AWS S3. I put a lot of effort into the readme.rst file, so look there for usage and examples.
The main purpose of s3p is to use …
So I'm working with an API (AWS ElastiCache) that offers mandatory pagination of results. I need to get all results, so I took some time to work out this logic.
def combine_results(function, key, marker=0, **kwargs):
"""deal with manditory pagination of AWS result descriptions"""
results = []
while marker != None:
result …
I'm currently refactoring a script that tags AWS resources and I came up with this one liner to generate pretty output. It basically turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2. Here is the code:
', '.join(['='.join(key_value) for key_value in {'a':'1','b':'2'}.items() ])
Oh, and …
First, I installed the mongodb 14.2.0 (uuid a5775e36-2a02-11e4-942a-67ae7a242985) dataset:
imgadm avail | grep mongo imgadm import a5775e36-2a02-11e4-942a-67ae7a242985
Next, I launched a new zone with this image.
Then I grabbed the uuid of the zone (211b992b-a448-40b4-94c9-00fa82615cec) and I connected into the zone
zlogin 211b992b-a448-40b4-94c9-00fa82615cec
The zone automatically creates a username …
I used project-fifo to launch the percona (14.2.0) MySQL dataset. I couldn't get into the MySQL instance so I reached out on IRC. Johngrasty, a friendly guy in the #smartos IRC channel, provided a command to display the randomly generated MySQL password emitted to the zone-init log:
cat …
Digital Rabbit Drawing.
Emotional overload
This article expands on my “Hello World” for Heka blog post. Check that one out first if you are new to Heka.
In this guide we introduce using Heka over the network by utilizing two Hekad processes on localhost. For discussion purposes we name one of the Hekad processes "sender …
I wrote these Salt States to install Mailpile on an Ubuntu host. Fun fact, it took me 20 minutes to write these states and they worked the first time I ran them. Disclaimer - I used a throw away server and wasn't concerned that buckets of packages were installed to the …
This post analyses the FreeNAS 9 code base and discusses the various places users may feel confident to hack on.
FreeNAS uses the following software stack:
Mixed content errors occur because Nginx (the front-end server) communicates to the upstream WSGI server using http. WSGI does not know (or care) about the SSL session between Nginx and the user. The WSGI server will naively generate URIs and serve assets as http.
To fix mixed content errors, we …
Lots of people claim that you need to upgrade openssl package, but this will not fix the issue.
The output of openssl version -a command …
I extended my IRC Bot Foxbot today to allow it to run canned remote executions on behalf of users in an IRC channel. This is only a prototype or proof-of-concept. Be very careful not to allow users to inject their own commands. Foxbot must be running on the Salt Master …
Sometimes you only want to see what has changed, and that is OK.
Create a file like this:
filter.py
#!/usr/bin/python
from json import loads
from json import dumps
import fileinput
stdin_lines = [line for line in fileinput.input()]
ret = loads(''.join(stdin_lines))
for minion_id, data in ret.items …
Note: I will update this post as I progress.
So the idea is to use Salt Stack's remote execution to communicate with all nodes and run the Nagios checks and collect the return output instead of using the NRPE client/service protocol. This reduces the number of agents running on …
When operations first became a thing, system administrators stood up servers using a base image from their favourite distribution. Things were done manually. Some administrators created their own distros, some wrote customised shell scripts to be run once-and-only-once to provision software and settings. This method worked, but it was slow …
My tools for creating automatic backups for various systems
I use tar-back in combination with cron to perform regular backups of all localhost filesystems into /archive/fs. I then have a central long term storage server that collects …
mysql-backis a backup utility script to dump (backup) and gzip every MySQL database on a host.
I use mysql-back in combination with cron to perform regular database dumps of MySQL servers to the /archive/db partition on localhost. I then have a central long term storage server that collects …
There are three deployment management strategies that could be used to maintain a system. Each has pros and cons which I outline in this document.
A proceedure that is run once and only once to setup a system's configuration values and settings. A semaphore or flag generally blocks …
Use the ipmitool which ships with most Unix based operating systems. I tested on SmartOS and Ubuntu Linux. Use a live boot disk if you must.
How I added two Seagate 240G SSDs as SmartOS L2ARC
How to list all drive installed in Solaris, Open Solaris, or SmartOS
iostat -eE
format
AVAILABLE …
Over this holiday season, Christmas and New Years, I took the time to mess with some Game Development. I wrote a demo game engine using Python and SFML. I plan to use this post to track my progress.
Video Evolution Playlist
2014-01-01
This post serves as a "Hello World" for the data collection and processing software called Heka. Heka is written in Go and was open sourced by Mozilla, the same fabulous group that brings us Firefox!
I intend to use Heka to replace Logstash agents by sending logs directly to ElasticSearch …
A while back I explained how to Create your own fleet of servers with Digital Ocean and salt-cloud. Today I will extend that post and show how I deployed a test environment for Sensu, an open source monitoring framework.
Before I test out new infrastructure software, I always attempt to …
As a warning before we dive into things, this post is less of a formal publication and more of a stream of conscience.
My employer newcars.com has allowed the technical staff to host hackathon! Over the past couple weeks I have had quite a few ideas tumbling around in …
This post will explain how to create a cronjob to backup of every virtual machine on a SmartOS hypervisor.
Create the following bash script in /opt/smart-back.sh:
#!/usr/bin/bash # Backup all virtual machines on a SmartOS hypervisor # Author: russell@ballestrini.net # Website: https://russell.ballestrini.net/ # Backup directory …
As you know from my previous post, I recently deleted LinkPeek.com and after struggling to get it back online, I vowed to start utilizing configuration management. During this exercise, I noticed that the architecture I use in production seems overly complicated.
The current production deployment stack:
This post explains how to use configuration management (Salt Stack) to completely control a MongoDB collection. In our example we want to control a store's collection of plans.
First we create a JSON representation of the collection.
mongodb/plan.json:
{
"_id" : { "$oid" : "4ef8b9e2be329f491d98f74b" },
"cost" : 20, "description" : "development",
"name" : "good", "count …
It's about time to learn the difference.
This state will create a user:
russell:
user:
- present
This state will create a user and a group. This also makes the user part of the group, and handles creating the group first:
russell:
group:
- present
user:
- present
- groups:
- russell
- require:
- group: russell
This state handles user and group generation …
Have you heard about Digital Ocean? They offer a polished user interface, KVM guests with SSD storage, and an API to interact with a cloud of hypervisors. API integration got you down? Don't worry, salt-cloud has already integrated Digital Ocean among it's list of providers! The rest of this post …
Salt-stack (salt) provides a solution for centralized configuration management and remote execution. One of the most basic things Salt provides is the ability to manage the contents of a file or a directory of files. Using Salt we can dictate the state of our minions and as a result we …
I started researching the best ways to use salt to provision minecraft servers. I wrote a salt state formula for the vanilla minecraft server deployment. The deployment worked out great so I decided to try my luck with plugins.
In order to use plugins and mods we need to use …
During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need …
baby monkey fxhp cartoon
Create an online only, hangout event
Create a new event with a date far into the future, like the year 2015. Go to the event's options > advanced and enable 'this event is online only' which will create a unique Hangout URI.
Create a cronjob
Create a cronjob on each device …
Pycon 2013 was excellent, in fact it was my first one I have attended.
I found it odd that django and Pyramid had plenty of talks but nobody mentioned tornado.
The only person that brought up tornado was Guido himself, who has been researching and developing async python since December …
Python Lambda functions are very powerful but I often forget how they work or the fun things they do. This post will document how to use a lambda to provide different default arguments to a function.
We will use the human function found in ago.py as an example - because …
Special thanks to David Beitey for supplying ideas and python code for this update!
All changes are backward compatible.
Change log:
Contents
You have just coded, implemented, and submitted a pull …
Extra points if you set the camera to record (just don't record yourself or sounds) which will fill up the iPhone capacity.
What is cache? I define cache as "a saved answer to a question". Caching can speed up an application if a computationally complex question is asked frequently. Instead of the computing the answer over and over, we can use the previously cached answer. This post will present one method of …
I greeted the UPS man in the middle of the street to sign for my new Lenovo ThinkPad T430. Because this was My first brand-new laptop purchase I rationalized the time I spent tracking the package from the factory in China to my hands in Connecticut. Once inside …
I just packaged and published a couple of python modules to pypi:
To install them, run:
Install:
pip install --upgrade ago miniuri
If you want to view their source code, look here:
I hope you enjoy them.
1. Write everyday. Build a blog for the project and write about milestones, progress, and hurdles. Also keep a personal blog and write about hobbies. Read some theory about "copy writing" and search engine optimization. Write personalized email responses to customers. Great communication skills will have the most impact on …
Update - I opensourced this script here: bash kira
I came up this this script to kill certain programs after they run for too long. This works like similar to a timeout. Warning this script is pretty harsh and kills the program.
#!/bin/bash PROGRAM=replace-with-program-name PIDSFILE=/tmp/kill-these.pids for …
I spent the weekend fretting because one of my servers was basically being DOS'd by paying customers. During the outage I started thinking about the best way to scale and how I could make the code-base more efficient.
Linux top reported high load, in the 20's. Eventually I figured out …
Charts that show the load difference.
Today our production Citrix NetScaler broke. The box wouldn't boot and our only backup copy of the config was on the NetScaler itself.
Being the only Unix guy around I attempted to help out the admins working the outage. I SSH'd into the development NetScaler and noticed it runs on …
How did you hear about us?
I first answered this question when joining Linode. I remember thinking "Wow, this is a great time to ask me!" because the real answer was still in my short term memory.
When I launched LinkPeek I applied this technique. After an amazing launch (thank …
This post was originally written from a place of jealousy and bitterness.
Turns out I was wrong about this transaction and for better-or-worse, Facebook (and Mark Zuckerburg) solidified their edge as the king of social for the last 6 years.
Facebook purchased Instagram on Monday, April …
Just found this out the hard way...
It looks like the attachment of /KVMROOT/guest-dev-app.img on guest-dev did not persist when the KVM host rebooted for patching.
As it appears the virsh attach-disk command works a lot like the mount command.
In order to have a disk attachment persist …
To utilize the no SSL option for your network, configure the DNS entry for www.google.com to be a CNAME for nosslsearch.google.com.
"I see" said the blind man, to the deaf dog, as he walked off the cliff.
As far as I can tell, I am the originator of this version of this quote.
EDIT: changed "originator of the quote" to "originator of this version of this quote".
I name my virtualenv 'virtpy'. Is there a standard name being used out there?
Maybe we can come to a consensus as a standard name? Please feel free to post your virtualenv names here as a sort of poll.
Jenn explains how she drastically decreased our grocery bill.
This article will explain how to use wireshark to capture TCP/IP packets. Specifically I will show how to capture encrypted (HTTPS) packets and attempt to document the "dance" a client and server do to build an SSL tunnel.
What is Wireshark?
Wireshark is a network protocol analyzer for Windows …
I recently needed to monitor an HTTPS API for response time and availability. At first I planned to just use the Nagios check_http command.
After gathering more requirements I learned that the API was protected by client certificate authentication. After some research I quickly found that no solution existed to …
My Adventure Time art, painted with MyPaint and wacom tablet.
Contents
A monoalphabetic cipher uses fixed substitution over the entire message.
You can build a monoalphabetic cipher using a Python dictionary, like so:
monoalpha_cipher = {
'a': 'm',
'b': 'n',
'c': 'b',
'd': 'v',
'e': 'c',
'f': 'x',
'g': 'z …Why does a Hash provide better message integrity then an Internet checksum?
Hash function and checksum function both return a value which cannot be reversed.
How many keys are involved for symmetric key encryption? How about public key encryption?
Suppose you have N people who want to communicate with each other using symmetric keys. All communication between any two people, i and j, is visible to group N. Only person i and person j can …
Consider an 8-block cipher and answer the following:
How many possible input blocks does this cipher have?
How many possible mappings are there?
If we view each mapping as a key, then how many possible keys does this cipher have?
To find the input blocks of this cipher we raise …
Today I lost a customer.
I added some new code to LinkPeek to accept coupons and I didn't think of an edge case. This ended up creating an uncaught exception in my server side code which ultimatly served the newly subscribing customer an HTTP 500 error page.
The damage was …
tldr; When faced with pivoting or killing a project, take a good look at all possible by-products. Don't miss the hidden gem in a project's slag!
Last year I built yoursitemakesmebarf.com, a novelty web application which allowed anonymous link submission. The software would automatically take screenshots of submitted links …
Flash Meeting
In a office or cubicle environment a group of uninvited people gather and hover around your desk to talk to you. A meeting forms in immaculate conception as you sit bewildered at your desk.
Other names: Flash Meeting, Flash Mob Meeting, Flash Office Meeting
Note: This post shows an old way to modify the request object. I discuss a new way in my Pyramid add_request_method post.
Imagine that you have just built a wiki, blog, or cms web application that will be deployed multiple times by different people. You would like to provide the …
find out what is the ladder.
But most of all I'm scared that nobody will like me. I'm scared of failure.
Now that I got …
Occupy Wall Street contributors claim to use a "stack" to determine speaking arrangements.
I plan to explain how the term "stack" used in this scenario does not align itself with the mathematical or computer science definition.
The term stack means First In Last Out or "FILO". For example: a person …
Enable ability to insert images into a message body. You can upload and insert image files in your computer, or insert images by URLs. This lab will not work if you have offline enabled.
I had network issues after upgrading to Ubuntu 11.10 which has the linux 3.0 kernel.
I used this guide to compile the r8168 driver however, I needed to alter the Makefile to support for linux 3.0 kernel.
Edit src/Makefile:
#KEXT := $(shell echo $(KVER) | sed -ne 's …I however advocate a revolutionary search pattern like -
"russell.ballestrini.net" -site:russell.ballestrini.net
to gather an improved representation of backlinks.
Read on for the one line fix.
Convert any webpage to an image.
Is my idea too minimal?
family += 1
Really fast and clean design.
Some operating systems depend on a specific version of python to function properly. For example, Yum on Redhat Enterprise Linux 5 (RHEL5) depends on python 2.4.3. This version of python lacks support from many utilities …
America has succumbed to "spectatoritis".
Download all the images from a URI with this simple tool!
In loom of recent internet attacks many institutions have started scrambling in attempt to "strengthen" their security stance. I agree that auditing our systems and networks for potential flaws seems appropriate at this time to prevent getting "caught with our pants down". Incidentally, I have recently witnessed the introduction of …
In a perfect world we should create backups but never need them. Although this statement holds truth, creating guest backups provides many more benefits.
The most common reasons system administrators restore from a virt-back guest backup:
except instead of exchanging matter, we programmers exchange time.
Please listen to my story before jumping to conclusions.
The best security acts like an onion.
This article will cover running a Voice Over IP service like TeamSpeak on a VPS.
Voice Over IP allows users to communicate using audio over the Internet.
When planning for this article I originally was going to cover ventrilo, but their download link was obfuscated behind a heinous php session …
backup your virtual maching guests.
Solve problems and write code in your sleep.
Stack Overflow was a progressive and natural evolution of the standard clunky forum.
Using ajax it created a more fun and clean user experience.
Using badges and karma to gain responsibility allow forums post to become a game. People naturally like to see progression and growth, being able to watch …
I have placed bread.py, a python breadcrumb module, into the public domain.
The bread object accepts a url string and grants access to the url crumbs (parts) or url links (list of hrefs to each crumb).
Tutorial: Add a Breadcrumb Subscriber to a Pyramid project using 4 simple steps …