Russell Ballestrini

Integrating OpenAI with Dry's Sample Chat Game

2024-02-17T14:11:00-05:00

This tutorial demonstrates enhancing Dry's Sample Chat Game by integrating OpenAI's language models, enabling the game to provide intelligent, AI-driven responses to user queries. Dry, the successor to Urho3D, offers a comprehensive framework for developing 2D and 3D games. Leveraging LLMs within the Dry engine opens up new possibilities.

Background

The Sample Chat game in Dry provides basic messaging functionality where multiple copies of the game client can connect to a server to communicate via text messages. Our aim is to extend this functionality to include responses from OpenAI's language models when messages contain specific triggers, such as "gpt-3".

Prerequisites

Ensure you have the following before starting:

An OpenAI API key.
A configured Dry build environment.
Some familiarity with build tooling or at least the ability to copy and paste commands into the terminal.

Step-by-Step Integration

Follow these steps to integrate OpenAI with the Sample Chat game:

Obtain the OpenAI C++ Client

Before modifying the chat game, you need to download the necessary files from the OpenAI C++ client repository. Use the following commands to create a directory for these files and download them:

cd ~/git/dry
mkdir -p Source/ThirdParty/openai
mkdir -p Source/ThirdParty/openai/nlohmann
cd Source/ThirdParty/openai
wget https://raw.githubusercontent.com/olrea/openai-cpp/main/include/openai/openai.hpp
cd nlohmann
wget https://raw.githubusercontent.com/olrea/openai-cpp/main/include/openai/nlohmann/json.hpp

I also found the need to modify the client to code slightly for the JSON import, pardon me if this is ignorant:

// #include <nlohmann/json.hpp>  // nlohmann/json
#include <Dry/ThirdParty/openai/nlohmann/json.hpp>

The client requires cURL development files, on Fedora:

sudo dnf install libcurl-devel

Update the CMake Configuration

Apply the following diff to CMakeLists.txt to include the OpenAI C++ client as well as cURL in your dry project:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index a1eff04..0f32bc7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -188,6 +188,20 @@ else ()
 endif ()
 file (MAKE_DIRECTORY ${THIRD_PARTY_INCLUDE_DIR})

+# Find the cURL library
+find_package(CURL REQUIRED)
+
+# Globally link cURL to all targets
+link_libraries(${CURL_LIBRARIES})
+
+# Create a symbolic link for openai
+execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink
+                ${PROJECT_SOURCE_DIR}/Source/ThirdParty/openai
+                ${CMAKE_BINARY_DIR}/include/Dry/ThirdParty/openai)
+

Modify Sample/16_Chat/chat.cpp file

Implement the changes outlined in the diffs below for chat.cpp:

diff --git a/Source/Samples/16_Chat/Chat.cpp b/Source/Samples/16_Chat/Chat.cpp
index ee7c2b7..9d0e454 100644
--- a/Source/Samples/16_Chat/Chat.cpp
+++ b/Source/Samples/16_Chat/Chat.cpp
@@ -41,6 +41,7 @@
 #include <Dry/UI/Text.h>
 #include <Dry/UI/UI.h>
 #include <Dry/UI/UIEvents.h>
+#include <Dry/ThirdParty/openai/openai.hpp>

 #include "Chat.h"

@@ -201,16 +202,58 @@ void Chat::HandleSend(StringHash /*eventType*/, VariantMap& eventData)

     if (serverConnection)
     {
-        // A VectorBuffer object is convenient for constructing a message to send
-        VectorBuffer msg;
-        msg.WriteString(text);
-        // Send the chat message as in-order and reliable
-        serverConnection->SendMessage(MSG_CHAT, true, true, msg);
+        // Check if the message contains "gpt-3"
+        if (text.Contains("gpt-3"))
+        {
+            // Initialize OpenAI
+            openai::start();
+
+            // Correctly construct the JSON payload as a std::string
+            std::string payload = std::string(R"({"model": "gpt-3.5-turbo", "messages":[{"role":"user", "content":")") + text.CString() + std::string(R"("}], "max_tokens": 600, "temperature": 0.5})");
+            nlohmann::json gptResponse;
+            try {
+                // Parse the payload to JSON and make the API call
+                gptResponse = openai::chat().create(nlohmann::json::parse(payload));
+            } catch (const std::exception& e) {
+                // Handle JSON parsing errors or API call failures
+                std::cerr << "Error making API call or parsing response: " << e.what() << '\n';
+                return;
+            }
+
+            std::string responseText;
+            try {
+                // Extract the response text from the JSON response
+                responseText = gptResponse["choices"][0]["message"]["content"].get<std::string>();
+            } catch (const std::exception& e) {
+                // Handle errors in accessing the response content
+                std::cerr << "Error extracting response text: " << e.what() << '\n';
+                return;
+            }
+
+            // send the user's message to gpt-3 to the server.
+            VectorBuffer msg1;
+            msg1.WriteString(text);
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg1);
+
+            // send the gpt-3 completion to the server.
+            VectorBuffer msg2;
+            msg2.WriteString(String(responseText.c_str()));
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg2);
+        }
+        else
+        {
+            // Normal chat message handling
+            VectorBuffer msg;
+            msg.WriteString(text);
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg);
+        }
+
         // Empty the text edit after sending
         textEdit_->SetText(String::EMPTY);
     }
 }

Prepare the Build Environment

Before running CMake, ensure that any previous build configurations are cleared to avoid conflicts. This might involve deleting the CMake cache file:
```
rm CMakeCache.txt # If exists
```
Then, generate the build configuration:
```
cmake .. -DCMAKE_BUILD_TYPE=Debug -DRY_64BIT=1
```
Build the Chat Game

Compile the Sample Chat game with the newly integrated OpenAI C++ client:
```
make 16_Chat/fast
```

Testing the Integration

After applying the changes and compiling the game, ensure your OpenAI API key is available to the game:

export OPENAI_API_KEY='your_openai_api_key_here'

Run the Sample Chat game and try sending a message containing "gpt-3". You should see an intelligent response generated by OpenAI's language model.

./bin/16_Chat

Remember you'll need at least one instance of the game running as server mode before a client can interact with the LLM.

That means you need to run ./bin/16_Chat at least twice in two different windows to see the experience.

What's Next?

You've now integrated OpenAI into Dry's Sample Chat game, enhancing it with AI-driven conversational capabilities. Explore further by customizing triggers, integrating other models using the same OpenAI client, or expanding the game's features.

I think personally I will try to get the client communicating with vllm likely running openchat.

Happy coding, and enjoy bringing LLM capabilities to your Dry games!

Optimizing Memory Management for Plausible Docker on Ubuntu

2024-02-05T17:40:00-05:00

Running Docker containers on a server with limited memory can lead to out-of-memory (OOM) issues, which can disrupt services and lead to downtime. This guide will show you how to increase swap space on an Ubuntu server to provide a buffer against OOM errors, using a real-world example from a server running multiple Docker containers.

Background

Consider a typical scenario where an Ubuntu server is running several Docker containers:

fox@analytics:~$ sudo docker ps
CONTAINER ID        IMAGE                                           PORTS                          NAMES
7677a77e5606        plausible/analytics:v2.0                        0.0.0.0:8000->8000/tcp         plausible-plausible-1
0ea79b7d0c03        clickhouse/clickhouse-server:23.3.7.5-alpine    8123/tcp, 9000/tcp, 9009/tcp   plausible-plausible_events_db-1
33f6e8a30da4        postgres:14-alpine                              5432/tcp                       plausible-plausible_db-1
40400c725d7c        bytemark/smtp                                   25/tcp                         plausible-mail-1

fox@analytics:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:            981         693          62          70         225          66
Swap:             0           0           0

In this example, the server has less than 1 GB of RAM and no swap space configured. This configuration is prone to OOM errors, especially when the containers require more memory than what is available. In my case the service stayed online but the configuration management service salt-minion was unable to be reached to deploy new TLS certificates, the cert expired and prevented clients from sending metrics.

Creating and Enabling Swap Space

To prevent OOM errors, we'll create a swap file. This script automates the process, ensuring that your system has additional virtual memory to handle peak loads.

Save the script as setup_swap.sh and execute it on your server:

#!/bin/bash

# Size of the swap file
SWAP_SIZE=1G
SWAP_FILE=/swapfile
SWAPPINESS=10
CACHE_PRESSURE=50

# Create a swap file
fallocate -l $SWAP_SIZE $SWAP_FILE || dd if=/dev/zero of=$SWAP_FILE bs=1024 count=1048576

# Secure swap file permissions
chmod 600 $SWAP_FILE

# Set up a Linux swap area
mkswap $SWAP_FILE

# Enable the swap file
swapon $SWAP_FILE

# Make the swap file permanent
echo "$SWAP_FILE none swap sw 0 0" | tee -a /etc/fstab

# Set the swappiness value
sysctl vm.swappiness=$SWAPPINESS
echo "vm.swappiness=$SWAPPINESS" | tee -a /etc/sysctl.conf

# Set the cache pressure value
sysctl vm.vfs_cache_pressure=$CACHE_PRESSURE
echo "vm.vfs_cache_pressure=$CACHE_PRESSURE" | tee -a /etc/sysctl.conf

# Output the results
echo "Swap file created and enabled:"
swapon --show
echo "Swappiness set to $SWAPPINESS and cache pressure set to $CACHE_PRESSURE."

This script will create a 1 GB swap file, configure the system to use it, and adjust kernel parameters to optimize memory usage.

Understanding Swappiness and Cache Pressure

The swappiness parameter influences how often the system uses swap space. A value of 10 encourages the system to keep processes in RAM, resorting to swap only when necessary.

The vfs_cache_pressure setting determines how aggressively the kernel reclaims memory from the cache. A value of 50 provides a balance between reclaiming memory and maintaining cache for quick file access.

Monitoring Your System

After increasing the swap space, monitor your system's memory usage with:

free -m

This will help you understand if the swap space is sufficient or if further adjustments are needed.

What's Next?

By adding swap space and tuning kernel parameters, you've bolstered your server's ability to handle memory-intensive Docker containers. However, swap is not a replacement for physical RAM. If your server consistently uses a lot of swap, consider upgrading the RAM for better performance.

Stay proactive in managing your server's resources to ensure uninterrupted service for your Dockerized applications. Happy hosting!

Understanding Pause Functionality in LucKey Park Built on Dry Engine

2024-02-05T08:57:00-05:00

In today's post, we're examining the pause functionality of the LucKey Park game code, which is built on the 'Dry' engine. We'll uncover how the game's status is managed and toggled, especially in response to the cancel button (ESC key), and provide insights into the game's input handling and state management.

Check out screenshots and videos of LucKey Park on itch.io

Understanding the Game's Status Management

The 'Park' game uses an enumeration GameStatus with values like GS_MAIN, GS_PLAY, GS_PAUSED, and GS_MODAL to represent the game's current state. The Game class in game.h maintains a status_ variable to track this state.

enum GameStatus{ GS_MAIN, GS_PLAY, GS_PAUSED, GS_MODAL };

class Game: public Object
{
    // ...
    private:
        GameStatus status_;
};

The Game class provides methods to get and set this status, as well as a TogglePause() method that switches between GS_PLAY and GS_PAUSED.

Binding the Cancel Action

In inputmaster.cpp, the InputMaster class binds the IA_CANCEL action to the ESC key. This setup is crucial for handling the pause functionality when the player presses ESC.

Bind(IA_CANCEL, KEY_ESCAPE);

Handling the Cancel Action

When the IA_CANCEL action is triggered, the HandleAction method in InputMaster responds accordingly. If no tool is active, it calls gui->ShowMainMenu(), which indirectly pauses the game by changing the game's status.

case IA_CANCEL:
    if (tool) sceneCursor->SetTool(nullptr);
    else gui->ShowMainMenu();
    break;

The Main Update Loop Controlled by Dry Engine

The Dry engine, like its predecessor Urho3D, manages the main game loop internally. Developers hook into this loop by subscribing to the E_UPDATE event, which the engine dispatches every frame. This event allows developers to perform per-frame updates to their game logic.

While the main update loop is not explicitly shown in the provided snippets, it's typically where the game checks the status_ variable each frame. If the status is GS_PAUSED or GS_MODAL, the loop skips updating the game world, pausing the game's logic.

End-to-End Pause Functionality

From the moment the player presses the ESC key to the game entering a paused state, the flow is as follows:

The ESC key is pressed.
InputMaster detects the IA_CANCEL action.
HandleAction calls gui->ShowMainMenu() if no tool is active.
ShowMainMenu() sets the game's status to GS_MODAL.
The main update loop, controlled by the Dry engine, respects this status and pauses the game.

This exploration has provided a clear understanding of how the pause functionality is implemented in Lucky Park using the Dry engine. It's a great example of how game state management and input handling work together to create responsive gameplay.

Stay tuned for more deep dives into game development.

Installing YaCy on Ubuntu

2024-02-04T11:02:00-05:00

This guide will walk you through installing YaCy on Ubuntu.

By default YaCy is configured to bind to 0.0.0.0 but it's admin interface is only accessible by default to a white list which includes localhost and 127.0.0.1, since my install is headless on a remote Ubuntu server, I access the admin interface remotely via an SSH tunnel.

Installing YaCy

run this script in a terminal, install_yacy.sh:

#!/bin/bash

# Update package lists
sudo apt-get update

# Install Java Development Kit (JDK)
sudo apt-get install -y openjdk-21-jdk

# Check if Java is correctly installed
java -version

# Download YaCy
wget https://download.yacy.net/yacy_v1.924_20210209_10069.tar.gz

# Extract the downloaded file
tar xfz yacy_v1.924_20210209_10069.tar.gz

# Change to the extracted directory
cd yacy

# Start YaCy
./startYACY.sh

Accessing the Admin Interface Remotely via an SSH Tunnel

You can create an SSH tunnel to your remote YaCy server, for example yacy.foxhop.net so that when you access localhost:8090, it tunnels to the remote host.

Open a terminal on your local machine and run the following command:

ssh -L 8090:localhost:8090 user@yacy.foxhop.net

Replace user with your username on the remote host. Now, when you access localhost:8090 on your local machine, it will be tunneled to the remote host.

Securing YaCy Portal and admin access with SSL/TLS

In the early days of YaCy, encryption was not as prevalent as it is today. However, securing your YaCy instance with SSL/TLS is essential for modern web safety standards. Here's how to enable SSL/TLS encryption for your YaCy server:

Generate a keystore using the keytool command:
```
keytool -keystore mySrvKeystore -genkey -keyalg RSA -alias yacy.foxhop.net
```
This command creates a keystore file named mySrvKeystore.
Move the mySrvKeystore file to the DATA/SETTINGS/ directory in your YaCy installation:
```
mv mySrvKeystore /path/to/YaCy/DATA/SETTINGS/
```
Edit the yacy.conf file to configure YaCy to use the new keystore:
```
vim /path/to/YaCy/DATA/SETTINGS/yacy.conf
```
Add or modify the following lines:
```
keyStore=DATA/SETTINGS/mySrvKeystore
keyStorePassword=YourKeystorePassword
```
Replace YourKeystorePassword with the password you chose when you created the keystore with the keytool command.

Restart YaCy to apply the SSL/TLS settings:

/path/to/YaCy/stopYACY.sh
/path/to/YaCy/startYACY.sh

Now, you can access the YaCy admin interface securely via https://localhost:8443. By default, YaCy listens on port 8443 for HTTPS, but this can be changed in the admin console, as was done in this case to use port 8091 so that https://localhost:8091 works instead. Ensure that HTTP remains on port 8090 for DHT network access by peers.

What's next?

With SSL/TLS enabled, it's time to start a local crawl and consider sharing the results with the YaCy network. To do this, you may need to open port 8090 on your router and forward it to your YaCy host. Before making changes to your network configuration, verify your peer mode status in the YaCy admin interface to understand your node's role in the network.

Sharing your crawl index with the network allows for a distributed scraping effort, meaning that the data you collect can benefit all users of YaCy, not just your local instance. This is the essence of the Distributed Hash Table (DHT) that underpins YaCy's decentralized architecture.

Continue to explore the capabilities of your YaCy server, and remember to update your documentation to assist others in their journey toward a more open and collaborative internet.`

Building 'Dry' and 'Park' from Source on Fedora Linux

2024-02-03T09:19:00-05:00

In this guide, we'll explore how to compile the 'Dry' game engine and the 'Park' game from source on Fedora Linux with debugging enabled. This process will give you a deeper understanding of the inner workings of game development and the satisfaction of running a game you built yourself.

For Ubuntu, go here for pre-compiled game:

https://luckeyproductions.itch.io/park

Prerequisites

Before diving in, ensure you have the necessary tools and libraries installed:

Git
CMake
GNU Make
GCC or Clang
X11 and audio system development libraries

Install these on Fedora with:

sudo dnf groupinstall "Development Tools"
sudo dnf install cmake git gcc-c++ libX11-devel libXcursor-devel libXinerama-devel libXi-devel libXrandr-devel libXrender-devel libXScrnSaver-devel libXxf86vm-devel pulseaudio-libs-devel nas-libs-devel qt5-qtbase-devel

Cloning the Repositories

Clone the 'Dry' and 'Park' repositories using Git:

cd ~/git
git clone https://gitlab.com/luckeyproductions/dry.git
git clone https://gitlab.com/luckeyproductions/games/park.git

Building Dry with Debugging Enabled

Navigate to the 'dry' directory and prepare the build environment:

cd dry
mkdir build && cd build

Configure the build with CMake and enable debugging:

cmake .. -DCMAKE_BUILD_TYPE=Debug -DRY_64BIT=1

Compile the Dry library with debug symbols:

make

Building Park with Debugging Enabled

In the 'Park' project, modify the Park.pro file to add the -g -O0 flags for debugging, for example:

QMAKE_CXXFLAGS += -std=c++17 -g -O0

Then set the DRY_HOME environment variable to the path of your compiled Dry library:

export DRY_HOME=/home/fox/git/dry/build

Navigate to the 'park' directory and compile the game:

cd /home/fox/git/park
mkdir build && cd build
qmake ../Park.pro
make
cp -r ../Resources .

Running Park

After a successful build, run the Park executable located in the build directory:

./park

Analyzing Core Dumps on Fedora

If your application crashes, Fedora can generate core dumps, which are snapshots of the program's state at the time of the crash. These can be invaluable for debugging.

List recent core dumps with:

coredumpctl list

To analyze a specific core dump, use gdb:

gdb /path/to/executable /path/to/coredump

For example:

gdb /home/fox/git/park/Park/park core.291993

Once in gdb, use the bt command to print a backtrace:

(gdb) bt

This will show you the call stack at the time of the crash, which can help pinpoint the source of the problem.

Happy building and debugging!

Ubuntu 22.04 Letsencrypt Docker Hints

2022-05-20T20:10:00-04:00

letsencrypt certbot is now installable via snap (the deb apt repository is no longer maintained).

alternatively you can use certbot via docker if you plan to use the certonly mode.

I did run into some issues & I will document my workarounds here:

domains=(
    example.com
    shop.example.com
)

for domain in ${domains[*]}; do
    echo "certifying: $domain"

    IFS='.' read -r -a domain_parts <<< "$domain"

    domain_parts_length=${#domain_parts[@]}

    if [ "$domain_parts_length" -eq 2 ]
    then
        # https://eff-certbot.readthedocs.io/en/stable/install.html#running-with-docker
        docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/var/log/letsencrypt:/var/log/letsencrypt" \
            -v "/www:/www" \
            certbot/certbot certonly -v --renew-by-default --webroot -w /www -d $domain -d www.$domain
    else
        # https://eff-certbot.readthedocs.io/en/stable/install.html#running-with-docker
        docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/var/log/letsencrypt:/var/log/letsencrypt" \
            -v "/www:/www" \
            certbot/certbot certonly -v --renew-by-default --webroot -w /www -d $domain
    fi

    # copy certificate links to a known file path and extention.
    cp /etc/letsencrypt/live/$domain/fullchain.pem /etc/letsencrypt/live/$domain/crt.crt
    cp /etc/letsencrypt/live/$domain/privkey.pem /etc/letsencrypt/live/$domain/key.key
done

# use minionfs to stage all certificates onto the salt master.
salt-call cp.push_dir "/etc/letsencrypt/live/" glob='*.crt'
salt-call cp.push_dir "/etc/letsencrypt/live/" glob='*.key

So the key is the -v mounts, I needed one for my webroot of /www & one for logs.

This is different or not assumed in the official guide notes.

Additionally on Ubuntu 22.04 LTS in Linode, I was not any to use the -v mounts until I ran these commands:

sudo su - root

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

The mount command never persists across reboots so you will want the following /etc/fstab entry:

Brian Amedro ended up modifying grub to avoid loading the certain systemd subsystem which were found to cause us the trouble:

https://github.com/docker/for-linux/issues/219#issuecomment-817318014

For now i will place the mkdir & mount commands into the renew script since the fstab solution doesn't work

because the directory doesn't exist, gets deleted on each reboot so it isn't present during boot mount time.

/etc/fstab

cgroup    /sys/fs/cgroup/systemd    cgroup    defaults

Vertically Scaling GitLab Server For As Cheap As Possible

2022-01-05T19:13:00-05:00

Today's essay acts as a power-up love story for the underdog.

A living document & quickstart for:

bootstrappers
small business under 99 employees
solo ops or devs
entrepreneurs
hackers
tinkerers

You also deserve a quick start to a GitLab Server power house!

Regardless of my intended audience, this strategy should scale to 500-1,000 concurrent engineers on the smallest of the instance classes which satisfy the current GitLab Server "minimum requirements".

https://docs.gitlab.com/ee/install/requirements.html

Why GitLab Server?

GitLab is open source (not black box)
GitLab CI is a game changer
GitLab CI is extendable to any workflow
GitLab Server wants 4G of memory

Expected Results

This guide prescribes the following setup:

A Physical or Virtual Machine with at least 4G of RAM
Docker installed
Docker container started via docker-compose to manage a monolithic Omnibus installation of GitLab Server
A Cronjob to backup GitLab Server to /tmp
A Cronjob to collect Gitlab Server backup tarball & /etc/gitlab/gitlab-secrets.json
A strong recommendation to test the complete backup and recovery process at least once to prepare for when times get rough.

GitLab Server Omnibus In Docker

The installation & configuration is documented using SaltStack to make it easy to spin up GitLab Servers.

To install SaltStack on the new host dedicated for GitLab Server, Run the following:

wget -O - https://bootstrap.saltstack.com | sudo sh -s -- stable;
echo "master: salt.example.com" >> /etc/salt/minion.d/custom.conf;
sudo service salt-minion restart

On the instance designated as salt-master, accept the new salt-minion key:

sudo salt-key -a git2.unturf.com
The following keys are going to be accepted:
Unaccepted Keys:
Git2.unturf.com
Proceed? [n/Y] y

and make sure communication is established:

sudo salt "git2*" test.ping
git2.unturf.com:
    True

and run a highstate to configure that new host instance:

sudo salt "git2*" state.highstate

Once the command returns, assuming we had no errors or failures we have docker installed and an Omnibus Gitlab Server booted inside a container.

Summary for git2.unturf.com
--------------
Succeeded: 104 (changed=69)
Failed:      0
--------------
Total states run:     104
Total run time:   194.431 s

The GitLab Server's internal services take over 3 minutes to come online.

Disaster Recovery

Prepare for real disaster by practicing the entire backup and restore process.

As extra homework, it is high advisable to spin up a 2nd host for GitLab Server in order to test the backup and restore process from start to finish:

https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab

GitLab CI is a game changer.: Gitlab is like Circle CI 2.0 only not black box.
GitLab CI is extendable to any workflow: but you should keep it simple, learn to use the tool as intended, and rethink legacy workflows rather than port.

Russell Open Sources Remarkbox & MakePostSell into Public Domain!

2021-12-23T12:08:00-05:00

Merry Christmas & Happy Holidays, Everyone!

My wife prompts students with this fun holiday writing idea:

"If you could gift the world anything, what would it be?"

I love to read responses in the comments!

As for me, I have wonderful news, my dream gift to the world will come true this year!

For the holiday season, I AM gifting to the public domain my most essential code in hopes to trigger a positive trend of doing in 2022!

"in 2022, dream & do!"

---

Each project has a link to guide on how to access the source code.

https://git.unturf.com/engineering/remarkbox/remarkbox

https://git.unturf.com/engineering/make-post-sell/make_post_sell

In 2021, I imagined many possible paths, yet all seemed to lead to one word:

OPEN

I will continue to operate the SaaS under Remarkbox and MakePostSell trademarks & operations will fall under unturf.

I will take an additional role of BDFL in regards to:

onboarding volunteers to maintain our common code in perpetuity
tie breaking

It feels vulnerable to share code without team support. Think from end & this feeling will pass.

Risk:

Alice may finally read the code & find vulnerabilities, burning a hole between us crippling our communication, taking down the system.

Solution:

More environments in wild, more eyes on problems, more options communicated.

Risk:

The masses may fork & divide the perfect circle.

Solution:

We volunteer to grow & multiply our systems, not worry about forks. We avoid focus on those who ignorantly mutilate or divide, instead focus on projects and forks which sprout, multiply, & thrive!

---

At a high level the unturf. stack currently consists of the following:

And of course we dogfood:

For analytics we self host our own:

Plausible

If the documentation works, people should be able to use the services without understanding the fundementals of each keyword listed above.

Pull requests welcome.

---

The writing of this essay has unfolded liberation in me and so, I speak words in favor of Truth, Freedom, and Love.

I AM now free to Grow, Explore, Document, and Multiply!

I love you, have a great day reader!

GNUnet GNS Nameserver Operator Notes, Quickstart, & Cheatsheet

2021-12-03T18:21:00-05:00

We GROW with Truth, Freedom, Love.

"You have an ally".

We scroll through the official GNUnet handbook, everything is covered in details, sections have examples. Seems verbose at first glance but also familiar .

Freedom of speech is under attack.

I AM not scared, we understand the technology we have and the technology we need. We will rescue ourselves and GROW together.

GNS right NOW!

Why?

Our Collective's Internet Nameservice (DNS) is UNDER ACTIVE ATTACK by groups of governments. Multitiple reports of DNS Name registrars breaking registrants Internet Domains on the behalf of NON-HUMAN ENTITIES.

To avoid any erosion of our Freedom of speech, we will use GNS to make sure our web domains continue to map to our resources for people looking for us.

I AM a seasoned 17 year Bind9 DNS administrator. I had my start on FreeBSD 5.0, virtualized over the years, now on Ubuntu VMs running on "the cloud" & local hypervisors hosting on broken laptops.

We know how to scale DNS, it sort of just works once you figure out the config files.

So let's figure out GNS and GROW a new Internet Culture, eh?

---

We pull down the source code using git:

git clone https://git.gnunet.org/gnunet.git

We see a lot of C code, interesting choice.

cd gnunet
[fox@blanka gnunet]$ ls -hal src/dns/
total 144K
drwxrwxr-x. 1 fox fox  362 Dec  3 18:09 .
drwxrwxr-x. 1 fox fox  834 Dec  3 18:09 ..
-rw-rw-r--. 1 fox fox 9.3K Dec  3 18:09 dns_api.c
-rw-rw-r--. 1 fox fox 1.3K Dec  3 18:09 dns.conf.in
-rw-rw-r--. 1 fox fox 2.2K Dec  3 18:09 dns.h
-rw-rw-r--. 1 fox fox  126 Dec  3 18:09 .gitignore
-rw-rw-r--. 1 fox fox  11K Dec  3 18:09 gnunet-dns-monitor.c
-rw-rw-r--. 1 fox fox 7.1K Dec  3 18:09 gnunet-dns-redirector.c
-rw-rw-r--. 1 fox fox  32K Dec  3 18:09 gnunet-helper-dns.c
-rw-rw-r--. 1 fox fox  35K Dec  3 18:09 gnunet-service-dns.c
-rw-rw-r--. 1 fox fox  14K Dec  3 18:09 gnunet-zonewalk.c
-rw-rw-r--. 1 fox fox 2.2K Dec  3 18:09 Makefile.am
-rw-rw-r--. 1 fox fox 7.5K Dec  3 18:09 plugin_block_dns.c
-rwxrwxr-x. 1 fox fox 1.6K Dec  3 18:09 test_gnunet_dns.sh

GNUnet 0.15.0 released: https://www.gnunet.org/en/news/2021-08-0.15.0.html

GNS First-come-first-served GNUnet top-level domain ".pin" zone key and website updated

Wow, Ok - We want to reserve our favorite names, right?

How do we generate our zone key`?

A section in the GNUnet handbook which seems to cover zones and keys:

https://docs.gnunet.org/handbook/gnunet.html#First-steps-_002d-Using-the-GNU-Name-System

To register a top-level domain of the .pin we need to generate a public/private keypair for each zone.

But before we do that we need to install GNUnet and configure the tool.

Ok, I would prefer some up-to-date binaries than the source tree.

Fedora package looks old and abandoned, same with Ubuntu, & Alpine...

I AM dreaming of a lightweight Alpine docker container to pass around the nets with with GNUnet prebaked in. Maybe a project for another day.

It seems like the NixOS package manager has an uptodate version 0.15.3!

To install GNUnet binaries, we must first learn how to install nix package manager.

Installing nix package manager, run the following:

# reference: https://nixos.org/download.html
curl -L https://nixos.org/nix/install | sh

Once installed try to install GNUnet, like this:

nix-env -iA nixpkgs.gnunet

Errors complaining about certificates might be an easy fix:

source /home/fox/.nix-profile/etc/profile.d/nix.sh

For example the install script placed the following into my ~/.bash_profile:

if [ -e /home/fox/.nix-profile/etc/profile.d/nix.sh ]; then . /home/fox/.nix-profile/etc/profile.d/nix.sh; fi # added by Nix installer

Once you get all that sorted you should have installed GNUnet!

To verify try to interact with ~/.nix-profile/bin/gnunet-config.

Battleship Operational, A Carrier Has Arrived.

[fox@blanka russell.ballestrini.net]$ gnunet-config --version
gnunet-config v0.15.3 release

Alright now we must somehow configure zones and public/private keypairs with this tool.

The handbook gives us this as an example, and we break down the anatomy of the command and outputs.

gnunet-config -s gns -o .myfriend -V PUBLIC_KEY

Let's try passing --help

[fox@blanka russell.ballestrini.net]$ gnunet-config --help

gnunet-config [OPTIONS]
Manipulate GNUnet configuration files
Arguments mandatory for long options are also mandatory for short options.
  -b, --supported-backend=BACKEND
                             test if the current installation supports the
                               specified BACKEND
  -c, --config=FILENAME      use configuration file FILENAME
  -d, --diagnostics          output extra diagnostics
  -F, --full                 write the full configuration file, including
                               default values
  -f, --filename             interpret option value as a filename (with
                               $-expansion)
  -h, --help                 print this help
  -L, --log=LOGLEVEL         configure logging to use LOGLEVEL
  -l, --logfile=FILENAME     configure logging to write logs to FILENAME
  -o, --option=OPTION        name of the option to access
  -r, --rewrite              rewrite the configuration file, even if nothing
                               changed
  -S, --list-sections        print available configuration sections
  -s, --section=SECTION      name of the section to access
  -V, --value=VALUE          value to set
  -v, --version              print the version number
Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

Ok so it seems like this command simply edits config files for GNUnet.

Ok but where are these enigmatic config files?

This is a living document we will continue the story toward a resolution and simple quickstart guide!

Leave comments below!

Copying files between cloud object stores like S3 GCP and Spaces using Boto3

2021-05-12T18:50:00-04:00

tl;dr if you just want something like aws s3 cp cli, try gsutil rsync.

At work one key item of team's sprint is properly utilizing and securing Google Cloud Platform (GCP).

For one of my projects, I'm learning GCP's Object Store called Google Cloud Storage (GCS).

I have prior experience using AWS S3 and Digital Ocean Spaces via aws s3 CLI and boto3 and I've even blogged in the past about some pretty advanced topics, like Pre-signed GET and POST requests. Pre-signing allows the client to perform specific actions on specific private objects, which is great for short time-to-live link downloads or direct browser uploads into the object store, all without compromising data security (hampering link sharing and preventing replay attacks)

At work, learning how to utilize and secure the GCS object store is critical to our teams interopability between clouds. We want to be able to store security artifacts into a single GCS bucket and sync all data to our main AWS S3 bucket as a long term archive.

Working with GCS using Boto3

Given my prior experience with boto3 I decided to test interoperability between it's s3 client and GCS.

For this test I created the following via the GCP Console:

IAM Service Account (with GCS Editor and GCS Viewer roles)
GCS Bucket
Service Account HMAC (linked to the new IAM Service Account)

I also set the default GCP project which is important because by default boto3 is not configured to pass the x-amz-project-id HTTP header.

At this point I have an access key and secret key (hmac) which in theory I may pass to my existing code to communicate with GCP in a similar way to how I communicate with S3 or Spaces.

Armed with this information I ran the following test and it worked:

# Reference:
# https://cloud.google.com/storage/docs/migrating#storage-list-objects-s3-python

import boto3

def list_gcs_objects(google_access_key_id, google_access_key_secret, bucket_name):
    """Lists GCS objects using boto3 SDK"""
    # Create a new client and do the following:
    # 1. Change the endpoint URL to use the
    #    Google Cloud Storage XML API endpoint.
    # 2. Use Cloud Storage HMAC Credentials.

    client = boto3.client(
        "s3",
        region_name="auto",
        endpoint_url="https://storage.googleapis.com",
        aws_access_key_id=google_access_key_id,
        aws_secret_access_key=google_access_key_secret,
    )

    # Call GCS to list objects in bucket_name
    response = client.list_objects(Bucket=bucket_name)

    # Print object names
    print("Objects:")
    for blob in response["Contents"]:
        print(blob["Key"])

If you are working with many GCP projects, you'll want to figure out how to configure boto3 to pass the x-amz-project-id. I found a good example reference but have not put it to practice: https://github.com/boto/boto3/issues/2251

The TL;DR is AWS doesn't have a concept of projects or default projects so this header is something GCP introduced for interopability.

GCP IAM first impressions

From a surface level look GCP IAM appears less complicated than AWS IAM but also less granular. I'm not sure how I feel at this point but I'm excited to see what the rest of my team uncovers when it comes to permissions, roles, and best practices.

Working with GCS using a CLI

Now that we covered boto3 (Python) interoperability between object stores, I started looking at some other simple CLI workflows which typically utilize aws s3. In the case of GCP the prefered CLI is gsutil.

The subcommand gsutil rsync in particular caught my eye as a simple way to setup a cross cloud object store synchronization!

For example:

gsutil rsync -d -r gs://my-gs-bucket s3://my-s3-bucket

For my next test, I'd like to try to setup a cronjob style automation to trigger gsutil rsync to copy and sync data from GCP GCS into AWS S3 for our long term security and governance artifacts, likely using a Gitlab CI Pipeline on a schedule.

Contents

Working with GCS using Boto3
GCP IAM first impressions
Working with GCS using a CLI

I tried to install GitLab on TrueNAS and failed

2020-10-25T14:28:00-04:00

Ok, so a month ago I changed employment and the new company uses GitLab exclusively for centralized code version control system.

This is my first time using GitLab and my first projects was integrating a dou/cloudmapper with a GitLab runner on a schedule.

After a couple weeks I've learned how GitLab runners work and wrote a wrapper to run cloudmapper concurrently so that each accounts is collected in parallel (the basic logic is running docker exec on a custom entrypoint and then backgrounds the task in a bash loop over each account configured in cloudmapper's config file)

GitLab runners are awesome.

I've used CircleCI for the better part of 3 years, having lifted the better part of 60 code bases from an internal app called conveyor to CircleCI 1.0 and performing CircleCI 2.0 conversions at Remind.

CircleCI 2.0 is awesome...

But GitLab runners are not black box which makes them so powerful.

Granted not all organizations need or even want this much power, but I'm a nerd and I want it, so I decided I also want to run GitLab at home.

My first thought was to try to just run GitLab on a VM on my SmartOS host called mbison, but that's a bit janky of a setup since it's an T420 ThinkPad after all. Also I learned the recommended minimum memory footprint of GitLab is 4G so a 4 of 16G allocation without even discussing a separate KVM for the runner. It feels weird that a single rails application would need _that_ much memory.

Anyways, I have a FreeNAS FreeNAS-11.3-U5 host at my house with 46G of memory and a bunch of idle cores, So I wondered if they have a plug-in for GitLab... I'm sure they do by now.

After a bit of research I found that iXsystems renamed the FreeNAS codebase to TrueNAS and version TrueNAS-12.0-RELEASE has a GitLab plugin! Sweet!

Thankfully I remembered that back in 2019 I replaced my FreeNAS USB boot device with an SSD so it should be a safe operation to attempt a TrueNAS upgrade on a Saturday morning since I'd have the weekend to try to recover if things went sour. Fun side story, I ran into trouble last year trying to upgrade FreeNAS train because I didn't have room for the upgrade because I was using a USB drive as the boot disk and it had like 5 previous version on the 8G file system.

The kind people in the freenode #FreeNAS channel helped me switch back to an older version using SSH and told me I should not boot from a USB drive which was a thing I picked up from back in the FreeNAS 6 or 7 days. Boy were they correct, I moved the OS to an SSD and the server boots _WAY_ faster now, plus I have more capacity to try new versions.

Anyways, here is a fun command to determine when a ZFS pool was created, I used this to get the exact date and time I moved from USB to SSD for this blog post:

root@guile[~]# zpool history -i freenas-boot | grep "create pool"

2019-11-12.13:00:30 [txg:5] create pool version 28; software version 5000/5; uts  11.2-STABLE 1102500 amd64

So weilding this confidence, I switched FreeNAS trains in the UI and clicked upgrade, following the prompts to create a backup of my NAS config. After the server rebooted, FreeNAS was now TrueNAS. I logged into the web UI to look around and make sure my data was still there.

Everything seemed fine, so I went to the plug-in tab and attempted to install GitLab. That failed with an error.

So I decided oh well, I'll try spinning up an Ubuntu VM on the TrueNAS box, apparently FreeBSD uses bhyve for that.

Attempting to start the VM does nothing in the UI, it appears like it will start but then the page refreshes and the VM is stopped.

I did eventually find the error message, by running

tail -f /var/log/libvirt/*/*

/usr/sbin/bhyve -c cpus=1,sockets=1,cores=1,threads=1 -m 4096 -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3:0,ahci,cd:/mnt/downloads/iso/ubuntu-20.04.1-live-server-amd64.iso -s 30:0,xhci,tablet -s 2:0,ahci -s 5:0,virtio-net,tap1,mac=00:a0:98:4f:98:13 -s 4:0,virtio-blk,/dev/zvol/downloads/gitlab-vcrg2z -s 31,lpc -l com1,/dev/nmdm1A -s 29,fbuf,vncserver,tcp=0.0.0.0:48009,w=1024,h=768 1_gitlab
25/10/2020 17:49:58 Listening for VNC connections on TCP port 48009
25/10/2020 17:49:58 Listening for VNC connections on TCP6 port 48009
ROM boot failed: unrestricted guest capability not available
fbuf frame buffer base: 0x941e00000 [sz 16777216]

The important part is:

ROM boot failed: unrestricted guest capability not available

It seems if you ask bhyve to use the UEFI bootloader it will require UG support in the Intel CPU : (

I attempted to switch to the grub bootloader but in that case I get the following error message:

File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1414, in do_update
  raise verrors
middlewared.service_exception.ValidationErrors: [EINVAL] vm_update.devices.3.dtype: VNC only works with UEFI bootloader.

and the TrueNAS UI does not let you disable VNC on VMs at this point...

Another limitation of a missing UG CPU flag is bhyve may only allocate 1 core, and 1 vcpu to a VM.

Anyways I'm out of luck, I can't use the plugin which uses jails due to a strange error and no other hints as to the issue and byhve / TrueNAS not supporting VMs for my CPU...

UPDATE I recently upgraded my TrueNAS server hardware to an old Dell PowerEdge R720 R720xd with two Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz each with 8 Physical Cores and 128G of ECC Memory! Booyah!

I was able to flash the included Dell H710 Mini RAID card into IT mode for ZFS support using the awesome guides and tools found here: https://fohdeesha.com/docs/perc/

I now have GitLab running on a byhve instance and also a dedicated runner instance. It's working great and really fun to have this extremely powerful gear in my home lab!

WeeChat on-boot in a tmux session

2020-09-15T13:32:00-04:00

Chronicles of a washed up systems administrator.

In the basement, M. Bison (mbision.foxhop.net.), a T430 with a cracked screen runs quietly with his lid closed, acting as a SmartOS hypervisor to 3 Solaris derived zones and 4 KVM ubuntu guests.

One of these guests is the oldest of the bunch and his name is Akuma (akuma.foxhop.net.).

Akuma and I go way back, over 13 years now. At one point Akuma was a physical machine, who ran FreeBSD and then later Ubuntu 6.04 with it's own guest kernel virtual machines.

Akuma has been "home base" for as long as I remember. A place where shit got done. A place to perform operations. A place to "jump" from with SSH.

It makes sense that Akuma would evolve and gain many roles over the years.

Akuma performs DNS caching and forwarding for my LAN, it acts as the internal authoritative Nameserver for foxhop.net., and is the Salt Master for all my hosts both internally (hosted at my house) and externally in the "cloud".

Akuma uses tmux with default settings to allow for a "remote shell" vibe.

This allows me attach to my session running on Akuma from anywhere in the world.

Only problem is, Akuma reboots weekly when M. Bison triggers a complete backup of all guests. Backups are stored on Guile, the FreeNAS server, uncompressed since disk space is cheap and this process allows for downtime, however I try to limit it.

Since Akuma reboots each week, I needed a way to create tmux sessions on boot if I wanted to continue to use it as my "home base", and I did so using SaltStack:

create-tmux-session-on-boot:
  cron.present:
    - comment: "create a new tmux session on system boot"
    - name: /bin/bash /home/fox/new-tmux
    - identifier: "create-tmux-session-on-boot"
    - special: "@reboot"
    - user: fox
    - require:
      - user: fox

This salt state results in the following crontab -l entry:

# create a new tmux session on system boot SALT_CRON_IDENTIFIER:create-tmux-session-on-boot
@reboot /bin/bash /home/fox/new-tmux

And of course I what tutorial wouldn't be complete without a script! (/home/fox/new-tmux) :

#!/bin/bash
# The -A flag makes new-session behave like attach-session if session-name
# already exists; in this case, -D behaves like -d to attach-session.
tmux new-session -d -A -s "remote-shell" "weechat-curses"
# also create a couple windows to use as "remote shells".
tmux new-window
tmux new-window
# finally attach to the new session!
tmux a -t "remote-shell"

Thanks for joining me on this escape into the world of computers.

This website is hosted on Ryu (ryu.foxhop.net) another Ubuntu guest running on M. Bison, humming away downstairs on that T430 with a cracked screen, but thats a story for another time.

Please feel free to comment below if you have suggestions for this post.

Dreaming of unlimited home computer storage capacity

2019-12-15T09:52:00-05:00

Unlimited computer storage capacity is a common science fiction trope and fun thought experiment.

What would be possible if we could potentially store near infinite data at your house in a normal sized desktop computer?

Even better using today's tech, what ideas could we dream up?

This post will explore ideas as we surf the bleeding edge of home computing.

Idea 1

Live stream everything in your life in HD and store it locally at your house as a hyper real augment reality. A person could store and recall any event that he or she has witnessed using a headcam with wifi syncing capabilities.

One of the many new features within the iPhone 6s is the ability for the 12MP camera on the back to record 4K video. It’s a big selling point for Apple, but what about storage space?

According to a video published by MKBHD, just how much a one-minute long video shot in 4K will take up in your iPhone 6s’ storage has been discovered. According to the screenshot captured by Ryan Jones (@rjones) of the hands-on video, a one-minute video captured at 720p HD at 30fps will take up 60MB of space on the iPhone 6s. At 1080p HD and 30fps, it’s 130MB of space. A 1080p HD video at 60fps will take up 200MB of space. And, finally, the 4K video at 30fps will take up 375MB of space.

reference: http://www.iphonehacks.com/2015/09/iphone-6s-4k-video-space.html

If we were streaming at 4k for 24 hours, it would be:

>>> 60 * 24 * 375
540000

540,000MB or 540GB or .54TB per day of streaming everything.

A years worth of 4k streaming from some guys headcam would consume:

>>> 365.25 * .54
197.235

or 197TB worth of capacity.

With today's technology and hardware available as home computing, including NAS systems, streaming 4k of a persons headcam would be possible... but not really practical without advancements in hard drive density.

A 6TB drive is $100, and by my math, we would need 34 drives (not really even counting for redundancy):

>>> 197.235 / 6
32.8725

So the cost of a years worth a capacity would be:

(197.235 / 6) * 100
3287.25

or $3,287 for 34 x 6TB drives.

It would fit in an oversized NAS, just barely... in another section I'll figure out what hard drive density is needed to make 4k video at 30fps more practical.

Ok so, 4k streaming is currently not practical, so how about 1080p HD at 30fps? Could we do that with today's home tech?

One minute of 1080p HD at 30fps is 130MB of space.

If we were streaming at 1080p at 30fps for 24 hours, it would be:

>>> 60 * 24 * 130
187200

187,200MB or 187GB or .187TB per day.

A years worth of non-stop 1080P streaming at 30fps off some guys headcam would consume:

>>> 365.25 * .184
67.235

Or about 68TB per year (rounded up).

Like before a 6TB drive is $100, and by my math, we would need 12 drives (not really even counting for redundancy):

>>> 67.206 / 6
11.201

so the cost of a years worth a capacity would be:

>>> 12 * 100
1200

or $1,200 for 12 x 6TB drives.

12 drives should easily fit in a mid-range home NAS. That said, a mid-range home NAS holding 12 drives is a clunky machine for most home owners.

It might be possible, but likely not practical without a support plan, in home repairs, and on-going maintenance.

It would likely require having a home NAS specialist in every region to offer up white gloved support for anyone who would be willing to pay for it.

How could we make this even easier with today's tech?

We could lower the quality even more...

Streaming 720P at 30fps you would need 36T per year or about 6 x 6TB drives for about $600 plus a NAS.

So at this time, if seems practical and economical to stream 720P at 30fps every minute of the year.

What would we need to build to make all this data useful?

Well we would need a way to index all this footage, and likely edit out useless footage.

We would need at least two headcams with wifi sync, so that we could have one charging while we use the other.

We would need software to sync the footage to the NAS, like syncthing.

Besides that, all we need is some software to host the videos, edit, and index them to be searchable.

Unity Collab not working on Fedora because of an invalid CA certificate path

2019-11-08T12:19:00-05:00

This issue blocked me for about two days and prevented me from collaborating with another engineer at gumyum co-operative video game studio.

The other engineer's environment was Windows and he was able to interact with collab and our shared project owned by our shared org. My environment was Fedora 30 and was not working with collab, however I was able to download the project files.

The issue?

Apparently Unity naively assumes that a Linux system's copy of the root CA certificates is located /etc/ssl/certs/ca-certificates.crt. This is the default location on Debian and Ubuntu based systems but not Fedora or Redhat, which uses the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem file.

The work around is simple, create a soft link between the paths:

sudo ln -s /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/ssl/certs/ca-certificates.crt

Please come back again soon to learn more about the games we are building over here at gumyum!

How to fallback to an older FreeNAS version on update failure

2019-10-31T17:06:00-04:00

try to log in as root locally, I needed to press ctrl-alt-f7 to get to a log in prompt.

Once logged in as root, I used the beadm command.

For example, beadm list and beadm activate <old-version>.

Finally reboot.

I followed this process and then got my FreeNAS server to boot again.

From the GUI I was able to create myself a config backup.

I then burned a new FreeNAS CD and used that to install a fresh new install on my USB drive and then once that booted I used the GUI to restore my config.

Installing Unity on Linux

2019-09-25T18:55:00-04:00

Russell, why are you installing Unity? I thought you were an open source developer and homegrown video game engine builder?!

The TL;DR we are starting a video game co-operative called gumyum. The basic idea is to build a large video game company owned completely by members. There will be no employees and most revenue after paying taxes, operational expenses, and marketing, will be divided among members.

We are actively looking for members, specifically engineers who want to build games. We already have a distributed team of 4 artists and 2 engineers who meet on Slack as we progress toward delivering our first two games.

Adopting Unity, at least for our first couple games, will lower the barrier of entry for other engineers when joining the co-operative.

Installing Unity On Linux

To get started first we will create a Unity Account and install the Unity Hub. You need both in order to aquire a free licence for personal use.

The Unity Hub software will help you download various Unity versions, manage your licence, and manage game projects.

You should go to the Unity website and created an account.

I found the Linux Unity Hub installer in this thread and downloaded it to my ~/Downloads directory.

Next I opened a terminal and made the file executable, and executed it.

cd ~/Downloads
chmod 755 UnityHub.AppImage
./UnityHub.AppImage

From there I needed to sign in and request a licence. I chose the free option since at this point we have a $0 revenue project.

On the Installs tab in Unity Hub you may download various stable and LTS (long-time-supported) versions of Unity. I suggest using one of these stable versions, I accidentally grabbed a non-stable one and all I had was pink screens inside of Unity!

Anyways, now you may start a Unity project. Unity Hub seems to store files locally and also allows you to sync to the cloud. This enables easy collaboration on the same project.

Thats all for now, please check back for more posts about gumyum!

Pre-signed GET and POST for Digital Ocean Spaces

2019-07-12T16:01:00-04:00

A pre-signed request grants a semi-trusted user temporary access to a private resource.

Let's unpack that statement ...

Pre-signed means, we bless a specific action on a specific private resource for a short duration of time.

Semi-trusted means, we have authenticated the user, but we don't trust them to have full access to our private resources.

That still seems a bit generic ...

No worries, our next example will appear more concrete:

Today we will pre-sign HTTP POST and GET requests to grant a semi-trusted user temporary access to objects in a private Digital Ocean Space.

Pre-signing allows the client to perform specific actions on specific private objects while still protecting us from link sharing and replay attacks.

The following examples use Python (Boto3), and Curl.

Before we start, you should reference the official Boto3 Presigned Urls documentation.

Pre-signed GET or Download

Allow a semi-trusted user the ability to download a specific file named my-object.zip from a private Digital Ocean Space named example-bucket for a short duration of 30 seconds.

Save this code into a file named pre_sign_get_test.py and run it with python pre_sign_get_test.py.

The result will be a URL that you may redirect a user to. For testing you should be able to load the URL into a web browser and download the file.

import boto3
from botocore.client import Config

# Initialize an S3 session/client to talk to DigitalOcean Spaces.
session = boto3.session.Session()
client = session.client(
    "s3",
    # configure the region you created the space in.
    region_name="nyc3",
    # configure the region you created the space in.
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    # configure your access key (generate this on the dashboard).
    aws_access_key_id="DOXXXXXXXXXXXXXXXXXX",
    # configure your secret key (generate this on the dashboard).
    aws_secret_access_key="DOO/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
)

signed_get_object_url = client.generate_presigned_url(
    ClientMethod='get_object',
    Params={
        'Bucket': 'example-bucket',
        'Key': 'my-object.zip',
    },
    ExpiresIn=30
)

# print the pre-signed GET request URL for downloading the object.
# You may redirect the user to this URL when they click a "download" button.
print(signed_get_object_url)

Pre-signed POST or Upload

Allow a semi-trusted user the ability to upload a specific file named my-object2.zip into a private Digital Ocean Space named example-bucket for a short duration of 60 seconds.

Save this code into a file named pre_sign_post_test.py and run it with python pre_sign_get_test.py.

The result will be a cURL command that you may run on your console for testing.

In a real situation, you would want to pass these parameters to the client and let it perform the authenticated upload. The file never hits your server and is uploaded directly to the private Space.

IMPORTANT: parameter order matters! The file parameter must be last.

import boto3
from botocore.client import Config

# Initialize an S3 session/client to talk to DigitalOcean Spaces.
session = boto3.session.Session()
client = session.client(
    "s3",
    # configure the region you created the space in.
    region_name="nyc3",
    # configure the region you created the space in.
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    # configure your access key (generate this on the dashboard).
    aws_access_key_id="DOXXXXXXXXXXXXXXXXXX",
    # configure your secret key (generate this on the dashboard).
    aws_secret_access_key="DOO/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
)

signed_post = client.generate_presigned_post(
    Bucket="example-bucket",
    Key="my-object2.zip",
    ExpiresIn=60,
)

params = []
for key, value in signed_post["fields"].items():
    params.append('--form "{}={}"'.format(key, value))

print("curl " + " ".join(params) + ' --form "file=@my-object2.zip;filename=my-object2.zip" ' + signed_post["url"])

Hybrid Hot Water Heater Saves 69 Percent On Energy Consumption

2019-02-05T09:28:00-05:00

Disclosure: This post contains affiliate links. See full disclosure page here.

Exactly one year ago I fufilled a longtime childhood dream when I invested in roof top solar on my house in Eastern Connecticut (Zone 6b).

Over the past 12 months, I have religiously tracked my families energy consumption using both a "Kill A Watt" and a Curb.

Along the way, I have found my largest energy consumer (abuser).

Heating water.

My traditional electric water heater had a monthly operating cost of $85. Worst yet, during some winter months, 100% of my solar production was soaked up by heating water!

I demanded a better way!

... and I found one ...

Hybrid Electric Hot Water Heat Pump

My new unit (A.O. Smith 50 Gallon Voltex Hybrid Electric Heat Pump) transfers ambient heat from the air into water.

The technology is relatively old, basically an air conditioner or refrigerator run in reverse. Instead of dumping the cold into the tank, it dumps hot and as a by-product produces cold air.

Yes that's right, this unit will suppliment my summer cooling because it will dump cold air while it produces hot water. The heat pump also serves as a dehumidifier, so I no longer run one of those!

The install was mostly identical to a normal water heater, however there are a couple extra requirements:

The unit needs at least 400 square feet to pull ambient air
The unit produces condensate, similar to a dehumidifier or air conditioner, which needs to drain somewhere.

Additionally the heat pump fan produces a bit of sound, not terrible, and seems quiet to me.

If you have a traditional electric water heater, you meet the extra requirements, and you consider yourself an environmentalist, you need to get a hybrid water heater.

Finances

These units currently run between $1200 - $1500 but I'll break down the complete finances of my project.

+ $1,149 purchase price
+    $65 delivery
+   $100 pipes and fittings
-   $300 instant rebate
-   $200 mail-in rebate
-   $200 craigslist 3 year old unit
===================================
+   $614

I replaced the unit myself, with help from my father-in-law. I don't own a truck, so I decided to have it delivered for $65. I bought $100 worth of pipe and fittings, mostly because I don't want to learn how to sweat copper, so I use sharkbite fittings which cost more but are fast and simple to use.

My state offers an instant $300 in store rebate, plus a $200 mail in rebate.

The old unit was only 3 years old (came with the new house purchase) so I sold it on craigslist for $200 (MSRP was $899).

Return on investment

Ok, so $614 to replace a working unit, am I crazy? What is the expected ROI?

Well, I've only had the unit for a month, however my energy consumption for heating hot water went from $85/mo to $30/mo.

Let's just round down to $50/mo savings.

Ok so in 12 months the unit will pay for itself. Even without the rebates you're still looking at only a 2 year ROI before the unit starts paying dividends.

That's not all.

I don't need to run a separate dehumidifier in my basement because the new unit removes humidity as a by-product!

... but wait there's more!

This unit also produces cold air as a by-product which means free supplimental cooling during the summer!

A water heater, dehumidifier, and air conditioner all-in-one!

What is the catch?

There is no catch, but you do have to agree on a small trade off, hot water recovery time.

Instead of using 4,500 watts for 45 minutes (.75 hours), a hybrid hot water heat pump uses 350 watts for 3 hours.

Trading the slightly longer hot water recovery times, gives you:

a significant reduction on your energy bill
dehumidification
supplimental air conditioning / cooling

# traditional.
4500 * .75 # == 3375 watts or 3.375 kWhr to recover

# hybrid heatpump.
350 * 3 # == 1050 watts or 1.050 kWhr to recover

The decrease in consumption means a huge savings of 69%!!!

# percentage decrease.
(3375 - 1050) / 3375 # == 69% !!!

So what are you waiting for? Honestly, if you are thinking about going solar, you should tackle this project first, right now!

Contents

Hybrid Electric Hot Water Heat Pump
Finances
Return on investment
What is the catch?

AWS nvme to block mapping

2019-01-19T14:50:00-05:00

Recently at work I transitioned our fleet from Ubuntu 14.04 LTS to Ubuntu 18.04 LTS. During the process I noticed an issue with our newer generation AWS EC2 "nitro" based instance types (specifically c5.2xlarge).

AWS was presenting my root block device as /dev/nvme1n1 and my data device as /dev/nvme0n1. For obvious reasons, this seemingly random and out-of-order assignment breaks my provisioning scripts.

After much research and deep thought, I came up with a solid solution.

I found a barely documented tool called ebsnvme-id on the official Amazon Linux AMI and wrote a wrapper (nvme-to-block-mapping) to iterate over all possible combinations of /dev/nvme[0-26]n1 to create a symlink to the block mapping selected when we launch the EC2 instance.

Since we have control over the block mapping, we end up with consistent and known symlink which we may confidently pass to our provisioning scripts when the time comes to partition, format, and use the block devices.

To save you time, I have added these scripts to this blog post!

Please consider donating here if my work has helped you!

Place the following two scripts into your AMI under /usr/sbin/ and trigger the wrapper just once prior to using the devices (don't worry the wrapper script is idempotent, running it more than once won't break anything).

nvme-to-block-mapping

/usr/sbin/nvme-to-block-mapping (click for file):

#!/bin/bash

# for details:
# https://russell.ballestrini.net/aws-nvme-to-block-mapping/

# this will create a symlinks like:
#
#     /dev/nvme1n1 -> /dev/xvdh
#
# these ebs block device paths are set by stacker and assumed by ansible.
#
# if the device is non ebs, it will use the following mapping:
non_ebs_mapping=("/dev/sdb1" "/dev/sdc1" "/dev/sdd1" "/dev/sde1" "/dev/sdf1" "/dev/sdg1")

# nvme0n1 uses ${non_ebs_mapping[0]} (the 0 index item in the array)
#
#     /dev/nvme0n1 -> /dev/sdb1
#
# nvme3n1 uses ${non_ebs_mapping[3]} (the 3 index item in the array)
#
#     /dev/nvme3n1 -> /dev/sde1
#

# why we only iterate from 0 to 26:
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html
for i in `seq 0 26`; do
    nvme_block_device="/dev/nvme${i}n1"

    # skip any nvme paths which don't exist.
    if [ -e  $nvme_block_device ]; then

        # get ebs block mapping device path set by stacker (or base AMI).
        mapping_device=$(/usr/sbin/ebsnvme-id ${nvme_block_device} --block-dev)

        # if the mapping_device is empty, it isn't an EBS device so
        # we will use the non_ebs_mapping to translate the device.
        if [[ -z "$mapping_device" ]]; then
            mapping_device="${non_ebs_mapping[$i]}"
        fi

        # if block mapping device path does not start with /dev/ fix it.
        if [[ "$mapping_device" != /dev/* ]]; then
            mapping_device="/dev/${mapping_device}"
        fi

        # if the block mapping device path already exists, skip it.
        if [ -e $mapping_device ]; then
            echo "path exists: ${mapping_device}"

        # otherwise, create a symlink from nvme block device to mapping device.
        else
            echo "symlink created: ${nvme_block_device} to ${mapping_device}"
            ln -s $nvme_block_device $mapping_device
        fi
    fi
done

ebsnvme-id

/usr/sbin/ebsnvme-id (click for file):

#!/usr/bin/env python2.7

# Copyright (C) 2017 Amazon.com, Inc. or its affiliates.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#    http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the
# specific language governing permissions and limitations under the
# License.
#
# Reference:
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html

"""
Usage:
Read EBS device information and provide information about
the volume.
"""

import argparse
from ctypes import *
from fcntl import ioctl
import sys

NVME_ADMIN_IDENTIFY = 0x06
NVME_IOCTL_ADMIN_CMD = 0xC0484E41
AMZN_NVME_VID = 0x1D0F
AMZN_NVME_EBS_MN = "Amazon Elastic Block Store"

class nvme_admin_command(Structure):
    _pack_ = 1
    _fields_ = [("opcode", c_uint8),      # op code
                ("flags", c_uint8),       # fused operation
                ("cid", c_uint16),        # command id
                ("nsid", c_uint32),       # namespace id
                ("reserved0", c_uint64),
                ("mptr", c_uint64),       # metadata pointer
                ("addr", c_uint64),       # data pointer
                ("mlen", c_uint32),       # metadata length
                ("alen", c_uint32),       # data length
                ("cdw10", c_uint32),
                ("cdw11", c_uint32),
                ("cdw12", c_uint32),
                ("cdw13", c_uint32),
                ("cdw14", c_uint32),
                ("cdw15", c_uint32),
                ("reserved1", c_uint64)]

class nvme_identify_controller_amzn_vs(Structure):
    _pack_ = 1
    _fields_ = [("bdev", c_char * 32),  # block device name
                ("reserved0", c_char * (1024 - 32))]

class nvme_identify_controller_psd(Structure):
    _pack_ = 1
    _fields_ = [("mp", c_uint16),       # maximum power
                ("reserved0", c_uint16),
                ("enlat", c_uint32),     # entry latency
                ("exlat", c_uint32),     # exit latency
                ("rrt", c_uint8),       # relative read throughput
                ("rrl", c_uint8),       # relative read latency
                ("rwt", c_uint8),       # relative write throughput
                ("rwl", c_uint8),       # relative write latency
                ("reserved1", c_char * 16)]

class nvme_identify_controller(Structure):
    _pack_ = 1
    _fields_ = [("vid", c_uint16),          # PCI Vendor ID
                ("ssvid", c_uint16),        # PCI Subsystem Vendor ID
                ("sn", c_char * 20),        # Serial Number
                ("mn", c_char * 40),        # Module Number
                ("fr", c_char * 8),         # Firmware Revision
                ("rab", c_uint8),           # Recommend Arbitration Burst
                ("ieee", c_uint8 * 3),      # IEEE OUI Identifier
                ("mic", c_uint8),           # Multi-Interface Capabilities
                ("mdts", c_uint8),          # Maximum Data Transfer Size
                ("reserved0", c_uint8 * (256 - 78)),
                ("oacs", c_uint16),         # Optional Admin Command Support
                ("acl", c_uint8),           # Abort Command Limit
                ("aerl", c_uint8),          # Asynchronous Event Request Limit
                ("frmw", c_uint8),          # Firmware Updates
                ("lpa", c_uint8),           # Log Page Attributes
                ("elpe", c_uint8),          # Error Log Page Entries
                ("npss", c_uint8),          # Number of Power States Support
                ("avscc", c_uint8),         # Admin Vendor Specific Command Configuration
                ("reserved1", c_uint8 * (512 - 265)),
                ("sqes", c_uint8),          # Submission Queue Entry Size
                ("cqes", c_uint8),          # Completion Queue Entry Size
                ("reserved2", c_uint16),
                ("nn", c_uint32),            # Number of Namespaces
                ("oncs", c_uint16),         # Optional NVM Command Support
                ("fuses", c_uint16),        # Fused Operation Support
                ("fna", c_uint8),           # Format NVM Attributes
                ("vwc", c_uint8),           # Volatile Write Cache
                ("awun", c_uint16),         # Atomic Write Unit Normal
                ("awupf", c_uint16),        # Atomic Write Unit Power Fail
                ("nvscc", c_uint8),         # NVM Vendor Specific Command Configuration
                ("reserved3", c_uint8 * (704 - 531)),
                ("reserved4", c_uint8 * (2048 - 704)),
                ("psd", nvme_identify_controller_psd * 32),     # Power State Descriptor
                ("vs", nvme_identify_controller_amzn_vs)]  # Vendor Specific

class ebs_nvme_device:
    def __init__(self, device):
        self.device = device
        self.ctrl_identify()

    def _nvme_ioctl(self, id_response, id_len):
        admin_cmd = nvme_admin_command(opcode = NVME_ADMIN_IDENTIFY,
                                       addr = id_response,
                                       alen = id_len,
                                       cdw10 = 1)

        with open(self.device, "rw") as nvme:
            ioctl(nvme, NVME_IOCTL_ADMIN_CMD, admin_cmd)

    def ctrl_identify(self):
        self.id_ctrl = nvme_identify_controller()
        self._nvme_ioctl(addressof(self.id_ctrl), sizeof(self.id_ctrl))

        if self.id_ctrl.vid != AMZN_NVME_VID or self.id_ctrl.mn.strip() != AMZN_NVME_EBS_MN:
            raise TypeError("[ERROR] Not an EBS device: '{0}'".format(self.device))

    def get_volume_id(self):
        vol = self.id_ctrl.sn

        if vol.startswith("vol") and vol[3] != "-":
            vol = "vol-" + vol[3:]

        return vol

    def get_block_device(self, stripped=False):
        dev = self.id_ctrl.vs.bdev

        if stripped and dev.startswith("/dev/"):
            dev = dev[5:]

        return dev

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Reads EBS information from NVMe devices.")
    parser.add_argument("device", nargs=1, help="Device to query")

    display = parser.add_argument_group("Display Options")
    display.add_argument("-v", "--volume", action="store_true",
            help="Return volume-id")
    display.add_argument("-b", "--block-dev", action="store_true",
            help="Return block device mapping")
    display.add_argument("-u", "--udev", action="store_true",
            help="Output data in format suitable for udev rules")

    if len(sys.argv) < 2:
        parser.print_help()
        sys.exit(1)

    args = parser.parse_args()

    get_all = not (args.udev or args.volume or args.block_dev)

    try:
        dev = ebs_nvme_device(args.device[0])
    except (IOError, TypeError) as err:
        print >> sys.stderr, err
        sys.exit(1)

    if get_all or args.volume:
        print "Volume ID: {0}".format(dev.get_volume_id())
    if get_all or args.block_dev or args.udev:
        print dev.get_block_device(args.udev)

Contents

nvme-to-block-mapping
ebsnvme-id

Yuletide Trains and Homegrown Video Games

2018-12-25T17:15:00-05:00

Each holiday season I find myself drawn to a side passion of mine.

While some build model trains and others create Christmas light shows with synchronized music you'll find me on my sofa where I build, explore, and tinker on my own video game engine.

Maybe the sound of wrapping paper tearing acts as a trigger, but I only prioritize time on my game engine during the holiday season.

Do you have a specific hobby that you do during the holiday season but not during other parts of the year?

Why am I working on Christmas every year - Am I a work-a-holic?

Work or Play.

Why do I absorb myself into this seasonal hobby? Am I escaping? Or am I making use of down time to work on a project that I am too occupied to even think about during the rest of the year?

For me, building my game on Christmas is fun, not work. Similar to how fishing while camping is fun, not work.

What is the last activity a commercial fisherman would do "for fun" on Christmas? Fishing.

What is fun for one person might be work for another. It is relative and depends on context.

I suspect that most people experience a tipping point, when an activity switches from fun to work.

I think, for an average person, this moment occurs when the answer to the following question changes:

Do you rely on the activity for the majority of your income?
- if yes, the activity often feels like work
- if no, the activity often feels like fun

Work and Play?

Now that's not to say you cannot find fun while at work, and vice versa. Personally, I aim to land a perfect balance between the two with anything I do.

In my situation, I use the computer "professionally" each work day, for 7-10 hours.

If I was in the games industry, I doubt I would find much fun in building a video game on Christmas. But I'm not in the games industry and I have more fun building games than playing games.

That is to say, for me building a game feels fun while playing a game feels like work. Crazy right?

Even though the activities of my "day job" overlap with the activities involved with building my game, there are enough differences. These differences make one feel like work while the other feel like fun.

I'm not making an excuse for working during Christmas; I honestly have a blast coding and tinkering on my game.

I love getting stumped and learning. I love showing different ideas to my boys and seeing their eyes light up. I love to see how fast I can implement a small suggestion one of them have. I love how there are no rules and only my own skills and understanding can block me.

While my family is occupied with their new holiday gifts, I use my free time to explore creative ideas on a non-stressful personal project. A project where the stakes are low and the pleasure is high.

Show me the game already!

This years game engine modifications:

ability to save the game state to a json file
ability to load the game state from a json file
added a few algorithms for auto-generating maps
refactored some common code (so I'm not reapeating myself)
implemented an action charge / recharge system

Here is a short video of the current state of the engine. Let me know what you think in the comments.

I also took the time to learn how to make pixel art using GIMP, a useful skill for making game assets.

Who else makes games during Christmas?

I reached out to a number of indie game developers while formulating this blog post and I had a solid conversation with Maverick Peppers a developer who runs a software company called ProtoComplete

He was working during Christmas on his game while making pudding and drinking white russians (recipe in the foot notes). He arrived at the conclusion that I was working from a passion-oriented mindset while he was from a goal-oriented mindset.

Maverick doesn't rely on the income from his games, so I asked "why are you 'working' on Christmas?" He explained that because he often has a type-a or "goal mindset", he has trouble relaxing until he finishes whatever he is working on. In a sense, he was working Christmas so that he could relax.

As for myself, building games is a passion-oreiented process, I don't rely on the income and I use it to unwind.

We talked about how people can approach activities with either:

type-a (goal-oriented mindset)
type-b (passion-oriented mindset)

The fun part is people are often both, depending on the activity or project. For example Maverick has a passion mindset when making music and DJing, but always takes a goal mindset when it comes to business.

Do you have a specific hobby that you do during the holiday season but not during other parts of the year?

Footnotes

Unwrapping My Christmas Commit History

Looking at my commit history, the first iteration of my current game engine was saved on Jan 01, 2014 with a few commits until Jan 19, 2014, at which point nothing until Dec 25, 2014 (Christmas itself) when I sprinted until Jan 10, 2015.

The next year, I must have hacked on something else, with no changes until Oct 09, 2016 where I had two commits.

Like clockwork on Dec 25, 2016 (Christmas) I tried to fix a regression in the engine's collision and intersection code. I left myself some breadcrumb comments to help me debug in the future... Nothing in 2017.

Today is Christmas 2018 and finally I have a work around for the regression I was looking into from Christmas 2016!

Porting SFML Rect from C++ to Python

This work around ports the Rect intersection logic of SFML from C++ to pure Python and avoids the following error message:

terminated by signal SIGSEGV (Address boundary error)

def get_rect_intersection(r1, r2):
    """
    Accept two sfml.graphics.Rect objects.
    Return a new sfml.graphics.Rect of the overlap or None.
    """

    # We allow Rects with negative dimensions, so handle them correctly.

    # Compute the min and max of the first Rect (r1).
    r1_min_x = min(r1.left, r1.left + r1.width)
    r1_max_x = max(r1.left, r1.left + r1.width)
    r1_min_y = min(r1.top, r1.top + r1.height)
    r1_max_y = max(r1.top, r1.top + r1.height)

    # Compute the min and max of the second Rect (r2).
    r2_min_x = min(r2.left, r2.left + r2.width)
    r2_max_x = max(r2.left, r2.left + r2.width)
    r2_min_y = min(r2.top, r2.top + r2.height)
    r2_max_y = max(r2.top, r2.top + r2.height)

    # compute the intersection boundaries.
    i_left   = max(r1_min_x, r2_min_x)
    i_top    = max(r1_min_y, r2_min_y)
    i_right  = min(r1_max_x, r2_max_x)
    i_bottom = min(r1_max_y, r2_max_y)

    # if the intersection is valid (positive non zero area),
    # then there is an intersection.
    if i_left < i_right and i_top < i_bottom:
        return sfml.graphics.Rect((i_left, i_top), (i_right - i_left, i_bottom - i_top))

White Russian Recipe

1/4 cup distilled водка
1/4 cup Kahlua coffee rum
1/2 cup cream

Older versions of the game engine

Some videos of older versions of this game engine.

index

Work or Play.
Work and Play?
Show me the game already!
Who else makes games during Christmas?
Footnotes

All Local Heros need a Gig Side Kick

2018-08-03T11:20:00-04:00

Five months ago during my preparation and ramp up to gardening season, I started thinking about what it would be like to have a partner, a side kick, a secretary to keep me honest and on task.

Somebody who could review old journal notes from previous years and give me hints about what seeds I should be starting, what worked well, what didn't, and why.

I started thinking I'm likely not alone and that other hobby and market gardeners likely feel the same way. I then expanded this, likely anyone bootstrapping a small business "enterprise" could use a side kick. Small craft brewers, artists, sign makers, the list goes on and on.

As you might know I'm deeply interested in systems like computer science and permaculture. Computer scientists often try to model complex natural systems. Often natural systems give us clues on how to make computers perform certain tasks.

One reoccuring theme I find between computer science and permaculture is the idea of inputs and outputs.

Brewing wine from berries requires the following inputs: boiling water, berries, sugar, yeast, and yeast nutrient. Combined all these inputs together in the right way, wait for a certain time, and the system returns an a tasty, alchoholic, product with a long shelf life as an output. The spent berries go straight into the compost for next years garden. Natural systems and unnatural systems model after natural ones, have no waste by-products (pollution). Every output of one process is always an input of another.

Only when we mess up the natural order of organic systems do we end up with polution. Polution is simply a holistically bad output. For example, when we artifically produce outputs, like plastics there is no natural system to take it as an input, it becomes polution and makes the system sick.

To fight off the production of unnatural outputs, like single use plastics, we need to create other human managed systems to use the pollution as inputs. For example, I'm insulating my shed with bags of single use polyethylene wrappers that wrap my families purchases. I'm keeping the valuable (although un-natural) resource out of the waste stream and using it to shrink the tempurature swing deltas in my shed. When we model the material world as inputs and outputs we can learn how to best use what we have on hand.

We can boost each of our local economies by providing proper tools to the local makers.

This is why today I'm announcing GigSideKick to flip the script on globalization and large enterprise giants.

GigSideKick is a "field" journal app to track inputs and outputs. The primary goal is to collect the right amount of data with the least amount of friction, have an intuitive user interface, and a user experience which mostly stays out of the way. In addition to tracking inputs and outputs, the user will optionally set ETAs on processes set in motion which will power reminders.

Each local hero deserves the unfair advantage that a smart and capabile Side Kick provides.

This type of input/output journal will not interfere with the task at hand, and would only require a few button presses. The data created from such a journal is important for the user's future self, and if properly aggregated and crowd sourced, the data will enable a powerful recommendation engine.

For example, a person subscribing to a "garden" knowledge pack might get the following recommendation: "gardeners in your area are known to typically start on average of 24 zucchini seeds this month, which typically result in 18 viable zucchini seedlings in 6 days", "How many will you start?"

None

12

24

48

Other

In the background, the app would also track human labor (which itself is valuable input), track inventory of parts, work-in-process, and finished goods. Finally the app's data would power a local marketplace to help the maker trade and sell finished products. After all, if you produce a surplus of apples without a system in place to use them, the apples will end up as an input to the compost pile instead of the bellies of deserving humans. : )

I believe we must give power back to local communities and economies. We, the side gig heros and local businesses operators, need all the help we can get to compete in today's global economy.

Local makers have many often un-exploited unfair advantages. We do not need to ship our goods which means we should have lower margins. We need to figure out how to market our products (outputs) to local people looking for our superior goods.

Another often untapped unfair advantage is the sale or trade of our by-products. We can even trade our waste! For example as a gardener, I would gladly trade a heaping bowl of same day cut, "beyond organic" salad greens, for your bucket of leaves, coffee grounds, sticks, sawdust, or grass clippings! It's honestly the timeless addage of "one mans trash is another mans treasure". Your waste is a very useful input to the correct system and local people trading outputs is by far the best thing we can do for our planet.

I plan to document my journy of building this vision, I hope you join me.

Like always, please feel free to leave comments below, I always respond.

Running DynamoDB Local service container on CircleCI 2.0

2018-07-18T11:56:00-04:00

tl;dr use a custom entrypoint in your CircleCI 2.0 config to limit Java memory to 1G.

The new CircleCI 2.0 docker configuration supports a "primary image" (listed first) which runs all the "steps" as well as zero or many "service images" (listed subsequently). The "service images", although not running in the same container as the primary present as if local to the primary.

It sort of feels like mixing many docker containers together.

Enough talk, in this example, I show how easy it is to run DynamoDB Local in a separate container but at the same time present it to the primary as 127.0.0.1:8000:

version: 2
workflows:
  version: 2
  tests:
    jobs:
      - test

jobs:

  test:

    working_directory: /go/src/github.com/remind101/r101-myapp

    docker:

      # Primary container image where all the steps run.
      - image: circleci/golang:1.8

      # Service container image made available to the primary container at `host: localhost`
      - image: dwmkerr/dynamodb:41
        # custom entrypoint to limit RAM to 1G to prevent OOM on CircleCI 2.0.
        # https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macosexecutor
        entrypoint: ["java", "-Xmx1G", "-jar", "DynamoDBLocal.jar"]

    steps:
      - checkout
      - run: make test_verbose

Troubleshooting

When I first tried to use this image, any test which tried to reach out to DynamoDB via 127.0.0.1:8000 had the following exception:

Test Panicked
Error creating table group_associations: RequestError: send request failed
caused by: Post http://127.0.0.1:8000/: dial tcp 127.0.0.1:8000: getsockopt: connection refused
/usr/local/go/src/runtime/panic.go:489

It seems the docker-dynamodb (DynamoDB Local) container failed to start and exited like this:

Initializing DynamoDB Local with the following configuration:
Port: 8000
InMemory:     false
DbPath:       null
SharedDb:     false
shouldDelayTransientStatuses: false
CorsParams:   *


Exited with code 137

Apparently Exit code 137 is a classic Docker + Java error code caused by an OOM (out of memory).

By default, projects on CircleCI build in virtual environments with 4GB of RAM but this is shared and Java acts greedy when inside a container so it needs a limit by adding -Xmx1G to the entrypoint!

References:

Pyramid SQLAlchemy bootstrap console script with transaction.manager

2018-06-15T10:54:00-04:00

So I've struggled for a while with the best way to properly setup a console script for my SQLAlchemy Pyramid apps.

I use the Pyramid Cookiecutter Alchemy to setup my projects and as such, I do not have a global and thus importable DBSession object. Instead my database session is attached to the request on creation.

Anyways, here is my recipe:

import argparse

from pyramid.paster import bootstrap, setup_logging

from remarkbox.models import invalidate_all_node_cache_objects


def get_arg_parser():
    parser = argparse.ArgumentParser(
        description="Invalidate all NodeCache objs to force recomputation."
    )
    parser.add_argument("-c", "--config", default="development.ini")
    return parser


def main():
    parser = get_arg_parser()
    args = parser.parse_args()
    setup_logging(args.config)

    # use bootstrap context manager to prepare app and request,
    # next use the resulting request's transaction manager!
    with bootstrap(args.config) as env, env["request"].tm:
        request = env["request"]
        invalidate_all_node_cache_objects(request.dbsession)

This pattern should help you solve this error:

NoTransaction error when using bootstrap in script

Quickstart to DKIM Sign Email with Python

2018-06-04T09:03:00-04:00

For a long time I have put off DKIM signing email sent from my web services because I couldn't wrap my head around all the indirection Postfix requires to make it work.

Honestly, I put it off for over 5 years...

Today a thought sprung into my head:

"Could I sign Email at the application level before passing to Postfix?"

As you may know, I primarily use the Python programming language, so I did some research and found a reference to a single library called dkimpy (previously pydkim). The codebase started over 10 years ago and appeared stable and mature.

The part that sold me was that dkimpy seemed compatible with the two Python standard library modules which I already use:

email which I use to prepare messages
smtplib which I use to transport messages to Postfix running on localhost

One issue I did have with dkimpy was the complete lack of examples or even a quickstart guide.

For this reason, I have written this post!

The Missing dkimpy Quickstart Guide

To install dkimpy you may use pip (requirements.txt) or in my case I added it to my setup.py.
Generate a public / private keypair. Don't let this step trip you up, the process easy. In a Unix -like environment you may run the following commands to create the keys.

generate private key (I name my file after the domain and DKIM selector I plan to use).
```
openssl genrsa -out remarkbox.com.20180605.pem 1024
```
generate public key from the private key.
```
openssl rsa -in remarkbox.com.20180605.pem -out remarkbox.com.20180605.pub -pubout
```
Install the public key (.pub) as a DNS TXT record, where the record name ("selector") is 20180605._domainkey and the value body is:
```
v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDcplYPRsqIFwXuggtH2XgQDMX+e+6sGnWdV8ld/FR9zgRAxB+DeiCEVooVvYt2JRZUEokgDFvys82Q+JTbN4qHNz19bdcBGrnTsnIFaQYpgeQYmPLdDtcWRKzTYMRNCnRmmEXyGv7WIDcaTapIq9NFgLmy1QT7ZTxuNjQtDB/2LwIDAQAB;
```
You may choose any selector, I happen to like to use YearMonthDay. Additionally you will substitute your public key in place of mine. Put each the line of the public key on a single line in the TXT record.
On each of my application servers I store my private portion of my DKIM key in /etc/dkim/remarkbox.com.20180605.pem. You may store your key any where on the filesystem that is accessible to the user or group running the application.

This is how I used to send Email with Python:

import smtplib

from email.mime.multipart import MIMEMultipart

from email.mime.text import MIMEText

# catch socket errors when postfix isn't running...
from socket import error as socket_error


def send_email(
    to_email,
    sender_email,
    subject,
    message_text,
    message_html,
    relay="localhost"
):
    msg = MIMEMultipart("alternative")
    msg.attach(MIMEText(message_text, "plain"))
    msg.attach(MIMEText(message_html, "html"))
    msg["Subject"] = subject
    msg["From"] = sender_email
    msg["To"] = to_email
    # TODO: react if connecting to postfix is a socket error.
    s = smtplib.SMTP(relay)
    s.sendmail(sender_email, [to_email], msg.as_string())
    s.quit()
    return msg

This is how I now send DKIM signed Email with Python:

# ref: https://github.com/russellballestrini/miscutils/blob/master/miscutils/mail.py

import dkim
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
# catch socket errors when postfix isn't running...
from socket import error as socket_error


def send_email(
    to_email,
    sender_email,
    subject,
    message_text,
    message_html,
    relay="localhost",
    dkim_private_key_path="",
    dkim_selector="",
):

    # the `email` library assumes it is working with string objects.
    # the `dkim` library assumes it is working with byte objects.
    # this function performs the acrobatics to make them both happy.

    if isinstance(message_text, bytes):
        # needed for Python 3.
        message_text = message_text.decode()

    if isinstance(message_html, bytes):
        # needed for Python 3.
        message_html = message_html.decode()

    sender_domain = sender_email.split("@")[-1]
    msg = MIMEMultipart("alternative")
    msg.attach(MIMEText(message_text, "plain"))
    msg.attach(MIMEText(message_html, "html"))
    msg["To"] = to_email
    msg["From"] = sender_email
    msg["Subject"] = subject

    try:
        # Python 3 libraries expect bytes.
        msg_data = msg.as_bytes()
    except:
        # Python 2 libraries expect strings.
        msg_data = msg.as_string()

    if dkim_private_key_path and dkim_selector:
        # the dkim library uses regex on byte strings so everything
        # needs to be encoded from strings to bytes.
        with open(dkim_private_key_path) as fh:
            dkim_private_key = fh.read()
        headers = [b"To", b"From", b"Subject"]
        sig = dkim.sign(
            message=msg_data,
            selector=str(dkim_selector).encode(),
            domain=sender_domain.encode(),
            privkey=dkim_private_key.encode(),
            include_headers=headers,
        )
        # add the dkim signature to the email message headers.
        # decode the signature back to string_type because later on
        # the call to msg.as_string() performs it's own bytes encoding...
        msg["DKIM-Signature"] = sig[len("DKIM-Signature: ") :].decode()

        try:
            # Python 3 libraries expect bytes.
            msg_data = msg.as_bytes()
        except:
            # Python 2 libraries expect strings.
            msg_data = msg.as_string()

    # TODO: react if connecting to relay (localhost postfix) is a socket error.
    s = smtplib.SMTP(relay)
    s.sendmail(sender_email, [to_email], msg_data)
    s.quit()
    return msg

As always, leave a comment or contact me for questions or help.

Fulfilling Childhood Dreams: Solar

2018-03-16T08:03:00-04:00

Ever since I was an 8 year old boy I have wanted solar. I remember reading about the environment and alternative energy sources in a monthly "socal studies" flyer my school subscribed our classroom to. I questioned, even at my young age, why the world wasn't actively switching over to solar, water, and wind!

25 years later I have finally fulfilled one of my childhood dreams: I installed a 11.375kWh solar system on my roof.

This post will act as an overview of the stuff I learned about solar, I hope you learn something too!

Contents

Solar Monitoring
Home Energy Monitoring
Solar Financing
Contact me

Disclosure: This post contains affiliate links. See full disclosure page here.

Solar Monitoring

The system is composed 35 x Panasonic 325 watt panels (VBHN325SA16). The panels are isolated into 3 arrays ("strings") of 8, 13, and 14.

>>> # 35 panels x 325 watts ~= 11.375kWh
>>> 35 * .325
11.375

Each panel has an "optimizer" which communicates to the centralized SolarEdge inverter installed in my basement near the breaker box. The optimizers allow each array to work efficently even when a panel is shaded or broken.

Here is the layout of the panels on my house:

Additionally the optimizers emit energy production metrics to the inverter, which in turn forwards this data to "the cloud" (aka SolarEdge Monitoring System). From there, I can watch daily playbacks of the whole system.

For example, this playback from March 19th, 2018 was a good "solar day" as my family calls:

You can see how the sun rises to cover the 8 panels on the front of the house and then later moves to cover the panels on the back of the house.

SolarEdge Monitoring Dashboard also keeps track of various high level metrics. For example, I input my electricity rates with date ranges and the dashboard calculates the "lifetime revenue" of my solar system. I plan to track my ROI on this number!

Home Energy Monitoring

Most of my key insights into home energy actually come from another device I had installed called Curb Home Energy Monitor. Curb uses a bunch of non-invasive CT clamps on each breaker circut to gather consumption. Metrics are gathered and sent to "the cloud" and power some really cool realtime dashboards.

For example, this chart shows solar production versus consumption in dollars for the last 15 days:

As you can see, about 50% of my solar production is being consumed by heating hot water for a family of 5!

This feels absurd...

Update: I switched to a Hybrid Water Heater and wrote a post to show how I now use 69% less electricity when heating water!

Anyways, here is a short video showing the Curb user interface.

Sorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

The Curb is likely the most accurate home energy monitor on the market. The draw back is the cost of parts and labor; I used my solar installer's electrician and the total cost was $800.

The competition has less accuracy but I could have likely installed a Sense myself and saved on the labor cost.

Here are the two other brands I was looking at, for reference:

Solar Financing

When talking with Tesla, I fully expected to pay for my solar system in full with cash savings. It wasn't until I reached out to SunLight Solar did I change my strategy. SunLight urged me to take advantage of the ".99% CT Green Bank" finance option. I took out a loan for the whole project and dumped my cash savings into my home mortgage as a bulk payment to the principle.

Additionally, the CT Green Bank granted me $3,600 toward my project and the US federal government will grant 30% or $8,400 on my next tax return.

After all the incentives, the parts and labor of my system came in just under $20,000:

$32,000 - $3,600 - 8,400 = $20,000

Putting solar on my house actually opened up my financial options and diversified my portfolio!

I now have:

a power plant on my roof with an expected 9-10 year ROI; after 10 years I'll be generating wealth, capital, and positive "cash flow" in the form of energy
paid down my 4.125% house mortgage by $35,000; saving tens of thousands over the life of the loan
increased the value of my house by $20-30,000; this is an asset I can sell with or without my house
shielded or insulated myself from electricity rate hikes; who knows what electricity will cost in 5 to 10 years

Contact me

As always, please feel free to leave comments below. I live in New England so you may also contact me to setup a time to tour my setup and ask questions. I look forward to meeting you!

How-to Work From Home

2017-11-09T09:51:00-05:00

"I work from home" — a phrase I have uttered hundreds of times and is often met with instant amazement, envy, or jokes about pants. My goal for this book is to teach you the hacks I have learned in my career to help you land your dream job. If you feel stuck in the job market and you don't want to relocate, this is the book for you.

Chapter 1: The Road to Remote

So you have made up your mind, you want to work from home.

The first step to attaining this goal is to make it a priority.

The suggestions and advice in this book come from my life experiences. Not all of my experiences will apply to you but I hope that we might find, together, at least a single thread of universal truth. In order to know, I think it makes sense to briefly review the anatomy of my life from childhood until now.

I'm currently 32 years old, but I started working with computers at a young age. As an elementary schooler, I had a bright childhood friend named Brian Brennan who although younger than I, was my first mentor regarding computers. At age 9 we played lots of video games, original playstation and Nintendo 64, as well as countless hours of computer games on his Gateway computer.

At one point Brian learned how to make the computer work for him instead of other the other way around. Together we started learning HTML and Javascript. I didn't have the Internet on my family computer at the time, but my parents did have a WebTV so I started building HTML web pages for things I enjoyed and used the WebTV to upload my pages Geocities.

Flash forward to age 13, I finally had dial-up. Now we didn't pay for dial-up, in fact my parents didn't know I was "connected" as we used to call it, but I was. I had the "setup" credentials for our regional phone company — I was living the dream of free Internet!

I covertly routed speaker wire from our pantry closet to the computer room. Yes, I learned early on that in a pinch speaker wire can work as telephone wire, although I don't recommend it. This was one of many hacks I stumbled on. Hacks, the combination of science, trial & error, and creativity, are a common theme through out my life.

I spent my highschool career building HTML pages instead of powerpoints slides for school projects. At home I built my own computers out of parts, as so many of us did back then and gamed all night during LAN parties where we traded "warez" and "pr0n". I was 15 when we got an A-DSL connection (125kbps down / 15kbps up) and I started hosting my websites from my house on a Windows server hiding in my bedroom closet — later this server was reborn running FreeBSD 3.

Most of my highschool education was spent figuring out how to get out of class and either into the computer lab or the library with friends. We basically met as an informal computer and tech club. For more details about this era, checkout my long time friend Charles Hooper's blog post titled "How I hacked my high school".

When I was going to community college for Computer Science, I worked two part time jobs — I worked for my parents and separately at a large retail chain in the electronics department, where I slung "HDTVs". I was working 40 hours a week, doing 4 courses a semester, and I didn't have time for anything.

I started hosting my parent's small family business website which I coded by hand and I mostly resented my mall job. I hated how cruel companies were and how they actively shit on their employees. I spent my lunch breaks in Borders reading computer and business books and dreaming of my perfect job.

In between Calculus and selling TVs, I also found time to build the best damn real estate search engine a local agency in South Eastern Connecticut could buy. I was taken advantage of; I worked at least 500 hours on that site which was commissioned for under $500. I did however learn a lot about business and tech. I was brave and dumb and took on more then I could chew but it was successful. Hands down it was the best solution on the market at the time, this was long before Zillow existed.

While working in the retail store, I met George, an IT guy working for my region's pharmaceutical company. George referred me to a job making $18/hr migrating Windows 2000 computers to Windows XP. I accepted the contracting job thinking I was on the top of the world, I finally had a "real" computer job.

I did awesome work but what I didn't know was this world is cruel to contractors. They don't get sick time, they don't get vacation, and they certainly don't get respect. Contractors are fed lies and false promises and they are the first to get cut. Managers treat contractors like physical resources. It's honestly sick.

From there I moved to a technical call center helpdesk, also as a contractor, of a contractor, of a contractor of the DoD. If you follow the money, I was making $17/hr but at the top of the pyramid they were billing me out at $55/hr. After some time I managed to move onsite and successfully cut out one of the contractor layers.

I still lacked negotiation skills and confidence so my increase was only a dollar, bringing me to $18/hr as a desktop support tech. I fixed computer problems whether it was hardware or software and I was fast and could basically solve anything handed to me. I quickly transitoned to a desktop support engineer role where I learned how to package and push software to small fleets of computers. A couple years later I transitioned to the Unix server team.

Each step was a very big promotion as far as responsibility (I was one of two employees who designed and operated a multi million dollar Hitachi SAN) but my salary didn't increase proportionally. People were coming "off the street" for the same or lower position and making more then double I was. At the time I was working my ass off for $32k a year and hardly had enough for what my family needed. I was watching people get hired for the same job for $65k a year ... After being let down for so long, this was the tipping point, this is where I started to shift my thinking.

The only way to move up is to move out.

I felt stuck. I worked for both the big industries for my region and they both failed me. I knew I did not want to relocate but I also didn't want to work for companies with obscene priorities and worse culture.

I started moonlighting on side projects. I had so many flops and failures, to many to count. Finally I launched an idea with traction called LinkPeek. During the code development phase of building LinkPeek I became active in the open source community for the framework I was using called Pyramid.

I was networking, even though I didn't know what it was called at the time. I taught myself some tricks to career development and ended up meeting a hiring manager who appreciated my skill and was willing to take a chance with me working remote. I flew across the country to Santa Monica and had a bit of culture shock but I landed the position!

This victory would be the first of 4 remote positions I have held since finally taking the leap to "work from home".

Tune in to the next chapter where I describe how I positioned myself to win the interview.

webwords: a minimal viable web app with docker in as many languages as possible

2017-10-24T12:11:00-04:00

The companion webwords git repo lives here.

This project shows how to code the same minimal web app called webwords in as many different programming languages as possible. It also provides guides for building and running webwords as a docker image.

Contents

what is webwords
why is webwords
go
python
ruby
js
debugging

what is webwords

A simple web application whose spec accepts the following two query parameters —

keyword:: The keyword you want to search for.
target:: The URI target that you want to search.

The application always returns an HTTP 200 response with the string true or false depending on if the keyword is found in the target web page body.

For example, to see if the word potato exists on Remarkbox, put the follwing in a browser:

http://127.0.0.1:32779/?keyword=potato&target=https://www.remarkbox.com

Spoiler:: potato does exist on Remarkbox, : )
Note:: You will need to replace the port of 32779 with the port from the docker ps output.

why is webwords

Webwords started as a programming Kata to practice writing code in different programming languages. Each port of webwords should behave the same to make comparing and functional testing simple.

why did you choose this programming problem?

I think the spec of webwords is small enough for people new to any language to digest but complete in that it does something useful and demonstrates two common tasks: running an HTTP server and using an HTTP client.

Also, I needed a way to verify if a user had possession of a domain name for the comment service I'm building and chose to code this verification program as a micro service, first with Python and later with Go. The tiny end result was webwords.

Shortly after, during a hackathon I used webwords to learn how to build Docker images for various languages and formalized the idea into a single project.

What's next?

Webwords is for tinkering. If you want to add a version or touch up an existing version, send a PR. Maybe a future fork will show a guide for adding a cache layer or teach how to add logging or gather metrics.

go

To build the docker image:

cd go
docker build -t webwords-go .

To run a test container from the new image:

docker run -d -p 8888 webwords-go

python

To build the docker image:

cd python
docker build -t webwords-python .

To run a test container from the new image:

docker run -d -p 8888 webwords-python

ruby

To build the docker image:

cd ruby
docker build -t webwords-ruby .

To run a test container from the new image:

docker run -d -p 8888 webwords-ruby

js

To build the docker image:

cd js
docker build -t webwords-js .

To run a test container from the new image:

docker run -d -p 8888 webwords-js

debugging

If you're anything like me, your programs rarely compile or work properly on the first try. Just like with programming, a docker image will rarely build correct the first time so you will need to learn how to debug.

To debug, get the failed docker container's id:

docker ps --all

Once you have the id, you can run the following to see the error:

docker logs <container-id>

Debug the issue, fix your Dockerfile, and retry the build process until you have it working.

You can delete old attempts by running:

docker rm <container-id>

So You're Planning a Beta Test

2017-10-22T13:20:00-04:00

I'm running a beta for Remarkbox and here is my biggest take away:

Always collect information from potential customers as soon as possible. Catch customers while they pursue a solution to their problems.

What do I mean?

Well for starters, don't just ask for an email address, have them fill out a short survey right away. If you only ask for an email, your out of luck if you need more information; people do not often respond later.

I know, I know ... nobody likes to fill out surveys, so power-up their motivation with an incentive or discount. Use this survey to earn qualifed leads, remember quality over quantity.

Come up with a few questions. Keep most of the questions open ended to allow them guide the dialog. Later, use the responses to break the ice when you send one-on-one emails. Each lead will present a unique situation, so take advantage of this and keep your on boarding emails personal!

tl;dr - Gather interest for the Beta as soon as possible and ask questions up front as part of the sign up.

My Search for the Perfect Alternative Hosted Comment System

2017-10-08T10:47:00-04:00

Nearly 6 years ago I launched the beta for LinkPeek, a web page screenshot service. After all this time, I'm writing to share something new I've been passionately working on called Remarkbox.

I started working on Remarkbox back in 2014 to solve a problem I had after moving my Wordpress blog to a static site. I knew my readers would still want to communicate with me so I started the search for the perfect hosted comment system to promote discussion.

My research found solutions which would slow down my site, or worse, serve ads! That very day, I set out to build my own solution because -

"How long could it take to build a comment system?" ...

3 years later, I'm finally ready to share Remarkbox: a hosted comment service that embeds in your pages to keep the conversation in the same place as your content.

Remarkbox increases user engagement because it:

allows a visitor to discuss your content right away without an account
supports Markdown with real-time comment previews
supports deeply nested replies and has an orderly user interface

... and the best part - fast page load speeds and absolutely NO ADS!

I'm in the process of hand selecting group of users to help me test. Would you be interested in joining a list to hear more about it?

Yes! I would like to follow the Remarkbox journey.

Selenium grid on Kubernetes

2017-03-23T16:48:00-04:00

This post continues from where we left off on the Minikube guide. If you do not already have a Kubernetes cluster, you should read that first.

Selenium Grid allows you to build a cluster of Selenium nodes. Today we will create a Selenium cluster with 1 hub and 4 nodes on Kubernetes.

Contents

Selenium Hub
- Selenium Hub Overlearning
Selenium Node
- Selenium Node Overlearning
Selenium Scale

Selenium Hub

Let's launch the hub:

kubectl get pods
kubectl run selenium-hub --image selenium/hub:2.53.1 --port 4444
kubectl get pods

To access the deployment externally, we need to expose it:

kubectl get services
kubectl expose deployment selenium-hub --type=NodePort
kubectl get services

My deployment was exposed here:

http://192.168.99.100:31136/grid/console

To find out where your deployment was exposed:

minikube service selenium-hub --url

You may open this URI in a web browser.

Selenium Hub Overlearning

You can skip this section or try it for extra credit.

We may use kubectl exec for container introspection:

kubectl exec selenium-hub-3216163580-7pqtx -- ps aux

As you can see we have a java process running

kubectl exec selenium-hub-3216163580-7pqtx -- ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
seluser      1  0.0  0.1  18044  2688 ?        Ss   17:20   0:00 /bin/bash /opt/bin/entry_point.sh
seluser      7  0.1  3.1 2928868 64864 ?       Sl   17:20   0:13 java -jar /opt/selenium/selenium-server-standalone.jar -role hub -hubConfig /opt/selenium/config.json
seluser     87  0.0  0.1  34424  2856 ?        Rs   20:54   0:00 ps aux

We may also look at the config file and test the service internally:

# inspect the selenium config file.
kubectl exec selenium-hub-3216163580-7pqtx -- cat /opt/selenium/config.json

# see if selenium is really listening on port 4444.
kubectl exec selenium-hub-3216163580-7pqtx -- wget 127.0.0.1:4444 -O -

You can even shell into the container:

kubectl exec -it elenium-grid-3216163580-7pqtx -- /bin/bash

Exit out of the container and lets setup some Selenium Nodes!

Selenium Node

Lets spin up a Selenium Chrome node:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"
kubectl get pods

Kubernetes will use service discovery to resolve selenium-hub to the service (pods) running the hub!

If you refresh the hub browser window, you should see a connected Chrome Node, like this:

Selenium Node Overlearning

You can skip this section or try it for extra credit.

The first time I tried to launch a Selenium node and I had trouble.

I ran this:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1
kubectl get pods

The new pod went into status CrashLoopBackOff:

NAME                                    READY     STATUS             RESTARTS   AGE
selenium-grid-3216163580-7pqtx          1/1       Running            1          3d
selenium-node-chrome-4019562870-mcpfg   0/1       CrashLoopBackOff   6          6m

To troubleshoot, I used the following commands:

kubectl describe pod selenium-node-chrome

This command allowed me to review the Kubernetes level logs. Everything seemed healthy so next I looked at the Docker level logs:

kubectl logs selenium-node-chrome-4019562870-mcpfg
Not linked with a running Hub container

Ok, the error Not linked with a running Hub container appears when Selenium node cannot find the hub.

Docker has a --link flag to link containers together, Kubernetes doesn't have this. After some research, it seems --link manages ENV vars.

You can see the environment vars of a pod using this command:

kubectl exec selenium-hub-3216163580-7pqtx -- printenv

I learned that the selinum-node-chrome docker image expects some ENV vars and if it doesn't get them, it goes into a crash loop.

I reached out over IRC in the #kubernetes and #selenium channels to ask about the ENV vars needed. A really helpful user named smccarthy linked me to this:

https://github.com/kubernetes/kubernetes/tree/master/examples/selenium

Apparently one of the example Kubernetes clusters is a Selenium Grid setup!

Looking over the example, I found the ENV vars that the selenium-node containers expect: HUB_PORT_4444_TCP_ADDR and HUB_PORT_4444_TCP_PORT

Man, why would they put the port (4444) in the key?

Anyways, we pass these key/values when creating the container like this:

kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"

Selenium Scale

Now we can scale up and down the Selenium Grid cluster. Lets scale the deployment to 4 replica node-chrome pods.

kubectl get pods
kubectl scale deployment selenium-node-chrome --replicas=4
kubectl get pods

Finally, if you refresh the hub browser window, you should see 4 connected Chrome nodes!

If you liked this post, leave me a message in the comments!

Minikube

2017-03-21T12:38:00-04:00

Minikube allows you to run a self contained, single node, Kubernetes cluster on your workstation. Once installed and configured, you may use kubectl to interact with it, just like a production Kubernetes cluster.

Contents

install
- install kubectl
- install minikube
start
demo

Reference: https://kubernetes.io/docs/getting-started-guides/minikube/

install

Requirements: Virtualbox

install kubectl

Reference: https://kubernetes.io/docs/tasks/kubectl/install/

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin

install minikube

Reference: https://github.com/kubernetes/minikube/releases

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

start

Supported hypervisors: virtualbox, vmwarefusion, kvm, xhyve

minikube start --vm-driver=virtualbox

If this is your first time starting minikube, it will perform the following:

Starting local Kubernetes cluster...
Starting VM...
Downloading Minikube ISO 89.24 MB / 89.24 MB [==============================================] 100.00% 0s
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster

Also, you should see a new VM running in VirtualBox, like this:

To verify that kubectl is configured to use minikube look at the config file (~/.kube/config).

Also try running:

kubectl get nodes
kubectl get services

You can also connect to the VirtualBox guest using SSH to have a look around. In my case the Minikube VM was assigned 192.168.99.100.

ssh -i ~/.minikube/machines/minikube/id_rsa docker@192.168.99.100

You can see all the containers running with:

docker ps
ps aux

Exit out, you really don't need to interact at this level

Instead we will treat Minikube as a "real" Kubernetes cluster and only use the kubectl tool.

demo

create a deployment

In this example we create an echoserver cluster.

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080

this command will create -

1 deployment:

kubectl get deployments

1 replicaset:

kubectl get replicasets

1 pod:

kubectl get pods

To make the echoserver accessible externally, you need to expose the deployment, like this:

kubectl expose deployment hello-minikube --type=NodePort

The expose command creates -

1 service:

kubectl get services
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes      10.0.0.1     <none>        443/TCP          2d
hello-minikube  10.0.0.225   <nodes>       8080:31136/TCP   58m

To access the service, you connect to the Minikube's IP address on the exposed port.

In my case the Minikube VirtualBox IP is 192.168.99.100 and the exposed port is 31136 as listed above.

The minikube tool has a shortcut for this info, try:

minikube service hello-minikube --url
http://192.168.99.100:31136

Toss this into a web browser on your local machine and it should echo back!

scale a deployment

Scale up the deployment named hello-minikube by setting the number of replicas to 3:

kubectl scale deployment hello-minikube --replicas=3

verify:

kubectl get deployments
kubectl get pods

delete a deployment

trash this demo (delete the deployment, replicaset, pods, and service):

kubectl delete deployment hello-minikube

Nginx throw HTTP 503 maintenance JSON for all requests

2016-12-21T15:09:00-05:00

I found this technique after stumbling on Aaron Parecki's blog. You can read his post here:

Setting a custom json 503 error page in nginx with proper http and content-type headers

Lets pretend you have an API and you need to turn on maintenance for a major change. All your client's (phones, web frontends) know what to do when they get an HTTP 503 Error code with JSON body.

Now you want to cause all requests to get the following JSON -

/opt/maint/maint.json:

{
 "errorCode": "maintenance",
 "errorDesc": "We are currently performing scheduled maintenance. Please try again later."
}

The nginx configuration looks like this -

maint.conf:

server {
    listen          80 default;
    listen          [::]:80 default_server;
    server_name     _;

    root /opt/maint;

    add_header Retry-After 30 always;

    error_page 503 /maint.json;

    location / {
        # all API calls will miss try_files on purpose and return custom 503.
        try_files $uri =503;
    }

}

All API calls will miss try_files on purpose and return our custom 503 maint.json.

Proof that this method rocks, it supports GET, POST, PUT, UPDATE, and DELETE.

It also automatically serves the proper Content-Type header, in this case application/json -

curl -v -X POST --data "param1=value1&param2=value2" http://127.0.0.1:80/my-very-favorite-api-call

> POST /my-very-favorite-api-call HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.10.2
< Date: Wed, 21 Dec 2016 20:32:44 GMT
< Content-Type: application/json
< Content-Length: 150
< Connection: keep-alive
< ETag: "585addfe-96"
< Retry-After: 30
<
{
  "errorCode": "maintenance",
  "errorDesc": "We are currently performing scheduled maintenance. Please try again later."
}

Need to make sure you always set a custom header? Don't forget the always directive:

add_header Retry-After 30 always;

Capability driven Presentation

2016-12-18T18:21:00-05:00

A web page does not need to look the same on every browser or device. We cannot control the capabilities of a user's browser or device. As web designers, we have the duty to give the viewer the best experience possible. A user will come with what they have and we should do our best to accommodate.

TL;DR: Capability drives presentation.

Resilient Web Design

A resilient web page should:

have a single canonical URI
serve the same content, regardless of a viewer's capibilities
use the viewer's capibilities to gracefully enhance and/or downgrade the presentation of the content

For more background and history please read this free online book: https://resilientwebdesign.com/

Techniques

The js-only class

The js-only class

Purpose: hide HTML elements when Javascript does not work.

In this technique we:

use a <noscript> tag to nest a class named js-only
declare display: none in the js-only class to hide elements
apply the js-only class to any element which should hide without js

Note: The js-only class will only exists when Javascript does not work.

<noscript>
  <style>
  .js-only {display: none;}
  </style>
</noscript>

Now, suppose we have a notification message which clears when a user clicks 'x'. To clear the notification, we must use a Javascript onclick event handler.

Without the Javascript capability the notification cannot clear. Therefore, to prevent confusion we should remove the 'x' from the presentation.

For example, with Javascript the notification will look like this:

But without Javascript the 'x' is removed and the notification will look this:

To do this we assign the js-only class to our label holding 'x':

<div id="alerts">
{%- for message, level in request.session.pop_flash() %}
<div class="alert alert-{{level}}" onclick="this.style.display='none'" name="alert">
  <p>
    <label class="close js-only" alt="dismiss" title="dismiss">x</label>
    {{message}}
  </p>
</div>
{%- endfor %}
</div>

The content is the same, but we hide the interface based on the user's capability! This small change has a huge impact on the overall user experience.

We no longer present a broken button to the user, and I think that is worth the effort.

Build RPM or DEB packages for Node.js using Jenkins and FPM

2016-12-01T15:30:00-05:00

This blog post assumes you already have:

a Jenkins master and none or many build servers
FPM installed on the build servers
Node.js installed on the build servers

I add jenkins-build.sh in the root of the Node.js code repo:

# example usage: JOB_NAME=example-api BUILD_NUMBER=101 bash jenkins-build.sh

# download and build nodejs application and 3rd party modules.
npm rebuild
npm install -l

version=$(cat package.json | python -c 'import sys, json; print json.load(sys.stdin)["version"]')

# change directory down below checkout directory
cd ..

# create an RPM using fpm.
#   JOB_NAME     = jenkins job name
#   BUILD_NUMBER = jenkins auto incremented build number
/usr/local/bin/fpm \
    -s dir -t rpm \
    --name "$JOB_NAME" \
    --iteration "$BUILD_NUMBER" \
    --version "$version" \
    --vendor "Example, Inc" \
    --rpm-user="node"  --deb-user="node" \
    --rpm-group="node" --deb-group="node" \
    --directories "/opt/$JOB_NAME" \
    "$JOB_NAME/$JOB_NAME.service=/lib/systemd/system/" \
    "$JOB_NAME=/opt/"

# make a copy of the rpm with generic name (without version or build).
cp *.rpm "$JOB_NAME.rpm"

# mv both the rpm and genericly named rpm to the workdir so jenkins can archive it.
mv *.rpm "$JOB_NAME"

Then in the Jenkins job, I have the following execute shell tasks:

# clean out old rpms.
rm -f *.rpm

# download all 3rd party Node.js modules and package into an installable RPM.
bash jenkins-build.sh

I then simply upload the rpm to an S3 repo which is accessible to my API hosts.

The same basic strategy may be used for other languages with subtle differences to the build script.

Register Super Powers with Pyramid add_request_method

2016-11-24T15:30:00-05:00

The Pyramid web application framework uses a request object to hold state regarding an inbound HTTP connection. A view must accept a request object as the first argument which makes it always available to our views and templates.

This behavior rocks, but Pyramid makes it even better by allowing us to enrich the request object!

As an example, lets pretend we want to randomly generate an integer from 1 to 999 and attach it to each request:

from pyramid.config import Configurator

from random import randint

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application."""

    # build app config object from ini.
    config = Configurator(
        settings = settings,
    )

   def add_random_number(request):
       """return a random number."""
       return randint(1, 1000)

   # register request methods.
   # each request instance will run these functions.
   # the result attaches to the request as an attribute.
   # cache the result with `reify=True` to prevent multiple computations.
   config.add_request_method(add_random_number, 'random_number', reify=True)

Now we have access to this attribute in our views and templates:

request.random_number

This strategy has a number of uses. For example, I use it to:

attach configuration settings and secrets like API keys
attach a user object, by querying the database for the user_id sourced from the cookie session
analyze requests for misuse like spam or flooding

What super powers do you register to your request? You should let the world know in the comments.

Another Pyramid related post: Sharing a Pyramid cookie with Flask or Tornado

Sharing a Pyramid cookie with Flask or Tornado

2016-11-21T18:35:00-05:00

Do you have a Pyramid application which authenticates users and uses a signed cookie as a session? Do you want to build a microservice using another framework and allow it to use the same cookie and session? Me too!

First we will review a bit of Pyramid code which describes the cookie session.

Teach Tornado how to read Pyramid cookies

In this section I'll show you how to access and deserialize the Pyramid cookie from a Tornado application.

To do this, I'm going to extend the Tornado Hello, world application:

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

This is a simple application which listens to port 8888 and serves the text Hello, world when / is requested.

Add the following imports:

# accessing Pyramid cookies.
from webob.cookies import SignedSerializer
from pyramid.session import PickleSerializer
from pyramid.compat import bytes_

For testing purposes, create a global serializer object:

# https://docs.webob.org/en/stable/api/cookies.html#webob.cookies.SignedSerializer
serializer = SignedSerializer(secret='test-secret', salt='pyramid.session.', serializer=PickleSerializer())

Adjust the get method in the MainHandler to look like this:

def get(self):
    raw_cookie = self.get_cookie('session', None)
    if raw_cookie is not None:
        session_data = serializer.loads(bytes_(raw_cookie))
        self.write(str(session_data))
        self.write(str("<br/>"))
    self.write("Hello, world")

The complete program follows:

import tornado.ioloop
import tornado.web

# accessing Pyramid cookies.
from webob.cookies import SignedSerializer
from pyramid.session import PickleSerializer
from pyramid.compat import bytes_

# https://docs.webob.org/en/stable/api/cookies.html#webob.cookies.SignedSerializer
serializer = SignedSerializer(secret='test-secret', salt='pyramid.session.', serializer=PickleSerializer())

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        raw_cookie = self.get_cookie('session', None)
        if raw_cookie is not None:
            session_data = serializer.loads(bytes_(raw_cookie))
            self.write(str(session_data))
            self.write(str("<br/>"))
        self.write("Hello, world")


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

Again, we are hardcoding the same secret. If you set everything up properly, loading http://127.0.0.1:8888 in a web browser should print the cookie session_data in plain-text.

In my testing, I saw my cookie and it looked like this:

(1479520270, 1479516714.062414, {'authenticated_user_id': 5, 'nodes_pending_verify': []})
Hello, world

Thats all for now, let me know what you think in the comments!

My first Systemd Service Script and Override

2016-10-28T15:54:00-04:00

At work we mostly run Centos and I have some NodeJS services to deploy. I feel most familiar with Ubuntu / Upstart so this post serves as my notes on systemd.

In this contrived example, we define a service for our taco-api application. The taco-api source code lives in /opt/taco-api.

We manage a static .service file using a package or config management.

[/usr]/lib/systemd/system/taco-api.service:

[Unit]
Description=Node.js service for taco API

[Service]

Environment=NODE_ENV=development

ExecStart=/usr/bin/node /opt/taco-api/app.js

# Restart service after all crashes but wait 10 seconds between restarts.
Restart=always
RestartSec=10

# output stdout and stderr to syslog. (/var/log/[syslog|messages])
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%N

# define the user and group to own the process.
#User=node
#Group=node

# change directory before running ExecStart command.
WorkingDirectory=/opt/taco-api

[Install]
WantedBy=multi-user.target

We also manage a static override.conf file using config management. In this file, we customize the environment variables present.

/etc/systemd/system/taco-api.service.d/override.conf:

[Service]
Environment=NODE_ENV=production

This allows us to only override and keep track of the deltas.

You can test, like this:

# emit the status of the service.
service taco-api status

# start the service.
service taco-api start

# look at the process list, note that node is running.
ps aux | grep node

# emit the status of the service.
service taco-api status

# start the service.
service taco-api stop

# look at the process list, note that node is not running.
ps aux | grep node

Thank you!

Set static hostname for RHEL Centos 7 on AWS

2016-09-06T11:18:00-04:00

This took me about 2 hours to figure out, hopefully it saves you time.

/etc/sysconfig/network:

HOSTNAME=desired-hostname.example.com

/etc/hostname:

desired-hostname.example.com

/etc/cloud/cloud.cfg:

preserve_hostname: true

If I swallow lots of air will I be lighter?

2016-06-22T19:17:00-04:00

After dinner tonight, Carter, my four year old asked me, "If I swallow lots of air will I be lighter?". I thought about the question for a moment and then told him it depends on what surrounds you.

If you are surrounded by:

air and you swallow lots of air you will not be lighter.
water and and swallow lots of air you will be lighter, relative to the water, and you will float better.
helium and you swallow lots of air you will become heavier, relative to the surrounding helium.

Whaaaaaat???!!?!1!

: )

Output all instance identifiers of an AWS VPC to JSON

2016-06-21T10:25:00-04:00

At work today I needed an easy way to collect private IP addresses of every instance in one of our production VPCs.

I ended up adding a tool to https://botoform.com to perform this task.

bf --profile <aws_profile> dump <vpc_name_tag> instances --output-format json

For example:

bf --profile customer3 dump prd instances --output-format json > ~/customer3-prd-instances.json

Checkout the Botoform Quickstart for installation instructions.

Set DNS resolver options

2015-12-16T10:17:00-05:00

I desire the following options in /etc/resolv.conf:

rotate timeout:1 attempts:1

Ubuntu or Debian

For Ubuntu or Debian based systems, place the options in /etc/resolvconf/resolv.conf.d/base:

rotate timeout:1 attempts:1

These options merge with the options from DHCP and end up in /etc/resolv.conf.

Then restart the host, or at least networking.

Redhat or Centos or Fedora

For Redhat, Centos or Fedora, add the following to /etc/sysconfig/network:

RES_OPTIONS="rotate timeout:1 attempts:1"

Then restart the host, or at least networking.

Bind9 on Joyent Triton

2015-11-29T18:47:00-05:00

I have a single DNS server running on a KVM at DigitalOcean for $5/mo. I might move to Joyent and use 2 x 128M Ubuntu containers for $2.23/mo each.

In my first test Bind9 (named RNDC) ended up using 111M of memory on a 256M Ubuntu Triton container.

The large 111M memory footprint correlated with the amount of worker threads running. Bind9 determines the number of worker threads to manage by the number of CPUs detected by the OS.

In my case, Ubuntu detected 48 CPUs because Joyent containers run on bare-metal.

We can see this by running the following commands:

cat /proc/cpuinfo | grep processor | wc -l
48

sudo rndc status
version: 9.9.5-3ubuntu0.5-Ubuntu <id:f9b8a50e>
CPUs found: 48
worker threads: 48
UDP listeners per interface: 1
number of zones: 216
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running

To fix this, at least on Ubuntu, we need to pass -n 2 to limit the worker threads to 2.

For Ubuntu, edit /etc/default/bind9:

OPTIONS="-u bind -n 2"

Then restart the bind9 service and verify using sudo rndc status and free -m.

sudo service bind9 restart
sudo rndc status
free -m

Migrating from WordPress to Pelican

2015-11-15T15:39:00-05:00

Its finally happening. I'm moving this blog from WordPress to Pelican. This task has persisted on my TODO list for over two years.

During the process of the move, I'm going to use this post to dump hints:

Hints:

WordPress XML to JSON
Pelican-import
Add date to post filenames
Alter category to tags
Alter attachment and image paths

WordPress XML to JSON

I wrote this tool to convert Wordpress XML dumps to JSON. The tool is opinionated and removes lots of data.

Pelican-import

A tool to convert WordPress .xml into .rst or .md files (ReStructuredText or MarkDown) is pelican-import.

I suggest checking it out, even if you do not plan to use Pelican as your static site generator.

Add date to post filenames

After using pelican-import I had about 150 .rst files and I decided to put the date in the filename, so I wrote this short bash script tool to do the renames:

files=`ls *.rst`

for file in $files:
  do
    the_date=`grep ':date:' "$file" | awk '{ print $2; }'`
    mv "$file" "$the_date-$file"
  done

Alter category to tags

category and tags have different meanings and assumptions between wordpress and pelican. As a result I decided to change all my categories to tags using this command:

sed -i'' -e 's/:category:/:tags:/g' *.rst

Alter attachment and image paths

fix paths to images / uploads to remove wp-content:

sed -i'' -e 's/\/wp-content//g' *.rst

Boto3 get main route table

2015-10-16T12:15:00-04:00

While developing Botoform I ran into an issue with Boto3 where I couldn't easily get the "main" route table of a VPC. I ended up adding a get_main_route_table method to do the duty.

def get_main_route_table(self):
    """Return the main (default) route table for VPC."""
    main_route_table = []
    for route_table in list(self.route_tables.all()):
        for association in list(route_table.associations.all()):
            if association.main == True:
                main_route_table.append(route_table)
    if len(main_route_table) != 1:
        raise Exception('cannot get main route table! {}'.format(main_route_table))
    return main_route_table[0]

That all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

List all installed package names in Python

2015-07-04T22:15:00-04:00

Here we define a function called pkgs. This function returns a list of package resource objects.

pkgs = lambda : list(__import__('pkg_resources').working_set)

We then define two more functions: pkg_names & pkg_versions

pkg_names = lambda : [x.project_name for x in pkgs()]

pkg_versions = lambda : [x.project_name + '==' + x.version for x in pkgs()]

Last we show how to use invoke these functions:

>>> pkg_names()
['ansible', 'pycrypto', 'PyYAML', 'Jinja2', '...truncated...', 'virt-back', 'Werkzeug', 'xmltodict']

>>> pkg_versions()
['ansible==1.7', 'pycrypto==2.6.1', '...truncated...', 'virt-back==0.1.0', 'xmltodict==0.9.2']

Filtering AWS resources with Boto3

2015-07-02T17:03:00-04:00

This post will be updated frequently when as I learn more about how to filter AWS resources using Boto3 library.

Filtering VPCs by tags

In this example we want to filter a particular VPC by the "Name" tag with the value of 'webapp01'.

>>> import boto3
>>> boto3.setup_default_session(profile_name='project1')
>>> ec2 = boto3.resource('ec2', region_name='us-west-2')
>>> filters = [{'Name':'tag:Name', 'Values':['webapp01']}]
>>> webapp01 = list(ec2.vpcs.filter(Filters=filters))[0]
>>> webapp01.vpc_id
'vpc-11111111'

You can also filter on the value of the 'tag-key' or the 'tag-value' like so:

>>> taco_key_filter = [{'Name':'tag-key', 'Values':['taco']}]
>>> nacho_value_filter = [{'Name':'tag-value', 'Values':['nacho']}]

You can also filter on multiple 'Values'. In this example want 2 VPCs named 'webapp01' and 'webapp02':

>>> filters = [{'Name':'tag:Name', 'Values':['webapp01','webapp02']}]
>>> list(ec2.vpcs.filter(Filters=filters))
[ec2.Vpc(id='vpc-11111111'), ec2.Vpc(id='vpc-22222222')]

You can also use the '*' wildcard to glob up results in your filter. In this example we want all 3 VPCs named 'webapp01', 'webapp02' and 'webapp03':

>>> filters = [{'Name':'tag:Name', 'Values':['webapp*']}]
>>> list(ec2.vpcs.filter(Filters=filters))
[ec2.Vpc(id='vpc-11111111'), ec2.Vpc(id='vpc-22222222'), ec2.Vpc(id='vpc-33333333')]

Thats all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

Working with botocore's ~/.aws/config

2015-07-01T18:14:00-04:00

I ran into a bug in botocore and this post will serve to document a work around as well as show how to use botocore session object to work with the values stored in ~/.aws/config.

Pretend you have an aws config with two accounts for two separate projects, like so:

*~/.aws/config:*

[profile project1]
account_id = 111111111111
aws_access_key_id=THISISNOTMYACCESSKEY1
aws_secret_access_key=THISISNOTMYSECRETKEY1
# Optional, to define default region for this profile.
region=us-west-1

[profile project2]
account_id = 222222222222
aws_access_key_id=THISISNOTMYACCESSKEY2
aws_secret_access_key=THISISNOTMYSECRETKEY2
# Optional, to define default region for this profile.
region=us-west-2

Now instead of using a single object, we create multiple objects, one for each profile we intend to use.

>>> import botocore.session
>>> session1 = botocore.session.Session(profile='project1')
>>> session2 = botocore.session.Session(profile='project2')
>>> session1.get_credentials().access_key
'THISISNOTMYACCESSKEY1'
>>> session2.get_credentials().access_key
'THISISNOTMYACCESSKEY2'

Also figured out how to get at the `account_id` integer:

>>> session1.get_scoped_config()['account_id']
'111111111111'

Here is another algorithm that returns a list of sessions objects, one for each profile listed in the config.

>>> import botocore.session
>>> sessions = []
>>> aws_config = botocore.session.get_session().full_config
>>> for profile_name in aws_config['profiles']:
...     session = botocore.session.Session(profile=profile_name)
...     sessions.append(session)

Thats all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

My Mentor

2015-06-21T13:36:00-04:00

The original purpose of this post was to recognize the teachers and mentors which have helped shape me. I planned to write about how Mr. Cassidy, my high school drafting teacher, taught me the importance precision. I prepared to explain how Mr. Mercuri, my college computer science professor, ignited my desire to engineer software. But I could not. I knew deep down that one person in particular affected me more then the rest.

This person could fix anything and could even talk through a broken heart. He didn't know the internals of computers but the hacker spirit sweats from his veins. He taught me how to problem solve, many times in unconventional ways. I once watched him clear brush with an old snow plow and cut our work day in half. If he didn't have access to a tool, he would furnish one out of raw materials and spare parts, like a real life MacGyver. With a little bit of thought and ingenuity he built anything he imagined.

Sometimes he would use tough love to prove a point. For example, when I had a sun burn in middle school, he taught me to push through pain and not let my team down even though my jersey was scratching my raw skin. He showed me how to work hard and how to have great work ethic. He taught me the importance of "saving for a rainy day" but also knew how to have fun and loved parties. He taught me self control and how to make sacrifices for payouts in the future.

He taught me the fundamentals of kindness and guided me on the righteous path. He gave great advice, and even though I didn't always listen, he respected my thoughts and wishes. If he ever caused an accident he would always admit fault. He emits honor, honesty, and courage and never lies to or manipulates the people around him. Although we have had our differences, I think about you each day.

I love you. Thank you and happy father's day, dad.

Change default gateway on all SmartOS Zones and KVM guests

2015-06-01T20:34:00-04:00

Today I needed to change the default gateway on every Zone and KVM guest on my SmartOS hypervisor because I switched my ISP and as a result my gateway changed from 192.168.1.254 to 192.168.1.1. After changing one guest I got lazy and put together this script.

update-all-gateways.sh

for VM in `vmadm list -p -o alias,uuid`

  do
    # create an array called VM_PARTS splitting on ':'
    IFS=':' VM_PARTS=($VM)

    # create some helper varibles for alias and uuid
    alias=${VM_PARTS[0]}
    uuid=${VM_PARTS[1]}

    mac=`vmadm get $uuid | json nics | json -a mac`

    echo "
{
   \"update_nics\": [
      {
        \"mac\": \"$mac\",
        \"gateway\": \"192.168.1.1\"
      }
   ]
}
" | vmadm update $uuid
done

[root@hypervisor /opt/setup-jsons/updates]# bash update-all-gateways.sh
Successfully updated VM 211b992b-a448-40b4-94c9-xxxxxxxxxxxx
Successfully updated VM 31baa6a5-aa98-4750-80df-xxxxxxxxxxxx
Successfully updated VM 65d176b4-c36d-4cbf-b6ed-xxxxxxxxxxxx
Successfully updated VM aa0f603c-9572-4cb0-b96f-xxxxxxxxxxxx
Successfully updated VM ad928301-f3e1-4fe8-a1c1-xxxxxxxxxxxx
Successfully updated VM b82a257e-5628-46db-aee4-xxxxxxxxxxxx
Successfully updated VM da72b638-51de-4d7d-9853-xxxxxxxxxxxx
Successfully updated VM ee42bf30-51ce-4ae2-915b-xxxxxxxxxxxx

You are welcome!

Also to change the global zone (head node) default gateway, edit /usbkey/config and then run the following commands:

[root@hypervisor /]# route delete default 192.168.1.254
delete net default: gateway 192.168.1.254

[root@hypervisor /]# route add default 192.168.1.1
add net default: gateway 192.168.1.1

[root@hypervisor /]# netstat -r

SmartOS Ubuntu guest, apt-get not working because IPv6

2015-05-09T20:29:00-04:00

Turns out I don't have IPv6 setup properly in my network so when apt attempts to connect to the Internet it tries IPv6 and fails.

To disable IPv6 on the ubuntu guest, add this to end of /etc/sysctl.conf and restart the guest:

sudo vim /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

This was a hacky work around mostly because although I desire to have IPv6 working, I desire to get this VM running more...

Thanks for reading, leave comments please.

A Python script which searches for available interpreters

2015-05-08T10:34:00-04:00

This post describes how to write a polyglot -- in this case a script which runs as valid Bash or Python, to search for available Python interpreters.

The script initially runs as Bash but upon finding a first match, the script will call itself again this time using the expected Python interpreter in interactive mode!

And now, for the polyglot code:

#!/bin/sh

# reinvoke this script with bpython -i, ipython -i, or python -i
# reference: https://unix.stackexchange.com/a/66242
''':'
if type bpython >/dev/null 2>/dev/null; then
  exec bpython -i "$0" "$@"
elif type ipython >/dev/null 2>/dev/null; then
  exec ipython -i "$0" "$@"
else
  exec python -i "$0" "$@"
fi
'''
from helpers import (
  base_parser,
  get_hvpc_from_args,
)
parser = base_parser('Interactive Python interpreter & connection to hvpc')
args = parser.parse_args()
hvpc = get_hvpc_from_args(args)
print('\nYou now have access to the hvpc object, for example: hvpc.roles\n')

In this case you can see that we setup the interactive interpreter's environment to create an hvpc (Husky VPC) object for exploration.

If you want a pure python version that doesn't use a bash/python polygot, checkout this code I wrote: https://github.com/russellballestrini/botoform/blob/master/botoform/plugins/repl.py

Migrating libvirt KVM guest to SmartOS KVM guest

2015-04-28T22:17:00-04:00

The following tutorial documents how to migrate a libvirt/KVM guest from Ubuntu to SmartOS.

akuma:

Ubuntu Hypervisor

guy:

SmartOS Hypervisor

sagat:

guest to migrate

These commands were run on akuma:

WORKDIR=/KVMROOT/migrate
sudo mkdir $WORKDIR
cd $WORKDIR

# this tool simply stops and tarballs up the qcow and xml for libvirt KVM guest.
sudo /usr/local/bin/virt-back --path=$WORKDIR --backup sagat

sudo tar -xzvf sagat.tar.gz

sudo qemu-img convert -O raw sagat.qcow2 sagat.raw
sudo qemu-img convert -O raw sagat-var.qcow2 sagat-var.raw

scp sagat*.raw root@guy:/opt/tmp

These commands were run on guy:

Create the following file on guy - setup-ubuntu-sagat.json:

{
  "brand": "kvm",
  "resolvers": [
    "192.168.1.22",
    "8.8.4.4"
  ],
  "ram": "4096",
  "vcpus": "4",
  "alias": "sagat",
  "hostname": "sagat",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.50",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.254",
      "model": "virtio",
      "primary": true
    }
  ],
  "disks": [
    {
      "boot": true,
      "model": "virtio",
      "size": 10240
    },
    {
      "model": "virtio",
      "size": 20480
    }
  ]
}

We then create a new KVM guest on guy:

[root@guy /opt/setup-jsons]# vmadm create -f setup-ubuntu-sagat.json
Successfully created VM aa0f603c-9572-4cb0-b96f-4c79eb431223

List out the virtual block devices on the new KVM guest:

[root@guy /opt/setup-jsons]# vmadm info aa0f603c-9572-4cb0-b96f-4c79eb431223 block
{
  "block": [
    {
      "device": "virtio0",
      "locked": false,
      "removable": false,
      "inserted": {
        "ro": false,
        "drv": "raw",
        "encrypted": false,
        "file": "/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk0"
      },
      "type": "hd"
    },
    {
      "device": "virtio1",
      "locked": false,
      "removable": false,
      "inserted": {
        "ro": false,
        "drv": "raw",
        "encrypted": false,
        "file": "/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk1"
      },
      "type": "hd"
    }
  ]
}

Stop the new guest, it needs to be off for the restore.

[root@guy /opt/setup-jsons]# vmadm stop aa0f603c-9572-4cb0-b96f-4c79eb431223
Successfully completed stop for VM aa0f603c-9572-4cb0-b96f-4c79eb431223

Use dd to:

write sagat.raw to disk0
write sagat-var.raw to disk1

dd if=sagat.raw of=/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk0 bs=1M
dd if=sagat-var.raw of=/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk1 bs=1M

Lastly start the new guest up, and check it out:

vmadm start aa0f603c-9572-4cb0-b96f-4c79eb431223

Setting region programmatically in Boto3

2015-04-24T14:07:00-04:00

At work I'm looking into the possibility of porting parts of our AWS automation codebase from Boto2 to Boto3. We desire to perform this port because Boto2's record and result pagination appears defective.

I started to familiarize myself with Boto3 by using the Interactive Python interpreter.

Here I show myself trying to connect to the RDS AWS endpoint following the docs:

>>> import boto3
>>> rds = boto3.client('rds')
Traceback (most recent call last):
...
NoRegionError: You must specify a region.

What, No region? OK - how do I set a region?

Well it turns out the docs want you to configure a region in a config file. This will not work for me, I need to set the region programatically...

So after stumbling around in the botocore source code I found the following solutions.

Solution 1 - Set region_name when creating client:

>>> import boto3
>>> rds = boto3.client('rds', region_name='us-west-2')

Solution 2 - Set default region_name on the session:

>>> import boto3
>>> rds = boto3.setup_default_session(region_name='us-west-2')
>>> rds = boto3.client('rds')

It seems Boto3 has two types of interfaces, clients and resources.

Clients:: return description objects and appear lower level. Description objects seem like AWS XML responses transformed into Python Dicts/Lists.
Resources:: return higher level Python objects and like Instances with stop/start methods.

At a quick glance, both clients and resources seem to properly implement pagination automatically!

>>> # client interface.
>>> ec2 = boto3.client('ec2', region_name='us-west-2')
>>> idesc = ec2.describe_instances()
>>> len(idesc['Reservations'])
273
>>> idesc['ResponseMetadata']
{'HTTPStatusCode': 200, 'RequestId': '7e431ff7-xxxx-xxxx-xxxx-xxxxxxxxxxxxx'}

>>> # resource interface.
>>> ec2 = boto3.resource('ec2', region_name='us-west-2')
>>> for instance in ec2.instances.all():
>>>     print instance, instance.tags

That all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

Securely publish Jenkins build artifacts on Salt Master

2015-02-22T12:39:00-05:00

Do you want a secure setup for publishing and staging build artifacts from a Jenkins build server to a Salt Master? This guide describes my fully automated pipeline to transport binaries using Salt's encrypted "bus".

We start off with some Salt States to stand up a Jenkins build server "client":

jenkins/client.sls:

# https://russell.ballestrini.net/securely-publish-jenkins-build-artifacts-on-salt-master/
# manage jenkins user, home dir, and Jenkins "master" public SSH key.
jenkins:
  user.present:
    - fullname: jenkins butler
    - shell: /bin/bash
    - home: /home/jenkins

  file.directory:
    - name: /home/jenkins
    - user: jenkins
    - group: jenkins
    - require:
      - user: jenkins

  ssh_auth.present:
    - user: jenkins
    - name: {{ pillar.get('jenkins-public-key') }}
    - require:
      - user: jenkins

# Manage a script to push artifacts to Salt Master.
# Note: jenkins user should _not_ have ability to change this file.
/opt/salt-call-put-artifacts-onto-salt-master.sh:
  file.managed:
    - user: root
    - group: jenkins
    - mode: 755
    - contents: |
        #!/bin/bash
        set -x
        salt-call cp.push_dir "$PWD" glob='*.tar.gz'
        salt-call cp.push "$PWD/commit-hash.txt"
    - require:
      - file: jenkins

# Allow jenkins to run push script as root via sudo.
jenkins-sudoers:
  file.append:
    - name: /etc/sudoers
    - text:
      - "jenkins    ALL = NOPASSWD: /opt/salt-call-put-artifacts-onto-salt-master.sh"

On the Salt Master we must enable MinionFS and restart Salt Master process:

fileserver_backend:
  - roots
  - minion

file_recv: True

We then use this command as the last build task in every Jenkins build job:

sudo /home/jenkins/salt-call-put-artifacts-onto-salt-master.sh

This causes the file to be staged on the Salt Master on a successful build.

The Salt States to deploy the software to production, look something like this:

# extract tarball from Salt Master using MinionFS.
extract-tarball-for-mysite:
  archive.extracted:
    - name: /www/mysite
    - archive_format: tar
    - source: salt://ubuntu-jenkins.foxhop.net/home/jenkins/workspace/job-name/env.tar.gz
    - user: uwsgi
    - group: uwsgi
    # I want to always extract, not sure a better way.
    - if_missing: /dev/taco
    - require:
      - service: make-mysite-dead-for-release

I also have build triggers which monitor remote git/hg repos for changes. Pushing code triggers a build which tests my code base and securely publishes to my Salt Master. When the time comes to perform a release, all I have to do is run highstate, because the pipeline did all the other work for me!

Update

Special thanks to an anonymous commentor regarding how to increase security. I have updated the scripts accordingly.

Set postgres user password on PostgreSQL SmartOS Zone

2015-01-21T22:28:00-05:00

Connect to zone and determine the auto generated password for postgres user:

cat /var/svc/log/system-zoneinit\:default.log | grep PGSQL_PW

document the result and log into postgres with the following command, entering the password when prompted:

[root@psql ~]# psql --user postgres

Alter the postgres role's password:

postgres=# ALTER ROLE postgres UNENCRYPTED PASSWORD 'new-password';

Now exit (\q) then try to log in with the new password.

In my case I was setting up PostgreSQL for testing out Zabbix monitoring system. So I did the following as the postgres user:

postgres=# CREATE USER zab UNENCRYPTED PASSWORD 'zabby'
postgres=# CREATE DATABASE zab OWNER zab;

Risk, Process, and Balance

2015-01-10T22:35:00-05:00

The operations of a company will have intrinsic risk. Risk occurs each time we decide to take an action or an inaction. This means that anything we choose to do, or not do, has associated risk.

An organization which has an unhealthy aversion to risk has a much higher chance of failure. As time goes on the intolerance of taking on additional risk to accomplish goals, will render a company petrified. Stuck in the fear of change.

In order to become unstuck, a company and it's employees must become great at calculating risk and assessment. In order to make a choice on what to work on, and what to not work on, they will need to research and collect data and continuously track progress and feedback. The company must use this data and feedback to move uncalculated bad risk into calculated good risk. Simply thinking through the problem and change before hand can uncover ways to accomplish the end result in the least risky way.

The biggest objection to the risk assessment phase is typically time constraints. A great company will learn how to rapidly assess and reduce the risk of action. They reduce risk by: gathering requirements, planning, building process, using process to drive automation, and iterating on automation to speed up feedback loops. These fast feedback loops will allow the company and it's employees to fail smaller and more frequently and facilitate reflection to optimize the pipeline. This means even more calculated and low risk actions!

So, a company should add more process to reduce risk? Not quite.

A process, when left unchecked or added for the wrong reasons, will become more detrimental then blindly performing an action without calculating the risk. Below I outline the key differences between a good process and a bad process.

A good process will:

reduce risk but not at the expense of blocking or preventing progress
be fast to facilitate quick feedback loops
have potential for automation and must not be tedious or manual (think approvals)
promote small and frequent changes
support both fail forward and fail backward
only fail backward to reduce time-to-recover
not be pushed down from upper management, rather upper management should help set goals and requirements, while the team builds and decides on process
be questioned and reviewed often to promote iteration to find and fix inefficiencies

A bad process will:

block real work from getting done
prevent creative and innovative solutions
have a higher chance of being ignored, skipped, or held in contempt
punish people who take risks (make change) and promote people who do nothing
cause more issues then it solves
become a crutch or a security blanket (a false sense of security)

I feel strongly that the essence behind the Agile and DevOps movements comes from the constant battle of balancing risk with process. This constant desire for balance in the organization's environment will lead to innovative ideas, tools, and workflows. These ideas, tools and workflows can not function properly if transplanted into a company who does not have the desire to find the optimal balance. Put simply, a company cannot buy Agile or DevOps, that culture needs to grow from with in.

autofs /net automount stopped working

2014-12-21T22:38:00-05:00

So autofs randomly stopped working on one of my Ubuntu hosts (this issue has been found on Arch as well so its most likely a change upstream). I found this error in the logs:

attempting to mount entry /net/freenas.example.net
get_exports: lookup(hosts): exports lookup failed for freenas.example.net
key "freenas.example.net" not found in map source(s).
failed to mount /net/freenas.example.net

The fix was to replace the following line in /etc/auto.master from

/net   -hosts

to this

/net /etc/auto.net --timeout=60

and then restart autofs (sudo service autofs restart).

Custom Rundeck HipChat notification templates

2014-12-03T01:37:00-05:00

Today I built a GUI and workflow around Ansible using Rundeck. Tonight I started diving into sending HipChat notifications and after a bit of research, I managed to create a custom notification template for each Rundeck project.

Modify your project's configuration file, on Ubuntu it was in /var/rundeck/projects/pname/etc/project.properties, and add the following line to the bottom:

framework.plugin.Notification.HipChatNotification.messageTemplateLocation=/var/rundeck/projects/pname/etc/custom-hipchat-template.ftl

Note: replace pname with your project name.

I ended up producing a single line chat notification template. I uploaded it here to act as an example:

Custom Rundeck HipChat Notification Template

The template syntax and renderer is called FreeMarker. Rundeck and the HipChat plugin pass many different context Map hash objects to FreeMarker for use in the templates. In this case I display "VPC name" selected by the user when starting the job.

References:

Rundeck HipChat Notification Plugin
Rundeck HipChat Notification Plugin (User Guide and Default Template)

Build release pipelines on S3 with s3p

2014-11-24T14:39:00-05:00

This weekend I finished my first sprint on s3p which is a Python library and CLI application that manages release pipelines on AWS S3. I put a lot of effort into the readme.rst file, so look there for usage and examples.

The main purpose of s3p is to use code to enforce process when promoting releases in the pipeline. Another goal was to make the CLI tool dead simple to use. As a side effect, I ended up using composition to extend boto.s3's Key and Bucket classes to produce S3Release and S3Pipeline.

I plan to incorporate s3p into Teamcity build jobs. At work I would like to use s3p to replace bash+s3cmd in pipeline management. Once s3p has a bit more production use, I will likely sprint on it again.

Dealing with pagination in Python

2014-11-13T21:06:00-05:00

So I'm working with an API (AWS ElastiCache) that offers mandatory pagination of results. I need to get all results, so I took some time to work out this logic.

def combine_results(function, key, marker=0, **kwargs):
    """deal with manditory pagination of AWS result descriptions"""
    results = []
    while marker != None:
        result = function(marker = marker, **kwargs)
        marker = nested_lookup('Marker', result)[0]
        results += nested_lookup(key, result)
    return results

Not only is the AWS ElastiCache API paginated but it also appears deeply nested in lists and dicts.

I use this to burn it with fire:

def nested_lookup(key, dictionary):
    """Lookup a key in a nested dictionary, return a list of values"""
    return list(_nested_lookup(key, dictionary))

def _nested_lookup(key, dictionary):
    """
    Lookup a key in a nested dictionary, return value

    Authors: Dougles Miranda and Russell Ballestrini
    """
    if isinstance(dictionary, list):
        for d in dictionary:
            for result in _nested_lookup(key, d):
                yield result

    if isinstance(dictionary, dict):
        for k, v in dictionary.iteritems():
            if k == key:
                yield v
            elif isinstance(v, dict):
                for result in _nested_lookup(key, v):
                    yield result
            elif isinstance(v, list):
                for d in v:
                    for result in _nested_lookup(key, d):
                        yield result

The end result is we have access to paginated and deeply nested data with a simple to use function:

>>> from lib import combine_results, nested_lookup
>>> d = elasticache_connection.describe_cache_clusters()
>>> nested_lookup('CacheClusterId', d)
[u'demo04-a-redis', u'demo04-b-redis', u'demo06-a-redis', u'demo06-b-redis', u'test-a-memcached', u'test-b-redis', u'ops01-redis', u'qa01-redis', u'ops02-redis', u'qa02-redis', u'int01-a-redis', u'int01-b-redis', u'ops03-redis', u'ops04-redis']

Here are some unit tests to prove these functions work like expected:

from unittest import TestCase

from lib.util import (
  combine_results,
  nested_lookup,
  _nested_lookup,
)

def my_func_that_paginates(max_results=3, marker=0):
    """this function sort of mocks the paginated AWS description results"""
    data = [
      {'desired_key' : 0},
      {'desired_key' : 1},
      {'desired_key' : 2},
      {'desired_key' : 3},
      {'desired_key' : 4},
      {'desired_key' : 5},
      {'desired_key' : 6},
      {'desired_key' : 7},
      {'desired_key' : 8},
      {'desired_key' : 9},
    ]
    new_marker = marker + max_results
    if new_marker > len(data):
        # last page!
        page = data[marker:]
        return {'results' : page, 'Marker' : None}
    page = data[marker:new_marker]
    return {'results' : page, 'Marker' : new_marker}

class TestCombineResults(TestCase):

    def test_combine_results_returns_all_results(self):
        expected_set = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
        f = my_func_that_paginates
        result_set = set(combine_results(f, 'desired_key'))
        self.assertSetEqual(expected_set, result_set)

class TestNestedLookup(TestCase):

    def setUp(self):
        self.subject_dict = {'a':1,'b':{'d':100},'c':{'d':200}}

    def test_nested_lookup(self):
        results = nested_lookup('d', self.subject_dict)
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list(self):
        results = nested_lookup('d', [{}, self.subject_dict, {}])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list_in_dict_in_list(self):
        results = nested_lookup('d', [{}, {'H' : [self.subject_dict]} ])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list_in_list(self):
        results = nested_lookup('d', [ {}, [self.subject_dict, {}] ])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

With this test, the steps of the algorithm looks like this:

{'Marker': 3, 'results': [{'desired_key': 0}, {'desired_key': 1}, {'desired_key': 2}]}
3
[0, 1, 2]
[0, 1, 2]
{'Marker': 6, 'results': [{'desired_key': 3}, {'desired_key': 4}, {'desired_key': 5}]}
6
[3, 4, 5]
[0, 1, 2, 3, 4, 5]
{'Marker': 9, 'results': [{'desired_key': 6}, {'desired_key': 7}, {'desired_key': 8}]}
9
[6, 7, 8]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
{'Marker': None, 'results': [{'desired_key': 9}]}
None
[9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
ok

Turn python dict into a key=value string and back again

2014-11-09T22:02:00-05:00

I'm currently refactoring a script that tags AWS resources and I came up with this one liner to generate pretty output. It basically turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2. Here is the code:

', '.join(['='.join(key_value) for key_value in {'a':'1','b':'2'}.items() ])

Oh, and if you like this, here is a function with additional functionality and protection:

def dict_to_key_value(data, sep='=', pair_sep=', '):
    """turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2"""
    return pair_sep.join([sep.join((unicode(key), unicode(value))) for key, value in data.items()])

Careful, this might blow up on dictionaries that nest other objects. Also here is a test:

def test_dict_to_key_value():
    data = {'tag1':'value1','tag2':'value2'}
    pretty_str = dict_to_key_value(data)
    assert('tag1=value1' in pretty_str)
    assert('tag2=value2' in pretty_str)
    assert('tag1=value1, tag2=value2' in pretty_str)
    not_as_pretty = dict_to_key_value(data,'x','x')
    assert('tag1xvalue1xtag2xvalue2' in not_as_pretty)

Here is the inverse, taking a list of key_value strings and returning a dictionary of the data:

def key_value_to_dict(key_value_list, sep='=', pair_sep=',' ):
    """
    Accept a key_value_list, like::

      key_value_list = ['a=1,b=2', 'c=3, d=4', 'e=5']

    Return a dict, like::

      {'a':'1', 'b':'2', 'c':'3', 'd':'4', 'e':'5'}
    """
    d = {}
    for speclist in key_value_list:
        for spec in speclist.strip().split(','):
            key, value = spec.strip().split('=')
            d[key] = value
    return d

And of course a test to prove it works how we expect:

def test_key_value_to_dict():
    key_value_list = ['a=1,b=2', 'c=3, d=4', 'e=5']
    desired_result = {'a':'1', 'b':'2', 'c':'3', 'd':'4', 'e':'5'}
    assert(key_value_to_dict(key_value_list) == desired_result)

Migrating MongoDB from Ubuntu to SmartOS

2014-10-11T20:52:00-04:00

First, I installed the mongodb 14.2.0 (uuid a5775e36-2a02-11e4-942a-67ae7a242985) dataset:

imgadm avail | grep mongo
imgadm import a5775e36-2a02-11e4-942a-67ae7a242985

Next, I launched a new zone with this image.

Then I grabbed the uuid of the zone (211b992b-a448-40b4-94c9-00fa82615cec) and I connected into the zone

zlogin 211b992b-a448-40b4-94c9-00fa82615cec

The zone automatically creates a username and password for admin and "quickbackup". You can find these passwords by running the following command inside the zone:

cat /var/svc/log/system-zoneinit\:default.log | grep -i mon

First thing I did was disable authentication by modifying /opt/local/etc/mongod.conf:

#auth = true
noauth = true

Then I restarted MongoDB to re-read its configuration:

svcadm restart mongodb

Next I attempted to restore the database BSON files, with the following command:

mongorestore --db=taco taco/taco.bson

But I got the following error:

connected to: 127.0.0.1
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Abort (core dumped)

After some research I learned that I needed to export the following variable before running the restore:

export LC_ALL=C

Set Root Password SmartOS Percona MySQL Zone

2014-10-11T00:12:00-04:00

I used project-fifo to launch the percona (14.2.0) MySQL dataset. I couldn't get into the MySQL instance so I reached out on IRC. Johngrasty, a friendly guy in the #smartos IRC channel, provided a command to display the randomly generated MySQL password emitted to the zone-init log:

cat /var/svc/log/system-zoneinit\:default.log | grep MYSQL_PW

I used this initial password to get into the mysql> shell and changed it with this SQL statement:

mysql> SET PASSWORD = PASSWORD('clear-text-password');

Johngrasty also supplied a snippet of JSON which shows how to declare root MySQL password:

{
  ... truncated ...
  "customer_metadata": {
    "salt-master": "10.0.0.101",
    "salt-id": "mysql1"
  },
  "internal_metadata": {
    "mysql_pw": "mypassword"
  }
}

I looked for help in the #project-fifo channel as to why the GUI does not work for assigning initial MySQL password. MerlinDMC, the author and operator of datasets.at, gave me the following command to run in the Global Zone to look at a particular zone's metadata:

cat /zones/uuid-of-zone-goes-here/config/metadata.json

Turns out the GUI is placing the "mysql_pw" parameter into "customer_metadata" instead of "internal_metadata" which is invalid.

rabbits

2014-09-18T21:11:00-04:00

reality shattered

2014-09-18T21:11:00-04:00

Heka, World2!

2014-08-23T22:54:00-04:00

This article expands on my “Hello World” for Heka blog post. Check that one out first if you are new to Heka.

In this guide we introduce using Heka over the network by utilizing two Hekad processes on localhost. For discussion purposes we name one of the Hekad processes "sender" and the other "receiver".

The "sender" will watch a log file and emit messages to localhost TCP port 9612.
The "receiver" will listen on localhost TCP port 9612 and emit message payloads to file.

hello-heka-file-in-tcp-out.toml (sender):

# watch /tmp/input.log and output to TCP port 9612 on localhost
[hello_heka_input_log]
type = "LogstreamerInput"
log_directory = "/tmp"
file_match = 'input\.log'

[tcp_out:9612]
type = "TcpOutput"
message_matcher = "TRUE"
address = "127.0.0.1:9612"
#encoder = "hello_heka_output_encoder"
#
#[hello_heka_output_encoder]
#type = "PayloadEncoder"
#append_newlines = false

hello-heka-tcp-in-file-out.toml (receiver):

# listen to TCP port 9612 and emit to /tmp/output.log
[tcp_in:9612]
type = "TcpInput"
parser_type = "message.proto"
decoder = "ProtobufDecoder"
address = ":9612"

[hello_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "hello_heka_output_encoder"

[hello_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false

Now that we have config files, let us start our hekad processes, Open three terminals.

in terminal 1:

sudo hekad -config=hello-heka-file-in-tcp-out.toml

in terminal 2:

sudo hekad -config=/tmp/hello-heka-tcp-in-file-out.toml

in terminal 3:
```
echo 'Heka, World2!' >> /tmp/input.log
```
in terminal 3:
```
cat /tmp/output.log
```

Again, like magic, the data echoed into input.log shows up in output.log. This time the data traveled over TCP between to separate Hekad processes. I leave changing the configuration to support separate hosts to the reader.

By default TCP sender encodes the message with Protobuf (ProtobufEncoder) and the TCP reciever decodes the message with Protobuf (ProtobufDecoder).

In my testing I decided to make the TCP sender use the PayloadEncoder and then instead of using a second hekad process, I used nc -l 9612 to listen on the port. When data was added to /tmp/input.log it showed up in the netcat terminal because hekad was watching the file and emiting just payload portion of the message to TCP 9612 which netcat was listening on. I left this configuration in the examples above, simply uncomment to reproduce.

Read this for more fun with netcat.

Mailpile Salt States for Ubuntu or Debian

2014-08-15T20:32:00-04:00

I wrote these Salt States to install Mailpile on an Ubuntu host. Fun fact, it took me 20 minutes to write these states and they worked the first time I ran them. Disclaimer - I used a throw away server and wasn't concerned that buckets of packages were installed to the system instead of using a virtualenv.

cd /opt/Mailpile
./mp --set sys.http_host=0.0.0.0
./mp

Then open a web browser the IP address of the host running the mp command and follow the prompts to setup the server/client/app.

mailpile/init.sls:

# Clone the source repository
mailpile-git-latest:
  git.latest:
    - name: https://github.com/pagekite/Mailpile.git
    - target: /opt/Mailpile

# install the system requirements
mailpile-system-packages:
  pkg.installed:
    - names:
      - make
      - python-imaging
      - python-lxml
      - python-jinja2
      - pep8
      - ruby-dev
      - yui-compressor
      - python-nose
      - spambayes
      - phantomjs
      - python-pip
      - python-mock
      - python-pexpect
      {% if grains['lsb_distrib_release']|float >= 14.04 %}
      - rubygems-integration
      {% else %}
      - rubygems
      {% endif %}

# install some python requirements with pip
mailpile-pip-packages:
  pip.installed:
    - names:
      - pgpdump
      - selenium >= 2.40.0
    - require:
      - pkg: mailpile-system-packages

# install some ruby requirements with gem
mailpile-gem-packages:
  gem.installed:
    - names:
      - therubyracer
      - less
    - require:
      - pkg: mailpile-system-packages

You can hack on FreeNAS 9

2014-05-15T01:06:00-04:00

This post analyses the FreeNAS 9 code base and discusses the various places users may feel confident to hack on.

FreeNAS uses the following software stack:

Django

A Python Web Application Framework which complies with WSGI

Nginx

A very fast web server which may act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and HTTP cache.

Dojo Toolkit

The Javascript toolkit used to create widgets and handle client side processing.

FreeBSD

FreeBSD is an advanced computer operating system used to power modern servers.

Want to hack on the frontend web application?

Start here if you enjoy Python, or if you really enjoy coding on Django applications:

https://github.com/freenas/freenas/tree/master/gui

Want to hack on the GUI?

Start here if you are a front end developer and enjoy writing HTML, CSS, and working with Javascript:

https://github.com/freenas/freenas/tree/master/gui/templates

Want to change Nginx?

Take a look here if you would like to review, change, or tune Nginx on FreeNAS:

https://github.com/freenas/freenas/tree/master/nanobsd/Files/usr/local/etc/nginx This directory holds the nginx "vhost" config files and CGI parameters.

Want to hack on the OS?

Start here, if you know about FreeBSD or operating systems in general:

https://github.com/freenas/freenas/tree/master/nanobsd This directory seems like a customized and completely version controlled nanoBSD install!

Nginx with SSL and mixed content errors with upstream WSGI servers

2014-05-08T16:28:00-04:00

Mixed content errors occur because Nginx (the front-end server) communicates to the upstream WSGI server using http. WSGI does not know (or care) about the SSL session between Nginx and the user. The WSGI server will naively generate URIs and serve assets as http.

To fix mixed content errors, we need to communicate the inbound request scheme or configure the WSGI server to always use https.

waitress

Waitress is meant to be a production-quality pure-Python WSGI server with very acceptable performance.

It commonly powers Pyramid and Substance D deployments.

To configure waitress to always use https in code:

from waitress import serve
serve(wsgiapp, host='0.0.0.0', port=8080, url_scheme='https')

configure waitress to always use https in paste deploy compatible configuration file:

[server:main]
host = 127.0.0.1
port = 6543
url_scheme = https

How to patch Heartbleed OpenSSL defect (libssl) on Ubuntu

2014-04-08T22:42:00-04:00

Lots of people claim that you need to upgrade openssl package, but this will not fix the issue.

The issue is not the openssl package, it is one of the libraries that the package relies on (libssl).

https://www.ubuntu.com/usn/usn-2165-1/

The output of openssl version -a command should have a built on date older then Mon Apr 7 20:33:29 UTC 2014. After patching openssl we still see the vulnerable date:

openssl version -a | egrep "OpenSSL|built"
OpenSSL 1.0.1 14 Mar 2012
built on: Tue Aug 21 05:18:48 UTC 2012

Now we patch libssl1.0.0:

sudo apt-get update
sudo apt-get install libssl1.0.0

Notice the patched built on date:

openssl version -a | egrep "OpenSSL|built"
OpenSSL 1.0.1 14 Mar 2012
built on: Mon Apr  7 20:33:29 UTC 2014

In my case I used a Salt remote execution to patch, verify, and restart nginx on all of my 14 hosts:

sudo salt '*' cmd.run 'apt-get update'

sudo salt '*' cmd.run 'apt-get -y install libssl1.0.0'

sudo salt '*' cmd.run 'openssl version -a | egrep "OpenSSL|built"'

sudo salt '*' service.restart nginx

xkcd 1353

IRC Bot (Foxbot) runs canned remote executions using Salt Stack

2014-03-15T16:33:00-04:00

I extended my IRC Bot Foxbot today to allow it to run canned remote executions on behalf of users in an IRC channel. This is only a prototype or proof-of-concept. Be very careful not to allow users to inject their own commands. Foxbot must be running on the Salt Master and must be running as the same user that runs the salt-master daemon.

The code lives here: foxbot/plugins/checks.py

Example usage:

16:18:25            * | russell checks uptime minion2.foxhop.net
16:18:25       foxbot | minion2.foxhop.net:  16:18:25 up 496 days, 22:36, 17 users,  load average: 0.33, 0.70, 0.87

16:29:10            * | russell checks procs *
16:29:17       foxbot | minion2.foxhop.net: PROCS WARNING: 165 processes
16:29:17       foxbot | minion5.foxhop.net: PROCS CRITICAL: 309 processes

Filter Salt Stack Return Data Output

2014-03-13T18:21:00-04:00

Sometimes you only want to see what has changed, and that is OK.

Create a file like this:

filter.py

#!/usr/bin/python

from json import loads
from json import dumps

import fileinput

stdin_lines = [line for line in fileinput.input()]

ret = loads(''.join(stdin_lines))

for minion_id, data in ret.items():
    print(minion_id)
    print('='*len(minion_id))
    for key, value in ret[minion_id].items():
        if value['changes'] or value['result'] == False:
            print('')
            print(dumps(value, indent=4))
            print('')

Make the file executable:

chmod 755 filter.py

Execute your remote execution like this:

sudo salt-call --out=json state.highstate | ./filter.py
sudo salt '*' --out=json  --timeout=60 --static state.highstate | ./filter.py
The flags --timeout=60 and --static will cause the Salt command to block until the specified seconds for each minion to return results. We then pipe the returned JSON into our filter.py script to filter out only the changes and failures!

Profit!

Change the conditional depending on what you want. For example, for just failures do this:

if value['result'] == False:

Example output:

# sudo salt 'graphite.foxhop.net' --out=json --static --timeout=60 state.highstate | ./filter.py

graphite.foxhop.net
===================

{
    "comment": "File /tmp/taco updated",
    "__run_num__": 15,
    "changes": {
        "diff": "New file",
        "mode": "0640"
    },
    "name": "/tmp/taco",
    "result": true
}

Replace the Nagios Scheduler and NRPE with Salt Stack

2014-03-08T15:53:00-05:00

Note: I will update this post as I progress.

So the idea is to use Salt Stack's remote execution to communicate with all nodes and run the Nagios checks and collect the return output instead of using the NRPE client/service protocol. This reduces the number of agents running on each host and appears significantly more secure. Salt Stack uses public/private crypto on top of the ZMQ publisher/subscriber model. This means the communication transport is very fast, very secure, and all nodes will run checks in parallel!

First, we need to install the Nagios checks and plugins on each host or minion.

I used the following State Formula on my Ubuntu and Debian hosts:

salt://nagios/plugins.sls

nagios-plugins:
  pkg:
    - installed

nagios-plugins-extra:
  pkg:
    - installed

salt://top.sls

base:
  '*'
    - nagios.plugins

I kicked off a highstate (salt '*' state.highstate) on all minions and eventually they all returned in the affirmative. We now have all the Nagios plugins and checks installed on each host.

This next part is FUN, We run a check on every host concurrently.

This particular check counts the number of processes running on each host and warns at 150 and criticals at 200.
salt '*' --out=json --static cmd.run_all '/usr/lib/nagios/plugins/check_procs -w 150 -c 200'
Output static JSON - Wait for all minions to return and then generate a single JSON object.
{
    "graphite.foxhop.net": {
        "pid": 24356,
        "retcode": 0,
        "stderr": "",
        "stdout": "PROCS OK: 78 processes"
    },
    "akuma.foxhop.net": {
        "pid": 4610,
        "retcode": 1,
        "stderr": "",
        "stdout": "PROCS WARNING: 158 processes"
    },
    "ken.foxhop.net": {
        "pid": 31254,
        "retcode": 2,
        "stderr": "",
        "stdout": "PROCS CRITICAL: 392 processes"
    }
}
Review the Nagios Plugin API for more information about return codes and STDOUT formats.

Last, we take this JSON data perform further processing or display it in a meaningful way.

For example, we could generate an HTML/JavaScript dashboard from the raw JSON objects. We could cut metrics out of the STDOUT and persist historic values in a data store. We could push the data into another system like Graphite or even send an email alerts.

This solution has all the perks of Nagios without ANY of the cons!

Returning the check output data to the CLI isn't super useful for further processing, so I hacked in a way to return data back to the master using the encrypted zeromq bus.

_returners/zeromq_return.py

# -*- coding: utf-8 -*-
'''
The zeromq returner will send return data back to the Salt Master over the
Encrypted 0MQ event bus with a custom tag for filtering on the other end.

Basically after the remote execution finishes, the ret data is "packaged" into
a special "envelope" which triggers the local Salt Minion Daemon to
forward the ret data to the Salt Master's event bus.

The "package" basically wraps the ret data and uses the tag 'fire_master'.

For example, a ret data object from the execution of test.ping
would be "packaged" like this::

  ret = {
    'graphite.foxhop.net': true
  }

  ret['tag'] = 'third-party'

  package = {
    'events': [ ret ],
    'tag': None,
    'pretag': None,
    'data': None
  }

The Salt Minion Daemon will forward this package to the Salt Master
where a 3rd party script may be filtering on the specified internal event tag.

To use the zeromq returner, append '--return zeromq' to the salt command. ex::

  salt --return zeromq '*' test.ping

TODO:

 figure out a way for user to define custom tag for filtering ...
 Most returners use the Salt Minion config file to supply returner
 details... that is not optimal, it would be ideal if the custom tag
 could be supplied on the CLI when the remote execution is run, like::

   --return=zeromq --tag=mytag

'''

# needed to log to log file
import logging

# needed for config to opts processing
import os
import salt.syspaths as syspaths
from salt.config import minion_config

# needed to send events over ZMQ
import salt.utils.event

log = logging.getLogger(__name__)

# needed to define the module's virtual name
__virtualname__ = 'zeromq'

def __virtual__():
    return __virtualname__


def returner(ret):
    '''
    Send the return data to the Salt Master over the encrypted
    0MQ bus with custom tag for 3rd party script filtering.
    '''

    # get opts from minion config file, supports minion.d drop dir!
    opts = minion_config(os.path.join(syspaths.CONFIG_DIR, 'minion'))

    # TODO: this needs to be customizable!
    tag = 'third-party'

    # add custom tag to return data for filtering
    ret['tag'] = tag

    # multi event example, supports a list of event ret objects.
    # single event does not currently expand/filter properly on Master side.
    package = {
      #'id': opts['id'],
      'events': [ ret ],
      'tag': None,
      'pretag': None,
      'data': None
    }

    # opts must contain valid minion ID else it binds to invalid 0MQ socket.
    event = salt.utils.event.SaltEvent('minion', **opts)

    # Fire event payload with 'fire_master' tag which triggers the
    # salt-minion daemon to forward payload to the master event bus!
    event.fire_event(package, 'fire_master')

Next we run a third party application on the Salt Master which subscribes to our events by filtering on the special tag ('third-party').

listen_to_master_bus.py

# event libary for events over ZMQ
import salt.utils.event

# create event object, attach to master socket ...
event = salt.utils.event.MasterEvent('/var/run/salt/master')

tag = 'third-party'

print('Listening for events tagged \'{}\' on Salt Master bus.'.format(tag))

# generator iterator yields events forever, we filter on tag
for data in event.iter_events(tag=tag):
    print(data)

This small application just prints the incoming return data, but it could easily be expanded to process the incoming return data and persist it somewhere.

Open two terminals on the Salt Master host.

# on terminal 1 run:
python listen_to_master_bus.py
# on terminal 2 run:
salt '*' --return zeromq cmd.run_all '/usr/lib/nagios/plugins/check_procs -w 150 -c 200'
Same remote execution check as before but now our new returner will make data appear in terminal 1!

This zeromq returner is more of a proof-of-concept. I think the salt remote execution command line tool should allow end-users to provide a --tag so that data may be feed directly back to a third-party script listening to the Salt Master's event bus which filters on the particular tag. My next step is to look into what it would take to build in this functionality.

In the future I want to rig up the Salt scheduler to invoke these remote execution checks on a steady and predictable cadence. Nagios historically runs checks every 5 minutes. The Salt scheduler will allow us schedule different checks with different frequencies. For example, I might want my load checks and metrics to be collected every 10 secs, but my disk capacity usage checked every 2 minutes. This fine-grain control is super powerful!

Configuration Management and the Golden Image

2014-02-21T17:19:00-05:00

When operations first became a thing, system administrators stood up servers using a base image from their favourite distribution. Things were done manually. Some administrators created their own distros, some wrote customised shell scripts to be run once-and-only-once to provision software and settings. This method worked, but it was slow, manual, and the human element caused defects. Then the request came in to stand up 100 servers the exact same way.

System admins resolved this large request by coming up with a Standard Operating Environment (SOE) image which would end up becoming the "golden image". This golden image was the source of truth, and we built hundreds, thousands of machines this way. All systems were the same, or rather started out the same, but it didn't take long for deviations occur.

(Norton Ghost, DD, ISOs)

https://en.wikipedia.org/wiki/Standard_Operating_Environment

The golden image by itself was a flawed idea. The task of creating a golden image was difficult and required a lot of work. Not only was it technical, but there was a lot of politics involved in what was worthy of inclusion. We didn't want to add cruft to every machine. Also the golden image was only updated every couple of years, so it would quickly become outdated and it still required provisioning scripts. Systems already in production didn't get configuration updates that were recently added to the golden image. There had to be a better way, and there was...

Some novel and smart system administrators who also knew how to program decided that they didn't want to maintain a golden image and deal with all the headaches involved and opted to use the light weight distribution image and then build sophisticated remote execution software to manage and maintain each server's configuration. Later on they built configuration management systems on top of this remote execution layer and were finally able to keep each system up-to-date, regardless of when it was deployed. They were geniuses and everyone who knew anything quickly rushed to implement remote execution and configuration management. It was the right way to manage servers, until the cloud drifted in.

(SaltStack, Ansible, Puppet, Chef, CFEngine)

In the day of cloud computing, we needed to scale up and down servers in seconds. A complex configuration manifest could take hours to run from start to finish. Configuration management was too slow. Each server needed to download, install, and configure the software stack, in real-time. Sure we could spin up multiple machines in parallel, but it was still slow. We started to look back at the golden age of the golden image, when a server was built and booted in moments. How could we pair the speed of the golden image with the flexibility of configuration management?

In the future could we use configuration management manifests and revision control to document how to build the golden image and then take a snapshot? Could we overlay multiple layers or dataset of images on top of each other? Perhaps we take a step way back and create golden images with a combination of a highly customizable distribution like Gentoo and configuration management.

(Joyant SmartOS zone datasets, Docker container images, Vagrant box files, AWS AMI, Digital Ocean Snapshots, etc)

Automatic Backups

2014-02-07T17:14:00-05:00

I maintain many tools for creating automatic backups for various systems. This page will act as a hub to each of those solutions.

tar-back:: tar-back is a backup utility to tar and gzip target filesystems. It supports a custom retention, filter exclusions, and backup directory.
virt-back:: virt-back virt-back is a python application that uses the libvirt API to safely shutdown, gzip, and restart guests. Supports KVM, QEMU, XEN, Virtualbox, and more.
mysql-back:: mysql-back is a backup utility script to dump (backup) and gzip every MySQL database on a host.
smart-back:: smart-back is a utility used to backup of every virtual machine on a SmartOS hypervisor.

tar-back

2014-02-07T16:49:00-05:00

tar-back is a backup utility to tar and gzip target filesystems.

It supports a custom retention, filter exclusions, and backup directory.

I use tar-back in combination with cron to perform regular backups of all localhost filesystems into /archive/fs. I then have a central long term storage server that collects the /archive partition from every host.

tar-back:

#!/usr/bin/env python

DESCRIPTION = """A backup utility to tar and gzip
target filesystems. Custom retention, filter exclusions and
backup directory.
"""

from os import path
from sys import exit
from shutil import move
from optparse import OptionParser
import tarfile

def filter_exclusions( tarinfo ):
    """Accept tarinfo, return tarinfo or None"""
    if o.filters:
        for filtr in o.filters:
            if filtr in tarinfo.name:
                return None
    return tarinfo

def exclude_exclusions( filename ):
    """Accept filename, return True or False"""
    """support for python 2.6 or lower"""
    if o.filters:
        for filtr in o.filters:
            if filtr in filename:
                return True
    return False

def file_rotate( target, retention = 3 ):
    """file rotation routine"""
    for i in range( retention-2, 0, -1 ): # count backwards
        old_name = "%s.%s" % ( target, i )
        new_name = "%s.%s" % ( target, i + 1 )
        try: move( old_name, new_name  )
        except IOError: pass
    move( target, target + '.1' )

if __name__ == '__main__':

    p = OptionParser()  # create an option parser object

    p.set_description( DESCRIPTION )

    p.add_option( '-t', '--targets',
      help='list of target filesystems to backup (coma delimited)',
      default=None, dest='targets', metavar='"/home"',
    )

    p.add_option( '-b', '--backup-dir',
      help='path to backup directory',
      dest='backdir', metavar='"PATH"',
    )

    p.add_option( '-f', '--filters',
      help='list of filter patterns to exclude (coma delimited)',
      default=None, dest='filters', metavar='".mp3"',
    )

    p.add_option( '-r', '--retention',
      help='backups to retain [default: 3]',
      default=3, type='int', dest='retention', metavar='amount',
    )

    o, args = p.parse_args()  # parse options and args

    extension = '.tar.gz'

    if not o.targets:
        exit( "missing filesystem target, run --help" )
    if not o.backdir:
        exit( "missing backup directory, run --help" )
    if not path.isdir( o.backdir ):
        exit( "backup-dir %s does not exist" % o.backdir )

    if o.filters:
        o.filters = o.filters.split(',')

    o.targets = o.targets.split(',')

    for target in o.targets:
        slug = target.replace( '/', '-' ).lstrip( '-' ) # change / to -

        if slug == '': slug = 'r' # / slug is empty (root)

        tarpath = path.join( o.backdir, slug + extension )

        if path.isfile( tarpath ):
            file_rotate( tarpath, o.retention )

        tar = tarfile.open( tarpath, 'w:gz' )
        try:
            tar.add( target, filter=filter_exclusions )
        except TypeError:
            tar.add( target, exclude=exclude_exclusions )
        tar.close()

mysql-back

2014-02-07T16:39:00-05:00

mysql-backis a backup utility script to dump (backup) and gzip every MySQL database on a host.

I use mysql-back in combination with cron to perform regular database dumps of MySQL servers to the /archive/db partition on localhost. I then have a central long term storage server that collects the /archive partition from every host.

mysql-back:

#!/bin/bash

USER="root"
PASSWORD=""
TODAY=`date +%Y-%m-%d`
OUTPUTDIR="/archive/db/$TODAY"
MYSQLDUMP=`which mysqldump`
MYSQL=`which mysql`
GZIP=`which gzip`

mkdir $OUTPUTDIR

# get a list of databases
databases=`$MYSQL --user=$USER --password=$PASSWORD \
 -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

# dump each database in turn
for db in $databases; do
    $MYSQLDUMP --force --opt --user=$USER --password=$PASSWORD \
    --databases $db > "$OUTPUTDIR/$db.sql"
done

# compress all of the files
$GZIP $OUTPUTDIR/*

# clean up all dumps 30 days old
find /archive/db -ctime +30 -type d | xargs rm -rf

If you ever need to restore, like in my case when I moved all my databases over to a brand new Percona MySQL SmartOS zone, I used the following command:

zcat *sql.gz | mysql -u root -p

The Three Deployment Management Strategies

2014-02-03T15:06:00-05:00

There are three deployment management strategies that could be used to maintain a system. Each has pros and cons which I outline in this document.

run once

A proceedure that is run once and only once to setup a system's configuration values and settings. A semaphore or flag generally blocks repeated executions to prevent an undesirable outcome.

Development Difficulty: LOW - binary state aware (has run/has not run)

run always

A proceedure that is safe to run multiple times but uses a brute force method to setup a system's configuration and settings. It will self-heal at the expense of clobbering existing state.

Development Difficulty: MEDIUM - not state aware - attention must be spent to allow multiple executions

run as needed

A proceedure that is safe to run multiple times and checks a system's configuration and settings before it sets a system's configuration or settings. "checks before it steps". It will self-heal only the values it needs to.

Development Difficulty: HIGH - state aware - must know how to both get and set values and make desicions based on what it finds.

How to reset HP iLO Lights-Out User and Password Settings with IPMItool

2014-01-25T09:18:00-05:00

Warning:: Do NOT follow guides that suggest to make a DOS boot disk, this is over complicated.

Use the ipmitool which ships with most Unix based operating systems. I tested on SmartOS and Ubuntu Linux. Use a live boot disk if you must.

Run ipmitool user list to list all users installed on the iLO.
Choose a user ID from the list and run ipmitool user set password [ID].
Last, make sure the user is enabled with ipmitool user enable [ID].

Here is a complete set of commands I used to reset user ID 6:

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user list
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
2   ROUSER           true    false      false      Unknown (0x00)
3   USERID           true    false      false      Unknown (0x00)
4   OEM              true    false      false      Unknown (0x00)
5   Operator         true    false      false      Unknown (0x00)
6   admin            true    false      false      Unknown (0x00)
... truncated ...
15  admin            true    false      false      Unknown (0x00)
16  OEM              true    false      false      Unknown (0x00)

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user set password 6 mynew-password

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user enable 6

Next I logged into the web GUI and cleaned up this crazy list of users. You may also need to change the privileges on a user ID to grant admin level access. Run ipmitool user for a list of possible sub-commands.

After I finished poking around with the web GUI, I decided to learn a bit more about IPMI and the ipmitool command.

I figured out how to get sensor information over the LAN using this command:

sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin sensor

I was even able to log into the server OS via Serial-over-LAN (SOL) using this command:

sudo ipmitool -I lanplus -H guy-ilo.foxhop.net -U admin sol activate

Serial-over-LAN completely resolves the need for a complicated JAVA/KVM (keyboard video mouse) setup, I was able to reboot the server and watch the machine POST over the serial connection! Now you don't need to shell out $229+ on an Advanced 1 yr single server Licence for HP Lights-Out 100i (LO100i)!

I uploaded a video showing Serial-Over-LAN with IPMItools to an HP Proliant DL160 G6 running SmartOS.

Update:

Documenting a few other useful commands for myself here.

Power on and off the server chassis:

sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power status
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power on
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power soft
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power off
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power cycle

Update:

Apparently after you know the ILO username and password you may also use SSH to connect and manage the server:

ssh admin@guy-ilo.foxhop.net
admin@guile-ilo.foxhop.net's password:

Lights-Out 100 Management
Copyright 2005-2007 ServerEngines Corporation
Copyright 2006-2007 Hewlett-Packard Development Company, L.P.

/./-> help
Root Directory

/./-> show
    /./
    Targets
        system1
        map1

    Properties

    Verbs
        cd
        version
        exit
        show
        help

/./-> cd system1
/./system1/-> show
    /./system1/
    Targets
        oemhp_sensors
        oemhp_frus
        console1
        led1

    Properties
        name=DL180(Aspen)    _R
        enabledstate=enabled

    Verbs
        cd
        version
        exit
        show
        reset
        start
        stop
        help

You can even trigger the server OS to stop change run levels or mess with chassis power for more extreme measures.

/./system1/-> stop
System1 stopped.

Update:

I run FreeNAS on an HP DL180 G6 and just replaced my p410 controller with an LSI SAS9220-8i (IBM M1015) flashed to IT mode. The stock cables are long enough. I did not have issues with fans running at high RPM.

(I did the same replacement on an HP DL160 G6 running SmartOS and it didn't have a high fan RPM issues either)

How I added two Seagate 240G SSDs as SmartOS L2ARC

2014-01-05T14:56:00-05:00

How I added two Seagate 240G SSDs as SmartOS L2ARC

removed icepacks from two western digital velociraptors
installed ssds into icepacks
installed icepacks into HP hotswap trays
installed trays into HP prolaient g6 server

How to list all drive installed in Solaris, Open Solaris, or SmartOS

iostat -eE

format

AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@0,0/pci103c,330b@1f,2/disk@0,0
       1. c1t1d0
          /pci@0,0/pci103c,330b@1f,2/disk@1,0
       2. c1t2d0
          /pci@0,0/pci103c,330b@1f,2/disk@2,0
       3. c1t3d0
          /pci@0,0/pci103c,330b@1f,2/disk@3,0

list zpools:

zpool list

NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones  1.81T  41.4G  1.77T         -     2%  1.00x  ONLINE  -

Add the two 240G SSDs as L2ARC devices:

zpool add zones cache c1t2d0 c1t3d0

Look at the iostats of the drives in the zpool

zpool iostat -v

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zones       41.4G  1.77T      5     46  78.7K   730K
  mirror    41.4G  1.77T      5     46  78.7K   730K
    c1t0d0      -      -      0     15  41.5K   733K
    c1t1d0      -      -      0     15  41.6K   733K
cache           -      -      -      -      -      -
  c1t2d0    14.6M   224G      0      3  2.48K   273K
  c1t3d0    45.5M   224G      0      6  2.48K   852K
----------  -----  -----  -----  -----  -----  -----

This machine is a test hypervisor and after reviewing the output from zpool iostat -v over the last couple days, I'm pretty sure adding L2ARC to this box was not needed, but it was a great learning experience.

Test Game Engine with Python and SFML

2014-01-02T00:30:00-05:00

Over this holiday season, Christmas and New Years, I took the time to mess with some Game Development. I wrote a demo game engine using Python and SFML. I plan to use this post to track my progress.

Video Evolution Playlist

All videos

2014-01-01

2014-01-02

Heka, World!

2013-12-20T18:25:00-05:00

This post serves as a "Hello World" for the data collection and processing software called Heka. Heka is written in Go and was open sourced by Mozilla, the same fabulous group that brings us Firefox!

I intend to use Heka to replace Logstash agents by sending logs directly to ElasticSearch and continuing to use Kibana3 for visualizations. Also I aim to start collecting metrics and sending to a central Whisper back-end to fuel Graphite charts. All that we need to make Heka take on these responsibilities is one binary and some custom configuration.

Heka: Hello, World!

This mostly contrived example will show how to use Heka to watch /tmp/input.log and write to /tmp/output.log.

Step 1: install Heka

Compile from source or install the Heka package for your operating system.

Step 2: create a Heka TOML configuration file

/tmp/hello-heka.conf:

[hello_heka_input_log]
type = "LogstreamerInput"
log_directory = "/tmp"
file_match = 'input\.log'

[hello_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "hello_heka_output_encoder"

[hello_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false

Step 3: start Hekad and test!

in terminal 1:

sudo hekad -config=/tmp/hello-heka.conf

in terminal 2:
```
tail -f /tmp/output.log
```
in terminal 3:
```
echo 'Heka, World!' >> /tmp/input.log
```

Like magic the data appended to input.log will appear in output.log

Heka also has great docs and a number of input and output plugins, but don't take my word for it, try it yourself!

sensu-salt

2013-12-17T11:35:00-05:00

A while back I explained how to Create your own fleet of servers with Digital Ocean and salt-cloud. Today I will extend that post and show how I deployed a test environment for Sensu, an open source monitoring framework.

Before I test out new infrastructure software, I always attempt to write deployment manifests. I subject myself to this exercise for the following reasons:

to get better at configuration management (self growth)
to enable quick setup and tear down of test environments (save money)
to make sure the new software may be deployed and maintained in configuration management (a must)
to document the install process and leave comments (my notes double as automation scripts!)
to allow knowledge transfer (sharing is caring, you're welcome!)

The state formulas in this post were tested on Ubuntu 12.04.3 x64bit.

Clone or fork the Salt-State tree

https://github.com/russellballestrini/sensu-salt

git clone https://github.com/russellballestrini/sensu-salt.git

Declare deployment targets

Before deployment, we must declare some targets in the top.sls file:

'*':
  - git

'sensu-client*':
  - sensu.client

'sensu-server*':
  - rabbitmq.server
  - redis.server
  - sensu.server
  - sensu.client

Stand up Sensu test environment

I used the following salt-cloud command to create the sensu monitor server:

salt-cloud --profile ubuntu_do sensu-server && salt 'sensu-server' state.highstate

Once the sensu-server was live, I altered the client-config.json and modified the RabbitMQ host with the new sensu-server's IP or DNS record.

I then spun up 4 sensu-clients using the following command:

salt-cloud -P --profile ubuntu_do sensu-client1 sensu-client2 sensu-client3 sensu-client4 && salt 'sensu-client*' state.highstate

This caused 4 cloud servers to be spawned in parallel, and when the provisioning finished, they instantly appeared in the sensu-dashboard which runs on the sensu-server.

Tear down Sensu test environment

Later when I was done messing with writing checks, I used the following salt-cloud commands to destroy the sensu server and clients:

salt-cloud --destroy sensu-server
salt-cloud --destroy sensu-client1 sensu-client2 sensu-client3 sensu-client4

Hackathon 2013 Virtualization

2013-12-13T20:56:00-05:00

As a warning before we dive into things, this post is less of a formal publication and more of a stream of conscience.

My employer newcars.com has allowed the technical staff to host hackathon! Over the past couple weeks I have had quite a few ideas tumbling around in my head:

Standup a central logging server
Standup Sensu for monitoring
Bake-off and document some KVM virtualization hypervisors (Ubuntu or SmartOS)
Test Docker and document findings

Ultimately I have chosen to dedicate my time to testing out virtualization on the new Cisco UCS Blade servers. I plan to test Ubuntu KVM first.

KVM (Ubuntu 12.04)

I decided to use a Salt-stack configuration management formulas to document how I transformed kvmtest02 (a regular Ubuntu 12.04 server) into a KVM hypervisor. I use kvmtest0a and kvmtest02b when refering to the virtual machines living on the hypervisor. Here is the formula:

kvm/init.sls:

# https://help.ubuntu.com/community/KVM
# Ubuntu server a KVM - Kernel Virtual Machine hypervisor

# the official ubuntu document suggests we install the following
kvm-hypervisor:
  pkg.installed:
    - names:
      - qemu-kvm
      - libvirt-bin
      - ubuntu-vm-builder
      - bridge-utils

# we need to make all of our ops people part of the libvirtd group
# so that they may create and manipulate guests. we skip this for now
# and assume all virtual machines will be owned by root.

# these are optional packages which gives us a GUI for the hypervisor
virt-manager-and-viewer:
  pkg.installed:
    - names:
      - virt-manager
      - virt-viewer
    - require:
      - pkg: kvm-hypervisor

# create a directory to hold virtual machine image files
kvm-image-dir:
  file.directory:
    - name: /cars/vms
    - user: root
    - group: root
    - mode: 775

I used the following to install the formula to the test hypervisor:

salt 'kvmtest02.example.com' state.highstate

The salt highstate reported everything was good (green), so I moved on to setting up the bridge networking interface. At this point I don't want to figure out the logistics setting up the network bridge in configuration management, so I simply manually edited /etc/network/interfaces to look like this (substitute your own network values):

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#iface eth0 inet manual

# https://help.ubuntu.com/community/KVM/Networking
# create a bridge so guest VMs may have their own identities
auto br0
iface br0 inet static
    address XXX.XX.89.42
    netmask 255.255.255.0
    network XXX.XX.89.0
    broadcast XXX.XX.89.255
    gateway XXX.XX.89.1

    # dns-* options are implemented by the resolvconf package
    dns-nameservers XXX.XX.254.225 XXX.XX.254.225
    dns-search example.com

    # bridge_* options are implemented by bridge-utils package
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

Then, I crossed my fingers and reloaded the network stack using this command:

/etc/init.d/networking restart

I also used this to "bounce" the bridge network interface:

ifdown br0 && ifup br0

I verified with ifconfig.

I'm ready to create my first VM. There are many different ways to boot the VM and install the operating system. KVM is fully virtualized so nearly any operating system may be install on the VM.

If you have not already, please get familiarized with the following two commands:

virsh
virt-install

The virsh command is an unified tool / API for working with hypervisors that support the libvirt library. Currently virsh supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox and VMware ESX. For more information run virsh help.

The virt-install command line tool is used to create new KVM, Xen, or Linux container guests using the "libvirt" hypervisor management library. For more information run man virt-install.

Woah, virsh and virt-install both support LXC?

We have decided to only support Ubuntu 12.04 at this time, so obviously we will choose that for our guest's OS. Now we need to decide on an installation strategy. We may use the following techniques to perform an install:

boot from local CD-rom
boot from local ISO
boot from PXE server on our local vLAN
boot from netboot image from anywhere in the world

We will choose the PXE boot strategy because our vLAN environment already uses that for physical hosts.

We will use the virt-install helper tool to create the virtual machine's "hardware" with various flags. Lets document the creation of this guest in a simple bash script so we may reference it again in the future.

/tmp/create-kvmtest02-a.sh:

HOSTNAME=kvmtest02-a
DOMAIN=example.com

sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $HOSTNAME \
  --vcpu 2 \
  --ram 4096 \
  --disk /cars/vms/$HOSTNAME.qcow2,size=20 \
  --os-type linux \
  --graphics vnc \
  --network bridge=br0,mac=RANDOM \
  --autostart \
  --pxe

This was not used, but shows the flags to perform a netboot from Internet:

--location=https://archive.ubuntu.com/ubuntu/dists/raring/main/installer-amd64/ \
--extra-args="auto=true priority=critical keymap=us locale=en_US hostname=$HOSTNAME domain=$DOMAIN url=http://192.168.1.22/my-debconf-preseed.txt"

I created the vm:

bash /tmp/create-kvmtest02-a.sh

virt-install drops you into the "console" of the VM, but this will not work yet, so we use ctrl+] to break out and get back to our hypervisor. Use virsh list to list all the currently running VMs.

Lets use virt-viewer to view the VMs display. For this we need to SSH to the hypervisor and forward our display to our workstation, we do this with the -X flag. For example:

ssh -X kvmtest02

Now we can launch virt-viewer on the remote hypervisor, and the GUI will be drawn on our local X display!

virt-viewer kvmtest02-a

Once I got that to work, I also tested virt-manager which gives a GUI to control all guests on the remote hypervisor.

virt-manager

Now we need to determine the auto-generated MAC Address of the new virtual machine.

virsh dumpxml kvmtest02-a | grep -i "mac "
  mac address='52:54:00:47:86:8e'

We need to add this MAC address to our PXE server's DHCP configuration to allocate the IP and tell it where to PXE-boot from.

During a real deployment we would get an IP address allocated and an A record and PTR setup for new servers. This is a test and I will be destroying all traces of this virtual machine after presenting during the hackathon, so for now I'm going to skip the DNS entries and "steal" an IP address. I must be VERY careful not to use an IP address already in production. First use dig to find an IP without a record, then attempt to ping and use NMAP on the IP.

dig -x XXX.XX.89.240 +short
ping XXX.XX.89.240
nmap XXX.XX.89.240 -PN

The IP address checked out, it didn't have a PTR, it didn't respond to pings, and using nmap proved there were no open ports. I'm very confident this IP address is not in use.

I added a record to our DHCP / PXE server for this Virtual Machine. I attempted multiple times to pxe boot the VM, but the network stack was never automatically configured... The DHCP server was discovering the new VMs MAC and offering the proper IP address, as noted by these log lines:

Dec 13 07:57:43 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:43 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0

I wasted about 4 hours attempting to troubleshoot and diagnose why the VM wouldn't work with DHCP. I ended the night without any guests online...

The Next DAY!

So today I decided to stop trying to get DHCP and PXE working. I downloaded an Ubuntu server ISO to the hypervisor, and used virt-manager to mount the ISO on the guest and booted for a manual operating system install.

This did two things, it proved that the hypervisor's network bridge br0 worked for static network assigned settings and that something between the DHCP server and the hypervisor was preventing the DHCPOFFER answer from getting back to the VM. I looked into iptables firewall, removed apparmor, removed SELINUX and reviewed countless logs looking for hints... then moved on...

I was able to get salt-minion installed on the vm using our post-install-salt-minion.sh script, which I manually downloaded from the salt master. But to keep this test self contained, I pointed the VM's salt-minion to kvmtest02 which we already had setup as a test salt-master.

The salt-master saw the salt-minion's key right away, so I decided to target an install. This what I applied to the VM:

salt/top.sls:

'kvmtest02.example.com':
  - kvm

'kvmtest02b.example.com':
  - virtualenv
  - python-ldap
  - nginx
  - the-gateway

salt-pillar/top.sls

# kvmtest02b gateway in a VM experiment
'kvmtest02b.example.com':
  - nginx
  - the-gateway.alpha
  - deployment-keys.the-gateway-alpha

The stack was successfully deployed to the VM and proved that virtual machines are a viable solution for stage or production. It also gave me the change to test out this particular deployment again and found a few gotchas we need to create maintenance tickets for.

Without configuration management, it would have taken weeks to deploy this custom application stack. The install with configuration management took less then 10 minutes!

One of the KVM related snags I ran into was that Nginx does some fun calculations with cpu cache to determine hash table sizes. As a temporary work around, until I can devote more research time, I raised up the following three hash table directives in the http section of nginx.conf:

server_names_hash_bucket_size 512;
types_hash_bucket_size 512;
types_hash_max_size 4096;

SmartOS

snippet from /etc/dhcp/dhcpd.conf

# SmartOS hypervisor group to boot image
group "smartos-hypervisors" {
  next-server xxx.xx.89.71;

   host smrtest01-eth0 {
        hardware ethernet 00:25:B5:02:07:DF;
        option host-name "ncstest01";
        fixed-address smrtest01.example.com;

        if exists user-class and option user-class = "iPXE" {
           filename = "smartos/menu.ipxe";
        } else {
           filename = "smartos/undionly.kpxe";
        }
    }

}

mkdir /cars/tftp/smartos
cd /cars/tftp/smartos
wget http://boot.ipxe.org/undionly.kpxe
wget https://download.joyent.com/pub/iso/platform-latest.tgz
tar -xzvf platform-latest.tgz
mv platform-20130629T040542Z 20130629T040542Z
mkdir platform
mv i86pc/ platform/

create boot menu that we referenced, vim /cars/tftp/smartos/menu.ipxe

#!ipxe

kernel /smartos/20130629T040542Z/platform/i86pc/kernel/amd64/unix
initrd /smartos/20130629T040542Z/platform/i86pc/amd64/boot_archive
boot

Make sure to replace platform version with current.

I was able to get the blade server to PXE boot the image, but it seems SmartOS doesn't really support the SANs. SmartOS really expects to see local disks, and to build a ZFS pool on top of that. Basically SmartOS could be used to build a SAN, so they didn't put much effort in supporting SANs. After I figured this out I abandoned this test. We could revist this again, using one of the Dell servers, or use it to stand up a really powerful Alpha server environment.

LXC

Run /usr/bin/httpd in a linux container guest (LXC). Resource usage is capped at 512 MB of ram and 2 host cpus:

virt-install \
--connect lxc:/// \
--name lxctest02-a \
--ram 512 \
--vcpus 2 \
--init /usr/bin/httpd

Discussion points

Why doesn't DHCP work on bridge?
If we use virtualization, we need to come up with a plan for IP addresses, like possibly allocate ~5 IP addresses to a hypervisor host
We need to come up with a naming convention for guests, in testing I appended a letter to the hypervisor name kvmtest02 so the guests names were kvmtest02a and kvmtest02b, is this plausible going forward?

If I had more time ...

I would have liked to test out LXC
I would have liked to test out Docker
I would have liked to test out physical to virtual migrations

Backup all virtual machines on a SmartOS hypervisor with smart-back.sh

2013-10-27T20:45:00-04:00

This post will explain how to create a cronjob to backup of every virtual machine on a SmartOS hypervisor.

Create the following bash script in /opt/smart-back.sh:

#!/usr/bin/bash

# Backup all virtual machines on a SmartOS hypervisor
# Author:  russell@ballestrini.net
# Website: https://russell.ballestrini.net/

# Backup directory without trailing slash
backupdir=/opt/backups

# temp dir where we ZFS send and gzip before moving to backupdir
tmpdir=/opt

svcadm enable autofs

for VM in `vmadm list -p -o alias,uuid`
  do
    # create an array called VM_PARTS splitting on ':'
    IFS=':' VM_PARTS=($VM)

    # create some helper varibles for alias and uuid
    alias=${VM_PARTS[0]}
    uuid=${VM_PARTS[1]}

    # echo "Backup started for $VM"
    vmadm send $uuid > $tmpdir/$alias

    # echo "Starting $VM"
    vmadm start $uuid

    # disk space is cheap, uncomment if you disagree.
    #pbzip2 $tmpdir/$alias

  done

mv $tmpdir/*.bz2 $backupdir

Create a cronjob entry to schedule the backups:

crontab -e

2 6 * * 0 /usr/bin/bash /opt/smart-back.sh

If I expand on this script much more, I plan to stick it into revision control.

If you look closely, I have also added a hack to enable autofs (svcadm enable autofs) which allows me to automount an NFS share on my remote FreeNAS by setting backupdir=/net/[ip-or-fqdn-of-freenas]/mnt/zfs-mirror/backup/vms.

We have scheduled a backup of each virtual machine on your SmartOS hypervisor!

If or when the time comes to restore a VM from a backup, use the following:

# decompress the backup file.
pbzip2 -d backup-file.bz2

# ingest the backup file into the hypervisor.
vmadm receive -f /path/to/backup-file

Just make sure the VM doesn't currently exist on the hypervisor.

This strategy is great for complete backups of machines which could be used during a manual migration, or if corruption happened to the VM and we wanted to restore to a previous version.

Simplify deployments with Upstart and uWSGI

2013-10-19T22:35:00-04:00

As you know from my previous post, I recently deleted LinkPeek.com and after struggling to get it back online, I vowed to start utilizing configuration management. During this exercise, I noticed that the architecture I use in production seems overly complicated.

The current production deployment stack:

Nginx listen on 80/443 proxy upstream 9901/9902
Upstart => Supervisord => Cherrypy/Paste listen on 9901/9902

Each of these services and processes have their own configuration files which must work together. Upstart needs to know the location of Supervisord's configuration files. Supervisord needs to know the location of Cherrypy/PasteDeploy's configuration files. Supervisord also must bring up a specified number of worker processes who listen on a pool of ports. Nginx needs to proxy upstream to that pool of worker ports.

This architecture seems difficult to automate because of the numerous places errors may sneak in. I started researching an alternative architecture and stumbled upon Nicholas Piel's Benchmark of Python Web Servers. The graphs Nicholas compiled allowed me to narrow down a list of potential replacements.

I chose to review uWSGI first because it was a top contender on every chart. After briefly testing uWSGI, I halted my explorations and selected it as the winner. I know you probably feel that was a premature decision but uWSGI fit what I was looking for.

The new simplified stack:

Nginx listen on 80/443 proxy upstream 5200
Upstart => uWSGI listen on 5200

The uWSGI server possesses great performance out-of-the-box but also presents many options for fine tuning. These options may be specified either directly in the Upstart script using flags or in a configuration file (like a Pyramid .ini). Either way, the stack uses less files, less processes, and less complexity then the original architecture. For LinkPeek deployments I decided to place all uWSGI related configuration directly in the Upstart script which I embedded below:

linkpeek/weblinkpeek.conf | Upstart for starting uWSGI

description "Start the uWSGI master for the LinkPeek WEB"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

# run service as linkpeek user
setuid linkpeek
setgid linkpeek

# uWSGI command to execute and treat as a daemon
exec /path/to/linkpeek/web/env/bin/uwsgi --master --processes=4 --http=127.0.0.1:5200 --die-on-term --logto2=/path/to/linkpeek/web/uwsgi.log --virtualenv=/path/to/linkpeek/web/env --ini-paste=/path/to/linkpeek/web/linkpeek/production.ini

The web application is now daemonized and listening on port 5200 with 4 workers. I leave implementing the Nginx front end to the reader.

Postfix Salt State Formula

2013-10-17T21:28:00-04:00

The following formula was tested on Ubuntu and Debian however it would not take much work to test on other operating systems.

This state formula will install postfix and mutt. The postfix service will watch various configuration files for changes and restart accordingly.

This formula will also manage and watch the /etc/aliases file and invoke the newaliases command to initialize or re-initialize the alias database. This formula will also manage and watch the /etc/postfix/virtual file and invoke the postmap command to create or update a managed Postfix lookup table.

postfix/init.sls:

# Install mutt and postfix mutt packages.
#
# This formula supports setting an optional:
#
#  * 'aliases' file
#  * 'virtual' map file
#
# Both aliases and virtual use a pillar data schema
# which takes the following form:
#
# postfix:
#   aliases: |
#       postmaster: root
#       root: testuser
#       testuser: russell@example.com
#   virtual: |
#       example.com             this is a comment
#       test1@example.com       me@example.com
#       test2@example.com       me@example.com
#

# install mutt
mutt:
  pkg:
    - installed

# install postfix have service watch main.cf
postfix:
  pkg:
    - installed
  service:
    - running
    - enable: True
    - watch:
      - pkg: postfix
      - file: /etc/postfix/main.cf

# postfix main configuration file
/etc/postfix/main.cf:
  file.managed:
    - source: salt://postfix/main.cf
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

# manage /etc/aliases if data found in pillar
{% if 'aliases' in pillar.get('postfix', '') %}
/etc/aliases:
  file.managed:
    - source: salt://postfix/aliases
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

run-newaliases:
  cmd.wait:
    - name: newaliases
    - cwd: /
    - watch:
      - file: /etc/aliases
{% endif %}

# manage /etc/postfix/virtual if data found in pillar
{% if 'virtual' in pillar.get('postfix', '') %}
/etc/postfix/virtual:
  file.managed:
    - source: salt://postfix/virtual
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

run-postmap:
  cmd.wait:
    - name: /usr/sbin/postmap /etc/postfix/virtual
    - cwd: /
    - watch:
      - file: /etc/postfix/virtual
{% endif %}

postfix/aliases:

# Managed by config management
# See man 5 aliases for format
{{pillar['postfix']['aliases']}}

postfix/virtual:

# Managed by config management
{{pillar['postfix']['virtual']}}

postfix/main.cf:

# Managed by config management
# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = {{ grains['fqdn'] }}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = {{ grains['fqdn'] }}, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

{% if 'virtual' in pillar.get('postfix','') %}
virtual_alias_maps = hash:/etc/postfix/virtual
{% endif %}

Control a MongoDB collection in configuration management

2013-10-14T21:39:00-04:00

This post explains how to use configuration management (Salt Stack) to completely control a MongoDB collection. In our example we want to control a store's collection of plans.

First we create a JSON representation of the collection.

mongodb/plan.json:

{
  "_id" : { "$oid" : "4ef8b9e2be329f491d98f74b" },
  "cost" : 20, "description" : "development",
  "name" : "good", "count" : 6000
}
{
  "_id" : { "$oid" : "4ef8b9e8be329f491d98f74c" },
  "cost" : 60, "description" : "freelancers",
  "name" : "better", "count" : 36000
}
{
  "_id" : { "$oid" : "4ef8b9f0be329f491d98f74d" },
  "cost" : 180, "description" : "production",
  "name" : "best", "count" : 162000
}

Next we configure a salt state formula to manage the JSON file and watch it for changes.

mongodb/init.sls:

# install mongodb server
mongodb-server:
  pkg:
    - installed

# manage the store's plan.json
/tmp/plan.json:
  file.managed:
    - source: salt://mongodb/plan.json
    - user: root
    - group: root
    - mode: 644

# import the plan collection if it changes
import-plan-collection:
    cmd.wait:
      - name: mongoimport --db=store --collection=plan --upsert /tmp/plan.json
      - require:
        - pkg: mongodb-server
      - watch:
        - file: /tmp/plan.json

Now whenever plan.json is altered in configuration management, the file on the minion will update which will trigger a mongoimport with upsert to occur.

Optionally, we could replace --upsert with --drop which will drop the collection before re-importing thus removing stale records.

We now have a version controlled JSON file in configuration management and the power of MongoDB Document Objects in our application code!

Configuration Management vs Remote Execution

2013-07-11T23:15:00-04:00

What is configuration management?

In a perfect world configuration management provides a centralized, revision controlled, self-documented, change management location for manifests and formulas which both define how to build a complete system and organize a means of knowledge transfer. An infrastructure perfectly described in configuration management allows any single part of the system to be created, reproduced, multiplied, self-healed or even re-purposed.

When should I use configuration management?

Use configuration management whenever you intend to permanently alter system state or infrastructure data.

System state - the current state of the system:

operating system (distro,kernel,patches,hot-fixes)
software installed (base,role-specific)
services running (base,role-specific)
user access
etc

Infrastructure data - the information that describes the infrastructure:

asset information (models,specs,IP,DNS,rack-elevation,etc)
roles (app,web,db,proxy,load-balancer,etc)
current allocations (allocated,unallocated,number of servers in each role,etc)
etc

What is remote execution?

Remote execution is the act of issuing commands to one or more remote systems. The most popular remote execution system in use today is SSH. SSH is great for maintaining a small group of dissimilar systems. Once the system count grows and similar roles present themselves, start looking at something like Fabric. Fabric is a python framework which builds on top of the SSH protocol and makes it possible to invoke the same command on hundreds of servers sequentially. Once the fleet count reaches thousands of servers and commands must run in parallel, look at Salt-stack's remote execution layer written with python and ZeroMQ.

When should I use remote execution?

Only use remote execution for ad hoc reports, data collection, or for temporary tests which have no expectation of persistence after research is complete. Frequent data collection jobs work best when implemented in a metrics collection or monitoring system. (I'll save that for another post)

When should I not use remote execution?

Do not use remote execution to change the state of the system or the data which describes the infrastructure! If a remote execution job changes either state or data it should be placed into the configuration management for the reasons mentioned above.

Understanding Salt Stack user and group management

2013-07-08T13:51:00-04:00

This state will create a user:

russell:
  user:
    - present

This state will create a user and a group. This also makes the user part of the group, and handles creating the group first:

russell:
  group:
    - present
  user:
    - present
    - groups:
      - russell
    - require:
      - group: russell

This state handles user and group generation along with password and ssh-key maintenance. This is all done securely using pillar to parameterize arguments:

# This state will create users accounts
#
# This state requires a pillar named 'users' with data formatted like:
#
# users:
#
#  tusername:
#    fullname: Test Username
#    uid: 1007
#    gid: 1007
#    groups:
#      - sudo
#      - ops
#    crypt: $password-hash-sha512-prefered
#    pub_ssh_keys:
#      - ssh-rsa list-of-public-keys tusername-sm
#
#  anotheruser: ... snipped ...

# loop over all users presented by pillar:
# create user's group, create user, then add pub keys
{% for username, details in pillar.get('users', {}).items() %}
{{ username }}:

  group:
    - present
    - name: {{ username }}
    - gid: {{ details.get('gid', '') }}

  user:
    - present
    - fullname: {{ details.get('fullname','') }}
    - name: {{ username }}
    - shell: /bin/bash
    - home: /home/{{ username }}
    - uid: {{ details.get('uid', '') }}
    - gid: {{ details.get('gid', '') }}
    - crypt: {{ details.get('crypt','') }}
    {% if 'groups' in details %}
    - groups:
      {% for group in details.get('groups', []) %}
      - {{ group }}
      {% endfor %}
    - require:
      {% for group in details.get('groups', []) %}
      - group: {{ group }}
      {% endfor %}
    {% endif %}

  {% if 'pub_ssh_keys' in details %}
  ssh_auth:
    - present
    - user: {{ username }}
    - names:
    {% for pub_ssh_key in details.get('pub_ssh_keys', []) %}
      - {{ pub_ssh_key }}
    {% endfor %}
    - require:
      - user: {{ username }}
  {% endif %}

{% endfor %}

Create your own fleet of servers with Digital Ocean and salt-cloud

2013-07-02T22:20:00-04:00

Have you heard about Digital Ocean? They offer a polished user interface, KVM guests with SSD storage, and an API to interact with a cloud of hypervisors. API integration got you down? Don't worry, salt-cloud has already integrated Digital Ocean among it's list of providers! The rest of this post illustrates the steps I took to configure salt-cloud to work with Digital Ocean.

This guide assumes you already have a:

salt-master
Digital Ocean account.

Step one, install the most recent version of salt-cloud.

On the salt-master:

sudo apt-get install salt-cloud

# or if you prefer ...
pip install salt-cloud==2015.5.0

# last verify it was successfully installed
salt-cloud --version

Step two, configure salt-cloud.

Salt-cloud uses the following files YAML files for configuration:

/etc/salt/cloud.conf.d/main.conf:: This is the main configuration file. I have the following statements:

minion:
    master: master.foxhop.net
    append_domain: foxhop.net

/etc/salt/cloud.providers/do.conf:: This is a provider configuration file for Digital Ocean (do). Collect your client_key and personal_access_token (api_key) from the Digital Ocean user dashboard. Also create an SSH key and add the public key using the dashboard:

# For Digital Ocean
do:
  provider: digital_ocean
  client_key: MyClientKeyLiftedFromDashboard
  personal_access_token: MyAPIKeyLiftedFromDashboard
  ssh_key_file: /keys/digital-ocean-salt-cloud
  ssh_key_name: digital-ocean-salt-cloud.pub

/etc/salt/cloud.profiles/do.conf:: This is the Digital Ocean profiles configuration file. We will create just two profiles for now, but you can create unlimited named combinations.

ubuntu-12-04-do-512:
  provider: do
  image: ubuntu-12-04-x64
  size: 512mb
  location: nyc1

ubuntu-14-04-do-512:
  provider: do
  image: ubuntu-14-04-x64
  size: 512mb
  location: nyc1

ssh_key_file:: This is your private SSH key located on your salt-master
ssh_key_name:: This is the name of the public key you added in your Digital Ocean dashboard
size:: The size or plan you would like to provision, 512mb is the smallest plan
location:: The geographical region, location, and/or data center
image:: The operating system image

After you configure the do provider in /etc/salt/cloud.providers you gain access to the following commands:

salt-cloud --list-sizes do
salt-cloud --list-locations do
salt-cloud --list-images do
salt-cloud --help

Lets provision a new cloud server!

salt-cloud --profile ubuntu-14-04-do-512 deejay

If all goes well you should have a newly provisioned server bootstrapped with salt-minion. The new minion's keys are already added to the salt-master. Now you just need to run highstate!

Add a custom header to all Salt managed files using pillar and jinja templates

2013-07-01T14:11:00-04:00

Salt-stack (salt) provides a solution for centralized configuration management and remote execution. One of the most basic things Salt provides is the ability to manage the contents of a file or a directory of files. Using Salt we can dictate the state of our minions and as a result we also gain auto-healing of configuration files.

Salt will clobber local changes to managed files and force the state to reflect the version in configuration management. In an effort to avoid confusing the uninformed, I place a header on each managed file which announces "THIS FILE IS MANAGED BY SALT".

To avoid repeating myself in each managed file, I came up with the following centralized solution -

First, I give all minions access to the headers pillar tree in top.sls:

base:
  '*':
    - headers

Next, I create a couple headers in in headers/init.sls:

headers:
  salt:
    file: |
        ################################
        # THIS FILE IS MANAGED BY SALT #
        ################################
    directory: |
        #####################################
        # THIS DIRECTORY IS MANAGED BY SALT #
        #####################################

Then, I parse each managed file in the state tree with jinja, for example hosts/init.sls:

/etc/hosts:
  file.managed:
    source: salt://hosts/hosts
    user: root
    group: group
    mode: 644
    template: jinja

Finally, in each file served by salt I add the jinja header substitution, for example hosts/hosts:

{{pillar['headers']['salt']['file']}}
127.0.0.1       localhost.localdomain localhost
::1     localhost6.localdomain6 localhost6

I use this same technique to declare most configuration options as parameters, like hostnames, IP addresses, ports, and more.

High load and CPU usage craftbukkit compared to vanilla minecraft

2013-06-20T23:35:00-04:00

I started researching the best ways to use salt to provision minecraft servers. I wrote a salt state formula for the vanilla minecraft server deployment. The deployment worked out great so I decided to try my luck with plugins.

In order to use plugins and mods we need to use a customized server package. I decided to try provisioning a craftbukkit server and quickly noticed something was very wrong. Vanilla Craftbukkit (with no plugins) produces very high load and CPU usage.

Here is a chart for proof. The spike is when I stopped the vanilla minecraft server and started the craftbukkit minecraft server:

Honey! I just DELETED LinkPeek.com

2013-05-26T13:19:00-04:00

During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need them. Boy was I lucky ...

Yes, leave it to me to inadvertently delete the VPS root disk. One of the major cloud providers places the "rename" and "remove" disk buttons right next to each other and I learned a nasty habit of clearing pop-ups without reading them (thanks Windows).

"Honey! I just deleted LinkPeek.com"

The horror... My stomach felt like I took a tumble in a roller-coaster. Instantly I tossed off my developer hat and put on my operations hat. I checked the off-site backups. I had nightly dumps of MongoDB and weekly tar backups of the /etc partition. The user data was in MongoDB and most of the system configuration information was in the tar. I used the tar to recover 2 upstart scripts, 2 supervisord scripts, 2 complex nginx confs, an ssl cert, and the pyramid production.ini.

I set out to stand up a new server, re-install the needed packages, recover the user data, and restore the service. After 1.5 hours of feverish typing, https://linkpeek.com/ was back online.

What I learned and my plan going forward

If you are a small team or a start-up, you must have somebody dedicated to operations. Without backups I would not have been able to gracefully recover. Most likely I would have reimbursed the existing members and shuttered the doors.

This experience was eye-opening. In my next couple of posts I will explain how I create and maintain backups and my next project will implement a configuration management and provisioning system.

This system will allow me to:

take out the human element of recovery
significantly reduce the time-to-recover from a catastrophic failure
test disaster recovery procedures before needing them
provision development and production environments without effort
have a reproducible blueprint of "how to build a LinkPeek server"

Survey Baby Monkey

2013-04-26T13:47:00-04:00

Automatic event hangout with cron

2013-04-09T09:17:00-04:00

Create an online only, hangout event

Create a new event with a date far into the future, like the year 2015. Go to the event's options > advanced and enable 'this event is online only' which will create a unique Hangout URI.

Create a cronjob

Create a cronjob on each device to open the web browser Monday-Friday at 12:49pm and open the unique Hangout URI.

Firefox cron example:

# Run Firefox at 12:49 EST each weekday and open hangout URI
49 12 * * 1-5 export DISPLAY=:0 && /usr/bin/firefox 'hangout-uri'

Chromium-browser cron example:

# Run Chromium-browser at 12:49 EST each weekday and open hangout URI
49 12 * * 1-5 export DISPLAY=:0 && /usr/bin/chromium-browser 'hangout-uri'

Guido name dropped tornado python tulip and pep-3156

2013-03-22T22:07:00-04:00

Pycon 2013 was excellent, in fact it was my first one I have attended.

I found it odd that django and Pyramid had plenty of talks but nobody mentioned tornado.

The only person that brought up tornado was Guido himself, who has been researching and developing async python since December 12th, 2012. Guido wants to add async API libraries to python core, and has been comparing his work to twisted and more importantly tornado. He is leveraging the existing solutions to stay on the correct track.

Async API's in Python core! This news was extra exciting for me, because I have already learned the power of async by messing around with tornado. Since the talk many people have approached me and asked for more information about async API's.

My favorite use for async is long polling for web applications. Here is a diagram that shows how great tornado is:

This type of polling can support thousands of concurrent users, all connected and waiting for a callback event to occur. I used tornado to build four2go, a real-time and multi-player browser-based-game.

virt-back's Domfetcher class returns doms from libvirt API

2013-03-02T15:04:00-05:00

I'm hooked...

I attended SCaLE 11x, my first technical conference, and had an amazing time. My favorite talk was Michael Day's "Advancements with Open Virtualization & KVM" (link to slides). Michael's presentation inspired me to continue my work on virt-back.

During my trip home I used the in-flight wifi to push this commit into the cloud from the clouds! This particular commit re-factored the dom object list generation into a simple-to-use class called Domfetcher. Domfetcher abstracts the libvirt API and grants access to the following helper methods:

get_all_doms( )

Return a list of all dom objects

get_doms_by_names( guest_names=[] )

Accept a list of guest_names, return a list of related dom objects

get_running_doms( )

Return a list of running dom objects

get_shutoff_doms( )

Return a list of shutoff but defined dom objects

This is an example of how to use Domfetcher:

import virtback

# optionally supply hypervisor uri
domfetcher = virtback.Domfetcher()

doms = domfetcher.get_running_doms()

for dom in doms:
    print dom.name()

for dom in doms:
    print dom.info()

for dom in doms:
    print dom.shutdown()

As always thanks for reading!

How to overload default function arguments in python using lambda

2013-01-12T11:22:00-05:00

Python Lambda functions are very powerful but I often forget how they work or the fun things they do. This post will document how to use a lambda to provide different default arguments to a function.

We will use the human function found in ago.py as an example - because I'm the module author and I really like it. Lets use the interactive python interpreter to run help on the human function.

>>> from ago import human
>>> help( human )

Help on function human in module ago:

human(dt, precision=2, past_tense='{} ago', future_tense='in {}')
    Accept a datetime or timedelta, return a human readable delta string

Shown above the human function accepts 1 argument and 3 named keyword arguments. The dt argument must be a datetime or timedelta object, the precision must be an integer, and the other two must be strings. If we didn't like the default arguments, we would need to specify (or pass in) new values each time we invoked the function.

Example: human(dt, 3, 'this happened {} ago!', 'in {} from now!'). If we know we will always want different default arguments we can create a lambda function to shorten the invocation length.

>>> h = lambda dt : human(dt, 3, 'this happened {} ago!', 'in {} from now!')
>>> print h( dt ) # h is much shorter then human, and still reusable!

Above creates a new function h who only accepts one argument dt. This function calls human with our default arguments.

This lambda is equivalent to this regular python function:

>>> def h( dt ):
...    return human(d, 3, 'this happened {} ago!', 'in {} from now!')
>>> print h( dt ) # h is much shorter then human, and still reusable!

Here is a working example to show the new lambda function h in action:

>>> from datetime import datetime
>>> from datetime import timedelta
>>> from ago import human
>>>
>>> h = lambda dt : human(dt, 3, 'this happened {} ago!', 'in {} from now!')
>>>
>>> present = datetime.now()
>>>
>>> for i in range( 1, 15 ):
...     if i % 2 == 0:
...         new_date = present - timedelta( i, i * i, i * i * i )
...     else:
...         new_date = present + timedelta( i, i * i, i * i * i )
...     h( new_date )
...
'in 22 hours, 9 minutes, 30 seconds from now!'
'this happened 2 days, 1 hour, 50 minutes ago!'
'in 2 days, 22 hours, 9 minutes from now!'
'this happened 4 days, 1 hour, 50 minutes ago!'
'in 4 days, 22 hours, 9 minutes from now!'
'this happened 6 days, 1 hour, 51 minutes ago!'
'in 6 days, 22 hours, 10 minutes from now!'
'this happened 8 days, 1 hour, 51 minutes ago!'
'in 8 days, 22 hours, 10 minutes from now!'
'this happened 10 days, 1 hour, 52 minutes ago!'
'in 10 days, 22 hours, 11 minutes from now!'
'this happened 12 days, 1 hour, 52 minutes ago!'
'in 12 days, 22 hours, 12 minutes from now!'
'this happened 14 days, 1 hour, 53 minutes ago!'

ago.py human readable timedelta 0.0.4 release

2013-01-09T00:04:00-05:00

We have released ago.py 0.0.4

pypi: https://pypi.python.org/pypi/ago

repo: https://bitbucket.org/russellballestrini/ago

Special thanks to David Beitey for supplying ideas and python code for this update!

All changes are backward compatible.

Change log:

added support for future dates
added optional past_tense keyword argument string for custom output
added optional future_tense keyword argument string for custom output
added 13 tests to help prevent regressions
Updated the README.rst documentation for better coverage

We were just shy of 800 downloads on pypi for ago-0.0.3, I hope ago-0.0.4 will perform even better!

Tips for getting pull requests approved

2012-12-12T12:50:00-05:00

Contents

Pull rejection sucks!
Start small
Don't add leading or trailing white space
Less is more
Only commit working code
Follow the leader
Comments, docs, and tests oh my!
Blog about the change
Pull requests are like paragraphs

Pull rejection sucks!

You have just coded, implemented, and submitted a pull request. A short while later the request is declined by an upstream maintainer and you feel crushed. We have all been there. Today I'm going to show you a better way. This article will teach you how to create pull requests that get approved.

Start small

You need to earn trust with the maintainers. Your first commit should be a small change which they cannot reject. Try to write a missing test or re-factor duplicate code. Correct a comment's accuracy or rename a variable to better reflect its purpose. Your first pull request should not alter how the program works.

Don't add leading or trailing white space

Additional whitespace will alter the diff output. This causes version control systems to flag lines as changed which is irritating and sometimes misleading.

Less is more

Minimize the number of changes to accomplish your goal. People are busy and at times lazy. Reduce the work the maintainers must do to perform a merge. Lowering the amount of lines to review should increase the chance of approval.

Only commit working code

Do not break the program, only commit working code. If the project has tests make sure they work before you commit.

Follow the leader

Try to mimic the maintainers. Follow the coding style and project layout even if it seems wrong. This is not your playground... yet. Before you commit, review the VCS logs to learn how verbose or terse your commit messages should be. When in Rome do as the romans do. This silly game of follow the leader reduces friction of an outsider committing to the project.

Comments, docs, and tests oh my!

Your first pull request should clarify or add to the existing documentation. Fix the README, adjust a comment, or write a test. These tasks might appear small but they serve to prove that you possess comprehension of the source code. They also do not alter the program's functionality.

Having a few of these pull requests under-your-belt will earn you trust which will eventuality translate to more responsibility in the future. You will also differentiate yourself from the rest because most people do not enjoy working on documentation.

Blog about the change

Write about the change, give reasons and examples. Include a link to your blog post in the pull request.

Pull requests are like paragraphs

If you were writing an essay, you would split up your ideas into separate paragraphs. A pull request has many qualities similar to a paragraph. Each commit should be related to the pull request's main objective. Commits of a pull request should stay focused and on topic.

A pull request is to a program as a paragraph is to an essay.

A commit is to a pull request as a sentence is to a paragraph.

In an essay, if you have more then one topic you should have more then one paragraph. Likewise when coding, only one idea or change per pull request.

Separating your ideas into different pull requests will grant the maintainers greater flexibility when they begin to integrate. They will have the ability to pick-and-choose which requests to merge and everybody wins!

how to drain an iPhone battery without needing passcode

2012-11-03T19:27:00-04:00

Press home button [ ]
Slide camera button up
Slide mode to video
Turn on flash
Put iPhone on table with light pointed down
Walk away inconspicuously

Extra points if you set the camera to record (just don't record yourself or sounds) which will fill up the iPhone capacity.

Explaining cache with python

2012-10-02T15:22:00-04:00

What is cache? I define cache as "a saved answer to a question". Caching can speed up an application if a computationally complex question is asked frequently. Instead of the computing the answer over and over, we can use the previously cached answer. This post will present one method of adding cache to a python program. Specifically we will write a program that computes prime numbers and saves the answers into cache for quick retrieval.

EDIT: The kind people of the Internet have expressed concern with my loose use of the term cache; the techniques that follow are most accurately described as memoization.

One algorithm for determining if a number is prime follows:

prime_flag = True # default state
x = 5 # number to test
if x == 1:
    prime_flag = False
else:
    for i in range( 2, x ):
        if x % i == 0:
            prime_flag = False
            break
print prime_flag

Create a prime_flag variable to hold the answer and default it to true. Let x be the number being tested and if x is equal to 1, the x is not prime. Otherwise iterate over each number in the range of 2 to x. Let i be the current number to be tested. if x is divided by i without any remainder, x is not prime. Set the prime_flag to False and break out of the loop. Print the result.

Next we will move the algorithm into a function which will allow for code reuse:

def is_prime( x ):
    """Determine if a number is prime, return Boolean"""
    prime_flag = True
    if x == 1:
        prime_flag = False
    else:
        for i in range( 2, x ):
            if x % i == 0:
                prime_flag = False
                break
    return prime_flag

# invoke function:
print is_prime( 5 ) # True
print is_prime( 4 ) # False

This function saves us a lot of typing and enables the ability to quickly determine if a given number is prime.

Next we will use a python dictionary to implement a result cache. Also by circumstance we introduce objects and classes.

class Primer( object ):
    def __init__( self ):
        """create a cache dictionary"""
        self.cache = {}

    def is_prime( self, x ):
        """Determine if x is prime, cache and return result"""
        if x in self.cache:
            return self.cache[x] # lookup result

        prime_flag = True

        if x == 1:
            prime_flag = False
        else:
            for i in range( 2, x ):
                if x % i == 0:
                    prime_flag = False
                    break

        self.cache[x] = prime_flag # cache result
        return prime_flag

p = Primer() # create a new primer object
p.is_prime( 5 ) # True
p.is_prime( 4 ) # False
p.is_prime( 5 ) # True and fetched from cache

What is great about this solution is that we can avoid looping and computation if an answer is already in cache. Looking up a cached result is much more efficient and will ultimately make a program feel more responsive.

Determining if 97352613 is prime takes my laptop nearly 18 seconds. Fetching the cached result seems to happen instantly.

>>> s1 = time();p.is_prime( 97352613 );e1 = time()
False # not prime
>>> s2 = time();p.is_prime( 97352613 );e2 = time()
False # not prime from cache
>>> e1 - s1
17.970067977905273 # seconds
>>> e2 - s2
2.5987625122070312e-05 # or approx .000026 seconds

A look-up will always beat a computation. Anything that can be cached, should be cached. I hope this helps clear things up.

The Pyramid community taught me the importance of test driven development

2012-09-22T14:03:00-04:00

Sontek's patch

I greeted the UPS man in the middle of the street to sign for my new Lenovo ThinkPad T430. Because this was My first brand-new laptop purchase I rationalized the time I spent tracking the package from the factory in China to my hands in Connecticut. Once inside, I opened the box and started installing Fedora 17. I couldn't help but to take in the new-electronics smell.

I've been without a laptop for more than a month so I was eager to get my development environment configured. Most of my tools ship with vanilla Linux, vim, python, hg mercurial, chromium browser, etc. My first goal was to get a development copy of linkpeek.com running locally. This took about 5 minutes and it seemed to be working fine until I noticed a few pages had errors. The errors seemed to be caused by a difference between Pyramid 1.3 and 1.4a1. But what was failing?

I posted a short message in #pyramid about the bug and minutes later I had multiple developers prodding for hints. "Could you post the whole traceback?", "What does your view look like?". I answered quickly and attempted to explain what I thought was going on. Turns out I was close but before I could finish explaining the problem, sontek had a working one-character-fix and was in the process finishing the tests to prove the patch. He also explained what I should do in the interim to patch locally.

In the next 4 hours a pull request was submitted to the upstream master and the patch was peer reviewed, accepted, and integrate by mcdonc. That impressed me, a lot. All Pylon Projects have strict policies about test coverage and now I understand why. Tests not only help produce better bug-free software but also act as a powerful tool when proving the validity of a patch. I plan to devote the next couple weeks to making test-driven-development a habit.

miniuri parser and ago human timedelta

2012-06-29T21:30:00-04:00

I just packaged and published a couple of python modules to pypi:

ago
miniuri

To install them, run:

Install:

pip install --upgrade ago miniuri

If you want to view their source code, look here:

miniuri
ago

I hope you enjoy them.

My top five suggestions for an independent developer creating a new product or service

2012-06-25T19:50:00-04:00

1. Write everyday. Build a blog for the project and write about milestones, progress, and hurdles. Also keep a personal blog and write about hobbies. Read some theory about "copy writing" and search engine optimization. Write personalized email responses to customers. Great communication skills will have the most impact on the success of your company. The best way to increase communication skills is to do it.

2. Spend at least 15 minutes on the company everyday. Even small or petty tasks will lift your company to the next goal. Over time the company will grow strong and people will give you attention. Building a company seems similar to building a character in an RPG. Instead however you will progress your company to its next level. Also, always look ahead when working, don't look back at prior achievements. Keep your eyes on the next milestone and keep moving forward. This type of behavior will promote growth and prevent getting stuck in the daily grind.

3. Do not fear manual processes. In the early stages of a company you should only build automation which will enhance customer experience. For everything else, build a repeatable manual process. Coding automation will typically cost more time to build than it will save. For example on LinkPeek I have not automated customer account de-activation because fortunately I don't have to do many of them. This manual process only takes a couple minutes to complete and gives me a chance to write a "thank you, and sorry to see you go" email to the fleeting customer. Manually de-activating (loosing) a customer is also a humbling experience. Keep a list of nice-to-have automations and review them in the far future. How far in the future? Far, like when your idea is validated and your company appears successful. Don't waste valuable time building automation for a service nobody will pay for or use. Time is your most scarce resource.

4. Always attempt to increase your luck. This might sound funny but the the following suggestions should reduce risk and also increase luck:

Meet fellow lucky people
Write on a blog
Advertise your company on side-projects that lack monetization
Think before wasting money using adsense or similar networks
Do market research before committing to an idea
Figure out how to make money before building product
Build the smallest version of the product that could still be sold
Try to stay focused on the things your customers will see
Don't build an admin dashboard until successful (profitable?)
Choose tasks that require the least effort but have the biggest impact
Don't be afraid to do things in the early stages that won't scale in the long run (example: personal email responses)
Listen to how early adopters describe your product; then on your website, marketing and emails reuse their words.
Attempt to keep marketing, newsletter, and support emails only few sentences long.
Start maintaining an opt-in email list

You might have the best mouse-trap but without luck it will never gain traction.

5. Have an unhealthy passion to support your customers and build your company. Six days ago I suffered a major chemical burn to my right eye. With my wife's help I continued to respond to support emails. Since I launched LinkPeek my focus and attention to my customers and company have never faltered. Dedication plays a key role to success.

Prevent a certain program from running too long in bash

2012-06-13T09:45:00-04:00

Update - I opensourced this script here: bash kira

I came up this this script to kill certain programs after they run for too long. This works like similar to a timeout. Warning this script is pretty harsh and kills the program.

#!/bin/bash
PROGRAM=replace-with-program-name
PIDSFILE=/tmp/kill-these.pids

for pid in `pidof $PROGRAM`
  do
    if grep -q $pid $PIDSFILE
      then
        kill $pid
    fi
  done

> $PIDSFILE

for pid in `pidof $PROGRAM`
  do
    echo $pid >> $PIDSFILE
  done

Then I wrote a cronjob to kill hung programs:

* * * * * /usr/local/sbin/killprogs.sh

Always attempt to scale vertically first

2012-06-10T21:52:00-04:00

I spent the weekend fretting because one of my servers was basically being DOS'd by paying customers. During the outage I started thinking about the best way to scale and how I could make the code-base more efficient.

Linux top reported high load, in the 20's. Eventually I figured out that the server was having IO performance issues.

I wasted a bunch of time attempting to fight fires. After about an hour of that I decided to scale my VPS vertically by giving it an extra 256mb of memory and a larger swap file (256mb to 1024mb).

These two changes were surprisingly effective and the IO issues resolved. Apparently the server was starving for memory which caused the host to swap which brought things to a crawl waiting for IO.

Crisis averted for the moment. Now I am free to think clearly and engineer a proper solution instead of attempting to put out fires.

If you ever encounter a similar situation, attempt the simplest fix. There is no shame in throwing more money at a problem if it will buy you time. In this case, an extra $10.00 a month relieved the performance issues and bought myself some time, for the moment.

High load on web server after updating from Ubuntu 10.04 to Ubuntu 12.04 LTS

2012-05-19T18:40:00-04:00

High load on web server after updating from Ubuntu 10.04 to Ubuntu 12.04 LTS

Check out charts which lineup to when I upgraded:

I couldn't determine the cause of the load average increase...

Update: The issue might be memory bound. Check out this graph that show much higher swap.

After much research this appears to be a load calculation and display problem with the newer Linux kernels. The community has found Commit-ID: c308b56b5398779cd3da0f62ab26b0453494c3d4 to be the problem. The commit causes incorrect high reported load averages can be reported under conditions of light load and high enter/exit idle frequency conditions (greater then 25 hertz).

A nice fellow named Doug from smythies.com researched the topic between tick and tickless linux kernels and the effect they have on load averages <http://www.smythies.com/~doug/network/load_average/new.html>. You should check it out.

How to rescue logs and config from a failed Citrix NetScaler App Gateway

2012-05-11T00:05:00-04:00

Today our production Citrix NetScaler broke. The box wouldn't boot and our only backup copy of the config was on the NetScaler itself.

Being the only Unix guy around I attempted to help out the admins working the outage. I SSH'd into the development NetScaler and noticed it runs on FreeBSD.

I suggested fetching the Hard drive and mounting it on a Linux computer. The NetScaler has one SATA (not SAS) disk so my desktop was compatible.

I installed the disk in the Linux tower and mounted the filesystem using the following command:

mount --read-only --type=ufs --test-opts ufstype=44bsd /dev/sda5 /mnt

Once mounted I was able to SCP interesting files to a safe location.

Warning, this procedure might void your warranty. If in doubt, call support first.

The most valuable registration field: How did you hear about us?

2012-04-26T22:21:00-04:00

How did you hear about us?

I first answered this question when joining Linode. I remember thinking "Wow, this is a great time to ask me!" because the real answer was still in my short term memory.

When I launched LinkPeek I applied this technique. After an amazing launch (thank you colleagues from HackerNews) the true significance of this field was exposed. I would argue that the answer to this simple question holds more value than collecting a username.

Here is why:

It may seem obvious after reading this post, but this is what I learned and I am sharing to help other startups.

Asking "How did you hear about us?" will help you determine:

how your customers found you
which of your marketing efforts are working
where to spend money or time marketing (and where to stop)
your true target market
how your customers will use your product

Here are some other benefits to this technique. It opens a dialog or conversation with the customer, which in return should help lower friction during the sale. You already have the the customer engaged during the signup process. I am firmly against the traditional method of surveying customers. I believe the same data can be collected without being abrasive or wasting the customers time.

"How did you hear about us?" is the gift that keeps on giving.

It is commonly accepted that shorter registration forms lead to better conversions. I totally agree, but in this case the sacrifice is worth learning more about my target market and customer needs.

How to ask this question properly:

Just ask! I second guessed my decision to add this field right before launch but that anxiety promptly faded after reading the first answer.
Don't use pre-filled answers! Having pre-filled answers is counter productive because you will not learn anything new. You are forcing the user into choosing one of your answers...
Allow the user to type as much as they like! A few of my customers nearly wrote a book in the text field.

If you liked this post, you should follow me on twitter `here <https://twitter.com/RussellBal>`__.

I just purchased Instagram for 1B and all I got was this lousy image filter

2012-04-09T17:47:00-04:00

Warning / Update!

This post was originally written from a place of jealousy and bitterness.

Turns out I was wrong about this transaction and for better-or-worse, Facebook (and Mark Zuckerburg) solidified their edge as the king of social for the last 6 years.

Original Post

Facebook purchased Instagram on Monday, April 9th 2012 for $1,000,000,000 US. Instagram, a free to use image sharing iPhone application, has absolutely NO REVENUE STREAM. The application is free to download, free to use, and has no monetization support from advertisements.

$1000000000.00 / 30000000 users = 33.33 per user

At the time of the sale Instagram had 30 million users in total (30,000,000). That means Facebook just paid heavy $33.33 per user... This is obviously a bad deal for Facebook.

Instagram did not have a method to monetize its application. I hedge my bets that Facebook will never recoup the one billion dollars spent on this deal.

Mark Zuckerburg, I just thought up the perfect T-Shirt for you:

I just purchased Instagram for 1B and all I got was this lousy image filter.

Trouble mounting filesystem on KVM guest after reboot

2012-03-27T12:04:00-04:00

Just found this out the hard way...

It looks like the attachment of /KVMROOT/guest-dev-app.img on guest-dev did not persist when the KVM host rebooted for patching.

As it appears the virsh attach-disk command works a lot like the mount command.

In order to have a disk attachment persist after a reboot, I think we still need to do a virsh edit.

the virsh attach-disk command is useful because it allows us to attach disk images to guests without restarting.

tldr;

virsh attach-disk is to mount as

virsh edit is to vim /etc/fstab

nosslsearch cname is a bad idea and solution

2012-03-26T11:51:00-04:00

Google SafeSearch and SSL Search for Schools suggests implementing the following changes to the network:

To utilize the no SSL option for your network, configure the DNS entry for www.google.com to be a CNAME for nosslsearch.google.com.

Here are the reasons why this is a bad idea and solution:

In order to create a CNAME record for www.google.com we need to become an authoritative master of that zone.
If you become an authoritative master you need to host all of Google's DNS resource records for the domain.
Google is asking us to DNS poison it's flag ship product on our networks.
If other companies follow suit the internet will quickly become unmanageable. DNS was not ment to work this way.
Not all networks have a local DNS server

This is a bad idea. Please change your stance on this matter.

Reference: https://support.google.com/websearch/bin/answer.py?hl=en&answer=186669

"I see" said the blind man, to the deaf dog, as he walked off the cliff.

2012-03-23T22:04:00-04:00

"I see" said the blind man, to the deaf dog, as he walked off the cliff.

As far as I can tell, I am the originator of this version of this quote.

EDIT: changed "originator of the quote" to "originator of this version of this quote".

What do you name your python virtualenv?

2012-03-07T21:42:00-05:00

What do you name your python virtualenv?

I name my virtualenv 'virtpy'. Is there a standard name being used out there?

Maybe we can come to a consensus as a standard name? Please feel free to post your virtualenv names here as a sort of poll.

How to save hundreds of dollars on groceries without clipping coupons

2012-03-03T19:54:00-05:00

This is the first time I've had a guest blogger on my site. It may sound campy but my wife Jenn wrote this article after explaining how our grocery bill decreased so drastically.

How to save hundreds of dollars on groceries without clipping coupons

I was recently re-working the cabinets of the kitchen, anticipating a new and organized year. With a growing family and limited space, I decided to do some consolidating.

I allocated a whole section of cabinet space to arts and craft supplies for the kids (play-doh, paints, etc), and another to bottles and formula. I suddenly realized that my food products remained scattered across the counter and there was only one lonely cabinet left to fill.

Among the food items on the counter, many were far expired, forgotten in the dusty back corners of the cabinets. I thought to myself “Hey! I worked hard to find coupons and sales for all these things.” and I regretfully filled a garbage bag with old food.

I had been clipping coupons and striving to match the savings of super couponers. These couponers, featured on popular reality shows, often recommend stock-piling when food items go on sale and you have coupons to purchase them.

I opened the refrigerator to find a similar and familiar situation: a head of lettuce that had seen better days, a tomato that had lost it's freshness, and other food items that were past the date of recommended consumption.

Looking into the garbage bag, I realized that... every week I was wastefully throwing food and money away. I decided to make a commitment that has cut my grocery bill in half! Do I still use coupons? YES, if they fit into my weekly plan. Do I still look at the sale flyers? ABSOLUTELY! But...

These two strategies have saved me much more money:

1. Cut down food storage spaces

If you currently use 4 food cabinets try designating only 2. In my case I consolidated my food items to 1 cabinet. This consolidation saved money in two distinct ways:

First, I am easily able to assess which food items I have and when they expire. This helps prevent re-buying a product we already have at home (Remember the time you were at the grocery store buying all the ingredients for those chocolate chip cookies and couldn't remember if you had any brown sugar? Undoubtebly you re-bought, probably to discover you had a brand new, unopened package in the cabinet).
Secondly, it prevents stock piling. I can buy only what I need for the week ahead, after all, it is all I have room for. If you generate the sense that “my kitchen is full of food”, it eliminates the need to buy for the sake of filling the cabinets.

2. Let your menu dictate your shopping list

My son complains “There are no more crackers." My husband sighs, “We're all out of bananas." My answer used to be probably a lot like yours “OK, I'll put it on my list." I had the perception that I constantly needed to replenish. It is unnecessary and wasteful to have three different brands of crackers, five types of cereal, and every kind of fruit known to the produce section. Without fail, I was throwing away money each week.

Instead, I sit down each week with two sheets of paper. One serves as dinner menu for the week and the other as my shopping list. When building my menu, I consider the sales and coupons I have on hand. If I notice a great deal on pasta sauce, we might have spaghetti one night and meatball subs another. I immediately begin a shopping list writing ONLY items used for the companion menu. Additionally, I add two breakfast choices and two lunch choices for the week if needed (some breakfast items such as pancake batter or a family size box of cereal often last longer). I also include 2 choices of fresh fruit or vegetable. To reduce waste and cost I ONLY add the staples (eggs, milk, and bread) when it is on the menu. If none of my breakfast, lunch, or dinner choices call for bread, I skip that aisle for this trip.

It took me only a few short weeks to notice that by cutting my food storage space in half, and adopting a disciplined approach to menu and list making, I had saved hundreds of dollars. I challenge you to cut down your food storage spaces and let your menu dictate your shopping list!

How to capture HTTPS SSL TLS packets with wireshark

2012-02-29T19:14:00-05:00

This article will explain how to use wireshark to capture TCP/IP packets. Specifically I will show how to capture encrypted (HTTPS) packets and attempt to document the "dance" a client and server do to build an SSL tunnel.

What is Wireshark?

Wireshark is a network protocol analyzer for Windows, OSX, and Linux. It lets you capture and interactively browse the traffic running on a computer network. Similar software includes tcpdump on Linux.

Install Wireshark

First step, acquire Wireshark for your operating system.

Ubuntu Linux: sudo apt-get install wireshark

Windows or Mac OSX: search for wireshark and download the binary.

How to capture packets

This is Wireshark's main menu:

To start a capture, click the following icon:

A new dialog box should have appeared. Click start on your preferred interface:

You are now capturing packets. The packet information is displayed in the table below the main menu:

Now browse to an HTTPS website with your browser. I went to https://linkpeek.com and after the page completely loaded, I stopped the Wireshark capture:

Depending on your network, you could have just captured MANY packets. To limit our view to only interesting packets you may apply a filter. Filter the captured packets by ssl and hit Apply:

Now we should be only looking at SSL packets.

Next we will analyze the SSL packets and answer a few questions

1. For each of the first 8 Ethernet frames, specify the source of the frame (client or server), determine the number of SSL records that are included in the frame, and list the SSL record types that are included in the frame. Draw a timing diagram between client and server, with one arrow for each SSL record.

Frame 1 client | 1 record | Arrival Time: Feb 15, 2012 15:38:55.601588000

Frame 2 server | 1 record | Arrival Time: Feb 15, 2012
15:38:55.688170000
Frame 3 server | 2 record | Arrival Time: Feb 15, 2012
15:38:55.688628000
Frame 4 client | 3 record | Arrival Time: Feb 15, 2012
15:38:55.697705000
frame 5 server | 2 record | Arrival Time: Feb 15, 2012
15:38:55.713139000
frame 6 client | 1 record | Arrival Time: Feb 15, 2012
15:38:55.713347000
frame 7 server | 0 record | Arrival Time: Feb 15, 2012
15:38:55.713753000
frame 8 server | 1 record | Arrival Time: Feb 15, 2012
15:38:55.715003000

2. Each of the SSL records begins with the same three fields (with possibly different values). One of these fields is “content type” and has length of one byte. List all three fields and their lengths.

Each hexadecimal digit (also called a "nibble") represents four binary digits (bits) so each pair of hexadecimal digits equals 1 byte.

a. Destination mac address | 6 btyes | 00 21 9b 31 99 51

b. Source mac address | 6 bytes | 00 10 db ff 20

c. Type: IP | 2 byte | 08 00

ClientHello Records

3.Expand the ClientHello record. (If your trace contains multiple ClientHello

records, expand the frame that contains the first one.) What is the value of the

content type?

hex: 16 (16+6=22) Handshake

4. Does the ClientHello record advertise the cipher suites it supports? If so, in the first listed suite, what are the public-key algorithm, the symmetric-key algorithm, and the hash algorithm?

MD5, SHA, RSA, DSS, DES, AES

ServertHello Records

5. Look to the ServerHello packet. What cipher suite does it choose?

Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)

6. Does this record include a nonce? If so, how long is it? What is the purpose of the

client and server nonces in SSL?

Yes, 28 bytes. The ClientHello packet also generated a nonces. They are used to make the session communication between the two nodes unique. It "salts" the communication to prevent replay attacks. A replay attack happens when data from old communications is used to "crack" a current communication.

7.Does this record include a session ID? What is the purpose of the session ID?

Yes, This is to make things efficient, in case the client has any plans of closing the current connection and reconnect in the near future.

8.How many frames does the SSL certificate take to send?

In this case it took 4 frames

Zenoss or Nagios monitoring of HTTPS using client certificate authentication

2012-02-26T19:21:00-05:00

I recently needed to monitor an HTTPS API for response time and availability. At first I planned to just use the Nagios check_http command.

After gathering more requirements I learned that the API was protected by client certificate authentication. After some research I quickly found that no solution existed to monitor HTTP protected by client certs. I needed to write my own plugin.

This is the python plugin I came up with: check_http_client_cert.py

#!/usr/bin/python

"""Nagios/Zenoss client cert https checker"""

import httplib
from optparse import OptionParser
from time import time
from sys import exit

def request( hostname, port, cert_file, path ):
    """request a resource and return response object"""
    try:
        c = httplib.HTTPSConnection( hostname, port, cert_file=cert_file )
        c.request( "GET", path )
        return c.getresponse()
    except:
        return False

if __name__ == '__main__':
    parser = OptionParser()
    parser.add_option('-H', '--hostname', dest='hostname')
    parser.add_option('-p', '--port', dest='port')
    parser.add_option('-c', '--cert_file', dest='cert_file')
    parser.add_option('-P', '--path', dest='path',
    help="Path relative to root, like /image/search")

    o, args = parser.parse_args()
    #print o

    start = time()
    r = request( o.hostname, o.port, o.cert_file, o.path )
    elapse = time() - start

    if r:
        if r.status >= 200 and r.status < 400:
            print "HTTP OK:", r.status, r.reason, "|time=" + str(elapse) + "s;;;"
            exit( 0 )
        print "HTTP Critical:", r.status, r.reason

    exit( 2 )

Jake, Finn, and Ice King

2012-02-18T13:34:00-05:00

Jake from Adventure Time, painted with MyPaint and wacom tablet.

I did this because my 4 month old "cracks-up" laughing whenever this character is on screen. I think the combination of colors and Jake's voice, played by John William DiMaggio, get him going.

Finn from Adventure Time.

Took a shot at painting Finn.

I'm on a roll! Here is Ice King.

Jake standing up, painted this this morning

Lumpy princess

I'm getting faster, this was completed in 20 minutes

My 4 month old's 15 minutes of fame

2012-02-16T18:42:00-05:00

My sister @VeronicaBal was watching my son (@RussellBal) tweeted a picture of him.

The official @RedSox re-tweeted the image to all 197,035 followers!

I'm a proud daddy!

Block cipher lab

2012-02-14T01:36:00-05:00

Consider the following block cipher. Suppose that each block
cipher T simply reverses the order of the eight input bits (so that,
for example 11110000 becomes 00001111).
Further suppose that the 64-bit scrambler does not modify any bits.
With n = 3 iterations and the original 64-bit input equal to 10100000
repeated eight times, what is the value of the output?
Now change the last bit of the original 64-bit input from 0 to a 1.
Now suppose that the 64-bit scrambler inverses the order of the 64
bits.
Solution in python:

def chunks( l, n ):
    """accept a list and chuck size, return chunks"""
    return [ l[ i:i+n ] for i in range( 0, len(l), n ) ]


def T( blocks ):
    """for each block, reverse block, return blocks"""
    result = []
    for block in blocks:
        result.append( ''.join( [bit for bit in reversed( block )] ) )

    return result

def scrambler( input ):
    """inverse the order of input"""
    return ''.join( [i for i in reversed( input ) ] )


def cipher1( input, n = 3, chunk_length = 8 ):
    """make chucks out of input, reverse each chunk return result"""
    blocks = chunks( input, chunk_length )
    for i in range( 0, n ): blocks = T( blocks )
    return ''.join( blocks )


def cipher2( input, n = 3, chunk_length = 8 ):
    """same as cipher1 but with scrambler"""
    blocks = chunks( input, chunk_length )
    for i in range( 0, n ):
        blocks = T( blocks )
        blocks = chunks( scrambler( ''.join( blocks ) ), chunk_length )
    return ''.join( blocks )


if __name__ == "__main__":

    input = "1010000010100000101000001010000010100000101000001010000010100000"
    print cipher1( input )
    # output: 0000010100000101000001010000010100000101000001010000010100000101

    input = "1010000010100000101000001010000010100000101000001010000010100001"
    print cipher1( input )
    # output: 0000010100000101000001010000010100000101000001010000010110000101

    input = "1010000010100000101000001010000010100000101000001010000010100000"
    print cipher2( input )
    # output: 1010000010100000101000001010000010100000101000001010000010100000

    input = "1010000010100000101000001010000010100000101000001010000010100001"
    print cipher2( input )
    # output: 1010000110100000101000001010000010100000101000001010000010100000

Monoalphabetic Cipher and Inverse Written in Python

2012-02-13T22:39:00-05:00

Contents

introduction and background
monoalphabetic_cipher.py
monoalphabetic_cipher.py example usage

introduction and background

A monoalphabetic cipher uses fixed substitution over the entire message.

You can build a monoalphabetic cipher using a Python dictionary, like so:

monoalpha_cipher = {
    'a': 'm',
    'b': 'n',
    'c': 'b',
    'd': 'v',
    'e': 'c',
    'f': 'x',
    'g': 'z',
    'h': 'a',
    'i': 's',
    'j': 'd',
    'k': 'f',
    'l': 'g',
    'm': 'h',
    'n': 'j',
    'o': 'k',
    'p': 'l',
    'q': 'p',
    'r': 'o',
    's': 'i',
    't': 'u',
    'u': 'y',
    'v': 't',
    'w': 'r',
    'x': 'e',
    'y': 'w',
    'z': 'q',
    ' ': ' ',
}

We can create an inverse of this cipher dictionary by switching the key and value places:

inverse_monoalpha_cipher = {}
for key, value in monoalpha_cipher.iteritems():
    inverse_monoalpha_cipher[value] = key

Now that we have both the cipher and the inverse_cipher, we may encrypt a message.

Encryption example:

message = "This is an easy problem"
encrypted_message = []
for letter in message:
    encrypted_message.append(monoalpha_cipher.get(letter, letter))

print(''.join(encrypted_message))

Result:: Tasi si mj cmiw lokngch

Using the inverse_cipher, We may decrypt a message.

Decryption example:

encrypted_message = "rmij'u uamu xyj?"
decrypted_message = []
for letter in encrypted_message:
     decrypted_message.append( inverse_monoalpha_cipher.get(letter, letter))

print(''.join( decrypted_message ))

Result:: wasn't that fun?

monoalphabetic_cipher.py

Here is a toy library I wrote to make the process repeatable -

monoalphabetic_cipher.py:

from string import letters, digits
from random import shuffle

def random_monoalpha_cipher(pool=None):
    """Generate a Monoalphabetic Cipher"""
    if pool is None:
        pool = letters + digits
    original_pool = list(pool)
    shuffled_pool = list(pool)
    shuffle(shuffled_pool)
    return dict(zip(original_pool, shuffled_pool))

def inverse_monoalpha_cipher(monoalpha_cipher):
    """Given a Monoalphabetic Cipher (dictionary) return the inverse."""
    inverse_monoalpha = {}
    for key, value in monoalpha_cipher.iteritems():
        inverse_monoalpha[value] = key
    return inverse_monoalpha

def encrypt_with_monoalpha(message, monoalpha_cipher):
    encrypted_message = []
    for letter in message:
        encrypted_message.append(monoalpha_cipher.get(letter, letter))
    return ''.join(encrypted_message)

def decrypt_with_monoalpha(encrypted_message, monoalpha_cipher):
    return encrypt_with_monoalpha(
               encrypted_message,
               inverse_monoalpha_cipher(monoalpha_cipher)
           )

monoalphabetic_cipher.py example usage

Here I show how to use the library:

>>> # load the module / library as 'mc'.
>>> import monoalphabetic_cipher as mc


>>> # generate a random cipher (only if needed).
>>> cipher = mc.random_monoalpha_cipher()

>>> # output the cipher (store for safe keeping).
>>> print(cipher)

>>> # encrypt a message with the cipher.
>>> mc.encrypt_with_monoalpha('Hello all you hackers out there!', cipher)
'sXGGt SGG Nt0 HSrLXFC t0U UHXFX!'

>>> # decrypt a message with the cipher.
>>> mc.decrypt_with_monoalpha('sXGGt SGG Nt0 HSrLXFC t0U UHXFX!', cipher)
'Hello all you hackers out there!'

Why does a Hash provide better message integrity then an Internet checksum?

2012-02-13T19:19:00-05:00

Why does a Hash provide better message integrity then an Internet checksum?

Hash function and checksum function both return a value which cannot be reversed.

An Internet checksum (TCP checksum or IP checksum) is designed to detect common errors quickly and efficiently. An Internet checksum does not attempt to prevent collisions.

Man cksum for more info.

A Hash provides better message integrity because it has less collisions then an Internet checksum. A collision means there is more then one way to produce the same sum. A great hash function aims to reduce the occurrence of collisions. Man md5 and sha for more info.

What is a collision

Let H() be a hash function. Let x and y be two differing messages.

H(x) = H(y) would be a collision.

I like to use python to show examples of hash functions

In this example I pass a message into the MD5 hash function to produce a resulting hash of the message. You can think of this hashed output as a finger print of the message.

import hashlib as h
message = "This message will be placed into an MD5 hash function to authenticate its integrity."
print h.md5(message).hexdigest()

Hash Output:

18f189f94b245ad8566206c199b4f60a

Now If I passed that message to you along with its MD5 hash hex representation, you could put the message into your own MD5 hash function and compare the resulting hash. This method is used to validate the message or verify data integrity.

Can you "decrypt" a hash of a message to get the original message

No! A hash may not be reversed, which means it cannot be decrypted.

By design a hash algorithm has no inverse, there is no way to get the original message from the hash. This is good news, turns out we have some really great applications for this type of function. We can validate messages, we can securely store passwords, and we can quickly determine if a message or file has been tampered with.

When using a publicly known hash function for storing password hashes, make sure to always use a salt or shared secret. Failure to do so will make your storage scheme susceptible to a rainbow table attack. A rainbow table allows a cracker to quickly match a list of hashes with a table of previously computed hash values and correlated passwords.

What is salt or a shared secret?

You can use salt or a shared secret to add extra data to a message before hashing with a publicly known algorithm. Below I will document how to properly add salt to a message before generating a SHA256 hash.

import hashlib as h
message = "This message and some salt will be hashed with SHA 256."
salt = "This is some secret salt data"
print h.sha256(message+salt).hexdigest()

Hash Output:

5e8d86bab9604620f19cfbc5f836f47feb9e8c9e74264fff1f4938bdaab1eeaa

Adding a salt to the message allows us to use a publicly know algorithm in a more protected manner.

Can you spot the error in the python code below?

import hashlib as h
message = "This message and some salt will be hashed with SHA 256."
salt = "This is some secret salt data"
print h.sha256(message).hexdigest()+salt

If you guessed that the message and salt BOTH need to be hashed together then you are correct!

The above code would have produced the following invalid hash:

Hash Output:

79cd4bfa1bcb71a7a1b5bfd5e8cfc8368a6cc6cb836d24bf04f2ef2bd0e81261This is some secret salt data

You should follow me on twitter: @russellbal

Symmetric Encryption vs Public Key Encryption

2012-02-13T17:25:00-05:00

How many keys are involved for symmetric key encryption? How about public key encryption?

Suppose you have N people who want to communicate with each other using symmetric keys. All communication between any two people, i and j, is visible to group N. Only person i and person j can decrypt each others messages.

How many keys would Symmetric Encryption require to protect group N?

I solved this with the following python function:

def count_symmetric_keys( N=2 ):
    """Provide the number of entities in group N.
    return the number of symmetric keys needed for this group"""
    keys = 0
    for i in range( 0, N ): keys += i
    return keys

A reader suggested the following optimized formula:

def calc_symmetric_keys( N=2 ):
    """Provide the number of entities in group N.
    return the number of symmetric keys needed for this group"""
    return N*(N-1)/2

If group N had 10 members, it would need to generate and maintain 45 Symmetric Keys.

If group N had 50 members, it would need to generate and maintain 1225 Symmetric Keys.

Symmetric keys are also susceptible to man-in-the-middle attacks. This attack occurs when an entity poses as a trusted entity. Let i and j be trusted entities. Let k be an untrusted attacker. If k determined the Symmetric key it could send or receive messages posing as i or j.

How many keys would Public-key Encryption require to protect group N?

Public Key Encryption requires 2n keys or two keys per person in group N. Public key encryption also does not require 'pre sharing' the secret key before communication may start. Each member would need 1 public key and 1 private key.

If group N had 10 members, it would need to generate and maintain 20 Public/Private Keys.

If group N had 50 members, it would need to generate and maintain 100 Public/Private Keys.

Attributes of an 8-block cipher

2012-02-13T00:50:00-05:00

Consider an 8-block cipher and answer the following:

How many possible input blocks does this cipher have?

How many possible mappings are there?

If we view each mapping as a key, then how many possible keys does this cipher have?

To find the input blocks of this cipher we raise 2 to the 8th power. 2^8 = 256 possible inputs.

To find the number of possible mappings we take the 256 input blocks and find it's factorial. There are 256! possible mappings.

We can view each of these mappings as a key, so this cipher has 256! keys.

Reasons why some Internet entities might want secure communication

2012-02-08T18:23:00-05:00

Internet entities often desire to communicate securely, some reasons include:

Web Servers: Communication on the Internet, or any network for that matter, should be encrypted before transmitting sensitive data. This will help prevent snooping from unauthorized parties. Most often SSL or HTTPS may be used to create a secure communication “tunnel” between a web server and a web client (browser).
Server Administration: Operators should always use a secure protocal when working on a server or remote computer. SSH (Secure Shell) wins the popularity contest with server admins.
DNS Servers: Using DNSSEC could help prevent DNS poisoning and certifies DNS data. DNS was first conceived as a distributed and highly scalable address lookup system. Security was not its top priority. Since then we have new DNSSEC extensions which allow for origin authentication of DNS data, authenticated denial of existence, and data integrity. DNSSEC does not attempt to solve availability and confidentiality.

What are the differences between message confidentiality and message integrity

2012-02-08T17:47:00-05:00

What are the differences between message confidentiality and message integrity? Can you have confidentiality without integrity? Can you have integrity without confidentiality?

message confidentiality: Two or more hosts communicate securely, typically using encryption. The communication cannot be monitored (sniffed) by untrusted hosts. The communication between trusted parties is confidential.
message integrity: The message transported has not been tampered with or altered. A message has integrity when the payload sent is the same as the payload received.

Sending a message confidentially does not guarantee data integrity. Even when two nodes have authenticated each other, the integrity of a message could be compromised during the transmission of a message.

Yes, you can have integrity of a message without confidentiality. One can take a hash or sum of the message on both sides to compare. Often we share downloadable files and provide data integrity using md5 hash sums.

Today I lost a customer

2012-01-18T23:30:00-05:00

Today I lost a customer.

I added some new code to LinkPeek to accept coupons and I didn't think of an edge case. This ended up creating an uncaught exception in my server side code which ultimatly served the newly subscribing customer an HTTP 500 error page.

The damage was done.

This error was catastrophic and ultimately killed the conversion. Here is an excerpt of how the user felt after the experience:

At this point the website has failed spectacularly enough that I can no longer trust you with my business. Please void the charge on my American Express card before it's processed. I need to hear from you ASAP.

Customers and prospects are forgiving for normal bugs in software. However, customers are intolerant to bugs in the sign up or payment flow. An error in payment flow will cause friction and friction will kill the sale.

Running a start up is hard work, and negative (but justified) feedback hurts more then I thought it would. This email made me feel awful.

Fortunately I learned from this mistake and the user's card was never charged.

Always make sure to extensively test payment and sign up code. Try all the edge cases. Try to make your software break. Don't inflict friction on your potential customers.

LinkPeek.com, webpage to image, was a by-product

2011-12-19T00:23:00-05:00

tldr; When faced with pivoting or killing a project, take a good look at all possible by-products. Don't miss the hidden gem in a project's slag!

Last year I built yoursitemakesmebarf.com, a novelty web application which allowed anonymous link submission. The software would automatically take screenshots of submitted links and curate a blog. I enjoyed building the site and the project served as my first Pyramid application.

The project's original intent was to jokingly poke fun at ugly design. The idea never caught on. Instead the application angered website owners and attracted undesirable people. Eventually, I decided to take it down and come up with less combative idea.

After witnessing the Goog release of "instant previews", I knew there was a market for a fast and reliable web screen shot service.

So I decided to bring instant previews to anyone who needed them. I wanted to build a fast, flexible, and easy to use screenshot API. After a few months I had a working prototype. I named the product LinkPeek because it described what the service was, and the domain was available.

Next I built a website thumbnail generator to show off the software. The generator application helped reinforce the simplicity of the underlying LinkPeek API. It didn't require any downloading, installation, or waiting.

About a week later on a whim I posted the generator to hacker news with an honest title (Convert Any Webpage to an Image) and in about 30 minutes it reached the number 1 spot.

Remember, when faced with pivoting or killing a project, take a good look at all possible by-products. Don't miss the hidden gem in a project's slag!

flash mob office meeting definition

2011-12-12T15:34:00-05:00

Flash Meeting

In a office or cubicle environment a group of uninvited people gather and hover around your desk to talk to you. A meeting forms in immaculate conception as you sit bewildered at your desk.

Other names: Flash Meeting, Flash Mob Meeting, Flash Office Meeting

LinkPeek.com Number One on Hacker News

2011-12-05T00:39:00-05:00

We made the number one spot on Hacker News.

LinkPeek was used to take a snapshot of the event:

How to Incorporate Custom Configuration in a Pyramid Application

2011-11-30T22:47:00-05:00

Note: This post shows an old way to modify the request object. I discuss a new way in my Pyramid add_request_method post.

Imagine that you have just built a wiki, blog, or cms web application that will be deployed multiple times by different people. You would like to provide the ability to configure some aspects of the program without the end user altering your python or template code. This guide explains how to incorporate custom configuration from the project's .ini file with the rest of the application.

To explain this process we will add a customizable Google Analytics key to our project.

Make a configuration key/value pair for Google Analytics
Add the Google Analytics javascript code to the template

Make a configuration key/value pair for Google Analytics

Inside production.ini place the following in the [app:main] section:

#google_analytics_key = UA-55555555-1
google_analytics_key =

Add the Google Analytics javascript code to the template

In this case I will show an example in mako. Other template solutions should look similar.

<%def name="google\_analytics()">
% if request.registry.settings['google_analytics_key']:


 <script type="text/javascript"></p>
 <p>var _gaq = _gaq || [];<br />
           _gaq.push(['_setAccount', "${request.registry.settings['google_analytics_key']}"]);<br />
           _gaq.push(['_trackPageview']);</p>
 <p>(function() {<br />
           var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;<br />
           ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'https://www') + '.google-analytics.com/ga.js';<br />
           var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);<br />
           })();</p>
 <p></script>

% endif

Then in the head section call the function:

${ google_analytics() }

That was easy because the configuration string just needed to be substituted into the JavaScript in the template. What if you needed to do something with the provided key/value before using it? Next I will show you a method for building renderer globals again showing a different way to configure Google Analytics:

Make an inject_renderer_globals subscriber

Define inject_renderer_globals(event) function in the project's __init__.py file.

I normally place it at the bottom of the file and it looks like this:

def inject_renderer_globals(event):
    """Inject some renderer globals before passing to template"""

    request = event['request']

    # Build ${google_analytics_key} from the configuration file
    event['google_analytics_key'] = request.registry.settings[ 'google_analytics_key' ]

Import BeforeRender at the top of the __init__.py:

from pyramid.events import BeforeRender

In the main function, add the inject_renderer_globals to the subscribers:

config.add_subscriber(inject_renderer_globals, BeforeRender)

Now you can use ${google_analytics_key} anywhere in your template.

Thank you for reading, and feel free to leave comments

Career development is a game of chutes and ladders

2011-11-29T20:42:00-05:00

If career development was a game of chutes and ladders, job networking would be the ladder. Ladders provide a shortcut to the top, a direct route to win, in this case you win your dream job.

At work today, a colleague was reviewing resumes for an open requisition within the Unix group. Later that night I decided to clean up my resume to make it more relevant.

I felt like I did something positive for my career. After coming down from the high of resume writing I began to question my rational and came to the contradictory conclusion that a great resume holds less importance than a mediocre recommendation.

It is better for an employer to learn about you from a recommendation then your resume.

Why?

Hiring new people holds risk. A hiring manager will reduce risk by promoting from within or using recommendations. In both cases the resume becomes a document of formality instead of a document of credentials.

"Its not what you know, its who you know."

What can we assume about the position they are extending to the public?

Can we assume that the manager has already promoted somebody internally for the high risk, enjoyable position? Depending on the industry, they might be looking for bottom feeder (with a resume) to fill the newly vacant position. Keep this in mind the next time a head hunter sends you a proposition.

Taking shortcuts normally goes against the conventional wisdom for success.

Job networking however, allows candidates to skip ahead and arrive at their dream occupation more directly. Meeting new people seems like the single best way to land a job doing what you love. Get out there and meet people with similar interests!

Did you like this post?

You should checkout the first chapter of my e-book, How-to Work From Home titled The Road to Remote

I'm petrified of launching my web application

2011-11-03T21:35:00-04:00

I'm petrified of launching my web application because I'm fearful that I won't ...

acquire users
support my users well
scale in a timely manner
react quickly to feedback
monetize the application

But most of all I'm scared that nobody will like me. I'm scared of failure.

Now that I got that out of my system please check out https://linkpeek.com

Webmaster tools alerted issue turned out Pylon session files flooded inodes

2011-10-29T15:30:00-04:00

This graph could happen to you if you ever forget to configure munin email alerting:

It only took approximately 1 hour to diagnoses and resolve this issue however most of my web applications hosted on this server were down for about 11 hours. I was lucky that this outage fell on a weekend otherwise I would not have known about the problem till around 6:30pm!

Diagnosis:

Two of my pylons apps had session files that slowly ran away on me. The session files don't consume much capacity however the shear quantity of them caused my inode usage to hit 100%.

Had I properly configured Munin's email alerting this issue would have been identified well before it was a problem.

Want to know what alerted me to the problem? G Webmaster's tools claimed it could not read my robots.txt on a few of my sites... After investigating I learned the site was down. Checking the Apache error logs pointed me to disk space issues. df -ha reported everything was fine, however df -hi reported 100% inode usage! At this point I started looking to cache and log locations to find lots of files, which lead me to my pylons web applications data/sessions directories.

Resolution:

Delete the session cache tree directories and allow the applications to rebuild them.

Todo: move /www off the root disk partition. This issue could have been much worse if I was unable to boot or login to remedy. Moving /www off root should prevent the web server from effecting the systems ability to boot.

Occupy Wall Street Stack vs Queue

2011-10-21T13:31:00-04:00

Occupy Wall Street contributors claim to use a "stack" to determine speaking arrangements.

I plan to explain how the term "stack" used in this scenario does not align itself with the mathematical or computer science definition.

The term stack means First In Last Out or "FILO". For example: a person placed on the stack in the morning would be the last to speak at the end of the day. This isn't happening like that...

Occupy Wall Street compatriots really use a technique called "queue". Mathematicians and Computer Scientists define a queue as First In First Out or "FIFO". A real world example of a queue would be a line at the grocery store. The first person in line is the first person to leave the line.

An object enters a stack from the rear and exits the rear.
An object enters a queue from the rear and exits from the front.

Adding inline image support to a gmail messages

2011-10-20T17:46:00-04:00

Enable ability to insert images into a message body. You can upload and insert image files in your computer, or insert images by URLs. This lab will not work if you have offline enabled.

Go to google labs: https://mail.google.com/mail/#settings/labs
Search "images".
Enable "Inserting images - by Kent T"

Welcome!

Cell shading practice, A sketched skull, and an old wind mill photograph

2011-10-16T12:56:00-04:00

Some of my resent works using my Ubuntu + Wacom Bamboo + MyPaint + Gimp Workflow!

First some cell shading practice. Inside MyPaint I only used two colors in my palate but to gain shadow and depth I used variations of opacity. This technique produces a neat cartoon effect.

Next I sketched a skull. I attempted to keep all my strokes in the same direction. This piece is monotone. I used only one color and deviated each layer with different opacity settings.

Last is a photograph I snapped during a family outing. I had to lower the quality of this image to prepare for use on the web, but I still adore the colors spectrum of this shot. The white birch tree frames the right side while the grass and brush frame the bottom. The wind mill was intentionally placed in the center with the apex extending upward. The ivy creeps up the structure and the cloud filled sky appears to stretch forever. The foliage emits a euphoric glow under the sun saturated rays.

Come back soon!

I cancelled my xbox live automatic renewal

2011-10-15T19:26:00-04:00

I cancelled my xbox live automatic renewal today because I no longer use the service.

I find humor in Microsoft's list of reasons to keep Xbox live:

6 out of 8 services above are FREE Internet services!

If I want to use those free Internet services on my TV, I'd rather use my tiny media center PC (nT330i) which is amazing and near silent.

Microsoft prevents subscribers from cancelling their service over the web, and thus they forced me to call their service center. After waiting 5 minutes I finally spoke to a sales representative who seemed friendly and understanding.

She asked me why I wanted to cancel automatic payments. I explained "I want to pay month to month". After listening to my position the sales representative offered 1 month of Xbox Live for $1.00. So I'll have Xbox live until December 2011 after all.

Microsoft, give your users a web browser on the Xbox and stop trying to sell subscriptions to Free Internet services.

*PlayStation 3 launched with a web browser and a FREE network.*

UPDATE: If you opt into the $1 for one month deal, you are also opting back into the automatic renewal program... Even if the person on the phone tells you otherwise.

Call us directly by dialing:

Toll free:
(800) 4MY-XBOX or (800) 469-9269
Hearing impaired (TDD device):
(866) 740-9269 or (425) 635-7102
9 am to 1 am Eastern Time
6 am to 10 pm Pacific Time

r8168 driver issues after Ubuntu 11.10 upgrade kernel linux 3.0

2011-10-15T11:15:00-04:00

I had network issues after upgrading to Ubuntu 11.10 which has the linux 3.0 kernel.

I used this guide to compile the r8168 driver however, I needed to alter the Makefile to support for linux 3.0 kernel.

Edit src/Makefile:

#KEXT  := $(shell echo $(KVER) | sed -ne 's/^2\.[567]\..*/k/p')o
KEXT := $(shell echo $(KVER) | sed -ne 's/^[23]\.[1-9]\..*/k/p')o-

Now you should follow the rest of the guide.

A better way to show website backlinks

2011-10-12T20:05:00-04:00

Early web pilgrims of the Internet fashioned search queries like

link:russell.ballestrini.net to gather backlinks for a domain.

I however advocate a revolutionary search pattern like -

"russell.ballestrini.net" -site:russell.ballestrini.net

to gather an improved representation of backlinks.

Click here to try now!

CSS frameworks not rendering properly on all browsers

2011-10-11T00:07:00-04:00

I ran into an issue when testing skeleton css framework and twitter bootstrap where some browsers were not rendering the pages properly. After some research I determined that I forgot the DOCTYPE declaration tag.

The DOCTYPE tag tells the browser what "rules" or standards to use when rendering markup.

The following code block displays the minimal approach to setting the document type:

<!DOCTYPE html>

This doctype tag forces the broswer into standards mode which allows the pages to render properly.

Thanks!

LinkPeek.com web address thumbnail api alpha release

2011-10-05T20:58:00-04:00

The LinkPeek API had an alpha release!

LinkPeek is a website screenshot service. Convert any webpage to an image. No software, no downloading and absolutely no waiting! Unlimited free 140 pixel thumbnails!

How do I calculate the M in my MVP?

2011-10-01T15:32:00-04:00

MVP (Minimal Viable Product):: The smallest version of your idea that produces value for customers.

Recently I have contemplated the idea of building a screen capture service for websites. I anticipate the service would provide functionality similar to Google's "Instant Preview". I also pondered the idea of archiving the screen captures and providing a history similar to the internet Wayback Machine.

After floundering on my first web application, https://four2go.gumyum.com, I'm afraid to spend lots of time and energy on another project if I cannot earn users or customers.

I feel the market segment for this type of service would include website directories, live portfolio generation for website designers, and websites in general for people who wish to keep track of the overall look of their site over time.

As an MVP I plan to build a free live web capture application to basically provide a "Gravatar" for website addresses.

Do you think anyone would use my MVP or service?

LinkPeek provides an easy to use API which enables anyone to instantly create live web page thumbnails. We provide the API free of charge, think of LinkPeek as Gravatar for web addresses.

Have I missed any other market segments?

New Baby Homecoming

2011-09-27T16:53:00-04:00

Hacker Olive Oil Lamp Crafted From Home Materials

2011-09-09T19:01:00-04:00

In loom of the recent and persisting hurricane season I present a "Hacker's Olive Oil Lamp"!

YES, this lamp is ...

Fun and simple to build
Fun and simple to use
Easy to Cleanup
Gentle on environment
Energy efficient (1 cup ~= 16 hours)
Safe (flame smothers if lamp tips over)

Odor
Smoke
Heat near base
Cost (household materials)
Waste (Store and seal oil in lamp with jar top)

Materials

1x Pickle jar with lid
1x Washcloth 100% cotton
1x Steel wire (salvaged from paint can handle)
2x Cups of olive oil

Tools

Needle nose plyers
Scissors

[gallery link="file"]

A system administrators guide to installing and maintaining multiple python environments

2011-09-08T19:28:00-04:00

Contents

Install Python 2.7
- Compile from source
- Install package from scl
virtualenv

Some operating systems depend on a specific version of python to function properly. For example, Yum on Redhat Enterprise Linux 5 (RHEL5) depends on python 2.4.3. This version of python lacks support from many utilities and 3rd party libraries. This guide will cover installing an alternative python instance while leaving the system's python alone.

This guide supports the following operating systems: Redhat, CentOS, and Fedora. As of this publication the latest Python version was 2.7.8; You might want to determine if a newer version exists.

Install Python 2.7

I found two methods which work for installing Python 2.7 side-by-side with the system's Python.

compile from source
install package via the scl repo

Choose which ever strategy you find easier.

Compile from source

Gather the dependencies:
```
yum install gcc zlib-devel python-setuptools readline-devel
```
- gcc is a compiler used to build python
- zlib-devel allows the python zlib module to be built
- python-setuptools provides the easy_install application
- readline-devel arrows readline and history handling in python shell

Download and untar the python sourcecode:

wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
tar -xzvf Python-2.7.8.tgz
cd Python-2.7.8

Compile the sourcecode:
```
./configure
make altinstall
```
Test new alternative python:
```
python2.7 --version
```
Document path to new python2.7, you will need it if you plan to use a virtualenv:
```
which python2.7
```
Now we can install third party libraries into our alternative python:
```
python2.7 -m easy_install
```

Install package from scl

install centos-release-scl repolists:
```
sudo yum install centos-release-scl
```
install python27:
```
sudo yum install python27
```
enable python27 via scl:
```
scl enable python27 bash
```
Document path to new python2.7, you will need it if you plan to use a virtualenv:
```
which python2.7
```

virtualenv

Optionally we can create a virtualenv (for development) based on the python 2.7 install. Virtual environments appear useful for testing packages and libraries without installing them to the system owned python site-packages directory.

Install pip using easy_install:
```
sudo easy_install pip
```
Install virtualenv using pip:
```
sudo pip install virtualenv
```
Create a new virtual python environment named virtpy:
```
cd ~
virtualenv -p /usr/local/bin/python2.7 virtpy
```
This will create a virtual python 2.7.2 environment named virtpy in your present working directory.

To invoke this environment run source virtpy/bin/activate and your prompt should change to reflect the active virtualenv.

Now you can run easy_install to install packages into virtpy/lib/python2.7/site-packages.

Thanks for reading, that's all for now.

Deliberating the Viewers vs. Doers concept

2011-09-06T10:20:00-04:00

Brett & Kate McKay from artofmanliness.com recently subscribed to an idea that America has succumbed to "spectatoritis, a blanket description to cover all kinds of passive amusement, an entering into the handiest activity merely to escape boredom." -Jay B. Nash, 1938

The basic concept discussed how people can fall into one of two groups when interacting in life, the viewers and the doers. Viewers or spectators watch an activity while doers perform the said activity. For example:

Viewers

Sports Fans
Audience
Couch Potatoes

Doers

Athletes
Musicians
Actors

Although the above patterns persists in the physical world, intellectual situations operate differently. For instance on the surface reading seems like a viewer or spectator task and writing appears to behave like a doer task. But what happens when a reader learns or responds to a blog post? We seem to need a stronger definition what constitutes doing a task.

We should not regard spectating as bad, however we should try to stay lucid of our current rolls when engaging in activities.

Spectating has some important purposes in our society and culture. Viewing a performance grants power to the actor, whether that person plays football or plays guitar. For this power and influence to occur a viewer must witness and pay attention to the event. What if the next great political leader took the podium and no one bothered to view his speech?

The truth is our American society has conditioned us to spend much of our lives spectating. During school a good student will view and listen to the lecture. While driving to work one radio DJ broadcasts his thoughts to many.

Spectating also plays and integral roll in learning. Human infants learn by first watching and then imitating. Experts also agree that inspiration for a creative works often occur after observation of prior productions.

tl;dr We should not battle the viewers vs the doers because both hold importance and need each other.

If you liked this article, you should follow me here

Python Image Grabber pig.py

2011-08-22T18:48:00-04:00

Python Image Grabber pig.py

Python Image Grabber or pig.py is a very simple python command line tool to download all the images from a given uri.

Enjoy!

Download and Installation:

https://bitbucket.org/russellballestrini/pig/raw/tip/pig.py

Usage:: python pig.py

Example:: python pig.py https://www.foxhop.net

Open Source:: pig.py has been placed in the public domain and its sourcecode may be viewed or branched here here: https://bitbucket.org/russellballestrini/pig

Pigpy Gimp Project: pigpy.xcf

Security Professionals: Yes we appear vulnerable but that attack vector will never happen

2011-08-18T21:36:00-04:00

In loom of recent internet attacks many institutions have started scrambling in attempt to "strengthen" their security stance. I agree that auditing our systems and networks for potential flaws seems appropriate at this time to prevent getting "caught with our pants down". Incidentally, I have recently witnessed the introduction of silly and at times ineffective security adjustments. Many of these new procedures, rules, and requirements do not make us more secure and worse instill a false sense of security.

I have previously addressed the fallacy of absolute security. No system is perfect. A successful security model accomplishes fortitude by implementing layers like an onion. Through the use of security layers we can significantly hamper attack vectors and create a safer complex.

When analyzing a potential attack vector we must first determine our current location in the security layers. This step serves two purposes:

to prevent wasting time and energy on vulnerabilities that don't matter at that point in our matrix.
to prevent causing outages and unneeded administrator and customer heartache.

If a vulnerability requires root or elevated privileges to occur, don't waste your time resolving it. If the attacker already has root, you have bigger problems on your hands.

Some real life examples:

Firewall denying a large range of IP addresses (like entire countries). This truly does not increase security, it just creates headaches for users. An attacker could just proxy to an open range (like a VPS based in a more trusted zone) and gain access from there. Also if you decide to ignore this advice and create blanket IP range deny rules, DON'T also block services intended to be internet-facing. For example, don't block your Internet-facing DNS server if it is authoritative for public domains. This will cause countless intermittent issues and will be a nightmare to diagnose.
Weekly Scanning for Windows viruses on network shares or data at rest. This hammers the servers for no reason. If all the desktops run antivirus then the file was already scanned when it was downloaded. That same file will be scanned again when retrieved on the share. If you want the warm and fuzzys of virus scanning network file shares, do it once a year. These scans waste time and resources. I feel even more outraged when asked to virus scan network shares hosted on UNIX servers or NAS.

I speculate that most of these arbitrary ideas come about because the people in charge make uninformed decisions out of fear without first consulting the appropriate subject matter experts.

Unfortunately, once a security mandate occurs it seems difficult to expunge. People are just not willing to put their neck on the chopping block to banish a legacy or silly mandates; So we end up living with nonsensical rules and procedures.

virt-back: restoring from backups

2011-08-10T23:20:00-04:00

In a perfect world we should create backups but never need them. Although this statement holds truth, creating guest backups provides many more benefits.

The most common reasons system administrators restore from a virt-back guest backup:

recovering from data corruption
recovering deleted files
recovering from a virus infection
recovering from a compromised server
backing out a failed change
rolling back to a previous state
testing disaster recovery plans
cloning a server
building test environments

During this article we will cover how to restore a system from a virt-back guest backup. This article will not cover how to restore a VM host server.

Virt-back guest restore procedure

In this guide our guest mbison has failed with a major corruption and we would like to restore from our backups. We have our running production guest images in /KVMROOT and our virt-back guest backups in /KVMBACK. We will be restoring the backup on the same hypervisor.

Overview:

Ensure the guest is shut off.
move the bad image file out of the way
untar the virt-back backup into place
power up the guest

Detailed Procedure:

Verify the guest is shut off by running:
```
virt-back --info-all
```
We noticed that mbison was still running so we invoked:
```
virt-back --shutdown mbison
```

Move the corrupted image file out of the way:

mv /KVMROOT/mbison.img /KVMROOT/mbison.img.NFG

Unzip and unarchive the backup using the following command:

sudo tar -xzvf /KVMBACK/mbison.tar.gz -C /KVMROOT --strip 1

When the untar completes, start the guest:
```
virt-back --create mbison
```
Connect to the guest over SSH and verify that all required services and applications start. Determine if the restore was successful.

Restore guest backup on new hypervisor:

The details in this section were adapted from a tutorial given by Fabian Rodriguez.

Re-create any bridge network interfaces on new hypervisor (/etc/network/interfaces for Debian)
Adjust mbison.xml if needed (for example if you are changing paths)

sudo mkdir /KVMROOT
sudo tar -xvzf mbison.tar.gz -C /KVMROOT --strip 1
virsh create /KVMROOT/mbison.xml

Note: We use virsh create instead of virt-back create. While both commands start guest DOMs, virsh create will also register the DOM into the hypervisor.

Programming is like Alchemy

2011-08-08T21:52:00-04:00

Programming is like Alchemy except instead of exchanging matter, we programmers exchange time. Also depending on the program the exchange of time worked (coding) increases the productivity (time) of its users.

On second thought, perhaps programmers correlate less with Alchemists and more with Time Travelers; Or at the very least time manipulators. For example, on a good day a programmer can easily complete a task that would take one thousand men. See, time created! On a bad week we can procrastinate and do nothing at all. Time lost!

Programming embodies other magic like wizardry. For example, our programs typically live as golems performing one task, repeatedly, over and over. Golem programs, without a soul, stuck in a loop of servitude.

Recently we have started coding creations with artificial intelligence. These smart programs act like familiar spirits (Wikipedia) and assist their creator in conjuring even more magic.

So we settled it! Programmers are like bad ass, time travelling, wizard alchemists!

Or maybe not ...

It is more accurate to group programs with technology then magic, but less fun. Programs are leveraged tools born to save people time and energy.

Thanks for reading, you should follow me on twitter here

The Great Gist Heist

2011-08-07T02:30:00-04:00

The Great Gist Heist

I have crawled, downloaded, and archived all of gist.github.com. Please hear my story before jumping to conclusions.

Why?

I'm currently building software that requires a large corpus of source code.

I began to search for a collection of source code documents but my pursuit appeared fruitless. Feeling displeased I attempted to gather all of my own source code. My collection lacked fidelity perhaps because of my revere for the python language.

Regardless of the reasoning, I needed a higher quantity of samples. I needed unbiased samples from all programming languages. I needed, most importantly, samples in a variation of quality that only the most popular paste sites have... sites like gist.

Why are you sharing it?

I feel a little bad about using Github's bandwidth.

Sharing this collection should reduce the chances that others will crawl for the same data. If you need a large collection of source code, download this torrent.

How did you do it?

I wrote a short, 30 line, python script. The script is part of the torrent.

At the peak of the scrape I had 14 threads running of the script, using approximately 580Kbps (I used iftop).

a hack to gain 80 percent efficiency when creating github projects

2011-07-11T21:47:00-04:00

Last night I decided to learn how to use git for its popularity and github to code more socially.

I have to admit early on that I enjoy hg and bitbucket so it came to a surprise that github would have me jump through hoops to create a new repository...

Below I have copied the seemingly bloated github instructions for creating a new project.

Github:

create project/repo using the form
mkdir scratch
cd scratch
git init
touch README
git add README
git commit -m 'first commit'
git remote add origin
git@github.com:russellballestrini/scratch.git
git push -u origin master

Wow...

I like my method better.

My Github clone hack:

create project/repo using the form
git clone https://russellballestrini@github.com/russellballestrini/scratch.git

Awesome, 2 steps versus 10. That hack appears 80% more efficient! Now if only my name was shorter... LOL

You should follow me on twitter here

Add a Breadcrumb Subscriber to a Pyramid project using 4 simple steps

2011-07-02T17:25:00-04:00

Note: This post shows an old way to modify the request object. I discuss a new way in my Pyramid add_request_method post.

This article will explain how to add a breadcrumb subscriber to a Pyramid project using 4 simple steps.

While programming https://school.yohdah.com/ I needed the ability to easily create breadcrumb links from the current url. You may view the bread.py source code here. The following guide describes the process I took to add this functionality.

1. Download and include bread.py at the top of your Pyramid project's __init__.py file:

from yourproject.bread import Bread
from pyramid.events import subscriber, NewRequest

2. At the bottom of the projects __init__.py file create the following subscriber function as follows:

def bread_subscriber( event ):
    """ Build Bread object and add to request """
    event.request.bread = Bread( event.request.url )

3. Now we can add our new subscriber to our Pyramid config inside main and above the routes:

config.add_subscriber(bread_subscriber, NewRequest)

4. You are done! Now you should have the ability to use the bread object from your template. Below I have provided a mako template snippet:

% for link in request.bread.links:
  ${ link | n } /
% endfor

Google Bot Attempts to Crawl Shortest Urls First

2011-06-25T09:37:00-04:00

Recently I built https://school.yohdah.com a Python, Pyramid, and mongoDB project during the last couple weekends.

The site features a directory style navigation of nearly every public school in the US. We have 61 state pages, approximately 19,000 city pages, and over 103,000 school pages.

It seems the Google Bots have noticed school.yohdah.com and started crawling the site. Since the initial crawl I started reviewing a sample of the sites apache logs in an attempt to track the bot's activity. After a few minutes of viewing the logs, I locked onto a pattern; Google Bot's algorithm appears to crawl the short URLs first!

PersonalCompute (a user) attached a graph of the fetched URL lengths here:

I have attached a zip containing the apache google bot crawl logs here: access-school.yohdah.log.zip

I found the pattern by opening the file in vim and scrolling very quickly down. You will notice the log lines will grow slowly to the right, as the urls being fetched increase by one character.

Why does Google do this? Does anyone have speculation as to what this means?

ATT Uverse Residential Gateway Broadband LED flashes red intermittently

2011-06-18T10:08:00-04:00

ATT Uverse Residential Gateway Broadband LED flashes red intermittently

I have struggled with my uverse Internet and TV service randomly and intermittently shutting off for the last 3 months. When this occurs the Residential Gateway's broadband LED flashes red.

Also I get Uverse DSL link errors:

When I 'live support chatted' with the uverse technical support they claimed I had bridge tap on my lines, and that my lines had other 'issues'.

They had to dispatch 3 technicians before one of them figured out the problem. It turns out I had a faulty 'voice/data' splitter.

I snapped a picture of the demarcation box before and after he replaced the faulty splitter.

Old Uverse filter on the left, New filter on the right

So far I have had no Uverse DSL Link Errors!

Connecticut killed affliate marketing with Amazon.com

2011-06-11T14:10:00-04:00

Connecticut killed affliate marketing with Amazon.com

I just opened my amazon account and read this:

Account Closed

This account is closed and will not generate referrals. Access to this site is for historical purposes only.

When I checked my email, I had this letter from Amazon.com:

Hello,

For well over a decade, the Amazon Associates Program has worked with thousands of Connecticut residents. Unfortunately, the budget signed by Governor Malloy contains a sales tax provision that compels us to terminate this program for Connecticut-based participants effective immediately. It specifically imposes the collection of taxes from consumers on sales by online retailers - including but not limited to those referred by Connecticut-based affiliates like you - even if those retailers have no physical presence in the state.

We opposed this new tax law because it is unconstitutional and counterproductive. It was supported by big-box retailers, most of which are based outside Connecticut, that seek to harm the affiliate advertising programs of their competitors. Similar legislation in other states has led to job and income losses, and little, if any, new tax revenue. We deeply regret that we must take this action.

As a result of the new law, contracts with all Connecticut residents participating in the Amazon Associates Program will be terminated today, June 10, 2011. Those Connecticut residents will no longer receive advertising fees for sales referred to Amazon.com , Endless.com , MYHABIT.COM or SmallParts.com . Please be assured that all qualifying advertising fees earned on or before today, June 10, 2011, will be processed and paid in full in accordance with the regular payment schedule.

You are receiving this email because our records indicate that you are a resident of Connecticut. If you are not currently a resident of Connecticut, or if you are relocating to another state in the near future, you can manage the details of your Associates account here . And if you relocate to another state after June 10, 2011, please contact us for reinstatement into the Amazon Associates Program.

To avoid confusion, we would like to clarify that this development will only impact our ability to offer the Associates Program to Connecticut residents and will not affect their ability to purchase from www.amazon.com .

We have enjoyed working with you and other Connecticut-based participants in the Amazon Associates Program and, if this situation is rectified, would very much welcome the opportunity to re-open our Associates Program to Connecticut residents.

Regards,

The Amazon Associates Team

Summary

Amazon.com does not have a facility in CT.
CT passed a new bill.
The bill declares: if a customer from any state clicks on a CT resident's affiliate link and purchases an item, that customer/Amazon.com should have to pay CT tax.

My Opinion

This seems bad.
I have less money to spend in Connecticut.
This smells like double taxation.
One more reason to move out of Connecticut when looking for Tech or IT jobs.

Dropbox Encryption with TrueCrypt

2011-04-16T12:50:00-04:00

Derek Newton recently invoked discussion about insecurities in Dropbox authentication. In his article he describes how an attacker could exploit Dropbox and gain access to unshared files. The concerns he raised do appear accurate however we must remember that security is an onion.

An onion, like security, has layers to protect its vital parts. The vital parts are more vulnerable when its security model only possess one layer. As we add layers to our security model, our system's protection grows exponentially.

In the case of Dropbox, the username and password act as the first layer. Experts agree that a simple authentication layer will provide enough protection for nonsensitive data. However when attempting to protect sensitive data we must pair authorization with encryption.

Generally speaking file systems have maintained a sense of insecurity, which makes them useful. Not encrypting files on Dropbox is akin to not encrypting files on a shared PC. Sensitive data should always be encrypted regardless of its location or media. We should treat sensitive data-at-rest on Dropbox the same way we treat sensitive data on local, optical or flash disk. We should encrypt it!

So how does a user encrypt their Dropbox?

My strongly opinionated solution uses TrueCrypt to create an encrypted volume in the Dropbox directory. Simply treat the Dropbox like a normal directory, follow the TrueCrypt documentation to build a volume, and give Dropbox a chance to sync the data. When the sync completes, the TrueCrypt volume will be mountable on each of your Dropbox enabled computers.

I have to admit at first I was skeptical, but the software cooperates surprisingly well and after the initial sync proceeding syncs occur quickly! I prefer TrueCrypt because it is open source, cross platform, and free (both in freedom and cost). TrueCrypt also functions and performs better then any other solution including commercial products like GuardianEdge or PGP both recently acquired by Symantec.

All security and encryption software should remain open sourced and peer reviewed to prevent harmful tampering. Commercial software, written in a black-box vacuum, prevents customers from viewing its code and procedures. We cannot trust software for security when we cannot view its source code.

You should follow me on twitter here

Voice Over IP with TeamSpeak

2011-04-03T12:11:00-04:00

This article will cover running a Voice Over IP service like TeamSpeak on a VPS.

Voice Over IP allows users to communicate using audio over the Internet.

When planning for this article I originally was going to cover ventrilo, but their download link was obfuscated behind a heinous php session script. Ventrilo also does not have a Linux client although they have been promising one for quite some time.

Instead we will cover how to install and configure TeamSpeak.

Installing a teamspeak server on your VPS

Create a user to run teamspeak.

sudo adduser teamspeak

follow the prompts

Personally I don't want the teamspeak user to have ssh access so I added the following to /etc/ssh/sshd_config:

DenyUsers teamspeak

Then reload ssh server config:

sudo service ssh reload

Setup the installation environment.

The following 4 commands will create a directory to hold your installation, change the ownership of the directory to the teamspeak user, change the working directory to the new folder and then become the teamspeak user:

sudo mkdir /opt/teamspeak
sudo chown teamspeak:teamspeak /opt/teamspeak
cd /opt/teamspeak
sudo su teamspeak

Download and extract TeamSpeak server software.

Find the proper package for your VPS and download it, in my case I ran:

wget https://teamspeak.gameserver.gamed.de/ts3/releases/beta-30/teamspeak3-server_linux-amd64-3.0.0-beta30.tar.gz

For best results download the latest version of teamspeak.

A teamspeak tarball should now exist in your present working directory. We can extract the files from the tarball by issuing the following command:

tar xvf teamspeak3-server_linux-amd64-3.0.0-beta30.tar.gz --strip-components=1

No you don't have to type that file name out! The bash shell has tab completion, type 'tar xvf teamsp' and then press tab. : )

Install the TeamSpeak server software.

I had success running:

./ts3server_startscript.sh start

Write down the auto generated serveradmin password.

Configure TeamSpeak to start at system bootup.

Create a cronjob under the teamspeak user:

crontab -e

Place the following into teamspeak's crontab:

@reboot /opt/teamspeak/ts3server_startscript.sh start

Porting the ChaosTheory Wordpress theme to Pylowiki

2011-03-26T22:00:00-04:00

As you may have noticed, I recently switched this blog over to the ChaosTheory Wordpress theme.

I had to tweak some of my pre-existing articles to make them look 'right' but upon completion I was very pleased with the aesthetics of each page.

Later on I reviewed the theme of foxhop.net, my pylowiki demo site. Pylowiki felt bland in comparison to this wordpress blog.

So, I made up my mind to build themes for Pylowik, and I started by porting ChaosTheory. You can view the finished product at https://www.foxhop.net.

You should also give Pylowiki a try. Pylowiki has futuristic, live preview, same page, section edit functionality.

virt-back: a python libvirt backup utility for kvm xen virtualbox

2011-03-20T11:32:00-04:00

Contents

Installation
Test installation
Example cronjob
Manual
Restoring

Over the weekend I wrote virt-back, a backup utility for QEMU, KVM, XEN, or Virtualbox guests.

virt-back is a python application that uses the libvirt API to safely shutdown, gzip, and restart guests.

The backup process logs to syslog for auditing and virt-back works great with cron for scheduling outages. Virt-back is in active development so feel free to give suggestions or branch the source.

virt-back has been placed in the public domain and the latest version may be downloaded here:

https://bitbucket.org/russellballestrini/virt-back

Installation

The fastest way to install virt-back is to use pip or setuptools.

Try sudo pip install virt-back or sudo easy_install virt-back

Otherwise you may manually install virt-back

sudo wget https://bitbucket.org/russellballestrini/virt-back/raw/tip/virt-back \
-O  /usr/local/bin/virt-back

sudo chmod 755 /usr/local/bin/virt-back

Test installation

virt-back --help

Example cronjob

15  2  *  *  1  /usr/local/bin/virt-back --quiet --backup sagat
15  23 *  *  5  /usr/local/bin/virt-back --quiet --backup mbison

Manual

russell@host:~$ virt-back --help
Usage: virt-back [options]
Options:
  -h, --help            show this help message and exit
  -q, --quiet           prevent output to stdout
  -d, --date            append date to tar filename [default: no date]
  -g, --no-gzip         do not gzip or tar the resulting files
  -a amount, --retention=amount
                        backups to retain [default: 3]
  -p 'PATH', --path='PATH'
                        backup path [default: '/KVMBACK']
  -u 'URI', --uri='URI'
                        optional hypervisor uri

  Actions for info testing:
    These options display info or test a list of guests.

    -i, --info          info/test a list of guests (space delimited dom names)
    --info-all          attempt to show info on ALL guests

  Actions for a list of dom names:
    WARNING:  These options WILL bring down guests!

    -b, --backup        backup a list of guests (space delimited dom names)
    -r, --reboot        reboot a list of guests (space delimited dom names)
    -s, --shutdown      shutdown a list of guests (space delimited dom names)
    -c, --create        start a list of guests (space delimited dom names)

  Actions for all doms:
    WARNING:  These options WILL bring down ALL guests!

    --backup-all        attempt to shutdown, backup, and start ALL guests
    --reboot-all        attempt to shutdown and then start ALL guests
    --shutdown-all      attempt to shutdown ALL guests
    --create-all        attempt to start ALL guests

Restoring

virt-back: restoring from backups

Response to L-Theanine: a 4000 Year Old Mind-Hack

2011-01-11T17:06:00-05:00

RE: https://worldoftea.org/caffeine-and-l-theanine

As the original article speculates, the combination and amount of L-Theanine and caffeine present in tea, appears to have notable affects on my programming and problem solving abilities. It's difficult to show conclusive evidence to this claim, but I generally feel most alert and organized when coding under the influence of 2 - 3 cups. In a typical day I drink about 4 cups, dosed an hour apart. I feel compelled to introduce another "mind hack" for problem solving or programming, which I call "Dream coding"

"Dream coding" is just that, coding at night during sleep. One can successfully invoke this "mind hack" by provoking thought about the problem while drifting to sleep. Once sleeping I'm able to see my code and my brain seems to iteratively problem solve. Most often I'm lucid during these events; aware of the problem I'm trying to solve and of the possible solutions. Other times I don't recall performing the solutions, but when I wake and after my first cup of tea, I'm able to solve my problem with a optimal and beautiful solution.

PBS NOVA published an episode about this phenomenon: What are dreams?

Does this happen to you? Have you experienced this?

You should follow me on twitter here.

I just used Google to purchase Chinese food takeout for two

2011-01-10T20:03:00-05:00

Yes, it has been done. Using my Google Chromium browser I searched Google Maps for the closest local Chinese restaurant. The small town venue doesn't have a website or menu posted online.

So I used Google Web to search for the restaurant name. I found takeouttonight.com (no longer works) which had the entire menu and VALID prices for the restaurant hosted online! I choose 'Sweet and Sour chicken', my wife selected Chicken and Broccoli.

I then logged into my Google Gmail account and clicked Call phone. Using the mouse I dialed the number I previously researched and after 3 rings an Asian speaking fellow answered the phone.

I said, "Hello, I'd like to place a delivery order." His reply, "Ok." resonated out of my 5.1 speaker setup.

I read off our selections and was super excited that it was WORKING! In my titillated state, I ended up ordering an extra side of egg rolls just to keep the conversation going.

He asked for my phone number to complete the order. I obliged by reading my Google phone number.

Google might be spamy these days, but it's still useful.

Who would have thought a lazy hacker could combine six different Google services to order Chinese food without leaving the desk?

I know it's possible now, I've seen me do it.

You should follow me on twitter here.

Graphic Design Powerhouse: Wacom Bamboo, Ubuntu, and MyPaint

2011-01-09T00:14:00-05:00

For Christmas this year my wife gifted me a Wacom Bamboo pen tablet. She claimed to have researched many products and looked for a model that worked well with Ubuntu and had positive user reviews. She ultimately chose the Wacom Bamboo model and I was so excited when I unwrapped it!

The driver install was flawless and I was up and running after about 5 mins and the first application I attempted to use was Gimp. Gimp worked but the flow felt "clunky".

I wanted an experience that better emulated a pencil or brush and canvas. I went to bed that night feeling a little unsatisfied.

The next day I decided to give the tablet another try, but instead of using Gimp I decided to try my luck with other software called MyPaint. MyPaint's interface feels perfect and it is loaded with brushes, pencils, and inks. MyPaint implements brush stroke pressure with unmatched precision!

The art on this page was created with my wacom BAMBOO tablet and MyPaint.

You should follow me on twitter here.

Thank you so much MyPaint developers for such a stellar piece of software!

MyPaint is a fast and easy open-source graphics application for digital painters. It lets you focus on the art instead of the program. You work on your canvas with minimum distractions, bringing up the interface only when you need it.

How did Stack Overflow get initial traction?

2011-01-05T02:57:00-05:00

Stack Overflow was a progressive and natural evolution of the standard clunky forum.

Using ajax it created a more fun and clean user experience.

Using badges and karma to gain responsibility allow forums post to become a game. People naturally like to see progression and growth, being able to watch my karma go up was a rush, similar to a video game. (some people plan elaborate flights to rack up frequent flyer miles, planning a trip properly could cost as little as 50 dollars to fly around the world in 24 hours, and gaining thousands of miles using point multipliers)

Stack Overflow was search-able! Here is a thought, create a forum that is easily index by search engines... One question per page, on topic with the most relevant posts at the top of the page. Who cares about user bounce backs, if they see your URL enough (StackOverflow.com) they will end up creating an account and becoming a user who creates even more quality content!

The site was geared toward geeks, and programmers! They adopt new tech the quickest.

Did I miss any reasons? Feel free to comment below.

You should follow me on twitter here

A homegrown python bread crumb module

2011-01-02T14:14:00-05:00

I have placed bread.py, a python breadcrumb module, into the public domain.

The bread object accepts a url string and grants access to the url crumbs (parts) or url links (list of hrefs to each crumb).

Tutorial: Add a Breadcrumb Subscriber to a Pyramid project using 4 simple steps.

Demo: https://school.yohdah.com/

You should follow me on twitter here

four2go!: A new spin on an old classic

2010-12-31T20:33:00-05:00

For the past two months I have feverishly worked on my side project. Initially I set out to work on this application for submission to Hacker News, lets make November "Launch an App Month".

The whole project took longer than expected but I was please with my progress so I continued development. Today, a month later, I have added the finishing touches to the app. It gives me great pleasure to announce the official launch of four2go!

A new spin ...

four2go! brings a new spin to the classic "four in a row" genre.

Instead of weighted tokens, four2go! uses balloons which float up rather than down. This new game mechanic requires players to retrain their brains and can easily throw off a novice player.

Another new spin, rather than playing on a coffee table, users play using a computer over the internet. The four2go! web application requires a simple account registration but does not require any download or installation!

Users can play with anyone, at anytime, anywhere in the world. The game sessions are persistent meaning the board will not disappear. This persistence allows players to check their games much like they would check their email.

The gumyum framework

The game itself was completed after the first weekend of coding. Programming the game was fun and exciting and originally four2go! was a command prompt application with a text-based gui. That version was not accessible and in order for people to play we needed to move to a different medium. So I set out to code a web front end.

After another weekend of coding I had the game working in the browser. It was very clunky (It didn't refresh after player moves) and didn't have any authentication, anyone could play any game.

Coding the game was simple, the framework was the difficult part...

Enter Stage Left: The gumyum framework. I needed a way to authentication users and a way to store and retrieve game sessions. I also needed a way to track user statistics and player history. Most importantly I needed a user interface that would be intuitive and fun to use! The remainder of the project set out to solve each of those needs. I needed the gumyum framework to make four2go! popular. I spent the rest of the time (1.5 months) working on gumyum and I feel safe claiming that the framework appears complete.

The great part about having a framework is the code reusability, I should be able to "plug" new games or new mechanics behind the gumyum framework with little trouble. My labor and efforts should pay off if I ever decide to build another game.

Artwork and graphic design

I was very fortunate to work with my brother Joey Ballestrini on some of the concept and game art. We designed the four2go! logo, the gumyum logo, and the balloons. Joey designed each of the "create" game icons. The clouds, the grass, and the dirt were created for another game and I decided they were a good fit so they were re-purposed. We both use gimp when creating graphical computer art.

LETS PLAY!

I have attached a video of the game, but I urge you to try four2go! out for yourself!

The Barrymores stole my heart then crushed it

2010-11-30T04:37:00-05:00

Recently while surfing pandora I stumbled upon a new favorite song, "Think Straight Again", composed by a ska band named The Barrymores. I embraced the bands alluring melodies after the first play.

What caught my attention?

Well the whole band rocks, hard. The main components of The Barrymores shape a full and energetic sound. They have a drummer, a bassist, a lead guitarist and two horn players (trumpet and trombone). Oh, I almost forgot to mention they contrast most ska bands by featuring a kickass lead female singer.

What compelled me to write about The Barrymores?

Though The Barrymores are a small town band, their technical abilities and prowess mimic more a band of a major record label. Their rocking upbeat style is similar to other ska bands. Each track creates a euphoric sensation and even the "sad" songs feel uplifting. When The Barrymores are on my stereo I typically end up chanting the lyrics while skanking around my office. I feel they bring the ska punk rock genre into another level of excellence.

And then I found the depressing news ...

I started researching The Barrymores because I was fascinated with their music. This research inevitably lead to some depressing news, The Barrymores are no longer together!

"I Am Jack's Broken Heart" -Fight Club.

Not often do I get excited about new music. In moments I grew attached to the group only to get crushed. I decided to investigate to uncover more.

Apparently The Barrymores went into hiatus after one of the founding members passed away. The Barrymore myspace page still seems active and continues to hosts a few of the bands tracks.

Sorry about the bummer, but you should still give the music a listen.

Russell Ballestrini

Integrating OpenAI with Dry's Sample Chat Game

Background

Prerequisites

Step-by-Step Integration

Testing the Integration

What's Next?

Optimizing Memory Management for Plausible Docker on Ubuntu

Background

Creating and Enabling Swap Space

Understanding Swappiness and Cache Pressure

Monitoring Your System

What's Next?

Understanding Pause Functionality in LucKey Park Built on Dry Engine

Understanding the Game's Status Management

Binding the Cancel Action

Handling the Cancel Action

Showing the Main Menu and Pausing the Game

The Main Update Loop Controlled by Dry Engine

End-to-End Pause Functionality

Installing YaCy on Ubuntu

Installing YaCy

Accessing the Admin Interface Remotely via an SSH Tunnel

Securing YaCy Portal and admin access with SSL/TLS

What's next?

Building 'Dry' and 'Park' from Source on Fedora Linux

Prerequisites

Cloning the Repositories

Building Dry with Debugging Enabled

Building Park with Debugging Enabled

Running Park

Analyzing Core Dumps on Fedora

Ubuntu 22.04 Letsencrypt Docker Hints

Vertically Scaling GitLab Server For As Cheap As Possible

Why GitLab Server?

Expected Results

GitLab Server Omnibus In Docker

Disaster Recovery

Russell Open Sources Remarkbox & MakePostSell into Public Domain!

GNUnet GNS Nameserver Operator Notes, Quickstart, & Cheatsheet

Copying files between cloud object stores like S3 GCP and Spaces using Boto3

I tried to install GitLab on TrueNAS and failed

WeeChat on-boot in a tmux session

Dreaming of unlimited home computer storage capacity

Idea 1

Unity Collab not working on Fedora because of an invalid CA certificate path

How to fallback to an older FreeNAS version on update failure

Installing Unity on Linux

Pre-signed GET and POST for Digital Ocean Spaces

Pre-signed GET or Download

Pre-signed POST or Upload

Hybrid Hot Water Heater Saves 69 Percent On Energy Consumption

AWS nvme to block mapping

Yuletide Trains and Homegrown Video Games

All Local Heros need a Gig Side Kick

Running DynamoDB Local service container on CircleCI 2.0

Troubleshooting

Pyramid SQLAlchemy bootstrap console script with transaction.manager

Quickstart to DKIM Sign Email with Python

The Missing dkimpy Quickstart Guide

Fulfilling Childhood Dreams: Solar

How-to Work From Home

Chapter 1: The Road to Remote

webwords: a minimal viable web app with docker in as many languages as possible

So You're Planning a Beta Test

My Search for the Perfect Alternative Hosted Comment System

Selenium grid on Kubernetes

Minikube

Nginx throw HTTP 503 maintenance JSON for all requests

Capability driven Presentation

Resilient Web Design

Techniques

Build RPM or DEB packages for Node.js using Jenkins and FPM

Register Super Powers with Pyramid add_request_method

Sharing a Pyramid cookie with Flask or Tornado

Setup Pyramid Signed Cookie Session

Teach Tornado how to read Pyramid cookies

My first Systemd Service Script and Override

Set static hostname for RHEL Centos 7 on AWS

If I swallow lots of air will I be lighter?