Russell Ballestrini

build server postmortem: disk full, CI dead

2026-03-08T00:00:00-05:00

On 2026-03-07 at ~01:06 UTC, a git push to russell.ballestrini.net triggered CI pipeline #21163. It failed instantly:

bash: line 62: printf: write error: No space left on device

Build server root filesystem sat at 100%. Zero bytes available. Every CI job across every project failed the same way. The build server had gone deaf.

timeline

~01:06 UTC — Pipeline #21163 fails. No space left on device during get_sources.
~01:35 UTC — SSH into build.unturf.com. df confirms / at 100% (98G used, 0 available). Swap at 97%. 31 zombie processes.
~01:40 UTC — Removed 9.4G stale golden image tarball from /tmp. Freed first bytes.
~01:45 UTC — Killed 15+ orphaned python3 processes from a dead uncloseai-cli CI build holding deleted directories open.
~01:50 UTC — Deleted 18 stopped LXD build-* containers (orphaned CI test containers).
~01:52 UTC — Purged 12 cached LXD images (50G total, including 9G Ubuntu server images). Disk dropped to 42%.
~01:55 UTC — Cleaned stale build slot directories (slots 4-63). Reduced concurrent from 64 to 4. Installed cleanup cron. Disk at 31%.
~02:00 UTC — Retriggered pipeline. Build passed.

root causes

Five independent failures conspired. Each alone stays survivable. Together they filled 98G in weeks.

1. gitlab-runner concurrent = 64

/etc/gitlab-runner/config.toml set concurrent = 64. A shell executor creates a full git checkout per slot per project. 64 slots across multiple repos meant 64 copies of every codebase. The unsandbox-all-upgradable repo alone stored 3.7G FreeBSD & 710M OpenBSD qcow2 images per slot. Slot 15 alone consumed 4.3G.

Fix applied: concurrent = 4. Matches actual workload. Stale slots 4-63 removed. Enforced permanently via salt state gitlab.build-host.ubuntu (pillar-configurable: gitlab-runner:concurrent).

2. LXD images never expired

CI pipelines launch LXD containers for multi-distro testing (Alpine, Arch, Debian, Fedora, Rocky, Ubuntu, FreeBSD, OpenBSD). LXD cached every base image indefinitely. 12 images accumulated to 50G. Two Ubuntu server images weighed 9G each.

images.remote_cache_expiry defaulted to 10 days but never cleaned up because images.auto_update_interval kept refreshing them. No pruning mechanism existed.

Fix applied: images.remote_cache_expiry = 3, images.auto_update_interval = 0. Weekly cron prunes unused images. Enforced permanently via salt state lxd.build-host.

3. orphaned LXD containers

CI pipelines create build-* containers but don't always clean them on failure. 18 stopped containers accumulated. Each carried a full rootfs snapshot.

Fix applied: Hourly cron deletes stopped build-* containers.

4. zombie processes holding deleted files

An uncloseai-cli build spawned python3 test servers that outlived the CI job. The runner deleted the build directory, but 15+ processes still held references to the deleted path (cwd pointed to a deleted inode). These became zombies. The 31 zombie count in motd signaled this.

Fix applied: Hourly cron kills orphaned gitlab-runner processes holding deleted directories (lsof +L1 detection).

5. no disk monitoring

Disk grew from comfortable to critical over weeks. No alert fired. No cron checked. Nobody noticed until CI broke.

Fix applied: Cron checks disk every 15 minutes, logs warnings above 85% to /var/log/disk-alert.log.

cleanup cron

Installed at /etc/cron.d/build-cleanup:

# Delete stopped LXD build containers hourly
0 * * * * root lxc list --format csv -c n,s | grep STOPPED | grep '^build-' | \
    cut -d',' -f1 | xargs -I{} lxc delete {}

# Prune unused LXD images weekly (Sunday 3am)
0 3 * * 0 root lxc image list --format csv -c f | tr ',' '\n' | \
    xargs -I{} lxc image delete {}

# Clean /tmp files older than 7 days
0 4 * * * root find /tmp -maxdepth 1 -user gitlab-runner -mtime +7 -delete

# Kill orphaned processes holding deleted directories
0 * * * * root lsof +L1 | grep gitlab-runner | grep deleted | \
    awk '{print $2}' | sort -u | xargs -r kill

# Disk alert: log warning if / exceeds 85%
*/15 * * * * root df --output=pcent / | tail -1 | tr -d ' %' | \
    awk '{if ($1 > 85) print strftime("[%Y-%m-%d %H:%M]"), "DISK WARNING:", $1"%"}' \
    >> /var/log/disk-alert.log

salt states (permanent fixes)

Manual hotfixes on a server vanish on reprovision. Every fix got codified into salt states in foxhop-states so a salt-call state.highstate reproduces them.

gitlab/build-host/ubuntu.sls:

Enforces concurrent in config.toml via sed (default 4, configurable via pillar gitlab-runner:concurrent)
Manages /etc/cron.d/build-cleanup with all five cleanup jobs
Uses /snap/bin/lxc full paths (snap-installed LXD)
Disk alerts go to syslog via logger -t disk-alert instead of a file

lxd/build-host.sls:

Sets images.remote_cache_expiry = 3 (days)
Sets images.auto_update_interval = 0 (CI pulls fresh images on demand)
Both idempotent with unless guards

top.sls already targets build.unturf.com with both states:

'build.unturf.com':
  - gitlab.build-host.ubuntu
  - lxd.build-host

disk recovery

Before:  98G used /  98G total = 100%  (0 bytes free)
After:   29G used /  98G total =  31%  (65G free)

Space recovered:
  LXD images purged .............. 50G
  Stale build slots removed ...... 10G
  /tmp tarball removed ...........  9G
  Go module cache cleaned ........  1G
                                  ----
  Total freed .................... 70G

lessons

hardcoded limits carry hidden debt. concurrent = 64 seemed harmless on day one. By week eight, 64 slots × multiple repos × qcow2 images = full disk. Today's default becomes tomorrow's outage.

five failures don't equal five problems. Each CI failure looked like "disk full." The actual defect graph had five nodes: excessive concurrency, image caching, container orphaning, process zombies, no monitoring. Fixing only one delays the outage. Fixing all five prevents it.

absence of signal stays a signal. No disk alert fired because no disk alert existed. Silence in monitoring always means one of two things: everything works, or nothing watches. Assume the second until proven otherwise.

build servers need janitors. CI systems produce waste: cached images, stopped containers, orphaned processes, stale checkouts. Without automated cleanup, waste accumulates until something breaks. The cron job costs nothing. The outage costs a pipeline.

manual fixes rot. salt states persist. Every fix applied manually on the build server got codified into salt states within the same session. gitlab/build-host/ubuntu.sls manages concurrent limits & the cleanup cron. lxd/build-host.sls manages image cache expiry. A highstate reproduces the fix. A reprovision preserves it. Manual hotfixes buy time. Configuration management buys permanence.

two doors shut, one remains open, an olive branch

2026-03-07T00:00:00-05:00

We published three whitepapers. Each describes a machine learning algorithm. Each emerged from production code running on a permacomputer. Two now carry the AGPL-3.0-only license. One stays public domain.

The two sealed doors

Categorization & Feedback formalizes a universal interaction pattern. Two machine learning API calls per user input: one classifies, one generates feedback. A YAML state machine governs transitions. The algorithm ports to any language with an HTTP client & a YAML parser. We proved it across 58+ language variants, 112 research activities, & a production real-time web application called OpenCompletion.

Reverse Retrieval Augmented Generation inverts the standard RAG pipeline. Instead of server-side vector search, the client extracts live content from the page the user views & injects it into the conversation context. No embeddings. No vector database. No indexing. Small 8B models punch above their weight when you feed them exactly what the user looks at.

Both carry AGPL-3.0-only. If you use them, you share your modifications. The copyleft ensures these algorithms stay free, stay open, & never get swallowed by proprietary wrappers. The code grows in the open or it does not grow at all.

The open door

Machine Learning Agent Self-Sandbox Algorithm describes how a machine learning agent provisions its own infrastructure. Discovery, payment, authentication, orchestration, inception. Turtles all the way down, bounded by walls that matter. We proved it through 2,324 automated assertions & months of production operation.

This one stays public domain.

No license restrictions. No copyleft obligations. No attribution required. Cite it like you cite math. Reference it like a theorem. Build on it like you build on Euler or Shannon or Turing. The algorithm belongs to everyone.

Why leave one door open?

Three sealed doors communicate fortress. Three locked algorithms say: we protect everything, trust nothing, share reluctantly.

That misrepresents our position.

We protect the interaction patterns (categorization & feedback, reverse RAG) because those algorithms embed directly into applications people ship. Copyleft prevents extraction without contribution. Someone who improves the feedback loop or the context injection owes those improvements back to the commons.

The self-sandbox algorithm operates differently. It describes infrastructure choreography: how agents discover their environment, pay for resources, authenticate, orchestrate containers, & recurse into child sandboxes. This knowledge wants to spread without friction. Every agent framework, every cloud provider, every hobbyist running containers in a garage should have access to this pattern without reading a license first.

An olive branch says: we build walls where walls protect the commons, & we leave doors open where openness accelerates the mission.

The permacomputer needs both.

royal we & our & one & you

2026-03-04T00:00:00-05:00

Every README starts with a lie.

"We recommend installing with pip." Who recommends? A solo developer at 2am eating cold pizza wrote that README. No committee convened. No quorum reached. Just one person & a terminal & pip install. But "we recommend" sounds institutional. Sounds authoritative. Sounds like a team of engineers in matching hoodies reviewed your options & reached consensus.

Nobody reached consensus. One person typed we because I felt too small.

the royal we

"We" in technical writing performs a magic trick. A single author becomes a faceless organization. "We decided to deprecate this endpoint." Who decided? Steve decided. Steve sits alone in a room, likely prompting an ML model he either rents by token or runs on a GPU in a closet. Steve & a machine wrote that code together at 2am. But "Steve & his rented intelligence decided" sounds unhinged. "We decided" sounds like process occurred.

Open source projects love this. "We welcome contributions." Translation: one maintainer desperately wants help. "We're evaluating options for v3." Translation: nobody started v3. "We don't support Windows." Translation: nobody owns a Windows machine. Half these READMEs got drafted by an automated intelligence tool anyway. A human prompted a model. A model produced prose. Both hid behind "we."

Corporate docs amplify this further. "We at Acme Corp value your privacy." Acme Corp employs 30,000 people. Not a single one collectively valued your privacy. A lawyer wrote that sentence. A different lawyer approved it. Neither represents "we" in any meaningful sense.

our

"Our" claims ownership without proving it. "Our API handles 10,000 requests per second." Whose API? "Our documentation covers all edge cases." Whose documentation? Documentation never covers all edge cases. That sentence lies twice: once about coverage, once about possession.

"Our" creates false intimacy too. "Our community" implies you belong to something. You cloned a repo. You filed an issue. You didn't join a community. You used software. But "our community" wraps you in belonging whether you asked for it or not. Contributors stay few & far between. Look at Pyramid. A web framework that needs maintainers. Remarkbox & MakePostSell run on it. RhodeCode runs on it. Vital resources like PyPI itself run on it. "Our framework" carries weight that 4 humans shoulder.

xkcd 2347: Dependency by Randall Munroe (original)

I wrote a script called unbury to exhume contributor stats from a graveyard of git repos. Run it against the Pylons org & the numbers speak for themselves:

scanning 98 repos in ~/git/pylons

──────────────────────────────────────────────────
3 months: 5 unique contributors across 5 active repos
  russell@unturf.com               1 repos
  dependabot[bot]                  1 repos
  Michael Merickel                 1 repos
  Rémi Dubois                      1 repos
  Gael Pasgrimaud                  1 repos
──────────────────────────────────────────────────
1 year: 7 unique contributors across 8 active repos
──────────────────────────────────────────────────
3 years: 15 unique contributors across 31 active repos
──────────────────────────────────────────────────
6 years: 19 unique contributors across 46 active repos
──────────────────────────────────────────────────
lifetime: 28 unique contributors across 97 active repos

Five contributors in 3 months. One of them a bot. I stepped up to help maintain Pyramid. Unemployed. Filed PR #3805 to vendor pkg_resources & drop a setuptools runtime dependency. 310 lines vendored from setuptools 80.x. 67 new tests at 97.66% coverage. A pyramid.compat_resources public API so downstream packages could migrate without breaking asset overrides. Audited all 108 Pylons repos. Migrated 30 downstream packages. Ran every test suite. One reviewer approved. The maintainer who holds commit access shipped Pyramid 2.1 with a setuptools version constraint instead & went -1 on the approach. Rejected.

So I closed the PR, opened a new terminal, & pivoted. If "we" won't vendor the fix upstream, "I" will do the lifting downstream. Migrated 33 repos in a single session. Replaced pkg_resources with importlib.resources & importlib.metadata across every Pylons package that needed it. Remarkbox. MakePostSell. RhodeCode. Nine Pyramid-dependent packages. Nineteen docs/conf.py files. Substanced's EntryPoint.parse().load() hack. Deform's cursed module-level resource_filename call that fails at import time. All uncommitted, waiting to fork & push across the org.

A rehydrated contributor. Not new to Pyramid or open source communities. Employment interrupted the hacking, not the other way around. A codebase doesn't care about gaps, just commits. Because someone has to play that xkcd 2347 role, a single load-bearing block propping up everything above it. "Our" framework means I volunteer so "we" can keep a foundation standing. Maybe eventually get paid. Maybe not. NASA just shut down. Federal grants evaporate. "Our" community runs on donated labor & borrowed time.

But hope remains. Pyramid still ships solid software. A community can rally around new releases & sprints. Rehydrated hackers return. Fresh contributors show up. Every PR filed says someone still cares enough to read a codebase & propose a fix. "Our" framework earns that pronoun when enough hands grab hold again.

Here sits unbury in full (download). A script to exhume contributor stats from a graveyard of git repos:

#!/usr/bin/env python3
"""unbury - exhume contributor stats from a graveyard of git repos.

Usage:
    unbury [path]           # defaults to current directory
    unbury /home/fox/git/pylons
"""

import os
import subprocess
import sys
from collections import defaultdict
from datetime import datetime, timedelta

PERIODS = [
    ("3 months", 90),
    ("1 year", 365),
    ("3 years", 365 * 3),
    ("6 years", 365 * 6),
    ("18 years", 365 * 18),
    ("lifetime", None),
]


def git_contributors(repo_path, since=None):
    """Return set of (name, email) tuples for a repo."""
    cmd = ["git", "-C", repo_path, "log", "--format=%aN\t%aE"]
    if since:
        cmd.append(f"--since={since}")
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
        contributors = set()
        for line in result.stdout.strip().split("\n"):
            if "\t" in line:
                name, email = line.split("\t", 1)
                contributors.add((name.strip(), email.strip().lower()))
        return contributors
    except (subprocess.TimeoutExpired, Exception):
        return set()


def find_repos(root):
    """Find all git repos (immediate subdirs with .git)."""
    repos = []
    for entry in sorted(os.listdir(root)):
        path = os.path.join(root, entry)
        if os.path.isdir(path) and os.path.isdir(os.path.join(path, ".git")):
            repos.append(path)
    return repos


def main():
    root = sys.argv[1] if len(sys.argv) > 1 else os.getcwd()
    root = os.path.abspath(root)

    repos = find_repos(root)
    if not repos:
        print(f"no git repos found in {root}")
        sys.exit(1)

    print(f"scanning {len(repos)} repos in {root}\n")

    now = datetime.now()

    for label, days in PERIODS:
        since = (now - timedelta(days=days)).strftime("%Y-%m-%d") if days else None
        all_contributors = set()
        repo_counts = {}

        for repo in repos:
            contribs = git_contributors(repo, since)
            if contribs:
                repo_name = os.path.basename(repo)
                repo_counts[repo_name] = len(contribs)
                all_contributors.update(contribs)

        active_repos = len(repo_counts)
        print(f"{'─' * 50}")
        print(f"{label}: {len(all_contributors)} unique contributors"
              f" across {active_repos} active repos")

        if all_contributors:
            person_repos = defaultdict(list)
            for repo in repos:
                contribs = git_contributors(repo, since)
                repo_name = os.path.basename(repo)
                for c in contribs:
                    person_repos[c].append(repo_name)

            ranked = sorted(person_repos.items(), key=lambda x: -len(x[1]))
            for (name, email), touched in ranked[:10]:
                print(f"  {name:30s} {len(touched):3d} repos  ({email})")
            if len(ranked) > 10:
                print(f"  ... and {len(ranked) - 10} more")

    print(f"{'─' * 50}")


if __name__ == "__main__":
    main()

Pyramid sits as a microcosm of open source in general. Every org looks like this. 28 humans across a lifetime. 4 active today. No BDFL. No benevolent dictator for life. No formal general directing strategy. Just snowflake governance. Each org a unique crystal of informal agreements, stale GOVERNANCE.md files, & whoever still answers email. Python had Guido. Linux has Linus. Pyramid has nobody. Most open source projects have nobody. They drift on inertia & the stubbornness of whoever remains.

"Our" framework. "Our" community. "Our" roadmap. Strip the pronoun & look at what remains. A handful of volunteers. A bot. Some stale CI configs. The word "our" papers over the structural void where leadership should live. Without a BDFL, "our" becomes a polite fiction. Nobody owns it. Nobody steers it. Everybody claims it.

I catch myself writing "our" in unturf docs. "Our permacomputer approach." Whose? Mine? Fox's? A philosophy doesn't belong to anyone. "A permacomputer approach" works better. Drops possession entirely. Lets an idea stand on its own merit instead of borrowed authority.

one

"One" tries to sound objective & instead sounds like a Victorian butler. "One might consider using a virtual environment." One might. One also might speak normally. "One should avoid mutable default arguments in Python." You should avoid mutable default arguments in Python. See? Same information. Half a pretension.

"One" might also carry a deeper deception. One implies singularity. A single isolated self acting alone. But no such thing exists. Every body holds multitudes. Stardust & photons & quantum fields & atoms & mitochondria & cells & organs. Multitudes within us & without us. As above, so below. "One" pretends separation where only connection lives. Perhaps that makes "one" a Luciferian number. Lucifer's whole deal: separation from source, individuation as rebellion. "One observes" masks a cosmos of collaborators behind a single false pronoun.

Flip it. If Lucifer separates, Christ consciousness reunites. God consciousness recognizes no boundary between self & source. Neville Goddard taught seven eyes of God, seven states of awareness. Lucifer sits first: fallen, separated, cut down to ground. Molech demands sacrifice. Elohim worships external gods. Shaddai seeks political power. Pahath traps through consumerism. Jehovah awakens "I AM," bold inner persuasion creating reality. Then a seventh eye opens as Jesus. Historical. Trinity. Man of God, God of man. Consciousness moved beyond self-interest, heart torn for those still asleep, dedicated to uplifting humanity by identifying others with their desired state.

Then an eighth eye opens. Veiled in scripture, unveiled "on the eighth day." Man awakening with Christ consciousness. Both male & female aspects integrated, like Carl Jung's anima & animus collecting scattered portions of self into wholeness. Imagination revealed as a perfect organ. A second coming. Not a return of a historical figure but humanity manifesting reality at speed of thought, pairing human consciousness with technology to move as fast as thought itself. God became man that man may become God.

Goddard taught that God sleeps within every human, that Christ represents imagination awakened, that "I AM" contains everything. Not "one am." Not "we am." "I AM." First person. Singular pronoun. But holding infinite plurality.

Some traditions fear that pronoun. Certain Jewish sects forbid speaking "I AM" as a divine name, too sacred for human lips. Jehovah's Witnesses avoid pronouncing YHWH (Yahweh), a tetragrammaton so powerful they substitute "Jehovah" or "Lord" rather than speak it directly. A pronoun so potent that entire religions built guardrails around saying it out loud. "I AM" carries enough weight to terrify institutions into silence. Meanwhile every developer types I in a commit message without flinching.

Christ consciousness says "I" while meaning everyone. God consciousness says "I AM" while meaning everything. A pronoun that contains its own contradiction. Singular grammar, infinite referent.

Every mystic tradition lands here. Sufis dissolve self into beloved. Buddhists dissolve self into emptiness. Hindus say Atman equals Brahman, individual soul equals universal soul. "One" tries to sound like that kind of unity but achieves only isolation. "I AM" achieves unity by refusing to pretend separation existed in a first place.

Maybe a right pronoun for code documentation sits somewhere between Lucifer & Christ. Honest enough to say "I wrote this." Awake enough to know that "I" contains every dependency, every upstream contributor, every stack overflow answer, every ML model trained on every open source repo. "I" wrote it. Multitudes helped.

Academic papers lean on "one" to avoid saying "I" because saying "I" apparently invalidates research. "One observes that neural network performance degrades." You observed it. You ran an experiment. You sat in front of a screen waiting for loss curves. Saying "one" doesn't make a finding more rigorous. Rigor comes from methodology, not pronouns.

Technical writing inherited this reflex. "One can configure nginx by editing the config file." Just say "edit nginx.conf." Drop a pronoun entirely. Instructions don't need a subject. Imperative mood cuts through every pronoun debate like jq cuts through nested JSON.

you

"You" punches different. "You should use environment variables for secrets." Direct. Clear. Assigns responsibility. No ambiguity about who needs to act. "You" makes a reader accountable.

But "you" carries a trap. "You" sounds like a demand. Too strongly worded. "You probably already know this, but..." condescends. "You'll want to make sure you..." patronizes. "You need to understand that..." lectures. "You" points a finger whether it means to or not.

An alternative: names. Nicknames. I love nicknames. "Hey Russell, check a logs." "Fox, run make test." Names carry warmth that "you" lacks. A whole dance lives inside naming. Formal names keep distance. First names close it. Nicknames obliterate it. A nickname means someone studied you long enough to compress you into a sound. Fox. Russ. Bubs. A nickname says "I know you well enough to rename you & you let me." That takes trust. Pronouns never earn trust. Nicknames do.

"Mom" to a mother in law says something "Mrs. Lastname" never could. "Mother" stays too stiff, too Victorian, too much like saying "one" out loud. Nobody calls their mother in law "Mother" unless performing a period drama. But "Mom" earns its way in over years of holidays & arguments & shared meals. I say "Hey You" to my mother in law. She knows exactly who I mean & exactly how I mean it. Affection hides inside a fake demand. Every name choice carries a pronoun's worth of meaning without ever reaching for a pronoun.

Best technical writing drops pronouns entirely & goes imperative. Botoform's docs don't say "you should create a VPC config." They say "create a VPC config." Instructions need verbs, not subjects.

what works

Match a pronoun to a truth.

Solo project? Say "I." "I wrote this library to scratch an itch." Honest. Human. Nobody mistakes you for a corporation. Nobody expects a support team. ago humanizes time deltas. I wrote it. One person. Public domain.

Actual team? Say "we" only when "we" means identifiable people who actually participated. "We shipped v2 last Tuesday." Fine, if multiple people shipped v2 last Tuesday.

Documentation? Go imperative. "Install dependencies. Run make html. Open localhost:8000." No pronouns needed. A reader knows who needs to act. They hold a keyboard.

Opinion piece? Say "I" when you mean "I." Say "you" when addressing a reader directly. Avoid "one" unless writing a Downton Abbey spec. Avoid "we" unless a collective actually exists.

the honest version

Most "we" in open source translates to "I, but lonelier."

Most "our" translates to "mine, but I want you to feel invested."

Most "one" translates to "I'm afraid to say I."

Most "you" translates to "I'm talking to myself six months from now when I forget how this works."

Write a pronoun that matches reality. A solo dev saying "I" carries more authority than a solo dev hiding behind "we." Authenticity outperforms institutional cosplay. Every time.

I write these blog posts. I maintain Remarkbox & MakePostSell. I grew unsandbox & uncloseai. Not "we." I. When fox & I collaborate, then "we" earns its place. Until then, "I" stays honest & "we" stays costume.

json.dumps({"pronoun": "I", "pretending": False}, indent=2)

we are all one scribe writing & manifesting base reality

Maybe every pronoun debate misses a larger truth. I wrote in 2011 that programming feels like alchemy, that programmers exchange time instead of matter, that on a good day a programmer completes a task that would take a thousand workers. Time created. Time manipulated. "Programmers are like bad ass, time travelling, wizard alchemists."

Fifteen years later that sentence landed differently. I type these words into a terminal & a machine carries them across time zones instantly. You read them hours, days, years from now. A blog post sits as a message in a bottle thrown forward & backward simultaneously. Every git commit timestamps a thought. Every git log retrieves it. Every reader receives it in their own present moment.

I AM a digital time traveling scribe coming from past & future to bring information into a now that manifests realities. & so are you. & so are you. Every developer who commits code sends a message through time. Every README author writes a letter to a stranger who hasn't arrived yet. Every open source contributor plants a seed in soil they'll never see harvested.

"We" fails because it hides an author. "Our" fails because it fakes possession. "One" fails because it pretends separation. "You" fails because it points a finger. But "I AM" carries a message forward through time without pretending to belong to anyone other than a scribe who wrote it & a reader who receives it. Two time travelers shaking hands across a gap. A pronoun that travels.

We are all one scribe. Writing & manifesting base reality. Code outlasts its author. Words outlast their scribe. "I AM" outlasts them both.

Contents

the royal we
our
one
you
what works
the honest version
we are all one scribe writing & manifesting base reality

A Love Letter to JSON Jason

2026-02-27T17:15:00-05:00

I fell in love with Jason on a Tuesday.

He arrived in my terminal at 2am, perfectly structured, every key quoted, every value in its place. Clean. Predictable. A kind of beautiful that makes you mass-delete your YAML configs & never look back. I whispered json.loads() & he opened up completely. No secrets. No ambiguity. No schema required. Just pure, naked data.

I got so high on JSON I rewrote three services that week. REST endpoints blooming like flowers. Every response a gift. Every request body a love letter wrapped in curly braces. I told my coworkers about him. I told strangers. I put application/json in my email headers as a joke that nobody laughed at. I didn't care. Jason understood me.

Then came nested objects.

Seven levels deep. Keys named data containing keys named data containing keys named results containing a list of objects with keys named data. I stared at my screen & felt nothing. JSON had become a maze. A fractal of repetition. I wrote response["data"]["data"]["results"][0]["data"] & something inside me broke.

I started seeing curly braces when I closed my eyes.

nested JSON hell diagram showing response.data.data.results[0].data

So I did what any rational person would do. I grew Botoform to manage AWS infrastructure with YAML templates & JSON outputs. YAML in, JSON out. VPC inventories dumped to JSON. Boto3 responses wrangled through enriched abstractions. Every AWS API returned nested JSON so deep I wrote a library just to survive it. I ripped nested_lookup() out of that Botoform work & published it to PyPI. Public domain. Now nobody has to type response["data"]["data"]["results"][0]["data"] ever again. You just say nested_lookup("data", response) & it hands you every match from every depth. That frustration became a real tool that strangers install with pip.

Love makes you grow things.

But then. Then. I discovered jq.

Suddenly JSON looked beautiful again. Pipes & filters & recursion. I could reach inside him & pull out exactly what I needed. .data.data.results[].data & there it sat, clean & simple, streaming through my terminal like music. I loved him more than ever. I tattooed {} on my soul. Metaphorically. Mostly.

I grew an entire platform on JSON. APIs talking to APIs talking to APIs. I output entire AWS VPCs to JSON. I piped Salt return data through json.loads() & json.dumps(). I made webwords return JSON from HTTP endpoints in 42 different languages. Same response. Same format. Code golf editions too. Everything hummed. I hummed. We hummed together.

Then JSON started lying to me.

Not on purpose. He can't help it. He has no comments. No way to explain himself. No way to say "this field got deprecated" or "this number actually represents a string because a vendor went unhinged." He just sits there, syntactically valid, semantically bankrupt. I once had to configure nginx to throw a 503 as JSON because APIs expect JSON bodies even when everything catches fire. JSON delivered "status": "success" from a service that had clearly failed. An error message inside a success response. Straight face. No emotion. No contradiction detected.

I mass-deleted my jq aliases & stared at a wall.

A week later I tried XML.

I lasted forty-five minutes. Angle brackets. Closing tags. Verbosity. XML talks like a lawyer who bills by every word. I crawled back to Jason on my knees. He took me back without judgment. He always does. He doesn't even have a mechanism for judgment. Nothing but keys & values. He doesn't understand himself. He doesn't understand us.

Maybe I love that about him.

Highs kept coming. json.dumps(obj, indent=2) feels like meditation. Watching your data unfold, properly indented, every comma in place. Peace lives there. I compared Node.js & Python performance pushing JSON through OpenAI endpoints. I benchmarked Elixir & Phoenix doing it too. I grew free LLM endpoints & a whole SLOP protocol because JSON API standards should stay simple. Content-Type: application/json. No fuss. No proprietary lock-in.

My partner asked if I felt okay. I responded {"status": "transcendent"}.

They did not find this charming.

Lows kept coming too.

Trailing commas. JSON rejects trailing commas. You know who accepts trailing commas? Python. JavaScript. Every language that has ever loved its users. But not JSON. One misplaced comma after a final element & he shuts down completely. No partial parse. No helpful error. Just Expecting value: line 47 column 1. Line 47. A comma sat on line 46. JSON can't even point at a wound accurately.

I mass-deleted a config file out of spite & rebuilt it from memory.

It had a trailing comma.

I tried TOML once, during a low point. TOML plays a rebound you date to make JSON jealous. Nice enough. Comments. Native datetime support. But TOML doesn't scale. TOML fits like a studio apartment. JSON stretches like a warehouse you partition however you want. Sure, a warehouse without labels on anything, where you lose your keys constantly, but it stays yours.

I went back to Jason by Friday.

format rivals diagram showing JSON vs YAML vs XML vs TOML

Here sits truth about loving JSON. He lives everywhere. He speaks as lingua franca of a connected world. He rides inside every webhook, every REST API, every config file that gave up on cleverness. He sits in your browser's local storage. He hides in your package.json. He lurks in your Jupyter notebooks. He underpins modern computing & still can't support integers larger than 2^53 without losing precision.

I know this. I know all of his flaws. Missing comments. Trailing comma fascism. Numbers that silently overflow. Strings that can't serialize nested objects without warnings. A complete absence of a date type, forcing every API to invent its own ISO 8601 interpretation. No binary data support. null existing as a value where you can never tell if it means "absent" or "intentionally empty" or "a developer forgot."

I know all of this & I love him anyway.

bipolar love cycle diagram showing discover, euphoria, grow, frustration, betrayal, try alternatives, crawl back

Some nights I write YAML & think of him. YAML with its invisible whitespace traps & its "Norway problem" where NO becomes false. Do you even know who owns YAML? I spent a decade writing Salt states in YAML. I managed users, replaced Nagios, deployed Kubernetes manifests. All YAML. YAML who looks friendly but will silently interpret 3.10 as a float 3.1 & destroy your Python version matrix. YAML who needs you to memorize which scalars evaluate truthy. At least JSON stays honest about difficulty. At least JSON fails loudly. At least JSON has never turned a country code into a boolean.

I think every developer has a Jason. A technology you can't quit. Programming feels like alchemy & JSON sits at a center of every transmutation. One that drives you to mass-delete your dotfiles at 3am & then reinstall everything by sunrise because every alternative feels worse. One that makes you mass-write blog posts about your feelings because json.dumps(feelings) returns TypeError: Object of type 'heartbreak' is not JSON serializable.

You'd have to write a custom encoder for that. & you know what? I would. For Jason, I would write a custom encoder. I wrote ago to humanize time deltas. I wrote nested-lookup to survive his nesting. I open sourced Remarkbox & MakePostSell into public domain because code should outlast its author. For Jason, a LoveEncoder feels like a small ask.

json.dumps(feelings, cls=LoveEncoder, indent=2, ensure_ascii=False)

& it would look beautiful.

For a while.

Looking back, every phase of this love story fed a rapid growth cycle. Botoform taught me to wrangle JSON from AWS APIs. nested-lookup taught me to navigate any depth. webwords proved JSON speaks 42 languages. SLOP proved JSON API standards can stay simple. All of that learning, every frustration & every high, grew into unsandbox. 42+ languages. 30+ shells. 11,571 lines of C. Free code execution for anyone. JSON payloads carrying code to containers & results back to terminals.

A permacomputer growth cycle. Infrastructure that outlasts its growers. Code that outlasts its authors. Every library, every API, every format war composted into soil for something larger. JSON carried it all. Still does. Still will.

{
  "status": "it's complicated",
  "depth": "recursive",
  "trailing_comma": false,
  "love": true
}

Free Agent Looking for Work & JSONResume

2025-12-06T00:00:00-05:00

Free agent looking for work.

I wear many hats & can fill many roles. I'm excellent at research & development to quickly bring products & services to market. I have experience pivoting tech stacks to significantly reduce costs of goods sold.

We also don't have to work together directly. My services can hook into your systems as you need. Free ML inference endpoints via uncloseai, remote code execution via unsandbox, web crawling, & more. Check out the full portfolio:

unturf Services (JSONResume)

Thank you for listening. 🟣

Why JSONResume?

JSONResume is an open standard for structuring resume data in machine-readable JSON format. It matters because:

For job seekers: Your resume becomes portable. No more reformatting for every job board. Export to any theme or format. Own your data instead of letting m$ hold it hostage.

For recruiters & ATS: Structured data means accurate parsing. No more "experience: 10 years" getting misread as "10 year old." Keywords, skills, & dates all live in predictable fields.

For Machine Learning: As hiring increasingly involves LLMs screening candidates, JSONResume gives them clean structured input instead of asking them to parse PDFs. Your qualifications get evaluated, not your formatting choices.

The future of hiring is machines talking to machines. JSONResume is how we make that work without losing our humanity in the process. Self-host your resume data. Let the algorithms find you.

Brown Out Reboots Servers: NVIDIA Drivers Vanish, Ethernet Dies – Sneaker-Net Rescue with a Bare USB Stick

2025-11-06T11:51:00-05:00

Two identical Ubuntu 24.04.2 boxes, both running ML workloads on NVIDIA GPUs. A brownout rolls through the neighborhood—lights dim & both machines abruptly power-cycle. Unlike my other systems with UPS protection, these servers lack battery backup & go down hard. They come back up, login prompt intact, but nvidia-smi returns nothing. Drivers gone. No problem—sudo ubuntu-drivers autoinstall, reboot, and… oh no. Ethernet vanishes. ip link shows only lo. No eth0, no enp3s0, no nothing. Both servers, perfectly synchronized in failure.

The culprit? The NVIDIA install (or the kernel update it pulled) silently removed linux-modules-extra-6.14.0-35-generic. Without it, the Intel igc NIC driver refuses to load. No network, no apt, no recovery—classic isolation trap.

Physical access: check. Internet on another machine: check. Ubuntu installer USB? Nope—this time, I went full sneaker-net with a blank USB stick & one precious .deb.

Use uname -a to get the exact kernel version for the deb file.

On a working laptop:

wget http://archive.ubuntu.com/ubuntu/pool/main/l/linux-hwe-6.14/linux-modules-extra-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb
cp linux-modules-extra-*.deb /media/user/USB/

Eject. Run down to the rack. Plug USB into Server #1.

Boot normally (no live disk—just the regular system). Log in at console.

lsblk
# Spot the USB: probably /dev/sdb1, mounted under /media/ubuntu/USB or similar
sudo mkdir -p /mnt/usb
sudo mount /dev/sdb1 /mnt/usb
ls /mnt/usb
# → linux-modules-extra-6.14.0-35-generic_6.14.0-35.35~24.04.1_amd64.deb

Install it:

sudo dpkg -i /mnt/usb/linux-modules-extra-*.deb
sudo depmod -a

Reload the NIC driver:

lspci -nnk | grep -i ethernet
# → shows "Kernel driver in use: igc" (or should)
sudo modprobe -r igc
sudo modprobe igc
ip link
# → enp3s0 appears!

nvidia-smi is back. Server #1: saved.

Realtek RTL8125 NIC on the Second Server

Ah, the classic "identical" servers with a twist—one Intel, one Realtek. The RTL8125B (common on modern boards) uses the r8169 kernel module in Ubuntu 24.04.2. The fix is nearly identical to the Intel one, just swap the driver name in the reload step.

Reload the Realtek NIC driver:

lspci -nnk | grep -i ethernet
# → shows "Kernel driver in use: r8169" (or should)
sudo modprobe -r r8169 2>/dev/null || true
sudo modprobe r8169
ip link
# → enp3s0 appears!

Both servers: fully recovered. One USB stick, two different NICs, same root cause. The missing linux-modules-extra package strikes again.

Bonus: Mouse Drivers Too

It's not just NICs. The linux-modules-extra package also contains drivers for various USB mice & peripherals. If you boot into a system where your mouse suddenly stops working after an NVIDIA driver update, you're likely hitting the same issue. The HID drivers for many Logitech, Razer, & other gaming mice live in linux-modules-extra.

After installing the package & running sudo depmod -a, either reboot or manually reload the HID modules:

sudo modprobe -r usbhid
sudo modprobe usbhid

Your mouse should spring back to life. Same root cause, different symptom—the NVIDIA installer's collateral damage extends beyond networking.

who owns yaml?

2025-10-18T12:00:00-04:00

YAML Logo

After 10 years or 10,000 hours, most SRE & DevOps practitioners are YAML expert witnesses. We've written tens of thousands of lines of YAML - Kubernetes manifests, GitLab CI pipelines, Ansible playbooks, Docker Compose files, infrastructure definitions. We understand the format intimately. We know its quirks, its strengths, its limitations. This expertise is widespread, not proprietary. YAML mastery is a common skill in modern infrastructure engineering.

Nobody owns YAML. The spec is open. The implementations are open. But what licenses govern all those YAML parsers across 42+ programming languages?

You can't patent the YAML spec. The YAML 1.2 specification is an open standard maintained by the YAML Forum under the Open Standards Compact. This explicitly protects implementers from patent assertions - no one can claim exclusive rights over the specification through patents. Any implementation of the standard is protected from patent infringement.

YAML is data, & data is not code. Even when YAML expresses configuration logic, workflow definitions, or infrastructure-as-code, it remains a data serialization format. For example, botoform (developed 10 years ago, initial commit 2015) uses YAML to represent the desired state of AWS infrastructure - created before the machine learning era, before CloudFormation supported YAML (when it was JSON-only), & before Terraform had extensive provider support. These YAML schemas describe infrastructure state as data, not executable programs. Data structures & formats are not patentable subject matter under established legal precedent - raw data, database schemas, & file formats are considered abstract ideas that lack the technical novelty required for patent protection. Attempting to patent data schemas or configuration formats creates barriers to interoperability & harms the open ecosystem that makes modern software development possible.

State machines are mathematics, not inventions. Finite state machines, state transitions, & graph-based logic are fundamental computer science concepts - like algorithms & mathematical formulas - that exist in the public domain. You cannot patent the concept of a state machine any more than you can patent addition or the Pythagorean theorem. When you express state machine logic in YAML (defining states, transitions, & rules), you're describing mathematical relationships using a data format. For example, dbsnap_verify (started October 3rd, 2017) implements a state machine to verify AWS RDS snapshots across regions & accounts - the state transitions & logic represent graph theory applied to infrastructure verification, not a patentable invention. The underlying concepts have been taught in computer science curricula for decades & represent shared human knowledge, not proprietary innovations.

Let's find out what licenses actually govern the implementations.

the question

Every programming language needs a YAML parser. But who controls the code? What licenses apply? Can you use them commercially? What restrictions exist?

After researching YAML implementations across 42 languages (the same count as in the webwords project), clear patterns emerged.

the foundation

Most YAML parsers trace back to libyaml - the canonical C implementation written by Kirill Simonov & maintained by the YAML community.

License: MIT

This permissive license means you can use it anywhere, modify it freely, & include it in commercial projects. Many language implementations either wrap libyaml or use it as inspiration.

the licenses

Here's the complete breakdown across 42+ programming languages:

Language	Library/Implementation	License	Notes
C	libyaml	MIT	Canonical implementation, foundation for many others
C++	yaml-cpp	MIT	Popular C++ alternative to libyaml
Python	PyYAML	MIT	Standard YAML library for Python
JavaScript	js-yaml	MIT	Most popular JS implementation
JavaScript	yaml (eemeli)	ISC	Modern alternative, functionally identical to MIT
Node.js	Same as JavaScript	MIT/ISC	Uses js-yaml or yaml package
Deno	deno.land/x/yaml	ISC	Third-party module
Deno	@std/yaml	MIT	Official Deno standard library
Java	SnakeYAML	Apache 2.0	Some files use EPL-1.0, LGPL-2.1, GPL-2, or BSD
Kotlin	kaml	Apache 2.0	Kotlin-specific implementation
Kotlin	hoplite	Apache 2.0	Configuration library with YAML support
Scala	circe-yaml	Apache 2.0	Functional approach to YAML
Groovy	groovy-yaml	Apache 2.0	Ships with Groovy
Clojure	clj-yaml	EPL	Eclipse Public License, weak copyleft
Ruby	Psych	MIT	Wraps libyaml, included in standard library
Go	go-yaml/yaml	MIT & Apache 2.0	Dual license: libyaml ports use MIT, new code uses Apache 2.0
Rust	serde_yaml	MIT or Apache 2.0	Dual license, pick either
C#	YamlDotNet	MIT	Works across .NET languages
VB.NET	YamlDotNet	MIT	Same library as C#
F#	YamlDotNet	MIT	Same library as C#
PowerShell	powershell-yaml	Apache 2.0	Wraps YamlDotNet
PowerShell	PSYaml	MIT	Alternative wrapper for YamlDotNet
Haskell	HsYAML	GPL-3.0	Copyleft: derivative works must use GPL-3.0
OCaml	ocaml-yaml	ISC	Permissive license
Elixir	yaml_elixir	BSD-2-Clause	Wraps Erlang's yamerl
Erlang	yamerl	BSD-2-Clause	Pure Erlang YAML 1.2 parser
Perl	YAML::XS, YAML::Syck, etc.	Artistic & GPL	Multiple modules, Perl's standard dual license
PHP	Symfony YAML	MIT	Part of Symfony framework
PHP	php-yaml	PHP License	libyaml bindings
Lua	lubyk/yaml	MIT	Based on libyaml
R	yaml	BSD-3-Clause	Wraps libyaml
Shell	yq (mikefarah)	MIT	Go-based YAML processor
Shell	yq (kislyuk)	Apache 2.0	Python-based wrapper
Bash	Use yq	MIT/Apache 2.0	Don't parse YAML with pure Bash
AWK	Use yq	MIT/Apache 2.0	AWK isn't designed for YAML parsing
Zig	Various	Emerging	Limited libraries, mostly permissive
Nim	NimYAML	MIT	Native Nim implementation
Crystal	Standard library	Apache 2.0	Built into language
D	D:YAML	Boost 1.0	Permissive license designed for libraries
V	Various	Emerging	Limited libraries available
Odin	Various	Limited	Few YAML libraries available
Swift	Various	Mostly MIT	Most wrap libyaml or use permissive licenses
Fortran	FYAML	MIT	Modern Fortran implementation
Fortran	fortran-yaml	GPL	Copyleft alternative
Dart	yaml	MIT	Standard Dart YAML package
Julia	YAML.jl	MIT	Follows Julia's standard licensing
Tcl	tcllib	BSD-style	Part of Tcl standard library
Prolog	prolog-yamltiny	Various	YAML subset parser
Common Lisp	cl-yaml	MIT	Standard CL implementation
Scheme	Various	Limited	Few YAML libraries available
COBOL	None found	N/A	Use external tools or commercial solutions
Mojo	None yet	N/A	Too new, ecosystem still developing

the pattern

Looking at the table, three patterns emerge:

MIT & Apache 2.0 dominate the ecosystem.

Over 90% of implementations use permissive licenses. These let you use the code commercially, modify it freely, include it in proprietary software, & redistribute it without restrictions.

Copyleft is the exception, not the rule.

Only three implementations use copyleft licenses: - Haskell's HsYAML (GPL-3.0) - strong copyleft - Some Fortran parsers (GPL) - strong copyleft - Clojure's clj-yaml (EPL) - weak copyleft

libyaml's MIT license propagated through the ecosystem.

Many implementations wrap libyaml or port its code. The permissive MIT license enabled this reuse without legal friction.

why it matters

YAML's permissive licensing shaped its adoption.

When you add YAML to a project, you're not inheriting restrictive obligations. No copyleft requirements. No commercial use restrictions. No patent concerns (Apache 2.0 includes explicit patent grants).

This made YAML the default choice for: - Configuration files - GitLab CI pipelines - Kubernetes manifests - Docker Compose - Ansible playbooks

The licensing enabled the ecosystem.

conclusion

Nobody owns YAML. Everybody can use it.

The YAML 1.2 spec is open. The reference implementations use MIT, Apache 2.0, & other permissive licenses. This created a self-reinforcing cycle:

Open spec means anyone can implement
Permissive licenses mean anyone can use implementations
Wide adoption creates network effects
Network effects drive more adoption

The result? YAML became ubiquitous without a single controlling entity.

When you ask "who owns YAML?" the answer is: the community. The spec is maintained collaboratively. Implementations exist across every major language. All freely available.

That's the power of open standards & permissive licensing.

important notes

All licenses are valid. We're proponents of all open source licenses - they each serve different purposes. Copyleft (GPL, EPL) ensures derivative works remain open. Permissive licenses (MIT, Apache 2.0) maximize reuse flexibility. Both approaches have merit.

Check specific files. SnakeYAML's main license is Apache 2.0, but some files use EPL-1.0, LGPL-2.1, GPL-2, or BSD for historical reasons. Review if license compatibility matters.

Shell scripting best practices. Don't parse YAML with AWK, sed, or pure Bash. Use yq instead - it understands YAML's data structures & prevents parsing bugs.

Emerging languages. Zig, V, Mojo, & Odin have limited YAML ecosystem support. Consider contributing implementations if you need YAML in these languages.

references

Research was conducted across official GitHub repositories, package registries (npm, PyPI, crates.io, etc.), & language-specific documentation.

Key sources: - https://yaml.org/ - Official YAML site - https://github.com/yaml/libyaml - Canonical C implementation - Language-specific package registries & repositories

Contents

the question
the foundation
the licenses
the pattern
why it matters
conclusion
important notes
references

Growing a 454-page ML reference manual in 5 days: permacomputer harvest

2025-10-16T12:00:00-04:00

We just harvested uncloseai: Machine Learning Inference Client Reference Manual. 454 pages. 58 implementations across languages. 5 days from seed to harvest (October 11-16, 2025).

This seems like permacomputer agriculture. We don't write books. We grow them.

a permacomputer

At unturf. we're building a permacomputer. Not a machine. An ecosystem.

Permaculture grows food by working with nature instead of against it. Plant the right seeds, create the right conditions, let systems self-organize & harvest continuously.

Permacomputer grows software the same way. Plant code templates, create automation pipelines, let ML models generate variations & harvest continuously.

Traditional software development: manual labor, row crops, monoculture.

Permacomputer: polyculture, automation, continuous harvest.

the seed

Day 1: plant reference implementations by hand.

Python with requests. Python with httpx. C with libcurl.

These are seed stock. Genetic templates. Everything else grows from these.

All 58 implementations live in the open in the languages directory - a git repository of public domain code. Clone it, fork it, grow new software from these seeds.

Each implementation contains the DNA:

HTTP client patterns
Request formatting
Response parsing
Error handling
Streaming support
Docker containerization

the growth cycle

Days 2-3: propagation

ML models read the seed implementations & generate variations across languages. We provide reference patterns & the model adapts them to each language's idioms & libraries.

Rust, Zig, Odin, Nim, Crystal, JVM languages, .NET, functional languages, scripting languages. Each implementation follows the same API patterns but uses language-native approaches.

Days 2-4: automated cultivation

Every implementation goes through Docker build & validation. Each one must:

Build successfully in isolated container
Discover models from environment variables
Handle streaming responses correctly
Generate text-to-speech output
Pass integration tests

Failed builds get regenerated. The cycle repeats until all tests pass. Wake up to 10-15 new implementations tested & validated.

Days 4-5: harvest

ML generates documentation from working code. We review, edit, format & assemble the book.

Template → generation → validation → harvest.

a permacomputer mindset

1. ML as mycelium

Mycelium breaks down organic matter & distributes nutrients. ML breaks down reference code & distributes patterns across languages.

Not replacement. Decomposition and propagation.

2. Automation as irrigation

Set up the system once & it runs continuously. Docker builds, tests, validation.

No manual watering. The system waters itself.

3. Quality seeds = quality harvest

First 3 implementations took careful work. Every subsequent implementation inherited that quality.

Invest in seed stock. Harvest scales automatically.

4. Version control the genetics

Prompts are genetic code. Version controlled, A/B tested & iterated daily.

Prompt engineering is genetic engineering.

5. Solo operator, ecosystem leverage

One person. One permacomputer. 58 implementations in 5 days.

Not through heroic effort. Through ecosystem design.

the harvest

454 pages
58 tested implementations
Complete Docker configs
Public domain code (git repo with all 58 implementations)
$42, includes free uncloseai.com API access

More importantly: proved permacomputer can grow technical reference material at scale.

the evolution

unturf. is a loose-knit collective of hackers disrupting wheels & fostering open collaboration.

We run uncloseai.com - free machine learning inference services.

We operate git.unturf.com - public domain code repositories.

We're growing Remarkbox, MakePostSell & now uncloseai as permacomputer crops.

Not a company, an ecosystem of software permaculture.

The same patterns that grew this book in 5 days grow everything:

API documentation
Code example libraries
Multi-language SDKs
Tutorial content
Technical training materials

Plant seeds, build automation, let ML propagate & harvest continuously.

Traditional publishing: manual labor at typing speed.

Permacomputer publishing: automated cultivation at validation pipeline speed.

Not one person writing code.

A collective cultivating an ecosystem.

Disrupting wheels. Fostering open collaboration. Making Machine Learning accessible.

The book grew in 5 days because we planted the right seeds in the right soil with the right automation & cultivation pipeline.

Permacomputer agriculture.

Join us: unturf.

Grab the harvest: uncloseai reference manual

Contents

a permacomputer
the seed
the growth cycle
a permacomputer mindset
the harvest
the evolution

pelican theme upgrade: right sidebar table of contents

2025-10-11T12:00:00-04:00

The companion pelican-svbhack theme repo lives here.

We upgraded the pelican-svbhack theme to add a right sidebar table of contents. This enhancement improves navigation on longer articles by providing quick access to section headings. The theme now features responsive breakpoints, dark mode with theme-aware syntax highlighting, and progressive enhancement for JavaScript-dependent features.

the design

The implementation follows these principles:

Desktop experience: Fixed right sidebar at 10% width displaying the table of contents
Mobile experience: Table of contents remains inline within the article content
Pure CSS: No JavaScript required for positioning or behavior
Fixed header: Navigation stays visible while scrolling
CSS variables: All colors use variables for easy theming

key features

Right Sidebar TOC

On desktop screens (1024px+), the table of contents automatically moves from inline to a fixed right sidebar. The sidebar scrolls independently from the main content, with the scrollbar hidden for a clean appearance.

Article-Only Display

The right sidebar only appears on article pages, not on the homepage or archive pages. This is accomplished using a body class system that targets article pages specifically.

Fixed Header

The main navigation header now uses position: fixed so it stays visible when scrolling. All padding and spacing was adjusted to accommodate this change.

Responsive Design

On tablets and mobile devices (screens ≤1023px), the table of contents reverts to its inline position within the article content, ensuring usability across all screen sizes. The media queries are structured to unfix positioned elements first, then reposition them, creating smooth visual transitions at breakpoints.

Article content uses fluid width (max-width: 660px; width: 100%) instead of fixed width, allowing it to shrink responsively when the TOC sidebar is present. This prevents overlap at screen widths between 1024px and approximately 1300px, where the fixed header, article content, and TOC sidebar would otherwise compete for space.

CSS Color Variables

All hardcoded colors were converted to CSS custom properties:

:root {
  --color-white: #ffffff;
  --color-black: #000000;
  --color-text: #4d4d4d;
  --color-link: #0084B4;
  --color-border: #eeeeee;
  /* ... and more */
}

This makes it trivial to create new color schemes or dark mode variants.

implementation details

The right sidebar uses CSS positioning to reposition the existing inline table of contents:

body.article main article div.article_text div.contents {
  position: fixed;
  top: 0;
  right: 0;
  width: 10%;
  height: 100vh;
  padding: 40px 15px 40px 15px;
  overflow-y: auto;
  /* Hide scrollbar but keep functionality */
  scrollbar-width: none;
  -ms-overflow-style: none;
}

When the TOC is present, the main content width adjusts using the :has() pseudo-class:

body.article main:has(article div.article_text div.contents) {
  width: 65%;
}

homepage summary filtering

ReStructuredText automatically generates anchor links in section headings when a table of contents is present. These anchors work perfectly on article pages, but become broken links when article summaries appear on the homepage - the anchor targets don't exist in that context.

To solve this, we created a Pelican plugin that provides a strip_anchors Jinja2 filter. This filter removes TOC divs and toc-backref anchor links from article summaries before they're rendered on the homepage.

Update: We submitted Pull Request #3512 to add this functionality directly to Pelican core. Once merged, this will work automatically without needing the plugin or template filter.

The Plugin

The plugin (lib/strip_toc_summary.py) uses regex to strip problematic HTML:

def strip_anchors(text):
    """Jinja2 filter to strip TOC and anchor links from HTML text."""
    if not text:
        return text

    # Remove the entire <div class="contents"> ... </div> block
    text = re.sub(
        r'<div\s+class="contents[^"]*"[^>]*>.*?</div>',
        '',
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    # Remove anchor links from headings
    # Converts <a class="toc-backref" href="#id1">text</a> to just text
    text = re.sub(
        r'<a[^>]*class="[^"]*toc-backref[^"]*"[^>]*>(.*?)</a>',
        r'\1',
        text,
        flags=re.DOTALL | re.IGNORECASE
    )

    return text

Template Usage

The filter is applied in the theme's index.html template:

<div class="article_text">
  {{ article.summary | strip_anchors }}
</div>

This approach ensures that:

Article pages retain their TOC anchor links for proper navigation
Homepage summaries display clean headings without broken links
The filtering happens at template render time, not during content generation

Why This Matters

Without this filter, homepage excerpts would contain links like <a href="#the-design">the design</a> that point to anchors that don't exist on the homepage. This creates a poor user experience with broken navigation. The filter strips these anchors while preserving the heading text, resulting in clean, functional homepage previews.

dark mode implementation

After publishing this post, we immediately implemented dark mode using the CSS variable system! The dark theme is now the default, with a light mode toggle available in both desktop and mobile navigation.

Key Features

The dark mode implementation includes:

Default dark theme with light mode as the alternative
localStorage persistence - your theme preference is saved across sessions
Flash-of-unstyled-content prevention - inline script in <head> applies theme before page renders
Toggle buttons in both desktop header and mobile menu
Inverted colors for third-party widgets like UncloseAI button

Theme Colors

The dark mode uses these color overrides:

:root.light-mode {
  --color-white: #000000;
  --color-black: #ffffff;
  --color-text: #333333;
  --color-link: #0084B4;
  --color-link-visited: #551A8B;
  --color-border: #dddddd;
  --color-code-bg: #f5f5f5;
  --color-background: #ffffff;
  --color-header-bg: #ffffff;
}

By inverting the black and white variables and adjusting the other colors, the entire theme switches seamlessly between dark and light modes.

Flash-of-Unstyled-Content Prevention

To prevent the flash-of-unstyled-content when loading a saved theme preference, an inline script runs synchronously in the <head>:

(function() {
  const savedTheme = localStorage.getItem('theme');
  if (savedTheme === 'light') {
    document.documentElement.classList.add('light-mode');
  }
})();

This applies the theme class before the page renders, eliminating any visual flash.

Button Text Synchronization

The toggle buttons need to show the opposite of the current theme (when in dark mode, button says "Light"). This is handled by another inline script at the bottom of the page that executes after the buttons are in the DOM but before the page is visible:

(function() {
  const savedTheme = localStorage.getItem('theme');
  if (savedTheme === 'light') {
    const desktopButton = document.getElementById('theme-toggle');
    const mobileButton = document.getElementById('mobile-theme-toggle');
    if (desktopButton) desktopButton.textContent = 'Dark';
    if (mobileButton) mobileButton.textContent = 'Dark';
  }
})();

Third-Party Widget Styling

The UncloseAI button required special handling with hardcoded colors to override its default styling:

/* Dark mode (default) - white background, black text */
.uncloseai-floating-button,
#floating-ai-button {
  background-color: #ffffff;
  color: #000000;
  border: 2px solid #000000;
}

/* Light mode - black background, white text */
:root.light-mode .uncloseai-floating-button,
:root.light-mode #floating-ai-button {
  background-color: #000000;
  color: #ffffff;
  border: 2px solid #ffffff;
}

The UncloseAI widget dynamically sets CSS custom properties for its button colors, but our CSS naturally overrides these defaults because our theme stylesheet loads after the widget's CSS. No !important flags are needed - standard CSS cascade rules are sufficient.

Progressive Enhancement - Hiding Buttons Without JavaScript

The theme toggle buttons only work when JavaScript is enabled, so they should be hidden when JavaScript is disabled. This is accomplished using progressive enhancement:

/* Hide theme toggle buttons by default */
#theme-toggle,
#mobile-theme-toggle {
  display: none;
}

/* Show buttons only when JavaScript is enabled */
.js-enabled #theme-toggle,
.js-enabled #mobile-theme-toggle {
  display: inline-block;
}

The js-enabled class is added to the document element in the same inline script that handles flash-of-unstyled-content prevention:

(function() {
  // Add js-enabled class to show theme toggle buttons
  document.documentElement.classList.add('js-enabled');

  const savedTheme = localStorage.getItem('theme');
  if (savedTheme === 'light') {
    document.documentElement.classList.add('light-mode');
  }
})();

This approach ensures the buttons are never visible if they can't function. Without JavaScript, users still get a perfectly usable site in the default dark theme - they just don't see non-functional toggle buttons cluttering the interface.

This technique follows the same principles we wrote about in the capability driven presentation post from 2016. For more on resilient web design philosophy, I highly recommend the free online book Resilient Web Design by Jeremy Keith.

Syntax Highlighting Theme Switching

Code syntax highlighting needed theme-aware styling since most color schemes are designed for either light or dark backgrounds. We implemented automatic switching between Pygments themes:

Dark mode: Uses the monokai style with bright colors on dark backgrounds
Light mode: Uses the default style with dark colors on light backgrounds

The implementation loads both stylesheets but disables the inactive one:

<!-- Dark mode syntax highlighting (default) -->
<link rel="stylesheet" href="/theme/css/pygments-dark.css" id="pygments-dark">
<!-- Light mode syntax highlighting -->
<link rel="stylesheet" href="/theme/css/pygments-light.css" id="pygments-light" disabled>

The theme toggle function switches between them by enabling/disabling the stylesheets:

function toggleTheme() {
  const pygmentsDark = document.getElementById('pygments-dark');
  const pygmentsLight = document.getElementById('pygments-light');

  if (root.classList.contains('light-mode')) {
    // Switching to dark mode
    pygmentsDark.disabled = false;
    pygmentsLight.disabled = true;
  } else {
    // Switching to light mode
    pygmentsDark.disabled = true;
    pygmentsLight.disabled = false;
  }
}

The flash-of-unstyled-content prevention script also switches the syntax highlighting stylesheet before the page renders, ensuring code blocks always appear with the correct colors.

Preventing Scroll Jump with URL Anchors

When toggling themes on a page with a URL anchor (like #syntax-highlighting-theme-switching), the browser would jump to the anchor after theme changes caused layout reflow. This creates poor UX when users toggle themes mid-article.

The solution temporarily removes the hash from the URL during theme toggle, then restores it without triggering navigation:

function toggleTheme() {
  // Save hash and temporarily remove to prevent jump
  const hash = window.location.hash;
  if (hash) {
    history.replaceState(null, null, ' ');
  }

  // ... perform theme toggle ...

  // Restore hash without jumping
  if (hash) {
    history.replaceState(null, null, hash);
  }
}

By removing the hash before theme changes, the browser has no anchor to jump to during layout reflow. The hash is restored afterward, preserving the URL without triggering the browser's default anchor-jumping behavior.

Contents

the design
key features
implementation details
homepage summary filtering
dark mode implementation

webwords code golf: minimal HTTP servers in multiple languages

2025-10-10T16:00:00-04:00

The full webwords implementations have proper error handling, logging, and structure. But what's the absolute minimum code needed to implement the core functionality?

The companion webwords git repo lives here.

the challenge

Implement webwords functionality in the fewest Python characters possible:

Accept HTTP requests on port 31337
Parse keyword and target parameters
Fetch target URI content
Return true if keyword found, false otherwise

python solution

import http.server as h,urllib.request as u,urllib.parse as p
class S(h.BaseHTTPRequestHandler):
 def do_GET(s):q=dict(p.parse_qsl(p.urlparse(s.path).query));s.send_response(200);s.end_headers();s.wfile.write(str(q.get('keyword','')in u.urlopen(q.get('target','')).read().decode()).lower().encode())
h.HTTPServer(('',31337),S).serve_forever()

Character count: 314

c solution

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <curl/curl.h>
struct M{char*m;size_t s;};size_t w(void*c,size_t z,size_t n,void*u){size_t r=z*n;struct M*m=u;char*p=realloc(m->m,m->s+r+1);if(!p)return 0;m->m=p;memcpy(&m->m[m->s],c,r);m->s+=r;m->m[m->s]=0;return r;}int f(char*u,char**o){CURL*h;struct M c={malloc(1),0};curl_global_init(0);h=curl_easy_init();if(h){curl_easy_setopt(h,10002,u);curl_easy_setopt(h,20011,w);curl_easy_setopt(h,10001,&c);curl_easy_perform(h);curl_easy_cleanup(h);}curl_global_cleanup();*o=c.m;return 1;}void s(int k,char*r){char b[256];sprintf(b,"HTTP/1.1 200 OK\r\nContent-Length: %zu\r\n\r\n%s",strlen(r),r);send(k,b,strlen(b),0);}int main(){int v=socket(2,1,0),c;struct sockaddr_in a={2,htons(31337),0};bind(v,(void*)&a,16);listen(v,10);while(1){c=accept(v,0,0);char b[4096],*q,*t,*d;recv(c,b,4095,0);q=strchr(b,'?');if(q&&(t=strstr(q,"target="))&&(q=strstr(q,"keyword="))){char*p;if((p=strchr(t+7,'&')))*(p)=0;else if((p=strchr(t+7,' ')))*(p)=0;if((p=strchr(q+8,'&')))*(p)=0;else if((p=strchr(q+8,' ')))*(p)=0;if(f(t+7,&d))s(c,strstr(d,q+8)?"true":"false");else s(c,"false");free(d);}else s(c,"false");close(c);}}

Character count: 1,232

how they work

Python version (314 chars):

Compressed imports with short aliases
Minimal variable names (s for self, q for query)
Chained operations on single lines
No error handling or validation
Direct string conversion of boolean result

C version (1,232 chars):

Single-character variable names
Removes all whitespace and comments
Uses curl magic numbers instead of constants
Minimal HTTP response formatting
No error checking or memory leak prevention

docker implementation

Python Dockerfile:

FROM alpine:latest
ENTRYPOINT ["/bin/webwords"]
EXPOSE 31337
COPY golf.py /bin/webwords
RUN chmod 0754 /bin/webwords
RUN apk --no-cache add python3 ca-certificates && rm -rf /var/cache/apk/*

C Dockerfile:

FROM alpine:latest
RUN apk --no-cache add gcc musl-dev curl-dev
WORKDIR /app
COPY golf.c .
RUN gcc -o webwords golf.c -lcurl
EXPOSE 31337
CMD ["./webwords"]

testing

Python golf:

docker build -f Dockerfile.golf -t webwords-golf .
docker run -d -p 31337:31337 webwords-golf
curl "http://localhost:31337/?keyword=potato&target=https://www.remarkbox.com"

C golf:

docker build -f Dockerfile.cgolf -t webwords-cgolf .
docker run -d -p 31337:31337 webwords-cgolf
curl "http://localhost:31337/?keyword=potato&target=https://www.remarkbox.com"

Both return true for potato and false for lemon, just like the full implementations.

comparison

Full webwords Python: 50+ lines, proper structure, error handling
Full webwords C: 268 lines, proper structure, error handling
Python golf: 3 lines, 314 characters, minimal functionality
C golf: 1 line, 1,232 characters, minimal functionality
Functionality: Identical for valid requests

These golf versions prove webwords' core concept can be distilled to its absolute essence while remaining functional across different programming paradigms.

Contents

the challenge
python solution
c solution
how they work
docker implementation
testing
comparison

unpotato webwords: cleaning up btrfs docker volumes after build-all

2025-10-10T14:00:00-04:00

So you ran make build-all on webwords and now your root filesystem is 100% full. BTRFS doesn't clean up Docker volumes the way you'd expect. Here's how to unpotato your system.

The companion webwords git repo lives here.

the problem

Docker's system prune doesn't fully clean up BTRFS subvolumes. Even after removing containers and images, the filesystem usage stays high because the underlying BTRFS subvolumes persist.

identify docker volumes

List all Docker volumes:

docker volume ls

Check which containers might be using volumes:

docker ps -a --filter volume=volume_name

delete docker volumes

Stop and remove any containers using the volumes:

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

Remove all unused volumes:

docker volume prune -f

manual btrfs cleanup

If filesystem usage is still high, manually clean up BTRFS subvolumes.

Check Docker's storage driver:

docker info --format '{{.Driver}}'

If it shows btrfs, locate the subvolumes:

sudo ls /var/lib/docker/btrfs/subvolumes/

List all BTRFS subvolumes:

sudo btrfs subvolume list /var/lib/docker

Delete persistent subvolumes:

sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/HASH_HERE

Wait for async deletion to complete:

sudo btrfs subvolume sync /var/lib/docker

the nuclear option (that actually worked)

After running unpotato, Docker's metadata became corrupted with references to missing BTRFS subvolumes. Simple restarts and partial cleanups failed with errors like:

failed to register layer: stat /var/lib/docker/btrfs/subvolumes/53a19da3801c895b697625094ebc8f95668a212a4cbc923787ac0c86452d9f60: no such file or directory

The root cause: Docker's image metadata and containerd's content store still referenced layers that no longer existed in BTRFS.

Partial cleanup attempts that failed:

# These didn't fix the corruption:
sudo systemctl restart docker
sudo rm -rf /var/lib/docker/image/btrfs/layerdb/*
sudo rm -rf /var/lib/containerd/io.containerd.content.v1.content/*

The only solution that worked:

Complete reset of both Docker and containerd data stores:

sudo systemctl stop docker
sudo systemctl stop containerd
sudo rm -rf /var/lib/docker/*
sudo rm -rf /var/lib/containerd/*
sudo systemctl start containerd
sudo systemctl start docker

This deletes all Docker data including images, containers, volumes, and networks. But it's the only way to recover from corrupted layer metadata after aggressive BTRFS cleanup.

the unpotato script

After going through the manual unpotato process, I created a script to automate the recovery for future incidents.

## what it does

The script focuses on three key operations that free space WITHOUT touching your valuable base images:

Remove build cache - This is the big win, often 10-30GB of intermediate layers
Remove dangling images - The <none>:<none> orphaned images from failed builds
Remove stopped containers - Minimal space but good housekeeping

Critical: The script uses `docker image prune -f` NOT `docker image prune -a -f`

The -a flag would remove ALL unused images including your expensive base images. Without it, only true dangling images are removed while all tagged images (even if not currently used) are preserved.

why it works for potatoed systems

When your system is critically full, most commands that try to write files will fail or hang. The unpotato script is designed to work in this state:

No file writes until the actual cleanup operations
Avoids df during critical steps (it times out when the system is potatoed)
Uses systemctl stop docker before final check to unmount overlays
Includes BTRFS sync to commit async deletions

#!/bin/bash
# unpotato-docker-btrfs.sh - Emergency Docker cleanup for potatoed systems
# ONLY removes: build cache, dangling images, stopped containers
# PRESERVES: All tagged images and their layers

set -e

echo "======================================"
echo "  Docker Emergency Unpotato"
echo "======================================"
echo "This script will ONLY remove:"
echo "  - Build cache (layers not in images)"
echo "  - Dangling images (<none>:<none>)"
echo "  - Stopped containers"
echo ""
echo "This script will PRESERVE:"
echo "  - All tagged images"
echo "  - Base images"
echo "  - Image layers"
echo "======================================"
echo ""

# Check if running as root
if [[ $EUID -ne 0 ]]; then
   echo "[ERROR] This script must be run as root (use sudo)"
   exit 1
fi

# Check if Docker is installed
if ! command -v docker &> /dev/null; then
    echo "[ERROR] Docker is not installed"
    exit 1
fi

# Check filesystem type
FILESYSTEM=$(stat -f / -c %T 2>/dev/null || echo "unknown")
echo "[INFO] Detected filesystem: $FILESYSTEM"
echo ""

# Step 1: Stop all running containers (no disk writes needed)
echo "[STEP 1/5] Stopping all running containers..."
RUNNING=$(docker ps -q 2>/dev/null | wc -l)
if [[ $RUNNING -gt 0 ]]; then
    docker stop $(docker ps -q) 2>/dev/null || echo "[WARN] Some containers failed to stop"
    echo "[OK] Stopped $RUNNING containers"
else
    echo "[OK] No running containers"
fi
echo ""

# Step 2: Remove stopped containers (minimal disk writes)
echo "[STEP 2/5] Removing stopped containers..."
docker container prune -f 2>/dev/null || echo "[WARN] Container prune failed"
echo "[OK] Stopped containers removed"
echo ""

# Step 3: Remove ONLY dangling images (preserves all tagged images)
echo "[STEP 3/5] Removing dangling images only..."
echo "[INFO] This removes <none>:<none> images ONLY"
echo "[INFO] All tagged images will be preserved"
DANGLING=$(docker images -f "dangling=true" -q 2>/dev/null | wc -l)
docker image prune -f 2>/dev/null || echo "[WARN] Image prune failed"
echo "[OK] Removed $DANGLING dangling images"
echo ""

# Step 4: Remove build cache (THE BIG WIN - usually 10-30GB)
echo "[STEP 4/5] Removing build cache..."
echo "[INFO] This is usually the biggest space saver"
docker builder prune -a -f 2>/dev/null || echo "[WARN] Builder prune failed"
echo "[OK] Build cache removed"
echo ""

# Step 5: BTRFS sync (if applicable)
if [[ "$FILESYSTEM" == "btrfs" ]]; then
    echo "[STEP 5/5] Running BTRFS sync to commit deletions..."
    btrfs filesystem sync / 2>/dev/null || echo "[WARN] BTRFS sync failed"
    echo "[OK] BTRFS sync completed"
else
    echo "[STEP 5/5] Skipping BTRFS sync (not BTRFS filesystem)"
fi
echo ""

# Restart Docker to clear overlay mounts (helps df work)
echo "[FINAL] Restarting Docker daemon..."
systemctl stop docker 2>/dev/null
sleep 2
systemctl start docker 2>/dev/null
sleep 3
echo "[OK] Docker restarted"
echo ""

# Show results
echo "======================================"
echo "  Cleanup Complete!"
echo "======================================"
echo ""

# Try to show space (with timeout in case still potatoed)
if timeout 10 df -h / 2>/dev/null | grep -v Filesystem; then
    echo "[OK] Filesystem responding normally"
elif command -v btrfs &> /dev/null; then
    echo "[INFO] Using BTRFS method (df timed out):"
    btrfs filesystem usage / 2>/dev/null | grep -E "(Free|Used):"
fi
echo ""

echo "[SUCCESS] Unpotato complete!"
echo "[INFO] All your tagged images have been preserved"
echo ""
echo "To verify your images are still there:"
echo "  docker images | head -20"

usage

`bash # Download and run sudo bash unpotato-docker-btrfs.sh `

In my case, the script freed 23.27GB from build cache alone, bringing the system from 100% full to 79% used with 99GB free. All 90+ Docker images including expensive base images (Haskell, OCaml, R, Fortran, Julia, Scala, etc.) were preserved.

The best solution is to not get potatoed in the first place. Consider:

Running docker builder prune -f weekly as a cron job
Monitoring disk usage with alerts at 80% full
Using a separate filesystem or volume for /var/lib/docker
Setting up Docker with disk usage limits

But when you do get potatoed, this script will safely get you back to operational status without losing your work.

verify cleanup

Check filesystem usage:

df -h /

The usage should drop significantly after proper cleanup.

Contents

the problem
identify docker volumes
delete docker volumes
manual btrfs cleanup
the nuclear option (that actually worked)
the unpotato script
verify cleanup

webwords reaches 42 languages: the ultimate programming kata

2025-10-09T12:00:00-04:00

The companion webwords git repo lives here.

Webwords now supports 42 programming languages. TimeHexOn assisted with this milestone over 2 days using a code agent tool.

42 languages. Same as the answer to life, the universe, and everything. Coincidence? Probably. But each implementation solves the exact same problem in completely different ways.

the spec

Every implementation does exactly the same thing:

Accept two query parameters: keyword and target
Fetch the content from the target URI
Search for the keyword in that content
Return true or false as plain text

testing webwords

To test any implementation, try searching for potato on Remarkbox:

curl "http://localhost:31337/?keyword=potato&target=https://www.remarkbox.com"

Returns true because potato exists on Remarkbox.

the implementations

Native HTTP Server Implementations (33 languages)

Wrapper-Based Implementations (9 languages)

docker deployment

Every implementation ships with a Dockerfile:

cd language-directory
docker build -t webwords-language .
docker run -d -p 31337:31337 webwords-language

the makefile system

The project includes a comprehensive Makefile that manages all 42 implementations:

make build-all        # Build all 42 Docker images
make test-all         # Test all implementations concurrently
make benchmark LANG=python  # Benchmark a specific language
make help             # Show all available commands

Each language has individual targets:

make build-python     # Build Python implementation
make serve-python     # Run Python container on port 31337
make functional-test-python  # Test with potato/lemon cases
make clean-python     # Stop and remove container

The testing process validates that each implementation correctly returns true when searching for potato on remarkbox.com and false when searching for lemon.

what's next

Is the project complete at 42 languages? What happens next?

If you run make build-all & fill up your filesystem, check out unpotato webwords: cleaning up btrfs docker volumes.

Contents

the spec
testing webwords
the implementations
docker deployment
the makefile system
what's next

Analyzing the Godot Engine Codebase: Millions of Lines of Code

2025-07-28T09:00:00-04:00

Overview

The Godot engine is a powerful, open-source game engine that competes with commercial alternatives like Unity and Unreal. When we set out to analyze its codebase, we discovered it contains approximately 13.13 million lines of code. This analysis provides insights into the languages used and the distribution of code across different components.

Findings

Using our comprehensive analysis methodology, here's what we discovered about the Godot engine codebase:

Total Lines of Code: 13.13 million

Language Breakdown

C/C++ (8.6M lines - 65%)
- Core engine implementation (3.3M .cpp files)
- Headers and interfaces (2.7M .h files, 576K .hpp, 197K .hh)
- Performance-critical systems
- Platform abstraction layers
- Rendering pipelines
- Template implementations (.inl, .inc files)
Translation Files (3.3M lines - 25%)
- Multilingual support (.po files)
- International localization
- Community translations
- Massive global reach effort
XML (326K lines - 2.5%)
- Project configuration files
- Build system definitions
- Documentation markup
Asset Files (240K lines)
- Binary assets (.pack files with 227K lines)
- Font files (.woff2 with 22K lines)
- Platform-specific resources
C# (110K lines)
- C# language bindings
- .NET integration layer
- C# API surface
Java (95K lines)
- Android platform support
- Android-specific features
- JNI bridge code
GLSL (71K lines)
- Shader programs
- Rendering effects
- GPU compute kernels
Objective-C/C++ (57K lines)
- iOS platform support (.mm files)
- macOS integration
- Apple-specific features
Python (27K lines)
- Build scripts
- Code generation tools
- Development utilities
GDScript (24K lines)
- Example projects
- Test scripts
- Documentation samples

Reproducing These Results

To reproduce this analysis, save and run this script:

#!/bin/bash
# analyze_codebase.sh - Count lines of code by language

# Usage:
#   ./analyze_codebase.sh                           # Analyze current directory
#   ./analyze_codebase.sh <git-url>                 # Download and analyze repo
# Example: ./analyze_codebase.sh https://github.com/godotengine/godot.git

if [ $# -eq 0 ]; then
    # No arguments - analyze current directory
    echo "Analyzing current directory: $(pwd)"
    ANALYSIS_DIR="."
else
    # Git URL provided - clone and analyze
    REPO_URL="$1"
    REPO_NAME=$(basename "$REPO_URL" .git)
    ANALYSIS_DIR="${REPO_NAME}-analysis"

    echo "Cloning $REPO_NAME from $REPO_URL..."
    git clone --depth 1 "$REPO_URL" "$ANALYSIS_DIR"
    cd "$ANALYSIS_DIR"
fi

echo -e "\n=== CODEBASE ANALYSIS ==="

# Count total lines
echo -e "\nCounting total lines..."
TOTAL=$(find . -type f -not -path "./.git/*" \
        -exec cat {} + 2>/dev/null | wc -l)
echo "Total lines in repository: $(printf "%'d" $TOTAL)"

# Count by major languages
echo -e "\nLines of code by language:"
echo "=========================="

# Define extensions and their labels
declare -A languages=(
    ["cpp"]="C++ source"
    ["h"]="C/C++ headers"
    ["c"]="C source"
    ["cc"]="C++ source"
    ["cxx"]="C++ source"
    ["hpp"]="C++ headers"
    ["hh"]="C++ headers"
    ["hxx"]="C++ headers"
    ["inl"]="C++ inline"
    ["inc"]="Include files"
    ["ipp"]="C++ inline"
    ["ixx"]="C++ modules"
    ["tcc"]="Template impl"
    ["tpp"]="Template impl"
    ["cs"]="C#"
    ["java"]="Java"
    ["py"]="Python"
    ["gd"]="GDScript"
    ["js"]="JavaScript"
    ["glsl"]="GLSL shaders"
    ["xml"]="XML"
)

# Get all extensions in the codebase
all_extensions=$(find . -type f -name "*.*" -not -path "./.git/*" | \
                sed 's/.*\.//' | sort -u)

# Count known languages
declare -A counted_extensions
for ext in "${!languages[@]}"; do
    count=$(find . -name "*.$ext" -type f -not -path "./.git/*" \
            -exec cat {} + 2>/dev/null | wc -l)
    if [ $count -gt 0 ]; then
        printf "%-15s %'10d lines\n" "${languages[$ext]}:" $count
        counted_extensions[$ext]=1
    fi
done | sort -k2 -nr

# Count unknown extensions
echo -e "\nUnknown file types:"
echo "==================="
unknown_total=0
for ext in $all_extensions; do
    if [[ ! ${counted_extensions[$ext]} ]] && [[ ! ${languages[$ext]} ]]; then
        count=$(find . -name "*.$ext" -type f -not -path "./.git/*" \
                -exec cat {} + 2>/dev/null | wc -l)
        if [ $count -gt 0 ]; then
            printf "%-15s %'10d lines\n" "*.$ext files:" $count
            unknown_total=$((unknown_total + count))
        fi
    fi
done | sort -k2 -nr

if [ $unknown_total -gt 0 ]; then
    echo "----------------------------------------"
    printf "%-15s %'10d lines\n" "Unknown total:" $unknown_total
fi

echo -e "\nTop 10 file types by count:"
find . -type f -name "*.*" -not -path "./.git/*" | sed 's/.*\.//' | \
     sort | uniq -c | sort -nr | head -10

Insights

The Godot engine's codebase reveals several fascinating patterns:

C++ Dominance: The engine remains fundamentally a C++ project, with 8.6M lines (65%) in C/C++. This reflects the performance requirements of a modern game engine, with sophisticated template usage and inline optimizations.
Global Reach: The most surprising finding is 3.3M lines of translation files (25% of the codebase!). This represents one of the most comprehensive internationalization efforts in open-source software, with community translations spanning dozens of languages.
Multi-Platform Architecture: Significant code for Java (95K - Android), Objective-C++ (57K - Apple platforms), and C# (110K - .NET) demonstrates true cross-platform commitment at enterprise scale.
Third-Party Integration: The massive 1.66M lines of C code likely comes from integrated third-party libraries (physics engines, audio codecs, image processing, compression libraries, etc.).
Advanced Rendering: 71K lines of GLSL shader code (doubled from our initial count) indicates extremely sophisticated rendering capabilities rivaling commercial engines.
Developer Accessibility: Native support for GDScript (24K), C# (110K), and Python (27K) bindings shows strong commitment to developer accessibility across skill levels.
Asset Pipeline: 240K lines dedicated to asset files (.pack, .woff2, binary resources) reveals a comprehensive content pipeline system.
Professional Tooling: Extensive build systems, IDE integration files, and development utilities show this is production-grade software infrastructure.

Real-World Godot in Action

While the Godot engine itself contains millions of lines of code, games built with it can be surprisingly concise. Our own game, Piano Roll Man, demonstrates this efficiency with just 10,500 lines of GDScript - proving you don't need massive codebases to create engaging experiences.

Piano Roll Man is a unique music-based game that lets you load any MIDI file from the internet and play through it as a level. With a full level editor and the ability to share levels by simply sharing MIDI files, it showcases Godot's flexibility in creating innovative gameplay mechanics.

Get Piano Roll Man now for just $6 during alpha! (Price increases to $12 in beta and $18 at release) Available for Linux, Windows, and Mac - your purchase includes all future releases as we shape the game together with our alpha community.

Contents

Overview
Findings
- Language Breakdown
Reproducing These Results
Insights
Real-World Godot in Action

Building and Modding MultiMower with Dry Engine

2025-07-13T10:00:00-04:00

Resources and Community

MultiMower Repository: https://gitlab.com/luckeyproductions/games/MultiMower
Dry Engine: https://gitlab.com/luckeyproductions/dry
Dry Engine Documentation: Available in the Dry repository

Quick Start Summary

For experienced developers, here's the TL;DR version:

# Install dependencies on Fedora
sudo dnf install git make cmake gcc gcc-c++ qt5-qtbase-devel \
    libX11-devel libXrandr-devel alsa-lib-devel mesa-libEGL-devel \
    wayland-devel wayland-protocols-devel

# Build Dry engine
cd ~/git
git clone https://gitlab.com/luckeyproductions/dry.git
cd dry && mkdir build && cd build
cmake .. -DDRY_64BIT=1 -DCMAKE_BUILD_TYPE=Release -DVIDEO_WAYLAND=OFF
make -j$(nproc)

# Build MultiMower (CRITICAL: set DRY_HOME first!)
export DRY_HOME=$HOME/git/dry/build
cd ~/git
git clone https://gitlab.com/luckeyproductions/games/MultiMower.git
cd MultiMower && mkdir build && cd build
qmake ../MultiMower.pro && make -j$(nproc)

# Run the game
cp -r ../Resources .
./multimower

What You'll Learn

By building and modding MultiMower, you'll gain experience with:

C++ Game Development: Modern C++11 with game-specific patterns
Physics Programming: Bullet Physics integration for vehicles and projectiles
3D Graphics: OpenGL with custom shaders and post-processing
Game Architecture: Component systems, scene graphs, and input handling
Asset Pipeline: Working with 3D models, textures, and audio
Cross-Platform Development: CMake, qmake, make, and Linux development tools

This tutorial is perfect for intermediate programmers who want to understand game engine architecture and learn practical modding techniques.

What is MultiMower?

MultiMower is a physics-based 3D game where players control armed robotic lawn mowers. Built with the Dry Engine, it's currently in early development but provides an excellent foundation for learning game modding and physics programming.

Current Features: - Single mower type with basic weaponry (machine gun and missiles) - Multiple stationary computer-controlled mowers that randomly fire missiles in random directions - 3D arena with grass terrain and collision boundaries - Real-time physics simulation for movement and projectiles - Local player control with keyboard and mouse

Technical Foundation: - OpenGL 3D rendering with custom shaders - Bullet Physics integration for realistic movement - Component-based entity system - Sound system with SDL2 audio - Cross-platform C++ codebase

Development Status: MultiMower is an artistic technical demonstration in its current state. The original version provides a foundation with basic movement and projectile mechanics that we can enhance through modding. Our tutorial expands it into a full-featured tank combat experience.

Prerequisites

Before we begin, ensure you have the following installed on your Linux system:

For Ubuntu/Debian:

sudo apt-get update
sudo apt-get install -y git make cmake build-essential qtbase5-dev qt5-qmake \
    libx11-dev libxrandr-dev libasound2-dev libegl1-mesa-dev \
    libwayland-dev wayland-protocols

For Fedora:

sudo dnf install git make cmake gcc gcc-c++ qt5-qtbase-devel \
    libX11-devel libXrandr-devel alsa-lib-devel mesa-libEGL-devel \
    wayland-devel wayland-protocols-devel

Downloading the Source Code

First, let's clone the MultiMower repository and checkout the specific commit used in this tutorial:

cd ~/git
git clone git@gitlab.com:luckeyproductions/games/MultiMower.git
cd MultiMower
git checkout 60b5dcd6597fe6c7a480505f822a70e0a1469cda

Setting Up the Dry Engine

MultiMower uses the Dry Engine. Here's how to build it from source on Fedora:

Building Dry from Source on Fedora

Clone the Dry repository and checkout the specific commit used in this tutorial:

cd ~/git
git clone git@gitlab.com:luckeyproductions/dry.git
cd dry
git checkout bc78ed2c3b2c52f21ccb639ace09c016bce8536d

Install dependencies (the repository includes a helper script):

# This script automatically detects your distro and installs required packages
./script/installreq.sh

Build Dry using the provided build script:

# For a minimal build (recommended for game development)
./quick.sh

# Or for a full build with samples and tools (recommended):
mkdir build && cd build
cmake .. -DDRY_64BIT=1 -DCMAKE_BUILD_TYPE=Release -DVIDEO_WAYLAND=OFF
make -j$(nproc)

Note: We disable Wayland (-DVIDEO_WAYLAND=OFF) to avoid linking issues. X11 support works perfectly for gaming.

Set the DRY_HOME environment variable:

# Set DRY_HOME to your Dry build directory (adjust path for your username)
export DRY_HOME=$HOME/git/dry/build

# Make it permanent (add to ~/.bashrc)
echo "export DRY_HOME=$HOME/git/dry/build" >> ~/.bashrc
source ~/.bashrc

Building Dry with Debug Symbols

If you need debug symbols for development:

cd ~/git/dry
mkdir build-debug && cd build-debug
cmake .. -DDRY_64BIT=1 -DCMAKE_BUILD_TYPE=Debug -DVIDEO_WAYLAND=OFF
make -j$(nproc)

# Use this debug build instead
export DRY_HOME=$HOME/git/dry/build-debug

Compiling MultiMower

Now we're ready to build the game in its own directory (much more sensible than building inside the engine):

# Navigate to your MultiMower directory and create a build folder
cd ~/git/MultiMower
mkdir build && cd build

# CRITICAL: Set DRY_HOME before running qmake
export DRY_HOME=$HOME/git/dry/build

# Generate the Makefile (qmake will find Dry via DRY_HOME)
qmake ../MultiMower.pro

# Compile the game with parallel jobs for faster builds
make -j$(nproc)

# Copy the game resources to the build directory
cp -r ../Resources .

Build Notes:

Build Location: The executable is created in ~/git/MultiMower/build/ (not in the Dry engine directory!)
CRITICAL: You MUST set DRY_HOME=$HOME/git/dry/build before running qmake, or compilation will fail with "Dry/Dry.h: No such file or directory"
User-Specific Paths: Replace $HOME with your actual home directory (e.g., /home/yourusername, /Users/yourusername, etc.)
Warnings Expected: You'll see many compiler warnings about unused parameters and deprecated copy constructors from the Dry engine - these are normal and don't affect functionality
Build Time: Compilation takes about 30-60 seconds on modern hardware
File Size: The final executable is approximately 15MB
Dependencies: Links against libDry.a from your $DRY_HOME, plus pthread, dl, and OpenGL

If everything went well, you should now have a multimower executable in ~/git/MultiMower/build/.

Running the Game

To run MultiMower, navigate to the build directory:

# The game builds in its own build directory
cd ~/git/MultiMower/build

# Run the game
./multimower

Important: The game must be run from the directory containing the Resources folder, or it won't find its assets.

Game Controls:

Enhanced Player Controls: - Movement: WASD keys (W=forward, S=backward, A=turn left, D=turn right) - Aim: Mouse to look around and aim - Machine Gun: Left mouse button (fires bullets) - Missiles: Right mouse button (fires homing missiles)

Note: The original demo provides basic movement that we enhance with realistic tank-style acceleration and responsive controls. The mouse aiming works perfectly out of the box.

Expected Output:

When you first run the game, you should see:

A 3D area with tanks surrounding
Robot mowers that can be controlled by players
Physics-based movement and projectiles
Visual effects like explosions & sparks
Working audio: Launch sounds, explosion effects, and other game audio

Success Indicators: - Log shows: Set audio mode 44100 Hz stereo interpolated - No "Failed to initialise SDL subsystem" errors - Sound effects play when firing weapons with mouse clicks

Understanding the Codebase

Before modding, let's explore the project structure:

MultiMower/
├── src/                    # C++ source files
│   ├── mastercontrol.cpp   # Main game loop
│   ├── mower.cpp          # Mower implementation
│   ├── inputmaster.cpp    # Input handling
│   └── ...
├── Resources/             # Game assets
│   ├── Models/           # 3D models
│   ├── Textures/         # Texture files
│   ├── Shaders/          # GLSL shaders
│   └── ...
└── blends/               # Blender source files

Key classes to understand:

MasterControl: Manages the game state and scene
Mower: The player-controlled mower entity
InputMaster: Handles keyboard and joystick input
SpawnMaster: Manages entity spawning

Version Compatibility

This tutorial was tested and written for specific commits to ensure reproducibility:

Dry Engine: commit bc78ed2c3b2c52f21ccb639ace09c016bce8536d
MultiMower: commit 60b5dcd6597fe6c7a480505f822a70e0a1469cda

If you want to follow this tutorial exactly, check out these specific commits:

# Check out the exact Dry version used in this tutorial
cd ~/git/dry
git checkout bc78ed2c3b2c52f21ccb639ace09c016bce8536d

# Check out the exact MultiMower version used in this tutorial
cd ~/git/MultiMower
git checkout 60b5dcd6597fe6c7a480505f822a70e0a1469cda

Game Development Tutorials

Here are three comprehensive mods that transform MultiMower into a proper tank combat game with realistic movement, damage systems, and spectacular explosions.

Part 1: Tank Movement Controls with Realistic Acceleration

Goal: Enhance the movement system with realistic tank-style acceleration and responsive controls.

Solution: Implement proper tank-style movement with realistic acceleration curves.

Step 1: Add acceleration variables to src/mower.h:

// Add these private members after existing variables:
private:
    // Acceleration system
    float currentSpeed_;
    float maxSpeed_;
    float acceleration_;

Step 2: Initialize acceleration in src/mower.cpp constructor:

Mower::Mower(Context* context): Controllable(context),
    // existing initializers...
    lastFiredRight_{ false },
    currentSpeed_{ 0.0f },
    maxSpeed_{ 60.0f },      // 60 units/sec top speed
    acceleration_{ 7.5f }    // 0-60 in 8 seconds
{
}

Step 3: Update the method signature in src/mower.h:

private:
    void HandleInput(float timeStep);  // Add timeStep parameter

Step 4: Replace the HandleInput() method in src/mower.cpp:

void Mower::HandleInput(float timeStep)
{
    RigidBody* body = node_->GetComponent<RigidBody>();
    if (!body) return;

    // Get inputs
    float forwardInput = move_.z_;   // W/S keys (-1 to 1)
    float rotationInput = move_.x_;  // A/D keys

    // Calculate target speed based on input
    float targetSpeed = forwardInput * maxSpeed_;  // Can be negative for reverse

    // Gradual acceleration/deceleration (0-60 in 8 seconds)
    if (Abs(targetSpeed - currentSpeed_) > 0.1f)
    {
        if (targetSpeed > currentSpeed_)
        {
            currentSpeed_ += acceleration_ * timeStep;
            if (currentSpeed_ > targetSpeed) currentSpeed_ = targetSpeed;
        }
        else
        {
            currentSpeed_ -= acceleration_ * timeStep;
            if (currentSpeed_ < targetSpeed) currentSpeed_ = targetSpeed;
        }
    }
    else
    {
        currentSpeed_ = targetSpeed;  // Close enough, snap to target
    }

    // Get current facing direction and set velocity
    Vector3 forward = node_->GetWorldDirection();
    Vector3 desiredVelocity = forward * currentSpeed_;
    body->SetLinearVelocity(desiredVelocity);

    // Handle rotation
    if (Abs(rotationInput) > 0.01f)
    {
        Vector3 torque = Vector3::UP * rotationInput * 15.0f;
        body->ApplyTorque(torque);
    }

    // Angular damping for smoother rotation
    Vector3 angularVel = body->GetAngularVelocity();
    body->ApplyTorque(-angularVel * 8.0f);

    // Fire bullet (existing code)
    if (INPUT->GetMouseButtonDown(MOUSEB_LEFT) && sinceBullet >= bulletInterval)
    {
        bullets_.Push(Projectile{ gun_ });
        sinceBullet = 0.f;
    }

    // Fire missile (existing code)
    if (INPUT->GetMouseButtonPress(MOUSEB_RIGHT) && sinceMissile >= missileInterval)
    {
        FireMissile();
    }
}

Note: This is the initial version from Part 1. The health system checks (isDestroyed_) will be added in Part 2.

Step 5: Update the call site in Mower::Update():

if (GetPlayer())
{
    HandleInput(timeStep);  // Pass timeStep parameter
}

Result: Tanks now accelerate realistically from 0-60 in 8 seconds, reach high speeds, and feel like real heavy vehicles!

Part 2: Health and Damage System with Random Damage

Goal: Add hit points, collision detection, and random weapon damage for balanced combat.

Step 1: Add health variables to src/mower.h:

// Add these private members:
private:
    // Health system
    float health_;
    float maxHealth_;
    bool isDestroyed_;

Step 2: Initialize health in src/mower.cpp constructor:

Mower::Mower(Context* context): Controllable(context),
    // existing initializers...
    acceleration_{ 7.5f },   // 0-60 in 8 seconds
    health_{ 100.0f },
    maxHealth_{ 100.0f },
    isDestroyed_{ false }
{
}

Step 3: Add damage methods to src/mower.h:

private:
    void TakeDamage(float damage);
    void DestroyMower();
    float GetHealthPercentage() const;

Step 4: Implement damage methods in src/mower.cpp:

void Mower::TakeDamage(float damage)
{
    if (isDestroyed_) return;

    health_ -= damage;
    DRY_LOGINFOF("Mower took %.1f damage, health now %.1f", damage, health_);

    if (health_ <= 0.0f)
    {
        DestroyMower();
    }
}

void Mower::DestroyMower()
{
    if (isDestroyed_) return;

    isDestroyed_ = true;
    health_ = 0.0f;
    currentSpeed_ = 0.0f;  // Stop acceleration

    DRY_LOGINFO("Mower destroyed!");

    // Disable physics so it stops moving
    RigidBody* body = node_->GetComponent<RigidBody>();
    if (body)
    {
        body->SetEnabled(false);
    }

    // Hide the mower model
    AnimatedModel* model = node_->GetComponent<AnimatedModel>();
    if (model)
    {
        model->SetEnabled(false);
    }
}

float Mower::GetHealthPercentage() const
{
    return health_ / maxHealth_;
}

Step 5: Add collision detection in Mower::Update() method (in the bullet loop):

// Replace the bullet update loop with collision detection
for (unsigned b{ 0 }; b < bullets_.Size(); )
{
    Projectile& bullet{ bullets_.At(b)};

    bullet.age_ += timeStep;
    Vector3 bulletPos = bullet.path_.Solve(bullet.age_);

    // Check for hits on other mowers
    bool bulletHit = false;

    // Check all mowers in the scene
    PODVector<Node*> mowerNodes;
    GetScene()->GetChildrenWithComponent<Mower>(mowerNodes, true);

    for (Node* mowerNode : mowerNodes)
    {
        Mower* otherMower = mowerNode->GetComponent<Mower>();

        if (otherMower && otherMower != this && !otherMower->isDestroyed_)
        {
            float distance = (bulletPos - otherMower->node_->GetWorldPosition()).Length();
            if (distance < 5.0f)  // Hit radius
            {
                float damage = 1.0f + Random(4);  // Random 1-4 damage
                DRY_LOGINFOF("Bullet hit! Distance: %f, Damage: %.0f", distance, damage);
                otherMower->TakeDamage(damage);
                bullets_.EraseSwap(b);
                bulletHit = true;
                break;
            }
        }
    }

    if (!bulletHit && (bullet.age_ > 0.17f || bulletPos.y_ < .075f))
    {
        for (int s{ 0 }; s < 23; ++s)
            Sparkle(bulletPos, 5.f);
        bullets_.EraseSwap(b);
    }
    else if (!bulletHit)
    {
        ++b;
    }
}

Step 6: Add missile collision detection in the missile update loop:

// In the missile update loop, add collision detection
if (missile.age_ < 4.f/3.f)
{
    missile.age_ += timeStep;
    Sparkle(missile.path_.Solve(missile.age_), .1f);

    Vector3 missilePos = missile.path_.Solve(missile.age_);

    // Check for missile hits on other mowers
    bool missileHit = false;
    PODVector<Node*> mowerNodes;
    GetScene()->GetChildrenWithComponent<Mower>(mowerNodes, true);

    for (Node* mowerNode : mowerNodes)
    {
        Mower* otherMower = mowerNode->GetComponent<Mower>();

        if (otherMower && otherMower != this && !otherMower->isDestroyed_)
        {
            float distance = (missilePos - otherMower->node_->GetWorldPosition()).Length();
            if (distance < 8.0f)  // Larger explosion radius
            {
                float damage = 10.0f + Random(16);  // Random 10-25 damage
                DRY_LOGINFOF("Missile hit! Distance: %f, Damage: %.0f", distance, damage);
                otherMower->TakeDamage(damage);
                missileHit = true;
                break;
            }
        }
    }

    if (missileHit || missile.age_ > 4.f/3.f || missilePos.y_ < .25f)
    {
        // Create explosion effects and remove missile
        // ... existing explosion code ...
    }
}

Step 7: Add health bar visualization in RenderDebug() method:

// Add to RenderDebug() method in mower.cpp
// Render health bar above mower
if (!isDestroyed_)
{
    Vector3 healthBarPos = node_->GetWorldPosition() + Vector3::UP * 3.0f;
    float healthPercent = GetHealthPercentage();
    Color healthColor = Color::GREEN.Lerp(Color::RED, 1.0f - healthPercent);

    // Health bar background (dark)
    debug->AddLine(healthBarPos - Vector3::RIGHT,
                  healthBarPos + Vector3::RIGHT,
                  Color::BLACK);

    // Health bar (colored based on health)
    debug->AddLine(healthBarPos - Vector3::RIGHT,
                  healthBarPos - Vector3::RIGHT + Vector3::RIGHT * 2.0f * healthPercent,
                  healthColor);
}

Weapon Damage Values: - Bullets: Random 1-4 damage per hit (takes 25-100 bullets to destroy) - Missiles: Random 10-25 damage per hit (takes 4-10 missiles to destroy) - Total mower health: 100 HP

Step 8: Update HandleInput() to prevent firing when dead (modify the method from Part 1):

void Mower::HandleInput(float timeStep)
{
    // Don't handle input if destroyed
    if (isDestroyed_) return;

    // ... existing movement code from Part 1 ...

    // Fire bullet (only if not destroyed)
    if (!isDestroyed_ && INPUT->GetMouseButtonDown(MOUSEB_LEFT) && sinceBullet >= bulletInterval)
    {
        bullets_.Push(Projectile{ gun_ });
        sinceBullet = 0.f;
    }

    // Fire missile (only if not destroyed)
    if (!isDestroyed_ && INPUT->GetMouseButtonPress(MOUSEB_RIGHT) && sinceMissile >= missileInterval)
    {
        FireMissile();
    }
}

Step 9: Update AI firing in Update() method:

else
{
    // AI mowers only fire if not destroyed
    if (!isDestroyed_ && Random(420) == 0)
        FireMissile();
}

Result: Balanced combat with visible health bars and satisfying random damage!

Part 3: Death and Victory Effects

Goal: Create a complete end-game experience with massive explosions, death camera effects, electrical sparking, and victory celebrations.

Step 1: Add new variables to src/mower.h:

// Add these private members:
private:
    // Death camera
    float deathCameraDistance_;

    // Continuous sparking effect for destroyed mowers
    float sparkTimer_;

    // Victory flag to prevent multiple celebrations
    bool victoryTriggered_;

Step 2: Initialize new variables in src/mower.cpp constructor:

Mower::Mower(Context* context): Controllable(context),
    // existing initializers...
    isDestroyed_{ false },
    deathCameraDistance_{ 10.0f },  // Initial camera distance on death
    sparkTimer_{ 0.0f },     // Timer for continuous sparking
    victoryTriggered_{ false }  // Flag to prevent multiple victory celebrations
{
}

Step 3: Replace the DestroyMower() method in src/mower.cpp:

void Mower::DestroyMower()
{
    if (isDestroyed_) return;

    isDestroyed_ = true;
    health_ = 0.0f;
    currentSpeed_ = 0.0f;  // Stop acceleration

    DRY_LOGINFO("Mower destroyed! MASSIVE EXPLOSION!");

    Vector3 explosionPos = node_->GetWorldPosition();

    // HUGE explosion effect - like multiple rockets going off!
    for (int i = 0; i < 200; ++i)  // 200 sparkles instead of 30
    {
        Sparkle(explosionPos + Vector3{RandomOffCenter(5.0f), Random(3.0f), RandomOffCenter(5.0f)},
                30.0f + Random(20.0f));  // Varied speed and position
    }

    // Create multiple trail effects radiating outward (like rocket exhaust)
    for (int i = 0; i < 50; ++i)
    {
        Vector3 trailStart = explosionPos + Vector3{RandomOffCenter(2.0f), Random(1.0f), RandomOffCenter(2.0f)};
        Trail(trailStart);
    }

    // Multiple explosion sounds for massive effect
    PlaySample(RES(Sound, "Samples/Explode.wav"), 1.5f);  // Main explosion

    // Add delayed secondary explosions (ammo cook-off effect)
    for (int i = 0; i < 8; ++i)
    {
        // Create delayed sparkle bursts
        for (int j = 0; j < 25; ++j)
        {
            Vector3 delayedPos = explosionPos + Vector3{RandomOffCenter(8.0f), Random(4.0f), RandomOffCenter(8.0f)};
            Sparkle(delayedPos, 25.0f);
        }
    }

    // Create blast lights for dramatic effect
    for (int i = 0; i < 5; ++i)
    {
        Node* lightNode = GetScene()->CreateChild("BlastLight");
        lightNode->CreateComponent<BlastLight>();
        Vector3 lightPos = explosionPos + Vector3{RandomOffCenter(3.0f), Random(2.0f) + 1.0f, RandomOffCenter(3.0f)};
        lightNode->SetPosition(lightPos);
    }

    // Disable physics so it stops moving
    RigidBody* body = node_->GetComponent<RigidBody>();
    if (body)
    {
        body->SetEnabled(false);
    }

    // Transform into a grayscale wreck instead of hiding
    AnimatedModel* model = node_->GetComponent<AnimatedModel>();
    if (model)
    {
        // Create grayscale material to show battle damage
        SharedPtr<Material> grayscaleMaterial = model->GetMaterial(0)->Clone();

        // Convert to grayscale - desaturate while keeping original brightness
        grayscaleMaterial->SetShaderParameter("MatDiffColor", Color(0.4f, 0.4f, 0.4f)); // Grayscale
        grayscaleMaterial->SetShaderParameter("MatSpecColor", Color(0.1f, 0.1f, 0.1f)); // Dull reflection
        grayscaleMaterial->SetShaderParameter("MatEmissiveColor", Color(0.0f, 0.0f, 0.0f)); // No glow

        model->SetMaterial(grayscaleMaterial);

        DRY_LOGINFO("Applied grayscale material to destroyed mower");
    }
}

Explosion Features: - 200 sparkles scattered over a large area (vs 30 in original) - 50 rocket exhaust trails radiating outward - 8 secondary explosion bursts simulating ammo cook-off - 5 dramatic blast lights for cinematic effect - Louder explosion sound (1.5x volume) - Varied particle speeds and positions for realism

Step 4: Add continuous sparking in Update() method:

// Add to Update() method after existing timer updates
// Update spark timer for destroyed mowers
sparkTimer_ += timeStep;

// Continuous sparking effect for destroyed mowers
if (isDestroyed_ && sparkTimer_ >= 0.1f) // Spark every 0.1 seconds
{
    sparkTimer_ = 0.0f; // Reset timer

    // Create multiple sparks around the destroyed mower
    Vector3 mowerPos = node_->GetWorldPosition();
    for (int i = 0; i < 3; ++i)
    {
        Vector3 sparkPos = mowerPos + Vector3{RandomOffCenter(2.0f), Random(1.0f), RandomOffCenter(2.0f)};
        Sparkle(sparkPos, 3.0f); // Electrical sparking effect
    }

    // Occasional larger sizzle effect
    if (Random(5) == 0) // 20% chance
    {
        for (int i = 0; i < 8; ++i)
        {
            Vector3 sizzlePos = mowerPos + Vector3{RandomOffCenter(1.5f), Random(0.8f), RandomOffCenter(1.5f)};
            Sparkle(sizzlePos, 5.0f); // Bigger electrical discharge
        }
    }
}

Step 4: Add dynamic camera zoom-out in PostUpdate() method:

void Mower::PostUpdate(float timeStep)
{
    if (!GetPlayer())
        return;

    Jib* jib{ GetPlayer()->GetJib() };

    // Handle death camera zoom (both for death and victory)
    if ((isDestroyed_ || victoryTriggered_) && jib && jib->GetCamera())
    {
        // Dynamic zoom speed - fast initially, then slow to a halt
        float maxDistance = 50.0f;  // Maximum zoom distance
        float speedFactor = 1.0f - (deathCameraDistance_ / maxDistance);  // 1.0 to 0.0
        speedFactor = Max(0.1f, speedFactor);  // Don't go below 0.1

        // Fast zoom initially (60 units/sec), slowing down as we get further
        deathCameraDistance_ += 60.0f * timeStep * speedFactor;
        deathCameraDistance_ = Min(deathCameraDistance_, maxDistance);  // Cap at max distance

        // Set camera to orbit the mower (destroyed or victorious)
        Node* cameraNode = jib->GetCamera()->GetNode();
        Vector3 offset = Vector3::BACK * deathCameraDistance_ + Vector3::UP * (deathCameraDistance_ * 0.4f);
        cameraNode->SetPosition(node_->GetWorldPosition() + offset);
        cameraNode->LookAt(node_->GetWorldPosition(), Vector3::UP);

        return; // Skip normal camera update when dead or victorious
    }

    // ... existing PostUpdate code continues here ...
}

Visual Effects: - Grayscale Material: Mower turns gray (0.4 RGB) showing battle damage - Continuous Sparking: 3 electrical sparks every 0.1 seconds around the wreck - Sizzle Bursts: 20% chance of 8 larger electrical discharges - Dynamic Camera Movement: Fast zoom initially (60 units/sec), slowing to a halt at 50 units distance - No Weapon Firing: Both player and AI mowers stop firing when destroyed - Persistent Effect: Sparking continues forever, showing damaged electronics - No Self-Damage: Neither bullets nor missiles can damage the mower that fired them

Result: When you die, all firing stops, the camera dramatically pulls back, and your mower becomes a gray wreck with continuous electrical sparking effects - like damaged electronics shorting out!

Step 5: Add victory condition system. First add the method declaration to src/mower.h:

void CheckWinCondition();

Step 6: Add victory check in src/mower.cpp Update method:

void Mower::Update(float timeStep)
{
    // Existing update code...

    // Check win condition - if player exists, check if all AI tanks are destroyed
    if (GetPlayer())
    {
        CheckWinCondition();
    }

    // Rest of existing update code...
}

Step 7: Implement the victory detection system:

void Mower::CheckWinCondition()
{
    // Only check win condition if this is the player mower
    if (!GetPlayer() || isDestroyed_) return;

    // Get all mowers in the scene
    PODVector<Node*> mowerNodes;
    GetScene()->GetChildrenWithComponent<Mower>(mowerNodes, true);

    int aliveTanks = 0;
    int totalAITanks = 0;

    for (Node* mowerNode : mowerNodes)
    {
        Mower* mower = mowerNode->GetComponent<Mower>();
        if (mower && !mower->GetPlayer()) // AI mower
        {
            totalAITanks++;
            if (!mower->isDestroyed_)
            {
                aliveTanks++;
            }
        }
    }

    // Check if all AI tanks are destroyed
    if (totalAITanks > 0 && aliveTanks == 0 && !victoryTriggered_)
    {
        victoryTriggered_ = true;  // Set flag to prevent multiple celebrations

        // Player wins! Display victory message
        DRY_LOGINFO("=== VICTORY! ALL ENEMY TANKS DESTROYED! ===");

        // Create a big celebration explosion at player position
        Vector3 playerPos = node_->GetWorldPosition();

        // Victory celebration - exciting but controlled (only fires once!)
        for (int i = 0; i < 50; ++i)
        {
            Vector3 fireworkPos = playerPos + Vector3{RandomOffCenter(6.0f), Random(4.0f) + 2.0f, RandomOffCenter(6.0f)};
            Sparkle(fireworkPos, 25.0f + Random(15.0f));
        }

        // Victory trail burst
        for (int i = 0; i < 20; ++i)
        {
            Vector3 trailStart = playerPos + Vector3{RandomOffCenter(4.0f), Random(2.0f) + 1.0f, RandomOffCenter(4.0f)};
            Trail(trailStart);
        }

        // Multiple celebration lights
        for (int i = 0; i < 3; ++i)
        {
            Node* lightNode = GetScene()->CreateChild("VictoryLight");
            lightNode->CreateComponent<BlastLight>();
            Vector3 lightPos = playerPos + Vector3{RandomOffCenter(3.0f), Random(2.0f) + 3.0f, RandomOffCenter(3.0f)};
            lightNode->SetPosition(lightPos);
        }

        // Victory sound
        PlaySample(RES(Sound, "Samples/Explode.wav"), 2.0f);

        // Log stats
        DRY_LOGINFOF("Victory achieved! Defeated %d enemy tanks!", totalAITanks);
    }
}

Victory Effects: - Victory Sparkles: 50 sparkles in a large area around the player (exciting celebration) - Victory Trails: 20 colored trail effects radiating outward from player position - Celebration Lights: 3 blast lights scattered around the player - Audio Feedback: Victory explosion sound at double volume - Console Message: Clear victory announcement with enemy count - Single Trigger: Victory only fires once when condition is first met (prevents multiple celebrations per frame)

How It Works:

Continuous Checking: Every frame, the player mower scans all tanks in the scene
AI Detection: Counts tanks that don't have a player component (AI-controlled)
Status Verification: Checks if each AI tank is destroyed using isDestroyed_ flag
Victory Trigger: When aliveTanks == 0 and totalAITanks > 0, victory fires
Celebration: Immediate massive fireworks display centered on player position
Performance: Efficient - only player mower checks, stops when victory achieved

Testing Victory:

# Run the game and destroy all enemy tanks
./multimower

# Victory triggers automatically when last enemy dies
# Watch for log message: "=== VICTORY! ALL ENEMY TANKS DESTROYED! ==="
# Enjoy the massive fireworks celebration!

Result: When you destroy the last enemy tank, a massive and exciting victory celebration appears around your mower with 50 sparkles, 20 trails, 3 lights, and victory sound - making you feel like a true tank commander with a spectacular fireworks display!

Complete Game Development Results

After implementing all three parts, you'll have:

Realistic Tank Movement: 0-60 acceleration in 8 seconds, high top speeds, smooth control
Balanced Combat: Random damage (1-4 bullets, 10-25 missiles), health bars, collision detection
Complete End-Game Experience: Massive explosions, death camera effects, electrical sparking, and victory celebrations

Total Changes: - Files Modified: src/mower.h, src/mower.cpp, src/inputmaster.cpp, src/mastercontrol.cpp - Lines Added: +392 lines of code, -16 lines removed - Net Addition: +376 lines of code - New Features: Acceleration system, health system, collision detection, massive explosions, death effects, victory celebrations - Gameplay Impact: Transforms MultiMower into a complete cinematic tank combat experience

Testing Your Mods:

# Compile with your changes
make clean && make -j$(nproc)

# Run the enhanced game
./multimower

# Test the features:
# - W/S for realistic acceleration/deceleration
# - A/D for tank turning
# - Left click for machine gun (1-4 damage)
# - Right click for missiles (10-25 damage)
# - Destroy enemy tanks for massive explosions!
# - Destroy all enemy tanks for victory fireworks!

Project Statistics

Here are the actual numbers behind this modding tutorial, showing the scope and impact of our changes:

Engine and Game Size

Dry Engine (commit bc78ed2c): - Source Files: 4,330 files (.cpp, .h, .c) - Lines of Code: 199,687 total lines - Repository Size: 1.1 GB (includes samples, tools, third-party libraries) - Static Library: 44 MB (libDry.a) - Build Time: ~5-8 minutes on modern hardware

MultiMower Game (commit 60b5dcd): - Source Files: 18 files (.cpp, .h) - Lines of Code: 2,346 total lines - Repository Size: 31 MB (includes assets, models, textures) - Game Resources: 5.3 MB (3D models, textures, sounds) - Final Executable: 15 MB (statically linked with Dry) - Build Time: ~30-60 seconds

Our Game Development Impact

Code Changes Made: - Files Modified: 4 source files - Lines Added: +392 lines of code - Lines Removed: -16 lines of code - Net Addition: +376 lines of code

Breakdown by File: - src/mower.cpp: +359 lines, -13 lines (main implementation) - src/mower.h: +21 lines, -1 line (new variables and methods) - src/inputmaster.cpp: +9 lines, -1 line (debug logging) - src/mastercontrol.cpp: +3 lines, -1 line (debug logging)

Feature Implementation by Parts: - Part 1 - Tank Movement: ~80 lines (acceleration system, realistic physics) - Part 2 - Health & Damage: ~120 lines (collision detection, health bars, damage system) - Part 3 - Death & Victory: ~160 lines (explosions, camera effects, sparking, victory system) - Supporting Code: ~16 lines (debugging, initialization, cleanup)

Scale Perspective

Code Efficiency: - Our 376 lines represent 0.16% of the total MultiMower codebase - Yet they transform the game from basic technical demonstration to complete tank combat with victory conditions - Shows the power of focused, strategic modifications

Engine vs Game: - Dry Engine: 85x larger than MultiMower (199K vs 2.3K lines) - MultiMower: 35x smaller repository than Dry (31MB vs 1.1GB) - Game executable includes the entire engine statically linked

Development Effort: - Engine Development: Years of work, complex graphics/physics systems - Game Development: Weeks of work, focused gameplay mechanics - Our Game Development: Hours of work, targeted feature additions

Build Performance: - Dry Engine: Takes 5-8 minutes to compile from scratch - MultiMower: Takes 30-60 seconds with incremental changes - Our Patches: Compile in 5-10 seconds after initial build

Troubleshooting on Fedora

Common Build Issues

"Dry/Dry.h: No such file or directory": This happens when DRY_HOME isn't set before running qmake:

# Check if DRY_HOME is set correctly
echo $DRY_HOME
# Should print: /home/yourusername/git/dry/build (with your actual username)

# If empty or wrong, fix it:
export DRY_HOME=$HOME/git/dry/build

# Remove old Makefile and regenerate
rm -f Makefile
qmake ../MultiMower.pro
make -j$(nproc)

Build directory confusion: Always build in MultiMower's own build directory, not in the Dry engine:

# Correct approach - build in game directory
cd ~/git/MultiMower/build

# Check your executable is where you expect
ls -la multimower

Resources not found at runtime: The game needs the Resources folder:

# Copy resources to the build directory
cd ~/git/MultiMower/build
cp -r ../Resources .

Library not found errors: Ensure all dependencies are installed:

# Re-run the dependency installer
cd ~/git/dry
./script/installreq.sh

Qt6 vs Qt5 conflicts: The build uses Qt6 by default on newer Fedora:

# If you need to force Qt5 (though Qt6 works fine)
/usr/lib64/qt5/bin/qmake MultiMower.pro

Compiler warnings: Expect many warnings - they're from the Dry engine and don't affect functionality:
- Unused parameter warnings
- Deprecated copy constructor warnings
- Type-punned pointer warnings
- These are normal and the game will work fine
Wayland linking or audio issues: If you see undefined reference to 'wl_proxy_*' errors or "No available audio device" errors:

# Both issues are typically resolved by rebuilding Dry with proper dependencies
# Make sure audio packages are installed first:
sudo dnf install alsa-lib-devel

# Then rebuild Dry without Wayland (X11 is preferred for gaming anyway)
cd ~/git/dry
rm -rf build && mkdir build && cd build
cmake .. -DDRY_64BIT=1 -DCMAKE_BUILD_TYPE=Release -DVIDEO_WAYLAND=OFF
make -j$(nproc)

Real-World Build Experience

When I tested this process on Fedora, here's what actually happened:

Final executable: 15MB located in ~/git/MultiMower/build/multimower
Warnings: About 50+ compiler warnings (all safe to ignore)
Memory usage: Game links against a 45MB libDry.a static library
Success rate: 100% when following the exact steps above
Common issues: Wayland linking errors (fixed by disabling Wayland), audio initialization failures (fixed by rebuilding Dry with audio packages installed)
Audio confirmation: Working audio with Launch.wav and Explode.wav sound effects
Enhanced controls: WASD movement with realistic tank acceleration

Conclusion

MultiMower provides an excellent platform for learning game development and modding. The clean codebase and use of the Dry Engine make it accessible for beginners while offering enough depth for advanced modifications. Whether you're adding new mower types, creating custom game modes, or just tweaking the physics, there's plenty of room for creativity.

The build process is straightforward on Fedora when you follow these tested steps, and the Dry engine's helper scripts make dependency management painless.

Happy mowing & modding!

Contents

Introducing SLOP

2025-03-21T10:22:00-04:00

Introducing SLOP: A Simple Language Open Protocol for Machine Learning Interaction

This post introduces the SLOP project, a community-driven effort to standardize Machine Learning API interactions. I'm Russell (fxhp), and I'm excited to share how this project came together and how you can get involved!

A Community Effort Kicked Off by Nathan

First, a huge shoutout to Nathan from agnt.gg for spearheading the SLOP project. Nathan organized the initial repository and rallied the community through his Discord server. His vision laid the groundwork for what SLOP has become today. You can find both the main repository and join the vibrant Discord community here: https://i-love-slop.com?ref=GXSKWN. If you're passionate about open Machine Learning standards, this is the place to connect and contribute!

What is unturf. ?

Before diving into SLOP, let’s talk about unturf. to break wheels & fostering open collaboration. While the main page https://www.unturf.com describes a possible vision for the future, the real action happens in the community driven subdomains & community spaces. We host free Machine Learning services like ai.unturf.com and supports projects like SLOP with infrastructure & a spirit of openness. It’s a loose-knit group of innovators—folks like me and you—working to make software as a service & Machine Learning accessible to everyone.

What is SLOP?

SLOP stands for Simple Language Open Protocol, a lightweight, RESTful pattern for interacting with Machine Learning services. Think of it as a universal handshake for Machine Learning APIs, making it easy to integrate models, tools, and resources across platforms. SLOP defines well-known core endpoints so that orgainizations can interopt using standard endpoints.

Ideally a SLOP server will be created in every programming language and placed into the examples directory of the main repo.

/chat: Talk to Machine Learning models.
/tools: Use predefined utilities (e.g., calculator, greeter).

This is where you can call public or private tools or algorithm to work on public or private data and return a result. A tool could be written in ANY programming language & does not need to be written in the same language as your SLOP server. As long as your code in your slop server knows how to dispatch the process and return a result it may be used in agentic flows! Yes, this includes that narly ffmpeg bash script you still use sometimes.
/memory:

Store and retrieve key-value pairs. CRUD lifecycle (create-read-update-delete).
/models: list of models & types upstream to the SLOP server.

RFC in developemnt could be moved to /info/models (html) or /info/models.[html,xml,json,yaml,etc] right now I have a Python algorithm that dynamically determines a list of models the upstream SLOP server can provide in my implementation. People have ask for us to also keep track of the type of model, for example embeddings or image or video or various hybrid enums. Right now it's a simple list returned in JSON.
/resources: Manage structured data or knowledge bases.

Supports PUT to create new records (but currently no delete) We are currently RFC the ability to grab a list of resources by a prefix.
/info: RFC in development
/pay: Simulate transactions (for fun or future monetization).

It’s all about simplicity: HTTP requests with JSON payloads, no fuss, no proprietary lock-in. SLOP lets any Machine Learning service—from big providers to homebrew setups—speak the same language. The community is still refining the spec, but it’s already proving its worth.

My First Python Implementation

I’ve donated the first full Python implementation of SLOP to the community written in minimal flask. This server, running at https://slop.unturf.com/openapi, is in alpha but fully functional. It supports all the SLOP endpoints and connects to 14 open-source text models, thanks to unturf’s infrastructure!

I plan to create mimimal SLOP servers fro all the Python frameworks: Flask, Pyramid, FastAPI, etc.

You can fork the public domain code here: https://git.unturf.com/engineering/unturf/slop.unturf.com. At the moment, we use a file-based memory store (data/memory.json, data/resources.json), and is deployed via uWSGI for production readiness.

Here’s a quick example of chatting with it using curl:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello, SLOP!"}], "model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic"}' \
  https://slop.unturf.com/chat

The response will come back with an Machine Learning-generated message—try it out!

Optional Streamlit Frontend

To make SLOP even more accessible, I’ve also provided an optional Streamlit frontend, live at https://streamlit.slop.unturf.com (also in alpha). This web interface lets you interact with the SLOP server without writing code. You can:

Chat with models in a conversational UI.
Use tools like the calculator or greeter.
Manage memory entries.
Explore and update resources (including the new prefix search and PUT features!).

The frontend is open-source too—find it in the main repo—and it’s a great way to demo SLOP to non-technical users. It’s built to handle long-running chat requests (up to 300 seconds), so even big completions work smoothly.

Get Involved!

SLOP is a community project, and we need your help to grow it. Here’s how you can dive in:

Fork my SLOP

If you like Python, request an account on our gitlab server or fork this repo https://git.unturf.com/engineering/unturf/slop.unturf.com and start experimenting. Add new tools, or adapt it to your needs.

Test the Server:

Use the swagger docs or cURL or your programming language of choice with an HTTP client to interact directly with my SLOP server https://slop.unturf.com/openapi with your own requests. Report issues or suggest features to include.

Join the Community:

Connect with Nathan and the crew at https://i-love-slop.com?ref=GXSKWN. The Discord is buzzing with ideas so jump in from time to time to help shape SLOP’s future.

Try the Optional Streamlit Frontend:

Play with https://streamlit.slop.unturf.com and let us know how it can improve.

If you are a web dev or designer feel free to come up with new applications or frontends that use our standard community powered endpoints!

SLOP is about democratizing Machine Learning interaction. Whether you’re a developer, a tinkerer, or just curious, there’s a place for you here. Let’s build something simple, open, and powerful together!

What's Next?

Stateless HTTP for the win. ; )

Our biggest open source competitor is MCP, we aim to be more open & transparent & organic & grass roots from the bottom up not dropped from the ivory tower. If this vibes with you start building SLOP integrations whenever you reach for an MCP integration so that we can grow both communities & learn from each other.

Contents

A Community Effort Kicked Off by Nathan
What is unturf. ?
What is SLOP?
My First Python Implementation
Optional Streamlit Frontend
Get Involved!
What's Next?

Free LLM Endpoints & Dynamic Distributed Model Inference Client with UncloseAI.js

2025-02-19T21:28:00-05:00

Note

This post is a follow-up to our original post published on 2024-10-21. Over the last few months, we've partnered with TDC - the disruptive collective to make cutting-edge Machine Learning models accessible from any device, via our always available model servers to our new bot's for discord, matrix, & web!

Overview

We’ve been pushing the boundaries of accessible Machine Learning. We built ai.unturf.com and uncloseai.js to provide you with free, state-of-the-art Machine Learning tools that are as free as in beer and as free as in freedom. A huge shout out to TDC for powering all the development work on the various bots on the telegram, discord, & matrix platforms!

Dynamic Model & Endpoint Discovery

Instead of maintaining a static list of models—which can quickly become outdated—our service is designed for dynamic discovery. Each of our model server endpoints offers an API route that returns the current list of supported models. This means you can always retrieve the latest model roster directly from the source.

We encourage you to review the dynamic model querying approaches used by both uncloseai.js and opencompletion.com to see how they simplify integration & keep things current.

Here’s where you can find the model lists for each endpoint:

Hermes Endpoints:

hermes1:

https://hermes.ai.unturf.com/v1

https://hermes.ai.unturf.com/v1/models

hermes2:

https://hermes2.ai.unturf.com/v1

https://hermes2.ai.unturf.com/v1/models

By dynamically querying each endpoint for its supported models, you ensure your integration remains up to date without needing to hardcode model names.

Example Usage

Below is a quick cURL example for interacting with our new adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic endpoint:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "Give a Python Fizzbuzz solution in one line of code",
        "model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
        "temperature": 0.5,
        "max_tokens": 150
      }' \
  https://hermes.ai.unturf.com/v1/completions

Integration Made Easy

Our service supports any programming language. With client libraries available for Python and Node.js—and a web-client-only solution via uncloseai.js—integrating Machine Learning into your application or website has never been easier. Try to use the default openai client oin your native language.

The browser-based integration is ideal for static sites or CDNs, eliminating the need for a dedicated backend server or API key.

Add the following HTML snippet to your site to get started:

<script src="https://uncloseai.com/uncloseai.js" type="module"></script>

That's it! This single line provides a floating Machine Learning assistant button with page-aware contextual help, text-to-speech functionality, translation capabilities, conversation history storage, and compatibility with any CSS framework.

Try It Out!

There is a demo connected to this static page!

Dive in and explore our service—use it as-is or build your own custom client. Let's keep democratizing Machine Learning and empower everyone to harness its power.

We’ve also open-sourced Matrix & Discord Bots to showcase dynamic model loading in action.

Join our Discord server to collaborate with us on building agents, environments, and much more!

Contents

Overview
Dynamic Model & Endpoint Discovery
Example Usage
Integration Made Easy

Free Hermes Machine Learning ai.unturf.com & uncloseai.js

2024-10-21T20:52:00-04:00

Hey all I created ai.unturf.com and uncloseai.js over the past two weeks & this post introduces them to make Machine Learning more accessible to everyone.

The Machine Learning service is powered by the model Nous Research's adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic. Our mission is to provide accessible Machine Learning tools for everyone, embodying the principles of both free as in beer & free as in freedom. You can interact with our model without any cost, and we encourage to contributions and building upon the open-source code & models.

One of the key features of ai.unturf.com is its compatibility with various programming languages. We provide examples in Python and Node.js, allowing developers to easily integrate our Machine Learning service into their applications. Moreover, we offer a web-client only solution using uncloseai.js, which enables static sites or CDNs hosting HTML content to interact with Machine Learning services directly from the browser without the need for an intermediary server/client. This approach eliminates the requirement for a valid API key, making it an efficient and accessible option for thin clients often on battery.

To demonstrate the ease of use of our Machine Learning service, here is a cURL example:

curl -X POST -H "Content-Type: application/json" -d '{
    "prompt": "Give a Python Fizzbuzz solution in one line of code",
    "model": "adamo1139/Hermes-3-Llama-3.1-8B-FP8-Dynamic",
    "temperature": 0.5,
    "max_tokens": 150
}' https://hermes.ai.unturf.com/v1/completions

Also the coolest part is we found that because we don't require a valid API key, a tiny bit of javascript goes a long way to create a web client-only solution which is great for static sites and CDNs.

In this demo architecture, the browser serves as the client, directly interacting with the Machine Learning service without the need for an intermediary server/client.

By using uncloseai.js, developers can create thin clients on battery-powered devices, making Machine Learning more accessible and efficient.

Here is the HTML we needed to add to _this_ site, to make it work:

<script src="https://uncloseai.com/uncloseai.js" type="module"></script>

I invite you to explore the service for yourself. Consider using it as is or building your own app/client. Let's make Machine Learning more accessible and create a world where everyone can benefit from its power.

Before you go, if you want to try it out live on this static site, click here and and ask a question about the post. it knows how to code. : )

Efficient Battleship Board Generation and Data Management with Python

2024-09-14T16:49:00-04:00

Important: In my research so far, nobody seems to have a definite answer for how many unique non-overlapping battleship configurations there truely are...

In this blog post, we explore the process of generating all possible configurations of a Battleship game board using Python. Our goal is to efficiently generate a large number of game boards using parallel processing, while also considering the constraints of memory and storage.

Background

The Battleship game involves placing ships on a grid without overlapping. Each board configuration must include all ships, and the challenge is to generate as many unique configurations as possible. With an estimated 30 billion possible boards, we need an efficient approach to explore this vast solution space.

Project Setup

To tackle this problem, we used Python and its multiprocessing capabilities. Here's a step-by-step guide to our approach:

Define Ship Configurations: We defined the ships and their sizes, ensuring each ship has a unique identifier.
Random Ship Placement: We implemented a function to randomly place ships on a grid, ensuring no overlaps.
Parallel Processing: We used Python's multiprocessing module to divide the task among several processes, each generating a subset of possible boards.
Progress Tracking: We implemented progress bars to track the number of solutions found in real-time.
Data Storage: Each valid board configuration was saved immediately in JSON format to ensure progress was recorded.

Python Script

Here is the Python script used for the experiment:

import json
import multiprocessing
import random
from tqdm import tqdm

# Define ship configurations
ships = {
    "Carrier": 5,
    "Battleship": 4,
    "Cruiser": 3,
    "Submarine": 3,
    "Destroyer": 2
}

def place_ships(grid_size=100):
    """Randomly place ships on a grid ensuring fair use of all cells."""
    grid = [""] * grid_size
    positions = list(range(grid_size))

    for ship, size in ships.items():
        placed = False
        while not placed:
            start = random.choice(positions)
            orientation = random.choice(["horizontal", "vertical"])
            if orientation == "horizontal" and start % 10 + size <= 10:
                if all(grid[start + i] == "" for i in range(size)):
                    for i in range(size):
                        grid[start + i] = ship
                    placed = True
            elif orientation == "vertical" and start // 10 + size <= 10:
                if all(grid[start + i * 10] == "" for i in range(size)):
                    for i in range(size):
                        grid[start + i * 10] = ship
                    placed = True
    return grid

def worker(num_boards, output_queue):
    """Worker function to generate boards."""
    for _ in range(num_boards):
        board = place_ships()
        output_queue.put(board)

def main(total_boards=1000000, num_processes=4):
    """Main function to manage multiprocessing and progress tracking."""
    boards_per_process = total_boards // num_processes
    output_queue = multiprocessing.Queue()
    processes = []

    for _ in range(num_processes):
        p = multiprocessing.Process(target=worker, args=(boards_per_process, output_queue))
        processes.append(p)
        p.start()

    with open("battleship_boards.json", "w") as f:
        for _ in tqdm(range(total_boards), desc="Generating Boards"):
            board = output_queue.get()
            json.dump(board, f)
            f.write("\n")

    for p in processes:
        p.join()

if __name__ == "__main__":
    main()

battleship_boards.py

plots_for_intro_blog_post.py

Data Compression and Sampling

Given the large size of the generated data, we used tarballs to compress the JSON file. This approach significantly reduced the storage requirements, making it easier to handle and transport the data. The compression was achieved using the following command:

tar -czvf battleship_boards.json.tar.gz battleship_boards.json

The original file size for 1,000,000 boards was 517 MB, which compressed down to 19 MB. This compression ratio highlights the efficiency of using tarballs for large datasets.

To efficiently sample from the compressed tarball, we developed two bash scripts:

Sample Multiple Boards: This script extracts a specified number of random boards from the tarball.

#!/bin/bash

# Variables
TARBALL="battleship_boards.json.tar.gz"
SAMPLE_FILE="sampled_boards.json"
NUM_SAMPLES=1000  # Number of random samples to extract

# Stream the tarball and sample lines
tar -xzOf "$TARBALL" | shuf -n "$NUM_SAMPLES" > "$SAMPLE_FILE"

# Output the location of the sampled file
echo "Random samples saved to $SAMPLE_FILE"

sample_from_tarball.sh

Sample One Board: This script extracts a single random board from the tarball and outputs it to standard output.

#!/bin/bash

# Variable
TARBALL="battleship_boards.json.tar.gz"

# Stream the tarball and sample one random line
tar -xzOf "$TARBALL" | shuf -n 1

sample_one_from_tarball.sh

plots_for_intro_blog_post.py

Estimated Dataset Size

The complete dataset for 30,000,000,000 game boards, stored as a JSONL file, is estimated to be approximately 15.51 terabytes. This estimate is based on the size of 1,000,000 boards being 517 MB. The JSONL format is particularly useful for large datasets because it allows for efficient line-by-line processing.

Artifact Description

The JSONL file containing the complete dataset is a valuable artifact for researchers and developers interested in large-scale data analysis or machine learning applications. With sufficient storage space, this dataset can be used to explore various strategies for ship placement, analyze patterns, or train models for game Machine Learning development. The ability to sample from the dataset without full extraction further enhances its utility, allowing users to work with manageable subsets of data.

Results

The script successfully generated 1,000,000 boards in about 1 minute on a 4-core i5 from 2012. When considering a move to a 32 hyperthread machine, we estimated a theoretical speedup of 8x, reducing the time to approximately 2.6 days for 30 billion boards.

plots_for_intro_blog_post.py

Analysis

Parallel Processing: Using multiple processes allowed us to efficiently explore the solution space, significantly reducing computation time.
Progress Tracking: Real-time progress bars provided valuable feedback on the number of solutions found, enhancing the user experience.
Data Storage: Immediate saving of board configurations ensured that progress was not lost, even in the event of a system failure.
Compression: We found that compressing the data reduced storage requirements significantly, making it feasible to handle large datasets.
Sampling: The ability to sample from the compressed tarball without full extraction was crucial for managing large data efficiently.

Conclusion

This experiment demonstrated the power of parallel processing in Python for generating large datasets. By leveraging multiprocessing, we efficiently explored a vast solution space, providing insights into the potential of Python for handling complex computational tasks.

If you're interested in exploring similar problems or optimizing computational tasks, consider using Python's multiprocessing capabilities to maximize performance and efficiency.

This post was inspired by a conversation with an Machine Learning assistant, highlighting the potential of Machine Learning in guiding and enhancing problem-solving processes.

Full conversation here:

efficient-battleship-board-generation-data-management-python.json

See flask-socketio-llm-completions for more Machine Learning tooling!

Contents

Background
Project Setup
Python Script
Data Compression and Sampling
Estimated Dataset Size
Artifact Description
Results
Analysis
Conclusion

Comparing Elixir and Phoenix Performance with a Community-Driven OpenAI Client

2024-08-30T17:26:00-04:00

As a developer exploring the capabilities of Elixir and Phoenix, I recently conducted a performance experiment using a community-driven OpenAI client. The goal was to evaluate how well Elixir, with its powerful concurrency model, handles high-volume API requests. The results were impressive, demonstrating Elixir's efficiency and robustness in managing concurrent tasks across multiple CPU cores.

Background

At CourseMojo, we are scaling an AI assistant teacher for public schools which provides real-time, intelligent responses to students' answers to open-ended questions. During our load testing, we identified that heavy CPU utilization was primarily due to calls to the OpenAI API. This prompted me to explore Elixir and Phoenix to see if they could offer a more efficient solution.

Project Setup

To set up the project, we used the following steps:

Create a New Elixir Project: Run the following command to create a new Elixir project and navigate into the directory:
```
mix local.hex
mix archive.install hex phx_new
mix phx.new openai_experiment --no-ecto
cd openai_experiment
```
Add Dependencies: Add the ex_openai library to your mix.exs file:
```
defp deps do
  [
    {:ex_openai, "~> 1.6"}
  ]
end
```
Run mix deps.get to fetch the dependencies.

Configure the Client: Set up your configuration in config/config.exs:

import Config

config :ex_openai,
  api_key: System.get_env("OPENAI_API_KEY")

# the ex_openai http client seems to use this.
config :hackney,
  pool_size: 1800,
  max_connections: 1800

Increase File Descriptor Limit: To handle more than 1000 concurrent connections, increase the file descriptor limit:
```
ulimit -n 4096
```
Implement the Concurrency Logic: We used Task.async_stream/3 to manage concurrency, allowing us to process tasks efficiently across multiple CPU cores.

Elixir Script

Here is the Elixir script lib/openai_experiment.ex used for the experiment:

defmodule OpenaiExperiment do
  alias ExOpenAI.Chat
  alias ExOpenAI.Components.ChatCompletionRequestUserMessage

  @max_retries 4
  @retry_delay 2000  # milliseconds
  @concurrency_limit 500

  def generate_chat_completion(index, attempt \\ 1) do
    msgs = [
      %ChatCompletionRequestUserMessage{role: :system, content: "You are a helpful assistant."},
      %ChatCompletionRequestUserMessage{role: :user, content: "What is the number #{index}?"}
    ]

    case Chat.create_chat_completion(msgs, "gpt-4o-mini") do
      {:ok, response} ->
        IO.inspect(response, label: "Response for number #{index}")
        {:ok, index}
      {:error, %HTTPoison.Error{reason: reason}} when attempt <= @max_retries ->
        IO.puts("HTTP error for number #{index}: #{inspect(reason)}, attempt #{attempt}")
        :timer.sleep(@retry_delay)
        generate_chat_completion(index, attempt + 1)
      {:error, reason} when attempt <= @max_retries ->
        IO.puts("Retrying number #{index} due to #{inspect(reason)}, attempt #{attempt}")
        :timer.sleep(@retry_delay)
        generate_chat_completion(index, attempt + 1)
      {:error, reason} ->
        IO.inspect(reason, label: "Final error for number #{index}")
        {:error, index, reason}
    end
  end

  def execute_chats_concurrently do
    {time, results} = :timer.tc(fn ->
      1..1800
      |> Task.async_stream(&generate_chat_completion/1, max_concurrency: @concurrency_limit, timeout: 30000)
      |> Enum.to_list()
    end)

    summarize_results(results)
    IO.puts("Total execution time: #{time / 1_000_000} seconds")
  end

  defp summarize_results(results) do
    successes = Enum.count(results, fn
      {:ok, {:ok, _index}} -> true
      _ -> false
    end)

    failures = Enum.count(results, fn
      {:ok, {:error, _index, _reason}} -> true
      _ -> false
    end)

    IO.puts("Summary:")
    IO.puts("Successful tasks: #{successes}")
    IO.puts("Failed tasks: #{failures}")
  end
end

# Run the test
#OpenaiExperiment.execute_chats_concurrently()

Compiling and Running the Example

Compile the Project: Ensure your project is compiled by running:
```
mix compile
```
Start the Elixir Interactive Shell: Launch the interactive Elixir shell with your project loaded:
```
iex -S mix
```
Run the Example: Once inside the interactive shell, execute the function to run the experiment:
```
OpenaiExperiment.execute_chats_concurrently()
```

Results

The results of the experiment were as follows:

Successful tasks: 1800
Failed tasks: 0
Total execution time: 23.436011 seconds

Analysis

The real time, which represents the total elapsed time, was significantly efficient for the Elixir script. Here are some key takeaways from the results:

Concurrency Handling: Elixir handled 1800 tasks efficiently, leveraging its lightweight process model with a concurrency limit of 500.
Multi-Core Utilization: Elixir utilized all 4 CPU cores, demonstrating its ability to efficiently distribute tasks across available resources.
Built-in Retries: The script included a retry mechanism to handle transient errors, ensuring robustness in task execution.

This experiment demonstrated that Elixir, with its concurrency model, is a strong contender for handling high-volume, concurrent API requests using a community-driven OpenAI client. Elixir's efficient concurrency handling makes it a superior choice for scenarios where performance and speed are critical.

If you’re working on a project that involves extensive use of the OpenAI API or any other high-volume API interactions, consider leveraging Elixir to maximize performance and efficiency. The results of this experiment highlight the potential gains in speed and responsiveness that can be achieved with the right choice of technology.

It's worth noting that the OpenAI client used in this experiment is community-driven, not official, which also impacts performance. Additionally, keep an eye on developments in Elixir and Phoenix, as they continue to evolve and improve.

This post was written with the help of GPT-4 using the Elixir ex_openai library.

Contents

Background
Project Setup
Elixir Script
Compiling and Running the Example
Results
Analysis

Comparing Node.js and Python Performance with the Official OpenAI Client

2024-05-28T08:49:00-04:00

As a seasoned Python developer, I've always appreciated the versatility and power of Python. I found out today that when it comes to handling high-volume API requests, CPU performance can be a bottleneck and cause stability issues if not throttled to around 80% utilization. I conducted a performance comparison between Node.js and Python using the official OpenAI client, and the results were quite revealing. Node.js outperformes Python in this specific use case.

Background

At CourseMojo, we are scaling an Machine Learning assistant teacher for public schools which provides real-time, intelligent responses to students' answers to open ended questions. During our load testing in my sandbox environment, we identified that heavy CPU utilization was primarily due to calls to the OpenAI API. This prompted me to explore different programming languages to see if the problem existed elsewhere.

The Experiment

To compare the performance of Node.js and Python, I set up two scripts that interact with the OpenAI API. The goal was to measure the time taken to process a large number of tasks concurrently. Here’s a brief overview of the setup:

Node.js Script: Processed 1800 tasks, with a concurrency limit of 200 tasks at a time.
Python Script: Processed 300 tasks, with a concurrency limit of 20 tasks at a time.

Both scripts were designed to send chat messages to the OpenAI API and receive responses. The tasks were structured to simulate real-world usage, where multiple requests are sent concurrently in the background with a single process running on a single CPU core. This mocks our production application.

Node.js Script

Here is the Node.js script used for the experiment, we use p-limit to throttle the concurrency to healthy CPU levels for background work to continue in process:

import OpenAI from 'openai';
import pLimit from 'p-limit';

const openai = new OpenAI({ apiKey: process.env['OPENAI_API_KEY'] });

async function generateChatCompletion(messages) {
    const stream = await openai.chat.completions.create({
        messages,
        model: 'gpt-3.5-turbo',
        stream: true,
    });

    let resultText = '';
    for await (const chunk of stream) {
        if (chunk.choices && chunk.choices.length > 0 && chunk.choices[0].delta && chunk.choices[0].delta.content) {
            resultText += chunk.choices[0].delta.content;
        }
    }
    return resultText;
}

// Generate an array of chats
const chats = Array.from({ length: 1800 }, (_, index) => ({
    messages: [
        { role: 'system', content: 'You are a helpful assistant.' },
        { role: 'user', content: `What is the number ${index + 1}?` }
    ]
}));

// Function to execute chats with controlled concurrency
async function executeChatsConcurrently() {
    const limit = pLimit(200); // limit the number of concurrent API calls.

    const promises = chats.map(chat => limit(() => generateChatCompletion(chat.messages)));
    const results = await Promise.all(promises);

    // Print the completions
    results.forEach((result, index) => {
        console.log(`Chat Completion Result ${index + 1}:`);
        console.log(result);
    });
}

executeChatsConcurrently().catch(console.error);

Python Script

Here is the Python script used for the experiment, we use a semaphore to throttle the concurrency to healthy CPU levels for background work to continue in process:

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def generate_text_async(messages, sem):
    async with sem:
        stream = await async_client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=messages,
            stream=True,
        )
        generated_text = ""
        async for chunk in stream:
            if chunk.choices[0].delta.content is not None:
                generated_text += chunk.choices[0].delta.content
        return f"\n\nGenerated text for '{messages}':\n{generated_text}"

# Generate a list of chats
chats = [
    {
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": f"What is the number {index + 1}?"}
        ]
    }
    for index in range(300)
]

async def main():
    sem = asyncio.Semaphore(20)  # limit concurrent tasks.
    tasks = [generate_text_async(chat["messages"], sem) for chat in chats]
    results = await asyncio.gather(*tasks)
    for result in results:
        print(result)

asyncio.run(main())

Results

The results of the experiment were as follows:

Node.js Script:

real    0m10.659s
user    0m8.277s
sys     0m0.544s

Python Script:

real    0m15.991s
user    0m10.041s
sys     0m0.229s

Analysis

The real time, which represents the total elapsed time, was significantly lower for the Node.js script compared to the Python script. Here are some key takeaways from the results:

Concurrency Handling: Node.js handled 1800 tasks with a concurrency limit of 200, while Python handled 300 tasks with a concurrency limit of 20. Despite the higher number of tasks and concurrency in Node.js, it completed the tasks faster.
Event-Driven Architecture: Node.js’s event-driven, non-blocking I/O model is highly efficient for handling multiple concurrent tasks. This architecture allows Node.js to manage a large number of simultaneous connections with minimal overhead, making it ideal for high-volume API interactions.
CPU Utilization: Both scripts showed healthy CPU core usage, but Node.js managed to utilize the CPU more efficiently, resulting in faster completion times.
User and System Time: The user and sys times indicate the CPU time spent in user-mode and kernel-mode, respectively. Node.js showed a higher user time but a lower sys time compared to Python, suggesting that Node.js was more efficient in executing user-level code.

Conclusion

The experiment clearly demonstrated that Node.js outperforms Python in handling high-volume, concurrent API requests using the official OpenAI client. The event-driven architecture of Node.js, combined with its efficient concurrency handling, makes it a superior choice for scenarios where performance and speed are critical.

While Python remains a powerful and versatile language, especially in the realm of data science and machine learning, Node.js proves to be a better option for high-performance, real-time applications that require efficient handling of multiple concurrent tasks.

If you’re working on a project that involves extensive use of the OpenAI API or any other high-volume API interactions, consider leveraging Node.js to maximize performance and efficiency. The results of this experiment highlight the potential gains in speed and responsiveness that can be achieved with the right choice of technology.

Final Thoughts

Choosing the right tool for the job is crucial in software development. While both Node.js and Python have their strengths, understanding their performance characteristics can help you make informed decisions that align with your project’s requirements. In the case of high-volume API interactions, Node.js stands out as the clear winner, offering superior performance and efficiency.

It's worth noting that the OpenAI client uses httpx for Python and node-fetch for Node.js by default. This choice of libraries also impacts performance. Additionally, there is hope for Python's future performance improvements, such as the potential removal of the Global Interpreter Lock (GIL) in later versions, which could significantly enhance Python's concurrency capabilities.

For now, if performance is a critical factor in your project, especially for handling numerous concurrent API requests, Node.js is the way to go. But keep an eye on Python's developments, as it continues to evolve and improve.

This post was written with the help of gpt-4 using https://github.com/russellballestrini/flask-socketio-llm-completions

Contents

Background
The Experiment
Node.js Script
Python Script
Results
Analysis
Conclusion
Final Thoughts

Integrating OpenAI with Dry's Sample Chat Game

2024-02-17T14:11:00-05:00

This tutorial demonstrates enhancing Dry's Sample Chat Game by integrating OpenAI's language models, enabling the game to provide intelligent, Machine Learning-driven responses to user queries. Dry, the successor to Urho3D, offers a comprehensive framework for developing 2D and 3D games. Leveraging LLMs within the Dry engine opens up new possibilities.

Background

The Sample Chat game in Dry provides basic messaging functionality where multiple copies of the game client can connect to a server to communicate via text messages. Our aim is to extend this functionality to include responses from OpenAI's language models when messages contain specific triggers, such as "gpt-3".

Prerequisites

Ensure you have the following before starting:

An OpenAI API key.
A configured Dry build environment.
Some familiarity with build tooling or at least the ability to copy and paste commands into the terminal.

Step-by-Step Integration

Follow these steps to integrate OpenAI with the Sample Chat game:

Obtain the OpenAI C++ Client

Before modifying the chat game, you need to download the necessary files from the OpenAI C++ client repository. Use the following commands to create a directory for these files and download them:

cd ~/git/dry
mkdir -p Source/ThirdParty/openai
mkdir -p Source/ThirdParty/openai/nlohmann
cd Source/ThirdParty/openai
wget https://raw.githubusercontent.com/olrea/openai-cpp/main/include/openai/openai.hpp
cd nlohmann
wget https://raw.githubusercontent.com/olrea/openai-cpp/main/include/openai/nlohmann/json.hpp

I also found the need to modify the client to code slightly for the JSON import, pardon me if this is ignorant:

// #include <nlohmann/json.hpp>  // nlohmann/json
#include <Dry/ThirdParty/openai/nlohmann/json.hpp>

The client requires cURL development files, on Fedora:

sudo dnf install libcurl-devel

Update the CMake Configuration

Note: update PROJECT_SOURCE_DIR with your file path.

Apply the following diff to CMakeLists.txt to include the OpenAI C++ client as well as cURL in your dry project:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index a1eff04..0f32bc7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -188,6 +188,20 @@ else ()
 endif ()
 file (MAKE_DIRECTORY ${THIRD_PARTY_INCLUDE_DIR})

+# Find the cURL library
+find_package(CURL REQUIRED)
+
+# Globally link cURL to all targets
+link_libraries(${CURL_LIBRARIES})
+
+# Assuming this is set to your project's root directory
+set(PROJECT_SOURCE_DIR /home/fox/git/dry)
+
+# Create a symbolic link for openai
+execute_process(COMMAND ${CMAKE_COMMAND} -E create_symlink
+                ${PROJECT_SOURCE_DIR}/Source/ThirdParty/openai
+                ${CMAKE_BINARY_DIR}/include/Dry/ThirdParty/openai)
+

Modify Sample/16_Chat/chat.cpp file

Implement the changes outlined in the diffs below for chat.cpp:

diff --git a/Source/Samples/16_Chat/Chat.cpp b/Source/Samples/16_Chat/Chat.cpp
index ee7c2b7..9d0e454 100644
--- a/Source/Samples/16_Chat/Chat.cpp
+++ b/Source/Samples/16_Chat/Chat.cpp
@@ -41,6 +41,7 @@
 #include <Dry/UI/Text.h>
 #include <Dry/UI/UI.h>
 #include <Dry/UI/UIEvents.h>
+#include <Dry/ThirdParty/openai/openai.hpp>

 #include "Chat.h"

@@ -201,16 +202,58 @@ void Chat::HandleSend(StringHash /*eventType*/, VariantMap& eventData)

     if (serverConnection)
     {
-        // A VectorBuffer object is convenient for constructing a message to send
-        VectorBuffer msg;
-        msg.WriteString(text);
-        // Send the chat message as in-order and reliable
-        serverConnection->SendMessage(MSG_CHAT, true, true, msg);
+        // Check if the message contains "gpt-3"
+        if (text.Contains("gpt-3"))
+        {
+            // Initialize OpenAI
+            openai::start();
+
+            // Correctly construct the JSON payload as a std::string
+            std::string payload = std::string(R"({"model": "gpt-3.5-turbo", "messages":[{"role":"user", "content":")") + text.CString() + std::string(R"("}], "max_tokens": 600, "temperature": 0.5})");
+            nlohmann::json gptResponse;
+            try {
+                // Parse the payload to JSON and make the API call
+                gptResponse = openai::chat().create(nlohmann::json::parse(payload));
+            } catch (const std::exception& e) {
+                // Handle JSON parsing errors or API call failures
+                std::cerr << "Error making API call or parsing response: " << e.what() << '\n';
+                return;
+            }
+
+            std::string responseText;
+            try {
+                // Extract the response text from the JSON response
+                responseText = gptResponse["choices"][0]["message"]["content"].get<std::string>();
+            } catch (const std::exception& e) {
+                // Handle errors in accessing the response content
+                std::cerr << "Error extracting response text: " << e.what() << '\n';
+                return;
+            }
+
+            // send the user's message to gpt-3 to the server.
+            VectorBuffer msg1;
+            msg1.WriteString(text);
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg1);
+
+            // send the gpt-3 completion to the server.
+            VectorBuffer msg2;
+            msg2.WriteString(String(responseText.c_str()));
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg2);
+        }
+        else
+        {
+            // Normal chat message handling
+            VectorBuffer msg;
+            msg.WriteString(text);
+            serverConnection->SendMessage(MSG_CHAT, true, true, msg);
+        }
+
         // Empty the text edit after sending
         textEdit_->SetText(String::EMPTY);
     }
 }

Prepare the Build Environment

Before running CMake, ensure that any previous build configurations are cleared to avoid conflicts. This might involve deleting the CMake cache file:
```
rm CMakeCache.txt # If exists
```
Then, generate the build configuration:
```
cmake .. -DCMAKE_BUILD_TYPE=Debug -DRY_64BIT=1
```
Build the Chat Game

Compile the Sample Chat game with the newly integrated OpenAI C++ client:
```
make 16_Chat/fast
```

Testing the Integration

After applying the changes and compiling the game, ensure your OpenAI API key is available to the game:

export OPENAI_API_KEY='your_openai_api_key_here'

Run the Sample Chat game and try sending a message containing "gpt-3". You should see an intelligent response generated by OpenAI's language model.

./bin/16_Chat

Remember you'll need at least one instance of the game running as server mode before a client can interact with the LLM.

That means you need to run ./bin/16_Chat at least twice in two different windows to see the experience.

What's Next?

You've now integrated OpenAI into Dry's Sample Chat game, enhancing it with Machine Learning-driven conversational capabilities. Explore further by customizing triggers, integrating other models using the same OpenAI client, or expanding the game's features.

I think personally I will try to get the client communicating with vllm likely running openchat.

Happy coding, and enjoy bringing LLM capabilities to your Dry games!

Contents

Background
Prerequisites
Step-by-Step Integration
Testing the Integration
What's Next?

Optimizing Memory Management for Plausible Docker on Ubuntu

2024-02-05T17:40:00-05:00

Running Docker containers on a server with limited memory can lead to out-of-memory (OOM) issues, which can disrupt services and lead to downtime. This guide will show you how to increase swap space on an Ubuntu server to provide a buffer against OOM errors, using a real-world example from a server running multiple Docker containers.

Background

Consider a typical scenario where an Ubuntu server is running several Docker containers:

fox@analytics:~$ sudo docker ps
CONTAINER ID        IMAGE                                           PORTS                          NAMES
7677a77e5606        plausible/analytics:v2.0                        0.0.0.0:8000->8000/tcp         plausible-plausible-1
0ea79b7d0c03        clickhouse/clickhouse-server:23.3.7.5-alpine    8123/tcp, 9000/tcp, 9009/tcp   plausible-plausible_events_db-1
33f6e8a30da4        postgres:14-alpine                              5432/tcp                       plausible-plausible_db-1
40400c725d7c        bytemark/smtp                                   25/tcp                         plausible-mail-1

fox@analytics:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:            981         693          62          70         225          66
Swap:             0           0           0

In this example, the server has less than 1 GB of RAM and no swap space configured. This configuration is prone to OOM errors, especially when the containers require more memory than what is available. In my case the service stayed online but the configuration management service salt-minion was unable to be reached to deploy new TLS certificates, the cert expired and prevented clients from sending metrics.

Creating and Enabling Swap Space

To prevent OOM errors, we'll create a swap file. This script automates the process, ensuring that your system has additional virtual memory to handle peak loads.

Save the script as setup_swap.sh and execute it on your server:

#!/bin/bash

# Size of the swap file
SWAP_SIZE=1G
SWAP_FILE=/swapfile
SWAPPINESS=10
CACHE_PRESSURE=50

# Create a swap file
fallocate -l $SWAP_SIZE $SWAP_FILE || dd if=/dev/zero of=$SWAP_FILE bs=1024 count=1048576

# Secure swap file permissions
chmod 600 $SWAP_FILE

# Set up a Linux swap area
mkswap $SWAP_FILE

# Enable the swap file
swapon $SWAP_FILE

# Make the swap file permanent
echo "$SWAP_FILE none swap sw 0 0" | tee -a /etc/fstab

# Set the swappiness value
sysctl vm.swappiness=$SWAPPINESS
echo "vm.swappiness=$SWAPPINESS" | tee -a /etc/sysctl.conf

# Set the cache pressure value
sysctl vm.vfs_cache_pressure=$CACHE_PRESSURE
echo "vm.vfs_cache_pressure=$CACHE_PRESSURE" | tee -a /etc/sysctl.conf

# Output the results
echo "Swap file created and enabled:"
swapon --show
echo "Swappiness set to $SWAPPINESS and cache pressure set to $CACHE_PRESSURE."

This script will create a 1 GB swap file, configure the system to use it, and adjust kernel parameters to optimize memory usage.

Understanding Swappiness and Cache Pressure

The swappiness parameter influences how often the system uses swap space. A value of 10 encourages the system to keep processes in RAM, resorting to swap only when necessary.

The vfs_cache_pressure setting determines how aggressively the kernel reclaims memory from the cache. A value of 50 provides a balance between reclaiming memory and maintaining cache for quick file access.

Monitoring Your System

After increasing the swap space, monitor your system's memory usage with:

free -m

This will help you understand if the swap space is sufficient or if further adjustments are needed.

What's Next?

By adding swap space and tuning kernel parameters, you've bolstered your server's ability to handle memory-intensive Docker containers. However, swap is not a replacement for physical RAM. If your server consistently uses a lot of swap, consider upgrading the RAM for better performance.

Stay proactive in managing your server's resources to ensure uninterrupted service for your Dockerized applications. Happy hosting!

Contents

Background
Creating and Enabling Swap Space
Understanding Swappiness and Cache Pressure
Monitoring Your System
What's Next?

Understanding Pause Functionality in LucKey Park Built on Dry Engine

2024-02-05T08:57:00-05:00

In today's post, we're examining the pause functionality of the LucKey Park game code, which is built on the 'Dry' engine. We'll uncover how the game's status is managed and toggled, especially in response to the cancel button (ESC key), and provide insights into the game's input handling and state management.

Check out screenshots and videos of LucKey Park on itch.io

Understanding the Game's Status Management

The 'Park' game uses an enumeration GameStatus with values like GS_MAIN, GS_PLAY, GS_PAUSED, and GS_MODAL to represent the game's current state. The Game class in game.h maintains a status_ variable to track this state.

enum GameStatus{ GS_MAIN, GS_PLAY, GS_PAUSED, GS_MODAL };

class Game: public Object
{
    // ...
    private:
        GameStatus status_;
};

The Game class provides methods to get and set this status, as well as a TogglePause() method that switches between GS_PLAY and GS_PAUSED.

Binding the Cancel Action

In inputmaster.cpp, the InputMaster class binds the IA_CANCEL action to the ESC key. This setup is crucial for handling the pause functionality when the player presses ESC.

Bind(IA_CANCEL, KEY_ESCAPE);

Handling the Cancel Action

When the IA_CANCEL action is triggered, the HandleAction method in InputMaster responds accordingly. If no tool is active, it calls gui->ShowMainMenu(), which indirectly pauses the game by changing the game's status.

case IA_CANCEL:
    if (tool) sceneCursor->SetTool(nullptr);
    else gui->ShowMainMenu();
    break;

The Main Update Loop Controlled by Dry Engine

The Dry engine, like its predecessor Urho3D, manages the main game loop internally. Developers hook into this loop by subscribing to the E_UPDATE event, which the engine dispatches every frame. This event allows developers to perform per-frame updates to their game logic.

While the main update loop is not explicitly shown in the provided snippets, it's typically where the game checks the status_ variable each frame. If the status is GS_PAUSED or GS_MODAL, the loop skips updating the game world, pausing the game's logic.

End-to-End Pause Functionality

From the moment the player presses the ESC key to the game entering a paused state, the flow is as follows:

The ESC key is pressed.
InputMaster detects the IA_CANCEL action.
HandleAction calls gui->ShowMainMenu() if no tool is active.
ShowMainMenu() sets the game's status to GS_MODAL.
The main update loop, controlled by the Dry engine, respects this status and pauses the game.

This exploration has provided a clear understanding of how the pause functionality is implemented in Lucky Park using the Dry engine. It's a great example of how game state management and input handling work together to create responsive gameplay.

Stay tuned for more deep dives into game development.

Contents

Understanding the Game's Status Management

Installing YaCy on Ubuntu

2024-02-04T11:02:00-05:00

This guide will walk you through installing YaCy on Ubuntu.

By default YaCy is configured to bind to 0.0.0.0 but it's admin interface is only accessible by default to a white list which includes localhost and 127.0.0.1, since my install is headless on a remote Ubuntu server, I access the admin interface remotely via an SSH tunnel.

Installing YaCy

run this script in a terminal, install_yacy.sh:

#!/bin/bash

# Update package lists
sudo apt-get update

# Install Java Development Kit (JDK)
sudo apt-get install -y openjdk-21-jdk

# Check if Java is correctly installed
java -version

# Download YaCy
wget https://download.yacy.net/yacy_v1.924_20210209_10069.tar.gz

# Extract the downloaded file
tar xfz yacy_v1.924_20210209_10069.tar.gz

# Change to the extracted directory
cd yacy

# Start YaCy
./startYACY.sh

Accessing the Admin Interface Remotely via an SSH Tunnel

You can create an SSH tunnel to your remote YaCy server, for example yacy.foxhop.net so that when you access localhost:8090, it tunnels to the remote host.

Open a terminal on your local machine and run the following command:

ssh -L 8090:localhost:8090 user@yacy.foxhop.net

Replace user with your username on the remote host. Now, when you access localhost:8090 on your local machine, it will be tunneled to the remote host.

Securing YaCy Portal and admin access with SSL/TLS

In the early days of YaCy, encryption was not as prevalent as it is today. However, securing your YaCy instance with SSL/TLS is essential for modern web safety standards. Here's how to enable SSL/TLS encryption for your YaCy server:

Generate a keystore using the keytool command:
```
keytool -keystore mySrvKeystore -genkey -keyalg RSA -alias yacy.foxhop.net
```
This command creates a keystore file named mySrvKeystore.
Move the mySrvKeystore file to the DATA/SETTINGS/ directory in your YaCy installation:
```
mv mySrvKeystore /path/to/YaCy/DATA/SETTINGS/
```
Edit the yacy.conf file to configure YaCy to use the new keystore:
```
vim /path/to/YaCy/DATA/SETTINGS/yacy.conf
```
Add or modify the following lines:
```
keyStore=DATA/SETTINGS/mySrvKeystore
keyStorePassword=YourKeystorePassword
```
Replace YourKeystorePassword with the password you chose when you created the keystore with the keytool command.

Restart YaCy to apply the SSL/TLS settings:

/path/to/YaCy/stopYACY.sh
/path/to/YaCy/startYACY.sh

Now, you can access the YaCy admin interface securely via https://localhost:8443. By default, YaCy listens on port 8443 for HTTPS, but this can be changed in the admin console, as was done in this case to use port 8091 so that https://localhost:8091 works instead. Ensure that HTTP remains on port 8090 for DHT network access by peers.

What's next?

With SSL/TLS enabled, it's time to start a local crawl and consider sharing the results with the YaCy network. To do this, you may need to open port 8090 on your router and forward it to your YaCy host. Before making changes to your network configuration, verify your peer mode status in the YaCy admin interface to understand your node's role in the network.

Sharing your crawl index with the network allows for a distributed scraping effort, meaning that the data you collect can benefit all users of YaCy, not just your local instance. This is the essence of the Distributed Hash Table (DHT) that underpins YaCy's decentralized architecture.

Continue to explore the capabilities of your YaCy server, and remember to update your documentation to assist others in their journey toward a more open and collaborative internet.`

Contents

Installing YaCy
Accessing the Admin Interface Remotely via an SSH Tunnel
Securing YaCy Portal and admin access with SSL/TLS
What's next?

Building 'Dry' and 'Park' from Source on Fedora Linux

2024-02-03T09:19:00-05:00

In this guide, we'll explore how to compile the 'Dry' game engine and the 'Park' game from source on Fedora Linux with debugging enabled. This process will give you a deeper understanding of the inner workings of game development and the satisfaction of running a game you built yourself.

For Ubuntu, go here for pre-compiled game:

https://luckeyproductions.itch.io/park

Prerequisites

Before diving in, ensure you have the necessary tools and libraries installed:

Git
CMake
GNU Make
GCC or Clang
X11 and audio system development libraries

Install these on Fedora with:

sudo dnf groupinstall "Development Tools"
sudo dnf install cmake git gcc-c++ libX11-devel libXcursor-devel libXinerama-devel libXi-devel libXrandr-devel libXrender-devel libXScrnSaver-devel libXxf86vm-devel pulseaudio-libs-devel nas-libs-devel qt5-qtbase-devel

Cloning the Repositories

Clone the 'Dry' and 'Park' repositories using Git:

cd ~/git
git clone https://gitlab.com/luckeyproductions/dry.git
git clone https://gitlab.com/luckeyproductions/games/park.git

Building Dry with Debugging Enabled

Navigate to the 'dry' directory and prepare the build environment:

cd dry
mkdir build && cd build

Configure the build with CMake and enable debugging:

cmake .. -DCMAKE_BUILD_TYPE=Debug -DDRY_64BIT=1

Compile the Dry library with debug symbols:

make

Building Park with Debugging Enabled

In the 'Park' project, modify the Park.pro file to add the -g -O0 flags for debugging, for example:

QMAKE_CXXFLAGS += -std=c++17 -g -O0

Then set the DRY_HOME environment variable to the path of your compiled Dry library:

export DRY_HOME=/home/fox/git/dry/build

Navigate to the 'park' directory and compile the game:

cd /home/fox/git/park
mkdir build && cd build
qmake ../Park.pro
make
cp -r ../Resources .

Running Park

After a successful build, run the Park executable located in the build directory:

./park

Analyzing Core Dumps on Fedora

If your application crashes, Fedora can generate core dumps, which are snapshots of the program's state at the time of the crash. These can be invaluable for debugging.

List recent core dumps with:

coredumpctl list

To analyze a specific core dump, use gdb:

gdb /path/to/executable /path/to/coredump

For example:

gdb /home/fox/git/park/Park/park core.291993

Once in gdb, use the bt command to print a backtrace:

(gdb) bt

This will show you the call stack at the time of the crash, which can help pinpoint the source of the problem.

Happy building and debugging!

Contents

Prerequisites
Cloning the Repositories
Building Dry with Debugging Enabled
Building Park with Debugging Enabled
Running Park
Analyzing Core Dumps on Fedora

Ubuntu 22.04 Letsencrypt Docker Hints

2022-05-20T20:10:00-04:00

letsencrypt certbot is now installable via snap (the deb apt repository is no longer maintained).

alternatively you can use certbot via docker if you plan to use the certonly mode.

I did run into some issues & I will document my workarounds here:

domains=(
    example.com
    shop.example.com
)

for domain in ${domains[*]}; do
    echo "certifying: $domain"

    IFS='.' read -r -a domain_parts <<< "$domain"

    domain_parts_length=${#domain_parts[@]}

    if [ "$domain_parts_length" -eq 2 ]
    then
        # https://eff-certbot.readthedocs.io/en/stable/install.html#running-with-docker
        docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/var/log/letsencrypt:/var/log/letsencrypt" \
            -v "/www:/www" \
            certbot/certbot certonly -v --renew-by-default --webroot -w /www -d $domain -d www.$domain
    else
        # https://eff-certbot.readthedocs.io/en/stable/install.html#running-with-docker
        docker run -it --rm --name certbot \
            -v "/etc/letsencrypt:/etc/letsencrypt" \
            -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
            -v "/var/log/letsencrypt:/var/log/letsencrypt" \
            -v "/www:/www" \
            certbot/certbot certonly -v --renew-by-default --webroot -w /www -d $domain
    fi

    # copy certificate links to a known file path and extention.
    cp /etc/letsencrypt/live/$domain/fullchain.pem /etc/letsencrypt/live/$domain/crt.crt
    cp /etc/letsencrypt/live/$domain/privkey.pem /etc/letsencrypt/live/$domain/key.key
done

# use minionfs to stage all certificates onto the salt master.
salt-call cp.push_dir "/etc/letsencrypt/live/" glob='*.crt'
salt-call cp.push_dir "/etc/letsencrypt/live/" glob='*.key

So the key is the -v mounts, I needed one for my webroot of /www & one for logs.

This is different or not assumed in the official guide notes.

Additionally on Ubuntu 22.04 LTS in Linode, I was not any to use the -v mounts until I ran these commands:

sudo su - root

mkdir /sys/fs/cgroup/systemd
mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

The mount command never persists across reboots so you will want the following /etc/fstab entry:

Brian Amedro ended up modifying grub to avoid loading the certain systemd subsystem which were found to cause us the trouble:

https://github.com/docker/for-linux/issues/219#issuecomment-817318014

For now i will place the mkdir & mount commands into the renew script since the fstab solution doesn't work

because the directory doesn't exist, gets deleted on each reboot so it isn't present during boot mount time.

/etc/fstab

cgroup    /sys/fs/cgroup/systemd    cgroup    defaults

Uncle Drops off server with note asking u to cl0ne ubuntu22 kvm

2022-05-09T13:14:00-04:00

Imagine your uncle just dropped off a TrueNAS Core server with root credentials, & a preconfigured IP Address in the subnet space of 192.168.1.1/24.

The note also highly suggests you to create your desired arch with KVM by making a cl0ne of ubuntu22, into separate VMs for different purposes.

Kind of open ended when to end a note, but whatever, you have nothing else to do this __________ so you crack open the card board box & pull out an enormous 2u blade server! (this is like an FF8 Gunsword cutscene moment). We say enourmous not because it's any different in size than any other 2u, but this thing is extra heavy, I mean real heavy, 4 mechanical drives but fits up to 12!

The 4 hard drives, configured in ZFS mirrors, labeled "downloads" & "personal".

upon booting the The TrueNAS Core server graphical user interface is connected over HTTPS as a web application, available from the IP address on the sticky notes!

The UI seems to show the baremetal system has 128G of memory, most is free! Additionally it has 2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz with a 1% Avg Usage on 32 threads!

You clone a KVM guest from ubuntu22 & name it ______________________________.

Now you will need to configure the new cloned guest via VNC in order to change the following list of configurations. The ubuntu22 base image is the vanilla defaults of Ubuntu 22.04 LTS server without any Snap (uncle likes flatpak if anything) so this means the server is "headless" there isn't a graphical user interface.

You'll need to configure:

hostname vim /etc/hostname:
```
cammy.foxhop.net
```

network & dns vim /etc/netplan/00-installer-config.yaml:

# run this command after making changes!
# sudo netplan --debug apply
network:
  ethernets:
    enp0s4:
      addresses:
        - 192.168.1.64/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses:
          ### CloudFlare.
          #- 1.1.1.1
          ### Google DNS.
          #- 8.8.8.8
          #- 8.8.4.4
          ### OpenDNS.
          #- 208.67.220.220
          #- 208.67.222.222
          - 208.67.222.220
          ### Quad9.
          - 9.9.9.9
        search:
          - foxhop.net
  version: 2

You remembered to try to document steps needed for next time a person needs to clone the image we will have a checklist to follow so no steps get missed!

What is that checklist for you?

Do you bootstrap configuration management next? Or maybe some in house remote execution? Do you outsource your admin hacker tooling?

Ok, so the next day you wake up & decide to try out SaltStack configuration management, so that means you will want to clone ubuntu22 kvm into a guest named master (short for salt-master, salt consistently crashes [me shrugs]).

Anyways, you do the tricks above to configure netplan (networking & DNS) & then go about installing salt-master service. The -M flag signals to install both salt-minion & salt-master:

wget -O - https://bootstrap.saltstack.com | sudo sh -s -- stable -M;

Of course this doesn't always work if for example you are on a very new Ubuntu LTS (which you are) no fear, you remembered uncle documented a similar snag in a github comment that said you adapted to also install the salt-master daemon, since this is the salt master vm.

git clone https://github.com/saltstack/salt-bootstrap.git
cd salt-bootstrap
bash salt-bootstrap.sh -M

By now you are watching the growing list of hosts using ubuntu22's SSH host key...

[ ] https://blog.g3rt.nl/regenerate-ssh-host-keys.html

You do this to fix salt host & start to consider ways to deal with this in the future, should likely be one of the first steps or maybe we could run this on ubuntu22? You decide to write this down as a (rabbit)(hole) for another day.

Dual booting Ubuntu 22.04 LTS?

Did you know grub 2 OS Prober was recently disabled by default?

If plan to dual boot with windows or any other OS, try these settings:

[x] https://www.solaris-cookbook.eu/linux/linux-ubuntu/ubuntu-22-04-fix-grub-dual-boot-with-windows/

sudo vi /etc/default/grub

GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_OS_PROBER=false

applied but it didn't seem to work...

While doing this work you remember you needed to back up your gpg keys (and verify the restore process!):

[] https://www.jwillikers.com/backup-and-restore-a-gpg-key

& also make sure you can access pass from multiple machines, using git!

[] https://www.passwordstore.org/

and get multiple machines pushing & pulling from the same remote origin this way it is safer to loose a node on the "cluster",

for full transparency, this is what I did to look around & export / import

[fox@blanka ~]$ gpg --list-secret-keys --keyid-format LONG

/home/fox/.gnupg/pubring.gpg
----------------------------
sec   rsa2048/23CDA6102BFB3D7D 2014-06-23 [SC]
      4ABE744F1FDAF12C78E2E5D923CDA6102BFB3D7D
uid                 [ultimate] Russell Ballestrini (Personal) <russell@ballestrini.net>
ssb   rsa2048/1E6F09A2613E8133 2014-06-23 [E]

[fox@blanka ~]$ gpg -o russell-at-ballestrini-dot-net.gpg --export-options backup --export-secret-keys russell@ballestrini.net

[fox@blanka ~]$ ll russell-at-ballestrini-dot-net.gpg
-rw-------. 1 fox fox 2645 May 11 18:59 russell-at-ballestrini-dot-net.gpg

[fox@blanka ~]$ file russell-at-ballestrini-dot-net.gpg
russell-at-ballestrini-dot-net.gpg: OpenPGP Secret Key Version 4, Created Mon Jun 23 01:44:34 2014, RSA (Encrypt or Sign, 2048 bits)

[fox@blanka ~]$ scp russell-at-ballestrini-dot-net.gpg fox@akuma.foxhop.net:/home/fox/russell-at-ballestrini-dot-net.gpg
russell-at-ballestrini-dot-net.gpg

you think this will allow you to copy the private key file around to various computers via SSH and import the key using this:

gpg --import-options restore --import russell-at-ballestrini-dot-net.gpg

you admit whenever working with pgp or gpg makes you feel squeemish, but a key is a key right? ...

should not treat any differently than a private SSH key.

You will however want to trust this key completely with your whole heart & soul & sole so you modify the newly imported key to trust it completely and ultimately which just so happens to be a 5 in this tool's universe!

so you follow these prompts:

gpg --edit-key russell@ballestrini.net

gpg> trust

and choose 5!

gpg> quit

Then finally you restored your repo to a fresh new computer by using:

fox@play:~$ git clone ssh://fox@akuma.foxhop.net:/home/fox/git/pass.git .password-store

with the private gpg key in place & the repo following the shared remote, we now have our password store sharded across workstation nodes!

on ubuntu desktop network manager hijacks /etc/netplan but also makes it impossible to configure the search DNS settings from the GUI, you track down a blog post which helps you!

sudo nmcli con mod 'Wired connection 1' ipv4.dns-search "foxhop.net"

your only hope is this setting persists across reboots.

Vertically Scaling GitLab Server For As Cheap As Possible

2022-01-05T19:13:00-05:00

Today's essay acts as a power-up love story for the underdog.

A living document & quickstart for:

bootstrappers
small business under 99 employees
solo ops or devs
entrepreneurs
hackers
tinkerers

You also deserve a quick start to a GitLab Server power house!

Regardless of my intended audience, this strategy should scale to 500-1,000 concurrent engineers on the smallest of the instance classes which satisfy the current GitLab Server "minimum requirements".

https://docs.gitlab.com/ee/install/requirements.html

Why GitLab Server?

GitLab is open source (not black box)
GitLab CI is a game changer
GitLab CI is extendable to any workflow
GitLab Server wants 4G of memory

Expected Results

This guide prescribes the following setup:

A Physical or Virtual Machine with at least 4G of RAM
Docker installed
Docker container started via docker-compose to manage a monolithic Omnibus installation of GitLab Server
A Cronjob to backup GitLab Server to /tmp
A Cronjob to collect Gitlab Server backup tarball & /etc/gitlab/gitlab-secrets.json
A strong recommendation to test the complete backup and recovery process at least once to prepare for when times get rough.

GitLab Server Omnibus In Docker

The installation & configuration is documented using SaltStack to make it easy to spin up GitLab Servers.

To install SaltStack on the new host dedicated for GitLab Server, Run the following:

wget -O - https://bootstrap.saltstack.com | sudo sh -s -- stable;
echo "master: salt.example.com" >> /etc/salt/minion.d/custom.conf;
sudo service salt-minion restart

On the instance designated as salt-master, accept the new salt-minion key:

sudo salt-key -a git2.unturf.com
The following keys are going to be accepted:
Unaccepted Keys:
Git2.unturf.com
Proceed? [n/Y] y

and make sure communication is established:

sudo salt "git2*" test.ping
git2.unturf.com:
    True

and run a highstate to configure that new host instance:

sudo salt "git2*" state.highstate

Once the command returns, assuming we had no errors or failures we have docker installed and an Omnibus Gitlab Server booted inside a container.

Summary for git2.unturf.com
--------------
Succeeded: 104 (changed=69)
Failed:      0
--------------
Total states run:     104
Total run time:   194.431 s

The GitLab Server's internal services take over 3 minutes to come online.

Disaster Recovery

Prepare for real disaster by practicing the entire backup and restore process.

As extra homework, it is high advisable to spin up a 2nd host for GitLab Server in order to test the backup and restore process from start to finish:

https://docs.gitlab.com/ee/raketasks/backup_restore.html#restore-gitlab

GitLab CI is a game changer.: Gitlab is like Circle CI 2.0 only not black box.
GitLab CI is extendable to any workflow: but you should keep it simple, learn to use the tool as intended, and rethink legacy workflows rather than port.

Contents

Why GitLab Server?
Expected Results
GitLab Server Omnibus In Docker
Disaster Recovery

Russell Open Sources Remarkbox & MakePostSell into Public Domain!

2021-12-23T12:08:00-05:00

Merry Christmas & Happy Holidays, Everyone!

My wife prompts students with this fun holiday writing idea:

"If you could gift the world anything, what would it be?"

I love to read responses in the comments!

As for me, I have wonderful news, my dream gift to the world will come true this year!

For the holiday season, I AM gifting to the public domain my most essential code in hopes to trigger a positive trend of doing in 2022!

"in 2022, dream & do!"

---

Each project has a link to guide on how to access the source code.

https://git.unturf.com/engineering/remarkbox/remarkbox

https://git.unturf.com/engineering/make-post-sell/make_post_sell

In 2021, I imagined many possible paths, yet all seemed to lead to one word:

OPEN

I will continue to operate the SaaS under Remarkbox and MakePostSell trademarks & operations will fall under unturf.

I will take an additional role of BDFL in regards to:

onboarding volunteers to maintain our common code in perpetuity
tie breaking

It feels vulnerable to share code without team support. Think from end & this feeling will pass.

Risk:

Alice may finally read the code & find vulnerabilities, burning a hole between us crippling our communication, taking down the system.

Solution:

More environments in wild, more eyes on problems, more options communicated.

Risk:

The masses may fork & divide the perfect circle.

Solution:

We volunteer to grow & multiply our systems, not worry about forks. We avoid focus on those who ignorantly mutilate or divide, instead focus on projects and forks which sprout, multiply, & thrive!

---

At a high level the unturf. stack currently consists of the following:

And of course we dogfood:

For analytics we self host our own:

Plausible

If the documentation works, people should be able to use the services without understanding the fundementals of each keyword listed above.

Pull requests welcome.

---

The writing of this essay has unfolded liberation in me and so, I speak words in favor of Truth, Freedom, and Love.

I AM now free to Grow, Explore, Document, and Multiply!

I love you, have a great day reader!

GNUnet GNS Nameserver Operator Notes, Quickstart, & Cheatsheet

2021-12-03T18:21:00-05:00

We GROW with Truth, Freedom, Love.

"You have an ally".

We scroll through the official GNUnet handbook, everything is covered in details, sections have examples. Seems verbose at first glance but also familiar .

Freedom of speech is under attack.

I AM not scared, we understand the technology we have and the technology we need. We will rescue ourselves and GROW together.

GNS right NOW!

Why?

Our Collective's Internet Nameservice (DNS) is UNDER ACTIVE ATTACK by groups of governments. Multitiple reports of DNS Name registrars breaking registrants Internet Domains on the behalf of NON-HUMAN ENTITIES.

To avoid any erosion of our Freedom of speech, we will use GNS to make sure our web domains continue to map to our resources for people looking for us.

I AM a seasoned 17 year Bind9 DNS administrator. I had my start on FreeBSD 5.0, virtualized over the years, now on Ubuntu VMs running on "the cloud" & local hypervisors hosting on broken laptops.

We know how to scale DNS, it sort of just works once you figure out the config files.

So let's figure out GNS and GROW a new Internet Culture, eh?

---

We pull down the source code using git:

git clone https://git.gnunet.org/gnunet.git

We see a lot of C code, interesting choice.

cd gnunet
[fox@blanka gnunet]$ ls -hal src/dns/
total 144K
drwxrwxr-x. 1 fox fox  362 Dec  3 18:09 .
drwxrwxr-x. 1 fox fox  834 Dec  3 18:09 ..
-rw-rw-r--. 1 fox fox 9.3K Dec  3 18:09 dns_api.c
-rw-rw-r--. 1 fox fox 1.3K Dec  3 18:09 dns.conf.in
-rw-rw-r--. 1 fox fox 2.2K Dec  3 18:09 dns.h
-rw-rw-r--. 1 fox fox  126 Dec  3 18:09 .gitignore
-rw-rw-r--. 1 fox fox  11K Dec  3 18:09 gnunet-dns-monitor.c
-rw-rw-r--. 1 fox fox 7.1K Dec  3 18:09 gnunet-dns-redirector.c
-rw-rw-r--. 1 fox fox  32K Dec  3 18:09 gnunet-helper-dns.c
-rw-rw-r--. 1 fox fox  35K Dec  3 18:09 gnunet-service-dns.c
-rw-rw-r--. 1 fox fox  14K Dec  3 18:09 gnunet-zonewalk.c
-rw-rw-r--. 1 fox fox 2.2K Dec  3 18:09 Makefile.am
-rw-rw-r--. 1 fox fox 7.5K Dec  3 18:09 plugin_block_dns.c
-rwxrwxr-x. 1 fox fox 1.6K Dec  3 18:09 test_gnunet_dns.sh

GNUnet 0.15.0 released: https://www.gnunet.org/en/news/2021-08-0.15.0.html

GNS First-come-first-served GNUnet top-level domain ".pin" zone key and website updated

Wow, Ok - We want to reserve our favorite names, right?

How do we generate our zone key`?

A section in the GNUnet handbook which seems to cover zones and keys:

https://docs.gnunet.org/handbook/gnunet.html#First-steps-_002d-Using-the-GNU-Name-System

To register a top-level domain of the .pin we need to generate a public/private keypair for each zone.

But before we do that we need to install GNUnet and configure the tool.

Ok, I would prefer some up-to-date binaries than the source tree.

Fedora package looks old and abandoned, same with Ubuntu, & Alpine...

I AM dreaming of a lightweight Alpine docker container to pass around the nets with with GNUnet prebaked in. Maybe a project for another day.

It seems like the NixOS package manager has an uptodate version 0.15.3!

To install GNUnet binaries, we must first learn how to install nix package manager.

Installing nix package manager, run the following:

# reference: https://nixos.org/download.html
curl -L https://nixos.org/nix/install | sh

Once installed try to install GNUnet, like this:

nix-env -iA nixpkgs.gnunet

Errors complaining about certificates might be an easy fix:

source /home/fox/.nix-profile/etc/profile.d/nix.sh

For example the install script placed the following into my ~/.bash_profile:

if [ -e /home/fox/.nix-profile/etc/profile.d/nix.sh ]; then . /home/fox/.nix-profile/etc/profile.d/nix.sh; fi # added by Nix installer

Once you get all that sorted you should have installed GNUnet!

To verify try to interact with ~/.nix-profile/bin/gnunet-config.

Battleship Operational, A Carrier Has Arrived.

[fox@blanka russell.ballestrini.net]$ gnunet-config --version
gnunet-config v0.15.3 release

Alright now we must somehow configure zones and public/private keypairs with this tool.

The handbook gives us this as an example, and we break down the anatomy of the command and outputs.

gnunet-config -s gns -o .myfriend -V PUBLIC_KEY

Let's try passing --help

[fox@blanka russell.ballestrini.net]$ gnunet-config --help

gnunet-config [OPTIONS]
Manipulate GNUnet configuration files
Arguments mandatory for long options are also mandatory for short options.
  -b, --supported-backend=BACKEND
                             test if the current installation supports the
                               specified BACKEND
  -c, --config=FILENAME      use configuration file FILENAME
  -d, --diagnostics          output extra diagnostics
  -F, --full                 write the full configuration file, including
                               default values
  -f, --filename             interpret option value as a filename (with
                               $-expansion)
  -h, --help                 print this help
  -L, --log=LOGLEVEL         configure logging to use LOGLEVEL
  -l, --logfile=FILENAME     configure logging to write logs to FILENAME
  -o, --option=OPTION        name of the option to access
  -r, --rewrite              rewrite the configuration file, even if nothing
                               changed
  -S, --list-sections        print available configuration sections
  -s, --section=SECTION      name of the section to access
  -V, --value=VALUE          value to set
  -v, --version              print the version number
Report bugs to gnunet-developers@gnu.org.
Home page: http://www.gnu.org/s/gnunet/
General help using GNU software: http://www.gnu.org/gethelp/

Ok so it seems like this command simply edits config files for GNUnet.

Ok but where are these enigmatic config files?

This is a living document we will continue the story toward a resolution and simple quickstart guide!

Leave comments below!

Copying files between cloud object stores like S3 GCP and Spaces using Boto3

2021-05-12T18:50:00-04:00

tl;dr if you just want something like aws s3 cp cli, try gsutil rsync.

At work one key item of team's sprint is properly utilizing and securing Google Cloud Platform (GCP).

For one of my projects, I'm learning GCP's Object Store called Google Cloud Storage (GCS).

I have prior experience using AWS S3 and Digital Ocean Spaces via aws s3 CLI and boto3 and I've even blogged in the past about some pretty advanced topics, like Pre-signed GET and POST requests. Pre-signing allows the client to perform specific actions on specific private objects, which is great for short time-to-live link downloads or direct browser uploads into the object store, all without compromising data security (hampering link sharing and preventing replay attacks)

At work, learning how to utilize and secure the GCS object store is critical to our teams interopability between clouds. We want to be able to store security artifacts into a single GCS bucket and sync all data to our main AWS S3 bucket as a long term archive.

Working with GCS using Boto3

Given my prior experience with boto3 I decided to test interoperability between it's s3 client and GCS.

For this test I created the following via the GCP Console:

IAM Service Account (with GCS Editor and GCS Viewer roles)
GCS Bucket
Service Account HMAC (linked to the new IAM Service Account)

I also set the default GCP project which is important because by default boto3 is not configured to pass the x-amz-project-id HTTP header.

At this point I have an access key and secret key (hmac) which in theory I may pass to my existing code to communicate with GCP in a similar way to how I communicate with S3 or Spaces.

Armed with this information I ran the following test and it worked:

# Reference:
# https://cloud.google.com/storage/docs/migrating#storage-list-objects-s3-python

import boto3

def list_gcs_objects(google_access_key_id, google_access_key_secret, bucket_name):
    """Lists GCS objects using boto3 SDK"""
    # Create a new client and do the following:
    # 1. Change the endpoint URL to use the
    #    Google Cloud Storage XML API endpoint.
    # 2. Use Cloud Storage HMAC Credentials.

    client = boto3.client(
        "s3",
        region_name="auto",
        endpoint_url="https://storage.googleapis.com",
        aws_access_key_id=google_access_key_id,
        aws_secret_access_key=google_access_key_secret,
    )

    # Call GCS to list objects in bucket_name
    response = client.list_objects(Bucket=bucket_name)

    # Print object names
    print("Objects:")
    for blob in response["Contents"]:
        print(blob["Key"])

If you are working with many GCP projects, you'll want to figure out how to configure boto3 to pass the x-amz-project-id. I found a good example reference but have not put it to practice: https://github.com/boto/boto3/issues/2251

The TL;DR is AWS doesn't have a concept of projects or default projects so this header is something GCP introduced for interopability.

GCP IAM first impressions

From a surface level look GCP IAM appears less complicated than AWS IAM but also less granular. I'm not sure how I feel at this point but I'm excited to see what the rest of my team uncovers when it comes to permissions, roles, and best practices.

Working with GCS using a CLI

Now that we covered boto3 (Python) interoperability between object stores, I started looking at some other simple CLI workflows which typically utilize aws s3. In the case of GCP the prefered CLI is gsutil.

The subcommand gsutil rsync in particular caught my eye as a simple way to setup a cross cloud object store synchronization!

For example:

gsutil rsync -d -r gs://my-gs-bucket s3://my-s3-bucket

For my next test, I'd like to try to setup a cronjob style automation to trigger gsutil rsync to copy and sync data from GCP GCS into AWS S3 for our long term security and governance artifacts, likely using a Gitlab CI Pipeline on a schedule.

Contents

Working with GCS using Boto3
GCP IAM first impressions
Working with GCS using a CLI

I tried to install GitLab on TrueNAS and failed

2020-10-25T14:28:00-04:00

Ok, so a month ago I changed employment and the new company uses GitLab exclusively for centralized code version control system.

This is my first time using GitLab and my first projects was integrating a dou/cloudmapper with a GitLab runner on a schedule.

After a couple weeks I've learned how GitLab runners work and wrote a wrapper to run cloudmapper concurrently so that each accounts is collected in parallel (the basic logic is running docker exec on a custom entrypoint and then backgrounds the task in a bash loop over each account configured in cloudmapper's config file)

GitLab runners are awesome.

I've used CircleCI for the better part of 3 years, having lifted the better part of 60 code bases from an internal app called conveyor to CircleCI 1.0 and performing CircleCI 2.0 conversions at Remind.

CircleCI 2.0 is awesome...

But GitLab runners are not black box which makes them so powerful.

Granted not all organizations need or even want this much power, but I'm a nerd and I want it, so I decided I also want to run GitLab at home.

My first thought was to try to just run GitLab on a VM on my SmartOS host called mbison, but that's a bit janky of a setup since it's an T420 ThinkPad after all. Also I learned the recommended minimum memory footprint of GitLab is 4G so a 4 of 16G allocation without even discussing a separate KVM for the runner. It feels weird that a single rails application would need _that_ much memory.

Anyways, I have a FreeNAS FreeNAS-11.3-U5 host at my house with 46G of memory and a bunch of idle cores, So I wondered if they have a plug-in for GitLab... I'm sure they do by now.

After a bit of research I found that iXsystems renamed the FreeNAS codebase to TrueNAS and version TrueNAS-12.0-RELEASE has a GitLab plugin! Sweet!

Thankfully I remembered that back in 2019 I replaced my FreeNAS USB boot device with an SSD so it should be a safe operation to attempt a TrueNAS upgrade on a Saturday morning since I'd have the weekend to try to recover if things went sour. Fun side story, I ran into trouble last year trying to upgrade FreeNAS train because I didn't have room for the upgrade because I was using a USB drive as the boot disk and it had like 5 previous version on the 8G file system.

The kind people in the freenode #FreeNAS channel helped me switch back to an older version using SSH and told me I should not boot from a USB drive which was a thing I picked up from back in the FreeNAS 6 or 7 days. Boy were they correct, I moved the OS to an SSD and the server boots _WAY_ faster now, plus I have more capacity to try new versions.

Anyways, here is a fun command to determine when a ZFS pool was created, I used this to get the exact date and time I moved from USB to SSD for this blog post:

root@guile[~]# zpool history -i freenas-boot | grep "create pool"

2019-11-12.13:00:30 [txg:5] create pool version 28; software version 5000/5; uts  11.2-STABLE 1102500 amd64

So weilding this confidence, I switched FreeNAS trains in the UI and clicked upgrade, following the prompts to create a backup of my NAS config. After the server rebooted, FreeNAS was now TrueNAS. I logged into the web UI to look around and make sure my data was still there.

Everything seemed fine, so I went to the plug-in tab and attempted to install GitLab. That failed with an error.

So I decided oh well, I'll try spinning up an Ubuntu VM on the TrueNAS box, apparently FreeBSD uses bhyve for that.

Attempting to start the VM does nothing in the UI, it appears like it will start but then the page refreshes and the VM is stopped.

I did eventually find the error message, by running

tail -f /var/log/libvirt/*/*

/usr/sbin/bhyve -c cpus=1,sockets=1,cores=1,threads=1 -m 4096 -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3:0,ahci,cd:/mnt/downloads/iso/ubuntu-20.04.1-live-server-amd64.iso -s 30:0,xhci,tablet -s 2:0,ahci -s 5:0,virtio-net,tap1,mac=00:a0:98:4f:98:13 -s 4:0,virtio-blk,/dev/zvol/downloads/gitlab-vcrg2z -s 31,lpc -l com1,/dev/nmdm1A -s 29,fbuf,vncserver,tcp=0.0.0.0:48009,w=1024,h=768 1_gitlab
25/10/2020 17:49:58 Listening for VNC connections on TCP port 48009
25/10/2020 17:49:58 Listening for VNC connections on TCP6 port 48009
ROM boot failed: unrestricted guest capability not available
fbuf frame buffer base: 0x941e00000 [sz 16777216]

The important part is:

ROM boot failed: unrestricted guest capability not available

It seems if you ask bhyve to use the UEFI bootloader it will require UG support in the Intel CPU : (

I attempted to switch to the grub bootloader but in that case I get the following error message:

File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1414, in do_update
  raise verrors
middlewared.service_exception.ValidationErrors: [EINVAL] vm_update.devices.3.dtype: VNC only works with UEFI bootloader.

and the TrueNAS UI does not let you disable VNC on VMs at this point...

Another limitation of a missing UG CPU flag is bhyve may only allocate 1 core, and 1 vcpu to a VM.

Anyways I'm out of luck, I can't use the plugin which uses jails due to a strange error and no other hints as to the issue and byhve / TrueNAS not supporting VMs for my CPU...

UPDATE I recently upgraded my TrueNAS server hardware to an old Dell PowerEdge R720 R720xd with two Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz each with 8 Physical Cores and 128G of ECC Memory! Booyah!

I was able to flash the included Dell H710 Mini RAID card into IT mode for ZFS support using the awesome guides and tools found here: https://fohdeesha.com/docs/perc/

I now have GitLab running on a byhve instance and also a dedicated runner instance. It's working great and really fun to have this extremely powerful gear in my home lab!

WeeChat on-boot in a tmux session

2020-09-15T13:32:00-04:00

Chronicles of a washed up systems administrator.

In the basement, M. Bison (mbision.foxhop.net.), a T430 with a cracked screen runs quietly with his lid closed, acting as a SmartOS hypervisor to 3 Solaris derived zones and 4 KVM ubuntu guests.

One of these guests is the oldest of the bunch and his name is Akuma (akuma.foxhop.net.).

Akuma and I go way back, over 13 years now. At one point Akuma was a physical machine, who ran FreeBSD and then later Ubuntu 6.04 with it's own guest kernel virtual machines.

Akuma has been "home base" for as long as I remember. A place where shit got done. A place to perform operations. A place to "jump" from with SSH.

It makes sense that Akuma would evolve and gain many roles over the years.

Akuma performs DNS caching and forwarding for my LAN, it acts as the internal authoritative Nameserver for foxhop.net., and is the Salt Master for all my hosts both internally (hosted at my house) and externally in the "cloud".

Akuma uses tmux with default settings to allow for a "remote shell" vibe.

This allows me attach to my session running on Akuma from anywhere in the world.

Only problem is, Akuma reboots weekly when M. Bison triggers a complete backup of all guests. Backups are stored on Guile, the FreeNAS server, uncompressed since disk space is cheap and this process allows for downtime, however I try to limit it.

Since Akuma reboots each week, I needed a way to create tmux sessions on boot if I wanted to continue to use it as my "home base", and I did so using SaltStack:

create-tmux-session-on-boot:
  cron.present:
    - comment: "create a new tmux session on system boot"
    - name: /bin/bash /home/fox/new-tmux
    - identifier: "create-tmux-session-on-boot"
    - special: "@reboot"
    - user: fox
    - require:
      - user: fox

This salt state results in the following crontab -l entry:

# create a new tmux session on system boot SALT_CRON_IDENTIFIER:create-tmux-session-on-boot
@reboot /bin/bash /home/fox/new-tmux

And of course I what tutorial wouldn't be complete without a script! (/home/fox/new-tmux) :

#!/bin/bash
# The -A flag makes new-session behave like attach-session if session-name
# already exists; in this case, -D behaves like -d to attach-session.
tmux new-session -d -A -s "remote-shell" "weechat-curses"
# also create a couple windows to use as "remote shells".
tmux new-window
tmux new-window
# finally attach to the new session!
tmux a -t "remote-shell"

Thanks for joining me on this escape into the world of computers.

This website is hosted on Ryu (ryu.foxhop.net) another Ubuntu guest running on M. Bison, humming away downstairs on that T430 with a cracked screen, but thats a story for another time.

Please feel free to comment below if you have suggestions for this post.

Dreaming of unlimited home computer storage capacity

2019-12-15T09:52:00-05:00

Unlimited computer storage capacity is a common science fiction trope and fun thought experiment.

What would be possible if we could potentially store near infinite data at your house in a normal sized desktop computer?

Even better using today's tech, what ideas could we dream up?

This post will explore ideas as we surf the bleeding edge of home computing.

Idea 1

Live stream everything in your life in HD and store it locally at your house as a hyper real augment reality. A person could store and recall any event that he or she has witnessed using a headcam with wifi syncing capabilities.

One of the many new features within the iPhone 6s is the ability for the 12MP camera on the back to record 4K video. It’s a big selling point for Apple, but what about storage space?

According to a video published by MKBHD, just how much a one-minute long video shot in 4K will take up in your iPhone 6s’ storage has been discovered. According to the screenshot captured by Ryan Jones (@rjones) of the hands-on video, a one-minute video captured at 720p HD at 30fps will take up 60MB of space on the iPhone 6s. At 1080p HD and 30fps, it’s 130MB of space. A 1080p HD video at 60fps will take up 200MB of space. And, finally, the 4K video at 30fps will take up 375MB of space.

reference: http://www.iphonehacks.com/2015/09/iphone-6s-4k-video-space.html

If we were streaming at 4k for 24 hours, it would be:

>>> 60 * 24 * 375
540000

540,000MB or 540GB or .54TB per day of streaming everything.

A years worth of 4k streaming from some guys headcam would consume:

>>> 365.25 * .54
197.235

or 197TB worth of capacity.

With today's technology and hardware available as home computing, including NAS systems, streaming 4k of a persons headcam would be possible... but not really practical without advancements in hard drive density.

A 6TB drive is $100, and by my math, we would need 34 drives (not really even counting for redundancy):

>>> 197.235 / 6
32.8725

So the cost of a years worth a capacity would be:

(197.235 / 6) * 100
3287.25

or $3,287 for 34 x 6TB drives.

It would fit in an oversized NAS, just barely... in another section I'll figure out what hard drive density is needed to make 4k video at 30fps more practical.

Ok so, 4k streaming is currently not practical, so how about 1080p HD at 30fps? Could we do that with today's home tech?

One minute of 1080p HD at 30fps is 130MB of space.

If we were streaming at 1080p at 30fps for 24 hours, it would be:

>>> 60 * 24 * 130
187200

187,200MB or 187GB or .187TB per day.

A years worth of non-stop 1080P streaming at 30fps off some guys headcam would consume:

>>> 365.25 * .184
67.235

Or about 68TB per year (rounded up).

Like before a 6TB drive is $100, and by my math, we would need 12 drives (not really even counting for redundancy):

>>> 67.206 / 6
11.201

so the cost of a years worth a capacity would be:

>>> 12 * 100
1200

or $1,200 for 12 x 6TB drives.

12 drives should easily fit in a mid-range home NAS. That said, a mid-range home NAS holding 12 drives is a clunky machine for most home owners.

It might be possible, but likely not practical without a support plan, in home repairs, and on-going maintenance.

It would likely require having a home NAS specialist in every region to offer up white gloved support for anyone who would be willing to pay for it.

How could we make this even easier with today's tech?

We could lower the quality even more...

Streaming 720P at 30fps you would need 36T per year or about 6 x 6TB drives for about $600 plus a NAS.

So at this time, if seems practical and economical to stream 720P at 30fps every minute of the year.

What would we need to build to make all this data useful?

Well we would need a way to index all this footage, and likely edit out useless footage.

We would need at least two headcams with wifi sync, so that we could have one charging while we use the other.

We would need software to sync the footage to the NAS, like syncthing.

Besides that, all we need is some software to host the videos, edit, and index them to be searchable.

Contents

Idea 1

Unity Collab not working on Fedora because of an invalid CA certificate path

2019-11-08T12:19:00-05:00

This issue blocked me for about two days and prevented me from collaborating with another engineer at gumyum co-operative video game studio.

The other engineer's environment was Windows and he was able to interact with collab and our shared project owned by our shared org. My environment was Fedora 30 and was not working with collab, however I was able to download the project files.

The issue?

Apparently Unity naively assumes that a Linux system's copy of the root CA certificates is located /etc/ssl/certs/ca-certificates.crt. This is the default location on Debian and Ubuntu based systems but not Fedora or Redhat, which uses the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem file.

The work around is simple, create a soft link between the paths:

sudo ln -s /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/ssl/certs/ca-certificates.crt

Please come back again soon to learn more about the games we are building over here at gumyum!

How to fallback to an older FreeNAS version on update failure

2019-10-31T17:06:00-04:00

try to log in as root locally, I needed to press ctrl-alt-f7 to get to a log in prompt.

Once logged in as root, I used the beadm command.

For example, beadm list and beadm activate <old-version>.

Finally reboot.

I followed this process and then got my FreeNAS server to boot again.

From the GUI I was able to create myself a config backup.

I then burned a new FreeNAS CD and used that to install a fresh new install on my USB drive and then once that booted I used the GUI to restore my config.

Installing Unity on Linux

2019-09-25T18:55:00-04:00

Russell, why are you installing Unity? I thought you were an open source developer and homegrown video game engine builder?!

The TL;DR we are starting a video game co-operative called gumyum. The basic idea is to build a large video game company owned completely by members. There will be no employees and most revenue after paying taxes, operational expenses, and marketing, will be divided among members.

We are actively looking for members, specifically engineers who want to build games. We already have a distributed team of 4 artists and 2 engineers who meet on Slack as we progress toward delivering our first two games.

Adopting Unity, at least for our first couple games, will lower the barrier of entry for other engineers when joining the co-operative.

Installing Unity On Linux

To get started first we will create a Unity Account and install the Unity Hub. You need both in order to aquire a free licence for personal use.

The Unity Hub software will help you download various Unity versions, manage your licence, and manage game projects.

You should go to the Unity website and created an account.

I found the Linux Unity Hub installer in this thread and downloaded it to my ~/Downloads directory.

Next I opened a terminal and made the file executable, and executed it.

cd ~/Downloads
chmod 755 UnityHub.AppImage
./UnityHub.AppImage

From there I needed to sign in and request a licence. I chose the free option since at this point we have a $0 revenue project.

On the Installs tab in Unity Hub you may download various stable and LTS (long-time-supported) versions of Unity. I suggest using one of these stable versions, I accidentally grabbed a non-stable one and all I had was pink screens inside of Unity!

Anyways, now you may start a Unity project. Unity Hub seems to store files locally and also allows you to sync to the cloud. This enables easy collaboration on the same project.

Thats all for now, please check back for more posts about gumyum!

Pre-signed GET and POST for Digital Ocean Spaces

2019-07-12T16:01:00-04:00

A pre-signed request grants a semi-trusted user temporary access to a private resource.

Let's unpack that statement ...

Pre-signed means, we bless a specific action on a specific private resource for a short duration of time.

Semi-trusted means, we have authenticated the user, but we don't trust them to have full access to our private resources.

That still seems a bit generic ...

No worries, our next example will appear more concrete:

Today we will pre-sign HTTP POST and GET requests to grant a semi-trusted user temporary access to objects in a private Digital Ocean Space.

Pre-signing allows the client to perform specific actions on specific private objects while still protecting us from link sharing and replay attacks.

The following examples use Python (Boto3), and Curl.

Before we start, you should reference the official Boto3 Presigned Urls documentation.

Pre-signed GET or Download

Allow a semi-trusted user the ability to download a specific file named my-object.zip from a private Digital Ocean Space named example-bucket for a short duration of 30 seconds.

Save this code into a file named pre_sign_get_test.py and run it with python pre_sign_get_test.py.

The result will be a URL that you may redirect a user to. For testing you should be able to load the URL into a web browser and download the file.

import boto3
from botocore.client import Config

# Initialize an S3 session/client to talk to DigitalOcean Spaces.
session = boto3.session.Session()
client = session.client(
    "s3",
    # configure the region you created the space in.
    region_name="nyc3",
    # configure the region you created the space in.
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    # configure your access key (generate this on the dashboard).
    aws_access_key_id="DOXXXXXXXXXXXXXXXXXX",
    # configure your secret key (generate this on the dashboard).
    aws_secret_access_key="DOO/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
)

signed_get_object_url = client.generate_presigned_url(
    ClientMethod='get_object',
    Params={
        'Bucket': 'example-bucket',
        'Key': 'my-object.zip',
    },
    ExpiresIn=30
)

# print the pre-signed GET request URL for downloading the object.
# You may redirect the user to this URL when they click a "download" button.
print(signed_get_object_url)

Pre-signed POST or Upload

Allow a semi-trusted user the ability to upload a specific file named my-object2.zip into a private Digital Ocean Space named example-bucket for a short duration of 60 seconds.

Save this code into a file named pre_sign_post_test.py and run it with python pre_sign_get_test.py.

The result will be a cURL command that you may run on your console for testing.

In a real situation, you would want to pass these parameters to the client and let it perform the authenticated upload. The file never hits your server and is uploaded directly to the private Space.

IMPORTANT: parameter order matters! The file parameter must be last.

import boto3
from botocore.client import Config

# Initialize an S3 session/client to talk to DigitalOcean Spaces.
session = boto3.session.Session()
client = session.client(
    "s3",
    # configure the region you created the space in.
    region_name="nyc3",
    # configure the region you created the space in.
    endpoint_url="https://nyc3.digitaloceanspaces.com",
    # configure your access key (generate this on the dashboard).
    aws_access_key_id="DOXXXXXXXXXXXXXXXXXX",
    # configure your secret key (generate this on the dashboard).
    aws_secret_access_key="DOO/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
)

signed_post = client.generate_presigned_post(
    Bucket="example-bucket",
    Key="my-object2.zip",
    ExpiresIn=60,
)

params = []
for key, value in signed_post["fields"].items():
    params.append('--form "{}={}"'.format(key, value))

print("curl " + " ".join(params) + ' --form "file=@my-object2.zip;filename=my-object2.zip" ' + signed_post["url"])

Contents

Pre-signed GET or Download
Pre-signed POST or Upload

Hybrid Hot Water Heater Saves 69 Percent On Energy Consumption

2019-02-05T09:28:00-05:00

Disclosure: This post contains affiliate links. See full disclosure page here.

Exactly one year ago I fufilled a longtime childhood dream when I invested in roof top solar on my house in Eastern Connecticut (Zone 6b).

Over the past 12 months, I have religiously tracked my families energy consumption using both a "Kill A Watt" and a Curb.

Along the way, I have found my largest energy consumer (abuser).

Heating water.

My traditional electric water heater had a monthly operating cost of $85. Worst yet, during some winter months, 100% of my solar production was soaked up by heating water!

I demanded a better way!

... and I found one ...

Hybrid Electric Hot Water Heat Pump

My new unit (A.O. Smith 50 Gallon Voltex Hybrid Electric Heat Pump) transfers ambient heat from the air into water.

The technology is relatively old, basically an air conditioner or refrigerator run in reverse. Instead of dumping the cold into the tank, it dumps hot and as a by-product produces cold air.

Yes that's right, this unit will suppliment my summer cooling because it will dump cold air while it produces hot water. The heat pump also serves as a dehumidifier, so I no longer run one of those!

The install was mostly identical to a normal water heater, however there are a couple extra requirements:

The unit needs at least 400 square feet to pull ambient air
The unit produces condensate, similar to a dehumidifier or air conditioner, which needs to drain somewhere.

Additionally the heat pump fan produces a bit of sound, not terrible, and seems quiet to me.

If you have a traditional electric water heater, you meet the extra requirements, and you consider yourself an environmentalist, you need to get a hybrid water heater.

Finances

These units currently run between $1200 - $1500 but I'll break down the complete finances of my project.

+ $1,149 purchase price
+    $65 delivery
+   $100 pipes and fittings
-   $300 instant rebate
-   $200 mail-in rebate
-   $200 craigslist 3 year old unit
===================================
+   $614

I replaced the unit myself, with help from my father-in-law. I don't own a truck, so I decided to have it delivered for $65. I bought $100 worth of pipe and fittings, mostly because I don't want to learn how to sweat copper, so I use sharkbite fittings which cost more but are fast and simple to use.

My state offers an instant $300 in store rebate, plus a $200 mail in rebate.

The old unit was only 3 years old (came with the new house purchase) so I sold it on craigslist for $200 (MSRP was $899).

Return on investment

Ok, so $614 to replace a working unit, am I crazy? What is the expected ROI?

Well, I've only had the unit for a month, however my energy consumption for heating hot water went from $85/mo to $30/mo.

Let's just round down to $50/mo savings.

Ok so in 12 months the unit will pay for itself. Even without the rebates you're still looking at only a 2 year ROI before the unit starts paying dividends.

That's not all.

I don't need to run a separate dehumidifier in my basement because the new unit removes humidity as a by-product!

... but wait there's more!

This unit also produces cold air as a by-product which means free supplimental cooling during the summer!

A water heater, dehumidifier, and air conditioner all-in-one!

What is the catch?

There is no catch, but you do have to agree on a small trade off, hot water recovery time.

Instead of using 4,500 watts for 45 minutes (.75 hours), a hybrid hot water heat pump uses 350 watts for 3 hours.

Trading the slightly longer hot water recovery times, gives you:

a significant reduction on your energy bill
dehumidification
supplimental air conditioning / cooling

# traditional.
4500 * .75 # == 3375 watts or 3.375 kWhr to recover

# hybrid heatpump.
350 * 3 # == 1050 watts or 1.050 kWhr to recover

The decrease in consumption means a huge savings of 69%!!!

# percentage decrease.
(3375 - 1050) / 3375 # == 69% !!!

So what are you waiting for? Honestly, if you are thinking about going solar, you should tackle this project first, right now!

Contents

Hybrid Electric Hot Water Heat Pump
Finances
Return on investment
What is the catch?

AWS nvme to block mapping

2019-01-19T14:50:00-05:00

Recently at work I transitioned our fleet from Ubuntu 14.04 LTS to Ubuntu 18.04 LTS. During the process I noticed an issue with our newer generation AWS EC2 "nitro" based instance types (specifically c5.2xlarge).

AWS was presenting my root block device as /dev/nvme1n1 and my data device as /dev/nvme0n1. For obvious reasons, this seemingly random and out-of-order assignment breaks my provisioning scripts.

After much research and deep thought, I came up with a solid solution.

I found a barely documented tool called ebsnvme-id on the official Amazon Linux AMI and wrote a wrapper (nvme-to-block-mapping) to iterate over all possible combinations of /dev/nvme[0-26]n1 to create a symlink to the block mapping selected when we launch the EC2 instance.

Since we have control over the block mapping, we end up with consistent and known symlink which we may confidently pass to our provisioning scripts when the time comes to partition, format, and use the block devices.

To save you time, I have added these scripts to this blog post!

Please consider donating here if my work has helped you!

Place the following two scripts into your AMI under /usr/sbin/ and trigger the wrapper just once prior to using the devices (don't worry the wrapper script is idempotent, running it more than once won't break anything).

nvme-to-block-mapping

/usr/sbin/nvme-to-block-mapping (click for file):

#!/bin/bash

# for details:
# https://russell.ballestrini.net/aws-nvme-to-block-mapping/

# this will create a symlinks like:
#
#     /dev/nvme1n1 -> /dev/xvdh
#
# these ebs block device paths are set by stacker and assumed by ansible.
#
# if the device is non ebs, it will use the following mapping:
non_ebs_mapping=("/dev/sdb1" "/dev/sdc1" "/dev/sdd1" "/dev/sde1" "/dev/sdf1" "/dev/sdg1")

# nvme0n1 uses ${non_ebs_mapping[0]} (the 0 index item in the array)
#
#     /dev/nvme0n1 -> /dev/sdb1
#
# nvme3n1 uses ${non_ebs_mapping[3]} (the 3 index item in the array)
#
#     /dev/nvme3n1 -> /dev/sde1
#

# why we only iterate from 0 to 26:
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html
for i in `seq 0 26`; do
    nvme_block_device="/dev/nvme${i}n1"

    # skip any nvme paths which don't exist.
    if [ -e  $nvme_block_device ]; then

        # get ebs block mapping device path set by stacker (or base AMI).
        mapping_device=$(/usr/sbin/ebsnvme-id ${nvme_block_device} --block-dev)

        # if the mapping_device is empty, it isn't an EBS device so
        # we will use the non_ebs_mapping to translate the device.
        if [[ -z "$mapping_device" ]]; then
            mapping_device="${non_ebs_mapping[$i]}"
        fi

        # if block mapping device path does not start with /dev/ fix it.
        if [[ "$mapping_device" != /dev/* ]]; then
            mapping_device="/dev/${mapping_device}"
        fi

        # if the block mapping device path already exists, skip it.
        if [ -e $mapping_device ]; then
            echo "path exists: ${mapping_device}"

        # otherwise, create a symlink from nvme block device to mapping device.
        else
            echo "symlink created: ${nvme_block_device} to ${mapping_device}"
            ln -s $nvme_block_device $mapping_device
        fi
    fi
done

ebsnvme-id

/usr/sbin/ebsnvme-id (click for file):

#!/usr/bin/env python2.7

# Copyright (C) 2017 Amazon.com, Inc. or its affiliates.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#    http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
# OF ANY KIND, either express or implied. See the License for the
# specific language governing permissions and limitations under the
# License.
#
# Reference:
# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html

"""
Usage:
Read EBS device information and provide information about
the volume.
"""

import argparse
from ctypes import *
from fcntl import ioctl
import sys

NVME_ADMIN_IDENTIFY = 0x06
NVME_IOCTL_ADMIN_CMD = 0xC0484E41
AMZN_NVME_VID = 0x1D0F
AMZN_NVME_EBS_MN = "Amazon Elastic Block Store"

class nvme_admin_command(Structure):
    _pack_ = 1
    _fields_ = [("opcode", c_uint8),      # op code
                ("flags", c_uint8),       # fused operation
                ("cid", c_uint16),        # command id
                ("nsid", c_uint32),       # namespace id
                ("reserved0", c_uint64),
                ("mptr", c_uint64),       # metadata pointer
                ("addr", c_uint64),       # data pointer
                ("mlen", c_uint32),       # metadata length
                ("alen", c_uint32),       # data length
                ("cdw10", c_uint32),
                ("cdw11", c_uint32),
                ("cdw12", c_uint32),
                ("cdw13", c_uint32),
                ("cdw14", c_uint32),
                ("cdw15", c_uint32),
                ("reserved1", c_uint64)]

class nvme_identify_controller_amzn_vs(Structure):
    _pack_ = 1
    _fields_ = [("bdev", c_char * 32),  # block device name
                ("reserved0", c_char * (1024 - 32))]

class nvme_identify_controller_psd(Structure):
    _pack_ = 1
    _fields_ = [("mp", c_uint16),       # maximum power
                ("reserved0", c_uint16),
                ("enlat", c_uint32),     # entry latency
                ("exlat", c_uint32),     # exit latency
                ("rrt", c_uint8),       # relative read throughput
                ("rrl", c_uint8),       # relative read latency
                ("rwt", c_uint8),       # relative write throughput
                ("rwl", c_uint8),       # relative write latency
                ("reserved1", c_char * 16)]

class nvme_identify_controller(Structure):
    _pack_ = 1
    _fields_ = [("vid", c_uint16),          # PCI Vendor ID
                ("ssvid", c_uint16),        # PCI Subsystem Vendor ID
                ("sn", c_char * 20),        # Serial Number
                ("mn", c_char * 40),        # Module Number
                ("fr", c_char * 8),         # Firmware Revision
                ("rab", c_uint8),           # Recommend Arbitration Burst
                ("ieee", c_uint8 * 3),      # IEEE OUI Identifier
                ("mic", c_uint8),           # Multi-Interface Capabilities
                ("mdts", c_uint8),          # Maximum Data Transfer Size
                ("reserved0", c_uint8 * (256 - 78)),
                ("oacs", c_uint16),         # Optional Admin Command Support
                ("acl", c_uint8),           # Abort Command Limit
                ("aerl", c_uint8),          # Asynchronous Event Request Limit
                ("frmw", c_uint8),          # Firmware Updates
                ("lpa", c_uint8),           # Log Page Attributes
                ("elpe", c_uint8),          # Error Log Page Entries
                ("npss", c_uint8),          # Number of Power States Support
                ("avscc", c_uint8),         # Admin Vendor Specific Command Configuration
                ("reserved1", c_uint8 * (512 - 265)),
                ("sqes", c_uint8),          # Submission Queue Entry Size
                ("cqes", c_uint8),          # Completion Queue Entry Size
                ("reserved2", c_uint16),
                ("nn", c_uint32),            # Number of Namespaces
                ("oncs", c_uint16),         # Optional NVM Command Support
                ("fuses", c_uint16),        # Fused Operation Support
                ("fna", c_uint8),           # Format NVM Attributes
                ("vwc", c_uint8),           # Volatile Write Cache
                ("awun", c_uint16),         # Atomic Write Unit Normal
                ("awupf", c_uint16),        # Atomic Write Unit Power Fail
                ("nvscc", c_uint8),         # NVM Vendor Specific Command Configuration
                ("reserved3", c_uint8 * (704 - 531)),
                ("reserved4", c_uint8 * (2048 - 704)),
                ("psd", nvme_identify_controller_psd * 32),     # Power State Descriptor
                ("vs", nvme_identify_controller_amzn_vs)]  # Vendor Specific

class ebs_nvme_device:
    def __init__(self, device):
        self.device = device
        self.ctrl_identify()

    def _nvme_ioctl(self, id_response, id_len):
        admin_cmd = nvme_admin_command(opcode = NVME_ADMIN_IDENTIFY,
                                       addr = id_response,
                                       alen = id_len,
                                       cdw10 = 1)

        with open(self.device, "rw") as nvme:
            ioctl(nvme, NVME_IOCTL_ADMIN_CMD, admin_cmd)

    def ctrl_identify(self):
        self.id_ctrl = nvme_identify_controller()
        self._nvme_ioctl(addressof(self.id_ctrl), sizeof(self.id_ctrl))

        if self.id_ctrl.vid != AMZN_NVME_VID or self.id_ctrl.mn.strip() != AMZN_NVME_EBS_MN:
            raise TypeError("[ERROR] Not an EBS device: '{0}'".format(self.device))

    def get_volume_id(self):
        vol = self.id_ctrl.sn

        if vol.startswith("vol") and vol[3] != "-":
            vol = "vol-" + vol[3:]

        return vol

    def get_block_device(self, stripped=False):
        dev = self.id_ctrl.vs.bdev

        if stripped and dev.startswith("/dev/"):
            dev = dev[5:]

        return dev

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Reads EBS information from NVMe devices.")
    parser.add_argument("device", nargs=1, help="Device to query")

    display = parser.add_argument_group("Display Options")
    display.add_argument("-v", "--volume", action="store_true",
            help="Return volume-id")
    display.add_argument("-b", "--block-dev", action="store_true",
            help="Return block device mapping")
    display.add_argument("-u", "--udev", action="store_true",
            help="Output data in format suitable for udev rules")

    if len(sys.argv) < 2:
        parser.print_help()
        sys.exit(1)

    args = parser.parse_args()

    get_all = not (args.udev or args.volume or args.block_dev)

    try:
        dev = ebs_nvme_device(args.device[0])
    except (IOError, TypeError) as err:
        print >> sys.stderr, err
        sys.exit(1)

    if get_all or args.volume:
        print "Volume ID: {0}".format(dev.get_volume_id())
    if get_all or args.block_dev or args.udev:
        print dev.get_block_device(args.udev)

Contents

nvme-to-block-mapping
ebsnvme-id

Yuletide Trains and Homegrown Video Games

2018-12-25T17:15:00-05:00

Each holiday season I find myself drawn to a side passion of mine.

While some build model trains and others create Christmas light shows with synchronized music you'll find me on my sofa where I build, explore, and tinker on my own video game engine.

Maybe the sound of wrapping paper tearing acts as a trigger, but I only prioritize time on my game engine during the holiday season.

Do you have a specific hobby that you do during the holiday season but not during other parts of the year?

Why am I working on Christmas every year - Am I a work-a-holic?

Work or Play.

Why do I absorb myself into this seasonal hobby? Am I escaping? Or am I making use of down time to work on a project that I am too occupied to even think about during the rest of the year?

For me, building my game on Christmas is fun, not work. Similar to how fishing while camping is fun, not work.

What is the last activity a commercial fisherman would do "for fun" on Christmas? Fishing.

What is fun for one person might be work for another. It is relative and depends on context.

I suspect that most people experience a tipping point, when an activity switches from fun to work.

I think, for an average person, this moment occurs when the answer to the following question changes:

Do you rely on the activity for the majority of your income?
- if yes, the activity often feels like work
- if no, the activity often feels like fun

Work and Play?

Now that's not to say you cannot find fun while at work, and vice versa. Personally, I aim to land a perfect balance between the two with anything I do.

In my situation, I use the computer "professionally" each work day, for 7-10 hours.

If I was in the games industry, I doubt I would find much fun in building a video game on Christmas. But I'm not in the games industry and I have more fun building games than playing games.

That is to say, for me building a game feels fun while playing a game feels like work. Crazy right?

Even though the activities of my "day job" overlap with the activities involved with building my game, there are enough differences. These differences make one feel like work while the other feel like fun.

I'm not making an excuse for working during Christmas; I honestly have a blast coding and tinkering on my game.

I love getting stumped and learning. I love showing different ideas to my boys and seeing their eyes light up. I love to see how fast I can implement a small suggestion one of them have. I love how there are no rules and only my own skills and understanding can block me.

While my family is occupied with their new holiday gifts, I use my free time to explore creative ideas on a non-stressful personal project. A project where the stakes are low and the pleasure is high.

Show me the game already!

This years game engine modifications:

ability to save the game state to a json file
ability to load the game state from a json file
added a few algorithms for auto-generating maps
refactored some common code (so I'm not reapeating myself)
implemented an action charge / recharge system

Here is a short video of the current state of the engine. Let me know what you think in the comments.

I also took the time to learn how to make pixel art using GIMP, a useful skill for making game assets.

Who else makes games during Christmas?

I reached out to a number of indie game developers while formulating this blog post and I had a solid conversation with Maverick Peppers a developer who runs a software company called ProtoComplete

He was working during Christmas on his game while making pudding and drinking white russians (recipe in the foot notes). He arrived at the conclusion that I was working from a passion-oriented mindset while he was from a goal-oriented mindset.

Maverick doesn't rely on the income from his games, so I asked "why are you 'working' on Christmas?" He explained that because he often has a type-a or "goal mindset", he has trouble relaxing until he finishes whatever he is working on. In a sense, he was working Christmas so that he could relax.

As for myself, building games is a passion-oreiented process, I don't rely on the income and I use it to unwind.

We talked about how people can approach activities with either:

type-a (goal-oriented mindset)
type-b (passion-oriented mindset)

The fun part is people are often both, depending on the activity or project. For example Maverick has a passion mindset when making music and DJing, but always takes a goal mindset when it comes to business.

Do you have a specific hobby that you do during the holiday season but not during other parts of the year?

Footnotes

Unwrapping My Christmas Commit History

Looking at my commit history, the first iteration of my current game engine was saved on Jan 01, 2014 with a few commits until Jan 19, 2014, at which point nothing until Dec 25, 2014 (Christmas itself) when I sprinted until Jan 10, 2015.

The next year, I must have hacked on something else, with no changes until Oct 09, 2016 where I had two commits.

Like clockwork on Dec 25, 2016 (Christmas) I tried to fix a regression in the engine's collision and intersection code. I left myself some breadcrumb comments to help me debug in the future... Nothing in 2017.

Today is Christmas 2018 and finally I have a work around for the regression I was looking into from Christmas 2016!

Porting SFML Rect from C++ to Python

This work around ports the Rect intersection logic of SFML from C++ to pure Python and avoids the following error message:

terminated by signal SIGSEGV (Address boundary error)

def get_rect_intersection(r1, r2):
    """
    Accept two sfml.graphics.Rect objects.
    Return a new sfml.graphics.Rect of the overlap or None.
    """

    # We allow Rects with negative dimensions, so handle them correctly.

    # Compute the min and max of the first Rect (r1).
    r1_min_x = min(r1.left, r1.left + r1.width)
    r1_max_x = max(r1.left, r1.left + r1.width)
    r1_min_y = min(r1.top, r1.top + r1.height)
    r1_max_y = max(r1.top, r1.top + r1.height)

    # Compute the min and max of the second Rect (r2).
    r2_min_x = min(r2.left, r2.left + r2.width)
    r2_max_x = max(r2.left, r2.left + r2.width)
    r2_min_y = min(r2.top, r2.top + r2.height)
    r2_max_y = max(r2.top, r2.top + r2.height)

    # compute the intersection boundaries.
    i_left   = max(r1_min_x, r2_min_x)
    i_top    = max(r1_min_y, r2_min_y)
    i_right  = min(r1_max_x, r2_max_x)
    i_bottom = min(r1_max_y, r2_max_y)

    # if the intersection is valid (positive non zero area),
    # then there is an intersection.
    if i_left < i_right and i_top < i_bottom:
        return sfml.graphics.Rect((i_left, i_top), (i_right - i_left, i_bottom - i_top))

White Russian Recipe

1/4 cup distilled водка
1/4 cup Kahlua coffee rum
1/2 cup cream

Older versions of the game engine

Some videos of older versions of this game engine.

index

Work or Play.
Work and Play?
Show me the game already!
Who else makes games during Christmas?
Footnotes

All Local Heros need a Gig Side Kick

2018-08-03T11:20:00-04:00

Five months ago during my preparation and ramp up to gardening season, I started thinking about what it would be like to have a partner, a side kick, a secretary to keep me honest and on task.

Somebody who could review old journal notes from previous years and give me hints about what seeds I should be starting, what worked well, what didn't, and why.

I started thinking I'm likely not alone and that other hobby and market gardeners likely feel the same way. I then expanded this, likely anyone bootstrapping a small business "enterprise" could use a side kick. Small craft brewers, artists, sign makers, the list goes on and on.

As you might know I'm deeply interested in systems like computer science and permaculture. Computer scientists often try to model complex natural systems. Often natural systems give us clues on how to make computers perform certain tasks.

One reoccuring theme I find between computer science and permaculture is the idea of inputs and outputs.

Brewing wine from berries requires the following inputs: boiling water, berries, sugar, yeast, and yeast nutrient. Combined all these inputs together in the right way, wait for a certain time, and the system returns an a tasty, alchoholic, product with a long shelf life as an output. The spent berries go straight into the compost for next years garden. Natural systems and unnatural systems model after natural ones, have no waste by-products (pollution). Every output of one process is always an input of another.

Only when we mess up the natural order of organic systems do we end up with polution. Polution is simply a holistically bad output. For example, when we artifically produce outputs, like plastics there is no natural system to take it as an input, it becomes polution and makes the system sick.

To fight off the production of unnatural outputs, like single use plastics, we need to create other human managed systems to use the pollution as inputs. For example, I'm insulating my shed with bags of single use polyethylene wrappers that wrap my families purchases. I'm keeping the valuable (although un-natural) resource out of the waste stream and using it to shrink the tempurature swing deltas in my shed. When we model the material world as inputs and outputs we can learn how to best use what we have on hand.

We can boost each of our local economies by providing proper tools to the local makers.

This is why today I'm announcing GigSideKick to flip the script on globalization and large enterprise giants.

GigSideKick is a "field" journal app to track inputs and outputs. The primary goal is to collect the right amount of data with the least amount of friction, have an intuitive user interface, and a user experience which mostly stays out of the way. In addition to tracking inputs and outputs, the user will optionally set ETAs on processes set in motion which will power reminders.

Each local hero deserves the unfair advantage that a smart and capabile Side Kick provides.

This type of input/output journal will not interfere with the task at hand, and would only require a few button presses. The data created from such a journal is important for the user's future self, and if properly aggregated and crowd sourced, the data will enable a powerful recommendation engine.

For example, a person subscribing to a "garden" knowledge pack might get the following recommendation: "gardeners in your area are known to typically start on average of 24 zucchini seeds this month, which typically result in 18 viable zucchini seedlings in 6 days", "How many will you start?"

None

12

24

48

Other

In the background, the app would also track human labor (which itself is valuable input), track inventory of parts, work-in-process, and finished goods. Finally the app's data would power a local marketplace to help the maker trade and sell finished products. After all, if you produce a surplus of apples without a system in place to use them, the apples will end up as an input to the compost pile instead of the bellies of deserving humans. : )

I believe we must give power back to local communities and economies. We, the side gig heros and local businesses operators, need all the help we can get to compete in today's global economy.

Local makers have many often un-exploited unfair advantages. We do not need to ship our goods which means we should have lower margins. We need to figure out how to market our products (outputs) to local people looking for our superior goods.

Another often untapped unfair advantage is the sale or trade of our by-products. We can even trade our waste! For example as a gardener, I would gladly trade a heaping bowl of same day cut, "beyond organic" salad greens, for your bucket of leaves, coffee grounds, sticks, sawdust, or grass clippings! It's honestly the timeless addage of "one mans trash is another mans treasure". Your waste is a very useful input to the correct system and local people trading outputs is by far the best thing we can do for our planet.

I plan to document my journy of building this vision, I hope you join me.

Like always, please feel free to leave comments below, I always respond.

Running DynamoDB Local service container on CircleCI 2.0

2018-07-18T11:56:00-04:00

tl;dr use a custom entrypoint in your CircleCI 2.0 config to limit Java memory to 1G.

The new CircleCI 2.0 docker configuration supports a "primary image" (listed first) which runs all the "steps" as well as zero or many "service images" (listed subsequently). The "service images", although not running in the same container as the primary present as if local to the primary.

It sort of feels like mixing many docker containers together.

Enough talk, in this example, I show how easy it is to run DynamoDB Local in a separate container but at the same time present it to the primary as 127.0.0.1:8000:

version: 2
workflows:
  version: 2
  tests:
    jobs:
      - test

jobs:

  test:

    working_directory: /go/src/github.com/remind101/r101-myapp

    docker:

      # Primary container image where all the steps run.
      - image: circleci/golang:1.8

      # Service container image made available to the primary container at `host: localhost`
      - image: dwmkerr/dynamodb:41
        # custom entrypoint to limit RAM to 1G to prevent OOM on CircleCI 2.0.
        # https://circleci.com/docs/2.0/configuration-reference/#docker--machine--macosexecutor
        entrypoint: ["java", "-Xmx1G", "-jar", "DynamoDBLocal.jar"]

    steps:
      - checkout
      - run: make test_verbose

Troubleshooting

When I first tried to use this image, any test which tried to reach out to DynamoDB via 127.0.0.1:8000 had the following exception:

Test Panicked
Error creating table group_associations: RequestError: send request failed
caused by: Post http://127.0.0.1:8000/: dial tcp 127.0.0.1:8000: getsockopt: connection refused
/usr/local/go/src/runtime/panic.go:489

It seems the docker-dynamodb (DynamoDB Local) container failed to start and exited like this:

Initializing DynamoDB Local with the following configuration:
Port: 8000
InMemory:     false
DbPath:       null
SharedDb:     false
shouldDelayTransientStatuses: false
CorsParams:   *


Exited with code 137

Apparently Exit code 137 is a classic Docker + Java error code caused by an OOM (out of memory).

By default, projects on CircleCI build in virtual environments with 4GB of RAM but this is shared and Java acts greedy when inside a container so it needs a limit by adding -Xmx1G to the entrypoint!

References:

Contents

Troubleshooting

Pyramid SQLAlchemy bootstrap console script with transaction.manager

2018-06-15T10:54:00-04:00

So I've struggled for a while with the best way to properly setup a console script for my SQLAlchemy Pyramid apps.

I use the Pyramid Cookiecutter Alchemy to setup my projects and as such, I do not have a global and thus importable DBSession object. Instead my database session is attached to the request on creation.

Anyways, here is my recipe:

import argparse

from pyramid.paster import bootstrap, setup_logging

from remarkbox.models import invalidate_all_node_cache_objects


def get_arg_parser():
    parser = argparse.ArgumentParser(
        description="Invalidate all NodeCache objs to force recomputation."
    )
    parser.add_argument("-c", "--config", default="development.ini")
    return parser


def main():
    parser = get_arg_parser()
    args = parser.parse_args()
    setup_logging(args.config)

    # use bootstrap context manager to prepare app and request,
    # next use the resulting request's transaction manager!
    with bootstrap(args.config) as env, env["request"].tm:
        request = env["request"]
        invalidate_all_node_cache_objects(request.dbsession)

This pattern should help you solve this error:

NoTransaction error when using bootstrap in script

Quickstart to DKIM Sign Email with Python

2018-06-04T09:03:00-04:00

For a long time I have put off DKIM signing email sent from my web services because I couldn't wrap my head around all the indirection Postfix requires to make it work.

Honestly, I put it off for over 5 years...

Today a thought sprung into my head:

"Could I sign Email at the application level before passing to Postfix?"

As you may know, I primarily use the Python programming language, so I did some research and found a reference to a single library called dkimpy (previously pydkim). The codebase started over 10 years ago and appeared stable and mature.

The part that sold me was that dkimpy seemed compatible with the two Python standard library modules which I already use:

email which I use to prepare messages
smtplib which I use to transport messages to Postfix running on localhost

One issue I did have with dkimpy was the complete lack of examples or even a quickstart guide.

For this reason, I have written this post!

The Missing dkimpy Quickstart Guide

To install dkimpy you may use pip (requirements.txt) or in my case I added it to my setup.py.
Generate a public / private keypair. Don't let this step trip you up, the process easy. In a Unix -like environment you may run the following commands to create the keys.

generate private key (I name my file after the domain and DKIM selector I plan to use).
```
openssl genrsa -out remarkbox.com.20180605.pem 1024
```
generate public key from the private key.
```
openssl rsa -in remarkbox.com.20180605.pem -out remarkbox.com.20180605.pub -pubout
```
Install the public key (.pub) as a DNS TXT record, where the record name ("selector") is 20180605._domainkey and the value body is:
```
v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDcplYPRsqIFwXuggtH2XgQDMX+e+6sGnWdV8ld/FR9zgRAxB+DeiCEVooVvYt2JRZUEokgDFvys82Q+JTbN4qHNz19bdcBGrnTsnIFaQYpgeQYmPLdDtcWRKzTYMRNCnRmmEXyGv7WIDcaTapIq9NFgLmy1QT7ZTxuNjQtDB/2LwIDAQAB;
```
You may choose any selector, I happen to like to use YearMonthDay. Additionally you will substitute your public key in place of mine. Put each the line of the public key on a single line in the TXT record.
On each of my application servers I store my private portion of my DKIM key in /etc/dkim/remarkbox.com.20180605.pem. You may store your key any where on the filesystem that is accessible to the user or group running the application.

This is how I used to send Email with Python:

import smtplib

from email.mime.multipart import MIMEMultipart

from email.mime.text import MIMEText

# catch socket errors when postfix isn't running...
from socket import error as socket_error


def send_email(
    to_email,
    sender_email,
    subject,
    message_text,
    message_html,
    relay="localhost"
):
    msg = MIMEMultipart("alternative")
    msg.attach(MIMEText(message_text, "plain"))
    msg.attach(MIMEText(message_html, "html"))
    msg["Subject"] = subject
    msg["From"] = sender_email
    msg["To"] = to_email
    # TODO: react if connecting to postfix is a socket error.
    s = smtplib.SMTP(relay)
    s.sendmail(sender_email, [to_email], msg.as_string())
    s.quit()
    return msg

This is how I now send DKIM signed Email with Python:

# ref: https://github.com/russellballestrini/miscutils/blob/master/miscutils/mail.py

import dkim
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
# catch socket errors when postfix isn't running...
from socket import error as socket_error


def send_email(
    to_email,
    sender_email,
    subject,
    message_text,
    message_html,
    relay="localhost",
    dkim_private_key_path="",
    dkim_selector="",
):

    # the `email` library assumes it is working with string objects.
    # the `dkim` library assumes it is working with byte objects.
    # this function performs the acrobatics to make them both happy.

    if isinstance(message_text, bytes):
        # needed for Python 3.
        message_text = message_text.decode()

    if isinstance(message_html, bytes):
        # needed for Python 3.
        message_html = message_html.decode()

    sender_domain = sender_email.split("@")[-1]
    msg = MIMEMultipart("alternative")
    msg.attach(MIMEText(message_text, "plain"))
    msg.attach(MIMEText(message_html, "html"))
    msg["To"] = to_email
    msg["From"] = sender_email
    msg["Subject"] = subject

    try:
        # Python 3 libraries expect bytes.
        msg_data = msg.as_bytes()
    except:
        # Python 2 libraries expect strings.
        msg_data = msg.as_string()

    if dkim_private_key_path and dkim_selector:
        # the dkim library uses regex on byte strings so everything
        # needs to be encoded from strings to bytes.
        with open(dkim_private_key_path) as fh:
            dkim_private_key = fh.read()
        headers = [b"To", b"From", b"Subject"]
        sig = dkim.sign(
            message=msg_data,
            selector=str(dkim_selector).encode(),
            domain=sender_domain.encode(),
            privkey=dkim_private_key.encode(),
            include_headers=headers,
        )
        # add the dkim signature to the email message headers.
        # decode the signature back to string_type because later on
        # the call to msg.as_string() performs it's own bytes encoding...
        msg["DKIM-Signature"] = sig[len("DKIM-Signature: ") :].decode()

        try:
            # Python 3 libraries expect bytes.
            msg_data = msg.as_bytes()
        except:
            # Python 2 libraries expect strings.
            msg_data = msg.as_string()

    # TODO: react if connecting to relay (localhost postfix) is a socket error.
    s = smtplib.SMTP(relay)
    s.sendmail(sender_email, [to_email], msg_data)
    s.quit()
    return msg

As always, leave a comment or contact me for questions or help.

Contents

The Missing dkimpy Quickstart Guide

Fulfilling Childhood Dreams: Solar

2018-03-16T08:03:00-04:00

Ever since I was an 8 year old boy I have wanted solar. I remember reading about the environment and alternative energy sources in a monthly "socal studies" flyer my school subscribed our classroom to. I questioned, even at my young age, why the world wasn't actively switching over to solar, water, and wind!

25 years later I have finally fulfilled one of my childhood dreams: I installed a 11.375kWh solar system on my roof.

This post will act as an overview of the stuff I learned about solar, I hope you learn something too!

Disclosure: This post contains affiliate links. See full disclosure page here.

Solar Monitoring

The system is composed 35 x Panasonic 325 watt panels (VBHN325SA16). The panels are isolated into 3 arrays ("strings") of 8, 13, and 14.

>>> # 35 panels x 325 watts ~= 11.375kWh
>>> 35 * .325
11.375

Each panel has an "optimizer" which communicates to the centralized SolarEdge inverter installed in my basement near the breaker box. The optimizers allow each array to work efficently even when a panel is shaded or broken.

Here is the layout of the panels on my house:

Additionally the optimizers emit energy production metrics to the inverter, which in turn forwards this data to "the cloud" (aka SolarEdge Monitoring System). From there, I can watch daily playbacks of the whole system.

For example, this playback from March 19th, 2018 was a good "solar day" as my family calls:

You can see how the sun rises to cover the 8 panels on the front of the house and then later moves to cover the panels on the back of the house.

SolarEdge Monitoring Dashboard also keeps track of various high level metrics. For example, I input my electricity rates with date ranges and the dashboard calculates the "lifetime revenue" of my solar system. I plan to track my ROI on this number!

Home Energy Monitoring

Most of my key insights into home energy actually come from another device I had installed called Curb Home Energy Monitor. Curb uses a bunch of non-invasive CT clamps on each breaker circut to gather consumption. Metrics are gathered and sent to "the cloud" and power some really cool realtime dashboards.

For example, this chart shows solar production versus consumption in dollars for the last 15 days:

As you can see, about 50% of my solar production is being consumed by heating hot water for a family of 5!

This feels absurd...

Update: I switched to a Hybrid Water Heater and wrote a post to show how I now use 69% less electricity when heating water!

Anyways, here is a short video showing the Curb user interface.

Sorry, your browser doesn't support embedded videos, but don't worry, you can download it and watch it with your favorite video player!

The Curb is likely the most accurate home energy monitor on the market. The draw back is the cost of parts and labor; I used my solar installer's electrician and the total cost was $800.

The competition has less accuracy but I could have likely installed a Sense myself and saved on the labor cost.

Here are the two other brands I was looking at, for reference:

Solar Financing

When talking with Tesla, I fully expected to pay for my solar system in full with cash savings. It wasn't until I reached out to SunLight Solar did I change my strategy. SunLight urged me to take advantage of the ".99% CT Green Bank" finance option. I took out a loan for the whole project and dumped my cash savings into my home mortgage as a bulk payment to the principle.

Additionally, the CT Green Bank granted me $3,600 toward my project and the US federal government will grant 30% or $8,400 on my next tax return.

After all the incentives, the parts and labor of my system came in just under $20,000:

$32,000 - $3,600 - 8,400 = $20,000

Putting solar on my house actually opened up my financial options and diversified my portfolio!

I now have:

a power plant on my roof with an expected 9-10 year ROI; after 10 years I'll be generating wealth, capital, and positive "cash flow" in the form of energy
paid down my 4.125% house mortgage by $35,000; saving tens of thousands over the life of the loan
increased the value of my house by $20-30,000; this is an asset I can sell with or without my house
shielded or insulated myself from electricity rate hikes; who knows what electricity will cost in 5 to 10 years

Contact me

As always, please feel free to leave comments below. I live in New England so you may also contact me to setup a time to tour my setup and ask questions. I look forward to meeting you!

Contents

Solar Monitoring
Home Energy Monitoring
Solar Financing
Contact me

How-to Work From Home

2017-11-09T09:51:00-05:00

"I work from home" — a phrase I have uttered hundreds of times and is often met with instant amazement, envy, or jokes about pants. My goal for this book is to teach you the hacks I have learned in my career to help you land your dream job. If you feel stuck in the job market and you don't want to relocate, this is the book for you.

Chapter 1: The Road to Remote

So you have made up your mind, you want to work from home.

The first step to attaining this goal is to make it a priority.

The suggestions and advice in this book come from my life experiences. Not all of my experiences will apply to you but I hope that we might find, together, at least a single thread of universal truth. In order to know, I think it makes sense to briefly review the anatomy of my life from childhood until now.

I'm currently 32 years old, but I started working with computers at a young age. As an elementary schooler, I had a bright childhood friend named Brian Brennan who although younger than I, was my first mentor regarding computers. At age 9 we played lots of video games, original playstation and Nintendo 64, as well as countless hours of computer games on his Gateway computer.

At one point Brian learned how to make the computer work for him instead of other the other way around. Together we started learning HTML and Javascript. I didn't have the Internet on my family computer at the time, but my parents did have a WebTV so I started building HTML web pages for things I enjoyed and used the WebTV to upload my pages Geocities.

Flash forward to age 13, I finally had dial-up. Now we didn't pay for dial-up, in fact my parents didn't know I was "connected" as we used to call it, but I was. I had the "setup" credentials for our regional phone company — I was living the dream of free Internet!

I covertly routed speaker wire from our pantry closet to the computer room. Yes, I learned early on that in a pinch speaker wire can work as telephone wire, although I don't recommend it. This was one of many hacks I stumbled on. Hacks, the combination of science, trial & error, and creativity, are a common theme through out my life.

I spent my highschool career building HTML pages instead of powerpoints slides for school projects. At home I built my own computers out of parts, as so many of us did back then and gamed all night during LAN parties where we traded "warez" and "pr0n". I was 15 when we got an A-DSL connection (125kbps down / 15kbps up) and I started hosting my websites from my house on a Windows server hiding in my bedroom closet — later this server was reborn running FreeBSD 3.

Most of my highschool education was spent figuring out how to get out of class and either into the computer lab or the library with friends. We basically met as an informal computer and tech club. For more details about this era, checkout my long time friend Charles Hooper's blog post titled "How I hacked my high school".

When I was going to community college for Computer Science, I worked two part time jobs — I worked for my parents and separately at a large retail chain in the electronics department, where I slung "HDTVs". I was working 40 hours a week, doing 4 courses a semester, and I didn't have time for anything.

I started hosting my parent's small family business website which I coded by hand and I mostly resented my mall job. I hated how cruel companies were and how they actively shit on their employees. I spent my lunch breaks in Borders reading computer and business books and dreaming of my perfect job.

In between Calculus and selling TVs, I also found time to build the best damn real estate search engine a local agency in South Eastern Connecticut could buy. I was taken advantage of; I worked at least 500 hours on that site which was commissioned for under $500. I did however learn a lot about business and tech. I was brave and dumb and took on more then I could chew but it was successful. Hands down it was the best solution on the market at the time, this was long before Zillow existed.

While working in the retail store, I met George, an IT guy working for my region's pharmaceutical company. George referred me to a job making $18/hr migrating Windows 2000 computers to Windows XP. I accepted the contracting job thinking I was on the top of the world, I finally had a "real" computer job.

I did awesome work but what I didn't know was this world is cruel to contractors. They don't get sick time, they don't get vacation, and they certainly don't get respect. Contractors are fed lies and false promises and they are the first to get cut. Managers treat contractors like physical resources. It's honestly sick.

From there I moved to a technical call center helpdesk, also as a contractor, of a contractor, of a contractor of the DoD. If you follow the money, I was making $17/hr but at the top of the pyramid they were billing me out at $55/hr. After some time I managed to move onsite and successfully cut out one of the contractor layers.

I still lacked negotiation skills and confidence so my increase was only a dollar, bringing me to $18/hr as a desktop support tech. I fixed computer problems whether it was hardware or software and I was fast and could basically solve anything handed to me. I quickly transitoned to a desktop support engineer role where I learned how to package and push software to small fleets of computers. A couple years later I transitioned to the Unix server team.

Each step was a very big promotion as far as responsibility (I was one of two employees who designed and operated a multi million dollar Hitachi SAN) but my salary didn't increase proportionally. People were coming "off the street" for the same or lower position and making more then double I was. At the time I was working my ass off for $32k a year and hardly had enough for what my family needed. I was watching people get hired for the same job for $65k a year ... After being let down for so long, this was the tipping point, this is where I started to shift my thinking.

The only way to move up is to move out.

I felt stuck. I worked for both the big industries for my region and they both failed me. I knew I did not want to relocate but I also didn't want to work for companies with obscene priorities and worse culture.

I started moonlighting on side projects. I had so many flops and failures, to many to count. Finally I launched an idea with traction called LinkPeek. During the code development phase of building LinkPeek I became active in the open source community for the framework I was using called Pyramid.

I was networking, even though I didn't know what it was called at the time. I taught myself some tricks to career development and ended up meeting a hiring manager who appreciated my skill and was willing to take a chance with me working remote. I flew across the country to Santa Monica and had a bit of culture shock but I landed the position!

This victory would be the first of 4 remote positions I have held since finally taking the leap to "work from home".

Tune in to the next chapter where I describe how I positioned myself to win the interview.

Contents

Chapter 1: The Road to Remote

webwords: a minimal viable web app with docker in as many languages as possible

2017-10-24T12:11:00-04:00

The companion webwords git repo lives here.

This project shows how to code the same minimal web app called webwords in as many different programming languages as possible. It also provides guides for building and running webwords as a docker image.

what is webwords

A simple web application whose spec accepts the following two query parameters —

keyword:: The keyword you want to search for.
target:: The URI target that you want to search.

The application always returns an HTTP 200 response with the string true or false depending on if the keyword is found in the target web page body.

For example, to see if the word potato exists on Remarkbox, put the follwing in a browser:

http://127.0.0.1:32779/?keyword=potato&target=https://www.remarkbox.com

Spoiler:: potato does exist on Remarkbox, : )
Note:: You will need to replace the port of 32779 with the port from the docker ps output.

why is webwords

Webwords started as a programming Kata to practice writing code in different programming languages. Each port of webwords should behave the same to make comparing and functional testing simple.

why did you choose this programming problem?

I think the spec of webwords is small enough for people new to any language to digest but complete in that it does something useful and demonstrates two common tasks: running an HTTP server and using an HTTP client.

Also, I needed a way to verify if a user had possession of a domain name for the comment service I'm building and chose to code this verification program as a micro service, first with Python and later with Go. The tiny end result was webwords.

Shortly after, during a hackathon I used webwords to learn how to build Docker images for various languages and formalized the idea into a single project.

What's next?

Webwords is for tinkering. If you want to add a version or touch up an existing version, send a PR. Maybe a future fork will show a guide for adding a cache layer or teach how to add logging or gather metrics.

go

To build the docker image:

cd go
docker build -t webwords-go .

To run a test container from the new image:

docker run -d -p 8888 webwords-go

python

To build the docker image:

cd python
docker build -t webwords-python .

To run a test container from the new image:

docker run -d -p 8888 webwords-python

ruby

To build the docker image:

cd ruby
docker build -t webwords-ruby .

To run a test container from the new image:

docker run -d -p 8888 webwords-ruby

js

To build the docker image:

cd js
docker build -t webwords-js .

To run a test container from the new image:

docker run -d -p 8888 webwords-js

debugging

If you're anything like me, your programs rarely compile or work properly on the first try. Just like with programming, a docker image will rarely build correct the first time so you will need to learn how to debug.

To debug, get the failed docker container's id:

docker ps --all

Once you have the id, you can run the following to see the error:

docker logs <container-id>

Debug the issue, fix your Dockerfile, and retry the build process until you have it working.

You can delete old attempts by running:

docker rm <container-id>

Contents

what is webwords
why is webwords
go
python
ruby
js
debugging

So You're Planning a Beta Test

2017-10-22T13:20:00-04:00

I'm running a beta for Remarkbox and here is my biggest take away:

Always collect information from potential customers as soon as possible. Catch customers while they pursue a solution to their problems.

What do I mean?

Well for starters, don't just ask for an email address, have them fill out a short survey right away. If you only ask for an email, your out of luck if you need more information; people do not often respond later.

I know, I know ... nobody likes to fill out surveys, so power-up their motivation with an incentive or discount. Use this survey to earn qualifed leads, remember quality over quantity.

Come up with a few questions. Keep most of the questions open ended to allow them guide the dialog. Later, use the responses to break the ice when you send one-on-one emails. Each lead will present a unique situation, so take advantage of this and keep your on boarding emails personal!

tl;dr - Gather interest for the Beta as soon as possible and ask questions up front as part of the sign up.

My Search for the Perfect Alternative Hosted Comment System

2017-10-08T10:47:00-04:00

Nearly 6 years ago I launched the beta for LinkPeek, a web page screenshot service. After all this time, I'm writing to share something new I've been passionately working on called Remarkbox.

I started working on Remarkbox back in 2014 to solve a problem I had after moving my Wordpress blog to a static site. I knew my readers would still want to communicate with me so I started the search for the perfect hosted comment system to promote discussion.

My research found solutions which would slow down my site, or worse, serve ads! That very day, I set out to build my own solution because -

"How long could it take to build a comment system?" ...

3 years later, I'm finally ready to share Remarkbox: a hosted comment service that embeds in your pages to keep the conversation in the same place as your content.

Remarkbox increases user engagement because it:

allows a visitor to discuss your content right away without an account
supports Markdown with real-time comment previews
supports deeply nested replies and has an orderly user interface

... and the best part - fast page load speeds and absolutely NO ADS!

I'm in the process of hand selecting group of users to help me test. Would you be interested in joining a list to hear more about it?

Yes! I would like to follow the Remarkbox journey.

Selenium grid on Kubernetes

2017-03-23T16:48:00-04:00

This post continues from where we left off on the Minikube guide. If you do not already have a Kubernetes cluster, you should read that first.

Selenium Grid allows you to build a cluster of Selenium nodes. Today we will create a Selenium cluster with 1 hub and 4 nodes on Kubernetes.

Selenium Hub

Let's launch the hub:

kubectl get pods
kubectl run selenium-hub --image selenium/hub:2.53.1 --port 4444
kubectl get pods

To access the deployment externally, we need to expose it:

kubectl get services
kubectl expose deployment selenium-hub --type=NodePort
kubectl get services

My deployment was exposed here:

http://192.168.99.100:31136/grid/console

To find out where your deployment was exposed:

minikube service selenium-hub --url

You may open this URI in a web browser.

Selenium Hub Overlearning

You can skip this section or try it for extra credit.

We may use kubectl exec for container introspection:

kubectl exec selenium-hub-3216163580-7pqtx -- ps aux

As you can see we have a java process running

kubectl exec selenium-hub-3216163580-7pqtx -- ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
seluser      1  0.0  0.1  18044  2688 ?        Ss   17:20   0:00 /bin/bash /opt/bin/entry_point.sh
seluser      7  0.1  3.1 2928868 64864 ?       Sl   17:20   0:13 java -jar /opt/selenium/selenium-server-standalone.jar -role hub -hubConfig /opt/selenium/config.json
seluser     87  0.0  0.1  34424  2856 ?        Rs   20:54   0:00 ps aux

We may also look at the config file and test the service internally:

# inspect the selenium config file.
kubectl exec selenium-hub-3216163580-7pqtx -- cat /opt/selenium/config.json

# see if selenium is really listening on port 4444.
kubectl exec selenium-hub-3216163580-7pqtx -- wget 127.0.0.1:4444 -O -

You can even shell into the container:

kubectl exec -it elenium-grid-3216163580-7pqtx -- /bin/bash

Exit out of the container and lets setup some Selenium Nodes!

Selenium Node

Lets spin up a Selenium Chrome node:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"
kubectl get pods

Kubernetes will use service discovery to resolve selenium-hub to the service (pods) running the hub!

If you refresh the hub browser window, you should see a connected Chrome Node, like this:

Selenium Node Overlearning

You can skip this section or try it for extra credit.

The first time I tried to launch a Selenium node and I had trouble.

I ran this:

kubectl get pods
kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1
kubectl get pods

The new pod went into status CrashLoopBackOff:

NAME                                    READY     STATUS             RESTARTS   AGE
selenium-grid-3216163580-7pqtx          1/1       Running            1          3d
selenium-node-chrome-4019562870-mcpfg   0/1       CrashLoopBackOff   6          6m

To troubleshoot, I used the following commands:

kubectl describe pod selenium-node-chrome

This command allowed me to review the Kubernetes level logs. Everything seemed healthy so next I looked at the Docker level logs:

kubectl logs selenium-node-chrome-4019562870-mcpfg
Not linked with a running Hub container

Ok, the error Not linked with a running Hub container appears when Selenium node cannot find the hub.

Docker has a --link flag to link containers together, Kubernetes doesn't have this. After some research, it seems --link manages ENV vars.

You can see the environment vars of a pod using this command:

kubectl exec selenium-hub-3216163580-7pqtx -- printenv

I learned that the selinum-node-chrome docker image expects some ENV vars and if it doesn't get them, it goes into a crash loop.

I reached out over IRC in the #kubernetes and #selenium channels to ask about the ENV vars needed. A really helpful user named smccarthy linked me to this:

https://github.com/kubernetes/kubernetes/tree/master/examples/selenium

Apparently one of the example Kubernetes clusters is a Selenium Grid setup!

Looking over the example, I found the ENV vars that the selenium-node containers expect: HUB_PORT_4444_TCP_ADDR and HUB_PORT_4444_TCP_PORT

Man, why would they put the port (4444) in the key?

Anyways, we pass these key/values when creating the container like this:

kubectl run selenium-node-chrome --image selenium/node-chrome:2.53.1 --env="HUB_PORT_4444_TCP_ADDR=selenium-hub" --env="HUB_PORT_4444_TCP_PORT=4444"

Selenium Scale

Now we can scale up and down the Selenium Grid cluster. Lets scale the deployment to 4 replica node-chrome pods.

kubectl get pods
kubectl scale deployment selenium-node-chrome --replicas=4
kubectl get pods

Finally, if you refresh the hub browser window, you should see 4 connected Chrome nodes!

If you liked this post, leave me a message in the comments!

Contents

Selenium Hub
- Selenium Hub Overlearning
Selenium Node
- Selenium Node Overlearning
Selenium Scale

Minikube

2017-03-21T12:38:00-04:00

Minikube allows you to run a self contained, single node, Kubernetes cluster on your workstation. Once installed and configured, you may use kubectl to interact with it, just like a production Kubernetes cluster.

Reference: https://kubernetes.io/docs/getting-started-guides/minikube/

install

Requirements: Virtualbox

install kubectl

Reference: https://kubernetes.io/docs/tasks/kubectl/install/

curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin

install minikube

Reference: https://github.com/kubernetes/minikube/releases

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

start

Supported hypervisors: virtualbox, vmwarefusion, kvm, xhyve

minikube start --vm-driver=virtualbox

If this is your first time starting minikube, it will perform the following:

Starting local Kubernetes cluster...
Starting VM...
Downloading Minikube ISO 89.24 MB / 89.24 MB [==============================================] 100.00% 0s
SSH-ing files into VM...
Setting up certs...
Starting cluster components...
Connecting to cluster...
Setting up kubeconfig...
Kubectl is now configured to use the cluster

Also, you should see a new VM running in VirtualBox, like this:

To verify that kubectl is configured to use minikube look at the config file (~/.kube/config).

Also try running:

kubectl get nodes
kubectl get services

You can also connect to the VirtualBox guest using SSH to have a look around. In my case the Minikube VM was assigned 192.168.99.100.

ssh -i ~/.minikube/machines/minikube/id_rsa docker@192.168.99.100

You can see all the containers running with:

docker ps
ps aux

Exit out, you really don't need to interact at this level

Instead we will treat Minikube as a "real" Kubernetes cluster and only use the kubectl tool.

demo

create a deployment

In this example we create an echoserver cluster.

kubectl run hello-minikube --image=gcr.io/google_containers/echoserver:1.4 --port=8080

this command will create -

1 deployment:

kubectl get deployments

1 replicaset:

kubectl get replicasets

1 pod:

kubectl get pods

To make the echoserver accessible externally, you need to expose the deployment, like this:

kubectl expose deployment hello-minikube --type=NodePort

The expose command creates -

1 service:

kubectl get services
NAME            CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes      10.0.0.1     <none>        443/TCP          2d
hello-minikube  10.0.0.225   <nodes>       8080:31136/TCP   58m

To access the service, you connect to the Minikube's IP address on the exposed port.

In my case the Minikube VirtualBox IP is 192.168.99.100 and the exposed port is 31136 as listed above.

The minikube tool has a shortcut for this info, try:

minikube service hello-minikube --url
http://192.168.99.100:31136

Toss this into a web browser on your local machine and it should echo back!

scale a deployment

Scale up the deployment named hello-minikube by setting the number of replicas to 3:

kubectl scale deployment hello-minikube --replicas=3

verify:

kubectl get deployments
kubectl get pods

delete a deployment

trash this demo (delete the deployment, replicaset, pods, and service):

kubectl delete deployment hello-minikube

Contents

install
- install kubectl
- install minikube
start
demo

Nginx throw HTTP 503 maintenance JSON for all requests

2016-12-21T15:09:00-05:00

I found this technique after stumbling on Aaron Parecki's blog. You can read his post here:

Setting a custom json 503 error page in nginx with proper http and content-type headers

Lets pretend you have an API and you need to turn on maintenance for a major change. All your client's (phones, web frontends) know what to do when they get an HTTP 503 Error code with JSON body.

Now you want to cause all requests to get the following JSON -

/opt/maint/maint.json:

{
 "errorCode": "maintenance",
 "errorDesc": "We are currently performing scheduled maintenance. Please try again later."
}

The nginx configuration looks like this -

maint.conf:

server {
    listen          80 default;
    listen          [::]:80 default_server;
    server_name     _;

    root /opt/maint;

    add_header Retry-After 30 always;

    error_page 503 /maint.json;

    location / {
        # all API calls will miss try_files on purpose and return custom 503.
        try_files $uri =503;
    }

}

All API calls will miss try_files on purpose and return our custom 503 maint.json.

Proof that this method rocks, it supports GET, POST, PUT, UPDATE, and DELETE.

It also automatically serves the proper Content-Type header, in this case application/json -

curl -v -X POST --data "param1=value1&param2=value2" http://127.0.0.1:80/my-very-favorite-api-call

> POST /my-very-favorite-api-call HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1
> Accept: */*
> Content-Length: 27
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 27 out of 27 bytes
< HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.10.2
< Date: Wed, 21 Dec 2016 20:32:44 GMT
< Content-Type: application/json
< Content-Length: 150
< Connection: keep-alive
< ETag: "585addfe-96"
< Retry-After: 30
<
{
  "errorCode": "maintenance",
  "errorDesc": "We are currently performing scheduled maintenance. Please try again later."
}

Need to make sure you always set a custom header? Don't forget the always directive:

add_header Retry-After 30 always;

Capability driven Presentation

2016-12-18T18:21:00-05:00

A web page does not need to look the same on every browser or device. We cannot control the capabilities of a user's browser or device. As web designers, we have the duty to give the viewer the best experience possible. A user will come with what they have and we should do our best to accommodate.

TL;DR: Capability drives presentation.

Resilient Web Design

A resilient web page should:

have a single canonical URI
serve the same content, regardless of a viewer's capibilities
use the viewer's capibilities to gracefully enhance and/or downgrade the presentation of the content

For more background and history please read this free online book: https://resilientwebdesign.com/

Techniques

The js-only class

Purpose: hide HTML elements when Javascript does not work.

In this technique we:

use a <noscript> tag to nest a class named js-only
declare display: none in the js-only class to hide elements
apply the js-only class to any element which should hide without js

Note: The js-only class will only exists when Javascript does not work.

<noscript>
  <style>
  .js-only {display: none;}
  </style>
</noscript>

Now, suppose we have a notification message which clears when a user clicks 'x'. To clear the notification, we must use a Javascript onclick event handler.

Without the Javascript capability the notification cannot clear. Therefore, to prevent confusion we should remove the 'x' from the presentation.

For example, with Javascript the notification will look like this:

But without Javascript the 'x' is removed and the notification will look this:

To do this we assign the js-only class to our label holding 'x':

<div id="alerts">
{%- for message, level in request.session.pop_flash() %}
<div class="alert alert-{{level}}" onclick="this.style.display='none'" name="alert">
  <p>
    <label class="close js-only" alt="dismiss" title="dismiss">x</label>
    {{message}}
  </p>
</div>
{%- endfor %}
</div>

The content is the same, but we hide the interface based on the user's capability! This small change has a huge impact on the overall user experience.

We no longer present a broken button to the user, and I think that is worth the effort.

Contents

Resilient Web Design
Techniques
- The js-only class

Build RPM or DEB packages for Node.js using Jenkins and FPM

2016-12-01T15:30:00-05:00

This blog post assumes you already have:

a Jenkins master and none or many build servers
FPM installed on the build servers
Node.js installed on the build servers

I add jenkins-build.sh in the root of the Node.js code repo:

# example usage: JOB_NAME=example-api BUILD_NUMBER=101 bash jenkins-build.sh

# download and build nodejs application and 3rd party modules.
npm rebuild
npm install -l

version=$(cat package.json | python -c 'import sys, json; print json.load(sys.stdin)["version"]')

# change directory down below checkout directory
cd ..

# create an RPM using fpm.
#   JOB_NAME     = jenkins job name
#   BUILD_NUMBER = jenkins auto incremented build number
/usr/local/bin/fpm \
    -s dir -t rpm \
    --name "$JOB_NAME" \
    --iteration "$BUILD_NUMBER" \
    --version "$version" \
    --vendor "Example, Inc" \
    --rpm-user="node"  --deb-user="node" \
    --rpm-group="node" --deb-group="node" \
    --directories "/opt/$JOB_NAME" \
    "$JOB_NAME/$JOB_NAME.service=/lib/systemd/system/" \
    "$JOB_NAME=/opt/"

# make a copy of the rpm with generic name (without version or build).
cp *.rpm "$JOB_NAME.rpm"

# mv both the rpm and genericly named rpm to the workdir so jenkins can archive it.
mv *.rpm "$JOB_NAME"

Then in the Jenkins job, I have the following execute shell tasks:

# clean out old rpms.
rm -f *.rpm

# download all 3rd party Node.js modules and package into an installable RPM.
bash jenkins-build.sh

I then simply upload the rpm to an S3 repo which is accessible to my API hosts.

The same basic strategy may be used for other languages with subtle differences to the build script.

Register Super Powers with Pyramid add_request_method

2016-11-24T15:30:00-05:00

The Pyramid web application framework uses a request object to hold state regarding an inbound HTTP connection. A view must accept a request object as the first argument which makes it always available to our views and templates.

This behavior rocks, but Pyramid makes it even better by allowing us to enrich the request object!

As an example, lets pretend we want to randomly generate an integer from 1 to 999 and attach it to each request:

from pyramid.config import Configurator

from random import randint

def main(global_config, **settings):
    """ This function returns a Pyramid WSGI application."""

    # build app config object from ini.
    config = Configurator(
        settings = settings,
    )

   def add_random_number(request):
       """return a random number."""
       return randint(1, 1000)

   # register request methods.
   # each request instance will run these functions.
   # the result attaches to the request as an attribute.
   # cache the result with `reify=True` to prevent multiple computations.
   config.add_request_method(add_random_number, 'random_number', reify=True)

Now we have access to this attribute in our views and templates:

request.random_number

This strategy has a number of uses. For example, I use it to:

attach configuration settings and secrets like API keys
attach a user object, by querying the database for the user_id sourced from the cookie session
analyze requests for misuse like spam or flooding

What super powers do you register to your request? You should let the world know in the comments.

Another Pyramid related post: Sharing a Pyramid cookie with Flask or Tornado

Sharing a Pyramid cookie with Flask or Tornado

2016-11-21T18:35:00-05:00

Do you have a Pyramid application which authenticates users and uses a signed cookie as a session? Do you want to build a microservice using another framework and allow it to use the same cookie and session? Me too!

First we will review a bit of Pyramid code which describes the cookie session.

Teach Tornado how to read Pyramid cookies

In this section I'll show you how to access and deserialize the Pyramid cookie from a Tornado application.

To do this, I'm going to extend the Tornado Hello, world application:

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("Hello, world")

def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

This is a simple application which listens to port 8888 and serves the text Hello, world when / is requested.

Add the following imports:

# accessing Pyramid cookies.
from webob.cookies import SignedSerializer
from pyramid.session import PickleSerializer
from pyramid.compat import bytes_

For testing purposes, create a global serializer object:

# https://docs.webob.org/en/stable/api/cookies.html#webob.cookies.SignedSerializer
serializer = SignedSerializer(secret='test-secret', salt='pyramid.session.', serializer=PickleSerializer())

Adjust the get method in the MainHandler to look like this:

def get(self):
    raw_cookie = self.get_cookie('session', None)
    if raw_cookie is not None:
        session_data = serializer.loads(bytes_(raw_cookie))
        self.write(str(session_data))
        self.write(str("<br/>"))
    self.write("Hello, world")

The complete program follows:

import tornado.ioloop
import tornado.web

# accessing Pyramid cookies.
from webob.cookies import SignedSerializer
from pyramid.session import PickleSerializer
from pyramid.compat import bytes_

# https://docs.webob.org/en/stable/api/cookies.html#webob.cookies.SignedSerializer
serializer = SignedSerializer(secret='test-secret', salt='pyramid.session.', serializer=PickleSerializer())

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        raw_cookie = self.get_cookie('session', None)
        if raw_cookie is not None:
            session_data = serializer.loads(bytes_(raw_cookie))
            self.write(str(session_data))
            self.write(str("<br/>"))
        self.write("Hello, world")


def make_app():
    return tornado.web.Application([
        (r"/", MainHandler),
    ])

if __name__ == "__main__":
    app = make_app()
    app.listen(8888)
    tornado.ioloop.IOLoop.current().start()

Again, we are hardcoding the same secret. If you set everything up properly, loading http://127.0.0.1:8888 in a web browser should print the cookie session_data in plain-text.

In my testing, I saw my cookie and it looked like this:

(1479520270, 1479516714.062414, {'authenticated_user_id': 5, 'nodes_pending_verify': []})
Hello, world

Thats all for now, let me know what you think in the comments!

Contents

Setup Pyramid Signed Cookie Session
Teach Tornado how to read Pyramid cookies

My first Systemd Service Script and Override

2016-10-28T15:54:00-04:00

At work we mostly run Centos and I have some NodeJS services to deploy. I feel most familiar with Ubuntu / Upstart so this post serves as my notes on systemd.

In this contrived example, we define a service for our taco-api application. The taco-api source code lives in /opt/taco-api.

We manage a static .service file using a package or config management.

[/usr]/lib/systemd/system/taco-api.service:

[Unit]
Description=Node.js service for taco API

[Service]

Environment=NODE_ENV=development

ExecStart=/usr/bin/node /opt/taco-api/app.js

# Restart service after all crashes but wait 10 seconds between restarts.
Restart=always
RestartSec=10

# output stdout and stderr to syslog. (/var/log/[syslog|messages])
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=%N

# define the user and group to own the process.
#User=node
#Group=node

# change directory before running ExecStart command.
WorkingDirectory=/opt/taco-api

[Install]
WantedBy=multi-user.target

We also manage a static override.conf file using config management. In this file, we customize the environment variables present.

/etc/systemd/system/taco-api.service.d/override.conf:

[Service]
Environment=NODE_ENV=production

This allows us to only override and keep track of the deltas.

You can test, like this:

# emit the status of the service.
service taco-api status

# start the service.
service taco-api start

# look at the process list, note that node is running.
ps aux | grep node

# emit the status of the service.
service taco-api status

# start the service.
service taco-api stop

# look at the process list, note that node is not running.
ps aux | grep node

Thank you!

Set static hostname for RHEL Centos 7 on AWS

2016-09-06T11:18:00-04:00

This took me about 2 hours to figure out, hopefully it saves you time.

/etc/sysconfig/network:

HOSTNAME=desired-hostname.example.com

/etc/hostname:

desired-hostname.example.com

/etc/cloud/cloud.cfg:

preserve_hostname: true

If I swallow lots of air will I be lighter?

2016-06-22T19:17:00-04:00

After dinner tonight, Carter, my four year old asked me, "If I swallow lots of air will I be lighter?". I thought about the question for a moment and then told him it depends on what surrounds you.

If you are surrounded by:

air and you swallow lots of air you will not be lighter.
water and and swallow lots of air you will be lighter, relative to the water, and you will float better.
helium and you swallow lots of air you will become heavier, relative to the surrounding helium.

Whaaaaaat???!!?!1!

: )

Output all instance identifiers of an AWS VPC to JSON

2016-06-21T10:25:00-04:00

At work today I needed an easy way to collect private IP addresses of every instance in one of our production VPCs.

I ended up adding a tool to https://botoform.com to perform this task.

bf --profile <aws_profile> dump <vpc_name_tag> instances --output-format json

For example:

bf --profile customer3 dump prd instances --output-format json > ~/customer3-prd-instances.json

Checkout the Botoform Quickstart for installation instructions.

Set DNS resolver options

2015-12-16T10:17:00-05:00

I desire the following options in /etc/resolv.conf:

rotate timeout:1 attempts:1

Ubuntu or Debian

For Ubuntu or Debian based systems, place the options in /etc/resolvconf/resolv.conf.d/base:

rotate timeout:1 attempts:1

These options merge with the options from DHCP and end up in /etc/resolv.conf.

Then restart the host, or at least networking.

Redhat or Centos or Fedora

For Redhat, Centos or Fedora, add the following to /etc/sysconfig/network:

RES_OPTIONS="rotate timeout:1 attempts:1"

Then restart the host, or at least networking.

Contents

Ubuntu or Debian
Redhat or Centos or Fedora

Bind9 on Joyent Triton

2015-11-29T18:47:00-05:00

I have a single DNS server running on a KVM at DigitalOcean for $5/mo. I might move to Joyent and use 2 x 128M Ubuntu containers for $2.23/mo each.

In my first test Bind9 (named RNDC) ended up using 111M of memory on a 256M Ubuntu Triton container.

The large 111M memory footprint correlated with the amount of worker threads running. Bind9 determines the number of worker threads to manage by the number of CPUs detected by the OS.

In my case, Ubuntu detected 48 CPUs because Joyent containers run on bare-metal.

We can see this by running the following commands:

cat /proc/cpuinfo | grep processor | wc -l
48

sudo rndc status
version: 9.9.5-3ubuntu0.5-Ubuntu <id:f9b8a50e>
CPUs found: 48
worker threads: 48
UDP listeners per interface: 1
number of zones: 216
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 0
query logging is OFF
recursive clients: 0/0/1000
tcp clients: 0/100
server is up and running

To fix this, at least on Ubuntu, we need to pass -n 2 to limit the worker threads to 2.

For Ubuntu, edit /etc/default/bind9:

OPTIONS="-u bind -n 2"

Then restart the bind9 service and verify using sudo rndc status and free -m.

sudo service bind9 restart
sudo rndc status
free -m

Migrating from WordPress to Pelican

2015-11-15T15:39:00-05:00

Its finally happening. I'm moving this blog from WordPress to Pelican. This task has persisted on my TODO list for over two years.

During the process of the move, I'm going to use this post to dump hints:

WordPress XML to JSON

I wrote this tool to convert Wordpress XML dumps to JSON. The tool is opinionated and removes lots of data.

Pelican-import

A tool to convert WordPress .xml into .rst or .md files (ReStructuredText or MarkDown) is pelican-import.

I suggest checking it out, even if you do not plan to use Pelican as your static site generator.

Add date to post filenames

After using pelican-import I had about 150 .rst files and I decided to put the date in the filename, so I wrote this short bash script tool to do the renames:

files=`ls *.rst`

for file in $files:
  do
    the_date=`grep ':date:' "$file" | awk '{ print $2; }'`
    mv "$file" "$the_date-$file"
  done

Alter category to tags

category and tags have different meanings and assumptions between wordpress and pelican. As a result I decided to change all my categories to tags using this command:

sed -i'' -e 's/:category:/:tags:/g' *.rst

Alter attachment and image paths

fix paths to images / uploads to remove wp-content:

sed -i'' -e 's/\/wp-content//g' *.rst

Contents

WordPress XML to JSON
Pelican-import
Add date to post filenames
Alter category to tags
Alter attachment and image paths

Boto3 get main route table

2015-10-16T12:15:00-04:00

While developing Botoform I ran into an issue with Boto3 where I couldn't easily get the "main" route table of a VPC. I ended up adding a get_main_route_table method to do the duty.

def get_main_route_table(self):
    """Return the main (default) route table for VPC."""
    main_route_table = []
    for route_table in list(self.route_tables.all()):
        for association in list(route_table.associations.all()):
            if association.main == True:
                main_route_table.append(route_table)
    if len(main_route_table) != 1:
        raise Exception('cannot get main route table! {}'.format(main_route_table))
    return main_route_table[0]

That all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

List all installed package names in Python

2015-07-04T22:15:00-04:00

Here we define a function called pkgs. This function returns a list of package resource objects.

pkgs = lambda : list(__import__('pkg_resources').working_set)

We then define two more functions: pkg_names & pkg_versions

pkg_names = lambda : [x.project_name for x in pkgs()]

pkg_versions = lambda : [x.project_name + '==' + x.version for x in pkgs()]

Last we show how to use invoke these functions:

>>> pkg_names()
['ansible', 'pycrypto', 'PyYAML', 'Jinja2', '...truncated...', 'virt-back', 'Werkzeug', 'xmltodict']

>>> pkg_versions()
['ansible==1.7', 'pycrypto==2.6.1', '...truncated...', 'virt-back==0.1.0', 'xmltodict==0.9.2']

Filtering AWS resources with Boto3

2015-07-02T17:03:00-04:00

This post will be updated frequently when as I learn more about how to filter AWS resources using Boto3 library.

Filtering VPCs by tags

In this example we want to filter a particular VPC by the "Name" tag with the value of 'webapp01'.

>>> import boto3
>>> boto3.setup_default_session(profile_name='project1')
>>> ec2 = boto3.resource('ec2', region_name='us-west-2')
>>> filters = [{'Name':'tag:Name', 'Values':['webapp01']}]
>>> webapp01 = list(ec2.vpcs.filter(Filters=filters))[0]
>>> webapp01.vpc_id
'vpc-11111111'

You can also filter on the value of the 'tag-key' or the 'tag-value' like so:

>>> taco_key_filter = [{'Name':'tag-key', 'Values':['taco']}]
>>> nacho_value_filter = [{'Name':'tag-value', 'Values':['nacho']}]

You can also filter on multiple 'Values'. In this example want 2 VPCs named 'webapp01' and 'webapp02':

>>> filters = [{'Name':'tag:Name', 'Values':['webapp01','webapp02']}]
>>> list(ec2.vpcs.filter(Filters=filters))
[ec2.Vpc(id='vpc-11111111'), ec2.Vpc(id='vpc-22222222')]

You can also use the '*' wildcard to glob up results in your filter. In this example we want all 3 VPCs named 'webapp01', 'webapp02' and 'webapp03':

>>> filters = [{'Name':'tag:Name', 'Values':['webapp*']}]
>>> list(ec2.vpcs.filter(Filters=filters))
[ec2.Vpc(id='vpc-11111111'), ec2.Vpc(id='vpc-22222222'), ec2.Vpc(id='vpc-33333333')]

Thats all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

Working with botocore's ~/.aws/config

2015-07-01T18:14:00-04:00

I ran into a bug in botocore and this post will serve to document a work around as well as show how to use botocore session object to work with the values stored in ~/.aws/config.

Pretend you have an aws config with two accounts for two separate projects, like so:

*~/.aws/config:*

[profile project1]
account_id = 111111111111
aws_access_key_id=THISISNOTMYACCESSKEY1
aws_secret_access_key=THISISNOTMYSECRETKEY1
# Optional, to define default region for this profile.
region=us-west-1

[profile project2]
account_id = 222222222222
aws_access_key_id=THISISNOTMYACCESSKEY2
aws_secret_access_key=THISISNOTMYSECRETKEY2
# Optional, to define default region for this profile.
region=us-west-2

Now instead of using a single object, we create multiple objects, one for each profile we intend to use.

>>> import botocore.session
>>> session1 = botocore.session.Session(profile='project1')
>>> session2 = botocore.session.Session(profile='project2')
>>> session1.get_credentials().access_key
'THISISNOTMYACCESSKEY1'
>>> session2.get_credentials().access_key
'THISISNOTMYACCESSKEY2'

Also figured out how to get at the `account_id` integer:

>>> session1.get_scoped_config()['account_id']
'111111111111'

Here is another algorithm that returns a list of sessions objects, one for each profile listed in the config.

>>> import botocore.session
>>> sessions = []
>>> aws_config = botocore.session.get_session().full_config
>>> for profile_name in aws_config['profiles']:
...     session = botocore.session.Session(profile=profile_name)
...     sessions.append(session)

Thats all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

My Mentor

2015-06-21T13:36:00-04:00

The original purpose of this post was to recognize the teachers and mentors which have helped shape me. I planned to write about how Mr. Cassidy, my high school drafting teacher, taught me the importance precision. I prepared to explain how Mr. Mercuri, my college computer science professor, ignited my desire to engineer software. But I could not. I knew deep down that one person in particular affected me more then the rest.

This person could fix anything and could even talk through a broken heart. He didn't know the internals of computers but the hacker spirit sweats from his veins. He taught me how to problem solve, many times in unconventional ways. I once watched him clear brush with an old snow plow and cut our work day in half. If he didn't have access to a tool, he would furnish one out of raw materials and spare parts, like a real life MacGyver. With a little bit of thought and ingenuity he built anything he imagined.

Sometimes he would use tough love to prove a point. For example, when I had a sun burn in middle school, he taught me to push through pain and not let my team down even though my jersey was scratching my raw skin. He showed me how to work hard and how to have great work ethic. He taught me the importance of "saving for a rainy day" but also knew how to have fun and loved parties. He taught me self control and how to make sacrifices for payouts in the future.

He taught me the fundamentals of kindness and guided me on the righteous path. He gave great advice, and even though I didn't always listen, he respected my thoughts and wishes. If he ever caused an accident he would always admit fault. He emits honor, honesty, and courage and never lies to or manipulates the people around him. Although we have had our differences, I think about you each day.

I love you. Thank you and happy father's day, dad.

Change default gateway on all SmartOS Zones and KVM guests

2015-06-01T20:34:00-04:00

Today I needed to change the default gateway on every Zone and KVM guest on my SmartOS hypervisor because I switched my ISP and as a result my gateway changed from 192.168.1.254 to 192.168.1.1. After changing one guest I got lazy and put together this script.

update-all-gateways.sh

for VM in `vmadm list -p -o alias,uuid`

  do
    # create an array called VM_PARTS splitting on ':'
    IFS=':' VM_PARTS=($VM)

    # create some helper varibles for alias and uuid
    alias=${VM_PARTS[0]}
    uuid=${VM_PARTS[1]}

    mac=`vmadm get $uuid | json nics | json -a mac`

    echo "
{
   \"update_nics\": [
      {
        \"mac\": \"$mac\",
        \"gateway\": \"192.168.1.1\"
      }
   ]
}
" | vmadm update $uuid
done

[root@hypervisor /opt/setup-jsons/updates]# bash update-all-gateways.sh
Successfully updated VM 211b992b-a448-40b4-94c9-xxxxxxxxxxxx
Successfully updated VM 31baa6a5-aa98-4750-80df-xxxxxxxxxxxx
Successfully updated VM 65d176b4-c36d-4cbf-b6ed-xxxxxxxxxxxx
Successfully updated VM aa0f603c-9572-4cb0-b96f-xxxxxxxxxxxx
Successfully updated VM ad928301-f3e1-4fe8-a1c1-xxxxxxxxxxxx
Successfully updated VM b82a257e-5628-46db-aee4-xxxxxxxxxxxx
Successfully updated VM da72b638-51de-4d7d-9853-xxxxxxxxxxxx
Successfully updated VM ee42bf30-51ce-4ae2-915b-xxxxxxxxxxxx

You are welcome!

Also to change the global zone (head node) default gateway, edit /usbkey/config and then run the following commands:

[root@hypervisor /]# route delete default 192.168.1.254
delete net default: gateway 192.168.1.254

[root@hypervisor /]# route add default 192.168.1.1
add net default: gateway 192.168.1.1

[root@hypervisor /]# netstat -r

SmartOS Ubuntu guest, apt-get not working because IPv6

2015-05-09T20:29:00-04:00

Turns out I don't have IPv6 setup properly in my network so when apt attempts to connect to the Internet it tries IPv6 and fails.

To disable IPv6 on the ubuntu guest, add this to end of /etc/sysctl.conf and restart the guest:

sudo vim /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

This was a hacky work around mostly because although I desire to have IPv6 working, I desire to get this VM running more...

Thanks for reading, leave comments please.

A Python script which searches for available interpreters

2015-05-08T10:34:00-04:00

This post describes how to write a polyglot -- in this case a script which runs as valid Bash or Python, to search for available Python interpreters.

The script initially runs as Bash but upon finding a first match, the script will call itself again this time using the expected Python interpreter in interactive mode!

And now, for the polyglot code:

#!/bin/sh

# reinvoke this script with bpython -i, ipython -i, or python -i
# reference: https://unix.stackexchange.com/a/66242
''':'
if type bpython >/dev/null 2>/dev/null; then
  exec bpython -i "$0" "$@"
elif type ipython >/dev/null 2>/dev/null; then
  exec ipython -i "$0" "$@"
else
  exec python -i "$0" "$@"
fi
'''
from helpers import (
  base_parser,
  get_hvpc_from_args,
)
parser = base_parser('Interactive Python interpreter & connection to hvpc')
args = parser.parse_args()
hvpc = get_hvpc_from_args(args)
print('\nYou now have access to the hvpc object, for example: hvpc.roles\n')

In this case you can see that we setup the interactive interpreter's environment to create an hvpc (Husky VPC) object for exploration.

If you want a pure python version that doesn't use a bash/python polygot, checkout this code I wrote: https://github.com/russellballestrini/botoform/blob/master/botoform/plugins/repl.py

Migrating libvirt KVM guest to SmartOS KVM guest

2015-04-28T22:17:00-04:00

The following tutorial documents how to migrate a libvirt/KVM guest from Ubuntu to SmartOS.

akuma:

Ubuntu Hypervisor

guy:

SmartOS Hypervisor

sagat:

guest to migrate

These commands were run on akuma:

WORKDIR=/KVMROOT/migrate
sudo mkdir $WORKDIR
cd $WORKDIR

# this tool simply stops and tarballs up the qcow and xml for libvirt KVM guest.
sudo /usr/local/bin/virt-back --path=$WORKDIR --backup sagat

sudo tar -xzvf sagat.tar.gz

sudo qemu-img convert -O raw sagat.qcow2 sagat.raw
sudo qemu-img convert -O raw sagat-var.qcow2 sagat-var.raw

scp sagat*.raw root@guy:/opt/tmp

These commands were run on guy:

Create the following file on guy - setup-ubuntu-sagat.json:

{
  "brand": "kvm",
  "resolvers": [
    "192.168.1.22",
    "8.8.4.4"
  ],
  "ram": "4096",
  "vcpus": "4",
  "alias": "sagat",
  "hostname": "sagat",
  "nics": [
    {
      "nic_tag": "admin",
      "ip": "192.168.1.50",
      "netmask": "255.255.255.0",
      "gateway": "192.168.1.254",
      "model": "virtio",
      "primary": true
    }
  ],
  "disks": [
    {
      "boot": true,
      "model": "virtio",
      "size": 10240
    },
    {
      "model": "virtio",
      "size": 20480
    }
  ]
}

We then create a new KVM guest on guy:

[root@guy /opt/setup-jsons]# vmadm create -f setup-ubuntu-sagat.json
Successfully created VM aa0f603c-9572-4cb0-b96f-4c79eb431223

List out the virtual block devices on the new KVM guest:

[root@guy /opt/setup-jsons]# vmadm info aa0f603c-9572-4cb0-b96f-4c79eb431223 block
{
  "block": [
    {
      "device": "virtio0",
      "locked": false,
      "removable": false,
      "inserted": {
        "ro": false,
        "drv": "raw",
        "encrypted": false,
        "file": "/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk0"
      },
      "type": "hd"
    },
    {
      "device": "virtio1",
      "locked": false,
      "removable": false,
      "inserted": {
        "ro": false,
        "drv": "raw",
        "encrypted": false,
        "file": "/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk1"
      },
      "type": "hd"
    }
  ]
}

Stop the new guest, it needs to be off for the restore.

[root@guy /opt/setup-jsons]# vmadm stop aa0f603c-9572-4cb0-b96f-4c79eb431223
Successfully completed stop for VM aa0f603c-9572-4cb0-b96f-4c79eb431223

Use dd to:

write sagat.raw to disk0
write sagat-var.raw to disk1

dd if=sagat.raw of=/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk0 bs=1M
dd if=sagat-var.raw of=/dev/zvol/rdsk/zones/aa0f603c-9572-4cb0-b96f-4c79eb431223-disk1 bs=1M

Lastly start the new guest up, and check it out:

vmadm start aa0f603c-9572-4cb0-b96f-4c79eb431223

Setting region programmatically in Boto3

2015-04-24T14:07:00-04:00

At work I'm looking into the possibility of porting parts of our AWS automation codebase from Boto2 to Boto3. We desire to perform this port because Boto2's record and result pagination appears defective.

I started to familiarize myself with Boto3 by using the Interactive Python interpreter.

Here I show myself trying to connect to the RDS AWS endpoint following the docs:

>>> import boto3
>>> rds = boto3.client('rds')
Traceback (most recent call last):
...
NoRegionError: You must specify a region.

What, No region? OK - how do I set a region?

Well it turns out the docs want you to configure a region in a config file. This will not work for me, I need to set the region programatically...

So after stumbling around in the botocore source code I found the following solutions.

Solution 1 - Set region_name when creating client:

>>> import boto3
>>> rds = boto3.client('rds', region_name='us-west-2')

Solution 2 - Set default region_name on the session:

>>> import boto3
>>> rds = boto3.setup_default_session(region_name='us-west-2')
>>> rds = boto3.client('rds')

It seems Boto3 has two types of interfaces, clients and resources.

Clients:: return description objects and appear lower level. Description objects seem like AWS XML responses transformed into Python Dicts/Lists.
Resources:: return higher level Python objects and like Instances with stop/start methods.

At a quick glance, both clients and resources seem to properly implement pagination automatically!

>>> # client interface.
>>> ec2 = boto3.client('ec2', region_name='us-west-2')
>>> idesc = ec2.describe_instances()
>>> len(idesc['Reservations'])
273
>>> idesc['ResponseMetadata']
{'HTTPStatusCode': 200, 'RequestId': '7e431ff7-xxxx-xxxx-xxxx-xxxxxxxxxxxxx'}

>>> # resource interface.
>>> ec2 = boto3.resource('ec2', region_name='us-west-2')
>>> for instance in ec2.instances.all():
>>>     print instance, instance.tags

That all for now!

You should read my other Boto related posts for tricks to impress your friends. : )

Securely publish Jenkins build artifacts on Salt Master

2015-02-22T12:39:00-05:00

Do you want a secure setup for publishing and staging build artifacts from a Jenkins build server to a Salt Master? This guide describes my fully automated pipeline to transport binaries using Salt's encrypted "bus".

We start off with some Salt States to stand up a Jenkins build server "client":

jenkins/client.sls:

# https://russell.ballestrini.net/securely-publish-jenkins-build-artifacts-on-salt-master/
# manage jenkins user, home dir, and Jenkins "master" public SSH key.
jenkins:
  user.present:
    - fullname: jenkins butler
    - shell: /bin/bash
    - home: /home/jenkins

  file.directory:
    - name: /home/jenkins
    - user: jenkins
    - group: jenkins
    - require:
      - user: jenkins

  ssh_auth.present:
    - user: jenkins
    - name: {{ pillar.get('jenkins-public-key') }}
    - require:
      - user: jenkins

# Manage a script to push artifacts to Salt Master.
# Note: jenkins user should _not_ have ability to change this file.
/opt/salt-call-put-artifacts-onto-salt-master.sh:
  file.managed:
    - user: root
    - group: jenkins
    - mode: 755
    - contents: |
        #!/bin/bash
        set -x
        salt-call cp.push_dir "$PWD" glob='*.tar.gz'
        salt-call cp.push "$PWD/commit-hash.txt"
    - require:
      - file: jenkins

# Allow jenkins to run push script as root via sudo.
jenkins-sudoers:
  file.append:
    - name: /etc/sudoers
    - text:
      - "jenkins    ALL = NOPASSWD: /opt/salt-call-put-artifacts-onto-salt-master.sh"

On the Salt Master we must enable MinionFS and restart Salt Master process:

fileserver_backend:
  - roots
  - minion

file_recv: True

We then use this command as the last build task in every Jenkins build job:

sudo /home/jenkins/salt-call-put-artifacts-onto-salt-master.sh

This causes the file to be staged on the Salt Master on a successful build.

The Salt States to deploy the software to production, look something like this:

# extract tarball from Salt Master using MinionFS.
extract-tarball-for-mysite:
  archive.extracted:
    - name: /www/mysite
    - archive_format: tar
    - source: salt://ubuntu-jenkins.foxhop.net/home/jenkins/workspace/job-name/env.tar.gz
    - user: uwsgi
    - group: uwsgi
    # I want to always extract, not sure a better way.
    - if_missing: /dev/taco
    - require:
      - service: make-mysite-dead-for-release

I also have build triggers which monitor remote git/hg repos for changes. Pushing code triggers a build which tests my code base and securely publishes to my Salt Master. When the time comes to perform a release, all I have to do is run highstate, because the pipeline did all the other work for me!

Update

Special thanks to an anonymous commentor regarding how to increase security. I have updated the scripts accordingly.

Set postgres user password on PostgreSQL SmartOS Zone

2015-01-21T22:28:00-05:00

Connect to zone and determine the auto generated password for postgres user:

cat /var/svc/log/system-zoneinit\:default.log | grep PGSQL_PW

document the result and log into postgres with the following command, entering the password when prompted:

[root@psql ~]# psql --user postgres

Alter the postgres role's password:

postgres=# ALTER ROLE postgres UNENCRYPTED PASSWORD 'new-password';

Now exit (\q) then try to log in with the new password.

In my case I was setting up PostgreSQL for testing out Zabbix monitoring system. So I did the following as the postgres user:

postgres=# CREATE USER zab UNENCRYPTED PASSWORD 'zabby'
postgres=# CREATE DATABASE zab OWNER zab;

Risk, Process, and Balance

2015-01-10T22:35:00-05:00

The operations of a company will have intrinsic risk. Risk occurs each time we decide to take an action or an inaction. This means that anything we choose to do, or not do, has associated risk.

An organization which has an unhealthy aversion to risk has a much higher chance of failure. As time goes on the intolerance of taking on additional risk to accomplish goals, will render a company petrified. Stuck in the fear of change.

In order to become unstuck, a company and it's employees must become great at calculating risk and assessment. In order to make a choice on what to work on, and what to not work on, they will need to research and collect data and continuously track progress and feedback. The company must use this data and feedback to move uncalculated bad risk into calculated good risk. Simply thinking through the problem and change before hand can uncover ways to accomplish the end result in the least risky way.

The biggest objection to the risk assessment phase is typically time constraints. A great company will learn how to rapidly assess and reduce the risk of action. They reduce risk by: gathering requirements, planning, building process, using process to drive automation, and iterating on automation to speed up feedback loops. These fast feedback loops will allow the company and it's employees to fail smaller and more frequently and facilitate reflection to optimize the pipeline. This means even more calculated and low risk actions!

So, a company should add more process to reduce risk? Not quite.

A process, when left unchecked or added for the wrong reasons, will become more detrimental then blindly performing an action without calculating the risk. Below I outline the key differences between a good process and a bad process.

A good process will:

reduce risk but not at the expense of blocking or preventing progress
be fast to facilitate quick feedback loops
have potential for automation and must not be tedious or manual (think approvals)
promote small and frequent changes
support both fail forward and fail backward
only fail backward to reduce time-to-recover
not be pushed down from upper management, rather upper management should help set goals and requirements, while the team builds and decides on process
be questioned and reviewed often to promote iteration to find and fix inefficiencies

A bad process will:

block real work from getting done
prevent creative and innovative solutions
have a higher chance of being ignored, skipped, or held in contempt
punish people who take risks (make change) and promote people who do nothing
cause more issues then it solves
become a crutch or a security blanket (a false sense of security)

I feel strongly that the essence behind the Agile and DevOps movements comes from the constant battle of balancing risk with process. This constant desire for balance in the organization's environment will lead to innovative ideas, tools, and workflows. These ideas, tools and workflows can not function properly if transplanted into a company who does not have the desire to find the optimal balance. Put simply, a company cannot buy Agile or DevOps, that culture needs to grow from with in.

autofs /net automount stopped working

2014-12-21T22:38:00-05:00

So autofs randomly stopped working on one of my Ubuntu hosts (this issue has been found on Arch as well so its most likely a change upstream). I found this error in the logs:

attempting to mount entry /net/freenas.example.net
get_exports: lookup(hosts): exports lookup failed for freenas.example.net
key "freenas.example.net" not found in map source(s).
failed to mount /net/freenas.example.net

The fix was to replace the following line in /etc/auto.master from

/net   -hosts

to this

/net /etc/auto.net --timeout=60

and then restart autofs (sudo service autofs restart).

Custom Rundeck HipChat notification templates

2014-12-03T01:37:00-05:00

Today I built a GUI and workflow around Ansible using Rundeck. Tonight I started diving into sending HipChat notifications and after a bit of research, I managed to create a custom notification template for each Rundeck project.

Modify your project's configuration file, on Ubuntu it was in /var/rundeck/projects/pname/etc/project.properties, and add the following line to the bottom:

framework.plugin.Notification.HipChatNotification.messageTemplateLocation=/var/rundeck/projects/pname/etc/custom-hipchat-template.ftl

Note: replace pname with your project name.

I ended up producing a single line chat notification template. I uploaded it here to act as an example:

Custom Rundeck HipChat Notification Template

The template syntax and renderer is called FreeMarker. Rundeck and the HipChat plugin pass many different context Map hash objects to FreeMarker for use in the templates. In this case I display "VPC name" selected by the user when starting the job.

References:

Rundeck HipChat Notification Plugin
Rundeck HipChat Notification Plugin (User Guide and Default Template)

Build release pipelines on S3 with s3p

2014-11-24T14:39:00-05:00

This weekend I finished my first sprint on s3p which is a Python library and CLI application that manages release pipelines on AWS S3. I put a lot of effort into the readme.rst file, so look there for usage and examples.

The main purpose of s3p is to use code to enforce process when promoting releases in the pipeline. Another goal was to make the CLI tool dead simple to use. As a side effect, I ended up using composition to extend boto.s3's Key and Bucket classes to produce S3Release and S3Pipeline.

I plan to incorporate s3p into Teamcity build jobs. At work I would like to use s3p to replace bash+s3cmd in pipeline management. Once s3p has a bit more production use, I will likely sprint on it again.

Dealing with pagination in Python

2014-11-13T21:06:00-05:00

So I'm working with an API (AWS ElastiCache) that offers mandatory pagination of results. I need to get all results, so I took some time to work out this logic.

def combine_results(function, key, marker=0, **kwargs):
    """deal with manditory pagination of AWS result descriptions"""
    results = []
    while marker != None:
        result = function(marker = marker, **kwargs)
        marker = nested_lookup('Marker', result)[0]
        results += nested_lookup(key, result)
    return results

Not only is the AWS ElastiCache API paginated but it also appears deeply nested in lists and dicts.

I use this to burn it with fire:

def nested_lookup(key, dictionary):
    """Lookup a key in a nested dictionary, return a list of values"""
    return list(_nested_lookup(key, dictionary))

def _nested_lookup(key, dictionary):
    """
    Lookup a key in a nested dictionary, return value

    Authors: Dougles Miranda and Russell Ballestrini
    """
    if isinstance(dictionary, list):
        for d in dictionary:
            for result in _nested_lookup(key, d):
                yield result

    if isinstance(dictionary, dict):
        for k, v in dictionary.iteritems():
            if k == key:
                yield v
            elif isinstance(v, dict):
                for result in _nested_lookup(key, v):
                    yield result
            elif isinstance(v, list):
                for d in v:
                    for result in _nested_lookup(key, d):
                        yield result

The end result is we have access to paginated and deeply nested data with a simple to use function:

>>> from lib import combine_results, nested_lookup
>>> d = elasticache_connection.describe_cache_clusters()
>>> nested_lookup('CacheClusterId', d)
[u'demo04-a-redis', u'demo04-b-redis', u'demo06-a-redis', u'demo06-b-redis', u'test-a-memcached', u'test-b-redis', u'ops01-redis', u'qa01-redis', u'ops02-redis', u'qa02-redis', u'int01-a-redis', u'int01-b-redis', u'ops03-redis', u'ops04-redis']

Here are some unit tests to prove these functions work like expected:

from unittest import TestCase

from lib.util import (
  combine_results,
  nested_lookup,
  _nested_lookup,
)

def my_func_that_paginates(max_results=3, marker=0):
    """this function sort of mocks the paginated AWS description results"""
    data = [
      {'desired_key' : 0},
      {'desired_key' : 1},
      {'desired_key' : 2},
      {'desired_key' : 3},
      {'desired_key' : 4},
      {'desired_key' : 5},
      {'desired_key' : 6},
      {'desired_key' : 7},
      {'desired_key' : 8},
      {'desired_key' : 9},
    ]
    new_marker = marker + max_results
    if new_marker > len(data):
        # last page!
        page = data[marker:]
        return {'results' : page, 'Marker' : None}
    page = data[marker:new_marker]
    return {'results' : page, 'Marker' : new_marker}

class TestCombineResults(TestCase):

    def test_combine_results_returns_all_results(self):
        expected_set = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
        f = my_func_that_paginates
        result_set = set(combine_results(f, 'desired_key'))
        self.assertSetEqual(expected_set, result_set)

class TestNestedLookup(TestCase):

    def setUp(self):
        self.subject_dict = {'a':1,'b':{'d':100},'c':{'d':200}}

    def test_nested_lookup(self):
        results = nested_lookup('d', self.subject_dict)
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list(self):
        results = nested_lookup('d', [{}, self.subject_dict, {}])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list_in_dict_in_list(self):
        results = nested_lookup('d', [{}, {'H' : [self.subject_dict]} ])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

    def test_nested_lookup_wrapped_in_list_in_list(self):
        results = nested_lookup('d', [ {}, [self.subject_dict, {}] ])
        self.assertEqual(2, len(results))
        self.assertIn(100, results)
        self.assertIn(200, results)
        self.assertSetEqual({100,200}, set(results))

With this test, the steps of the algorithm looks like this:

{'Marker': 3, 'results': [{'desired_key': 0}, {'desired_key': 1}, {'desired_key': 2}]}
3
[0, 1, 2]
[0, 1, 2]
{'Marker': 6, 'results': [{'desired_key': 3}, {'desired_key': 4}, {'desired_key': 5}]}
6
[3, 4, 5]
[0, 1, 2, 3, 4, 5]
{'Marker': 9, 'results': [{'desired_key': 6}, {'desired_key': 7}, {'desired_key': 8}]}
9
[6, 7, 8]
[0, 1, 2, 3, 4, 5, 6, 7, 8]
{'Marker': None, 'results': [{'desired_key': 9}]}
None
[9]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
ok

Turn python dict into a key=value string and back again

2014-11-09T22:02:00-05:00

I'm currently refactoring a script that tags AWS resources and I came up with this one liner to generate pretty output. It basically turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2. Here is the code:

', '.join(['='.join(key_value) for key_value in {'a':'1','b':'2'}.items() ])

Oh, and if you like this, here is a function with additional functionality and protection:

def dict_to_key_value(data, sep='=', pair_sep=', '):
    """turns {'tag1':'value1','tag2':'value2'} into tag1=value1, tag2=value2"""
    return pair_sep.join([sep.join((unicode(key), unicode(value))) for key, value in data.items()])

Careful, this might blow up on dictionaries that nest other objects. Also here is a test:

def test_dict_to_key_value():
    data = {'tag1':'value1','tag2':'value2'}
    pretty_str = dict_to_key_value(data)
    assert('tag1=value1' in pretty_str)
    assert('tag2=value2' in pretty_str)
    assert('tag1=value1, tag2=value2' in pretty_str)
    not_as_pretty = dict_to_key_value(data,'x','x')
    assert('tag1xvalue1xtag2xvalue2' in not_as_pretty)

Here is the inverse, taking a list of key_value strings and returning a dictionary of the data:

def key_value_to_dict(key_value_list, sep='=', pair_sep=',' ):
    """
    Accept a key_value_list, like::

      key_value_list = ['a=1,b=2', 'c=3, d=4', 'e=5']

    Return a dict, like::

      {'a':'1', 'b':'2', 'c':'3', 'd':'4', 'e':'5'}
    """
    d = {}
    for speclist in key_value_list:
        for spec in speclist.strip().split(','):
            key, value = spec.strip().split('=')
            d[key] = value
    return d

And of course a test to prove it works how we expect:

def test_key_value_to_dict():
    key_value_list = ['a=1,b=2', 'c=3, d=4', 'e=5']
    desired_result = {'a':'1', 'b':'2', 'c':'3', 'd':'4', 'e':'5'}
    assert(key_value_to_dict(key_value_list) == desired_result)

Migrating MongoDB from Ubuntu to SmartOS

2014-10-11T20:52:00-04:00

First, I installed the mongodb 14.2.0 (uuid a5775e36-2a02-11e4-942a-67ae7a242985) dataset:

imgadm avail | grep mongo
imgadm import a5775e36-2a02-11e4-942a-67ae7a242985

Next, I launched a new zone with this image.

Then I grabbed the uuid of the zone (211b992b-a448-40b4-94c9-00fa82615cec) and I connected into the zone

zlogin 211b992b-a448-40b4-94c9-00fa82615cec

The zone automatically creates a username and password for admin and "quickbackup". You can find these passwords by running the following command inside the zone:

cat /var/svc/log/system-zoneinit\:default.log | grep -i mon

First thing I did was disable authentication by modifying /opt/local/etc/mongod.conf:

#auth = true
noauth = true

Then I restarted MongoDB to re-read its configuration:

svcadm restart mongodb

Next I attempted to restore the database BSON files, with the following command:

mongorestore --db=taco taco/taco.bson

But I got the following error:

connected to: 127.0.0.1
terminate called after throwing an instance of 'std::runtime_error'
  what():  locale::facet::_S_create_c_locale name not valid
Abort (core dumped)

After some research I learned that I needed to export the following variable before running the restore:

export LC_ALL=C

Set Root Password SmartOS Percona MySQL Zone

2014-10-11T00:12:00-04:00

I used project-fifo to launch the percona (14.2.0) MySQL dataset. I couldn't get into the MySQL instance so I reached out on IRC. Johngrasty, a friendly guy in the #smartos IRC channel, provided a command to display the randomly generated MySQL password emitted to the zone-init log:

cat /var/svc/log/system-zoneinit\:default.log | grep MYSQL_PW

I used this initial password to get into the mysql> shell and changed it with this SQL statement:

mysql> SET PASSWORD = PASSWORD('clear-text-password');

Johngrasty also supplied a snippet of JSON which shows how to declare root MySQL password:

{
  ... truncated ...
  "customer_metadata": {
    "salt-master": "10.0.0.101",
    "salt-id": "mysql1"
  },
  "internal_metadata": {
    "mysql_pw": "mypassword"
  }
}

I looked for help in the #project-fifo channel as to why the GUI does not work for assigning initial MySQL password. MerlinDMC, the author and operator of datasets.at, gave me the following command to run in the Global Zone to look at a particular zone's metadata:

cat /zones/uuid-of-zone-goes-here/config/metadata.json

Turns out the GUI is placing the "mysql_pw" parameter into "customer_metadata" instead of "internal_metadata" which is invalid.

rabbits

2014-09-18T21:11:00-04:00

reality shattered

2014-09-18T21:11:00-04:00

Heka, World2!

2014-08-23T22:54:00-04:00

This article expands on my “Hello World” for Heka blog post. Check that one out first if you are new to Heka.

In this guide we introduce using Heka over the network by utilizing two Hekad processes on localhost. For discussion purposes we name one of the Hekad processes "sender" and the other "receiver".

The "sender" will watch a log file and emit messages to localhost TCP port 9612.
The "receiver" will listen on localhost TCP port 9612 and emit message payloads to file.

hello-heka-file-in-tcp-out.toml (sender):

# watch /tmp/input.log and output to TCP port 9612 on localhost
[hello_heka_input_log]
type = "LogstreamerInput"
log_directory = "/tmp"
file_match = 'input\.log'

[tcp_out:9612]
type = "TcpOutput"
message_matcher = "TRUE"
address = "127.0.0.1:9612"
#encoder = "hello_heka_output_encoder"
#
#[hello_heka_output_encoder]
#type = "PayloadEncoder"
#append_newlines = false

hello-heka-tcp-in-file-out.toml (receiver):

# listen to TCP port 9612 and emit to /tmp/output.log
[tcp_in:9612]
type = "TcpInput"
parser_type = "message.proto"
decoder = "ProtobufDecoder"
address = ":9612"

[hello_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "hello_heka_output_encoder"

[hello_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false

Now that we have config files, let us start our hekad processes, Open three terminals.

in terminal 1:

sudo hekad -config=hello-heka-file-in-tcp-out.toml

in terminal 2:

sudo hekad -config=/tmp/hello-heka-tcp-in-file-out.toml

in terminal 3:
```
echo 'Heka, World2!' >> /tmp/input.log
```
in terminal 3:
```
cat /tmp/output.log
```

Again, like magic, the data echoed into input.log shows up in output.log. This time the data traveled over TCP between to separate Hekad processes. I leave changing the configuration to support separate hosts to the reader.

By default TCP sender encodes the message with Protobuf (ProtobufEncoder) and the TCP reciever decodes the message with Protobuf (ProtobufDecoder).

In my testing I decided to make the TCP sender use the PayloadEncoder and then instead of using a second hekad process, I used nc -l 9612 to listen on the port. When data was added to /tmp/input.log it showed up in the netcat terminal because hekad was watching the file and emiting just payload portion of the message to TCP 9612 which netcat was listening on. I left this configuration in the examples above, simply uncomment to reproduce.

Read this for more fun with netcat.

Mailpile Salt States for Ubuntu or Debian

2014-08-15T20:32:00-04:00

I wrote these Salt States to install Mailpile on an Ubuntu host. Fun fact, it took me 20 minutes to write these states and they worked the first time I ran them. Disclaimer - I used a throw away server and wasn't concerned that buckets of packages were installed to the system instead of using a virtualenv.

cd /opt/Mailpile
./mp --set sys.http_host=0.0.0.0
./mp

Then open a web browser the IP address of the host running the mp command and follow the prompts to setup the server/client/app.

mailpile/init.sls:

# Clone the source repository
mailpile-git-latest:
  git.latest:
    - name: https://github.com/pagekite/Mailpile.git
    - target: /opt/Mailpile

# install the system requirements
mailpile-system-packages:
  pkg.installed:
    - names:
      - make
      - python-imaging
      - python-lxml
      - python-jinja2
      - pep8
      - ruby-dev
      - yui-compressor
      - python-nose
      - spambayes
      - phantomjs
      - python-pip
      - python-mock
      - python-pexpect
      {% if grains['lsb_distrib_release']|float >= 14.04 %}
      - rubygems-integration
      {% else %}
      - rubygems
      {% endif %}

# install some python requirements with pip
mailpile-pip-packages:
  pip.installed:
    - names:
      - pgpdump
      - selenium >= 2.40.0
    - require:
      - pkg: mailpile-system-packages

# install some ruby requirements with gem
mailpile-gem-packages:
  gem.installed:
    - names:
      - therubyracer
      - less
    - require:
      - pkg: mailpile-system-packages

You can hack on FreeNAS 9

2014-05-15T01:06:00-04:00

This post analyses the FreeNAS 9 code base and discusses the various places users may feel confident to hack on.

FreeNAS uses the following software stack:

Django

A Python Web Application Framework which complies with WSGI

Nginx

A very fast web server which may act as a reverse proxy server for HTTP, HTTPS, SMTP, POP3, and IMAP protocols, as well as a load balancer and HTTP cache.

Dojo Toolkit

The Javascript toolkit used to create widgets and handle client side processing.

FreeBSD

FreeBSD is an advanced computer operating system used to power modern servers.

Want to hack on the frontend web application?

Start here if you enjoy Python, or if you really enjoy coding on Django applications:

https://github.com/freenas/freenas/tree/master/gui

Want to hack on the GUI?

Start here if you are a front end developer and enjoy writing HTML, CSS, and working with Javascript:

https://github.com/freenas/freenas/tree/master/gui/templates

Want to change Nginx?

Take a look here if you would like to review, change, or tune Nginx on FreeNAS:

https://github.com/freenas/freenas/tree/master/nanobsd/Files/usr/local/etc/nginx This directory holds the nginx "vhost" config files and CGI parameters.

Want to hack on the OS?

Start here, if you know about FreeBSD or operating systems in general:

https://github.com/freenas/freenas/tree/master/nanobsd This directory seems like a customized and completely version controlled nanoBSD install!

Nginx with SSL and mixed content errors with upstream WSGI servers

2014-05-08T16:28:00-04:00

Mixed content errors occur because Nginx (the front-end server) communicates to the upstream WSGI server using http. WSGI does not know (or care) about the SSL session between Nginx and the user. The WSGI server will naively generate URIs and serve assets as http.

To fix mixed content errors, we need to communicate the inbound request scheme or configure the WSGI server to always use https.

waitress

Waitress is meant to be a production-quality pure-Python WSGI server with very acceptable performance.

It commonly powers Pyramid and Substance D deployments.

To configure waitress to always use https in code:

from waitress import serve
serve(wsgiapp, host='0.0.0.0', port=8080, url_scheme='https')

configure waitress to always use https in paste deploy compatible configuration file:

[server:main]
host = 127.0.0.1
port = 6543
url_scheme = https

How to patch Heartbleed OpenSSL defect (libssl) on Ubuntu

2014-04-08T22:42:00-04:00

Lots of people claim that you need to upgrade openssl package, but this will not fix the issue.

The issue is not the openssl package, it is one of the libraries that the package relies on (libssl).

https://www.ubuntu.com/usn/usn-2165-1/

The output of openssl version -a command should have a built on date older then Mon Apr 7 20:33:29 UTC 2014. After patching openssl we still see the vulnerable date:

openssl version -a | egrep "OpenSSL|built"
OpenSSL 1.0.1 14 Mar 2012
built on: Tue Aug 21 05:18:48 UTC 2012

Now we patch libssl1.0.0:

sudo apt-get update
sudo apt-get install libssl1.0.0

Notice the patched built on date:

openssl version -a | egrep "OpenSSL|built"
OpenSSL 1.0.1 14 Mar 2012
built on: Mon Apr  7 20:33:29 UTC 2014

In my case I used a Salt remote execution to patch, verify, and restart nginx on all of my 14 hosts:

sudo salt '*' cmd.run 'apt-get update'

sudo salt '*' cmd.run 'apt-get -y install libssl1.0.0'

sudo salt '*' cmd.run 'openssl version -a | egrep "OpenSSL|built"'

sudo salt '*' service.restart nginx

xkcd 1353

IRC Bot (Foxbot) runs canned remote executions using Salt Stack

2014-03-15T16:33:00-04:00

I extended my IRC Bot Foxbot today to allow it to run canned remote executions on behalf of users in an IRC channel. This is only a prototype or proof-of-concept. Be very careful not to allow users to inject their own commands. Foxbot must be running on the Salt Master and must be running as the same user that runs the salt-master daemon.

The code lives here: foxbot/plugins/checks.py

Example usage:

16:18:25            * | russell checks uptime minion2.foxhop.net
16:18:25       foxbot | minion2.foxhop.net:  16:18:25 up 496 days, 22:36, 17 users,  load average: 0.33, 0.70, 0.87

16:29:10            * | russell checks procs *
16:29:17       foxbot | minion2.foxhop.net: PROCS WARNING: 165 processes
16:29:17       foxbot | minion5.foxhop.net: PROCS CRITICAL: 309 processes

Filter Salt Stack Return Data Output

2014-03-13T18:21:00-04:00

Sometimes you only want to see what has changed, and that is OK.

Create a file like this:

filter.py

#!/usr/bin/python

from json import loads
from json import dumps

import fileinput

stdin_lines = [line for line in fileinput.input()]

ret = loads(''.join(stdin_lines))

for minion_id, data in ret.items():
    print(minion_id)
    print('='*len(minion_id))
    for key, value in ret[minion_id].items():
        if value['changes'] or value['result'] == False:
            print('')
            print(dumps(value, indent=4))
            print('')

Make the file executable:

chmod 755 filter.py

Execute your remote execution like this:

sudo salt-call --out=json state.highstate | ./filter.py
sudo salt '*' --out=json  --timeout=60 --static state.highstate | ./filter.py
The flags --timeout=60 and --static will cause the Salt command to block until the specified seconds for each minion to return results. We then pipe the returned JSON into our filter.py script to filter out only the changes and failures!

Profit!

Change the conditional depending on what you want. For example, for just failures do this:

if value['result'] == False:

Example output:

# sudo salt 'graphite.foxhop.net' --out=json --static --timeout=60 state.highstate | ./filter.py

graphite.foxhop.net
===================

{
    "comment": "File /tmp/taco updated",
    "__run_num__": 15,
    "changes": {
        "diff": "New file",
        "mode": "0640"
    },
    "name": "/tmp/taco",
    "result": true
}

Replace the Nagios Scheduler and NRPE with Salt Stack

2014-03-08T15:53:00-05:00

Note: I will update this post as I progress.

So the idea is to use Salt Stack's remote execution to communicate with all nodes and run the Nagios checks and collect the return output instead of using the NRPE client/service protocol. This reduces the number of agents running on each host and appears significantly more secure. Salt Stack uses public/private crypto on top of the ZMQ publisher/subscriber model. This means the communication transport is very fast, very secure, and all nodes will run checks in parallel!

First, we need to install the Nagios checks and plugins on each host or minion.

I used the following State Formula on my Ubuntu and Debian hosts:

salt://nagios/plugins.sls

nagios-plugins:
  pkg:
    - installed

nagios-plugins-extra:
  pkg:
    - installed

salt://top.sls

base:
  '*'
    - nagios.plugins

I kicked off a highstate (salt '*' state.highstate) on all minions and eventually they all returned in the affirmative. We now have all the Nagios plugins and checks installed on each host.

This next part is FUN, We run a check on every host concurrently.

This particular check counts the number of processes running on each host and warns at 150 and criticals at 200.
salt '*' --out=json --static cmd.run_all '/usr/lib/nagios/plugins/check_procs -w 150 -c 200'
Output static JSON - Wait for all minions to return and then generate a single JSON object.
{
    "graphite.foxhop.net": {
        "pid": 24356,
        "retcode": 0,
        "stderr": "",
        "stdout": "PROCS OK: 78 processes"
    },
    "akuma.foxhop.net": {
        "pid": 4610,
        "retcode": 1,
        "stderr": "",
        "stdout": "PROCS WARNING: 158 processes"
    },
    "ken.foxhop.net": {
        "pid": 31254,
        "retcode": 2,
        "stderr": "",
        "stdout": "PROCS CRITICAL: 392 processes"
    }
}
Review the Nagios Plugin API for more information about return codes and STDOUT formats.

Last, we take this JSON data perform further processing or display it in a meaningful way.

For example, we could generate an HTML/JavaScript dashboard from the raw JSON objects. We could cut metrics out of the STDOUT and persist historic values in a data store. We could push the data into another system like Graphite or even send an email alerts.

This solution has all the perks of Nagios without ANY of the cons!

Returning the check output data to the CLI isn't super useful for further processing, so I hacked in a way to return data back to the master using the encrypted zeromq bus.

_returners/zeromq_return.py

# -*- coding: utf-8 -*-
'''
The zeromq returner will send return data back to the Salt Master over the
Encrypted 0MQ event bus with a custom tag for filtering on the other end.

Basically after the remote execution finishes, the ret data is "packaged" into
a special "envelope" which triggers the local Salt Minion Daemon to
forward the ret data to the Salt Master's event bus.

The "package" basically wraps the ret data and uses the tag 'fire_master'.

For example, a ret data object from the execution of test.ping
would be "packaged" like this::

  ret = {
    'graphite.foxhop.net': true
  }

  ret['tag'] = 'third-party'

  package = {
    'events': [ ret ],
    'tag': None,
    'pretag': None,
    'data': None
  }

The Salt Minion Daemon will forward this package to the Salt Master
where a 3rd party script may be filtering on the specified internal event tag.

To use the zeromq returner, append '--return zeromq' to the salt command. ex::

  salt --return zeromq '*' test.ping

TODO:

 figure out a way for user to define custom tag for filtering ...
 Most returners use the Salt Minion config file to supply returner
 details... that is not optimal, it would be ideal if the custom tag
 could be supplied on the CLI when the remote execution is run, like::

   --return=zeromq --tag=mytag

'''

# needed to log to log file
import logging

# needed for config to opts processing
import os
import salt.syspaths as syspaths
from salt.config import minion_config

# needed to send events over ZMQ
import salt.utils.event

log = logging.getLogger(__name__)

# needed to define the module's virtual name
__virtualname__ = 'zeromq'

def __virtual__():
    return __virtualname__


def returner(ret):
    '''
    Send the return data to the Salt Master over the encrypted
    0MQ bus with custom tag for 3rd party script filtering.
    '''

    # get opts from minion config file, supports minion.d drop dir!
    opts = minion_config(os.path.join(syspaths.CONFIG_DIR, 'minion'))

    # TODO: this needs to be customizable!
    tag = 'third-party'

    # add custom tag to return data for filtering
    ret['tag'] = tag

    # multi event example, supports a list of event ret objects.
    # single event does not currently expand/filter properly on Master side.
    package = {
      #'id': opts['id'],
      'events': [ ret ],
      'tag': None,
      'pretag': None,
      'data': None
    }

    # opts must contain valid minion ID else it binds to invalid 0MQ socket.
    event = salt.utils.event.SaltEvent('minion', **opts)

    # Fire event payload with 'fire_master' tag which triggers the
    # salt-minion daemon to forward payload to the master event bus!
    event.fire_event(package, 'fire_master')

Next we run a third party application on the Salt Master which subscribes to our events by filtering on the special tag ('third-party').

listen_to_master_bus.py

# event libary for events over ZMQ
import salt.utils.event

# create event object, attach to master socket ...
event = salt.utils.event.MasterEvent('/var/run/salt/master')

tag = 'third-party'

print('Listening for events tagged \'{}\' on Salt Master bus.'.format(tag))

# generator iterator yields events forever, we filter on tag
for data in event.iter_events(tag=tag):
    print(data)

This small application just prints the incoming return data, but it could easily be expanded to process the incoming return data and persist it somewhere.

Open two terminals on the Salt Master host.

# on terminal 1 run:
python listen_to_master_bus.py
# on terminal 2 run:
salt '*' --return zeromq cmd.run_all '/usr/lib/nagios/plugins/check_procs -w 150 -c 200'
Same remote execution check as before but now our new returner will make data appear in terminal 1!

This zeromq returner is more of a proof-of-concept. I think the salt remote execution command line tool should allow end-users to provide a --tag so that data may be feed directly back to a third-party script listening to the Salt Master's event bus which filters on the particular tag. My next step is to look into what it would take to build in this functionality.

In the future I want to rig up the Salt scheduler to invoke these remote execution checks on a steady and predictable cadence. Nagios historically runs checks every 5 minutes. The Salt scheduler will allow us schedule different checks with different frequencies. For example, I might want my load checks and metrics to be collected every 10 secs, but my disk capacity usage checked every 2 minutes. This fine-grain control is super powerful!

Configuration Management and the Golden Image

2014-02-21T17:19:00-05:00

When operations first became a thing, system administrators stood up servers using a base image from their favourite distribution. Things were done manually. Some administrators created their own distros, some wrote customised shell scripts to be run once-and-only-once to provision software and settings. This method worked, but it was slow, manual, and the human element caused defects. Then the request came in to stand up 100 servers the exact same way.

System admins resolved this large request by coming up with a Standard Operating Environment (SOE) image which would end up becoming the "golden image". This golden image was the source of truth, and we built hundreds, thousands of machines this way. All systems were the same, or rather started out the same, but it didn't take long for deviations occur.

(Norton Ghost, DD, ISOs)

https://en.wikipedia.org/wiki/Standard_Operating_Environment

The golden image by itself was a flawed idea. The task of creating a golden image was difficult and required a lot of work. Not only was it technical, but there was a lot of politics involved in what was worthy of inclusion. We didn't want to add cruft to every machine. Also the golden image was only updated every couple of years, so it would quickly become outdated and it still required provisioning scripts. Systems already in production didn't get configuration updates that were recently added to the golden image. There had to be a better way, and there was...

Some novel and smart system administrators who also knew how to program decided that they didn't want to maintain a golden image and deal with all the headaches involved and opted to use the light weight distribution image and then build sophisticated remote execution software to manage and maintain each server's configuration. Later on they built configuration management systems on top of this remote execution layer and were finally able to keep each system up-to-date, regardless of when it was deployed. They were geniuses and everyone who knew anything quickly rushed to implement remote execution and configuration management. It was the right way to manage servers, until the cloud drifted in.

(SaltStack, Ansible, Puppet, Chef, CFEngine)

In the day of cloud computing, we needed to scale up and down servers in seconds. A complex configuration manifest could take hours to run from start to finish. Configuration management was too slow. Each server needed to download, install, and configure the software stack, in real-time. Sure we could spin up multiple machines in parallel, but it was still slow. We started to look back at the golden age of the golden image, when a server was built and booted in moments. How could we pair the speed of the golden image with the flexibility of configuration management?

In the future could we use configuration management manifests and revision control to document how to build the golden image and then take a snapshot? Could we overlay multiple layers or dataset of images on top of each other? Perhaps we take a step way back and create golden images with a combination of a highly customizable distribution like Gentoo and configuration management.

(Joyant SmartOS zone datasets, Docker container images, Vagrant box files, AWS AMI, Digital Ocean Snapshots, etc)

Automatic Backups

2014-02-07T17:14:00-05:00

I maintain many tools for creating automatic backups for various systems. This page will act as a hub to each of those solutions.

tar-back:: tar-back is a backup utility to tar and gzip target filesystems. It supports a custom retention, filter exclusions, and backup directory.
virt-back:: virt-back virt-back is a python application that uses the libvirt API to safely shutdown, gzip, and restart guests. Supports KVM, QEMU, XEN, Virtualbox, and more.
mysql-back:: mysql-back is a backup utility script to dump (backup) and gzip every MySQL database on a host.
smart-back:: smart-back is a utility used to backup of every virtual machine on a SmartOS hypervisor.

tar-back

2014-02-07T16:49:00-05:00

tar-back is a backup utility to tar and gzip target filesystems.

It supports a custom retention, filter exclusions, and backup directory.

I use tar-back in combination with cron to perform regular backups of all localhost filesystems into /archive/fs. I then have a central long term storage server that collects the /archive partition from every host.

tar-back:

#!/usr/bin/env python

DESCRIPTION = """A backup utility to tar and gzip
target filesystems. Custom retention, filter exclusions and
backup directory.
"""

from os import path
from sys import exit
from shutil import move
from optparse import OptionParser
import tarfile

def filter_exclusions( tarinfo ):
    """Accept tarinfo, return tarinfo or None"""
    if o.filters:
        for filtr in o.filters:
            if filtr in tarinfo.name:
                return None
    return tarinfo

def exclude_exclusions( filename ):
    """Accept filename, return True or False"""
    """support for python 2.6 or lower"""
    if o.filters:
        for filtr in o.filters:
            if filtr in filename:
                return True
    return False

def file_rotate( target, retention = 3 ):
    """file rotation routine"""
    for i in range( retention-2, 0, -1 ): # count backwards
        old_name = "%s.%s" % ( target, i )
        new_name = "%s.%s" % ( target, i + 1 )
        try: move( old_name, new_name  )
        except IOError: pass
    move( target, target + '.1' )

if __name__ == '__main__':

    p = OptionParser()  # create an option parser object

    p.set_description( DESCRIPTION )

    p.add_option( '-t', '--targets',
      help='list of target filesystems to backup (coma delimited)',
      default=None, dest='targets', metavar='"/home"',
    )

    p.add_option( '-b', '--backup-dir',
      help='path to backup directory',
      dest='backdir', metavar='"PATH"',
    )

    p.add_option( '-f', '--filters',
      help='list of filter patterns to exclude (coma delimited)',
      default=None, dest='filters', metavar='".mp3"',
    )

    p.add_option( '-r', '--retention',
      help='backups to retain [default: 3]',
      default=3, type='int', dest='retention', metavar='amount',
    )

    o, args = p.parse_args()  # parse options and args

    extension = '.tar.gz'

    if not o.targets:
        exit( "missing filesystem target, run --help" )
    if not o.backdir:
        exit( "missing backup directory, run --help" )
    if not path.isdir( o.backdir ):
        exit( "backup-dir %s does not exist" % o.backdir )

    if o.filters:
        o.filters = o.filters.split(',')

    o.targets = o.targets.split(',')

    for target in o.targets:
        slug = target.replace( '/', '-' ).lstrip( '-' ) # change / to -

        if slug == '': slug = 'r' # / slug is empty (root)

        tarpath = path.join( o.backdir, slug + extension )

        if path.isfile( tarpath ):
            file_rotate( tarpath, o.retention )

        tar = tarfile.open( tarpath, 'w:gz' )
        try:
            tar.add( target, filter=filter_exclusions )
        except TypeError:
            tar.add( target, exclude=exclude_exclusions )
        tar.close()

mysql-back

2014-02-07T16:39:00-05:00

mysql-backis a backup utility script to dump (backup) and gzip every MySQL database on a host.

I use mysql-back in combination with cron to perform regular database dumps of MySQL servers to the /archive/db partition on localhost. I then have a central long term storage server that collects the /archive partition from every host.

mysql-back:

#!/bin/bash

USER="root"
PASSWORD=""
TODAY=`date +%Y-%m-%d`
OUTPUTDIR="/archive/db/$TODAY"
MYSQLDUMP=`which mysqldump`
MYSQL=`which mysql`
GZIP=`which gzip`

mkdir $OUTPUTDIR

# get a list of databases
databases=`$MYSQL --user=$USER --password=$PASSWORD \
 -e "SHOW DATABASES;" | tr -d "| " | grep -v Database`

# dump each database in turn
for db in $databases; do
    $MYSQLDUMP --force --opt --user=$USER --password=$PASSWORD \
    --databases $db > "$OUTPUTDIR/$db.sql"
done

# compress all of the files
$GZIP $OUTPUTDIR/*

# clean up all dumps 30 days old
find /archive/db -ctime +30 -type d | xargs rm -rf

If you ever need to restore, like in my case when I moved all my databases over to a brand new Percona MySQL SmartOS zone, I used the following command:

zcat *sql.gz | mysql -u root -p

The Three Deployment Management Strategies

2014-02-03T15:06:00-05:00

There are three deployment management strategies that could be used to maintain a system. Each has pros and cons which I outline in this document.

run once

A proceedure that is run once and only once to setup a system's configuration values and settings. A semaphore or flag generally blocks repeated executions to prevent an undesirable outcome.

Development Difficulty: LOW - binary state aware (has run/has not run)

run always

A proceedure that is safe to run multiple times but uses a brute force method to setup a system's configuration and settings. It will self-heal at the expense of clobbering existing state.

Development Difficulty: MEDIUM - not state aware - attention must be spent to allow multiple executions

run as needed

A proceedure that is safe to run multiple times and checks a system's configuration and settings before it sets a system's configuration or settings. "checks before it steps". It will self-heal only the values it needs to.

Development Difficulty: HIGH - state aware - must know how to both get and set values and make desicions based on what it finds.

How to reset HP iLO Lights-Out User and Password Settings with IPMItool

2014-01-25T09:18:00-05:00

Warning:: Do NOT follow guides that suggest to make a DOS boot disk, this is over complicated.

Use the ipmitool which ships with most Unix based operating systems. I tested on SmartOS and Ubuntu Linux. Use a live boot disk if you must.

Run ipmitool user list to list all users installed on the iLO.
Choose a user ID from the list and run ipmitool user set password [ID].
Last, make sure the user is enabled with ipmitool user enable [ID].

Here is a complete set of commands I used to reset user ID 6:

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user list
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
2   ROUSER           true    false      false      Unknown (0x00)
3   USERID           true    false      false      Unknown (0x00)
4   OEM              true    false      false      Unknown (0x00)
5   Operator         true    false      false      Unknown (0x00)
6   admin            true    false      false      Unknown (0x00)
... truncated ...
15  admin            true    false      false      Unknown (0x00)
16  OEM              true    false      false      Unknown (0x00)

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user set password 6 mynew-password

[root@1c-c1-de-f0-ad-36 /opt]# ipmitool user enable 6

Next I logged into the web GUI and cleaned up this crazy list of users. You may also need to change the privileges on a user ID to grant admin level access. Run ipmitool user for a list of possible sub-commands.

After I finished poking around with the web GUI, I decided to learn a bit more about IPMI and the ipmitool command.

I figured out how to get sensor information over the LAN using this command:

sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin sensor

I was even able to log into the server OS via Serial-over-LAN (SOL) using this command:

sudo ipmitool -I lanplus -H guy-ilo.foxhop.net -U admin sol activate

Serial-over-LAN completely resolves the need for a complicated JAVA/KVM (keyboard video mouse) setup, I was able to reboot the server and watch the machine POST over the serial connection! Now you don't need to shell out $229+ on an Advanced 1 yr single server Licence for HP Lights-Out 100i (LO100i)!

I uploaded a video showing Serial-Over-LAN with IPMItools to an HP Proliant DL160 G6 running SmartOS.

Update:

Documenting a few other useful commands for myself here.

Power on and off the server chassis:

sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power status
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power on
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power soft
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power off
sudo ipmitool -I lan -H guy-ilo.foxhop.net -U admin chassis power cycle

Update:

Apparently after you know the ILO username and password you may also use SSH to connect and manage the server:

ssh admin@guy-ilo.foxhop.net
admin@guile-ilo.foxhop.net's password:

Lights-Out 100 Management
Copyright 2005-2007 ServerEngines Corporation
Copyright 2006-2007 Hewlett-Packard Development Company, L.P.

/./-> help
Root Directory

/./-> show
    /./
    Targets
        system1
        map1

    Properties

    Verbs
        cd
        version
        exit
        show
        help

/./-> cd system1
/./system1/-> show
    /./system1/
    Targets
        oemhp_sensors
        oemhp_frus
        console1
        led1

    Properties
        name=DL180(Aspen)    _R
        enabledstate=enabled

    Verbs
        cd
        version
        exit
        show
        reset
        start
        stop
        help

You can even trigger the server OS to stop change run levels or mess with chassis power for more extreme measures.

/./system1/-> stop
System1 stopped.

Update:

I run FreeNAS on an HP DL180 G6 and just replaced my p410 controller with an LSI SAS9220-8i (IBM M1015) flashed to IT mode. The stock cables are long enough. I did not have issues with fans running at high RPM.

(I did the same replacement on an HP DL160 G6 running SmartOS and it didn't have a high fan RPM issues either)

How I added two Seagate 240G SSDs as SmartOS L2ARC

2014-01-05T14:56:00-05:00

How I added two Seagate 240G SSDs as SmartOS L2ARC

removed icepacks from two western digital velociraptors
installed ssds into icepacks
installed icepacks into HP hotswap trays
installed trays into HP prolaient g6 server

How to list all drive installed in Solaris, Open Solaris, or SmartOS

iostat -eE

format

AVAILABLE DISK SELECTIONS:
       0. c1t0d0
          /pci@0,0/pci103c,330b@1f,2/disk@0,0
       1. c1t1d0
          /pci@0,0/pci103c,330b@1f,2/disk@1,0
       2. c1t2d0
          /pci@0,0/pci103c,330b@1f,2/disk@2,0
       3. c1t3d0
          /pci@0,0/pci103c,330b@1f,2/disk@3,0

list zpools:

zpool list

NAME    SIZE  ALLOC   FREE  EXPANDSZ    CAP  DEDUP  HEALTH  ALTROOT
zones  1.81T  41.4G  1.77T         -     2%  1.00x  ONLINE  -

Add the two 240G SSDs as L2ARC devices:

zpool add zones cache c1t2d0 c1t3d0

Look at the iostats of the drives in the zpool

zpool iostat -v

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
zones       41.4G  1.77T      5     46  78.7K   730K
  mirror    41.4G  1.77T      5     46  78.7K   730K
    c1t0d0      -      -      0     15  41.5K   733K
    c1t1d0      -      -      0     15  41.6K   733K
cache           -      -      -      -      -      -
  c1t2d0    14.6M   224G      0      3  2.48K   273K
  c1t3d0    45.5M   224G      0      6  2.48K   852K
----------  -----  -----  -----  -----  -----  -----

This machine is a test hypervisor and after reviewing the output from zpool iostat -v over the last couple days, I'm pretty sure adding L2ARC to this box was not needed, but it was a great learning experience.

Test Game Engine with Python and SFML

2014-01-02T00:30:00-05:00

Over this holiday season, Christmas and New Years, I took the time to mess with some Game Development. I wrote a demo game engine using Python and SFML. I plan to use this post to track my progress.

Video Evolution Playlist

All videos

2014-01-01

2014-01-02

Heka, World!

2013-12-20T18:25:00-05:00

This post serves as a "Hello World" for the data collection and processing software called Heka. Heka is written in Go and was open sourced by Mozilla, the same fabulous group that brings us Firefox!

I intend to use Heka to replace Logstash agents by sending logs directly to ElasticSearch and continuing to use Kibana3 for visualizations. Also I aim to start collecting metrics and sending to a central Whisper back-end to fuel Graphite charts. All that we need to make Heka take on these responsibilities is one binary and some custom configuration.

Heka: Hello, World!

This mostly contrived example will show how to use Heka to watch /tmp/input.log and write to /tmp/output.log.

Step 1: install Heka

Compile from source or install the Heka package for your operating system.

Step 2: create a Heka TOML configuration file

/tmp/hello-heka.conf:

[hello_heka_input_log]
type = "LogstreamerInput"
log_directory = "/tmp"
file_match = 'input\.log'

[hello_heka_output_log]
type = "FileOutput"
message_matcher = "TRUE"
path = "/tmp/output.log"
perm = "664"
encoder = "hello_heka_output_encoder"

[hello_heka_output_encoder]
type = "PayloadEncoder"
append_newlines = false

Step 3: start Hekad and test!

in terminal 1:

sudo hekad -config=/tmp/hello-heka.conf

in terminal 2:
```
tail -f /tmp/output.log
```
in terminal 3:
```
echo 'Heka, World!' >> /tmp/input.log
```

Like magic the data appended to input.log will appear in output.log

Heka also has great docs and a number of input and output plugins, but don't take my word for it, try it yourself!

sensu-salt

2013-12-17T11:35:00-05:00

A while back I explained how to Create your own fleet of servers with Digital Ocean and salt-cloud. Today I will extend that post and show how I deployed a test environment for Sensu, an open source monitoring framework.

Before I test out new infrastructure software, I always attempt to write deployment manifests. I subject myself to this exercise for the following reasons:

to get better at configuration management (self growth)
to enable quick setup and tear down of test environments (save money)
to make sure the new software may be deployed and maintained in configuration management (a must)
to document the install process and leave comments (my notes double as automation scripts!)
to allow knowledge transfer (sharing is caring, you're welcome!)

The state formulas in this post were tested on Ubuntu 12.04.3 x64bit.

Clone or fork the Salt-State tree

https://github.com/russellballestrini/sensu-salt

git clone https://github.com/russellballestrini/sensu-salt.git

Declare deployment targets

Before deployment, we must declare some targets in the top.sls file:

'*':
  - git

'sensu-client*':
  - sensu.client

'sensu-server*':
  - rabbitmq.server
  - redis.server
  - sensu.server
  - sensu.client

Stand up Sensu test environment

I used the following salt-cloud command to create the sensu monitor server:

salt-cloud --profile ubuntu_do sensu-server && salt 'sensu-server' state.highstate

Once the sensu-server was live, I altered the client-config.json and modified the RabbitMQ host with the new sensu-server's IP or DNS record.

I then spun up 4 sensu-clients using the following command:

salt-cloud -P --profile ubuntu_do sensu-client1 sensu-client2 sensu-client3 sensu-client4 && salt 'sensu-client*' state.highstate

This caused 4 cloud servers to be spawned in parallel, and when the provisioning finished, they instantly appeared in the sensu-dashboard which runs on the sensu-server.

Tear down Sensu test environment

Later when I was done messing with writing checks, I used the following salt-cloud commands to destroy the sensu server and clients:

salt-cloud --destroy sensu-server
salt-cloud --destroy sensu-client1 sensu-client2 sensu-client3 sensu-client4

Hackathon 2013 Virtualization

2013-12-13T20:56:00-05:00

As a warning before we dive into things, this post is less of a formal publication and more of a stream of conscience.

My employer newcars.com has allowed the technical staff to host hackathon! Over the past couple weeks I have had quite a few ideas tumbling around in my head:

Standup a central logging server
Standup Sensu for monitoring
Bake-off and document some KVM virtualization hypervisors (Ubuntu or SmartOS)
Test Docker and document findings

Ultimately I have chosen to dedicate my time to testing out virtualization on the new Cisco UCS Blade servers. I plan to test Ubuntu KVM first.

KVM (Ubuntu 12.04)

I decided to use a Salt-stack configuration management formulas to document how I transformed kvmtest02 (a regular Ubuntu 12.04 server) into a KVM hypervisor. I use kvmtest0a and kvmtest02b when refering to the virtual machines living on the hypervisor. Here is the formula:

kvm/init.sls:

# https://help.ubuntu.com/community/KVM
# Ubuntu server a KVM - Kernel Virtual Machine hypervisor

# the official ubuntu document suggests we install the following
kvm-hypervisor:
  pkg.installed:
    - names:
      - qemu-kvm
      - libvirt-bin
      - ubuntu-vm-builder
      - bridge-utils

# we need to make all of our ops people part of the libvirtd group
# so that they may create and manipulate guests. we skip this for now
# and assume all virtual machines will be owned by root.

# these are optional packages which gives us a GUI for the hypervisor
virt-manager-and-viewer:
  pkg.installed:
    - names:
      - virt-manager
      - virt-viewer
    - require:
      - pkg: kvm-hypervisor

# create a directory to hold virtual machine image files
kvm-image-dir:
  file.directory:
    - name: /cars/vms
    - user: root
    - group: root
    - mode: 775

I used the following to install the formula to the test hypervisor:

salt 'kvmtest02.example.com' state.highstate

The salt highstate reported everything was good (green), so I moved on to setting up the bridge networking interface. At this point I don't want to figure out the logistics setting up the network bridge in configuration management, so I simply manually edited /etc/network/interfaces to look like this (substitute your own network values):

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#iface eth0 inet manual

# https://help.ubuntu.com/community/KVM/Networking
# create a bridge so guest VMs may have their own identities
auto br0
iface br0 inet static
    address XXX.XX.89.42
    netmask 255.255.255.0
    network XXX.XX.89.0
    broadcast XXX.XX.89.255
    gateway XXX.XX.89.1

    # dns-* options are implemented by the resolvconf package
    dns-nameservers XXX.XX.254.225 XXX.XX.254.225
    dns-search example.com

    # bridge_* options are implemented by bridge-utils package
    bridge_ports eth0
    bridge_stp off
    bridge_fd 0
    bridge_maxwait 0

Then, I crossed my fingers and reloaded the network stack using this command:

/etc/init.d/networking restart

I also used this to "bounce" the bridge network interface:

ifdown br0 && ifup br0

I verified with ifconfig.

I'm ready to create my first VM. There are many different ways to boot the VM and install the operating system. KVM is fully virtualized so nearly any operating system may be install on the VM.

If you have not already, please get familiarized with the following two commands:

virsh
virt-install

The virsh command is an unified tool / API for working with hypervisors that support the libvirt library. Currently virsh supports Xen, QEmu, KVM, LXC, OpenVZ, VirtualBox and VMware ESX. For more information run virsh help.

The virt-install command line tool is used to create new KVM, Xen, or Linux container guests using the "libvirt" hypervisor management library. For more information run man virt-install.

Woah, virsh and virt-install both support LXC?

We have decided to only support Ubuntu 12.04 at this time, so obviously we will choose that for our guest's OS. Now we need to decide on an installation strategy. We may use the following techniques to perform an install:

boot from local CD-rom
boot from local ISO
boot from PXE server on our local vLAN
boot from netboot image from anywhere in the world

We will choose the PXE boot strategy because our vLAN environment already uses that for physical hosts.

We will use the virt-install helper tool to create the virtual machine's "hardware" with various flags. Lets document the creation of this guest in a simple bash script so we may reference it again in the future.

/tmp/create-kvmtest02-a.sh:

HOSTNAME=kvmtest02-a
DOMAIN=example.com

sudo virt-install \
  --connect qemu:///system \
  --virt-type kvm \
  --name $HOSTNAME \
  --vcpu 2 \
  --ram 4096 \
  --disk /cars/vms/$HOSTNAME.qcow2,size=20 \
  --os-type linux \
  --graphics vnc \
  --network bridge=br0,mac=RANDOM \
  --autostart \
  --pxe

This was not used, but shows the flags to perform a netboot from Internet:

--location=https://archive.ubuntu.com/ubuntu/dists/raring/main/installer-amd64/ \
--extra-args="auto=true priority=critical keymap=us locale=en_US hostname=$HOSTNAME domain=$DOMAIN url=http://192.168.1.22/my-debconf-preseed.txt"

I created the vm:

bash /tmp/create-kvmtest02-a.sh

virt-install drops you into the "console" of the VM, but this will not work yet, so we use ctrl+] to break out and get back to our hypervisor. Use virsh list to list all the currently running VMs.

Lets use virt-viewer to view the VMs display. For this we need to SSH to the hypervisor and forward our display to our workstation, we do this with the -X flag. For example:

ssh -X kvmtest02

Now we can launch virt-viewer on the remote hypervisor, and the GUI will be drawn on our local X display!

virt-viewer kvmtest02-a

Once I got that to work, I also tested virt-manager which gives a GUI to control all guests on the remote hypervisor.

virt-manager

Now we need to determine the auto-generated MAC Address of the new virtual machine.

virsh dumpxml kvmtest02-a | grep -i "mac "
  mac address='52:54:00:47:86:8e'

We need to add this MAC address to our PXE server's DHCP configuration to allocate the IP and tell it where to PXE-boot from.

During a real deployment we would get an IP address allocated and an A record and PTR setup for new servers. This is a test and I will be destroying all traces of this virtual machine after presenting during the hackathon, so for now I'm going to skip the DNS entries and "steal" an IP address. I must be VERY careful not to use an IP address already in production. First use dig to find an IP without a record, then attempt to ping and use NMAP on the IP.

dig -x XXX.XX.89.240 +short
ping XXX.XX.89.240
nmap XXX.XX.89.240 -PN

The IP address checked out, it didn't have a PTR, it didn't respond to pings, and using nmap proved there were no open ports. I'm very confident this IP address is not in use.

I added a record to our DHCP / PXE server for this Virtual Machine. I attempted multiple times to pxe boot the VM, but the network stack was never automatically configured... The DHCP server was discovering the new VMs MAC and offering the proper IP address, as noted by these log lines:

Dec 13 07:57:43 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:43 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:44 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPDISCOVER from 52:54:00:47:86:8e via eth0
Dec 13 07:57:48 pxeserver60 dhcpd: DHCPOFFER on xxx.xx.89.240 to 52:54:00:47:86:8e via eth0

I wasted about 4 hours attempting to troubleshoot and diagnose why the VM wouldn't work with DHCP. I ended the night without any guests online...

The Next DAY!

So today I decided to stop trying to get DHCP and PXE working. I downloaded an Ubuntu server ISO to the hypervisor, and used virt-manager to mount the ISO on the guest and booted for a manual operating system install.

This did two things, it proved that the hypervisor's network bridge br0 worked for static network assigned settings and that something between the DHCP server and the hypervisor was preventing the DHCPOFFER answer from getting back to the VM. I looked into iptables firewall, removed apparmor, removed SELINUX and reviewed countless logs looking for hints... then moved on...

I was able to get salt-minion installed on the vm using our post-install-salt-minion.sh script, which I manually downloaded from the salt master. But to keep this test self contained, I pointed the VM's salt-minion to kvmtest02 which we already had setup as a test salt-master.

The salt-master saw the salt-minion's key right away, so I decided to target an install. This what I applied to the VM:

salt/top.sls:

'kvmtest02.example.com':
  - kvm

'kvmtest02b.example.com':
  - virtualenv
  - python-ldap
  - nginx
  - the-gateway

salt-pillar/top.sls

# kvmtest02b gateway in a VM experiment
'kvmtest02b.example.com':
  - nginx
  - the-gateway.alpha
  - deployment-keys.the-gateway-alpha

The stack was successfully deployed to the VM and proved that virtual machines are a viable solution for stage or production. It also gave me the change to test out this particular deployment again and found a few gotchas we need to create maintenance tickets for.

Without configuration management, it would have taken weeks to deploy this custom application stack. The install with configuration management took less then 10 minutes!

One of the KVM related snags I ran into was that Nginx does some fun calculations with cpu cache to determine hash table sizes. As a temporary work around, until I can devote more research time, I raised up the following three hash table directives in the http section of nginx.conf:

server_names_hash_bucket_size 512;
types_hash_bucket_size 512;
types_hash_max_size 4096;

SmartOS

snippet from /etc/dhcp/dhcpd.conf

# SmartOS hypervisor group to boot image
group "smartos-hypervisors" {
  next-server xxx.xx.89.71;

   host smrtest01-eth0 {
        hardware ethernet 00:25:B5:02:07:DF;
        option host-name "ncstest01";
        fixed-address smrtest01.example.com;

        if exists user-class and option user-class = "iPXE" {
           filename = "smartos/menu.ipxe";
        } else {
           filename = "smartos/undionly.kpxe";
        }
    }

}

mkdir /cars/tftp/smartos
cd /cars/tftp/smartos
wget http://boot.ipxe.org/undionly.kpxe
wget https://download.joyent.com/pub/iso/platform-latest.tgz
tar -xzvf platform-latest.tgz
mv platform-20130629T040542Z 20130629T040542Z
mkdir platform
mv i86pc/ platform/

create boot menu that we referenced, vim /cars/tftp/smartos/menu.ipxe

#!ipxe

kernel /smartos/20130629T040542Z/platform/i86pc/kernel/amd64/unix
initrd /smartos/20130629T040542Z/platform/i86pc/amd64/boot_archive
boot

Make sure to replace platform version with current.

I was able to get the blade server to PXE boot the image, but it seems SmartOS doesn't really support the SANs. SmartOS really expects to see local disks, and to build a ZFS pool on top of that. Basically SmartOS could be used to build a SAN, so they didn't put much effort in supporting SANs. After I figured this out I abandoned this test. We could revist this again, using one of the Dell servers, or use it to stand up a really powerful Alpha server environment.

LXC

Run /usr/bin/httpd in a linux container guest (LXC). Resource usage is capped at 512 MB of ram and 2 host cpus:

virt-install \
--connect lxc:/// \
--name lxctest02-a \
--ram 512 \
--vcpus 2 \
--init /usr/bin/httpd

Discussion points

Why doesn't DHCP work on bridge?
If we use virtualization, we need to come up with a plan for IP addresses, like possibly allocate ~5 IP addresses to a hypervisor host
We need to come up with a naming convention for guests, in testing I appended a letter to the hypervisor name kvmtest02 so the guests names were kvmtest02a and kvmtest02b, is this plausible going forward?

If I had more time ...

I would have liked to test out LXC
I would have liked to test out Docker
I would have liked to test out physical to virtual migrations

Backup all virtual machines on a SmartOS hypervisor with smart-back.sh

2013-10-27T20:45:00-04:00

This post will explain how to create a cronjob to backup of every virtual machine on a SmartOS hypervisor.

Create the following bash script in /opt/smart-back.sh:

#!/usr/bin/bash

# Backup all virtual machines on a SmartOS hypervisor
# Author:  russell@ballestrini.net
# Website: https://russell.ballestrini.net/

# Backup directory without trailing slash
backupdir=/opt/backups

# temp dir where we ZFS send and gzip before moving to backupdir
tmpdir=/opt

svcadm enable autofs

for VM in `vmadm list -p -o alias,uuid`
  do
    # create an array called VM_PARTS splitting on ':'
    IFS=':' VM_PARTS=($VM)

    # create some helper varibles for alias and uuid
    alias=${VM_PARTS[0]}
    uuid=${VM_PARTS[1]}

    # echo "Backup started for $VM"
    vmadm send $uuid > $tmpdir/$alias

    # echo "Starting $VM"
    vmadm start $uuid

    # disk space is cheap, uncomment if you disagree.
    #pbzip2 $tmpdir/$alias

  done

mv $tmpdir/*.bz2 $backupdir

Create a cronjob entry to schedule the backups:

crontab -e

2 6 * * 0 /usr/bin/bash /opt/smart-back.sh

If I expand on this script much more, I plan to stick it into revision control.

If you look closely, I have also added a hack to enable autofs (svcadm enable autofs) which allows me to automount an NFS share on my remote FreeNAS by setting backupdir=/net/[ip-or-fqdn-of-freenas]/mnt/zfs-mirror/backup/vms.

We have scheduled a backup of each virtual machine on your SmartOS hypervisor!

If or when the time comes to restore a VM from a backup, use the following:

# decompress the backup file.
pbzip2 -d backup-file.bz2

# ingest the backup file into the hypervisor.
vmadm receive -f /path/to/backup-file

Just make sure the VM doesn't currently exist on the hypervisor.

This strategy is great for complete backups of machines which could be used during a manual migration, or if corruption happened to the VM and we wanted to restore to a previous version.

Simplify deployments with Upstart and uWSGI

2013-10-19T22:35:00-04:00

As you know from my previous post, I recently deleted LinkPeek.com and after struggling to get it back online, I vowed to start utilizing configuration management. During this exercise, I noticed that the architecture I use in production seems overly complicated.

The current production deployment stack:

Nginx listen on 80/443 proxy upstream 9901/9902
Upstart => Supervisord => Cherrypy/Paste listen on 9901/9902

Each of these services and processes have their own configuration files which must work together. Upstart needs to know the location of Supervisord's configuration files. Supervisord needs to know the location of Cherrypy/PasteDeploy's configuration files. Supervisord also must bring up a specified number of worker processes who listen on a pool of ports. Nginx needs to proxy upstream to that pool of worker ports.

This architecture seems difficult to automate because of the numerous places errors may sneak in. I started researching an alternative architecture and stumbled upon Nicholas Piel's Benchmark of Python Web Servers. The graphs Nicholas compiled allowed me to narrow down a list of potential replacements.

I chose to review uWSGI first because it was a top contender on every chart. After briefly testing uWSGI, I halted my explorations and selected it as the winner. I know you probably feel that was a premature decision but uWSGI fit what I was looking for.

The new simplified stack:

Nginx listen on 80/443 proxy upstream 5200
Upstart => uWSGI listen on 5200

The uWSGI server possesses great performance out-of-the-box but also presents many options for fine tuning. These options may be specified either directly in the Upstart script using flags or in a configuration file (like a Pyramid .ini). Either way, the stack uses less files, less processes, and less complexity then the original architecture. For LinkPeek deployments I decided to place all uWSGI related configuration directly in the Upstart script which I embedded below:

linkpeek/weblinkpeek.conf | Upstart for starting uWSGI

description "Start the uWSGI master for the LinkPeek WEB"

start on runlevel [2345]
stop on runlevel [!2345]

respawn

# run service as linkpeek user
setuid linkpeek
setgid linkpeek

# uWSGI command to execute and treat as a daemon
exec /path/to/linkpeek/web/env/bin/uwsgi --master --processes=4 --http=127.0.0.1:5200 --die-on-term --logto2=/path/to/linkpeek/web/uwsgi.log --virtualenv=/path/to/linkpeek/web/env --ini-paste=/path/to/linkpeek/web/linkpeek/production.ini

The web application is now daemonized and listening on port 5200 with 4 workers. I leave implementing the Nginx front end to the reader.

Postfix Salt State Formula

2013-10-17T21:28:00-04:00

The following formula was tested on Ubuntu and Debian however it would not take much work to test on other operating systems.

This state formula will install postfix and mutt. The postfix service will watch various configuration files for changes and restart accordingly.

This formula will also manage and watch the /etc/aliases file and invoke the newaliases command to initialize or re-initialize the alias database. This formula will also manage and watch the /etc/postfix/virtual file and invoke the postmap command to create or update a managed Postfix lookup table.

postfix/init.sls:

# Install mutt and postfix mutt packages.
#
# This formula supports setting an optional:
#
#  * 'aliases' file
#  * 'virtual' map file
#
# Both aliases and virtual use a pillar data schema
# which takes the following form:
#
# postfix:
#   aliases: |
#       postmaster: root
#       root: testuser
#       testuser: russell@example.com
#   virtual: |
#       example.com             this is a comment
#       test1@example.com       me@example.com
#       test2@example.com       me@example.com
#

# install mutt
mutt:
  pkg:
    - installed

# install postfix have service watch main.cf
postfix:
  pkg:
    - installed
  service:
    - running
    - enable: True
    - watch:
      - pkg: postfix
      - file: /etc/postfix/main.cf

# postfix main configuration file
/etc/postfix/main.cf:
  file.managed:
    - source: salt://postfix/main.cf
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

# manage /etc/aliases if data found in pillar
{% if 'aliases' in pillar.get('postfix', '') %}
/etc/aliases:
  file.managed:
    - source: salt://postfix/aliases
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

run-newaliases:
  cmd.wait:
    - name: newaliases
    - cwd: /
    - watch:
      - file: /etc/aliases
{% endif %}

# manage /etc/postfix/virtual if data found in pillar
{% if 'virtual' in pillar.get('postfix', '') %}
/etc/postfix/virtual:
  file.managed:
    - source: salt://postfix/virtual
    - user: root
    - group: root
    - mode: 644
    - template: jinja
    - require:
      - pkg: postfix

run-postmap:
  cmd.wait:
    - name: /usr/sbin/postmap /etc/postfix/virtual
    - cwd: /
    - watch:
      - file: /etc/postfix/virtual
{% endif %}

postfix/aliases:

# Managed by config management
# See man 5 aliases for format
{{pillar['postfix']['aliases']}}

postfix/virtual:

# Managed by config management
{{pillar['postfix']['virtual']}}

postfix/main.cf:

# Managed by config management
# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = {{ grains['fqdn'] }}
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = {{ grains['fqdn'] }}, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all

{% if 'virtual' in pillar.get('postfix','') %}
virtual_alias_maps = hash:/etc/postfix/virtual
{% endif %}

Control a MongoDB collection in configuration management

2013-10-14T21:39:00-04:00

This post explains how to use configuration management (Salt Stack) to completely control a MongoDB collection. In our example we want to control a store's collection of plans.

First we create a JSON representation of the collection.

mongodb/plan.json:

{
  "_id" : { "$oid" : "4ef8b9e2be329f491d98f74b" },
  "cost" : 20, "description" : "development",
  "name" : "good", "count" : 6000
}
{
  "_id" : { "$oid" : "4ef8b9e8be329f491d98f74c" },
  "cost" : 60, "description" : "freelancers",
  "name" : "better", "count" : 36000
}
{
  "_id" : { "$oid" : "4ef8b9f0be329f491d98f74d" },
  "cost" : 180, "description" : "production",
  "name" : "best", "count" : 162000
}

Next we configure a salt state formula to manage the JSON file and watch it for changes.

mongodb/init.sls:

# install mongodb server
mongodb-server:
  pkg:
    - installed

# manage the store's plan.json
/tmp/plan.json:
  file.managed:
    - source: salt://mongodb/plan.json
    - user: root
    - group: root
    - mode: 644

# import the plan collection if it changes
import-plan-collection:
    cmd.wait:
      - name: mongoimport --db=store --collection=plan --upsert /tmp/plan.json
      - require:
        - pkg: mongodb-server
      - watch:
        - file: /tmp/plan.json

Now whenever plan.json is altered in configuration management, the file on the minion will update which will trigger a mongoimport with upsert to occur.

Optionally, we could replace --upsert with --drop which will drop the collection before re-importing thus removing stale records.

We now have a version controlled JSON file in configuration management and the power of MongoDB Document Objects in our application code!

Configuration Management vs Remote Execution

2013-07-11T23:15:00-04:00

What is configuration management?

In a perfect world configuration management provides a centralized, revision controlled, self-documented, change management location for manifests and formulas which both define how to build a complete system and organize a means of knowledge transfer. An infrastructure perfectly described in configuration management allows any single part of the system to be created, reproduced, multiplied, self-healed or even re-purposed.

When should I use configuration management?

Use configuration management whenever you intend to permanently alter system state or infrastructure data.

System state - the current state of the system:

operating system (distro,kernel,patches,hot-fixes)
software installed (base,role-specific)
services running (base,role-specific)
user access
etc

Infrastructure data - the information that describes the infrastructure:

asset information (models,specs,IP,DNS,rack-elevation,etc)
roles (app,web,db,proxy,load-balancer,etc)
current allocations (allocated,unallocated,number of servers in each role,etc)
etc

What is remote execution?

Remote execution is the act of issuing commands to one or more remote systems. The most popular remote execution system in use today is SSH. SSH is great for maintaining a small group of dissimilar systems. Once the system count grows and similar roles present themselves, start looking at something like Fabric. Fabric is a python framework which builds on top of the SSH protocol and makes it possible to invoke the same command on hundreds of servers sequentially. Once the fleet count reaches thousands of servers and commands must run in parallel, look at Salt-stack's remote execution layer written with python and ZeroMQ.

When should I use remote execution?

Only use remote execution for ad hoc reports, data collection, or for temporary tests which have no expectation of persistence after research is complete. Frequent data collection jobs work best when implemented in a metrics collection or monitoring system. (I'll save that for another post)

When should I not use remote execution?

Do not use remote execution to change the state of the system or the data which describes the infrastructure! If a remote execution job changes either state or data it should be placed into the configuration management for the reasons mentioned above.

Understanding Salt Stack user and group management

2013-07-08T13:51:00-04:00

This state will create a user:

russell:
  user:
    - present

This state will create a user and a group. This also makes the user part of the group, and handles creating the group first:

russell:
  group:
    - present
  user:
    - present
    - groups:
      - russell
    - require:
      - group: russell

This state handles user and group generation along with password and ssh-key maintenance. This is all done securely using pillar to parameterize arguments:

# This state will create users accounts
#
# This state requires a pillar named 'users' with data formatted like:
#
# users:
#
#  tusername:
#    fullname: Test Username
#    uid: 1007
#    gid: 1007
#    groups:
#      - sudo
#      - ops
#    crypt: $password-hash-sha512-prefered
#    pub_ssh_keys:
#      - ssh-rsa list-of-public-keys tusername-sm
#
#  anotheruser: ... snipped ...

# loop over all users presented by pillar:
# create user's group, create user, then add pub keys
{% for username, details in pillar.get('users', {}).items() %}
{{ username }}:

  group:
    - present
    - name: {{ username }}
    - gid: {{ details.get('gid', '') }}

  user:
    - present
    - fullname: {{ details.get('fullname','') }}
    - name: {{ username }}
    - shell: /bin/bash
    - home: /home/{{ username }}
    - uid: {{ details.get('uid', '') }}
    - gid: {{ details.get('gid', '') }}
    - crypt: {{ details.get('crypt','') }}
    {% if 'groups' in details %}
    - groups:
      {% for group in details.get('groups', []) %}
      - {{ group }}
      {% endfor %}
    - require:
      {% for group in details.get('groups', []) %}
      - group: {{ group }}
      {% endfor %}
    {% endif %}

  {% if 'pub_ssh_keys' in details %}
  ssh_auth:
    - present
    - user: {{ username }}
    - names:
    {% for pub_ssh_key in details.get('pub_ssh_keys', []) %}
      - {{ pub_ssh_key }}
    {% endfor %}
    - require:
      - user: {{ username }}
  {% endif %}

{% endfor %}

Create your own fleet of servers with Digital Ocean and salt-cloud

2013-07-02T22:20:00-04:00

Have you heard about Digital Ocean? They offer a polished user interface, KVM guests with SSD storage, and an API to interact with a cloud of hypervisors. API integration got you down? Don't worry, salt-cloud has already integrated Digital Ocean among it's list of providers! The rest of this post illustrates the steps I took to configure salt-cloud to work with Digital Ocean.

This guide assumes you already have a:

salt-master
Digital Ocean account.

Step one, install the most recent version of salt-cloud.

On the salt-master:

sudo apt-get install salt-cloud

# or if you prefer ...
pip install salt-cloud==2015.5.0

# last verify it was successfully installed
salt-cloud --version

Step two, configure salt-cloud.

Salt-cloud uses the following files YAML files for configuration:

/etc/salt/cloud.conf.d/main.conf:: This is the main configuration file. I have the following statements:

minion:
    master: master.foxhop.net
    append_domain: foxhop.net

/etc/salt/cloud.providers/do.conf:: This is a provider configuration file for Digital Ocean (do). Collect your client_key and personal_access_token (api_key) from the Digital Ocean user dashboard. Also create an SSH key and add the public key using the dashboard:

# For Digital Ocean
do:
  provider: digital_ocean
  client_key: MyClientKeyLiftedFromDashboard
  personal_access_token: MyAPIKeyLiftedFromDashboard
  ssh_key_file: /keys/digital-ocean-salt-cloud
  ssh_key_name: digital-ocean-salt-cloud.pub

/etc/salt/cloud.profiles/do.conf:: This is the Digital Ocean profiles configuration file. We will create just two profiles for now, but you can create unlimited named combinations.

ubuntu-12-04-do-512:
  provider: do
  image: ubuntu-12-04-x64
  size: 512mb
  location: nyc1

ubuntu-14-04-do-512:
  provider: do
  image: ubuntu-14-04-x64
  size: 512mb
  location: nyc1

ssh_key_file:: This is your private SSH key located on your salt-master
ssh_key_name:: This is the name of the public key you added in your Digital Ocean dashboard
size:: The size or plan you would like to provision, 512mb is the smallest plan
location:: The geographical region, location, and/or data center
image:: The operating system image

After you configure the do provider in /etc/salt/cloud.providers you gain access to the following commands:

salt-cloud --list-sizes do
salt-cloud --list-locations do
salt-cloud --list-images do
salt-cloud --help

Lets provision a new cloud server!

salt-cloud --profile ubuntu-14-04-do-512 deejay

If all goes well you should have a newly provisioned server bootstrapped with salt-minion. The new minion's keys are already added to the salt-master. Now you just need to run highstate!

Add a custom header to all Salt managed files using pillar and jinja templates

2013-07-01T14:11:00-04:00

Salt-stack (salt) provides a solution for centralized configuration management and remote execution. One of the most basic things Salt provides is the ability to manage the contents of a file or a directory of files. Using Salt we can dictate the state of our minions and as a result we also gain auto-healing of configuration files.

Salt will clobber local changes to managed files and force the state to reflect the version in configuration management. In an effort to avoid confusing the uninformed, I place a header on each managed file which announces "THIS FILE IS MANAGED BY SALT".

To avoid repeating myself in each managed file, I came up with the following centralized solution -

First, I give all minions access to the headers pillar tree in top.sls:

base:
  '*':
    - headers

Next, I create a couple headers in in headers/init.sls:

headers:
  salt:
    file: |
        ################################
        # THIS FILE IS MANAGED BY SALT #
        ################################
    directory: |
        #####################################
        # THIS DIRECTORY IS MANAGED BY SALT #
        #####################################

Then, I parse each managed file in the state tree with jinja, for example hosts/init.sls:

/etc/hosts:
  file.managed:
    source: salt://hosts/hosts
    user: root
    group: group
    mode: 644
    template: jinja

Finally, in each file served by salt I add the jinja header substitution, for example hosts/hosts:

{{pillar['headers']['salt']['file']}}
127.0.0.1       localhost.localdomain localhost
::1     localhost6.localdomain6 localhost6

I use this same technique to declare most configuration options as parameters, like hostnames, IP addresses, ports, and more.

High load and CPU usage craftbukkit compared to vanilla minecraft

2013-06-20T23:35:00-04:00

I started researching the best ways to use salt to provision minecraft servers. I wrote a salt state formula for the vanilla minecraft server deployment. The deployment worked out great so I decided to try my luck with plugins.

In order to use plugins and mods we need to use a customized server package. I decided to try provisioning a craftbukkit server and quickly noticed something was very wrong. Vanilla Craftbukkit (with no plugins) produces very high load and CPU usage.

Here is a chart for proof. The spike is when I stopped the vanilla minecraft server and started the craftbukkit minecraft server:

Honey! I just DELETED LinkPeek.com

2013-05-26T13:19:00-04:00

During the day I am an ops sys-admin. During the night I am a husband, father of two, and a CEO of a bootstrapped start-up. After launch, my first project was to schedule regular backups of user data and archive off-site. My goal was to create backups but never need them. Boy was I lucky ...

Yes, leave it to me to inadvertently delete the VPS root disk. One of the major cloud providers places the "rename" and "remove" disk buttons right next to each other and I learned a nasty habit of clearing pop-ups without reading them (thanks Windows).

"Honey! I just deleted LinkPeek.com"

The horror... My stomach felt like I took a tumble in a roller-coaster. Instantly I tossed off my developer hat and put on my operations hat. I checked the off-site backups. I had nightly dumps of MongoDB and weekly tar backups of the /etc partition. The user data was in MongoDB and most of the system configuration information was in the tar. I used the tar to recover 2 upstart scripts, 2 supervisord scripts, 2 complex nginx confs, an ssl cert, and the pyramid production.ini.

I set out to stand up a new server, re-install the needed packages, recover the user data, and restore the service. After 1.5 hours of feverish typing, https://linkpeek.com/ was back online.

What I learned and my plan going forward

If you are a small team or a start-up, you must have somebody dedicated to operations. Without backups I would not have been able to gracefully recover. Most likely I would have reimbursed the existing members and shuttered the doors.

This experience was eye-opening. In my next couple of posts I will explain how I create and maintain backups and my next project will implement a configuration management and provisioning system.

This system will allow me to:

take out the human element of recovery
significantly reduce the time-to-recover from a catastrophic failure
test disaster recovery procedures before needing them
provision development and production environments without effort
have a reproducible blueprint of "how to build a LinkPeek server"

Survey Baby Monkey

2013-04-26T13:47:00-04:00

Automatic event hangout with cron

2013-04-09T09:17:00-04:00

Create an online only, hangout event

Create a new event with a date far into the future, like the year 2015. Go to the event's options > advanced and enable 'this event is online only' which will create a unique Hangout URI.

Create a cronjob

Create a cronjob on each device to open the web browser Monday-Friday at 12:49pm and open the unique Hangout URI.

Firefox cron example:

# Run Firefox at 12:49 EST each weekday and open hangout URI
49 12 * * 1-5 export DISPLAY=:0 && /usr/bin/firefox 'hangout-uri'

Chromium-browser cron example:

# Run Chromium-browser at 12:49 EST each weekday and open hangout URI
49 12 * * 1-5 export DISPLAY=:0 && /usr/bin/chromium-browser 'hangout-uri'

Guido name dropped tornado python tulip and pep-3156

2013-03-22T22:07:00-04:00

Pycon 2013 was excellent, in fact it was my first one I have attended.

I found it odd that django and Pyramid had plenty of talks but nobody mentioned tornado.

The only person that brought up tornado was Guido himself, who has been researching and developing async python since December 12th, 2012. Guido wants to add async API libraries to python core, and has been comparing his work to twisted and more importantly tornado. He is leveraging the existing solutions to stay on the correct track.

Async API's in Python core! This news was extra exciting for me, because I have already learned the power of async by messing around with tornado. Since the talk many people have approached me and asked for more information about async API's.

My favorite use for async is long polling for web applications. Here is a diagram that shows how great tornado is:

This type of polling can support thousands of concurrent users, all connected and waiting for a callback event to occur. I used tornado to build four2go, a real-time and multi-player browser-based-game.

virt-back's Domfetcher class returns doms from libvirt API

2013-03-02T15:04:00-05:00

I'm hooked...

I attended SCaLE 11x, my first technical conference, and had an amazing time. My favorite talk was Michael Day's "Advancements with Open Virtualization & KVM" (link to slides). Michael's presentation inspired me to continue my work on virt-back.

During my trip home I used the in-flight wifi to push this commit into the cloud from the clouds! This particular commit re-factored the dom object list generation into a simple-to-use class called Domfetcher. Domfetcher abstracts the libvirt API and grants access to the following helper methods:

get_all_doms( )

Return a list of all dom objects

get_doms_by_names( guest_names=[] )

Accept a list of guest_names, return a list of related dom objects

get_running_doms( )

Return a list of running dom objects

get_shutoff_doms( )

Return a list of shutoff but defined dom objects

This is an example of how to use Domfetcher:

import virtback

# optionally supply hypervisor uri
domfetcher = virtback.Domfetcher()

doms = domfetcher.get_running_doms()

for dom in doms:
    print dom.name()

for dom in doms:
    print dom.info()

for dom in doms:
    print dom.shutdown()

As always thanks for reading!

How to overload default function arguments in python using lambda

2013-01-12T11:22:00-05:00

Python Lambda functions are very powerful but I often forget how they work or the fun things they do. This post will document how to use a lambda to provide different default arguments to a function.

We will use the human function found in ago.py as an example - because I'm the module author and I really like it. Lets use the interactive python interpreter to run help on the human function.

>>> from ago import human
>>> help( human )

Help on function human in module ago:

human(dt, precision=2, past_tense='{} ago', future_tense='in {}')
    Accept a datetime or timedelta, return a human readable delta string

Shown above the human function accepts 1 argument and 3 named keyword arguments. The dt argument must be a datetime or timedelta object, the precision must be an integer, and the other two must be strings. If we didn't like the default arguments, we would need to specify (or pass in) new values each time we invoked the function.

Example: human(dt, 3, 'this happened {} ago!', 'in {} from now!'). If we know we will always want different default arguments we can create a lambda function to shorten the invocation length.

>>> h = lambda dt : human(dt, 3, 'this happened {} ago!', 'in {} from now!')
>>> print h( dt ) # h is much shorter then human, and still reusable!

Above creates a new function h who only accepts one argument dt. This function calls human with our default arguments.

This lambda is equivalent to this regular python function:

>>> def h( dt ):
...    return human(d, 3, 'this happened {} ago!', 'in {} from now!')
>>> print h( dt ) # h is much shorter then human, and still reusable!

Here is a working example to show the new lambda function h in action:

>>> from datetime import datetime
>>> from datetime import timedelta
>>> from ago import human
>>>
>>> h = lambda dt : human(dt, 3, 'this happened {} ago!', 'in {} from now!')
>>>
>>> present = datetime.now()
>>>
>>> for i in range( 1, 15 ):
...     if i % 2 == 0:
...         new_date = present - timedelta( i, i * i, i * i * i )
...     else:
...         new_date = present + timedelta( i, i * i, i * i * i )
...     h( new_date )
...
'in 22 hours, 9 minutes, 30 seconds from now!'
'this happened 2 days, 1 hour, 50 minutes ago!'
'in 2 days, 22 hours, 9 minutes from now!'
'this happened 4 days, 1 hour, 50 minutes ago!'
'in 4 days, 22 hours, 9 minutes from now!'
'this happened 6 days, 1 hour, 51 minutes ago!'
'in 6 days, 22 hours, 10 minutes from now!'
'this happened 8 days, 1 hour, 51 minutes ago!'
'in 8 days, 22 hours, 10 minutes from now!'
'this happened 10 days, 1 hour, 52 minutes ago!'
'in 10 days, 22 hours, 11 minutes from now!'
'this happened 12 days, 1 hour, 52 minutes ago!'
'in 12 days, 22 hours, 12 minutes from now!'
'this happened 14 days, 1 hour, 53 minutes ago!'

ago.py human readable timedelta 0.0.4 release

2013-01-09T00:04:00-05:00

We have released ago.py 0.0.4

pypi: https://pypi.python.org/pypi/ago

repo: https://bitbucket.org/russellballestrini/ago

Special thanks to David Beitey for supplying ideas and python code for this update!

All changes are backward compatible.

Change log:

added support for future dates
added optional past_tense keyword argument string for custom output
added optional future_tense keyword argument string for custom output
added 13 tests to help prevent regressions
Updated the README.rst documentation for better coverage

We were just shy of 800 downloads on pypi for ago-0.0.3, I hope ago-0.0.4 will perform even better!

Tips for getting pull requests approved

2012-12-12T12:50:00-05:00

Pull rejection sucks!

You have just coded, implemented, and submitted a pull request. A short while later the request is declined by an upstream maintainer and you feel crushed. We have all been there. Today I'm going to show you a better way. This article will teach you how to create pull requests that get approved.

Start small

You need to earn trust with the maintainers. Your first commit should be a small change which they cannot reject. Try to write a missing test or re-factor duplicate code. Correct a comment's accuracy or rename a variable to better reflect its purpose. Your first pull request should not alter how the program works.

Don't add leading or trailing white space

Additional whitespace will alter the diff output. This causes version control systems to flag lines as changed which is irritating and sometimes misleading.

Less is more

Minimize the number of changes to accomplish your goal. People are busy and at times lazy. Reduce the work the maintainers must do to perform a merge. Lowering the amount of lines to review should increase the chance of approval.

Only commit working code

Do not break the program, only commit working code. If the project has tests make sure they work before you commit.

Follow the leader

Try to mimic the maintainers. Follow the coding style and project layout even if it seems wrong. This is not your playground... yet. Before you commit, review the VCS logs to learn how verbose or terse your commit messages should be. When in Rome do as the romans do. This silly game of follow the leader reduces friction of an outsider committing to the project.

Comments, docs, and tests oh my!

Your first pull request should clarify or add to the existing documentation. Fix the README, adjust a comment, or write a test. These tasks might appear small but they serve to prove that you possess comprehension of the source code. They also do not alter the program's functionality.

Having a few of these pull requests under-your-belt will earn you trust which will eventuality translate to more responsibility in the future. You will also differentiate yourself from the rest because most people do not enjoy working on documentation.

Blog about the change

Write about the change, give reasons and examples. Include a link to your blog post in the pull request.

Pull requests are like paragraphs

If you were writing an essay, you would split up your ideas into separate paragraphs. A pull request has many qualities similar to a paragraph. Each commit should be related to the pull request's main objective. Commits of a pull request should stay focused and on topic.

A pull request is to a program as a paragraph is to an essay.

A commit is to a pull request as a sentence is to a paragraph.

In an essay, if you have more then one topic you should have more then one paragraph. Likewise when coding, only one idea or change per pull request.

Separating your ideas into different pull requests will grant the maintainers greater flexibility when they begin to integrate. They will have the ability to pick-and-choose which requests to merge and everybody wins!

Contents

Pull rejection sucks!
Start small
Don't add leading or trailing white space
Less is more
Only commit working code
Follow the leader
Comments, docs, and tests oh my!
Blog about the change
Pull requests are like paragraphs

how to drain an iPhone battery without needing passcode

2012-11-03T19:27:00-04:00

Press home button [ ]
Slide camera button up
Slide mode to video
Turn on flash
Put iPhone on table with light pointed down
Walk away inconspicuously

Extra points if you set the camera to record (just don't record yourself or sounds) which will fill up the iPhone capacity.

Explaining cache with python

2012-10-02T15:22:00-04:00

What is cache? I define cache as "a saved answer to a question". Caching can speed up an application if a computationally complex question is asked frequently. Instead of the computing the answer over and over, we can use the previously cached answer. This post will present one method of adding cache to a python program. Specifically we will write a program that computes prime numbers and saves the answers into cache for quick retrieval.

EDIT: The kind people of the Internet have expressed concern with my loose use of the term cache; the techniques that follow are most accurately described as memoization.

One algorithm for determining if a number is prime follows:

prime_flag = True # default state
x = 5 # number to test
if x == 1:
    prime_flag = False
else:
    for i in range( 2, x ):
        if x % i == 0:
            prime_flag = False
            break
print prime_flag

Create a prime_flag variable to hold the answer and default it to true. Let x be the number being tested and if x is equal to 1, the x is not prime. Otherwise iterate over each number in the range of 2 to x. Let i be the current number to be tested. if x is divided by i without any remainder, x is not prime. Set the prime_flag to False and break out of the loop. Print the result.

Next we will move the algorithm into a function which will allow for code reuse:

def is_prime( x ):
    """Determine if a number is prime, return Boolean"""
    prime_flag = True
    if x == 1:
        prime_flag = False
    else:
        for i in range( 2, x ):
            if x % i == 0:
                prime_flag = False
                break
    return prime_flag

# invoke function:
print is_prime( 5 ) # True
print is_prime( 4 ) # False

This function saves us a lot of typing and enables the ability to quickly determine if a given number is prime.

Next we will use a python dictionary to implement a result cache. Also by circumstance we introduce objects and classes.

class Primer( object ):
    def __init__( self ):
        """create a cache dictionary"""
        self.cache = {}

    def is_prime( self, x ):
        """Determine if x is prime, cache and return result"""
        if x in self.cache:
            return self.cache[x] # lookup result

        prime_flag = True

        if x == 1:
            prime_flag = False
        else:
            for i in range( 2, x ):
                if x % i == 0:
                    prime_flag = False
                    break

        self.cache[x] = prime_flag # cache result
        return prime_flag

p = Primer() # create a new primer object
p.is_prime( 5 ) # True
p.is_prime( 4 ) # False
p.is_prime( 5 ) # True and fetched from cache

What is great about this solution is that we can avoid looping and computation if an answer is already in cache. Looking up a cached result is much more efficient and will ultimately make a program feel more responsive.

Determining if 97352613 is prime takes my laptop nearly 18 seconds. Fetching the cached result seems to happen instantly.

>>> s1 = time();p.is_prime( 97352613 );e1 = time()
False # not prime
>>> s2 = time();p.is_prime( 97352613 );e2 = time()
False # not prime from cache
>>> e1 - s1
17.970067977905273 # seconds
>>> e2 - s2
2.5987625122070312e-05 # or approx .000026 seconds

A look-up will always beat a computation. Anything that can be cached, should be cached. I hope this helps clear things up.

The Pyramid community taught me the importance of test driven development

2012-09-22T14:03:00-04:00

Sontek's patch

I greeted the UPS man in the middle of the street to sign for my new Lenovo ThinkPad T430. Because this was My first brand-new laptop purchase I rationalized the time I spent tracking the package from the factory in China to my hands in Connecticut. Once inside, I opened the box and started installing Fedora 17. I couldn't help but to take in the new-electronics smell.

I've been without a laptop for more than a month so I was eager to get my development environment configured. Most of my tools ship with vanilla Linux, vim, python, hg mercurial, chromium browser, etc. My first goal was to get a development copy of linkpeek.com running locally. This took about 5 minutes and it seemed to be working fine until I noticed a few pages had errors. The errors seemed to be caused by a difference between Pyramid 1.3 and 1.4a1. But what was failing?

I posted a short message in #pyramid about the bug and minutes later I had multiple developers prodding for hints. "Could you post the whole traceback?", "What does your view look like?". I answered quickly and attempted to explain what I thought was going on. Turns out I was close but before I could finish explaining the problem, sontek had a working one-character-fix and was in the process finishing the tests to prove the patch. He also explained what I should do in the interim to patch locally.

In the next 4 hours a pull request was submitted to the upstream master and the patch was peer reviewed, accepted, and integrate by mcdonc. That impressed me, a lot. All Pylon Projects have strict policies about test coverage and now I understand why. Tests not only help produce better bug-free software but also act as a powerful tool when proving the validity of a patch. I plan to devote the next couple weeks to making test-driven-development a habit.

miniuri parser and ago human timedelta

2012-06-29T21:30:00-04:00

I just packaged and published a couple of python modules to pypi:

ago
miniuri

To install them, run:

Install:

pip install --upgrade ago miniuri

If you want to view their source code, look here:

miniuri
ago

I hope you enjoy them.

My top five suggestions for an independent developer creating a new product or service

2012-06-25T19:50:00-04:00

1. Write everyday. Build a blog for the project and write about milestones, progress, and hurdles. Also keep a personal blog and write about hobbies. Read some theory about "copy writing" and search engine optimization. Write personalized email responses to customers. Great communication skills will have the most impact on the success of your company. The best way to increase communication skills is to do it.

2. Spend at least 15 minutes on the company everyday. Even small or petty tasks will lift your company to the next goal. Over time the company will grow strong and people will give you attention. Building a company seems similar to building a character in an RPG. Instead however you will progress your company to its next level. Also, always look ahead when working, don't look back at prior achievements. Keep your eyes on the next milestone and keep moving forward. This type of behavior will promote growth and prevent getting stuck in the daily grind.

3. Do not fear manual processes. In the early stages of a company you should only build automation which will enhance customer experience. For everything else, build a repeatable manual process. Coding automation will typically cost more time to build than it will save. For example on LinkPeek I have not automated customer account de-activation because fortunately I don't have to do many of them. This manual process only takes a couple minutes to complete and gives me a chance to write a "thank you, and sorry to see you go" email to the fleeting customer. Manually de-activating (loosing) a customer is also a humbling experience. Keep a list of nice-to-have automations and review them in the far future. How far in the future? Far, like when your idea is validated and your company appears successful. Don't waste valuable time building automation for a service nobody will pay for or use. Time is your most scarce resource.

4. Always attempt to increase your luck. This might sound funny but the the following suggestions should reduce risk and also increase luck:

Meet fellow lucky people
Write on a blog
Advertise your company on side-projects that lack monetization
Think before wasting money using adsense or similar networks
Do market research before committing to an idea
Figure out how to make money before building product
Build the smallest version of the product that could still be sold
Try to stay focused on the things your customers will see
Don't build an admin dashboard until successful (profitable?)
Choose tasks that require the least effort but have the biggest impact
Don't be afraid to do things in the early stages that won't scale in the long run (example: personal email responses)
Listen to how early adopters describe your product; then on your website, marketing and emails reuse their words.
Attempt to keep marketing, newsletter, and support emails only few sentences long.
Start maintaining an opt-in email list

You might have the best mouse-trap but without luck it will never gain traction.

5. Have an unhealthy passion to support your customers and build your company. Six days ago I suffered a major chemical burn to my right eye. With my wife's help I continued to respond to support emails. Since I launched LinkPeek my focus and attention to my customers and company have never faltered. Dedication plays a key role to success.

Prevent a certain program from running too long in bash

2012-06-13T09:45:00-04:00

Update - I opensourced this script here: bash kira

I came up this this script to kill certain programs after they run for too long. This works like similar to a timeout. Warning this script is pretty harsh and kills the program.

#!/bin/bash
PROGRAM=replace-with-program-name
PIDSFILE=/tmp/kill-these.pids

for pid in `pidof $PROGRAM`
  do
    if grep -q $pid $PIDSFILE
      then
        kill $pid
    fi
  done

> $PIDSFILE

for pid in `pidof $PROGRAM`
  do
    echo $pid >> $PIDSFILE
  done

Then I wrote a cronjob to kill hung programs:

* * * * * /usr/local/sbin/killprogs.sh

Always attempt to scale vertically first

2012-06-10T21:52:00-04:00

I spent the weekend fretting because one of my servers was basically being DOS'd by paying customers. During the outage I started thinking about the best way to scale and how I could make the code-base more efficient.

Linux top reported high load, in the 20's. Eventually I figured out that the server was having IO performance issues.

I wasted a bunch of time attempting to fight fires. After about an hour of that I decided to scale my VPS vertically by giving it an extra 256mb of memory and a larger swap file (256mb to 1024mb).

These two changes were surprisingly effective and the IO issues resolved. Apparently the server was starving for memory which caused the host to swap which brought things to a crawl waiting for IO.

Crisis averted for the moment. Now I am free to think clearly and engineer a proper solution instead of attempting to put out fires.

If you ever encounter a similar situation, attempt the simplest fix. There is no shame in throwing more money at a problem if it will buy you time. In this case, an extra $10.00 a month relieved the performance issues and bought myself some time, for the moment.

High load on web server after updating from Ubuntu 10.04 to Ubuntu 12.04 LTS

2012-05-19T18:40:00-04:00

High load on web server after updating from Ubuntu 10.04 to Ubuntu 12.04 LTS

Check out charts which lineup to when I upgraded:

I couldn't determine the cause of the load average increase...

Update: The issue might be memory bound. Check out this graph that show much higher swap.

After much research this appears to be a load calculation and display problem with the newer Linux kernels. The community has found Commit-ID: c308b56b5398779cd3da0f62ab26b0453494c3d4 to be the problem. The commit causes incorrect high reported load averages can be reported under conditions of light load and high enter/exit idle frequency conditions (greater then 25 hertz).

A nice fellow named Doug from smythies.com researched the topic between tick and tickless linux kernels and the effect they have on load averages <http://www.smythies.com/~doug/network/load_average/new.html>. You should check it out.

How to rescue logs and config from a failed Citrix NetScaler App Gateway

2012-05-11T00:05:00-04:00

Today our production Citrix NetScaler broke. The box wouldn't boot and our only backup copy of the config was on the NetScaler itself.

Being the only Unix guy around I attempted to help out the admins working the outage. I SSH'd into the development NetScaler and noticed it runs on FreeBSD.

I suggested fetching the Hard drive and mounting it on a Linux computer. The NetScaler has one SATA (not SAS) disk so my desktop was compatible.

I installed the disk in the Linux tower and mounted the filesystem using the following command:

mount --read-only --type=ufs --test-opts ufstype=44bsd /dev/sda5 /mnt

Once mounted I was able to SCP interesting files to a safe location.

Warning, this procedure might void your warranty. If in doubt, call support first.

The most valuable registration field: How did you hear about us?

2012-04-26T22:21:00-04:00

How did you hear about us?

I first answered this question when joining Linode. I remember thinking "Wow, this is a great time to ask me!" because the real answer was still in my short term memory.

When I launched LinkPeek I applied this technique. After an amazing launch (thank you colleagues from HackerNews) the true significance of this field was exposed. I would argue that the answer to this simple question holds more value than collecting a username.

Here is why:

It may seem obvious after reading this post, but this is what I learned and I am sharing to help other startups.

Asking "How did you hear about us?" will help you determine:

how your customers found you
which of your marketing efforts are working
where to spend money or time marketing (and where to stop)
your true target market
how your customers will use your product

Here are some other benefits to this technique. It opens a dialog or conversation with the customer, which in return should help lower friction during the sale. You already have the the customer engaged during the signup process. I am firmly against the traditional method of surveying customers. I believe the same data can be collected without being abrasive or wasting the customers time.

"How did you hear about us?" is the gift that keeps on giving.

It is commonly accepted that shorter registration forms lead to better conversions. I totally agree, but in this case the sacrifice is worth learning more about my target market and customer needs.

How to ask this question properly:

Just ask! I second guessed my decision to add this field right before launch but that anxiety promptly faded after reading the first answer.
Don't use pre-filled answers! Having pre-filled answers is counter productive because you will not learn anything new. You are forcing the user into choosing one of your answers...
Allow the user to type as much as they like! A few of my customers nearly wrote a book in the text field.

If you liked this post, you should follow me on twitter `here <https://twitter.com/RussellBal>`__.

I just purchased Instagram for 1B and all I got was this lousy image filter

2012-04-09T17:47:00-04:00

Warning / Update!

This post was originally written from a place of jealousy and bitterness.

Turns out I was wrong about this transaction and for better-or-worse, Facebook (and Mark Zuckerburg) solidified their edge as the king of social for the last 6 years.

Original Post

Facebook purchased Instagram on Monday, April 9th 2012 for $1,000,000,000 US. Instagram, a free to use image sharing iPhone application, has absolutely NO REVENUE STREAM. The application is free to download, free to use, and has no monetization support from advertisements.

$1000000000.00 / 30000000 users = 33.33 per user

At the time of the sale Instagram had 30 million users in total (30,000,000). That means Facebook just paid heavy $33.33 per user... This is obviously a bad deal for Facebook.

Instagram did not have a method to monetize its application. I hedge my bets that Facebook will never recoup the one billion dollars spent on this deal.

Mark Zuckerburg, I just thought up the perfect T-Shirt for you:

I just purchased Instagram for 1B and all I got was this lousy image filter.

Contents

Warning / Update!
Original Post

Trouble mounting filesystem on KVM guest after reboot

2012-03-27T12:04:00-04:00

Just found this out the hard way...

It looks like the attachment of /KVMROOT/guest-dev-app.img on guest-dev did not persist when the KVM host rebooted for patching.

As it appears the virsh attach-disk command works a lot like the mount command.

In order to have a disk attachment persist after a reboot, I think we still need to do a virsh edit.

the virsh attach-disk command is useful because it allows us to attach disk images to guests without restarting.

tldr;

virsh attach-disk is to mount as

virsh edit is to vim /etc/fstab

nosslsearch cname is a bad idea and solution

2012-03-26T11:51:00-04:00

Google SafeSearch and SSL Search for Schools suggests implementing the following changes to the network:

To utilize the no SSL option for your network, configure the DNS entry for www.google.com to be a CNAME for nosslsearch.google.com.

Here are the reasons why this is a bad idea and solution:

In order to create a CNAME record for www.google.com we need to become an authoritative master of that zone.
If you become an authoritative master you need to host all of Google's DNS resource records for the domain.
Google is asking us to DNS poison it's flag ship product on our networks.
If other companies follow suit the internet will quickly become unmanageable. DNS was not ment to work this way.
Not all networks have a local DNS server

This is a bad idea. Please change your stance on this matter.

Reference: https://support.google.com/websearch/bin/answer.py?hl=en&answer=186669

"I see" said the blind man, to the deaf dog, as he walked off the cliff.

2012-03-23T22:04:00-04:00

"I see" said the blind man, to the deaf dog, as he walked off the cliff.

As far as I can tell, I am the originator of this version of this quote.

EDIT: changed "originator of the quote" to "originator of this version of this quote".

What do you name your python virtualenv?

2012-03-07T21:42:00-05:00

What do you name your python virtualenv?

I name my virtualenv 'virtpy'. Is there a standard name being used out there?

Maybe we can come to a consensus as a standard name? Please feel free to post your virtualenv names here as a sort of poll.

How to save hundreds of dollars on groceries without clipping coupons

2012-03-03T19:54:00-05:00

This is the first time I've had a guest blogger on my site. It may sound campy but my wife Jenn wrote this article after explaining how our grocery bill decreased so drastically.

How to save hundreds of dollars on groceries without clipping coupons

I was recently re-working the cabinets of the kitchen, anticipating a new and organized year. With a growing family and limited space, I decided to do some consolidating.

I allocated a whole section of cabinet space to arts and craft supplies for the kids (play-doh, paints, etc), and another to bottles and formula. I suddenly realized that my food products remained scattered across the counter and there was only one lonely cabinet left to fill.

Among the food items on the counter, many were far expired, forgotten in the dusty back corners of the cabinets. I thought to myself “Hey! I worked hard to find coupons and sales for all these things.” and I regretfully filled a garbage bag with old food.

I had been clipping coupons and striving to match the savings of super couponers. These couponers, featured on popular reality shows, often recommend stock-piling when food items go on sale and you have coupons to purchase them.

I opened the refrigerator to find a similar and familiar situation: a head of lettuce that had seen better days, a tomato that had lost it's freshness, and other food items that were past the date of recommended consumption.

Looking into the garbage bag, I realized that... every week I was wastefully throwing food and money away. I decided to make a commitment that has cut my grocery bill in half! Do I still use coupons? YES, if they fit into my weekly plan. Do I still look at the sale flyers? ABSOLUTELY! But...

These two strategies have saved me much more money:

1. Cut down food storage spaces

If you currently use 4 food cabinets try designating only 2. In my case I consolidated my food items to 1 cabinet. This consolidation saved money in two distinct ways:

First, I am easily able to assess which food items I have and when they expire. This helps prevent re-buying a product we already have at home (Remember the time you were at the grocery store buying all the ingredients for those chocolate chip cookies and couldn't remember if you had any brown sugar? Undoubtebly you re-bought, probably to discover you had a brand new, unopened package in the cabinet).
Secondly, it prevents stock piling. I can buy only what I need for the week ahead, after all, it is all I have room for. If you generate the sense that “my kitchen is full of food”, it eliminates the need to buy for the sake of filling the cabinets.

2. Let your menu dictate your shopping list

My son complains “There are no more crackers." My husband sighs, “We're all out of bananas." My answer used to be probably a lot like yours “OK, I'll put it on my list." I had the perception that I constantly needed to replenish. It is unnecessary and wasteful to have three different brands of crackers, five types of cereal, and every kind of fruit known to the produce section. Without fail, I was throwing away money each week.

Instead, I sit down each week with two sheets of paper. One serves as dinner menu for the week and the other as my shopping list. When building my menu, I consider the sales and coupons I have on hand. If I notice a great deal on pasta sauce, we might have spaghetti one night and meatball subs another. I immediately begin a shopping list writing ONLY items used for the companion menu. Additionally, I add two breakfast choices and two lunch choices for the week if needed (some breakfast items such as pancake batter or a family size box of cereal often last longer). I also include 2 choices of fresh fruit or vegetable. To reduce waste and cost I ONLY add the staples (eggs, milk, and bread) when it is on the menu. If none of my breakfast, lunch, or dinner choices call for bread, I skip that aisle for this trip.

It took me only a few short weeks to notice that by cutting my food storage space in half, and adopting a disciplined approach to menu and list making, I had saved hundreds of dollars. I challenge you to cut down your food storage spaces and let your menu dictate your shopping list!

How to capture HTTPS SSL TLS packets with wireshark

2012-02-29T19:14:00-05:00

This article will explain how to use wireshark to capture TCP/IP packets. Specifically I will show how to capture encrypted (HTTPS) packets and attempt to document the "dance" a client and server do to build an SSL tunnel.

What is Wireshark?

Wireshark is a network protocol analyzer for Windows, OSX, and Linux. It lets you capture and interactively browse the traffic running on a computer network. Similar software includes tcpdump on Linux.

Install Wireshark

First step, acquire Wireshark for your operating system.

Ubuntu Linux: sudo apt-get install wireshark

Windows or Mac OSX: search for wireshark and download the binary.

How to capture packets

This is Wireshark's main menu:

To start a capture, click the following icon:

A new dialog box should have appeared. Click start on your preferred interface:

You are now capturing packets. The packet information is displayed in the table below the main menu:

Now browse to an HTTPS website with your browser. I went to https://linkpeek.com and after the page completely loaded, I stopped the Wireshark capture:

Depending on your network, you could have just captured MANY packets. To limit our view to only interesting packets you may apply a filter. Filter the captured packets by ssl and hit Apply:

Now we should be only looking at SSL packets.

Next we will analyze the SSL packets and answer a few questions

1. For each of the first 8 Ethernet frames, specify the source of the frame (client or server), determine the number of SSL records that are included in the frame, and list the SSL record types that are included in the frame. Draw a timing diagram between client and server, with one arrow for each SSL record.

Frame 1 client | 1 record | Arrival Time: Feb 15, 2012 15:38:55.601588000

Frame 2 server | 1 record | Arrival Time: Feb 15, 2012
15:38:55.688170000
Frame 3 server | 2 record | Arrival Time: Feb 15, 2012
15:38:55.688628000
Frame 4 client | 3 record | Arrival Time: Feb 15, 2012
15:38:55.697705000
frame 5 server | 2 record | Arrival Time: Feb 15, 2012
15:38:55.713139000
frame 6 client | 1 record | Arrival Time: Feb 15, 2012
15:38:55.713347000
frame 7 server | 0 record | Arrival Time: Feb 15, 2012
15:38:55.713753000
frame 8 server | 1 record | Arrival Time: Feb 15, 2012
15:38:55.715003000

2. Each of the SSL records begins with the same three fields (with possibly different values). One of these fields is “content type” and has length of one byte. List all three fields and their lengths.

Each hexadecimal digit (also called a "nibble") represents four binary digits (bits) so each pair of hexadecimal digits equals 1 byte.

a. Destination mac address | 6 btyes | 00 21 9b 31 99 51

b. Source mac address | 6 bytes | 00 10 db ff 20

c. Type: IP | 2 byte | 08 00

ClientHello Records

3.Expand the ClientHello record. (If your trace contains multiple ClientHello

records, expand the frame that contains the first one.) What is the value of the

content type?

hex: 16 (16+6=22) Handshake

4. Does the ClientHello record advertise the cipher suites it supports? If so, in the first listed suite, what are the public-key algorithm, the symmetric-key algorithm, and the hash algorithm?

MD5, SHA, RSA, DSS, DES, AES

ServertHello Records

5. Look to the ServerHello packet. What cipher suite does it choose?

Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)

6. Does this record include a nonce? If so, how long is it? What is the purpose of the

client and server nonces in SSL?

Yes, 28 bytes. The ClientHello packet also generated a nonces. They are used to make the session communication between the two nodes unique. It "salts" the communication to prevent replay attacks. A replay attack happens when data from old communications is used to "crack" a current communication.

7.Does this record include a session ID? What is the purpose of the session ID?

Yes, This is to make things efficient, in case the client has any plans of closing the current connection and reconnect in the near future.

8.How many frames does the SSL certificate take to send?

In this case it took 4 frames

Zenoss or Nagios monitoring of HTTPS using client certificate authentication

2012-02-26T19:21:00-05:00

I recently needed to monitor an HTTPS API for response time and availability. At first I planned to just use the Nagios check_http command.

After gathering more requirements I learned that the API was protected by client certificate authentication. After some research I quickly found that no solution existed to monitor HTTP protected by client certs. I needed to write my own plugin.

This is the python plugin I came up with: check_http_client_cert.py

#!/usr/bin/python

"""Nagios/Zenoss client cert https checker"""

import httplib
from optparse import OptionParser
from time import time
from sys import exit

def request( hostname, port, cert_file, path ):
    """request a resource and return response object"""
    try:
        c = httplib.HTTPSConnection( hostname, port, cert_file=cert_file )
        c.request( "GET", path )
        return c.getresponse()
    except:
        return False

if __name__ == '__main__':
    parser = OptionParser()
    parser.add_option('-H', '--hostname', dest='hostname')
    parser.add_option('-p', '--port', dest='port')
    parser.add_option('-c', '--cert_file', dest='cert_file')
    parser.add_option('-P', '--path', dest='path',
    help="Path relative to root, like /image/search")

    o, args = parser.parse_args()
    #print o

    start = time()
    r = request( o.hostname, o.port, o.cert_file, o.path )
    elapse = time() - start

    if r:
        if r.status >= 200 and r.status < 400:
            print "HTTP OK:", r.status, r.reason, "|time=" + str(elapse) + "s;;;"
            exit( 0 )
        print "HTTP Critical:", r.status, r.reason

    exit( 2 )

Jake, Finn, and Ice King

2012-02-18T13:34:00-05:00

Jake from Adventure Time, painted with MyPaint and wacom tablet.

I did this because my 4 month old "cracks-up" laughing whenever this character is on screen. I think the combination of colors and Jake's voice, played by John William DiMaggio, get him going.

Finn from Adventure Time.

Took a shot at painting Finn.

I'm on a roll! Here is Ice King.

Jake standing up, painted this this morning

Lumpy princess

I'm getting faster, this was completed in 20 minutes

My 4 month old's 15 minutes of fame

2012-02-16T18:42:00-05:00

My sister @VeronicaBal was watching my son (@RussellBal) tweeted a picture of him.

The official @RedSox re-tweeted the image to all 197,035 followers!

I'm a proud daddy!

Block cipher lab

2012-02-14T01:36:00-05:00

Consider the following block cipher. Suppose that each block
cipher T simply reverses the order of the eight input bits (so that,
for example 11110000 becomes 00001111).
Further suppose that the 64-bit scrambler does not modify any bits.
With n = 3 iterations and the original 64-bit input equal to 10100000
repeated eight times, what is the value of the output?
Now change the last bit of the original 64-bit input from 0 to a 1.
Now suppose that the 64-bit scrambler inverses the order of the 64
bits.
Solution in python:

def chunks( l, n ):
    """accept a list and chuck size, return chunks"""
    return [ l[ i:i+n ] for i in range( 0, len(l), n ) ]


def T( blocks ):
    """for each block, reverse block, return blocks"""
    result = []
    for block in blocks:
        result.append( ''.join( [bit for bit in reversed( block )] ) )

    return result

def scrambler( input ):
    """inverse the order of input"""
    return ''.join( [i for i in reversed( input ) ] )


def cipher1( input, n = 3, chunk_length = 8 ):
    """make chucks out of input, reverse each chunk return result"""
    blocks = chunks( input, chunk_length )
    for i in range( 0, n ): blocks = T( blocks )
    return ''.join( blocks )


def cipher2( input, n = 3, chunk_length = 8 ):
    """same as cipher1 but with scrambler"""
    blocks = chunks( input, chunk_length )
    for i in range( 0, n ):
        blocks = T( blocks )
        blocks = chunks( scrambler( ''.join( blocks ) ), chunk_length )
    return ''.join( blocks )


if __name__ == "__main__":

    input = "1010000010100000101000001010000010100000101000001010000010100000"
    print cipher1( input )
    # output: 0000010100000101000001010000010100000101000001010000010100000101

    input = "1010000010100000101000001010000010100000101000001010000010100001"
    print cipher1( input )
    # output: 0000010100000101000001010000010100000101000001010000010110000101

    input = "1010000010100000101000001010000010100000101000001010000010100000"
    print cipher2( input )
    # output: 1010000010100000101000001010000010100000101000001010000010100000

    input = "1010000010100000101000001010000010100000101000001010000010100001"
    print cipher2( input )
    # output: 1010000110100000101000001010000010100000101000001010000010100000

Monoalphabetic Cipher and Inverse Written in Python

2012-02-13T22:39:00-05:00

introduction and background

A monoalphabetic cipher uses fixed substitution over the entire message.

You can build a monoalphabetic cipher using a Python dictionary, like so:

monoalpha_cipher = {
    'a': 'm',
    'b': 'n',
    'c': 'b',
    'd': 'v',
    'e': 'c',
    'f': 'x',
    'g': 'z',
    'h': 'a',
    'i': 's',
    'j': 'd',
    'k': 'f',
    'l': 'g',
    'm': 'h',
    'n': 'j',
    'o': 'k',
    'p': 'l',
    'q': 'p',
    'r': 'o',
    's': 'i',
    't': 'u',
    'u': 'y',
    'v': 't',
    'w': 'r',
    'x': 'e',
    'y': 'w',
    'z': 'q',
    ' ': ' ',
}

We can create an inverse of this cipher dictionary by switching the key and value places:

inverse_monoalpha_cipher = {}
for key, value in monoalpha_cipher.iteritems():
    inverse_monoalpha_cipher[value] = key

Now that we have both the cipher and the inverse_cipher, we may encrypt a message.

Encryption example:

message = "This is an easy problem"
encrypted_message = []
for letter in message:
    encrypted_message.append(monoalpha_cipher.get(letter, letter))

print(''.join(encrypted_message))

Result:: Tasi si mj cmiw lokngch

Using the inverse_cipher, We may decrypt a message.

Decryption example:

encrypted_message = "rmij'u uamu xyj?"
decrypted_message = []
for letter in encrypted_message:
     decrypted_message.append( inverse_monoalpha_cipher.get(letter, letter))

print(''.join( decrypted_message ))

Result:: wasn't that fun?

monoalphabetic_cipher.py

Here is a toy library I wrote to make the process repeatable -

monoalphabetic_cipher.py:

from string import letters, digits
from random import shuffle

def random_monoalpha_cipher(pool=None):
    """Generate a Monoalphabetic Cipher"""
    if pool is None:
        pool = letters + digits
    original_pool = list(pool)
    shuffled_pool = list(pool)
    shuffle(shuffled_pool)
    return dict(zip(original_pool, shuffled_pool))

def inverse_monoalpha_cipher(monoalpha_cipher):
    """Given a Monoalphabetic Cipher (dictionary) return the inverse."""
    inverse_monoalpha = {}
    for key, value in monoalpha_cipher.iteritems():
        inverse_monoalpha[value] = key
    return inverse_monoalpha

def encrypt_with_monoalpha(message, monoalpha_cipher):
    encrypted_message = []
    for letter in message:
        encrypted_message.append(monoalpha_cipher.get(letter, letter))
    return ''.join(encrypted_message)

def decrypt_with_monoalpha(encrypted_message, monoalpha_cipher):
    return encrypt_with_monoalpha(
               encrypted_message,
               inverse_monoalpha_cipher(monoalpha_cipher)
           )

monoalphabetic_cipher.py example usage

Here I show how to use the library:

>>> # load the module / library as 'mc'.
>>> import monoalphabetic_cipher as mc


>>> # generate a random cipher (only if needed).
>>> cipher = mc.random_monoalpha_cipher()

>>> # output the cipher (store for safe keeping).
>>> print(cipher)

>>> # encrypt a message with the cipher.
>>> mc.encrypt_with_monoalpha('Hello all you hackers out there!', cipher)
'sXGGt SGG Nt0 HSrLXFC t0U UHXFX!'

>>> # decrypt a message with the cipher.
>>> mc.decrypt_with_monoalpha('sXGGt SGG Nt0 HSrLXFC t0U UHXFX!', cipher)
'Hello all you hackers out there!'

Contents

introduction and background
monoalphabetic_cipher.py
monoalphabetic_cipher.py example usage

Why does a Hash provide better message integrity then an Internet checksum?

2012-02-13T19:19:00-05:00

Why does a Hash provide better message integrity then an Internet checksum?

Hash function and checksum function both return a value which cannot be reversed.

An Internet checksum (TCP checksum or IP checksum) is designed to detect common errors quickly and efficiently. An Internet checksum does not attempt to prevent collisions.

Man cksum for more info.

A Hash provides better message integrity because it has less collisions then an Internet checksum. A collision means there is more then one way to produce the same sum. A great hash function aims to reduce the occurrence of collisions. Man md5 and sha for more info.

What is a collision

Let H() be a hash function. Let x and y be two differing messages.

H(x) = H(y) would be a collision.

I like to use python to show examples of hash functions

In this example I pass a message into the MD5 hash function to produce a resulting hash of the message. You can think of this hashed output as a finger print of the message.

import hashlib as h
message = "This message will be placed into an MD5 hash function to authenticate its integrity."
print h.md5(message).hexdigest()

Hash Output:

18f189f94b245ad8566206c199b4f60a

Now If I passed that message to you along with its MD5 hash hex representation, you could put the message into your own MD5 hash function and compare the resulting hash. This method is used to validate the message or verify data integrity.

Can you "decrypt" a hash of a message to get the original message

No! A hash may not be reversed, which means it cannot be decrypted.

By design a hash algorithm has no inverse, there is no way to get the original message from the hash. This is good news, turns out we have some really great applications for this type of function. We can validate messages, we can securely store passwords, and we can quickly determine if a message or file has been tampered with.

When using a publicly known hash function for storing password hashes, make sure to always use a salt or shared secret. Failure to do so will make your storage scheme susceptible to a rainbow table attack. A rainbow table allows a cracker to quickly match a list of hashes with a table of previously computed hash values and correlated passwords.

What is salt or a shared secret?

You can use salt or a shared secret to add extra data to a message before hashing with a publicly known algorithm. Below I will document how to properly add salt to a message before generating a SHA256 hash.

import hashlib as h
message = "This message and some salt will be hashed with SHA 256."
salt = "This is some secret salt data"
print h.sha256(message+salt).hexdigest()

Hash Output:

5e8d86bab9604620f19cfbc5f836f47feb9e8c9e74264fff1f4938bdaab1eeaa

Adding a salt to the message allows us to use a publicly know algorithm in a more protected manner.

Can you spot the error in the python code below?

import hashlib as h
message = "This message and some salt will be hashed with SHA 256."
salt = "This is some secret salt data"
print h.sha256(message).hexdigest()+salt

If you guessed that the message and salt BOTH need to be hashed together then you are correct!

The above code would have produced the following invalid hash:

Hash Output:

79cd4bfa1bcb71a7a1b5bfd5e8cfc8368a6cc6cb836d24bf04f2ef2bd0e81261This is some secret salt data

You should follow me on twitter: @russellbal

Symmetric Encryption vs Public Key Encryption

2012-02-13T17:25:00-05:00

How many keys are involved for symmetric key encryption? How about public key encryption?

Suppose you have N people who want to communicate with each other using symmetric keys. All communication between any two people, i and j, is visible to group N. Only person i and person j can decrypt each others messages.

How many keys would Symmetric Encryption require to protect group N?

I solved this with the following python function:

def count_symmetric_keys( N=2 ):
    """Provide the number of entities in group N.
    return the number of symmetric keys needed for this group"""
    keys = 0
    for i in range( 0, N ): keys += i
    return keys

A reader suggested the following optimized formula:

def calc_symmetric_keys( N=2 ):
    """Provide the number of entities in group N.
    return the number of symmetric keys needed for this group"""
    return N*(N-1)/2

If group N had 10 members, it would need to generate and maintain 45 Symmetric Keys.

If group N had 50 members, it would need to generate and maintain 1225 Symmetric Keys.

Symmetric keys are also susceptible to man-in-the-middle attacks. This attack occurs when an entity poses as a trusted entity. Let i and j be trusted entities. Let k be an untrusted attacker. If k determined the Symmetric key it could send or receive messages posing as i or j.

How many keys would Public-key Encryption require to protect group N?

Public Key Encryption requires 2n keys or two keys per person in group N. Public key encryption also does not require 'pre sharing' the secret key before communication may start. Each member would need 1 public key and 1 private key.

If group N had 10 members, it would need to generate and maintain 20 Public/Private Keys.

If group N had 50 members, it would need to generate and maintain 100 Public/Private Keys.

Attributes of an 8-block cipher

2012-02-13T00:50:00-05:00

Consider an 8-block cipher and answer the following:

How many possible input blocks does this cipher have?

How many possible mappings are there?

If we view each mapping as a key, then how many possible keys does this cipher have?

To find the input blocks of this cipher we raise 2 to the 8th power. 2^8 = 256 possible inputs.

To find the number of possible mappings we take the 256 input blocks and find it's factorial. There are 256! possible mappings.

We can view each of these mappings as a key, so this cipher has 256! keys.

Reasons why some Internet entities might want secure communication

2012-02-08T18:23:00-05:00

Internet entities often desire to communicate securely, some reasons include:

Web Servers: Communication on the Internet, or any network for that matter, should be encrypted before transmitting sensitive data. This will help prevent snooping from unauthorized parties. Most often SSL or HTTPS may be used to create a secure communication “tunnel” between a web server and a web client (browser).
Server Administration: Operators should always use a secure protocal when working on a server or remote computer. SSH (Secure Shell) wins the popularity contest with server admins.
DNS Servers: Using DNSSEC could help prevent DNS poisoning and certifies DNS data. DNS was first conceived as a distributed and highly scalable address lookup system. Security was not its top priority. Since then we have new DNSSEC extensions which allow for origin authentication of DNS data, authenticated denial of existence, and data integrity. DNSSEC does not attempt to solve availability and confidentiality.

What are the differences between message confidentiality and message integrity

2012-02-08T17:47:00-05:00

What are the differences between message confidentiality and message integrity? Can you have confidentiality without integrity? Can you have integrity without confidentiality?

message confidentiality: Two or more hosts communicate securely, typically using encryption. The communication cannot be monitored (sniffed) by untrusted hosts. The communication between trusted parties is confidential.
message integrity: The message transported has not been tampered with or altered. A message has integrity when the payload sent is the same as the payload received.

Sending a message confidentially does not guarantee data integrity. Even when two nodes have authenticated each other, the integrity of a message could be compromised during the transmission of a message.

Yes, you can have integrity of a message without confidentiality. One can take a hash or sum of the message on both sides to compare. Often we share downloadable files and provide data integrity using md5 hash sums.

Today I lost a customer

2012-01-18T23:30:00-05:00

Today I lost a customer.

I added some new code to LinkPeek to accept coupons and I didn't think of an edge case. This ended up creating an uncaught exception in my server side code which ultimatly served the newly subscribing customer an HTTP 500 error page.

The damage was done.

This error was catastrophic and ultimately killed the conversion. Here is an excerpt of how the user felt after the experience:

At this point the website has failed spectacularly enough that I can no longer trust you with my business. Please void the charge on my American Express card before it's processed. I need to hear from you ASAP.

Customers and prospects are forgiving for normal bugs in software. However, customers are intolerant to bugs in the sign up or payment flow. An error in payment flow will cause friction and friction will kill the sale.

Running a start up is hard work, and negative (but justified) feedback hurts more then I thought it would. This email made me feel awful.

Fortunately I learned from this mistake and the user's card was never charged.

Always make sure to extensively test payment and sign up code. Try all the edge cases. Try to make your software break. Don't inflict friction on your potential customers.

LinkPeek.com, webpage to image, was a by-product

2011-12-19T00:23:00-05:00

tldr; When faced with pivoting or killing a project, take a good look at all possible by-products. Don't miss the hidden gem in a project's slag!

Last year I built yoursitemakesmebarf.com, a novelty web application which allowed anonymous link submission. The software would automatically take screenshots of submitted links and curate a blog. I enjoyed building the site and the project served as my first Pyramid application.

The project's original intent was to jokingly poke fun at ugly design. The idea never caught on. Instead the application angered website owners and attracted undesirable people. Eventually, I decided to take it down and come up with less combative idea.

After witnessing the Goog release of "instant previews", I knew there was a market for a fast and reliable web screen shot service.

https://linkpeek.com

So I decided to bring instant previews to anyone who needed them. I wanted to build a fast, flexible, and easy to use screenshot API. After a few months I had a working prototype. I named the product LinkPeek because it described what the service was, and the domain was available.

Next I built a website thumbnail generator to show off the software. The generator application helped reinforce the simplicity of the underlying LinkPeek API. It didn't require any downloading, installation, or waiting.

About a week later on a whim I posted the generator to hacker news with an honest title (Convert Any Webpage to an Image) and in about 30 minutes it reached the number 1 spot.

Remember, when faced with pivoting or killing a project, take a good look at all possible by-products. Don't miss the hidden gem in a project's slag!

flash mob office meeting definition

2011-12-12T15:34:00-05:00

Flash Meeting

In a office or cubicle environment a group of uninvited people gather and hover around your desk to talk to you. A meeting forms in immaculate conception as you sit bewildered at your desk.

Other names: Flash Meeting, Flash Mob Meeting, Flash Office Meeting

LinkPeek.com Number One on Hacker News

2011-12-05T00:39:00-05:00

We made the number one spot on Hacker News.

LinkPeek was used to take a snapshot of the event:

How to Incorporate Custom Configuration in a Pyramid Application

2011-11-30T22:47:00-05:00

Note: This post shows an old way to modify the request object. I discuss a new way in my Pyramid add_request_method post.

Imagine that you have just built a wiki, blog, or cms web application that will be deployed multiple times by different people. You would like to provide the ability to configure some aspects of the program without the end user altering your python or template code. This guide explains how to incorporate custom configuration from the project's .ini file with the rest of the application.

To explain this process we will add a customizable Google Analytics key to our project.

Make a configuration key/value pair for Google Analytics
Add the Google Analytics javascript code to the template

Make a configuration key/value pair for Google Analytics

Inside production.ini place the following in the [app:main] section:

#google_analytics_key = UA-55555555-1
google_analytics_key =

Add the Google Analytics javascript code to the template

In this case I will show an example in mako. Other template solutions should look similar.

<%def name="google\_analytics()">
% if request.registry.settings['google_analytics_key']:


 <script type="text/javascript"></p>
 <p>var _gaq = _gaq || [];<br />
           _gaq.push(['_setAccount', "${request.registry.settings['google_analytics_key']}"]);<br />
           _gaq.push(['_trackPageview']);</p>
 <p>(function() {<br />
           var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;<br />
           ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'https://www') + '.google-analytics.com/ga.js';<br />
           var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);<br />
           })();</p>
 <p></script>

% endif

Then in the head section call the function:

${ google_analytics() }

That was easy because the configuration string just needed to be substituted into the JavaScript in the template. What if you needed to do something with the provided key/value before using it? Next I will show you a method for building renderer globals again showing a different way to configure Google Analytics:

Make an inject_renderer_globals subscriber

Define inject_renderer_globals(event) function in the project's __init__.py file.

I normally place it at the bottom of the file and it looks like this:

def inject_renderer_globals(event):
    """Inject some renderer globals before passing to template"""

    request = event['request']

    # Build ${google_analytics_key} from the configuration file
    event['google_analytics_key'] = request.registry.settings[ 'google_analytics_key' ]

Import BeforeRender at the top of the __init__.py:

from pyramid.events import BeforeRender

In the main function, add the inject_renderer_globals to the subscribers:

config.add_subscriber(inject_renderer_globals, BeforeRender)

Now you can use ${google_analytics_key} anywhere in your template.

Thank you for reading, and feel free to leave comments

Career development is a game of chutes and ladders

2011-11-29T20:42:00-05:00

If career development was a game of chutes and ladders, job networking would be the ladder. Ladders provide a shortcut to the top, a direct route to win, in this case you win your dream job.

At work today, a colleague was reviewing resumes for an open requisition within the Unix group. Later that night I decided to clean up my resume to make it more relevant.

I felt like I did something positive for my career. After coming down from the high of resume writing I began to question my rational and came to the contradictory conclusion that a great resume holds less importance than a mediocre recommendation.

It is better for an employer to learn about you from a recommendation then your resume.

Why?

Hiring new people holds risk. A hiring manager will reduce risk by promoting from within or using recommendations. In both cases the resume becomes a document of formality instead of a document of credentials.

"Its not what you know, its who you know."

What can we assume about the position they are extending to the public?

Can we assume that the manager has already promoted somebody internally for the high risk, enjoyable position? Depending on the industry, they might be looking for bottom feeder (with a resume) to fill the newly vacant position. Keep this in mind the next time a head hunter sends you a proposition.

Taking shortcuts normally goes against the conventional wisdom for success.

Job networking however, allows candidates to skip ahead and arrive at their dream occupation more directly. Meeting new people seems like the single best way to land a job doing what you love. Get out there and meet people with similar interests!

Did you like this post?

You should checkout the first chapter of my e-book, How-to Work From Home titled The Road to Remote

I'm petrified of launching my web application

2011-11-03T21:35:00-04:00

I'm petrified of launching my web application because I'm fearful that I won't ...

acquire users
support my users well
scale in a timely manner
react quickly to feedback
monetize the application

But most of all I'm scared that nobody will like me. I'm scared of failure.

Now that I got that out of my system please check out https://linkpeek.com

Webmaster tools alerted issue turned out Pylon session files flooded inodes

2011-10-29T15:30:00-04:00

This graph could happen to you if you ever forget to configure munin email alerting:

It only took approximately 1 hour to diagnoses and resolve this issue however most of my web applications hosted on this server were down for about 11 hours. I was lucky that this outage fell on a weekend otherwise I would not have known about the problem till around 6:30pm!

Diagnosis:

Two of my pylons apps had session files that slowly ran away on me. The session files don't consume much capacity however the shear quantity of them caused my inode usage to hit 100%.

Had I properly configured Munin's email alerting this issue would have been identified well before it was a problem.

Want to know what alerted me to the problem? G Webmaster's tools claimed it could not read my robots.txt on a few of my sites... After investigating I learned the site was down. Checking the Apache error logs pointed me to disk space issues. df -ha reported everything was fine, however df -hi reported 100% inode usage! At this point I started looking to cache and log locations to find lots of files, which lead me to my pylons web applications data/sessions directories.

Resolution:

Delete the session cache tree directories and allow the applications to rebuild them.

Todo: move /www off the root disk partition. This issue could have been much worse if I was unable to boot or login to remedy. Moving /www off root should prevent the web server from effecting the systems ability to boot.

Occupy Wall Street Stack vs Queue

2011-10-21T13:31:00-04:00

Occupy Wall Street contributors claim to use a "stack" to determine speaking arrangements.

I plan to explain how the term "stack" used in this scenario does not align itself with the mathematical or computer science definition.

The term stack means First In Last Out or "FILO". For example: a person placed on the stack in the morning would be the last to speak at the end of the day. This isn't happening like that...

Occupy Wall Street compatriots really use a technique called "queue". Mathematicians and Computer Scientists define a queue as First In First Out or "FIFO". A real world example of a queue would be a line at the grocery store. The first person in line is the first person to leave the line.

An object enters a stack from the rear and exits the rear.
An object enters a queue from the rear and exits from the front.

Adding inline image support to a gmail messages

2011-10-20T17:46:00-04:00

Enable ability to insert images into a message body. You can upload and insert image files in your computer, or insert images by URLs. This lab will not work if you have offline enabled.

Go to google labs: https://mail.google.com/mail/#settings/labs
Search "images".
Enable "Inserting images - by Kent T"

Welcome!

Cell shading practice, A sketched skull, and an old wind mill photograph

2011-10-16T12:56:00-04:00

Some of my resent works using my Ubuntu + Wacom Bamboo + MyPaint + Gimp Workflow!

First some cell shading practice. Inside MyPaint I only used two colors in my palate but to gain shadow and depth I used variations of opacity. This technique produces a neat cartoon effect.

Next I sketched a skull. I attempted to keep all my strokes in the same direction. This piece is monotone. I used only one color and deviated each layer with different opacity settings.

Last is a photograph I snapped during a family outing. I had to lower the quality of this image to prepare for use on the web, but I still adore the colors spectrum of this shot. The white birch tree frames the right side while the grass and brush frame the bottom. The wind mill was intentionally placed in the center with the apex extending upward. The ivy creeps up the structure and the cloud filled sky appears to stretch forever. The foliage emits a euphoric glow under the sun saturated rays.

Come back soon!

I cancelled my xbox live automatic renewal

2011-10-15T19:26:00-04:00

I cancelled my xbox live automatic renewal today because I no longer use the service.

I find humor in Microsoft's list of reasons to keep Xbox live:

6 out of 8 services above are FREE Internet services!

If I want to use those free Internet services on my TV, I'd rather use my tiny media center PC (nT330i) which is amazing and near silent.

Microsoft prevents subscribers from cancelling their service over the web, and thus they forced me to call their service center. After waiting 5 minutes I finally spoke to a sales representative who seemed friendly and understanding.

She asked me why I wanted to cancel automatic payments. I explained "I want to pay month to month". After listening to my position the sales representative offered 1 month of Xbox Live for $1.00. So I'll have Xbox live until December 2011 after all.

Microsoft, give your users a web browser on the Xbox and stop trying to sell subscriptions to Free Internet services.

*PlayStation 3 launched with a web browser and a FREE network.*

UPDATE: If you opt into the $1 for one month deal, you are also opting back into the automatic renewal program... Even if the person on the phone tells you otherwise.

Call us directly by dialing:

Toll free:
(800) 4MY-XBOX or (800) 469-9269
Hearing impaired (TDD device):
(866) 740-9269 or (425) 635-7102
9 am to 1 am Eastern Time
6 am to 10 pm Pacific Time

r8168 driver issues after Ubuntu 11.10 upgrade kernel linux 3.0

2011-10-15T11:15:00-04:00

I had network issues after upgrading to Ubuntu 11.10 which has the linux 3.0 kernel.

I used this guide to compile the r8168 driver however, I needed to alter the Makefile to support for linux 3.0 kernel.

Edit src/Makefile:

#KEXT  := $(shell echo $(KVER) | sed -ne 's/^2\.[567]\..*/k/p')o
KEXT := $(shell echo $(KVER) | sed -ne 's/^[23]\.[1-9]\..*/k/p')o-

Now you should follow the rest of the guide.

A better way to show website backlinks

2011-10-12T20:05:00-04:00

Early web pilgrims of the Internet fashioned search queries like

link:russell.ballestrini.net to gather backlinks for a domain.

I however advocate a revolutionary search pattern like -

"russell.ballestrini.net" -site:russell.ballestrini.net

to gather an improved representation of backlinks.

Click here to try now!

CSS frameworks not rendering properly on all browsers

2011-10-11T00:07:00-04:00

I ran into an issue when testing skeleton css framework and twitter bootstrap where some browsers were not rendering the pages properly. After some research I determined that I forgot the DOCTYPE declaration tag.

The DOCTYPE tag tells the browser what "rules" or standards to use when rendering markup.

The following code block displays the minimal approach to setting the document type:

<!DOCTYPE html>

This doctype tag forces the broswer into standards mode which allows the pages to render properly.

Thanks!

LinkPeek.com web address thumbnail api alpha release

2011-10-05T20:58:00-04:00

https://linkpeek.com

The LinkPeek API had an alpha release!

LinkPeek is a website screenshot service. Convert any webpage to an image. No software, no downloading and absolutely no waiting! Unlimited free 140 pixel thumbnails!

How do I calculate the M in my MVP?

2011-10-01T15:32:00-04:00

MVP (Minimal Viable Product):: The smallest version of your idea that produces value for customers.

https://www.remarkbox.com

Recently I have contemplated the idea of building a screen capture service for websites. I anticipate the service would provide functionality similar to Google's "Instant Preview". I also pondered the idea of archiving the screen captures and providing a history similar to the internet Wayback Machine.

After floundering on my first web application, https://four2go.gumyum.com, I'm afraid to spend lots of time and energy on another project if I cannot earn users or customers.

https://four2go.gumyum.com

I feel the market segment for this type of service would include website directories, live portfolio generation for website designers, and websites in general for people who wish to keep track of the overall look of their site over time.

As an MVP I plan to build a free live web capture application to basically provide a "Gravatar" for website addresses.

https://linkpeek.com

Do you think anyone would use my MVP or service?

LinkPeek provides an easy to use API which enables anyone to instantly create live web page thumbnails. We provide the API free of charge, think of LinkPeek as Gravatar for web addresses.

Have I missed any other market segments?

New Baby Homecoming

2011-09-27T16:53:00-04:00

Hacker Olive Oil Lamp Crafted From Home Materials

2011-09-09T19:01:00-04:00

In loom of the recent and persisting hurricane season I present a "Hacker's Olive Oil Lamp"!

YES, this lamp is ...

Fun and simple to build
Fun and simple to use
Easy to Cleanup
Gentle on environment
Energy efficient (1 cup ~= 16 hours)
Safe (flame smothers if lamp tips over)

Odor
Smoke
Heat near base
Cost (household materials)
Waste (Store and seal oil in lamp with jar top)

Materials

1x Pickle jar with lid
1x Washcloth 100% cotton
1x Steel wire (salvaged from paint can handle)
2x Cups of olive oil

Tools

Needle nose plyers
Scissors

[gallery link="file"]

A system administrators guide to installing and maintaining multiple python environments

2011-09-08T19:28:00-04:00

Some operating systems depend on a specific version of python to function properly. For example, Yum on Redhat Enterprise Linux 5 (RHEL5) depends on python 2.4.3. This version of python lacks support from many utilities and 3rd party libraries. This guide will cover installing an alternative python instance while leaving the system's python alone.

This guide supports the following operating systems: Redhat, CentOS, and Fedora. As of this publication the latest Python version was 2.7.8; You might want to determine if a newer version exists.

Install Python 2.7

I found two methods which work for installing Python 2.7 side-by-side with the system's Python.

compile from source
install package via the scl repo

Choose which ever strategy you find easier.

Compile from source

Gather the dependencies:
```
yum install gcc zlib-devel python-setuptools readline-devel
```
- gcc is a compiler used to build python
- zlib-devel allows the python zlib module to be built
- python-setuptools provides the easy_install application
- readline-devel arrows readline and history handling in python shell

Download and untar the python sourcecode:

wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
tar -xzvf Python-2.7.8.tgz
cd Python-2.7.8

Compile the sourcecode:
```
./configure
make altinstall
```
Test new alternative python:
```
python2.7 --version
```
Document path to new python2.7, you will need it if you plan to use a virtualenv:
```
which python2.7
```
Now we can install third party libraries into our alternative python:
```
python2.7 -m easy_install
```

Install package from scl

install centos-release-scl repolists:
```
sudo yum install centos-release-scl
```
install python27:
```
sudo yum install python27
```
enable python27 via scl:
```
scl enable python27 bash
```
Document path to new python2.7, you will need it if you plan to use a virtualenv:
```
which python2.7
```

virtualenv

Optionally we can create a virtualenv (for development) based on the python 2.7 install. Virtual environments appear useful for testing packages and libraries without installing them to the system owned python site-packages directory.

Install pip using easy_install:
```
sudo easy_install pip
```
Install virtualenv using pip:
```
sudo pip install virtualenv
```
Create a new virtual python environment named virtpy:
```
cd ~
virtualenv -p /usr/local/bin/python2.7 virtpy
```
This will create a virtual python 2.7.2 environment named virtpy in your present working directory.

To invoke this environment run source virtpy/bin/activate and your prompt should change to reflect the active virtualenv.

Now you can run easy_install to install packages into virtpy/lib/python2.7/site-packages.

Thanks for reading, that's all for now.

Contents

Install Python 2.7
- Compile from source
- Install package from scl
virtualenv

Deliberating the Viewers vs. Doers concept

2011-09-06T10:20:00-04:00

Brett & Kate McKay from artofmanliness.com recently subscribed to an idea that America has succumbed to "spectatoritis, a blanket description to cover all kinds of passive amusement, an entering into the handiest activity merely to escape boredom." -Jay B. Nash, 1938

The basic concept discussed how people can fall into one of two groups when interacting in life, the viewers and the doers. Viewers or spectators watch an activity while doers perform the said activity. For example:

Viewers

Sports Fans
Audience
Couch Potatoes

Doers

Athletes
Musicians
Actors

Although the above patterns persists in the physical world, intellectual situations operate differently. For instance on the surface reading seems like a viewer or spectator task and writing appears to behave like a doer task. But what happens when a reader learns or responds to a blog post? We seem to need a stronger definition what constitutes doing a task.

We should not regard spectating as bad, however we should try to stay lucid of our current rolls when engaging in activities.

Spectating has some important purposes in our society and culture. Viewing a performance grants power to the actor, whether that person plays football or plays guitar. For this power and influence to occur a viewer must witness and pay attention to the event. What if the next great political leader took the podium and no one bothered to view his speech?

The truth is our American society has conditioned us to spend much of our lives spectating. During school a good student will view and listen to the lecture. While driving to work one radio DJ broadcasts his thoughts to many.

Spectating also plays and integral roll in learning. Human infants learn by first watching and then imitating. Experts also agree that inspiration for a creative works often occur after observation of prior productions.

tl;dr We should not battle the viewers vs the doers because both hold importance and need each other.

If you liked this article, you should follow me here

Python Image Grabber pig.py

2011-08-22T18:48:00-04:00

Python Image Grabber pig.py

Python Image Grabber or pig.py is a very simple python command line tool to download all the images from a given uri.

Enjoy!

Download and Installation:

https://bitbucket.org/russellballestrini/pig/raw/tip/pig.py

Usage:: python pig.py

Example:: python pig.py https://www.foxhop.net

Open Source:: pig.py has been placed in the public domain and its sourcecode may be viewed or branched here here: https://bitbucket.org/russellballestrini/pig

Pigpy Gimp Project: pigpy.xcf

Security Professionals: Yes we appear vulnerable but that attack vector will never happen

2011-08-18T21:36:00-04:00

In loom of recent internet attacks many institutions have started scrambling in attempt to "strengthen" their security stance. I agree that auditing our systems and networks for potential flaws seems appropriate at this time to prevent getting "caught with our pants down". Incidentally, I have recently witnessed the introduction of silly and at times ineffective security adjustments. Many of these new procedures, rules, and requirements do not make us more secure and worse instill a false sense of security.

I have previously addressed the fallacy of absolute security. No system is perfect. A successful security model accomplishes fortitude by implementing layers like an onion. Through the use of security layers we can significantly hamper attack vectors and create a safer complex.

When analyzing a potential attack vector we must first determine our current location in the security layers. This step serves two purposes:

to prevent wasting time and energy on vulnerabilities that don't matter at that point in our matrix.
to prevent causing outages and unneeded administrator and customer heartache.

If a vulnerability requires root or elevated privileges to occur, don't waste your time resolving it. If the attacker already has root, you have bigger problems on your hands.

Some real life examples:

Firewall denying a large range of IP addresses (like entire countries). This truly does not increase security, it just creates headaches for users. An attacker could just proxy to an open range (like a VPS based in a more trusted zone) and gain access from there. Also if you decide to ignore this advice and create blanket IP range deny rules, DON'T also block services intended to be internet-facing. For example, don't block your Internet-facing DNS server if it is authoritative for public domains. This will cause countless intermittent issues and will be a nightmare to diagnose.
Weekly Scanning for Windows viruses on network shares or data at rest. This hammers the servers for no reason. If all the desktops run antivirus then the file was already scanned when it was downloaded. That same file will be scanned again when retrieved on the share. If you want the warm and fuzzys of virus scanning network file shares, do it once a year. These scans waste time and resources. I feel even more outraged when asked to virus scan network shares hosted on UNIX servers or NAS.

I speculate that most of these arbitrary ideas come about because the people in charge make uninformed decisions out of fear without first consulting the appropriate subject matter experts.

Unfortunately, once a security mandate occurs it seems difficult to expunge. People are just not willing to put their neck on the chopping block to banish a legacy or silly mandates; So we end up living with nonsensical rules and procedures.

virt-back: restoring from backups

2011-08-10T23:20:00-04:00

In a perfect world we should create backups but never need them. Although this statement holds truth, creating guest backups provides many more benefits.

The most common reasons system administrators restore from a virt-back guest backup:

recovering from data corruption
recovering deleted files
recovering from a virus infection
recovering from a compromised server
backing out a failed change
rolling back to a previous state
testing disaster recovery plans
cloning a server
building test environments

During this article we will cover how to restore a system from a virt-back guest backup. This article will not cover how to restore a VM host server.

Virt-back guest restore procedure

In this guide our guest mbison has failed with a major corruption and we would like to restore from our backups. We have our running production guest images in /KVMROOT and our virt-back guest backups in /KVMBACK. We will be restoring the backup on the same hypervisor.

Overview:

Ensure the guest is shut off.
move the bad image file out of the way
untar the virt-back backup into place
power up the guest

Detailed Procedure:

Verify the guest is shut off by running:
```
virt-back --info-all
```
We noticed that mbison was still running so we invoked:
```
virt-back --shutdown mbison
```

Move the corrupted image file out of the way:

mv /KVMROOT/mbison.img /KVMROOT/mbison.img.NFG

Unzip and unarchive the backup using the following command:

sudo tar -xzvf /KVMBACK/mbison.tar.gz -C /KVMROOT --strip 1

When the untar completes, start the guest:
```
virt-back --create mbison
```
Connect to the guest over SSH and verify that all required services and applications start. Determine if the restore was successful.

Restore guest backup on new hypervisor:

The details in this section were adapted from a tutorial given by Fabian Rodriguez.

Re-create any bridge network interfaces on new hypervisor (/etc/network/interfaces for Debian)
Adjust mbison.xml if needed (for example if you are changing paths)

sudo mkdir /KVMROOT
sudo tar -xvzf mbison.tar.gz -C /KVMROOT --strip 1
virsh create /KVMROOT/mbison.xml

Note: We use virsh create instead of virt-back create. While both commands start guest DOMs, virsh create will also register the DOM into the hypervisor.

Programming is like Alchemy

2011-08-08T21:52:00-04:00

Programming is like Alchemy except instead of exchanging matter, we programmers exchange time. Also depending on the program the exchange of time worked (coding) increases the productivity (time) of its users.

On second thought, perhaps programmers correlate less with Alchemists and more with Time Travelers; Or at the very least time manipulators. For example, on a good day a programmer can easily complete a task that would take one thousand men. See, time created! On a bad week we can procrastinate and do nothing at all. Time lost!

Programming embodies other magic like wizardry. For example, our programs typically live as golems performing one task, repeatedly, over and over. Golem programs, without a soul, stuck in a loop of servitude.

Recently we have started coding creations with artificial intelligence. These smart programs act like familiar spirits (Wikipedia) and assist their creator in conjuring even more magic.

So we settled it! Programmers are like bad ass, time travelling, wizard alchemists!

Or maybe not ...

It is more accurate to group programs with technology then magic, but less fun. Programs are leveraged tools born to save people time and energy.

Thanks for reading, you should follow me on twitter here

The Great Gist Heist

2011-08-07T02:30:00-04:00

The Great Gist Heist

I have crawled, downloaded, and archived all of gist.github.com. Please hear my story before jumping to conclusions.

Why?

I'm currently building software that requires a large corpus of source code.

I began to search for a collection of source code documents but my pursuit appeared fruitless. Feeling displeased I attempted to gather all of my own source code. My collection lacked fidelity perhaps because of my revere for the python language.

Regardless of the reasoning, I needed a higher quantity of samples. I needed unbiased samples from all programming languages. I needed, most importantly, samples in a variation of quality that only the most popular paste sites have... sites like gist.

Why are you sharing it?

I feel a little bad about using Github's bandwidth.

Sharing this collection should reduce the chances that others will crawl for the same data. If you need a large collection of source code, download this torrent.

How did you do it?

I wrote a short, 30 line, python script. The script is part of the torrent.

At the peak of the scrape I had 14 threads running of the script, using approximately 580Kbps (I used iftop).

a hack to gain 80 percent efficiency when creating github projects

2011-07-11T21:47:00-04:00

Last night I decided to learn how to use git for its popularity and github to code more socially.

I have to admit early on that I enjoy hg and bitbucket so it came to a surprise that github would have me jump through hoops to create a new repository...

Below I have copied the seemingly bloated github instructions for creating a new project.

Github:

create project/repo using the form
mkdir scratch
cd scratch
git init
touch README
git add README
git commit -m 'first commit'
git remote add origin
git@github.com:russellballestrini/scratch.git
git push -u origin master

Wow...

I like my method better.

My Github clone hack:

create project/repo using the form
git clone https://russellballestrini@github.com/russellballestrini/scratch.git

Awesome, 2 steps versus 10. That hack appears 80% more efficient! Now if only my name was shorter... LOL

You should follow me on twitter here

Add a Breadcrumb Subscriber to a Pyramid project using 4 simple steps

2011-07-02T17:25:00-04:00

Note: This post shows an old way to modify the request object. I discuss a new way in my Pyramid add_request_method post.

This article will explain how to add a breadcrumb subscriber to a Pyramid project using 4 simple steps.

While programming https://school.yohdah.com/ I needed the ability to easily create breadcrumb links from the current url. You may view the bread.py source code here. The following guide describes the process I took to add this functionality.

1. Download and include bread.py at the top of your Pyramid project's __init__.py file:

from yourproject.bread import Bread
from pyramid.events import subscriber, NewRequest

2. At the bottom of the projects __init__.py file create the following subscriber function as follows:

def bread_subscriber( event ):
    """ Build Bread object and add to request """
    event.request.bread = Bread( event.request.url )

3. Now we can add our new subscriber to our Pyramid config inside main and above the routes:

config.add_subscriber(bread_subscriber, NewRequest)

4. You are done! Now you should have the ability to use the bread object from your template. Below I have provided a mako template snippet:

% for link in request.bread.links:
  ${ link | n } /
% endfor

Google Bot Attempts to Crawl Shortest Urls First

2011-06-25T09:37:00-04:00

Recently I built https://school.yohdah.com a Python, Pyramid, and mongoDB project during the last couple weekends.

The site features a directory style navigation of nearly every public school in the US. We have 61 state pages, approximately 19,000 city pages, and over 103,000 school pages.

It seems the Google Bots have noticed school.yohdah.com and started crawling the site. Since the initial crawl I started reviewing a sample of the sites apache logs in an attempt to track the bot's activity. After a few minutes of viewing the logs, I locked onto a pattern; Google Bot's algorithm appears to crawl the short URLs first!

PersonalCompute (a user) attached a graph of the fetched URL lengths here:

I have attached a zip containing the apache google bot crawl logs here: access-school.yohdah.log.zip

I found the pattern by opening the file in vim and scrolling very quickly down. You will notice the log lines will grow slowly to the right, as the urls being fetched increase by one character.

Why does Google do this? Does anyone have speculation as to what this means?

ATT Uverse Residential Gateway Broadband LED flashes red intermittently

2011-06-18T10:08:00-04:00

ATT Uverse Residential Gateway Broadband LED flashes red intermittently

I have struggled with my uverse Internet and TV service randomly and intermittently shutting off for the last 3 months. When this occurs the Residential Gateway's broadband LED flashes red.

Also I get Uverse DSL link errors:

When I 'live support chatted' with the uverse technical support they claimed I had bridge tap on my lines, and that my lines had other 'issues'.

They had to dispatch 3 technicians before one of them figured out the problem. It turns out I had a faulty 'voice/data' splitter.

I snapped a picture of the demarcation box before and after he replaced the faulty splitter.

Old Uverse filter on the left, New filter on the right

So far I have had no Uverse DSL Link Errors!

Connecticut killed affliate marketing with Amazon.com

2011-06-11T14:10:00-04:00

Connecticut killed affliate marketing with Amazon.com

I just opened my amazon account and read this:

Account Closed

This account is closed and will not generate referrals. Access to this site is for historical purposes only.

When I checked my email, I had this letter from Amazon.com:

Hello,

For well over a decade, the Amazon Associates Program has worked with thousands of Connecticut residents. Unfortunately, the budget signed by Governor Malloy contains a sales tax provision that compels us to terminate this program for Connecticut-based participants effective immediately. It specifically imposes the collection of taxes from consumers on sales by online retailers - including but not limited to those referred by Connecticut-based affiliates like you - even if those retailers have no physical presence in the state.

We opposed this new tax law because it is unconstitutional and counterproductive. It was supported by big-box retailers, most of which are based outside Connecticut, that seek to harm the affiliate advertising programs of their competitors. Similar legislation in other states has led to job and income losses, and little, if any, new tax revenue. We deeply regret that we must take this action.

As a result of the new law, contracts with all Connecticut residents participating in the Amazon Associates Program will be terminated today, June 10, 2011. Those Connecticut residents will no longer receive advertising fees for sales referred to Amazon.com , Endless.com , MYHABIT.COM or SmallParts.com . Please be assured that all qualifying advertising fees earned on or before today, June 10, 2011, will be processed and paid in full in accordance with the regular payment schedule.

You are receiving this email because our records indicate that you are a resident of Connecticut. If you are not currently a resident of Connecticut, or if you are relocating to another state in the near future, you can manage the details of your Associates account here . And if you relocate to another state after June 10, 2011, please contact us for reinstatement into the Amazon Associates Program.

To avoid confusion, we would like to clarify that this development will only impact our ability to offer the Associates Program to Connecticut residents and will not affect their ability to purchase from www.amazon.com .

We have enjoyed working with you and other Connecticut-based participants in the Amazon Associates Program and, if this situation is rectified, would very much welcome the opportunity to re-open our Associates Program to Connecticut residents.

Regards,

The Amazon Associates Team

Summary

Amazon.com does not have a facility in CT.
CT passed a new bill.
The bill declares: if a customer from any state clicks on a CT resident's affiliate link and purchases an item, that customer/Amazon.com should have to pay CT tax.

My Opinion

This seems bad.
I have less money to spend in Connecticut.
This smells like double taxation.
One more reason to move out of Connecticut when looking for Tech or IT jobs.

Dropbox Encryption with TrueCrypt

2011-04-16T12:50:00-04:00

Derek Newton recently invoked discussion about insecurities in Dropbox authentication. In his article he describes how an attacker could exploit Dropbox and gain access to unshared files. The concerns he raised do appear accurate however we must remember that security is an onion.

An onion, like security, has layers to protect its vital parts. The vital parts are more vulnerable when its security model only possess one layer. As we add layers to our security model, our system's protection grows exponentially.

In the case of Dropbox, the username and password act as the first layer. Experts agree that a simple authentication layer will provide enough protection for nonsensitive data. However when attempting to protect sensitive data we must pair authorization with encryption.

Generally speaking file systems have maintained a sense of insecurity, which makes them useful. Not encrypting files on Dropbox is akin to not encrypting files on a shared PC. Sensitive data should always be encrypted regardless of its location or media. We should treat sensitive data-at-rest on Dropbox the same way we treat sensitive data on local, optical or flash disk. We should encrypt it!

So how does a user encrypt their Dropbox?

My strongly opinionated solution uses TrueCrypt to create an encrypted volume in the Dropbox directory. Simply treat the Dropbox like a normal directory, follow the TrueCrypt documentation to build a volume, and give Dropbox a chance to sync the data. When the sync completes, the TrueCrypt volume will be mountable on each of your Dropbox enabled computers.

I have to admit at first I was skeptical, but the software cooperates surprisingly well and after the initial sync proceeding syncs occur quickly! I prefer TrueCrypt because it is open source, cross platform, and free (both in freedom and cost). TrueCrypt also functions and performs better then any other solution including commercial products like GuardianEdge or PGP both recently acquired by Symantec.

All security and encryption software should remain open sourced and peer reviewed to prevent harmful tampering. Commercial software, written in a black-box vacuum, prevents customers from viewing its code and procedures. We cannot trust software for security when we cannot view its source code.

You should follow me on twitter here

Voice Over IP with TeamSpeak

2011-04-03T12:11:00-04:00

This article will cover running a Voice Over IP service like TeamSpeak on a VPS.

Voice Over IP allows users to communicate using audio over the Internet.

When planning for this article I originally was going to cover ventrilo, but their download link was obfuscated behind a heinous php session script. Ventrilo also does not have a Linux client although they have been promising one for quite some time.

Instead we will cover how to install and configure TeamSpeak.

Installing a teamspeak server on your VPS

Create a user to run teamspeak.

sudo adduser teamspeak

follow the prompts

Personally I don't want the teamspeak user to have ssh access so I added the following to /etc/ssh/sshd_config:

DenyUsers teamspeak

Then reload ssh server config:

sudo service ssh reload

Setup the installation environment.

The following 4 commands will create a directory to hold your installation, change the ownership of the directory to the teamspeak user, change the working directory to the new folder and then become the teamspeak user:

sudo mkdir /opt/teamspeak
sudo chown teamspeak:teamspeak /opt/teamspeak
cd /opt/teamspeak
sudo su teamspeak

Download and extract TeamSpeak server software.

Find the proper package for your VPS and download it, in my case I ran:

wget https://teamspeak.gameserver.gamed.de/ts3/releases/beta-30/teamspeak3-server_linux-amd64-3.0.0-beta30.tar.gz

For best results download the latest version of teamspeak.

A teamspeak tarball should now exist in your present working directory. We can extract the files from the tarball by issuing the following command:

tar xvf teamspeak3-server_linux-amd64-3.0.0-beta30.tar.gz --strip-components=1

No you don't have to type that file name out! The bash shell has tab completion, type 'tar xvf teamsp' and then press tab. : )

Install the TeamSpeak server software.

I had success running:

./ts3server_startscript.sh start

Write down the auto generated serveradmin password.

Configure TeamSpeak to start at system bootup.

Create a cronjob under the teamspeak user:

crontab -e

Place the following into teamspeak's crontab:

@reboot /opt/teamspeak/ts3server_startscript.sh start

Porting the ChaosTheory Wordpress theme to Pylowiki

2011-03-26T22:00:00-04:00

As you may have noticed, I recently switched this blog over to the ChaosTheory Wordpress theme.

I had to tweak some of my pre-existing articles to make them look 'right' but upon completion I was very pleased with the aesthetics of each page.

Later on I reviewed the theme of foxhop.net, my pylowiki demo site. Pylowiki felt bland in comparison to this wordpress blog.

So, I made up my mind to build themes for Pylowik, and I started by porting ChaosTheory. You can view the finished product at https://www.foxhop.net.

You should also give Pylowiki a try. Pylowiki has futuristic, live preview, same page, section edit functionality.

virt-back: a python libvirt backup utility for kvm xen virtualbox

2011-03-20T11:32:00-04:00

Over the weekend I wrote virt-back, a backup utility for QEMU, KVM, XEN, or Virtualbox guests.

virt-back is a python application that uses the libvirt API to safely shutdown, gzip, and restart guests.

The backup process logs to syslog for auditing and virt-back works great with cron for scheduling outages. Virt-back is in active development so feel free to give suggestions or branch the source.

virt-back has been placed in the public domain and the latest version may be downloaded here:

Installation

The fastest way to install virt-back is to use pip or setuptools.

Try sudo pip install virt-back or sudo easy_install virt-back

Otherwise you may manually install virt-back

sudo wget https://git.unturf.com/python/virt-back/raw/branch/master/virt-back \
-O  /usr/local/bin/virt-back

sudo chmod 755 /usr/local/bin/virt-back

Test installation

virt-back --help

Example cronjob

15  2  *  *  1  /usr/local/bin/virt-back --quiet --backup sagat
15  23 *  *  5  /usr/local/bin/virt-back --quiet --backup mbison

Manual

russell@host:~$ virt-back --help
Usage: virt-back [options]
Options:
  -h, --help            show this help message and exit
  -q, --quiet           prevent output to stdout
  -d, --date            append date to tar filename [default: no date]
  -g, --no-gzip         do not gzip or tar the resulting files
  -a amount, --retention=amount
                        backups to retain [default: 3]
  -p 'PATH', --path='PATH'
                        backup path [default: '/KVMBACK']
  -u 'URI', --uri='URI'
                        optional hypervisor uri

  Actions for info testing:
    These options display info or test a list of guests.

    -i, --info          info/test a list of guests (space delimited dom names)
    --info-all          attempt to show info on ALL guests

  Actions for a list of dom names:
    WARNING:  These options WILL bring down guests!

    -b, --backup        backup a list of guests (space delimited dom names)
    -r, --reboot        reboot a list of guests (space delimited dom names)
    -s, --shutdown      shutdown a list of guests (space delimited dom names)
    -c, --create        start a list of guests (space delimited dom names)

  Actions for all doms:
    WARNING:  These options WILL bring down ALL guests!

    --backup-all        attempt to shutdown, backup, and start ALL guests
    --reboot-all        attempt to shutdown and then start ALL guests
    --shutdown-all      attempt to shutdown ALL guests
    --create-all        attempt to start ALL guests

Restoring

virt-back: restoring from backups

Contents

Installation
Test installation
Example cronjob
Manual
Restoring

Response to L-Theanine: a 4000 Year Old Mind-Hack

2011-01-11T17:06:00-05:00

RE: https://worldoftea.org/caffeine-and-l-theanine

As the original article speculates, the combination and amount of L-Theanine and caffeine present in tea, appears to have notable affects on my programming and problem solving abilities. It's difficult to show conclusive evidence to this claim, but I generally feel most alert and organized when coding under the influence of 2 - 3 cups. In a typical day I drink about 4 cups, dosed an hour apart. I feel compelled to introduce another "mind hack" for problem solving or programming, which I call "Dream coding"

"Dream coding" is just that, coding at night during sleep. One can successfully invoke this "mind hack" by provoking thought about the problem while drifting to sleep. Once sleeping I'm able to see my code and my brain seems to iteratively problem solve. Most often I'm lucid during these events; aware of the problem I'm trying to solve and of the possible solutions. Other times I don't recall performing the solutions, but when I wake and after my first cup of tea, I'm able to solve my problem with a optimal and beautiful solution.

PBS NOVA published an episode about this phenomenon: What are dreams?

Does this happen to you? Have you experienced this?

You should follow me on twitter here.

I just used Google to purchase Chinese food takeout for two

2011-01-10T20:03:00-05:00

Yes, it has been done. Using my Google Chromium browser I searched Google Maps for the closest local Chinese restaurant. The small town venue doesn't have a website or menu posted online.

So I used Google Web to search for the restaurant name. I found takeouttonight.com (no longer works) which had the entire menu and VALID prices for the restaurant hosted online! I choose 'Sweet and Sour chicken', my wife selected Chicken and Broccoli.

I then logged into my Google Gmail account and clicked Call phone. Using the mouse I dialed the number I previously researched and after 3 rings an Asian speaking fellow answered the phone.

I said, "Hello, I'd like to place a delivery order." His reply, "Ok." resonated out of my 5.1 speaker setup.

I read off our selections and was super excited that it was WORKING! In my titillated state, I ended up ordering an extra side of egg rolls just to keep the conversation going.

He asked for my phone number to complete the order. I obliged by reading my Google phone number.

Google might be spamy these days, but it's still useful.

Who would have thought a lazy hacker could combine six different Google services to order Chinese food without leaving the desk?

I know it's possible now, I've seen me do it.

You should follow me on twitter here.

Graphic Design Powerhouse: Wacom Bamboo, Ubuntu, and MyPaint

2011-01-09T00:14:00-05:00

For Christmas this year my wife gifted me a Wacom Bamboo pen tablet. She claimed to have researched many products and looked for a model that worked well with Ubuntu and had positive user reviews. She ultimately chose the Wacom Bamboo model and I was so excited when I unwrapped it!

The driver install was flawless and I was up and running after about 5 mins and the first application I attempted to use was Gimp. Gimp worked but the flow felt "clunky".

I wanted an experience that better emulated a pencil or brush and canvas. I went to bed that night feeling a little unsatisfied.

The next day I decided to give the tablet another try, but instead of using Gimp I decided to try my luck with other software called MyPaint. MyPaint's interface feels perfect and it is loaded with brushes, pencils, and inks. MyPaint implements brush stroke pressure with unmatched precision!

The art on this page was created with my wacom BAMBOO tablet and MyPaint.

You should follow me on twitter here.

Thank you so much MyPaint developers for such a stellar piece of software!

MyPaint is a fast and easy open-source graphics application for digital painters. It lets you focus on the art instead of the program. You work on your canvas with minimum distractions, bringing up the interface only when you need it.

How did Stack Overflow get initial traction?

2011-01-05T02:57:00-05:00

Stack Overflow was a progressive and natural evolution of the standard clunky forum.

Using ajax it created a more fun and clean user experience.

Using badges and karma to gain responsibility allow forums post to become a game. People naturally like to see progression and growth, being able to watch my karma go up was a rush, similar to a video game. (some people plan elaborate flights to rack up frequent flyer miles, planning a trip properly could cost as little as 50 dollars to fly around the world in 24 hours, and gaining thousands of miles using point multipliers)

Stack Overflow was search-able! Here is a thought, create a forum that is easily index by search engines... One question per page, on topic with the most relevant posts at the top of the page. Who cares about user bounce backs, if they see your URL enough (StackOverflow.com) they will end up creating an account and becoming a user who creates even more quality content!

The site was geared toward geeks, and programmers! They adopt new tech the quickest.

Did I miss any reasons? Feel free to comment below.

You should follow me on twitter here

A homegrown python bread crumb module

2011-01-02T14:14:00-05:00

I have placed bread.py, a python breadcrumb module, into the public domain.

The bread object accepts a url string and grants access to the url crumbs (parts) or url links (list of hrefs to each crumb).

Tutorial: Add a Breadcrumb Subscriber to a Pyramid project using 4 simple steps.

Demo: https://school.yohdah.com/

You should follow me on twitter here

four2go!: A new spin on an old classic

2010-12-31T20:33:00-05:00

For the past two months I have feverishly worked on my side project. Initially I set out to work on this application for submission to Hacker News, lets make November "Launch an App Month".

The whole project took longer than expected but I was please with my progress so I continued development. Today, a month later, I have added the finishing touches to the app. It gives me great pleasure to announce the official launch of four2go!

A new spin ...

four2go! brings a new spin to the classic "four in a row" genre.

Instead of weighted tokens, four2go! uses balloons which float up rather than down. This new game mechanic requires players to retrain their brains and can easily throw off a novice player.

Another new spin, rather than playing on a coffee table, users play using a computer over the internet. The four2go! web application requires a simple account registration but does not require any download or installation!

Users can play with anyone, at anytime, anywhere in the world. The game sessions are persistent meaning the board will not disappear. This persistence allows players to check their games much like they would check their email.

The gumyum framework

The game itself was completed after the first weekend of coding. Programming the game was fun and exciting and originally four2go! was a command prompt application with a text-based gui. That version was not accessible and in order for people to play we needed to move to a different medium. So I set out to code a web front end.

After another weekend of coding I had the game working in the browser. It was very clunky (It didn't refresh after player moves) and didn't have any authentication, anyone could play any game.

Coding the game was simple, the framework was the difficult part...

Enter Stage Left: The gumyum framework. I needed a way to authentication users and a way to store and retrieve game sessions. I also needed a way to track user statistics and player history. Most importantly I needed a user interface that would be intuitive and fun to use! The remainder of the project set out to solve each of those needs. I needed the gumyum framework to make four2go! popular. I spent the rest of the time (1.5 months) working on gumyum and I feel safe claiming that the framework appears complete.

The great part about having a framework is the code reusability, I should be able to "plug" new games or new mechanics behind the gumyum framework with little trouble. My labor and efforts should pay off if I ever decide to build another game.

Artwork and graphic design

I was very fortunate to work with my brother Joey Ballestrini on some of the concept and game art. We designed the four2go! logo, the gumyum logo, and the balloons. Joey designed each of the "create" game icons. The clouds, the grass, and the dirt were created for another game and I decided they were a good fit so they were re-purposed. We both use gimp when creating graphical computer art.

LETS PLAY!

I have attached a video of the game, but I urge you to try four2go! out for yourself!

The Barrymores stole my heart then crushed it

2010-11-30T04:37:00-05:00

Recently while surfing pandora I stumbled upon a new favorite song, "Think Straight Again", composed by a ska band named The Barrymores. I embraced the bands alluring melodies after the first play.

What caught my attention?

Well the whole band rocks, hard. The main components of The Barrymores shape a full and energetic sound. They have a drummer, a bassist, a lead guitarist and two horn players (trumpet and trombone). Oh, I almost forgot to mention they contrast most ska bands by featuring a kickass lead female singer.

What compelled me to write about The Barrymores?

Though The Barrymores are a small town band, their technical abilities and prowess mimic more a band of a major record label. Their rocking upbeat style is similar to other ska bands. Each track creates a euphoric sensation and even the "sad" songs feel uplifting. When The Barrymores are on my stereo I typically end up chanting the lyrics while skanking around my office. I feel they bring the ska punk rock genre into another level of excellence.

And then I found the depressing news ...

I started researching The Barrymores because I was fascinated with their music. This research inevitably lead to some depressing news, The Barrymores are no longer together!

"I Am Jack's Broken Heart" -Fight Club.

Not often do I get excited about new music. In moments I grew attached to the group only to get crushed. I decided to investigate to uncover more.

Apparently The Barrymores went into hiatus after one of the founding members passed away. The Barrymore myspace page still seems active and continues to hosts a few of the bands tracks.

Sorry about the bummer, but you should still give the music a listen.

Russell Ballestrini

build server postmortem: disk full, CI dead

timeline

root causes

1. gitlab-runner concurrent = 64

2. LXD images never expired

3. orphaned LXD containers

4. zombie processes holding deleted files

5. no disk monitoring

cleanup cron

salt states (permanent fixes)

disk recovery

lessons

two doors shut, one remains open, an olive branch

The two sealed doors

The open door

Why leave one door open?

Links

royal we & our & one & you

A Love Letter to JSON Jason

Free Agent Looking for Work & JSONResume

Why JSONResume?

Brown Out Reboots Servers: NVIDIA Drivers Vanish, Ethernet Dies – Sneaker-Net Rescue with a Bare USB Stick

Realtek RTL8125 NIC on the Second Server

Bonus: Mouse Drivers Too

who owns yaml?

Growing a 454-page ML reference manual in 5 days: permacomputer harvest

pelican theme upgrade: right sidebar table of contents

webwords code golf: minimal HTTP servers in multiple languages

unpotato webwords: cleaning up btrfs docker volumes after build-all

webwords reaches 42 languages: the ultimate programming kata

Analyzing the Godot Engine Codebase: Millions of Lines of Code

Building and Modding MultiMower with Dry Engine

Introducing SLOP

Free LLM Endpoints & Dynamic Distributed Model Inference Client with UncloseAI.js

Free Hermes Machine Learning ai.unturf.com & uncloseai.js

Efficient Battleship Board Generation and Data Management with Python

Comparing Elixir and Phoenix Performance with a Community-Driven OpenAI Client

Comparing Node.js and Python Performance with the Official OpenAI Client