Ok, so a month ago I changed employment and the new company uses GitLab exclusively for centralized code version control system.
This is my first time using GitLab and my first projects was integrating a dou/cloudmapper with a GitLab runner on a schedule.
After a couple weeks I've learned how GitLab runners work and wrote a wrapper to run cloudmapper concurrently so that each accounts is collected in parallel (the basic logic is running docker exec on a custom entrypoint and then backgrounds the task in a bash loop over each account configured in cloudmapper's config file)
GitLab runners are awesome.
I've used CircleCI for the better part of 3 years, having lifted the better part of 60 code bases from an internal app called conveyor to CircleCI 1.0 and performing CircleCI 2.0 conversions at Remind.
CircleCI 2.0 is awesome...
But GitLab runners are not black box which makes them so powerful.
Granted not all organizations need or even want this much power, but I'm a nerd and I want it, so I decided I also want to run GitLab at home.
My first thought was to try to just run GitLab on a VM on my SmartOS host called mbison, but that's a bit janky of a setup since it's an T420 ThinkPad after all. Also I learned the recommended minimum memory footprint of GitLab is 4G so a 4 of 16G allocation without even discussing a separate KVM for the runner. It feels weird that a single rails application would need _that_ much memory.
Anyways, I have a FreeNAS FreeNAS-11.3-U5 host at my house with 46G of memory and a bunch of idle cores, So I wondered if they have a plug-in for GitLab... I'm sure they do by now.
After a bit of research I found that iXsystems renamed the FreeNAS codebase to TrueNAS and version TrueNAS-12.0-RELEASE has a GitLab plugin! Sweet!
Thankfully I remembered that back in 2019 I replaced my FreeNAS USB boot device with an SSD so it should be a safe operation to attempt a TrueNAS upgrade on a Saturday morning since I'd have the weekend to try to recover if things went sour. Fun side story, I ran into trouble last year trying to upgrade FreeNAS train because I didn't have room for the upgrade because I was using a USB drive as the boot disk and it had like 5 previous version on the 8G file system.
The kind people in the freenode #FreeNAS channel helped me switch back to an older version using SSH and told me I should not boot from a USB drive which was a thing I picked up from back in the FreeNAS 6 or 7 days. Boy were they correct, I moved the OS to an SSD and the server boots _WAY_ faster now, plus I have more capacity to try new versions.
Anyways, here is a fun command to determine when a ZFS pool was created, I used this to get the exact date and time I moved from USB to SSD for this blog post:
root@guile[~]# zpool history -i freenas-boot | grep "create pool"
2019-11-12.13:00:30 [txg:5] create pool version 28; software version 5000/5; uts 11.2-STABLE 1102500 amd64
So weilding this confidence, I switched FreeNAS trains in the UI and clicked upgrade, following the prompts to create a backup of my NAS config. After the server rebooted, FreeNAS was now TrueNAS. I logged into the web UI to look around and make sure my data was still there.
Everything seemed fine, so I went to the plug-in tab and attempted to install GitLab. That failed with an error.
So I decided oh well, I'll try spinning up an Ubuntu VM on the TrueNAS box, apparently FreeBSD uses bhyve for that.
Attempting to start the VM does nothing in the UI, it appears like it will start but then the page refreshes and the VM is stopped.
I did eventually find the error message, by running
tail -f /var/log/libvirt/*/*
/usr/sbin/bhyve -c cpus=1,sockets=1,cores=1,threads=1 -m 4096 -A -w -H -s 0:0,hostbridge -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -s 3:0,ahci,cd:/mnt/downloads/iso/ubuntu-20.04.1-live-server-amd64.iso -s 30:0,xhci,tablet -s 2:0,ahci -s 5:0,virtio-net,tap1,mac=00:a0:98:4f:98:13 -s 4:0,virtio-blk,/dev/zvol/downloads/gitlab-vcrg2z -s 31,lpc -l com1,/dev/nmdm1A -s 29,fbuf,vncserver,tcp=0.0.0.0:48009,w=1024,h=768 1_gitlab
25/10/2020 17:49:58 Listening for VNC connections on TCP port 48009
25/10/2020 17:49:58 Listening for VNC connections on TCP6 port 48009
ROM boot failed: unrestricted guest capability not available
fbuf frame buffer base: 0x941e00000 [sz 16777216]
The important part is:
ROM boot failed: unrestricted guest capability not available
It seems if you ask bhyve to use the UEFI bootloader it will require UG support in the Intel CPU : (
I attempted to switch to the grub bootloader but in that case I get the following error message:
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/vm.py", line 1414, in do_update
raise verrors
middlewared.service_exception.ValidationErrors: [EINVAL] vm_update.devices.3.dtype: VNC only works with UEFI bootloader.
and the TrueNAS UI does not let you disable VNC on VMs at this point...
Another limitation of a missing UG CPU flag is bhyve may only allocate 1 core, and 1 vcpu to a VM.
Anyways I'm out of luck, I can't use the plugin which uses jails due to a strange error and no other hints as to the issue and byhve / TrueNAS not supporting VMs for my CPU...
UPDATE I recently upgraded my TrueNAS server hardware to an old Dell PowerEdge R720 R720xd with two Intel(R) Xeon(R) CPU E5-2670 @ 2.60GHz each with 8 Physical Cores and 128G of ECC Memory! Booyah!
I was able to flash the included Dell H710 Mini RAID card into IT mode for ZFS support using the awesome guides and tools found here: https://fohdeesha.com/docs/perc/
I now have GitLab running on a byhve instance and also a dedicated runner instance. It's working great and really fun to have this extremely powerful gear in my home lab!