IPXE and Coreboot Ramblings

This is more a collection of notes compiled from my command history. It hasn’t been repeated…

I have a motherboard that is Coreboot compatible. Coreboot comes with a basic payload that permits booting from local disks, but requires other payloads to be added if required – such as IPXE.

Need some info first, such as the NIC PCI ID

root@blackbird:~# lspci  | grep -i net
00:0a.0 Bridge: NVIDIA Corporation CK804 Ethernet Controller (rev a3)
root@blackbird:~# lspci -s 00:0a.0 -nnn
00:0a.0 Bridge [0680]: NVIDIA Corporation CK804 Ethernet Controller [10de:0057] (rev a3)

Get, configure and build IPXE.

git clone git://git.ipxe.org/ipxe.git
cd ipxe/src/

Insert this into config/local/general.h – it defines what gets included into the ROM image, such as menu support, iscsi support, etc

#undef PXE_STACK               /* PXE stack in iPXE – you want this! */
#undef PXE_MENU                /* PXE menu booting */
#undef DOWNLOAD_PROTO_TFTP     /* Trivial File Transfer Protocol */
#undef SANBOOT_PROTO_ISCSI     /* iSCSI protocol */
#undef SANBOOT_PROTO_AOE       /* AoE protocol */
#undef SANBOOT_PROTO_IB_SRP    /* Infiniband SCSI RDMA protocol */
#undef SANBOOT_PROTO_FCP       /* Fibre Channel protocol */
#undef CRYPTO_80211_WEP        /* WEP encryption (deprecated and insecure!) */
#undef CRYPTO_80211_WPA        /* WPA Personal, authenticating with passphrase */
#undef CRYPTO_80211_WPA2       /* Add support for stronger WPA cryptography */
#undef IMAGE_NBI               /* NBI image support */
#undef IMAGE_ELF               /* ELF image support */
#undef IMAGE_MULTIBOOT         /* MultiBoot image support */
#undef IMAGE_PXE               /* PXE image support */
#define        IMAGE_SCRIPT            /* iPXE script image support */
#define        IMAGE_BZIMAGE           /* Linux bzImage image support */
#undef IMAGE_COMBOOT           /* SYSLINUX COMBOOT image support */
#undef IMAGE_EFI               /* EFI image support */
#undef IMAGE_SDI               /* SDI image support */
#undef NVO_CMD                 /* Non-volatile option storage commands */
#define CONFIG_CMD              /* Option configuration console */
#undef FCMGMT_CMD              /* Fibre Channel management commands */
#undef ROUTE_CMD               /* Routing table management commands */
#define IMAGE_CMD               /* Image management commands */
#define SANBOOT_CMD             /* SAN boot commands */
#define MENU_CMD                /* Menu commands */
#undef LOGIN_CMD               /* Login command */
#undef SYNC_CMD                /* Sync command */
#undef NSLOOKUP_CMD            /* DNS resolving command */
#undef TIME_CMD                /* Time commands */
#undef DIGEST_CMD              /* Image crypto digest commands */
#undef LOTEST_CMD              /* Loopback testing commands */
#undef VLAN_CMD                /* VLAN commands */
#undef PXE_CMD         /* PXE commands */
#undef REBOOT_CMD             /* Reboot command */
#undef IMAGE_TRUST_CMD /* Image trust management commands */

And the command that will be run at startup of ipxe should be inserted into shell.ipxe

#!ipxe

 

dhcp
chain –autofree http://core.vpn.glasgownet.com/menu.ipxe

Take the PCI ID, concatenate it as the ROM filename, and embed the script.

make -j3 bin/10de0057.rom EMBED=./shell.ipxe

Now the IPXE payload is ready.

 

For Coreboot,

git clone https://review.coreboot.org/coreboot
cd coreboot
git submodule update –init –checkout
make nconfig

In nconfig, select the appropriate board.

Save, exit.

Build the cross compilers

make crossgcc CPUS=4

Build the firmware

make

The coreboot firmware is saved as build/coreboot.rom, but needs the payload added.

./build/cbfstool ./build/coreboot.rom add -f ../ipxe/src/bin/10de0057.rom -n pci10de,0057.rom -t raw
./build/cbfstool ./build/coreboot.rom print

To remove the payload, in the event of wanting to add a new version

./build/cbfstool ./build/coreboot.rom remove -n pci10de,0057.rom

The coreboot.rom file is now ready to be copied to the target machine and flashed.

If a backup is required, use

flashrom -p internal -r backup.bin

To flash,

flashrom -p internal -w coreboot.rom

AWS G2 GPU vs unattended-upgrade

Recently we noticed that our AWS G2 GPU instances were no longer working correctly after a reboot. We were being greeted with the joyful message of

[ 6710.061115] NVRM: The NVIDIA GRID K520 GPU installed in this system is
NVRM:  supported through the NVIDIA 367.xx Legacy drivers. Please
NVRM:  visit http://www.nvidia.com/object/unix.html for more
NVRM:  information.  The 375.39 NVIDIA driver will ignore
NVRM:  this GPU.  Continuing probe…

It was evident that at some point in the past, the NVIDIA driver had been upgraded to a version that now no longer supports the GRID K520 GPU card in the machine. Of course the first thought is to blame whoever had root access on the system. Let’s have a look at /var/log/apt/history.log then…

Start-Date: 2017-03-21  08:51:00
Commandline: /usr/bin/unattended-upgrade
Install: libcuda1-375:amd64 (375.39-0ubuntu0.16.04.1, automatic), nvidia-opencl-icd-375:amd64 (375.39-0ubuntu0.16.04.1, automatic), nvidia-375:amd64 (375.39-0ubuntu0.16.04.1, automatic)
Upgrade: libc6-dev:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), libcuda1-367:amd64 (367.57-0ubuntu0.16.04.1, 375.39-0ubuntu0.16.04.1), libc6:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), locales:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), libc-bin:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), libc6-
i386:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), libc-dev-bin:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), multiarch-support:amd64 (2.23-0ubuntu5, 2.23-0ubuntu6), libfreetype6:amd64 (2.6.1-0.1ubuntu2, 2.6.1-0.1ubuntu2.1), nvidia-opencl-icd-367:amd64 (367.57-0ubuntu0.16.04.1, 375.39-0
ubuntu0.16.04.1), nvidia-367:amd64 (367.57-0ubuntu0.16.04.1, 375.39-0ubuntu0.16.04.1)
End-Date: 2017-03-21  08:52:28

There we go, it was the unattended-upgrade feature of Ubuntu that’s upgrading NVIDIA drivers to an unsupported version for AWS G2 GPU machines.

To fix this, since version 367 of NVIDIA is no longer available in the Ubuntu archives, it has to be obtained as a build artifact. It’s not the cleanest way, but it would seem that the quickest way to resolve this is to apt-get remove nvidia-375, and any dependencies, and then install the build artifacts from https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/11078476

Namely,

apt-get remove libcuda1-375 nvidia-opencl-icd-375 nvidia-375 nvidia-cuda-toolkit

wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/11078476/+files/nvidia-367_367.57-0ubuntu0.16.04.1_amd64.deb

wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/11078476/+files/nvidia-opencl-icd-367_367.57-0ubuntu0.16.04.1_amd64.deb

wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/11078476/+files/libcuda1-367_367.57-0ubuntu0.16.04.1_amd64.deb

dpkg -i –auto-deconfigure libcuda1-367_367.57-0ubuntu0.16.04.1_amd64.deb nvidia-opencl-icd-367_367.57-0ubuntu0.16.04.1_amd64.deb nvidia-367_367.57-0ubuntu0.16.04.1_amd64.deb

If there are any lingering versions of a package that depends on nvidia-375, uninstall it, rinse and repeat, and re-install it. It most likely does not depend on -375 explicitly, but a metapackage provided by -375, which we’re providing instead from -365

Once the core -367 packages are installed and happy, check dmesg to make sure the GPU has been discovered, and then reinstall nvidia-cuda-toolkit and any other packages. Assuming all goes well, you can now test your software against the installed package suit.

If things are working as expected, simply mark your now critical packages as held in order to prevent them from being upgraded again

apt-mark hold libcuda1-367 nvidia-367 nvidia-opencl-icd-367

This has been reported to Canonical at https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-375/+bug/1674666

1-wire, Node-Red, Domoticz & Grafana

Recently I posted a shiny graph of my garage temperature after I’d put a car with a hot engine in there. The spikes were fairly pronounced, and it was possibly to see where I’d left the door open whilst I worked on the car, before going for a test drive in the evening.

I was subsequently asked…

tl;dr 1-Wire -> ESP8266 -> MQTT -> Node-Red -> MQTT -> Domoticz -> MQTT -> Node-Red -> InfluxDB -> Grafana

It starts out simply enough, with a string of DS18B20 1-wire sensors hooked up to a WeMos D1 Mini NodeMCU board. On that board, there’s a copy of https://github.com/kylegordon/mqtt_esp8266_ds1820_arduino flashed onto, and it scans the bus periodically, reads the values, and publishes them to an MQTT broker. Each ROM ID (1-wire device) gets its own topic, and a plain number is published to the relevant topic.

My home automation controller of choice is Domoticz , and it likes a particular flavour of JSON being published on the /domoticz/in topic. This is where Node-Red steps in to do a translation.

[{“id”:”aa0d4469.8e7458″,”type”:”mqtt in”,”z”:”d5caab33.2a3558″,”name”:”esp8266 on temp/#”,”topic”:”temp/#”,”qos”:”2″,”broker”:”66a92c76.9956d4″,”x”:117,”y”:349,”wires”:[[“5f4ec201.7bbdac”]]},{“id”:”5f4ec201.7bbdac”,”type”:”function”,”z”:”d5caab33.2a3558″,”name”:”Device to IDX”,”func”:”temp = msg.payload/16;\nrom_id = msg.topic.split(\”/\”)[1];\n\nmsg.payload = {};\nswitch (rom_id) {\n case \”28b8c81d300e5\”:\n msg.payload.idx = 186;\n break;\n case \”28ac871d300e4\”:\n msg.payload.idx = 187;\n break;\n}\nmsg.payload.rom_id = rom_id;\ntemp = temp.toString();\nmsg.payload.svalue = temp;\n\n\nreturn msg;”,”outputs”:1,”noerr”:0,”x”:347,”y”:348,”wires”:[[“71da0e3.d7e8ff”]]},{“id”:”71da0e3.d7e8ff”,”type”:”mqtt out”,”z”:”d5caab33.2a3558″,”name”:””,”topic”:”domoticz/in”,”qos”:””,”retain”:””,”broker”:”66a92c76.9956d4″,”x”:535,”y”:349,”wires”:[]},{“id”:”66a92c76.9956d4″,”type”:”mqtt-broker”,”z”:””,”broker”:”homeauto.vpn.glasgownet.com”,”port”:”1883″,”clientid”:””,”usetls”:false,”compatmode”:true,”keepalive”:”15″,”cleansession”:true,”willTopic”:””,”willQos”:”0″,”willPayload”:””,”birthTopic”:””,”birthQos”:”0″,”birthPayload”:””}]

In short, this flow subscribes to all messages on temp/#, takes the payload and topic, and formulates a JSON message with the correct IDX. The IDX is the unique ID for a virtual device (in this case, a temperature sensor) in Domoticz. The JSON message is then published on /domoticz/in, where it is consumed by Domoticz and used for its own home automation purposes.

Now, every value of every device in Domoticz is also published on /domoticz/out. I use this for a few MQTT to Python services I run, but I also have another Node-Red flow that takes the Domoticz JSON messages and inserts them into InfluxDB. This flow was taken from here and relies on the node-red-contrib-influxdb node.

The rest is plain sailing really – there’s a Grafana install that is configured to use InfluxDB as a datasource. Grafana can extract the data using the IDX that’s mentioned above, and will display it in a nice fashion.

SELECT mean(“svalue1”) FROM “domoticz” WHERE “idx” = ‘171’ AND $timeFilter GROUP BY time($interval) fill(null)

Job done.

The Old Backnet

Spurred on by the Twitter discussion regarding the old Backnet network and Electron Club founding, I went for a rummage through some backups.

Fond memories of the Backnet Assigned Names And Number Authority, BANANA. Whilst I still use 172.24.32.0, I’ve expanded somewhat from the /27 to /24. My home network won’t fit on a /27 these days! Gordon is on 172.24.33.0/24, my parents on 172.24.34.0/24. The legacy lives on a little, and still technically within the allocated BANANA range 🙂

Archive.org to the rescue! https://web.archive.org/web/20050421022318/http://wiki.backnet.org/BANANA/Addresses

Like I say, these are from backups. The file this came from was last modified in 2004!

Current Address Allocations

The majority of these addresses also route to the Backnet network

GlasgowNet currently runs on the IP range 172.24.32.0/21, and we have 172.24.40.0/21 as well for overflow usage.

We also use AS numbers 65088 – 65103 from the private AS numbers range defined by IANA

You can see the current routing table on one of the routers at the looking glass

172.24.32.000/27 – AS65089 – Kyle Gordon
172.24.32.032/27 – AS65096 – Justin Hayes
172.24.32.064/27 – AS65090 – Neil McKillop
172.24.32.096/27 – ICM
172.24.32.128/27 – AS65092 – Colin Petrie
172.24.32.160/28 – Kenny Duffus
172.24.32.176/30 – Andrew Elwell
172.24.32.180/30 – AS65094 – William Anderson
172.24.32.184/30 – AS65091 – Gordon Pearce
172.24.32.188/28 – AS65093 – Stinkpad
172.24.32.204/27 – AS65095 – Matthew Keay

AWS IOT with Mosquitto

Amazon AWS recently released the IOT service, a utility for lightweight devices to create and consume messages on the internet, and also in the case of AWS to leverage the rest of their feature set, such as Kinesis, Lambda, S3, DynamoDB, etc.

Of course, I didn’t fancy using the AWS SDK to do this. I just wanted to get mosquitto_pub and mosquitto_sub working on the command line, so see how easy it would be to get plain old MQTT working with it. It’s not that difficult.

First off, create a working directory, and download the root CA file for your client to use

cd ~
mkdir aws_iot
wget https://www.symantec.com/content/en/us/enterprise/verisign/roots/VeriSign-Class%203-Public-Primary-Certification-Authority-G5.pem -O rootCA.pem

Now that you have the root certificate, head to the AWS IOT console, sign up or sign in, and click on the “Create a Resource”, and then the “Create a thing” button. Give it a name, and optional attributes, and hit “Create”. Once the page loads, you can now select your new “Thing” and aim for the “Connect a Device” button. You can then choose which SDK to use. That’s up to you, but after you select it and hit the “Generate Certificate and Policy” button you will be invited to download three files – public and private keys, and a cert. Do so, save them in your working directory, and also somewhere safe if you plan on deleting the directory later. You can’t download them again.

Now you have to ascertain what your broker endpoint is. The AWS IOT UI isn’t the clearest on this, but it’s helpfully hidden inside the parameter “REST API Endpoint”. You’ll need to select a thing from the console, and it appears at the top right. It’s just the domain part of the REST endpoint, so “https://A2HAT5HHRF2IFT.iot.eu-west-1.amazonaws.com/things/KyleG_Test/shadow” becomes A2HAT5HHRF2IFT.iot.eu-west-1.amazonaws.com

Once you have that information, and have the certificate files in place, it’s a simple case of passing some SSL options to the mosquitto client tools.

mosquitto_pub –cafile rootCA.pem –cert dec39df945-certificate.pem.crt –key dec39df945-private.pem.key -h A2HAT5HHRF2IFT.iot.eu-west-1.amazonaws.com -p 8883 -q 1 -d -t ‘$aws/things/KyleG_Desktop/shadow/update’ -m ‘testing’