Text manipulation idioms in linux

awk: select columns
sed: stream editor (operations like select, substitute, add/delete lines, modify)
sed expressions can be separated by ";"
sed can substitute all occurrences with 'g' modified at the end: 's/(find)/(replace)/g'

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement

# arg I/O
$@: unpack all input args
$*: join all inputs as ONE arg, separated by FIRST character of IFS (empty space if unspecified)

# Remember the double quotes around "$*" or "$array[*]" usages or else IFS won't function

array[@]: entire array
${array[@]}: unpacks entire array into MULTIPLE arguments
${array[*]}: join entire array into ONE argument separated by FIRST character of IFS (defaults to an empty space if unspecified)
( IFS=$'\n'; echo "${my_array[*]}" )

${#str}: length of string
${#array[@]}: length of array
${#array[@]:start:after_stop}: select array[start] ... array[after_stop-1]

${str:="my_string"}: initializes variable str with "my_string" (useful for side-effect)

$(str##my_pattern}: delete front matching my_pattern
${str%%my_pattern}: deletes tail matching my_pattern (can use one % instead)
$(str%?}: delete last character (the my_pattern is a single character wildcard "?")

$( whatever_command ): captures stdout created by running whatever_command
( $str ): tokenize to string array, governed by IFS (specify delimiter)
( $( whatever_command ) ): combines the two operations above: capture stdout from command and tokenize the results

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement
function strjoin { local IFS="$1"; shift; echo "$*"; }

Loading

Auto mount USB drives

Raspbian OS (Raspberry Pi) do not mount USB drives automatically out of the box.

I’m pretty annoyed by the lack of easy to use packages by 2021 and I still have to do it myself with the instructions here: https://github.com/avanc/mopidy-vintage/wiki/Automount-USB-sticks

These are cookbook instructions, but I’ll add some insights to what each component means so it’s easier to remember the steps.

At top level, to auto-detect and mount USB drives, we need the following components

  • udev: analogous to what happens behind device manager, it keeps track of and updates devices as they are connected and disconnected immediately. Need to register USB sticks by adding a event handler, which triggers a systemd service (see below)
  • systemd: analogous to Windows’ services. Need to register the service by stating what commands it will call on start (mostly mounting) and cleanup (mostly unmounting)
  • automount script: it’s a user defined script that abstracts most of the hard work detecting the partitions on the USB stick, assign the mount point names, and mount them
# Register udev event handler (rules)
# /etc/udev/rules.d/usbstick.rules

ACTION=="add", KERNEL=="sd[a-z][0-9]", TAG+="systemd", ENV{SYSTEMD_WANTS}="usbstick-handler@%k"

# It triggers a systemd call to "usbstick-handler@" service registered under /etc/systemd/system/
# Register systemd service
# /etc/systemd/system/usbstick-handler@.service
# (Note: instructions used /lib instead of /etc. It's better to add it as /etc as this is manually registered as user-defined service rather than from a package)

[Unit]
Description=Mount USB sticks
BindsTo=dev-%i.device
After=dev-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/automount %I
ExecStop=/usr/bin/pumount /dev/%I

# %I is the USB stick's device name under /dev, usually sda
# Abstracted the logic of determining the mount point name and mounting to 'automount' (see below)

Create the file /usr/local/bin/automount and give it execution permission: chmod +x /usr/local/bin/automount

#!/bin/bash

# $1 (first argument) is usually "sda" (supposedly USB stick device name) seen from %I in the systemd service commands
PART=$1

# Within the "sda" (USB stick device of interest), extract the partition labels (if applicable) from lsblk command. The first column (name) is dropped
FS_LABEL=`lsblk -o name,label | grep ${PART} | awk '{print $2}'`

# Decide the mount point name {partition label}_{partition name}
# e.g. MS-DOS_sda1
tokens=($FS_LABEL)
tokens+=($PART)
MOUNT_LABEL=$(IFS='_'; echo "${tokens[*]}")
# Using string array makes it easier to drop the prefix if there's no {partition label}
# Bash use IFS to specify separators for listing all elements of the array

# Suggestion: drop --sync for faster USB access (if you can umount properly)
/usr/bin/pmount --umask 000 --noatime -w --sync /dev/${PART} /media/${MOUNT_LABEL}

This automount script is adapted from https://raspberrypi.stackexchange.com/questions/66169/auto-mount-usb-stick-on-plug-in-without-uuid with my improvements.

Loading

Get myself comfortable with Raspberry Pi

I2C is disabled by default. Use raspi-config to enable it. Editing config file /boot/config.txt directly might not work

Locale & Keyboard (105 keys) defaults to UK out of the box. Shift+3 “#” (hash) sign became “£” pound sign. Use raspi-config to change the keyboard.

It reads random garbage partitions for MFT assigned to FAT16 drives. Just use FAT32

USB drives does not automount by default. usbmount is messy as it creates dummy /media/usb[0-7] folders. Do this instead.

Loading

Flashing router firmware through Serial Port: CFE bootloader (usually Broadcom) based routers

Here’s a summary of learnings from dd-wrt’s serial recovery instructions:

  1. Use a UART controller that signals at 3.3V (e.g. FTDI TTL-232R-3V3) to talk to the board. Regular serial RS-232 ports requires a voltage level shifter converting the signal to swing between 0V to 3.3V.
  2. You only need 3 pins: Tx, Rx and Ground. There’s voltage contention if you plug in the Vcc from TTL-232R-3V3 (It’s the USB’s 5V despite the signaling is 3.3V) to the 3.3V supply of the router. You don’t need the Vcc pin. It didn’t harm anything or do anything when I connected the Vcc pin.
  3. Stick with all serial port defaults and only set the baud to 115200 (default is 9600) and turn off flow control (default is xon/xoff). I use Putty for terminal.
  4. The terminal serves as the monitor for the computer on the router that shows a text console. Broadcom uses CFE bootloader (others use U-Boot with busybox).
  5. CFE bootloader defaults to 192.168.1.1 with subnet mask 255.255.255.0 (aka /24). Set up the network interface to have a static IP on the same subnet to talk to the board.
  6. Good habit: nvram erase
  7. The flash program relies on TFTP protocol to receive the firmware file. So get your TFTP client ready. Microsoft included a TFTP client/server since Windows 7 but usually disabled (turn it on in Windows OptionalFeatures.exe).
  8. TFTP is a simple push(put)/pull(get) design. You can either “push a file on your computer” or “get a file as filename”. You’d want to specify -i switch (binary image transfer) with Windows tftp.exe.
  9. So type this command at the command prompt but don not press enter until your router is ready to grab the file: tftp -i 192.168.1.1 put {path to whatever TRX firmware file}
  10. Go back to the serial terminal and tell the router to accept a TFTP push (in a window of a few seconds before it time out) and flash the memory region flash1.trx with this command: flash -ctheader : flash1.trx
  11. Immediately initiate the TFTP push from your computer (Windows command line example in Step #9 above)
  12. Wait for a couple of hours! The terminal might tell you that it has received the file completely, but it won’t show anything when it’s writing to the flash! It’s a painfully slow process with no feedback. Just be patient!

Some observations

  • FreshTomato firmware absolutely won’t tell you on the screen after it has done flashing (Merlin-WRT does). Just turn the router back on after a couple of hours.
  • Merlin firmware repeats (exposes) the raw passwords to the serial port!
  • FreshTomato firmware boots to a linux prompt on the serial port

Loading