Text manipulation idioms in linux

awk: select columns
sed: stream editor (operations like select, substitute, add/delete lines, modify)
sed expressions can be separated by ";"
sed can substitute all occurrences with 'g' modified at the end: 's/(find)/(replace)/g'

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement

# arg I/O
$@: unpack all input args
$*: join all inputs as ONE arg, separated by FIRST character of IFS (empty space if unspecified)

# Remember the double quotes around "$*" or "$array[*]" usages or else IFS won't function

array[@]: entire array
${array[@]}: unpacks entire array into MULTIPLE arguments
${array[*]}: join entire array into ONE argument separated by FIRST character of IFS (defaults to an empty space if unspecified)
( IFS=$'\n'; echo "${my_array[*]}" )

${#str}: length of string
${#array[@]}: length of array
${#array[@]:start:after_stop}: select array[start] ... array[after_stop-1]

${str:="my_string"}: initializes variable str with "my_string" (useful for side-effect)

$(str##my_pattern}: delete front matching my_pattern
${str%%my_pattern}: deletes tail matching my_pattern (can use one % instead)
$(str%?}: delete last character (the my_pattern is a single character wildcard "?")

$( whatever_command ): captures stdout created by running whatever_command
( $str ): tokenize to string array, governed by IFS (specify delimiter)
( $( whatever_command ) ): combines the two operations above: capture stdout from command and tokenize the results

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement
function strjoin { local IFS="$1"; shift; echo "$*"; }

Loading

Improved code for Toner Reset SP C250SF/DN

This is based on the Raspberry Pi implementation of the Toner chip reset:

https://gist.github.com/joeljacobs/c57550cdb4e68e3b86d6b89fb58f305d

I am using a Raspberry Pi Zero W so the chip is BCM2835 instead and I can use 100Kbps/400KBps instead of 9600 baud as in the original code

The electrical pins we need is clustered on to top left, Pins 1 (3.3V), 3 (I2C SDATA), 5 (I2C SCLK), 9 (Ground)

Raspberry Pi Zero GPIO Pinout, Specifications and Programming language

While looking for the pinouts (https://pinout.xyz/pinout/i2c), I discovered a useful tool called i2cdetect that allows me to find out the address of the chips which means I can write a program automatically figure out the right image to load to the chip without looking:

sudo apt-get install i2c-tools
sudo i2cdetect -y 1
Sorry I forgot where I got this image from.
Please remind me in the comments section if you find out who should I credit it to.

Since I don’t have cheap pogo pins lying around, I took the 2.4mm pitch (the standard size used in PC, Arduino and Raspberry Pi) jumper block I have (so all pins are set at equal lengths to make simultaneous contact) and hope somehow there’s 4 pins that kind of align with the contact, and it did. See pictures here:

Can press the pins down by using jumpers

You might be worried about shorting into the next pin or hooking something up in reverse damaging the chips, but luckily the chips survived. My guess is that it’s a good design to put the Vcc next to Ground on one side instead of making it symmetric so the polarity can be reversed. When reversed, SCL is set to low (Ground), SDA is pulled up to Vcc while there is no power supply, so no damage is done. Brilliant! The worst case for my poorly aligned jumper block is that SDA and Vcc might touch each other, but it doesn’t matter because it’s a perfectly legal hookup (just not communicating)!

So no worries if you didn’t touch the pins right! The only case it might go wrong is if you intentionally flip the block and slide it by two pins (reversing Vcc and Ground). Other cases (you are likely going to run into) are pretty much data lines getting hooked high or low levels while power lines not getting any supplies.

I’ve designed the program that it’ll detect the chip if you hook it up right and immediately program the chip (takes only a second), so you don’t have to hold the jumper for too long to worry about unstable contacts.

#!/bin/bash

# This program detects rewrite the toner chips to "full" for a Ricoh SP C250SF/DN Printer using Raspberry PI (defaults to BCM2835 models such as Raspberry PI Zero)

# The chip data is in file named "black" "cyan" "magenta" and "yellow".
# The pad closest to the edge is GND (-> Pin 9), followed by VCC (-> Pin 1) , DATA (-> Pin 3), and Clock (-> Pin 5).

# Be sure i2c is enabled and installed (it's turned off by default) on Raspbian

# This line is disabled because it takes too long to unregister i2c_bcm2835 to start from a clean slate
# modprobe -r i2c_bcm2835

# This program needs sudo permissions as it involves direct access to hardware
if [[ $EUID -ne 0 ]]; then
  echo "Need root permissions as this code involves direct hardware access. Try sudo"
  exit 2
fi

# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Install i2cdetect if not exist (This idiom uses short-circuit OR for conditional exec)
command -v i2cdetect > /dev/null 2>&1 || {sudo apt-get install i2c-tools}

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )
# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )
# Keep the 0x5* address lines since only 0x50~0x53 is valid. Strip the 50: header, discard all "--" entries, and you are left with the detected address
HEX_I2C_address="0x$I2C_address"

# LED flash function(s)
target_device="/sys/class/leds/led0/brightness"
# This accounts for breaking changes (by renaming led0 to ACT) from Bookworm
[ -f ${target_device} ] || target_device="/sys/class/leds/ACT/brightness"

function turn_off {
  echo 0 > ${target_device}
}
function turn_on {
  echo 1 > ${target_device}
}
function toggle {
  # PI Zero only has 1 power LED which is on/off 
  # Reads 0 for OFF and 255 for ON
  readin=$(cat ${target_device})
  # Condition 255 to 1 to reduce platform dependency
  state=$[$readin>0]
  # Not equal to 1 is XOR (flip)
  flip=$[$state != 1]
  # Display output
  echo $flip > ${target_device}
}

function flash_once {
  period=${1:-0.5}

  toggle
  sleep $period

  toggle
  sleep $period
}

function flash {
  times=${1:-1}
  period=$2
  for((i=1; i<=times; i++)); do
    flash_once $period
  done
}

echo "Detecting toner chip ..."
turn_off
if [ -v COLORS[I2C_address] ]; then
  # Meat
  color=${COLORS[I2C_address]}
  echo "Detected toner chip for color: $color"

   # "address" counter sync up with the hex code index in file
   printf "Writing"
   address=0;
   datafile="${BASH_SOURCE%/*}/${color}"
   for i in $(cat $datafile); do
     i2cset -y 1 ${HEX_I2C_address} $address $i; || break
     address=$(($address +1));
     printf .
     flash 1 0
   done
   echo "Done!"
   turn_on
else
  echo "Invalid I2C address for SP C250DN/SF toner chips. ${I2C_address}"
  turn_on
  exit 4
fi

Turns out Linux’s dot sourcing (bash calling) syntax is relative to your current folder, not the scripts’ folder. So if you have one script calling another in the same folder when you are not currently at the same folder as the script (e.g. startup automation), the relative paths will be wrong. So instead of ./ , you should use this prefix instead:

${BASH_SOURCE%/*}/

I chose to flash the board’s only LED light quickly before starting and blink slowly a few times after it’s done for visual clues. It’s entirely optional. Here’s the guts of the code without the fancy indicators:

#!/bin/bash

# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )

# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )
HEX_I2C_address="0x$I2C_address"

if [ -v COLORS[I2C_address] ]; then
  # Meat
  color=${COLORS[I2C_address]}

  # "address" counter sync up with the hex code index in file
  address=0;
  for i in $(cat ${color}); do
    i2cset -y 1 ${HEX_I2C_address} $address $i || break
    address=$(($address +1));
  done
else
  echo "Invalid I2C address for SP C250DN/SF toner chips: ${I2C_address}"
fi

Download the package. Run program_toner

Note: the blog page’

Just in case if people are wondering. The L01 chip’s datasheet is here:


To make a dedicated Raspberry Pi Zero device to run this script (program_toner) repeatedly on boot, I made a script loop_program_toner that runs the same program as infinite loop.

Since writing starts immediately once the chip is identified (no reason to hold the pogo pins for longer than needed), once the chip is detected, whether the writing finished successfully or not, threre’d be 5 seconds of delay so you can reseat the pogo pins (on write failure) or remove the pogo pins (so it won’t unnecessarily rewrite the chip again).

#!/bin/bash

if [[ $EUID -ne 0 ]]; then
  echo "Needs root permissions. Try sudo"
  exit -2
fi

while :
do
  bash "${BASH_SOURCE%/*}/program_toner"
  if (($?)); then
    # Wait longer to give time to detatch probe once chip is detected
    # whether the process completed successfully or not
    sleep 5;
  else
    sleep 1;
  fi
done

There are a few steps needs to auto make this automated:

  1. Have a user account dedicate to autorun the program on boot. In the example here, we’ll use Raspberry Pi’s default admin account
  2. Auto-logged on by running sudo raspi-config, select (1) System Option > (S5) Boot / Auto Login > (B2) Console Autologin
  3. Bless the said account with password free sudo rights: Run sudo visudo and add this to the end (replace admin with something else if you chose a different username)

admin ALL=(ALL) NOPASSWD:ALL
  1. Call loop_program_toner in ./profile or ~/.bashrc (startup script). Can place this at the bottom if your package folder is ~/spc250dn :
sudo ./spc250dn/loop_program_toner

The path can start with . or ~ because you are starting with at home folder anyway.


EDIT (2023-11-1): Linux people lack the backward compatibility decency again! They changed /led0 and /led1 to /ACT and /PWR under /sys/class/leds and they don’t have the decency to put in a symbolic link so old cold won’t break! WTF!

Do they have an idea how many code they’ll break and how many man hours they will waste the world to discover this breaking change when updating the distro? Non-caught exceptions just fall through and shows up as mysterious error messages!

If you want people to get the memo and change their code to match a more sensible notation, feel free to put in a warning stub, but hell no to just flip a switch and watch the world scream! This is apparently not a difficult decision to make when adding backward compatibility would be a huge project because of some catch-22 conditions!

ARM: dts: Standardise on the upstream LED names · raspberrypi/linux@ea14f14 (github.com)

Raspberry Pi Zero only has one LED which is /led0 (/ACT), so ignore /led1 (/PWR).

Loading

Auto mount USB drives

Raspbian OS (Raspberry Pi) do not mount USB drives automatically out of the box.

I’m pretty annoyed by the lack of easy to use packages by 2021 and I still have to do it myself with the instructions here: https://github.com/avanc/mopidy-vintage/wiki/Automount-USB-sticks

These are cookbook instructions, but I’ll add some insights to what each component means so it’s easier to remember the steps.

At top level, to auto-detect and mount USB drives, we need the following components

  • udev: analogous to what happens behind device manager, it keeps track of and updates devices as they are connected and disconnected immediately. Need to register USB sticks by adding a event handler, which triggers a systemd service (see below)
  • systemd: analogous to Windows’ services. Need to register the service by stating what commands it will call on start (mostly mounting) and cleanup (mostly unmounting)
  • automount script: it’s a user defined script that abstracts most of the hard work detecting the partitions on the USB stick, assign the mount point names, and mount them
# Register udev event handler (rules)
# /etc/udev/rules.d/usbstick.rules

ACTION=="add", KERNEL=="sd[a-z][0-9]", TAG+="systemd", ENV{SYSTEMD_WANTS}="usbstick-handler@%k"

# It triggers a systemd call to "usbstick-handler@" service registered under /etc/systemd/system/
# Register systemd service
# /etc/systemd/system/usbstick-handler@.service
# (Note: instructions used /lib instead of /etc. It's better to add it as /etc as this is manually registered as user-defined service rather than from a package)

[Unit]
Description=Mount USB sticks
BindsTo=dev-%i.device
After=dev-%i.device

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/automount %I
ExecStop=/usr/bin/pumount /dev/%I

# %I is the USB stick's device name under /dev, usually sda
# Abstracted the logic of determining the mount point name and mounting to 'automount' (see below)

Create the file /usr/local/bin/automount and give it execution permission: chmod +x /usr/local/bin/automount

#!/bin/bash

# $1 (first argument) is usually "sda" (supposedly USB stick device name) seen from %I in the systemd service commands
PART=$1

# Within the "sda" (USB stick device of interest), extract the partition labels (if applicable) from lsblk command. The first column (name) is dropped
FS_LABEL=`lsblk -o name,label | grep ${PART} | awk '{print $2}'`

# Decide the mount point name {partition label}_{partition name}
# e.g. MS-DOS_sda1
tokens=($FS_LABEL)
tokens+=($PART)
MOUNT_LABEL=$(IFS='_'; echo "${tokens[*]}")
# Using string array makes it easier to drop the prefix if there's no {partition label}
# Bash use IFS to specify separators for listing all elements of the array

# Suggestion: drop --sync for faster USB access (if you can umount properly)
/usr/bin/pmount --umask 000 --noatime -w --sync /dev/${PART} /media/${MOUNT_LABEL}

This automount script is adapted from https://raspberrypi.stackexchange.com/questions/66169/auto-mount-usb-stick-on-plug-in-without-uuid with my improvements.

Loading