Text manipulation idioms in linux

awk: select columns
sed: stream editor (operations like select, substitute, add/delete lines, modify)
sed expressions can be separated by ";"
sed can substitute all occurrences with 'g' modified at the end: 's/(find)/(replace)/g'

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement

# arg I/O
$@: unpack all input args
$*: join all inputs as ONE arg, separated by FIRST character of IFS (empty space if unspecified)

# Remember the double quotes around "$*" or "$array[*]" usages or else IFS won't function

array[@]: entire array
${array[@]}: unpacks entire array into MULTIPLE arguments
${array[*]}: join entire array into ONE argument separated by FIRST character of IFS (defaults to an empty space if unspecified)
( IFS=$'\n'; echo "${my_array[*]}" )

${#str}: length of string
${#array[@]}: length of array
${#array[@]:start:after_stop}: select array[start] ... array[after_stop-1]

${str:="my_string"}: initializes variable str with "my_string" (useful for side-effect)

$(str##my_pattern}: delete front matching my_pattern
${str%%my_pattern}: deletes tail matching my_pattern (can use one % instead)
$(str%?}: delete last character (the my_pattern is a single character wildcard "?")

$( whatever_command ): captures stdout created by running whatever_command
( $str ): tokenize to string array, governed by IFS (specify delimiter)
( $( whatever_command ) ): combines the two operations above: capture stdout from command and tokenize the results

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement
function strjoin { local IFS="$1"; shift; echo "$*"; }

 7 total views

Improved code for Toner Reset SP C250SF/DN

This is based on the Raspberry Pi implementation of the Toner chip reset:


I am using a Raspberry Pi Zero W so the chip is BCM2835 instead and I can use 100Kbps/400KBps instead of 9600 baud as in the original code

The electrical pins we need is clustered on to top left, Pins 1 (3.3V), 3 (I2C SDATA), 5 (I2C SCLK), 9 (Ground)

Raspberry Pi Zero GPIO Pinout, Specifications and Programming language

While looking for the pinouts (https://pinout.xyz/pinout/i2c), I discovered a useful tool called i2cdetect that allows me to find out the address of the chips which means I can write a program automatically figure out the right image to load to the chip without looking:

sudo apt-get install i2c-tools
sudo i2cdetect -y 1
Sorry I forgot where I got this image from.
Please remind me in the comments section if you find out who should I credit it to.

Since I don’t have cheap pogo pins lying around, I took the 2.4mm pitch (the standard size used in PC, Arduino and Raspberry Pi) jumper block I have (so all pins are set at equal lengths to make simultaneous contact) and hope somehow there’s 4 pins that kind of align with the contact, and it did. See pictures here:

Can press the pins down by using jumpers

You might be worried about shorting into the next pin or hooking something up in reverse damaging the chips, but luckily the chips survived. My guess is that it’s a good design to put the Vcc next to Ground on one side instead of making it symmetric so the polarity can be reversed. When reversed, SCL is hooked forced to Ground, SDA is pulled up to Vcc while there is no power supply, so no damage is done. Brilliant! The worst case for my poorly aligned jumper block is that SDA and Vcc might touch each other, but it doesn’t matter because it’s a perfectly legal hookup (just not communicating)!

So no worries if you didn’t touch the pins right! The only case it might go wrong is if you intentionally flip the block and slide it by two pins (reversing Vcc and Ground). Other cases are pretty much data lines getting hooked high or low levels while power lines not getting any supplies.

I’ve designed the program that it’ll detect the chip if you hook it up right and immediately program the chip (takes only a second), so you don’t have to hold the jumper for too long to worry about unstable contacts.


# This program detects rewrite the toner chips to "full" for a Ricoh SP C250SF/DN Printer using Raspberry PI (defaults to BCM2835 models such as Raspberry PI Zero W)

# The chip data is in file named "black" "cyan" "magenta" and "yellow". 
# The pad closest to the edge is GND (-> Pin 9), followed by VCC (-> Pin 1) , DATA (-> Pin 3), and Clock (-> Pin 5).

# Be sure i2c is enabled and installed (it's turned off by default) on Raspbian

# This line is disabled because it takes too long to unregister i2c_bcm2835 to start from a clean slate
# modprobe -r i2c_bcm2835 

# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )
# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )
# Keep the 0x5* address lines since only 0x50~0x53 is valid. Strip the 50: header, discard all "--" entries, and you are left with the detected address

# LED flash function
function flash_once {

  echo 0 > ${target_device}
  sleep $period

  echo 1 > ${target_device}
  sleep $period

function flash {
  for((i=1; i<=times; i++)); do
    flash_once $period

if [ -v COLORS[I2C_address] ]; then
  # Meat
  echo "Detected toner chip for color: $color"

  echo "Short flashes before starting. Long flash after done"
  flash 5 0.1

   # "address" counter sync up with the hex code index in file
   printf "Writing"   
   for i in $(cat ${color}); do
     i2cset -y 1 ${HEX_I2C_address} $address $i;
     address=$(($address +1));
     printf .
   echo "Done!"
  flash 3 0.5
  echo "Invalid I2C address for SP C250DN/SF toner chips: ${I2C_address}"

I chose to flash the board’s only LED light quickly before starting and blink slowly a few times after it’s done for visual clues. It’s entirely optional. Here’s the guts of the code without the fancy indicators:


# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )

# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )

if [ -v COLORS[I2C_address] ]; then
  # Meat

  # "address" counter sync up with the hex code index in file
  for i in $(cat ${color}); do
    i2cset -y 1 ${HEX_I2C_address} $address $i;
    address=$(($address +1));
  echo "Invalid I2C address for SP C250DN/SF toner chips: ${I2C_address}"

Download the package. Run program_toner

Just in case if people are wondering. The L01 chip’s datasheet is here:

 12 total views

Auto mount USB drives

Raspbian OS (Raspberry Pi) do not mount USB drives automatically out of the box.

I’m pretty annoyed by the lack of easy to use packages by 2021 and I still have to do it myself with the instructions here: https://github.com/avanc/mopidy-vintage/wiki/Automount-USB-sticks

These are cookbook instructions, but I’ll add some insights to what each component means so it’s easier to remember the steps.

At top level, to auto-detect and mount USB drives, we need the following components

  • udev: analogous to what happens behind device manager, it keeps track of and updates devices as they are connected and disconnected immediately. Need to register USB sticks by adding a event handler, which triggers a systemd service (see below)
  • systemd: analogous to Windows’ services. Need to register the service by stating what commands it will call on start (mostly mounting) and cleanup (mostly unmounting)
  • automount script: it’s a user defined script that abstracts most of the hard work detecting the partitions on the USB stick, assign the mount point names, and mount them
# Register udev event handler (rules)
# /etc/udev/rules.d/usbstick.rules

ACTION=="add", KERNEL=="sd[a-z][0-9]", TAG+="systemd", ENV{SYSTEMD_WANTS}="usbstick-handler@%k"

# It triggers a systemd call to "usbstick-handler@" service registered under /etc/systemd/system/
# Register systemd service
# /etc/systemd/system/usbstick-handler@.service
# (Note: instructions used /lib instead of /etc. It's better to add it as /etc as this is manually registered as user-defined service rather than from a package)

Description=Mount USB sticks

ExecStart=/usr/local/bin/automount %I
ExecStop=/usr/bin/pumount /dev/%I

# %I is the USB stick's device name under /dev, usually sda
# Abstracted the logic of determining the mount point name and mounting to 'automount' (see below)

Create the file /usr/local/bin/automount and give it execution permission: chmod +x /usr/local/bin/automount


# $1 (first argument) is usually "sda" (supposedly USB stick device name) seen from %I in the systemd service commands

# Within the "sda" (USB stick device of interest), extract the partition labels (if applicable) from lsblk command. The first column (name) is dropped
FS_LABEL=`lsblk -o name,label | grep ${PART} | awk '{print $2}'`

# Decide the mount point name {partition label}_{partition name}
# e.g. MS-DOS_sda1
MOUNT_LABEL=$(IFS='_'; echo "${tokens[*]}")
# Using string array makes it easier to drop the prefix if there's no {partition label}
# Bash use IFS to specify separators for listing all elements of the array

# Suggestion: drop --sync for faster USB access (if you can umount properly)
/usr/bin/pmount --umask 000 --noatime -w --sync /dev/${PART} /media/${MOUNT_LABEL}

This automount script is adapted from https://raspberrypi.stackexchange.com/questions/66169/auto-mount-usb-stick-on-plug-in-without-uuid with my improvements.

 13 total views