Text manipulation idioms in linux

awk: select columns
sed: stream editor (operations like select, substitute, add/delete lines, modify)
sed expressions can be separated by ";"
sed can substitute all occurrences with 'g' modified at the end: 's/(find)/(replace)/g'

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement

# arg I/O
$@: unpack all input args
$*: join all inputs as ONE arg, separated by FIRST character of IFS (empty space if unspecified)

# Remember the double quotes around "$*" or "$array[*]" usages or else IFS won't function

array[@]: entire array
${array[@]}: unpacks entire array into MULTIPLE arguments
${array[*]}: join entire array into ONE argument separated by FIRST character of IFS (defaults to an empty space if unspecified)
( IFS=$'\n'; echo "${my_array[*]}" )

${#str}: length of string
${#array[@]}: length of array
${#array[@]:start:after_stop}: select array[start] ... array[after_stop-1]

${str:="my_string"}: initializes variable str with "my_string" (useful for side-effect)

$(str##my_pattern}: delete front matching my_pattern
${str%%my_pattern}: deletes tail matching my_pattern (can use one % instead)
$(str%?}: delete last character (the my_pattern is a single character wildcard "?")

$( whatever_command ): captures stdout created by running whatever_command
( $str ): tokenize to string array, governed by IFS (specify delimiter)
( $( whatever_command ) ): combines the two operations above: capture stdout from command and tokenize the results

# https://unix.stackexchange.com/questions/92187/setting-ifs-for-a-single-statement
function strjoin { local IFS="$1"; shift; echo "$*"; }

 12 total views

Improved code for Toner Reset SP C250SF/DN

This is based on the Raspberry Pi implementation of the Toner chip reset:


I am using a Raspberry Pi Zero W so the chip is BCM2835 instead and I can use 100Kbps/400KBps instead of 9600 baud as in the original code

The electrical pins we need is clustered on to top left, Pins 1 (3.3V), 3 (I2C SDATA), 5 (I2C SCLK), 9 (Ground)

Raspberry Pi Zero GPIO Pinout, Specifications and Programming language

While looking for the pinouts (https://pinout.xyz/pinout/i2c), I discovered a useful tool called i2cdetect that allows me to find out the address of the chips which means I can write a program automatically figure out the right image to load to the chip without looking:

sudo apt-get install i2c-tools
sudo i2cdetect -y 1
Sorry I forgot where I got this image from.
Please remind me in the comments section if you find out who should I credit it to.

Since I don’t have cheap pogo pins lying around, I took the 2.4mm pitch (the standard size used in PC, Arduino and Raspberry Pi) jumper block I have (so all pins are set at equal lengths to make simultaneous contact) and hope somehow there’s 4 pins that kind of align with the contact, and it did. See pictures here:

Can press the pins down by using jumpers

You might be worried about shorting into the next pin or hooking something up in reverse damaging the chips, but luckily the chips survived. My guess is that it’s a good design to put the Vcc next to Ground on one side instead of making it symmetric so the polarity can be reversed. When reversed, SCL is set to low (Ground), SDA is pulled up to Vcc while there is no power supply, so no damage is done. Brilliant! The worst case for my poorly aligned jumper block is that SDA and Vcc might touch each other, but it doesn’t matter because it’s a perfectly legal hookup (just not communicating)!

So no worries if you didn’t touch the pins right! The only case it might go wrong is if you intentionally flip the block and slide it by two pins (reversing Vcc and Ground). Other cases (you are likely going to run into) are pretty much data lines getting hooked high or low levels while power lines not getting any supplies.

I’ve designed the program that it’ll detect the chip if you hook it up right and immediately program the chip (takes only a second), so you don’t have to hold the jumper for too long to worry about unstable contacts.


# This program detects rewrite the toner chips to "full" for a Ricoh SP C250SF/DN Printer using Raspberry PI (defaults to BCM2835 models such as Raspberry PI Zero W)

# The chip data is in file named "black" "cyan" "magenta" and "yellow". 
# The pad closest to the edge is GND (-> Pin 9), followed by VCC (-> Pin 1) , DATA (-> Pin 3), and Clock (-> Pin 5).

# Be sure i2c is enabled and installed (it's turned off by default) on Raspbian

# This line is disabled because it takes too long to unregister i2c_bcm2835 to start from a clean slate
# modprobe -r i2c_bcm2835 

# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )
# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )
# Keep the 0x5* address lines since only 0x50~0x53 is valid. Strip the 50: header, discard all "--" entries, and you are left with the detected address

# LED flash function
function flash_once {

  echo 0 > ${target_device}
  sleep $period

  echo 1 > ${target_device}
  sleep $period

function flash {
  for((i=1; i<=times; i++)); do
    flash_once $period

if [ -v COLORS[I2C_address] ]; then
  # Meat
  echo "Detected toner chip for color: $color"

  echo "Short flashes before starting. Long flash after done"
  flash 5 0.1

   # "address" counter sync up with the hex code index in file
   printf "Writing"   
   for i in $(cat ${color}); do
     i2cset -y 1 ${HEX_I2C_address} $address $i;
     address=$(($address +1));
     printf .
   echo "Done!"
  flash 3 0.5
  echo "Invalid I2C address for SP C250DN/SF toner chips: ${I2C_address}"

I chose to flash the board’s only LED light quickly before starting and blink slowly a few times after it’s done for visual clues. It’s entirely optional. Here’s the guts of the code without the fancy indicators:


# Sets the baud rate
modprobe i2c_bcm2835 baudrate=400000

# Create I2C address to color map
COLORS=( [50]="yellow" [51]="magenta" [52]="cyan" [53]="black" )

# Detect chip I2C address
I2C_address=$( sudo i2cdetect -y 1 | grep 50 | sed -e 's/50: //;s/-- //g' )

if [ -v COLORS[I2C_address] ]; then
  # Meat

  # "address" counter sync up with the hex code index in file
  for i in $(cat ${color}); do
    i2cset -y 1 ${HEX_I2C_address} $address $i;
    address=$(($address +1));
  echo "Invalid I2C address for SP C250DN/SF toner chips: ${I2C_address}"

Download the package. Run program_toner

Just in case if people are wondering. The L01 chip’s datasheet is here:

 21 total views

Auto mount USB drives

Raspbian OS (Raspberry Pi) do not mount USB drives automatically out of the box.

I’m pretty annoyed by the lack of easy to use packages by 2021 and I still have to do it myself with the instructions here: https://github.com/avanc/mopidy-vintage/wiki/Automount-USB-sticks

These are cookbook instructions, but I’ll add some insights to what each component means so it’s easier to remember the steps.

At top level, to auto-detect and mount USB drives, we need the following components

  • udev: analogous to what happens behind device manager, it keeps track of and updates devices as they are connected and disconnected immediately. Need to register USB sticks by adding a event handler, which triggers a systemd service (see below)
  • systemd: analogous to Windows’ services. Need to register the service by stating what commands it will call on start (mostly mounting) and cleanup (mostly unmounting)
  • automount script: it’s a user defined script that abstracts most of the hard work detecting the partitions on the USB stick, assign the mount point names, and mount them
# Register udev event handler (rules)
# /etc/udev/rules.d/usbstick.rules

ACTION=="add", KERNEL=="sd[a-z][0-9]", TAG+="systemd", ENV{SYSTEMD_WANTS}="usbstick-handler@%k"

# It triggers a systemd call to "usbstick-handler@" service registered under /etc/systemd/system/
# Register systemd service
# /etc/systemd/system/usbstick-handler@.service
# (Note: instructions used /lib instead of /etc. It's better to add it as /etc as this is manually registered as user-defined service rather than from a package)

Description=Mount USB sticks

ExecStart=/usr/local/bin/automount %I
ExecStop=/usr/bin/pumount /dev/%I

# %I is the USB stick's device name under /dev, usually sda
# Abstracted the logic of determining the mount point name and mounting to 'automount' (see below)

Create the file /usr/local/bin/automount and give it execution permission: chmod +x /usr/local/bin/automount


# $1 (first argument) is usually "sda" (supposedly USB stick device name) seen from %I in the systemd service commands

# Within the "sda" (USB stick device of interest), extract the partition labels (if applicable) from lsblk command. The first column (name) is dropped
FS_LABEL=`lsblk -o name,label | grep ${PART} | awk '{print $2}'`

# Decide the mount point name {partition label}_{partition name}
# e.g. MS-DOS_sda1
MOUNT_LABEL=$(IFS='_'; echo "${tokens[*]}")
# Using string array makes it easier to drop the prefix if there's no {partition label}
# Bash use IFS to specify separators for listing all elements of the array

# Suggestion: drop --sync for faster USB access (if you can umount properly)
/usr/bin/pmount --umask 000 --noatime -w --sync /dev/${PART} /media/${MOUNT_LABEL}

This automount script is adapted from https://raspberrypi.stackexchange.com/questions/66169/auto-mount-usb-stick-on-plug-in-without-uuid with my improvements.

 22 total views

Get myself comfortable with Raspberry Pi

I2C is disabled by default. Use raspi-config to enable it. Editing config file /boot/config.txt directly might not work

Locale & Keyboard (105 keys) defaults to UK out of the box. Shift+3 “#” (hash) sign became “£” pound sign. Use raspi-config to change the keyboard.

It reads random garbage partitions for MFT assigned to FAT16 drives. Just use FAT32

USB drives does not automount by default. usbmount is messy as it creates dummy /media/usb[0-7] folders. Do this instead.

 11 total views

InfiniiMax AutoProbe 1 Caveats

For most mortal souls probing up to 12Ghz, Agilent’s integrated active probe system is called the the AutoProbe 1, which looks like this:

Regular full blown Windows-based Infiniium oscilloscopes takes any AutoProbe 1 probes (as long as the shape fits), but I noticed my DSO6104A (InfiniiVision 6000A series) do not take my 1152A (2.5Ghz) probes nor my fancy 1168A (10Ghz) and 1169A (12Ghz) probes.

Turns out the more compact, embedded (VxWorks) Agilent scopes that boots almost immediately. It’s called the InfiniiVision Series, which covers 1000 X-, 2000 X-, 3000A/T X-, 4000 X-, 6000 X-, 5000, 6000, and 7000 Series.

I’m not rich enough to get my hands on the X series, but I know from the architecture that 5000, 6000 and 7000 series are basically the same scope. 5000 and 6000 series looks almost identical while the 7000 series adds a giant screen and a slightly different keypad layout (the BNC ports do not align with the channel buttons and dials).

Turns out the datasheets shows two caveats:

  • 100Mhz model uses different hardware. They don’t take Autoprobe interface as there’s absolutely no reason why you need an active probe to get 100Mhz single ended. Agilent skipped the hardware for it (thus the autoprobe pins) altogether although they kept the recessed space reserved for Autoprobe so they don’t have to mold a different front bezel just for the 100Mhz models.
  • They basically take only Generation I AutoProbe I, namely the 1130 series
  • Gen 0 (not an official name) AutoProbe 1 does not work: 1152A (2.5Ghz single ended) for 54845A. These differential probes: 1153A, 1154A, 1155A, 1159A are also considered too old. They were intended to work with old Infiniiums such as 54845A
  • Gen 2 AutoProbe 1 (only 10Ghz 1168A/B and 12Ghz 1169A/B models) does not work. These embedded scopes usually max out at 1.5Ghz, with the exception that 6000X goes up to 6Ghz, which is still way below 10Ghz
  • N2800 series are Autoprobe I, but it’s Gen III (has a bigger butt extending away from the AutoProbe I hole), so it doesn’t work
  • The rest are Autoprobe II and III that’s beyond our mortal souls (and way out of the league of InfiniiVision scopes)

 11 total views

Evoluent Vertical Mouse 4 Cable Mod

The mouse cable for Evoluent Vertical Mouse 4 is extremely long, which creates a lot of clutters especially when my keyboard has a USB hub relay built in (it’s the mouse is less than a feet away from it). Instead of splicing the cable, which creates a hard junction that’s not flexible, I modified the mouse to take a micro-USB cable instead.

 10 total views

AC Coupling (removing DC drift) 3-Amp INA Instrumentation Amplifier (DC restoration)

While clearing out my old data, I came across the teaching materials I’ve helped rewritten as a teaching assistant of Biomedical Electronics Lab (Stanford EE122A).

It’s a generic concept in electronics that often used in EKG/ECG circuits to remove the baseline drift on the fly so the analog signal won’t drift off the rails (exceed the dynamic range limited by the op-amps) before the post-processing filter (whether it’s analog or digital) kicks in to remove the DC component.

This concept is called ‘DC restoration’, which is often not taught in standard electronics textbooks. Instead it’s detailed in one of the instrumentation amp (INA) Burr-Brown (now TI) application notes.

It’s a slick trick but the rationale wasn’t very well explained even in the application note itself. It was presented as a feedback design but it doesn’t tell you intuitively what was fed back and why INA chips, and why the reference pin is the right injection point.

Most textbooks don’t even teach the existence of the reference pin (they always short the reference to the ground without explaining). Application notes talk about the REF pin, but they often jump too quickly into cookbook recipes and equations (likely because customers just want quick answers) so they never tell you the thought process.

This blog post shares my intuition of DC-restoration that’s exposed to EE122A students after I’ve updated the lab document. Hope electronic hobbyists and industry people will find it useful.

Before we get to DC restoration, we must see that the purpose of a 3-amp INA is basically a non-inverting buffer stage (primarily done to increase input impedance) followed by a single difference amp (output) stage.

Non-inverting configuration has higher input impedance as the input goes directly to the high impedance non-inverting (+) input pin without taking material current from the input (loading), so there’s no good reason to lose this property by doing inverting op-amp configuration twice.

Most practical INA chips assign some of the user-adjustable gains at the buffer stage because ‘mirrored-ground’ (superposition) allows one resistor to program the gains for the 2 buffer amps without adding more mismatch, but conceptually the first stage’s primary purpose is a buffer. The rest of the gains can be hard-coded by manufacturing with matched ‘resistors’ inside the IC, mostly at the output difference amp stage.

But for illustration (so not to drown the readers in math), let’s assume the design choice of assigning the gains at the input buffer stage and make the output stage a unity gain difference amplifier (V_pV_n), which I use small letters (n, p) to denote the outputs of the buffer stage internal to the INA chip.


The REF pin (the part of R_4 exposed in Figure 6 above) is often advertised as an offset adjustment pin. This is just one of the many uses if you really understand the idea behind the 1-amp difference amplifier configuration.

The slightly more shallow perspective (more modular or system perspective that I’ve shared with EE122A students) is that if you look at an standard 3-amp INA configuration in regular textbooks like this,

Understanding CMR and instrumentation amplifiers - EDN

the entire sub-circuit (3 amp INA) is floating so it has absolutely no idea what the reference (‘ground’) would have been if we did not ARBITRARILY define it through R_4 by tying it to the CM ground, by forcing REF pin (V_{REF}) to be 0V (relative to the common mode ground shared with the inputs)!

In other words, we now have an INA ‘ground’ and a Common Mode (CM) ground, which they do not have to be the same unless we force them to be equal by shorting the REF pin to the CM ground.

This means whatever voltage V_{REF} we set the REF pin to be, it’s the baseline of the system (amplifier) and the whole output shifts moves up and down by whatever V_{REF} relative to common mode ground we are feeding into the REF pin for the moment.

The DC restoration takes advantage of the user-definable baseline (INA ‘ground’) by extracting a low-frequency (drift) portion of the output signal V_o with an INVERTING low-pass filter (LPF) with frequency response L(\omega), and re-define it as the INA’s ‘ground’ level. This is the LPF:

e.g. if the signal’s baseline drifted up by 1V, a -1V is generated by the inverting LPF and the INA ‘ground’ respond by moving from 0V down to -1V, which pulls the entire signal down by 1V, cancelling the 1V increase in baseline. All the voltages used here are relative to the common mode ground.

As with any AC coupling circuit, there is no precise definition of what ‘DC’ or ‘baseline’ is. It’s up to the experimenter to consider what cutoff frequency in the LPF is close enough. Technically DSP engineers can call a running window of trimmed-median the baseline if they wanted to.

The feedback (how fast the INA ‘ground’ is readjusted) is as responsive as the phase delay introduced by the LPF’s time constant. If you only consider anything below 0.00001Hz to be DC, you have to pay a price for the long delay catching up to the changes which might or might not be considered a baseline drift (it’s an application specific context).

I also have an alternate view of DC restoration which does not use the concept of INA ‘ground’ (not taught in EE122A). This is based on seeing the final stage op-amp (1-amp INA) not just as a simple difference amplifier, but as a 3-input arithmetic circuit (summing and subtractions) through super-position (setting one input to 0V at a time and add the results up).

This is the gut of a 1-amp INA difference amplifier

We can break it down into a (1 input) inverting amplifier plus a summing (2 inputs) non-inverting amplifier.

The equations for inverting amplifier and non-inverting amplifiers are not symmetric! The core part of the feedback gain in EITHER CONFIGURATION are ALWAYS set at the feedback branch which ONLY goes to the inverting input (-), aka R_1 and R_2!

  1. Inverting amplifier portion do not care about the resistors at the non-inverting input (+), but
  2. Non-inverting amplifier portion’s gain is determined by the 2 resistors at the inverting input (-)! The resistors at the non-inverting inputs (+) never boost the amplifier gains! They only attenuate signal from external sources (like voltage dividers). The gain boost happens ONLY at the inverting branch!

Why? By superposition (short out other inputs you are not considering)

  1. Inputs to R_3 (V_p) and R_4 (V_{REF}) shorted to the common ground gives an inverting amplifier. They don’t matter to V_n. The gain to V_n is -\frac{R_2}{R_1}.
  2. Input to R_1 (V_n) shorted to the common ground gives a non-inverting amplifier, which the gain boost (1+\frac{R_2}{R_1}) is relative to voltage V_+ showing up at the non-inverting input (+), which is the result of attenuating V_p through R_3 and V_{REF} through R_4.

The output contributions, if R_1=R_2 and R_3=R_4,

  1. [Inverting amplifier gain of -1 relative to V_n] contributes -V_n to the output
  2. [Non-inverting amplifier gain of 2 relative to V_+] if R_4 is set to ground through V_{REF}, R_3 and R_4 forms a 1:1 potential divider which halves V_p to give \frac{V_p}{2} at V_+. Doubling (2x gain) the halved input gives an overall gain of 1, therefore contributing V_p to the output

So the overall output equation is V_p - V_n if V_{REF} is grounded to 0V.

The intuition for DC restoration is to untie the REF pin (going to R_4) from CM ground and treat it as equals to V_p pin (going through R_3, so instead of a potential divider, they form a non-inverting summing amplifier:


So the DC restoration circuit can be seen as a 3-input arithmetic amplifier that gives the equation V_{REF}+V_p-V_n and you can subtract the baseline by setting V_{REF} to be whatever baseline your inverting LPF feedback branch judged. The overall AC-coupled system response is \frac{1}{1+L(\omega)}.

Note that ALL 3 inputs (V_p, V_n, V_{REF}) should be driven from sources with low output impedance. V_p and V_n is the outputs of a buffer op-amp so they already have good low impedance outputs feeding to the last stage. We’ll need to do the same for V_{REF} by using an op-amp to lower the output resistance whether it is an active low pass filter or active potential divider, because V_+ do not see V_p differently from V_{REF}. Noise showing up from high resistance output driving REF pin do not simultaneously appear on V_n, so it’s not canceled and therefore it’s worsening the common mode rejection (CMR).

 23 total views