Modifying mutable (like bytearray) arguments’ data in Python functions Use slice assignment x[:] = ... to replace the entire contents. Dynamically typed language lets you shadow input arguments with a local variable!

I’d like to write a function to selectively modify lines read from a file handle and write it back. By default, lines are read as byte()  objects that are immutable, so I converted it to bytearray() instead so it can be modified because only a few lines meeting certain criteria needs to be changed.

When I try to refactor similar operation into a function, I was hoping to pass the mutable bytearray() as an argument and directly modify the caller’s content like in C++, given Python variables works LIKE reference binding.

I know bytearray.replace() does not modify the data in place, but instead outputs the modified line to a new variable. Normally, I can simply do this:

line = line.replace(b'\tCLASS', b'')

and the code will work. However, it doesn’t do anything when I try to pass it as an argument to a Python function (unless I return line as output). Although I am well aware that Python variables assignments to existing variables means orphaning the old data and re-purposing the label, the variable assignment behavior in Python requires careful thought when used in non-idiomatic situation.

In other words, I want this function to have side effects on the variable ‘line‘, but I wasn’t doing it right. This is a tempting mistake for people with a C/C++ background: in C/C++, it is not possible to shadow an input parameter even if we were to explicitly declare it, so the innocent assignment I did above has to modify the object in the caller (passed as a reference to the function) in C/C++, as if I did this directly in the caller.

However, in Python, variables do not need to be declared (aka, dynamically typed). This opens up the possibility of unwittingly shadowing the input parameters, which is what happened here. Mutable arguments on the stack still can be modified through the function, but when you assign a variable using ‘=’ operator, a new local variable with the name on the LHS is created, which shadows the input parameter.

This means the connection to the caller objects is lost during shadowing.

The correct way to do this is use slice assignment (which the logic/concept is very different despite the syntax is similar) to replace all the contents of the input variable with the output of bytearray.replace():

def remove_from_header_token_CLASS(tokens, line):
     # line is expected to be byte array (mutable)    
    try:
        column_CLASS = tokens.index(b'CLASS')
    except:
        column_CLASS = None
    else:
        line[:] = line.replace(b'\tCLASS', b'')  
                
    return column_CLASS

Since Python has a clear distinct concept of parameter variable (from local variable), trying to apply nonlocal keyword over it (in hopes to broaden the scope) will not parse/compile.


This is actually the same behavior as in MATLAB (dynamic typing) for the same reason that variables does not have to be declared like in C/C++ (static typing). In MATLAB, if you choose to have a handle object (which works like references), you can shadow the input argument by creating a local variable of the same name:

classdef DemoHandleClass < handle
    properties
        x = 3;
    end
end

function demo_shadowing()
    C = DemoHandleClass();
    f(C);
    
    disp(C.x)
end

function f(C)
    C = DemoHandleClass();  // Shadowing
    C.x = 14;
end

The above MATLAB program will display 14 without shadowing and 3 with shadowing (C became a new local variable that has nothing to do with the input argument C). MATLAB users rarely run into this because the language design heavily discourage side-effects: we are supposed to return the changed local variable to the caller. The only way to do side-effects in MATLAB is through handles (which you need to establish a class, which is clumsy). Technically you can write the data to external resources (e.g. file) and read it back. But guess what? Resources are accessed through handles, so there’s no escape.


Of course, there’s a better way to do so (MATLAB’s preferred way): return the modified object back to the caller as if they are immutable:

def remove_from_header_token_CLASS(tokens, line):
     # line DOES NOT HAVE TO BE MUTABLE    
    try:
        column_CLASS = tokens.index(b'CLASS')
    except:
        column_CLASS = None
    else:
        line = line.replace(b'\tCLASS', b'')  
                
    return column_CLASS, line

This is what I ultimately used (so I ended up not converting the byte lines to bytearray), given that Python’s tuple syntax make it easy to return multiple outputs like MATLAB. The call ended up looking like this:

column_SPL_CLASS, line = remove_from_header_token_CLASS(tokens, line)                

Nonetheless, I think there’s an important lesson to be learned for doing side-effects in dynamically typed languages. Maybe I’ll need this one day if I get an excuse to do something more complicated that genuinely requires side-effects.


In summary, variable assignments in most dynamically typed languages will shadow the input argument with a newly generated local variable instead of modifying the data in the original input argument. This implies that there function side-effects cannot be carried out through variable assignment.

The most common implication is: do not (equality) assign to a input variable to modify its contents in a dynamically typed language.

22 total views, 1 views today

A Positive Tektronix Customer Support Experience

I often have bad experience with Tektronix product’s design (user convenience, reliability and repairability issues), and the by policy poor support for discontinued products. So far I have yet to get a chance to say good things about Tektronix while Agilent blew them out of water in these 4 areas.

Nonetheless, I have something good to say about Tektronix today. I have a DPO4000 series oscilloscope that the knob and busing popped out during shipping and disappeared, so I had to order them from Tektronix.

The operator on the phone noticed the part numbers and was aware that it’s a common problem that the jog shuttle’s knob and busing (for the Wave Inspector feature) often come loose, and offered to send it to me for free. That’s excellent customer service. The part that I admired the most is that they proactively acknowledged their design weakness and make amends.

Seems like Tektronix takes good care of their customers as long as the models are still supported. Definitely a redeeming quality!

32 total views, no views today

Samsung Galaxy Note 3 Charge and USB-OTG simultaneously

I’d like to charge my phone and use USB devices at the same time, but it seems like it requires a 64.9kOhm resistor from sensor ID pin (micro USB) to ground. Instead of melting a USB-OTG cable, I bought this adapter (schematics here)

micro USB3.0 Type B Male to USB3.0 Type A Female adapter

so that I can have direct access to the ID pin. This is a USB 3.0 give that I have a Galaxy Note 3. The same principles apply to the USB 2.0 versions for Galaxy Note 4.


According to this website, fsa9480_i2c.h has the table for the resistor ID values. Turns out 64.9kOhm is the one for both charging (slowly) and using USB devices (like mouse, network adapter, etc.).

RID_USB_OTG_MODE,	/* 0 0 0 0 0 	GND

USB OTG Mode

              */
RID_AUD_SEND_END_BTN,	/* 0 0 0 0 1 	2K		Audio Send_End Button*/
RID_AUD_REMOTE_S1_BTN,	/* 0 0 0 1 0 	2.604K		Audio Remote S1 Button */
RID_AUD_REMOTE_S2_BTN,	/* 0 0 0 1 1 	3.208K		Audio Remote S2 Button                         */
RID_AUD_REMOTE_S3_BTN,	/* 0 0 1 0 0 	4.014K		Audio Remote S3 Button */
RID_AUD_REMOTE_S4_BTN,	/* 0 0 1 0 1 	4.82K		Audio Remote S4 Button */
RID_AUD_REMOTE_S5_BTN,	/* 0 0 1 1 0 	6.03K		Audio Remote S5 Button */
RID_AUD_REMOTE_S6_BTN,	/* 0 0 1 1 1 	8.03K		Audio Remote S6 Button */
RID_AUD_REMOTE_S7_BTN,	/* 0 1 0 0 0 	10.03K		Audio Remote S7 Button */
RID_AUD_REMOTE_S8_BTN,	/* 0 1 0 0 1 	12.03K		Audio Remote S8 Button */
RID_AUD_REMOTE_S9_BTN,	/* 0 1 0 1 0 	14.46K		Audio Remote S9 Button */
RID_AUD_REMOTE_S10_BTN,	/* 0 1 0 1 1 	17.26K		Audio Remote S10 Button */
RID_AUD_REMOTE_S11_BTN,	/* 0 1 1 0 0 	20.5K		Audio Remote S11 Button */
RID_AUD_REMOTE_S12_BTN,	/* 0 1 1 0 1 	24.07K		Audio Remote S12 Button */
RID_RESERVED_1,		/* 0 1 1 1 0 	28.7K		Reserved Accessory #1 */
RID_RESERVED_2,		/* 0 1 1 1 1 	34K 		Reserved Accessory #2 */
RID_RESERVED_3,		/* 1 0 0 0 0 	40.2K		Reserved Accessory #3 */
RID_RESERVED_4,		/* 1 0 0 0 1 	49.9K		Reserved Accessory #4 */
RID_RESERVED_5,		/* 1 0 0 1 0 	64.9K		Reserved Accessory #5 */
RID_AUD_DEV_TY_2,	/* 1 0 0 1 1 	80.07K		Audio Device Type 2 */
RID_PHONE_PWD_DEV,	/* 1 0 1 0 0 	102K		Phone Powered Device */
RID_TTY_CONVERTER,	/* 1 0 1 0 1 	121K		TTY Converter */
RID_UART_CABLE,		/* 1 0 1 1 0 	150K		UART Cable */
RID_CEA936A_TY_1,	/* 1 0 1 1 1 	200K		CEA936A Type-1 Charger(1) */
RID_FM_BOOT_OFF_USB,	/* 1 1 0 0 0 	255K		Factory Mode Boot OFF-USB */
RID_FM_BOOT_ON_USB,	/* 1 1 0 0 1 	301K		Factory Mode Boot ON-USB */
RID_AUD_VDO_CABLE,	/* 1 1 0 1 0 	365K		Audio/Video Cable */
RID_CEA936A_TY_2,	/* 1 1 0 1 1 	442K		CEA936A Type-2 Charger(1) */
RID_FM_BOOT_OFF_UART,	/* 1 1 1 0 0 	523K		Factory Mode Boot OFF-UART */
RID_FM_BOOT_ON_UART,	/* 1 1 1 0 1 	619K		Factory Mode Boot ON-UART */
RID_AUD_DEV_TY_1_REMOTE,	/* 1 1 1 1 0 	1000.07K	Audio Device Type 1 with Remote(1) */
RID_AUD_DEV_TY_1_SEND = RID_AUD_DEV_TY_1_REMOTE ,		/* 1 1 1 1 0 	1002K		Audio Device Type 1 / Only Send-End(2) */
RID_USB_MODE,		/* 1 1 1 1 1 	Open		USB Mode, Dedicated Charger or Accessory Detach */

 

51 total views, 1 views today

Windows path length limit

Windows has a path length limit that are typically at the order of 250 (260 for Windows 10) that’s a pain in the butt when moving files. Despite you can override it, it’s no fun when you copy a jillion files just to find out a few can’t make it because the path is too long and you have to find out which ones are not copied!

There’s a short command to check if the path exceed certain number of characters, which I recommend testing for 240 character so you can at least have a 10+ character folder on the root folder to put the files in:

powershell: cmd /c dir /s /b |? {$_.length -gt 240}

26 total views, no views today

Dual-booting: Linux and Windows fight for the system clock

Turns out it’s a common problem when dual-booting Windows and Linux, they keep changing the hardware system clock on each other (unless you live in GMT+0 zone) because Windows assume the system time is the one at the set timezone while Linux think the system time is the UTC+0 time (and offset it afterwards).

Linux updates the time through NTP server blindly while Windows 7 check if the current time is within 1hr from the NTP server to avoid unintended time changes (I have to give Microsoft credit for that). EDIT: Windows 10 blindly updates the time like Linux too.

The easy solution is to have Linux follow Windows’ suit:

timedatectl set-local-rtc 1 --adjust-system-clock

43 total views, no views today

X11VNC for Linux setup notes

x11vnc is a relatively smooth experience, but there are quite a few common use cases that would have been automated away if it’s a Windows program, namely have it start as a service on boot (before logging in)

It’s from babelmonk’s solution on StackExchange. Paraphrased here to make it easier to understand:

After installation, create the password file with -storepasswd switch AND specify the where you want the password saved as an optional argument, and I prefer /etc/x11vnc.pass:

sudo x11vnc -storepasswd {your password goes here} /etc/x11vnc.pass

which will be read by -rfbauth switch for the x11vnc program.


Build your own (systemctl) service by creating /etc/systemd/system/x11vnc.service:

[Unit]
Description="x11vnc"
Requires=display-manager.service
After=display-manager.service

[Service]
ExecStart=/usr/bin/x11vnc -xkb -noxrecord -noxfixes -noxdamage -display :0 -auth guess -rfbauth /etc/x11vnc.pass
ExecStop=/usr/bin/killall x11vnc
Restart=on-failure
Restart-sec=2

[Install]
WantedBy=multi-user.target

Then, start with:

sudo systemctl daemon-reload
sudo systemctl start x11vnc

Enable the service (if not already done by previous commands) so it will start on boot

sudo systemctl enable x11vnc

41 total views, no views today

Acrobat reader on Linux

Adobe gave up supporting Acrobat reader for Linux long time ago, so it’s stuck at the old 32-bit version (9.5.5):

http://ardownload.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/AdbeRdr9.5.5-1_i386linux_enu.deb

ftp://ftp.adobe.com/pub/adobe/reader/unix/9.x/9.5.5/enu/AdbeRdr9.5.5-1_i386linux_enu.deb

The tutorial websites tells you to use wget, but sometimes you might run into authentication problems. You can simply use the links above and double-click the .deb file to install.

Nonetheless, it doesn’t work right out of the box in modern 64-bit Linux. You’ll run into a missing library

libxml2

on run because you didn’t install the 32-bit version of it. Enable i386 (x86 or 32-bit) packages first then get the 32-bit library:

dpkg --add-architecture i386
apt-get install libxml2:i386 ia32-libs

There are some GTK complaints if you run it on a command line, but it doesn’t affect the uses so you can safely ignore them

36 total views, no views today

systemd-resolved DNS resolution nightmare

Linux Mint 19 does not resolve local hostsnames (nothing to do with SMB, which does not rely exclusively on DNS) out of the box! Damn. MX Linux does.

systemd-resolve, which act as a local DNS server on 127.0.0.53, despite it points to the DNS server assigned by the router’s DHCP (aka, the router’s IP address itself), managed not to resolve the local hostnames out of the box!

Time to disable this mofo (no need to mess with /etc/nsswitch.conf and install Winbind to use WINS):

Disable and stop the systemd-resolved service:

sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved

Then put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf:

dns=default

Delete the symlink /etc/resolv.conf

sudo rm /etc/resolv.conf

Restart NetworkManager

sudo systemctl restart NetworkManager

I know it has good reasons to exist (like breaking VPN ties), but if Linux Mint decide to have it on as out-of-the-box defaults, at least tell the users that local network DNS resolution won’t work by default!

This is a choice that caters the 5% elite at the expense of frustrating 95% of the target audience!

70 total views, 4 views today

Windscribe Linux breaks internet (messed up DNS resolution) on first use

I lost the internet (actually just DNS not resolving correctly) after installing Winscribe for Linux and disconnecting the session. WTF windscribe! I know it’s beta version, but at least you should check if it bricks people’s internet after a fresh install and first use!

Turns out that on first connection, it re-binds /etc/resolv.conf to /run/resolvconf/resolv.conf which has this line:

# Generated by resolvconf
nameserver 10.255.255.2

So like systemd-resolve, Windscribe lets resolvconf steal the DNS redirection that’s supposed to go straight to my router to an intermediary 10.255.255.2 that doesn’t do the job! Aargh!

To fix it (needs to be done every time after a Windscribe connection, so I’m getting rid of this lamely written Windscribe CLI for now), remove the symlink /etc/resolv.conf:

sudo rm /etc/resolv.conf

and restart NetworkManager

sudo systemctl restart NetworkManager

so NetworkManager will re-generate /etc/resolv.conf directly (no symlink) with the correct name server from the GUI config program (in my case, automatically obtained from my router).

Turns out it’s a common scene that in Linux, many DNS resolution program fight over the control over /etc/resolv.conf. NetworkManager kicks in after you disabled the rest.

76 total views, no views today