F#*@ing newline in Putty

I tried to use Putty to connect to my serial port test instrument (Agilent’s 33120A or E3600 series power supplies) and to my dismay it doesn’t respond to my commands.

I figured it might be newlines not being recognized properly. Checked the programming manuals and found that HP/Agilent accepts LF (‘\n’)as newline and optionally allowing a CR (‘\r’) before it (i.e. CR+LF like Windows).

I thought this configuration (Implicit LF in every CR) would work:

However it doesn’t! I had to dig through internet forums to find out that the ‘Terminal’ settings page controls what gets SPIT OUT TO THE TERMINAL SCREEN, not how your keystrokes are treated/sent! This page controls what your keystrokes mean:

However, it says nothing about what Putty does with ENTER key. I found from Stackflow that Putty sends out only CR (‘\r’) when ENTER key is pressed.

Turns out with terminals,

  • Ctrl+J is LF
  • Ctrl+M is CR,

and the convention for ENTER is Ctrl+M (also mentioned here):

https://ss64.com/bash/syntax-keyboard.html

So if you want ENTER/RETURN key to generate LF (instead of CR) with the official putty, you have to press Ctrl+J each time!

Putty sending CR (Ctrl+M) with ENTER key by default is also hinted by the default Telnet (only applies to telnet, not serial) setting in the Putty docs “Return key sends Telnet New Line instead of Ctrl+M”, which means outside Telnet, it the default behavior or ENTER/RETURN key is Ctrl+M (CR)!

Ironically even HyperTerminal came with the option to send out CR+LF on ENTER key! Luckily some kind soul (Grzegorz Niemirowski) compiled a mod (currently v0.74) which added the option and and posted it on Stack overflow:

Loading

RS-232 motherboard header mapping (DB9)

The pin ordering for RS-232 (DB9) pin is sequential is row-wise (the long side is a row) while the IDC-10 (ribbon cable) header is column-wise (zig-zag pattern).

This might be a little confusing because geometrically, they are in-place on both sides (you can overlay the pins of DB9 on top of IDC-10 and they align perfectly, except pin 6-9 was lowered by half a notch on the DB9 side). I am writing this post so nobody waste their time separating the wires in a ribbon just to find out the DB9 was designed so it aligns with the flat ribbon cable perfectly.

Here’s a great tutorial building your own RS-232 cable which I took the images from to illustrate the point. Please pay them a visit to show some love: https://developer.toradex.com/knowledge-base/assembling-serial-idc-to-db9-cable

If you split the table on the right in half (cut after pin 5) and place pins 6 (DSR) ~ 9 (RI) on the right, you’ll see it align with the IDC10

1DCD (Data Carrier Detect)
Check if connection dropped
DSR (Data Set Ready)
DTR-DSR Handshaking
6
2RxD (Receive Data)
RTS (Request to Send)
RTS-CTS Flow control
7
3TxD (Transmit Data)
CTS (Clear to Send)
RTS-CTS Flow control
8
4DTR (Data Terminal Ready
DTR-DSR Handshaking
RI (Ring Indicator)
For phone rings
9
5GND (Ground pin)– (Not connected)10
Rearranged DB9M RS-232 to align with IDC10

Cable-tester.com has a clearer annotated picture that matches the physical mapping above:

http://www.cable-tester.com/rs232-pin-out/

Note that the tutorial itself has Tx(D) and Rx(D) reserved it was building a null modem cable and they skipped all the handshake lines. I’m doing a straight cable (which should be done for internal board header cables where the DB9 socket is male, hence DB9M).

The DTE/DCE might be confusing. Hope these properties can help people make sense out of it (so you can figure it out in your head confidently instead of randomly trying null modem adapters till it work)

  • DTE device (colloquially ‘computer’) has the pattern as shown in the pictures above (receive pins above/before the transmit pins), which is usually the computer end and the port/socket is male. Think of it as the ‘driver/master’ (though it’s arbitrary)
  • DCE device (colloquially ‘modem’) reverses all the sends and receives of the DTE. Can think of it as the ‘receiver/slave’ (though it’s arbitrary). It’s usually the modem and the port/socket is female.
  • For DTE-DTE (like data transfer between two PCs), the send lines on one side should go to the receive lines on the other side. A null modem cable that swaps the send pins with receive pins. You can think of it as making one side DCE. Given that the topology is symmetric, it’s up to the software set up to decide which side is the initiator/client (master) and which side is the reactor/server (slave)
  • The handshaking (optional) and flow control (optional) lines also have their initiator/reactor roles reversed with null modem cable.
http://www.lm-technologies.com/wp-content/downloads/wireless%20adapters/LM048/Application%20Notes/DTR-DSR-RTS-CTS_Difference.pdf

Here’s an image for the above mapping:

Puertos E/S COM
https://itigic.com/input-and-output-ports-on-pc-types-uses-and-characteristics/

It turns out the direct geometric mapping (IDC male pins match the relative locations of the DB9 male pins) mentioned above is the less common type of motherboard header configuration. IDC ribbon crimp-on DB9 headers like this:

ฅนบ้ายอ: [34+] Db9 Male And Female Connector Pinout

has to follow the above geometric layout since the pins cannot be remapped (so it has to follow the ribbon order). The soldered version looks like this:

https://store.cwc-group.com/lowprldb9mat1.html

However, the more commonly seen soldered RS-232M to IDC10 header uses a transposed configuration (which DB-9M pin numbers matches IDC-10 pin numbering EXACTLY despite one is row-major and the other is column-major), which has nothing to do with the IDC10 pin layout mentioned above.

https://store.cwc-group.com/lowprldb9mat.html
(I don’t think it’s a good idea to call it ‘crossed-config’ like CWC did.
It almost mislead me to think it happens to swap the roles of Tx and Rx.
I did the mapping on paper and it didn’t make any sense.
Let’s call it ‘transposed-configuration’)

The crazy thing about the existence of these 2 pin layouts is that there’s no easy way to tell which pin layout/mapping it is until you open the connector up and inspect the solder joints! Taking a pin and probe it with a multimeter is more work than taking the screw posts out and disassemble the connector.

So if you just buy some old scrap parts that came with old motherboards, this might confuse the heck out of you until you tested the pin mapping with a multimeter and realize things doesn’t add up!

Note to self: just open the DB9 side up whenever I see a DB9-IDC cable and mark the configuration on the DB9 end directly on the cable!

Loading

Off the Goolag (2): Low cost shared hosting

For absolutely privacy, avoid using email (say, talk on Signal instead). Unless it’s inter-server mail in zero-knowledge encrypted providers like ProtonMail that also encrypt the message headers (meta-data, especially who’s sending to who), expect determined people with enough social engineering or authority can see it naked. It’s the same deal as snail mail where people in the post office can see what’s written on the envelope.

For big files like photos and typical cloud storage, which you should self-host these at home anyway. If you are worried about slow internet connection and downtime, you can pay for Zero-Knowledge cloud storage (which the server owners don’t have the master keys to your files) to add redundancy.

The next step down is to self-host your email, contacts, calendar, tasks (productivity suite) which you physically own so nobody can peek into it as long as you guard your home.

Hosting these services from home might be more work and risks (downtime), especially when it’s possible that your ISP’s IP address block is on the spammer’s list or if your ISP blocks the ports needed. The less secure alternative is to pay for extremely cheap shared web hosting services (we are talking about <$4/mo regular price and <$2 for the first year) which

  • you can make as many email accounts as you wanted
  • each email account comes with contacts, calendar, tasks as a bundle
  • use your own domain name
  • also host your own webpage and wordpress site

With Google, Microsoft, Apple and other big providers, they have big security teams to protect your data from hackers, but because of their centralized nature, it’s much more rewarding for hackers to breach one big provider than going after little accounts spread across different servers and IPs. Unless you are a high profile person or expect to be specifically targeted, you are better off managing your own productivity suite’s hosting/storage.

More importantly, it feels creepy when Google harvest my email and suggest I allow them to automatically register my appointment on my calendar. Random staff might not be reading our emails, but bots are and god knows what else they can do just by updating their code if they someday want to turn on us. They’ve become so powerful that with enough bankroll, they can make our politicians look the other way so there’s no way to stop them if we become dependent on their platforms.


cPanel

The instructions below assumes your shared hosting provider adopted cPanel as the account management interface which you have access to.

Like Google, your Gmail (email) account is also your account for a variety of productivity services (contacts, calendar, tasks). You can set it up by logging into cPanel, often https://(your server here)/cpanel.

There are a few naming conventions in cPanel that are different from Google’s ecosystem:

  • Login name is your ENTIRE email address because you can have different domains attached to the same hosting storage so you must enter the domain name after the @ sign for it to tell the accounts apart

Email

In modern times, I’d stick with IMAP for email (which is enabled by default in cPanel). Since Google would like to keep you in their ecosystem as much as possible, IMAP is not enabled by default for Gmail.

Note that due to tougher security settings in shared hosting email server (EXIM Internet Mailer), you might not be able to receive email sent from servers with shady practice (often done by spammers) such as the source address’s server not resolving with DNS (no A or MX records). Skype server might have a typo in their verification email server so I cannot use my shared hosting email address for it.

Webmail

Web email interface (you have a choice between Horde or RoundCube) is at port 2096. You can access it by

https://{name or IP to the shared host server assigned by your provider}:2096

or

https://{name or IP to the shared host server assigned by your provider}/webmail
(which will redirect you to port 2096 above)

In most cases, your domain name attached to the hosting points to the actual underlying shared hosting server assigned by your provider. I’d prefer not to use the underlying server address/IP because it might change when you move between hosting plans.

Also, per security design, WebMail doesn’t warn you when you enter non-existent email addresses (login). I’ll just silently loop you back to the login page again without explanation if you got any part of the login or password wrong.

DavX5 for calendar/tasks (CalDAV) and contacts (CardDAV)

In Android, calendar and contacts (also known as address book) are stored in a standard place shared by apps that picks them up from the system (email storage is per app, since POP3 and IMAP itself already does things very differently)

The default Calendar/Contacts app made it look like you have to use Google Calendar/Contacts to set up an online account (by default it came with Device/Local and Google accounts as option), but you can inject CalDAV/CardDAV accounts into the Android’s calendar/contacts system with an app called DAVx5.

The App is FREE if you download it from F-droid but costs $5.99 if you download it from Google Play. It’s not a loophole, but the authors want people to move away from Google Play and use F-droid, a Free-and-Open-Source (FOSS) app store.

DAVx5 works in a little unusual way that accounts are NOT added through calendar/contacts app but instead you register your CalDAV/CardDAV accounts, select the folders to sync, SYNC IT, then each sync’ed FOLDER (you hear me right) will show up as standard Android Accounts (just like Google/Samsung Accounts) which will work with any standard Calendar/Contacts app. All management (add/removal) happens in DAVx5.

You must enter the URL which points to Port 2080 of the shared hosting server! If you forget to enter the port number, the account will be set up with CalDAV/WebCAL only, without CardDAV! This is buried in cPanel’s instructions:

When you set up an CalDAV/CardDAV account, remember NOT to use the first option “Login with email address” (or else the server URL will be whatever that’s after the @ mark and therefore no port number entered, which means CalDAV only, no CardDAV, and DAVx5 will simply remove the CalDAV tab and won’t warn you about it)!

Select “Groups are separate vCards“:

Help - Davx5 (Davdroid): How do I use the Posteo address book and the  Posteo calendar on Android devices? - posteo.de
https://posteo.de/en/help/synchronising-contacts-and-calendar-entries-with-the-address-book-and-calendar-using-davdroid-android

Basically CardDAV is just a folder storing each contact as VCF (vCARD) file and CalDAV is just a folder storing each event/task as an ICS file. Basically it’s just a primitive HTTP file manager hosted with HTTPS login and apps are supposed to find the folder using a consistent naming scheme.

Loading

Freshtomato for Netgear (Nighthawk) R7000

Netgear R7000 supports these major forms of firmware

  • DD-WRT (Powerful, but very messy web interface that are sometimes non-intuitively organized)
  • FreshTomato (Powerful. I wouldn’t say easy to use but mortal souls can understand it)
  • XWRT-Vortex (Easy to use AsusWRT Merlin web interface adapted for non-Asus routers)

There are other forms of Tomato that support R7000 but only FreshTomato is actively maintained as of late 2021.

However updating it from stock firmware to FreshTomato has some model-specific quirks (that you cannot extrapolate from general procedures for other models)

First of all. You cannot update directly to the latest firmware. There’s a bootstrap (intermediate) firmware called INITIAL (usually downloaded from ‘Netgear R-series initial files’ folder) that must be installed (upgraded from stock firmware to) FIRST so the router is ready to accept the latest/full firmware.

Here’s the model specific quirk: the default login/password is non-standard for R7000! It’s not root/admin (unless you press the button to reset the NVRAM)! It’s admin/@newdig!

After logging into the bootstrap/INITIAL freshtomato with the password above, upgrade the firmware to the latest (the one intended) and choose clear NVRAM along the way. The default login/password will be root/admin as standard for freshtomato.


There’s another twist for SSH connections! The username in your web admin interface do NOT matter! The username is ‘root’ for SSH regardless of what you set in the web interface, your password is the one entered in web admin interface! This is super counterintuitive!

Loading

Off the Goolag (1): Android (AOSP)

App Stores

The first thing to worry about after a fresh install is getting the Apps you need. I’d recommend installing all these app stores to start with:

  • F-droid (Only free and open source, privacy-respecting and community verified apps. Some paid apps on Google Play such as DavX and FairMail is free on F-droid to promote it! Note that sometimes the F-droid repository might be a little behind)
  • Aptoide (App writers self-publishes app under community scrutiny, which sometimes let you download geoblocked app. Fairly updated but not as updated as Auora. The apps are not tightly guarded as F-droid)
  • Auora (It’s a proxy to downloading the APK from Google Play store without letting Google track you. It’s anonymous account is international, which makes the search for the basic apps very difficult as it’s seeing a worldwide scope. You can have Auora sign in with a disposable Google account for apps that are relevant to your regions. Remember to disable your Google Play if installed and have Google Play links open with Auora.)

Yalp store is outdated and not actively maintained. Google Play store proxies needs to constantly fight with Google’s changes so they need to be updated frequently. Just stick with Auora for now.

Install Island (Sandboxing apps)

This is the very first app you need to install after you get F-droid or Auora. This app is a must even if your phone is not DeGoogled. A lot of apps asks for more permissions than they actually needed, but sometimes we are stuck with using them (like required at work).

The solution is to create a sandbox (another copy/instance of the app) that has a different space for app data and they cannot see your actual call logs, contact list, photos, etc even if you gave them the permission to do so. It’s called an ‘Island’ and the native space is called ‘Mainland’.

Putting an app in Island (work mode) doesn’t protect you from other access requests such as location, etc. Nonetheless the mainland and island app has their own data space (i.e. you have to configure it twice as if the apps are freshly installed) so they can have two sets of settings and application permissions.

Apps installed through the Island instance of the App Stores / Browsers above will stay in Island mode. Apps installed through Mainland instance of the App Stores will stay in Mainland mode. They are totally separate as intended. You can clone the APK/app between Island and Mainland through the Island App.

Note that some VPN clients will have a split personality (which is a good thing) between Mainland and Island! This means you get to have a group of apps that’s on VPN and a group of apps that’s on the direct network without managing them one by one with split tunneling!

Also note that the Island won’t be able to access external storage like SD cards. It’s by design so that Island apps are trapped in their own virtual space so they cannot snoop around your personal data even if you gave them the permissions (demanded during installation)

Browsers: Brave + DuckDuckGo

DuckDuckGo (sometimes the permissions and default app lunching do not work correctly with it. I use DuckDuckGo first whenever it doesn’t break)

Brave (based on Chromium). Replaces Chrome. It has a chain sync feature that syncs passwords, bookmarks, etc like Google Chrome does but WITHOUT AN ACCOUNT. As long as you have one device with Brave connected to the Internet and you did the steps to match the devices, they will sync up.

Keyboard: MS Swift Keyboard

I speak 5 languages and found Microsoft Swift on screen keyboard having nice IME (Input Method Engine) for all of them and also have an intuitive interface that’s not clumsy to use. I prefer it over Gboard and AOSP that came with LineageOS). I do not log in to Microsoft Swift and share data with them (the petty convenience of sharing the clipboard is just not worth it).

NON-CLOUD based Email client: Fairmail / MailDroid

Email is a huge topic which I’ll discuss in a separate post as it often come as a bundle with contact list, task lists, calendars, and taking notes.

Do NOT use free apps that sends your credentials to the provider‘s BACK-END SERVERS which you don’t own and manage, such as BlueMail if you are doing all these to protect your privacy (you might as well use Google if you do that)! NextCloud is OK as long as you host it.

K-9 mail client is promising but I do not like it implicitly forcing you to log in through Google’s web interface to set it up instead of doing the traditional IMAP setup if you use Gmail.

Fairmail has a lot of extra steps during setup because it let you customize the heck out of it and by default (out of the box), it protects you from tracking images and malicious HTML dingleberries out of the box (which hurts readability).

MailDroid is is supported by in-app advertisements (you can remove ads with Pro version). It’s more intuitive than Fairmail and K-9. It automatically detects Gmail’s IMAP settings (not Google OAuth2 login) correctly, but doesn’t autodetect does not work with namecheap’s cPanel mail while Fairmail does.

There’s an unintuitive design choice by MailDroid that by default Sent email are saved on the device (local folder), not the IMAP ‘Sent’ folder. The fix is also awkward, but it’s doable and you only have to do it once for every account added to Maildroild.

I don’t use Aqua Mail because the free version do not allow multiple accounts like MailDroid does.

K-@ Mail feels like Gmail. Support multiple accounts. Also autodetect Gmail IMAP correctly but not cPanel email correctly like Maildroid. IMAP Folders do not work correctly for either Gmail/cPanel (it shows nothing): the folder button (bottom left) shows generic Inbox/Draft/Outbox/Sent/Trash/{Folder list} which do not match the IMAP folders. “Folder list” shows the IMAP folders, but when I clicked on them it shows nothing.

Just because of the non-working IMAP folders, I chose to not use K-@ Mail and stick with MailDroid.

I personally like Fairmail because the interface wastes no visual space showing all my IMAP folders. MailDroid is visually pleasing but the folder panel took up too much white space and I cannot tuck away the special (local) folders in the side panel.

Chat: Signal, VoIP: Telegram

Fascist book now requires data sharing with Whatsapp, so people in Hong Kong who don’t trust the fascist Chinese Communist Party regime is dropping it like a hot potato.

Signal App do not keep a master key on your message (if you lose the key, you lock yourself out and there’s no recovery) and is my preferred app for chat. Although Telegram’s owner has a good track record of protecting political dissidents (that’s why it was used in 2019 Hong Kong Protests), it’s not fully open source and the owner still has the master key. Telegram is still way better than Whatsapp but for full privacy I stick with Signal App.

Signal App’s voice over IP is a little weak and it can break up easily on spotty network connection. Telegram is much better in terms of voice quality so I basically use Telegram as a VoIP phone and leave the chats to Signal.

Loading