KVM-Qemu Duality

Qemu started off as a software emulator at user space (Type 2 hypervisor like VirtualBox) when there weren’t hardware features specifically supporting VM use cases.

KVM extended Qemu to take advantage of hardware features that accelerates virtualization which requires kernel space (OS low level direct) access, more directly handing off requests to hardware instead of going exclusively through the emulated fabric.

Conceptually I think of KVM as the kernel space (low-level hardware drivers / hardware accelerator) portion of the software while Qemu is the user space (interactive) portion of the software.

What made it confusing is that KVM was a fork of Qemu, so they were seen as one package when people said ‘KVM’ (when they should have said qemu-kvm). The responsibilities are not that clearly separated as far as Linux is concerned but when we get to MacOS/Windows, we start to appreciate the distinction that KVM is the kernel-based hardware accelerator itself while Qemu is the whole idea of virtualization.

This is confirmed by Qemu’s docs:

KVM made no sense in Windows, as Microsoft is obviously going to implement KVM in its kernel. Straight ports of Qemu to Windows retained the KVM lingo, which confused the hell out of me.

Since KVM does not exist on Windows. Intel use to make HAXM for its processor on Windows, but they discontinued it. Microsoft’s version of real hardware accelerator is WPHX. These are the only 2 available hardware acceleration options for Qemu on Windows.

TCG (Tiny Code Generator) is not real hardware acceleration, but more like an on-the-fly machine code (think of it as assembly for those who don’t know the difference) instruction translator that matches the VM guests’ instruction set with the host hardware’s instruction set.

People usually call ‘KVM’ (qemu-kvm) a Type 1 hypervisor but the line is blurry. Does it still count as Type 1 if you add an FTP server on the linux distro that hosts the KVM? What makes KVM fast was its hardware acceleration through the kernel, but the user-space qemu calls can make these kernel calls by using KVM as a hardware accelerator.

In my opinion, whether it’s Type 1 or Type 2 mostly boils down to intent on whether you are dedicating the computer to serving VMs. Most people nowadays won’t insist on foregoing available hardware acceleration (which is kernel space) to solely run the VM as a purely software emulator. The VM idea is to decouple the hardware from the software, who cares if you let the host computer take a side job (like running a FTP server) as long as virtualiation overhead is the same?

Loading

Maildroid does not save sent email to IMAP ‘Sent’ folder by default

This behavior is so frustrating. We don’t live in a time where IMAP storage space is at a premium anymore! Fairmail saves a copy of sent mail in the IMAP’s ‘Sent’ folder by default.

I did some research and it seems like MailDroid insists on ‘saving the sent mail in “Sent from device” folder (I tried disabling it thinking Maildroid will be smart enough to save it in the IMAP’s sent folder, but no, it doesn’t).

Fairmail’s Settings->Send->Message->[Last Item] On Replying to a message in user folder, save the reply in the same folder was disabled (should be, or else it’d be messy), but there’s a subtext that says “The email server could still add the message to the sent message folder”, which points me to think saving a copy to the Sent folder is an IMAP server managed behavior, so it doesn’t rely on email clients specifically telling it where (which folder) to save the sent email.

I was about to give up Maildroid and stick to Fairmail, and did a little research on “imap put sent mail in sent folder” and viola, Maildroid is the first one that came up. Seems like I wasn’t the only one perplexed by Maildroid’s weird design choices and it seems like most other email apps do not have this awkward behavior.

I did as what the dev for maildroid said and it turns out all of the email accounts in use says “Not specified”. When I clicked on one of the accounts, the root folder /Inbox shows up. I had to expand it with the ‘>‘ so the (IMAP) sub-folders show up.

This GUI was designed horribly and it’s clearly an afterthought. The reason I’m saying that instead of just assigning the IMAP sent folder right away after I tapped on it, the UI changes the text at the bottom of the screen (WTF) to what I’ve selected (despite there were no multiple choices) and expect me to click the DONE next to it. It’s so unintuitive in many levels.

Loading

How to get a CAP (cantonese IME /w swearwords) and coexist with mozc (Japanese IME) in Linux

I’ve been using CPIME for ages and I’m comfortable with Sidney Lau’s phonetic scheme. Jyutping is unnatural to Hongkongers because we do not consider ‘j’ a ‘y’ sound like Germans do.

However, since Windows 10, there aren’t much choices when it comes Cantonese IME that defaults to Sidney Lau’s and yet it accommodates common swearwords (including the most common ones that were technically incorrect) well. The only reasonable choice is Andrew Choi’s CAP, which I will write about how to get it working for Windows 10 on another blog post.

There aren’t much choices for Linux either. There’s RIME, but it’s super hard to install and Sidney Lau’s phonetic scheme is buried deep down that can only be changed with shortcut keys during the IME composition mode. The deal breaker is t.he lack of swearword support. Being able to type 林鄭我?你老母 is essential for every self-respecting Cantonese speaker. 「屌」你老母 just won’t cut it. Lol.

CAP takes a quite bit of wrestling to get it to install in Windows (in another post) and quite a bit of wrestling to get it to function in Linux. Once you get it working, it’s a very powerful Cantonese IME that allows superfast typing unless you plan to play with words (玩食字). I just can’t praise the IME design enough and I was willing to deal with the quirks which curbs its wide adoption.

Andrew Choi made a few release at his blog page in 2012 (ibus), 2015 (ibus), 2018 (fcitx), 2019 (fcitx4) and 2021 (fcitx4). For the linux version, this blog post is only concerned with the 2021 version (latest at the time of writing).

For Linux CAP, installing the debian package is the easiest part:

sudo dpkg -i fcitx-cap_1.0.0_amd64.deb

The thorns are

  1. get the CAP show up on the list of valid input methods
  2. fend of fcitx5 which is trying to kill CAP
  3. deal with IME settings state corruption (especially when working with other IME)
  4. live with being unable to select characters from subsequent pages of selection candidates

CAP is not immediately available as an IME out of the box
(even after installing the .deb package)

.

You will need to add Chinese to “Install / Remove Language” under “Language Support” to get anything to show up there!

Fend off fcitx5

fcitx5 was recently released and Ubuntu is aggressively trying to push it onto every user. However, its very existence kills the currently available CAP which is written for fcitx4 as of the 2021 release. This means you will have to give up fcitx5 if you want to use CAP!

Fcitx5 is considered as a replacement for fcitx4, so whenever Ubuntu sees that you have fcitx installed (which is likely fcitx4), it’ll tempt you into installing fcitx5. DO NOT ACCEPT THE INVITATION! However, fcitx5 do not coexist with fcitx4. Your fcitx4 will be removed the moment you installed fcitx4. To go back to fcitx4, you have first to remove fcitx5 completely then re-install fcitx.

What makes things more complicated is that Ubuntu’s gnome Language Support GUI keeps prompting you to install fcitx5 whenever you start it or do something with it such as installing new languages (which is required as the first non-obvious step). It’ll typically try to deceive you into installing fcitx5 with a dialog box like this:

but if you open up the details it’s fcitx5 which will cockblock CAP

However, when you try to perform the first step, if you already have fcitx (fcitx4) installed, adding new languages (required to get CAP) to work will come bundled with upgrades to fcitx5! It’d be super frustrating. So you can choose between the two paths

1) Concede to fcitx5 and downgrade to fcitx4

  • install the languages first (with fcitx5 IMEs),
  • remove fcitx5
  • install fcitx4

2) Prevent fcitx5 in the first place

  • remove fcitx4,
  • install the languages (no fcitx IMEs)
  • install fcitx4

Remove fcitx4

sudo apt purge fcitx

Remove fcitx5

sudo apt purge fcitx5*

Install languages

If you already have fcitx installed (path #1), you’ll have to click yes and live with fcitx4 being upgraded to fcitx5 which you’ll have to destroy it later and reinstall fcitx4.

If you already removed fcitx, Language Support will only install IME for other systems such as ibus associated with installed languages

Install fcitx4 AND activate it

sudo apt install fcitx

Remember to select the installed Fcitx 4 as your IME system (not ibus, etc, or none):


Now CAP is on the list of available IMEs in fcitx-configuration


Learn the new shortcut keys that are different from Windows

One default out of the box that’s hard to guess is moving from page to page. It used to be PageUp/PageDown but CAP follows fcitx’s global configuration moving between pages, which is the lower case of +/- keys which is basically =/- because + is upper case while it was intended to be lower case. I know, this is confusing!

IME switching follows the OLD windows shortcut keys (like Windows 98 and XP days), which

  • Ctrl+Space means turning IME on/off (global sense),
  • Ctrl+Shift changes IME languages (newer Windows use Alt+Shift by default)
  • Shift to temporarily disable/enable the IME (i.e. English mode) but stay within the language state

More customizations to get it closer to Windows IME behavior

CAP follows the global config settings in fcitx, unlike mozc (Japanese IME) which sometimes play by its own rules which behaves similar to its Windows’ counterpart.

If you are used to CPIME’s vertical lists, you can change it in ‘Appearance’ tab.


CAP candidate selection quirks when used with mozc (bug?)

For some reason, when both CAP and mozc are freshly installed, the first time you use the candidate list in mozc by selecting space/tab, the candidate list will disappear!

I installed and uninstalled fcitx, mozc and CAP and realized narrowed the bug to this reproducible path. My suspicion is that there’s a setting regarding the candidate selection shortcut (usually by ‘1’~’9’+’0′) parameter state that’s not exposed in fcitx-configuration that was being changed my mozc. And this guess puts me closer as I was able to play around with mozc’s config and found a candidate selection shortcut option

Note that mozc only has a max shortcut of 9 items (instead of 10, that means the ‘0’ key is not available as shortcut key) despite fcitx-configuration’s Global Config has a different idea (which CAP can use the 10th key, aka ‘0’ as candidate selection shortcut)

I noticed that after switching ‘1’-‘9’ to ‘a’-‘l’ (or no shortcut) mode, activate it in mozc by using space key to expand selection (this is necessary or the change won’t happen), I get the ‘1’-‘9’+’0’ candidate selection shortcut when I go back to CAP. I also noticed if I messed with the maximum number of suggestions in mozc a few times, I can get into an undefined state in CAP where it shows the candidate selection shortcut for the first few but not the rest, such as this:

I also noticed CAP has one consistent bug that the candidate selection (not just the keys) ONLY WORKS FOR THE FIRST PAGE! I tried to use the candidate selection shortcuts or click on the character with mouse for subsequent pages, it only commit the current word choice disregarding the selection!

Loading

Taking screenshots in Ubuntu with Flameshot

I’m spoiled by Greeshot in Windows and I find the gnome-screenshot that came with Cinnamon lacking.

With Greenshot, Print Screen key by default selects an area (which should be the most versatile and productive mode which should be prioritized with least complicated keys) and prompts you on whether you want to copy to clipboard or save a file.

Gnome-screenshot went with the most traditional behavior where Print Screen keys captures the entire screen AND it saves to a default file in ~/Pictures folder with timestamped file names, which I don’t want (I prefer copying to the clipboard).

So natively in gnome, Ctrl is the modifier for copying to clipboard. Shift is the modifier for selecting a section of the screen. So the most common operation I want to do became a bit of finger gymnastics Ctrl+Shift+Print and there are no immediate access to image editor like Greenshot (don’t even bother with GIMP, it’s slow to load and convoluted).

I discovered a much neater app called Flameshot. It has a much quicker design that you can select a section of a screen and do the most common screenshot edits on the fly and copy to the clipboard or save to files, even faster than Greenshot, which opens the captured image into a separate editor and you have to click file->copy-to-clipboard after edits!

Turns out that Flameshot does not use the native gnome’s print screen categories,

so establishing the shortcut has to be done as just a simple shortcut for running the command:

/usr/bin/flameshot gui

and I chose to use Win/Super+Print for Flameshot

Again, the command is /usr/bin/flameshot gui (or where flameshot is located if it’s not under /usr/bin)

Loading

Ubuntu cannot ping Windows hostnames out of the box (resolving NETBIOS announcements)

Out of the box, Ubuntu cannot resolve hostnames announced by Windows out of the box.

The internet had many solutions from ditching NETBIOS (winbind, wins) but it involves replacing systemd-resolved with the old NetworkManager (systemd-resolved was an extra level of indirection to break VPN ties), which I illustrated in this now deprecated blog post.

Having router assign host names you specify often included a local domain name (must not choose one that conflicts with the internet) such as local or lan. So the computers are accessed in the format of myPC.local or myPC.lan depending on the local domain name you picked. However, it doesn’t take advantage of the hostname announced by Windows computers.

I decided to give the NETBIOS service a second research today and found the missing link to the common solution of installing winbind and adding wins entry to host search order in /etc/nsswitch.conf (you can put it at the end or earlier if you want). I put it at the end as I wanted it to be the last resort

hosts:          files {a bunch of things depending on your system} wins 

Of course having a wins entry in the hosts search order involves installing winbind make sure the winbind service is running

sudo apt install winbind

The missing piece is editing /etc/samba/smb.conf to inject a name resolve order list after installing samba and winbind:

name resolve order = wins lmhosts bcast

You will need to install samba package first if you haven’t already installed it (for sharing folders with Windows)

sudo apt install samba

The post said the name resolve order section was commented out, but in newer version of samba, the line is simply non-existent. You’ll have to add it somewhere in /etc/samba/smb.conf, I chose to put it right at the beginning of [global] section.

Restart the services after editing to reflect the changes and you can start pinging!

sudo systemctl restart nmbd smbd winbind

So in the process above (installing samba and winbind and editing nsswitch.conf), you’ve also enabled linux to announce its hostname to Windows, which I’ve discussed in this blog post.

So to summarize the concepts,

  1. You need to install winbind to add wins to host search list in nsswitch.conf, but it doesn’t do you anything yet!
  2. Once you installed samba, your linux computer start announcing its hostname to Windows computers
  3. To be able to use the hostnames announced by Windows, i.e. the other direction, you’ll need to add the name resolve order line to smb.conf (samba config file) and restart samba and winbind.

Loading