Category: Windows

Deltacopy is a GUI wrapper around rsync, a feature-packed tool to copy files locally AND remotely, AND differentially (automatically figure out the parts that are different and resend. Excellent for repair) through hash comparisons. For non-programmers, hash is a unique ID computed for a chunk of data that are expected to change wildly even at the slightest data/file change/corruption).

Deltacopy is very useful if you just want to do the basic stuff and not know the rsync syntax and switch combinations off the top of your head. It also provides a windows port of rsync based on Cygwin (a tiny Linux runtime environment for windows). This is the only free alternative to cwRsync, a paid Windows port of rsync.

rsync is a Swiss Army Knife that can also work from one local path to another. Deltacopy is intended for remote file transfer.

Deltacopy server is basically this:

rsync --daemon 

However in Windows, since it’s cygwin, it’s looking for linux’s /etc/rsyncd.conf by default if you do not specify the config file through --config switch.

Deltacopy client basically help you generate the command to transfer files. Most of the features are done through right-click (context) menu, not toolbar or pull-down menus, which might confuse some people. You set up your tasks as Profile, which can be scheduled (the bottom panel) or executed immediately by right clicking on the profile:

Run is pushing file to the server, Restore is pulling files from the server. Run Now and Restore are for executing the command (aka task) immediately. You can peek into what it generated by right-clicking on the profile and choose “Display Run/Restore Command”. First time users might not be able to find it since the only place to access it is through context menus.

There are some tricky parts (gotchas) for specifying the files/folders to copy. First of all, even though you use Add Folder/Add Files button for entries

Basically you can make a (source, destination) pair by modifying the selection and target path. It’s just passed onto rsync command verbatim. The target path is relative to the virtual directory set on he server (see Deltacopy Server’s directory)

The destination path which is endowed with the branch folder name (one-level). In other words, if your source is C:/foo/bar, Deltacopy by default set the destination to /bar instead of /. This is probably to avoid the temptation of lumping all contents in the same remote destination root. If you just want to simply lay the files at the root virtual folder at the destination (my most common use case), you’ll have to edit and clear out the (relative) destination path.

As for the source, the author of rsync chose to do it the logical (more conservative) way but not intuitive way: by default it reconstruct the source folder’s FULL path structure at the destination! For example if you intend to copy everything under C:\foo over, the destination will create {destination root}\foo in the process and put everything under it instead of directly at {destination root}. The design choice was supposed to prevent accidental overwrites as multiple source subfolders try to write over each other with the same names at the destination.

Luckily, there’s a way around it! See man pages for -R –relative: Put a dot (.) at the place where the relative path starts! For example, the source is C:\foo\bar\baz and you do not want /foo to be created at the destination and want it to start with /bar instead. You should enter C:\foo\.\bar\baz as source. Everything the left of the dot (refers to self-folder) are stripped from the destination path structure.

ACL support for Windows sucks because rsync lives on cygwin, which has POSIX (unix/linux) type of permissions/ACL.

https://unix.stackexchange.com/questions/547275/how-do-i-use-rsync-to-reliably-transfer-permissions-acls-when-copying-from-ntf

In my opinion, the best way to go about it is to not transfer ACLs from the source and follow the preexisting ACLs at the destination. I’d also leave the groups and owners alone (inherit at destination) as well I might not be on the same active directory (or workgroup user management) as the destination computer so accounts with the same name might not be actually the same accounts.

--no-p --no-g --no-o

–no-{command} is the complement prefix that does the opposite of the -{command}, so the above means skipping -p (perms/permissions), -g (group), -o (owner) and make sure it has full permissions for everybody.

Sometimes a remote path can be mistaken as a relative local path with the hostname/IP address as the folder name if there’s no username. Start it with rsync:// as the URL scheme and the syntax is like ftp:// as far as username is concerned.

Deltacopy protects the source and destination paths with double quotes (“). It’s a good practice that we should do it even with direct rsync calls.

 34 total views

I’m trying to cross compile my router’s firmware as I made a few edits override the update DDNS update frequency. Turns out it doesn’t work on the latest Linux so I’d need to run an older Ubuntu just to keep it happy.

RANT: Package servers keeps pulling to rug on outdated linux is frustrating. Very often developers didn’t make a whole installer it so we are often wedged between downloading a package at the mercy of its availability from package managers and their servers or compiling the damn source code!

With the promise that Qemu might have less overhead than Hyper-V or VirtualBox (indeed it observably is), I tried installing Qemu on Windows host and it turned out to be a frustrating nightmare.

RANT: Linux is not free. The geniuses did the most sophisticated work for free but users pay time and energy cleaning after them (aka a support network dealing with daily frustrations) to made these inventions useable. There’s a company that does the clean up to make BSD (same umbrella as Linux/Unix) useable and made a lot of money: it’s called Apple Computers since Steve Jobs return.

qemu is just the core components. System integration (simplifying common use cases) are practically non-existent. Think of them as the one who produced an ASIC (chip) and the end-user happens to be the application engineers. There’s a few tutorials on qemu Linux hosts for moderately complex scenarios, but you are pretty much on your own trying to piece it altogether for Windows because there are some conceptual and terminology differences. The man page --help for the qemu’s Windows host’s VM engine was blindly copied from the Linux hosts counterpart, so it tells you about qemu-bridge-helper which is missing.

I stupidly went down the rabbit hole and drained my time on qemu. So I documented the quirks to help the next poor sap who has to get qemu running on Windows 10 host efficiently over Bridged-Adapter (VirtualBox lingo) networking mode.

• Preparation work to get HAXM accelerator set up
  • Release VT-d (hardware assisted virtualizations) so HAXM can acquire it
    • You’ll need to remove Hyper-V completely as it will hoard VT-d’s control
      • Windows Sandbox and Windows Subsystem for Linux (WSL2) uses Hyper-V. If you just unchecked Hyper-V in Windows Optional Features leaving any of these 2 on, Hyper-V is still active (it only removes the icons)
  • HAXM v7.6.6 not recognized by qemu on clean install. Install v7.6.5 first, then remove it and install v7.6.6. Likely they forgot a step in v7.6.6’s installer
  • Turn on optimization by: -accel hax
• Command line qemu engine
  • qemu-system-{architecture name}.exe is what runs the show
  • qemu-system-{architecture name}w.exe is the silent version of the above engine. Won’t give you a clue if something fails (like invalid parameters)
  • qemu-img create -f {format such as vhd/qcow2} {hard drive image name} {size like 10G}
• QtEmu sucks, and they lack any better GUIs out there!
  • It’s basically a rudimentary command line’s GUI wrapper
  • It only has user mode (SLIRP) networking (default)
  • It’s not maintained actively so it doesn’t keep up with the parameter syntax changes (i.e. can generate invalid combinations)
  • Since it uses the silent (with a w suffix) engine, likely to avoid a lingering command window, it also won’t tell you shit and why if something fails. It just ignores you when you press the start button unless all the stars align (you got everything right)
• Basic command line parameters
  • Set aside 10G for the VM: -m 10G
  • 1 core if unspecified. Number of available threads (in hyper-threaded system) show up as # of processors. It’s referring to logical processors, not physical cores.
    • Windows: -smp %NUMBER_OF_PROCESSORS%
    • Linux: -smp $(nproc)
  • Attach virtual hard drive: -hda {virtual hard drive file name}
  • Attach optical drive (iso): -cdrom {iso file}

I typically want Bridged-Adapter option from VirtualBox, which means the virtual NIC plugs into the same router as the host and just appears as another computer on the same network as host. This is broken into a few components in qemu and you have to manage them separately. Great for learning about how Bridged-Adapter really works, but a lot of swearwords coming from people who just want to get basic things done.

Networking in QEMU is another can of worms if you deviate from the default SLIRP (user mode). I figured out how to work it, but the network bridge is faulty and it keeps crashing my windows with BSOD on bridge.sys with varying error tag. I have short glimpse of it working if I move very fast. Looks like the TAP driver is corrupting the memory as the bridge became very erratic that I see error messages deleting it and have persistent BSOD when the bridge starts after the VM hanged at the TAP bridge on boot.

I listed the steps below to show what should have been done to get the Bridge-Adapter (VirtualBox) equivalent function if there are no bugs in the software, but hell I’m throwing qemu for Windows to trash as it’s half-baked.

First, of all, you need to install OpenVPN to steal its TAP-Win32 virtual network card. It’s not VMware or Virtualbox that it’s part of the package. Qemu didn’t care to tightly integrate or test this driver properly.

Then you’ll need to bridge the “TAP-Windows Adapter (V#) for OpenVPN” with the network interface you want it to piggy back on.

The name of the TAP adapter is what you enter as ifname= parameter of the tap interface in qemu command line. You have to tell qemu which specifically interface you want to engage in. I named the virtual network card as ‘TAP’ above. After bridging it looks like this:

You are not done yet! The bridged network (seen as one logical interface) is confused and it won’t be able to configure with your physical network card’s DHCP client. You’ll have to go to the properties of the Network Bridge and configure the IPv4 with static IP.

You can use ipconfig /all to find out the relevant adapters acquired DHCP settings and enter it as static IP. Coordinate with the network administrator (can be yourself) to make sure you own that IP address so you won’t run into IP conflict if you reboot and somebody took your IP.

After these are all set up the parameter to add to qemu call is:

-nic tap,ifname=TAP

There are complicated settings like -net nic and -netdev -device. These are old ways to do it and have bloated abstractions. -nic switch combined them into one switch.

Then welcome to the world of Windows 10 bridge.sys crashing frequently and you might get a short window of opportunity that it boots and ifconfig acquire the IP address settings from your router (or network the physical adapter is on)’s DHCP server.

It’s like a damn research project finding out something is technically feasible but definitely not ready for production. Welcome to FOSS jungle!

Postscript: I put Hyper-V back and realized it’s insanely slow with Linux Mint as it does not support hardware graphics acceleration. It’s night and day of a difference. Qemu is fast, but it crashes on Windows 10 if I bridge the adapters!

 102 total views

My email service provider has recently pulled the plug on TLS 1.0/1.1 support as they reached end-of-life. This means old email clients not written for TLS 1.2 and above will not work when it tries to connect to the server with SSL support!

Google did this in 2014 but offered a compatibility option called “allow less secure clients”. Back then I didn’t know it means TLS 1.0/1.1 until I learned it the hard way when my shared hosting email provider pulled the plug on the old TLS protocols and I scrambled to figure out my email stopped working with cryptic IMAP errors (like suggesting my computer might be lacking memory, which is not true).

Stunnel config that needs to be changed from defaults. If stunnel was installed by entware (opkg), the config file is in /opt/etc/stunnel/stunnel.conf.

1. Disable (comment out) drop privileges
2. Remove the [dummy] section since we are going to set up sections for each (server, port) pair. stunnel won’t start without any port forwarding sections.
3. It already has an [imap] section that’s commented out. Change the local port number and the target server url:port to your liking. Do [pop] if you use POP3 email instead of SMTP
4. Do the same by adding a [smtp] section for outgoing email

You might need to comment out

; You might need to comment out this drop privilege (3rd line in default conf file)
;setgid = nogroup

; Examples of forward sections with a '50' prefix to SSL port numbers to 'convert it to non-SSL' heading to stunnel
[imap]
client = yes
accept = 50993
connect = target_server:993


[smtp]
client = yes
accept = 50465
connect = target_server:465

Can look at the log by just executing stunnel. Use Ctrl+C to quit monitoring the logs.

Of course you want to make sure the stunnel service/server is always started on boot. If you are using entware (or jffs scripts) for your router, add the call to stunnel to /jffs/scripts/post-mount and make sure you set the script to executable so it’ll run:

#!/bin/sh
...
stunnel

Note that it’s post-mount because entware packages are installed on persistent storage (like USB drive or SD card in your router) that needs to be mounted before the files can even be read.

Remember to go to your old email client and change the email server address to computer running stunnel service (can be the same computer as the client, a raspberry pi, or a router). Unless you are managing a company with many old email clients, I recommend installing and configuring stunnel on the same OS which the old mail program runs instead of relying on a centralized router to manage it because it’d be very confusing if the centralized stunnel service is down and you don’t get any warnings that are not cryptic (as mentioned at the beginning of this article).

 90 total views