Monthly Archives: April 2019

Getting pyinstaller 3.4 to work with Python 3.7

Posted on by
Reply

Python is an excellent language, but given that it’s free, it also comes with a lot of conspicuous loose-ends that you will not expect in commercially supported platforms like MATLAB.

Don’t expect everything to work right out of the box in Python. Everything is like 98% there, with the last 2% frustrate the heck out of you when you are rushing to get from point A to point B and you have to iron out a few dozen kinks before you can really start working.

When I tried use pyinstaller (v3.4) to compile my Python (v3.7) program into an executable, I ended up having to jump through a bunch of hoops:

  • pip install pyinstaller gives: 
    ModuleNotFoundError: No module named 'cffi'
  • Then I looked up and installed cffi 
    pip install cffi
  • After the dependency was addressed manually (it shouldn’t )  pip install pyinstaller worked
  • Then I tried to compile my first Python executable with pyinstaller, and I got this exception: 
    File "C:\Python37\lib\site-packages\win32ctypes\core\cffi\_advapi32.py", line 198

        ^
    SyntaxError: invalid syntax
  • I searched the exact string and learned that pyinstaller (v3.4) is not ready for Python 3.7 yet! How come pip installer didn’t check for it? I opened up the offending file and looked for line 198 and saw this: 
    c_creds.CredentialBlobSize = \

    ffi.sizeof(blob_data) - ffi.sizeof('wchar_t')

    It’s a freaking line continuation character \ (actually the extraneous CR before CRLF) that rooster-blocked it.

  • I just deleted the line continuation and merged the two lines, and saved _advapi32.py, then I was able to compile my Python v3.7 code (using pyinstaller 3.4) with no issues.

This is not something you’ll experience as a MATLAB user. The same company, TMW, wrote the MATLAB compiler as well as the rest. The toolbox/packages are released together in one piece so breaking changes that causes failure for the most obvious use case are caught before they get out of the door.

Another example of breaking changes that I ran into: ipdb does not allow you to move cursor backward.

Again, this is the cost associated with free software and access to the latest updates and new features without waiting for April/October (it’s the MATLAB regular release cycle). If hassle and the extra engineering time far exceed licensing MATLAB licensing costs, MATLAB is a better choice, especially if software is just a chore to get your company from point A to point B, and you are willing to pay big bucks to get there quickly and reliably.

Even with free software on the table, your platform choice is always determined by:

  • how much your time is worth wrestling problems
  • how much flexibility do you need (for customizing to your needs)
  • how much you are willing to pay for the licenses and support

In any case, Python community did good work. Please consider sponsoring PyInstaller and PSF if you profit immensely from their work. But if your company already paid for MATLAB and has real MATLAB experts on it*, don’t switch to Python just for the sake of chasing the trendy thing. You will loose time. Lots of time! Precious engineering hours! Over really stupid things like the ones above that is totally not your fault!  It’s better to dedicate a few days dealing with MATLAB-Python interface than wasting months over the clumsiness of Python if you have a long, complex workflow

Python’s largest value over MATLAB is that there’s a function developed for nearly every less than common situations (like all different variations of path and file management tools) so you don’t need MATLAB File Exchange to fill in the gap, but the downside is that Python, just like the rest of FOSS world (or Tektronix), has absolutely no sense of user experience studies: common operations are tucked under clumsy maneuvers such as making user jump through hoops to run a Python script in the interpreter (import do not execute the script after its cached, execfile was lile eval in MATLAB so they made it hell, or you need to import a freaking library like runpy to do something basic like this. I understand they cannot Why can’t they just add a root-level function like run() and call it a day? Python was just too eager to protect the namespace, and this comes at the expense of adding a lot of stupid maneuvers for really basic bread-and-butter stuff). Python is productive in the sense that there are many sophisticated features that are readily available that you don’t have to write your own library for the advanced cases, but the edge is like 1% of the use cases which I invest the time when I run into it: it’s not worth making 99% of my workflow miserable!

If you are into test instruments, MATLAB is like HP/Agilent/Keysight’s user interface while Python is like Tektronix’s UI where you’ll need to get to 4 levels of context menus to measure a freaking peak-to-peak voltage! Spending your smart people’s time to figure these stupid shit that came from poor UX (user experience) design is a poor investment! At least Python is free and has more advanced features; Tek charges similar to HP and offer nearly the same features yet it has absolutely no respect for the users’ time: they think all people have Stockholm syndrome after going through their steep learning curve for absolutely no benefits.

Why pay a junior engineer $50/hr for a whole week to go through the learning curve, plus months of maintenance work because they’ve used clumsy tools and clumsy methods, when I can code the same thing up in MATLAB in one hour for $500 and give you one page of easy to read code that you likely don’t have to debug through it down the line? If I designed a moderately complex system, most often I can get to the root cause of any bugs in around 15 minutes: the reason is that I spent my time thinking through the abstractions instead of diving into writing for-loops to hack through any common scenarios without first researching for the neatest mechanism. This way I don’t spend an afternoon writing for loops extracting a strings in the object properties of a categorical data type (a efficient way of marking complex repeating data by indexing unique items) and check if it’s really correct when I could have done some research and realize I can just cast it to a cell/monad/array of strings. The value of my experiences comes in knowing the right words and common lingo in many forms to describe many problems, like spotting a low level database operation implementation is just a RLE (run length encoding) in disguise. The hardest part of using search engines to find what you want quickly is knowing the right keywords. If you don’t know the concepts, you’d have jump into reinventing the wheel poorly when you could have used off-the-shelf proven code had you been able to frame the problem succinctly.

* Don’t be mislead by how many years of MATLAB one has! Majority of people those who claimed decades of MATLAB under the belt didn’t really learn the advanced concepts through studying documentation, MATLAB blogs or get training to take it to the next level of super-productivity! You can tell if they have a lot of for-loops for situations that doesn’t absolutely need it (memory saving, injecting elements to LHS assignment, parallel computing, not loading too many files at the same time, drawing graphics (order matters), big tasks that needs to be resumed if it throws an error) and using cells everywhere when they should have used structs or more modern types such as table(). Most of the modern, super-productive MATLAB language constructs happens since 2013, so I can safely say that MATLAB experiences accumulated before that aren’t too valuable for developing business logic part of the code (might be useful if you are writing tools and lower level MATLAB code like those in TMW).

Loading

Picking an IDE for Python

Posted on by
Reply

The native features in MATLAB are often very good most of the time, as I’ve yet to hear anybody spending time to shop for a IDE outside the official one.

Atom has the feel of Maple/MathCAD, and Jupyter Notebook has the feel of Mathematica. Spyder feels like MATLAB the most, but it’s hugely primitive.

IDLE is more miserable than a command prompt. It doesn’t even have the decency to recall command history with up arrow. It’s like freaking DOS before loading doskey.com. Not to mention that single clicking on the window won’t set the cursor to the active command line, which you have to scroll all the way down to click on the bottom line. WTF! I’d rather use the command prompt and give up meaningless syntax coloring.

IPython (in Spyder) is unbearably slow (compare to MATLAB’s editor which I consider slow to the extent that it’s marginally bearable for the interactive features it offers), but at least usable unlike IDLE, and most importantly the output display is pprint (pretty printer) formatted so it’s legible. Just type locals() and see what kind of sh*t Python spits out in IDLE/cmd.exe and you’ll see what I meant.

I simply cannot live without who/whos provided in IPython, but I still don’t like it showing the accessible functions/modules along with the variables (I know, Python doesn’t tell them apart). Nonetheless it’s still weak because these are automagics that doesn’t return the results as Python data (just print). Spyder’s ‘variable explorer’ is the only place I can find that doesn’t include loaded functions/modules. Python should have provided facilities to get the user-introduced variables exclusively and leave the modules to a different function like MATLAB’s import command that shows imported packages/classes.

However, pretty printer doesn’t even come close to MATLAB in terms of the amount of dirty work disp() did to format the text to make it easy to read. Keys in the dictionary shown in pretty printer in Python are not right-aligned like MATLAB struct so we can easily tell keys and values apart. For example:

MATLAB struct shows:
          name: 'S'
          size: [9 1]
         bytes: 7765
         class: 'struct'
        global: 0
        sparse: 0
       complex: 0
       nesting: [1×1 struct]
    persistent: 0

Python with Pretty Printer shows:
{'__name__': '__main__',
 '__doc__': 'Automatically created module for IPython interactive environment',
 '__package__': None,
 '__loader__': None,
 '__spec__': None,
 '__builtin__': <module 'builtins' (built-in)>,
 '__builtins__': <module 'builtins' (built-in)>,
 '_ih': ['', 'locals()'],
 '_oh': {},
 '_dh': ['C:\\Users\\Administrator'],
 'In': ['', 'locals()'],
 'Out': {},
 'get_ipython': <bound method InteractiveShell.get_ipython of <ipykernel.zmqshell.ZMQInteractiveShell object at 0x00000000059B7828>>,
 'exit': <IPython.core.autocall.ZMQExitAutocall at 0x5a3b198>,
 'quit': <IPython.core.autocall.ZMQExitAutocall at 0x5a3b198>,
 '_': '',
 '__': '',
 '___': '',
 '_i': '',
 '_ii': '',
 '_iii': '',
 '_i1': 'locals()'}

I often convert things to MATLAB dataset() because the disp() method is excellent, such as struct2dataset(ver()). table/disp() is nice, but I think they overdid it by defaulting to fancy rich-text that bold the header, which makes it a magnitude of orders slower, and it’s not using the limited visual space effectively to show more data. Python still has a lot more to do in the user-friendly department.

Loading

Implicit ways to store data in a program

Posted on by
Reply

The most obvious way to store data is in plain data structures such as arrays and queues, or even a hashtable, but don’t forget these implicit ones:

  • Call stack. As the name say, it’s a stack data structure. It saves the local variables before making another function call. This is often exploited in recursion to avoid passing around an explicit data structure
  • Closures. What closure means is that when you create an anonymous function, the variables involved (other than the arguments) that is saved along (captured) with the function object you created. This can be exploited to make forward iterators or generators
  • Functions. You can make a function that does nothing other than returning a certain piece of data. It’s an excellent way to make avoid the overhead of managing (reading, updating promptly) a config file. Works best when your programming language requires so little typing to specify data (such as MATLAB) that your code is almost as short as a plain text config file.

Loading

不是懶趴是卵脬，不是雞是膣

Posted on by
Reply

 

粵 國 台 日
屄/閪 屄 膣 (誤讀 雞) 膣
屌/? 姦 (誤讀 幹) 肏 (誤讀 操) 姦
?/?，㞗/?，杘/?，
脧脧/JerJer		 屌 ?/卵鳥 (誤讀 懶鳥) チンポ　(珍宝)
春袋 蛋蛋 卵脬 きんたま（金玉）
儸柚 屁股 尻(kha)川(tshng) おしり　(お尻)

 

 

真趣味的台灣俗諺(尻川): http://www.tma.tw/ltk/107611213.pdf

Loading

Obscure differences between Kanji and Chinese characters

Posted on by
Reply

People who already know Chinese characters are often said to have the advantage of being able to pick up Japanese quickly. However, to learn it properly, in addition to the  difference between infix (English, Chinese) and reverse polish (Japanese) notations, it also comes with quite a bit of baggage. It’s the differences that requires work to observe, such as:

  • some made up ‘Chinese’ characters (和製漢語),
  • some are written slightly differently, including artistic variations
  • some has a completely different meaning,
  • some has opposite preferences for using which character in the pair when simplifying
  • and some has drastically different overtones despite they technically mean the same thing
  • the mixture of simplified and traditional characters, occasionally a character written like simplified Chinese means something totally different from traditional Chinese, such as 机（つくえ）which means desk vs 機（キ）which means machines or chances depending on the context.
  • the roles of historical and modern writings are randomly reversed

学習 is a good example. Modern Chinese considers 学 to be more colloquial (e.g. 学武功）and 習 to be more formal (e.g. 習武). Japanese is the other way round for 学ぶ and 習う。学ぶ has a more serious tone.

Actually, the kinds of variations mentioned above applies to regional differences in Chinese languages (such as Taiwanese, Cantonese and Mandarin). Most places agree to write Chinese in a way that can be read directly using Mandarin so that we can at least communicate on paper. So as time goes by, we lost the ability to write in Taiwanese and Cantonese. I hope it’ll change as both dialects are very colorful. Re-expressing them in Mandarin will take away all the flavors in them.

It’s evident that humans can pick up more than one language, so there is no reason to compromise dialects in the process of standardization. People advocating to kill other languages are simpletons who believe in the kind of logic supporting a competitive system: you find ways to make your peers do worse to stay ahead, instead of improving yourself.

Different regions occasionally have different preferences for character order in phrases. Basically we have to watch out for all kinds of combinations. Like 介紹 is used in the same order for Taiwanese/Cantonese/Mandarin to mean introduction, but it’s reversed 紹介(しょうかい) in Japanese. To make it a total mindfuck, Mandarin sticks with 客人 for guests, which is used the same way as Japanese’s 客人(きゃくにん), Taiwanese mostly says 人客, while Cantonese uses both with slight overtones: 客人 is usually used as a particular noun (e.g. 呢位客人) while 人客 is often used as a collective noun (e.g. 人客嚟齊未?), most likely because 客人 sounds more formal than 人客.

Putting traditional and simplified Chinese aside, different regions have different preferences for Chinese characters. I couldn’t tell the difference between traditional Chinese characters used in Hongkong/Macau (港澳繁體) and Taiwan (台灣正體) on Wikipedia, and later learned that it was because I’ve been randomly mixing both all along and nobody ever pointed it out.

裏/着 (Hongkong) vs 裡/著 (Taiwan) are good examples. For these two, modern Japanese sided with Hongkong in the character choices for 裏(うら) and 着(ちゃく). On the other hand, 峰(みね) in Japanese sided with the Taiwanese’s preferred writing 峰, while the 峯 is the ‘officially’ preferred writing in Hongkong.

I remember writing 峰 most of the time even when I was a kid and only used 峯 for names that specifically calls for it. We respect the original writing for names. This is the similar situation as in Japanese: 沢(さわ/たく) is used in most cases and reserve 澤(サワ) for names that specifically requests to be written in this form. The only difference is that I used the official character 峯 exclusively for names, while using the off-label 峰 for the rest.

Speaking of names, there are some similar-looking characters that has the same Japanese sound (かな) but are actually different in both writing and meaning. 斉藤 and 斎藤 are different, but they are easily confused for native Japanese speakers who don’t have any Chinese language background. Here’s the table for comparison:

齊／齐・斉 齋／齋・斎
Meaning Gathered, organized Plain, house, recitations
Cantonese chai (cai4) jaai (zaai1)
Taiwanese tsè tsai
Mandarin qi2 zhai1
Japanese (音読み：さい) 斉しい・等しく いつき・（潔斎）物忌み

The bottom line is: as language evolves, different regions have different preferences about what can they be sloppy about and what they must be meticulous about. They also reorder/tweak things to make them flow smoothly with their dialect. This means traps for for those learning a new language that are close to what they’ve already mastered.

I came across a document called 常用漢字表 released by the Agency for Cultural Affairs (文化庁) that explains all the quirks of Kanji that was carefully collecting on my own while taking the classes. Wish I had it back in the days. Here’s the link, but I also saved a local copy of 常用漢字表 just in case if their website moves around in the future.

Loading