Dart Language: late binding in lambda (no capture syntax)

Dart’s documentation at the time of writing (2021-10-27) is not as detailed as I hoped for so I don’t know if the Lambda (anonymous function) is late-binding or early-binding.

For those who don’t know what late/early-binding is:

Bindingearlylate
Expressionclosed (closure)open (not a closure)
Self-containedyesno
Free symbols
(relevant variables
in outer workspace)
bound/captured at creation
(no free symbols)
left untouched until evaluated/executed
(has free symbols)
Snapshotduring creationduring execution

Late-binding is almost a always a gotcha as it’s not natural. Makes great Python interview problems since the combination of ‘everything is a reference‘ and ‘reference to unnamed objects‘ spells trouble with lazy evaluation.

I’d say unlike MATLAB, which has a more mature thinking that is not willing to trade a slow-but-right program for a fast-but-wrong program, Python tries to squeeze all the new cool conceptual gadgets without worrying about how to avoid certain toxic combinations.

I wrote a small program to find out Dart’s lambda is late-bound just like Python:

void main()
{
  int y=3;
    var f = (int x) => y+x;	
    // can also write it as 
    // f(int x) => y+x;
  y=1000;	

  print( f(10) );
}
The result showed 1010, which means Dart use open lambdas (late-binding)

LanguageBinding rules for lambda (anonymous functions)
DartLate binding
Need partial application (discussed below) to enforce early binding.
PythonLate binding.
Can capture (bind early) by endowing running (free) variables (symbols) with defaults based on workspace variables
C++11Early binding (no access to caller workspace variables not captured within the lambda)
MATLABEarly binding (universal snapshot, like [=] capture with C++11 lambdas)

I tried to see if there’s a ‘capture’ syntax in Dart like in C+11 but I couldn’t find any.

I also tried to see if I can endow a free symbol with a default (using an outer workspace variable) in Dart, but the answer is no because unlike Python, default value expressions are required to be constexpr (literals or variables that cannot change during runtime) in Dart.

So far the only way to do early binding is with Partial Application (currying is different: it’s doing partial application of multiple variables by breaking it into a cascade of function compositions when you partially substitute one free variable/symbol at a time):

  1. Add an extra outer function layer (I call it the capture/binder wrapper) putting all free variables you want to bind as input arguments (which shadows the variables names outside the lambda expression if you chose not to rename the parameter variables to avoid conceptual confusion),
  2. then immediately EVALUATE the wrapper with the with the outer workspace variables (free variables) to capture them into the functor (closed lambda, or simply closure) which binder wrapper spits out.

Note I used a different style of lambda syntax in the partial application code example below

void main()
{
  var y=3;
    f(int x) => y+x;
	
    // Partial Application
    g(int w) => (int x) => w+x;	
    // Meat: EVALUATING g() AT y saves the content of y in h()
    var h = g(y);
	
  y=1000;	

  // y is free in f 
  print( f(10) );	

  // y is not involved in g until now (y=1000)	
  print( g(y)(10) );		

  // y is bounded for h when it was still y=3
  print( h(10) );
}

Only h() captures y when it’s still 3. The snapshot won’t happen if you cascade it with other lambda (since the y remains free as they show up in the lambda expression).

You MUST evaluate it at y at the instance you want y captured. In other words, you can defer capturing y until later points in your code, though most likely you’d want to do it right after the lambda expression was declared.


As a side note, I could have used ‘y’ instead of ‘w’ as parameter in the partial application statement

g(int y) => (int x) => y + x

but the ‘y‘ inside g() has shadowed (no longer refers to) the ‘y‘ in the outer workspace!

What makes it confusing here is that quite a few authors think it’s helpful (mnemonics-wise) to use the free variable (outer scope) name you are going to inject into the dummy (argument) as the name of the dummy! This gives the false impression of the variable being free while it’s bounded (through feeding it as a parameter in the wrapper/outer function)!

Scoping rules is a nasty source of confusion in understanding lambda calculus so I decide to give it a different name ‘w‘. I’m generally dismissive of shadowing under any circumstances, to the extent that I find Python’s @ syntactic sugar for decorators shadowing the underlying function which could have been reused somewhere. See my rants here.

Notational ambiguity, even if resolvable, is NOT helpful at all here especially when there are so many level of abstractions squeezed into so few symbols! People jumping into learning the subject quickly for the first time should not have unnecessarily keep track of the obscure scoping rules in their head to resolve the ambiguities!

 12 total views

Modifying mutable (like bytearray) arguments’ data in Python functions Use slice assignment x[:] = ... to replace the entire contents. Dynamically typed language lets you shadow input arguments with a local variable!

I’d like to write a function to selectively modify lines read from a file handle and write it back. By default, lines are read as byte()  objects that are immutable, so I converted it to bytearray() instead so it can be modified because only a few lines meeting certain criteria needs to be changed.

When I try to refactor similar operation into a function, I was hoping to pass the mutable bytearray() as an argument and directly modify the caller’s content like in C++, given Python variables works LIKE reference binding.

I know bytearray.replace() does not modify the data in place, but instead outputs the modified line to a new variable. Normally, I can simply do this:

line = line.replace(b'\tCLASS', b'')

and the code will work. However, it doesn’t do anything when I try to pass it as an argument to a Python function (unless I return line as output). Although I am well aware that Python variables assignments to existing variables means orphaning the old data and re-purposing the label, the variable assignment behavior in Python requires careful thought when used in non-idiomatic situation.

In other words, I want this function to have side effects on the variable ‘line‘, but I wasn’t doing it right. This is a tempting mistake for people with a C/C++ background: in C/C++, it is not possible to shadow an input parameter even if we were to explicitly declare it, so the innocent assignment I did above has to modify the object in the caller (passed as a reference to the function) in C/C++, as if I did this directly in the caller.

However, in Python, variables do not need to be declared (aka, dynamically typed). This opens up the possibility of unwittingly shadowing the input parameters, which is what happened here. Mutable arguments on the stack still can be modified through the function, but when you assign a variable using ‘=’ operator, a new local variable with the name on the LHS is created, which shadows the input parameter.

This means the connection to the caller objects is lost during shadowing.

The correct way to do this is use slice assignment (which the logic/concept is very different despite the syntax is similar) to replace all the contents of the input variable with the output of bytearray.replace():

def remove_from_header_token_CLASS(tokens, line):
     # line is expected to be byte array (mutable)    
    try:
        column_CLASS = tokens.index(b'CLASS')
    except:
        column_CLASS = None
    else:
        line[:] = line.replace(b'\tCLASS', b'')  
                
    return column_CLASS

Since Python has a clear distinct concept of parameter variable (from local variable), trying to apply nonlocal keyword over it (in hopes to broaden the scope) will not parse/compile.


This is actually the same behavior as in MATLAB (dynamic typing) for the same reason that variables does not have to be declared like in C/C++ (static typing). In MATLAB, if you choose to have a handle object (which works like references), you can shadow the input argument by creating a local variable of the same name:

classdef DemoHandleClass < handle
    properties
        x = 3;
    end
end

function demo_shadowing()
    C = DemoHandleClass();
    f(C);
    
    disp(C.x)
end

function f(C)
    C = DemoHandleClass();  // Shadowing
    C.x = 14;
end

The above MATLAB program will display 14 without shadowing and 3 with shadowing (C became a new local variable that has nothing to do with the input argument C). MATLAB users rarely run into this because the language design heavily discourage side-effects: we are supposed to return the changed local variable to the caller. The only way to do side-effects in MATLAB is through handles (which you need to establish a class, which is clumsy). Technically you can write the data to external resources (e.g. file) and read it back. But guess what? Resources are accessed through handles, so there’s no escape.


Of course, there’s a better way to do so (MATLAB’s preferred way): return the modified object back to the caller as if they are immutable:

def remove_from_header_token_CLASS(tokens, line):
     # line DOES NOT HAVE TO BE MUTABLE    
    try:
        column_CLASS = tokens.index(b'CLASS')
    except:
        column_CLASS = None
    else:
        line = line.replace(b'\tCLASS', b'')  
                
    return column_CLASS, line

This is what I ultimately used (so I ended up not converting the byte lines to bytearray), given that Python’s tuple syntax make it easy to return multiple outputs like MATLAB. The call ended up looking like this:

column_SPL_CLASS, line = remove_from_header_token_CLASS(tokens, line)                

Nonetheless, I think there’s an important lesson to be learned for doing side-effects in dynamically typed languages. Maybe I’ll need this one day if I get an excuse to do something more complicated that genuinely requires side-effects.


In summary, variable assignments in most dynamically typed languages will shadow the input argument with a newly generated local variable instead of modifying the data in the original input argument. This implies that there function side-effects cannot be carried out through variable assignment.

The most common implication is: do not (equality) assign to a input variable to modify its contents in a dynamically typed language.

 448 total views

Spyder on MX Linux

This is another example that non-commercial (open-source) Linux/Python does not have a feel of a finished product: things break out of the box when installed fresh, in the most simple, expected ways, without any tweaks.

Again, don’t get me wrong, open-source free software are good stuff (more modern concepts and people working on it for free), but it’s never going to beat professional companies (like Microsoft/MATLAB) in how well-funded they are so they can maintain their software and the user experience using their profits. So far, users are still expected to put up with a bunch of unjustifiably unnecessary work to get to where they want to go with community-maintained software like Linux/Python.

This time I’m installing Spyder on MX Linux. Look at how many damn hoops I have to jump to get Spyder 3 to function properly there:

  1. I installed python3-spyder from MX Package Installer
  2. Spyder complained about missing rope on start
  3. Installed python-rope on MX Package Installer. The complaint still won’t go away
  4. I tried follow the instruction sudo pip3 install rope_py3k  and realized pip3 was not installed already with the Python that came with Spyder! (Didn’t have the problem with the Windows counterpart).
  5. Installed python3-pip from MX Package Installer.
  6. Came back and run pip3 install rope_py3k. It choked at "Command 'python  setup.oy egg_info' failed with error code 1 in /tmp/pip-build-0nnknjhi/rope-py3k". Again, known problem.
  7. Followed the solution in the comments pip3 install --upgrade setuptools
  8. Then come back and run pip3 install rope_py3k again. It says "Failed building wheel for rope-py3k" in between, but nonetheless I’ll try to move on since it says "Successfully installed rope-py3k-0.9.4.post1"

Then Spyder launch uneventfully.

These are not design decisions (sacrificing one quality for another), but inter-operability wrinkles that nobody are paid enough to do the grunt-work babysitting it. So if your business profits heavily from it, consider sponsoring the developers!


It’s also slightly annoying that the version of Spyder maitained in MX Linux’s most recent repository is a little older than what’s actually available (3.1.3+dfsg1-3 instead of 3.3.4).

At first I followed instructions to have PIP to update it: pip3 install -U spyder, but it doesn’t work. I got a lot of “failed building wheel for (package)” error.

I also realized the Python that came with it is 3.5, not the 3.7(.3) I had in Windows. I checked the MX package manager and indeed it stopped at 3.5. After some searching, I learned the base system package was frozen in 2016! MX Test Repo (at your own risk) has Python 3.7 though.

 757 total views

Watch out for ‘const’ method in Python

One thing I feel a little bit not quite as intuitive when I switch to Python is I constantly have to look up whether the method directly updates the contents or it’ll return a different object (of the same type) that I’ll have to overwrite the input variable myself.

An example would be strings and bytes object. replace() sounded like an updating method, but it’s actually a ‘const’ method (a term borrowed from C++ to say that the method does not have side-effects) that does not change the state of the object.

I initially thought this has to do with whether the object is immutable or not, but I tried it on bytearray objects (which is mutable), replace() behaves consistently with the identically named methods in other immutable objects (bytes object, string object): you’ll need to assign the output to self (basically bind the name to the temporary and throw away the original).

bts = b'test'
bts.replace('es', 'oas')       # dumps the output to workspace (can be accessed by _) and do nothing else
bts = bts.replace('es', 'oas') # actually updates bts

 

 513 total views

Anonymous Functions (MATLAB) vs Lambdas (Python) Anonymous Functions in MATLAB is closure while Lambdas in Python are not

Lambdas in Python does not play by the same rules as anonymous functions in MATLAB

  • MATLAB takes a snapshot of (capture) the workspace variables involved in the anonymous function AT the time the anonymous function handle is created, thus the captured values will live on afterwards (by definition a proper closure).
  • Lambda in Python is NOT closure! [EDIT: I’ll need to investigate the definition of closure more closely before I use the term here] The free variables involved in lambda expressions are simply read on-the-fly (aka, the last state) when the functor is executed.

It’s kind of a mixed love-and-hate situation for both. Either design choice will be confusing for some use cases. I was at first thrown off by MATLAB’s anonymous function’s full variable capture behavior, then after I get used to it, Python’s Lambda’s non-closure tripped me. Even in the official FAQ, it address the surprise that people are not getting what they expected creating lambdas in a for-loop.

To enable capture in Python, you assign the value you wanted to capture to a lambda input argument (aka, using a bound variable as an intermediary and initialize it with the free variable that needs to be captured), then use the intermediary in the expression. For example:

lambda: ser.close()      # does not capture 'ser'
lambda s=ser: s.close()  # 'ser' is captured by s.

I usually keep the usage of nested functions to the minimum, even in MATLAB, because effectively it’s kind of a compromised ‘global’ between nested levels, or a little bit like protected classes in C++. It breaks encapsulation (intentionally) for functions in your inner circle (nest).

It’s often useful for coding up GUI in MATLAB quick because you need to share access to the UI controls within the same group. For GUI that gets more complicated, I actually avoided nested functions altogether and used *appdata() to share UI object handles.

Functors of nested functions are closures in both MATLAB and Python! Only Lambdas in Python behave slightly differently.

 729 total views

Handling resources that needs to be cleaned up in Python and MATLAB Files, sockets, ports, etc.

Using try/catch to handle resource (files, ports, etc) cleanup is out of fashion in both MATLAB and Python. In MATLAB, it uses a very slick idea based on closures to let you sneak your cleanup function (as a function object) into an onCleanup object that will be executed (in destructor) when that object is cleaned up by being out of scope.

Python does not provide the same mechanism. Instead, it relies on the resource class (like file IO or PySerial) to implement as a Context Manager (has __enter__) and provide the cleanup in the manager’s __exit__ method. Then you use the with keyword with the returned resource object put after as keyword, like this:

with File('test.txt', 'w') as f:
    f.write('SPFCCsMfT!')

The body of with-block will not run (and therefore object f won’t be created) if the with-statement throws an exception. Unfortunately, it’s a fill-or-kill (or try/finally) instead of try/catch. So if the resource failed to open, resource object f is simply not created. No other clue is generated. This is what I hate about the with-statement. There are two ways to kind of get around it, but they are not reliable and might cause other bugs if you don’t keep track of the variable names in the local context:

  1. Check for the existence of the resource object
    if 'f' in dir(): del f   # Avoid name conflicts
    with File() as f
        print("Done");
    if not 'f' in dir():
        print('Cannot create file');
  2. Use the body code to indicate that the with-block is executed
    isSuccess = false     # Signaling
    with File() as f
        isSuccess = true
        print("Done')
    if not isSuccess
        print("Cannot create file");
    
    
  3. Back to the old way
    try:
        f = File();
    except:
        print("Cannot open file")
    else:
        cleanup_obj = onCleanup(lambda x = f: x.custom_cleanup())
        # run core code that uses resource f
    

Actually there’s another mess here. In PySerial, creating the Serial object with a wrong port string will throw an exception as well, which with-as statements cannot handle. Therefore you’ll need to do both:

try:
    ser = serial.Serial(dev_str)
except:
    print(dev_str + " not accessible (either the wrong port of something else opened it)");
else:
    with ser:
        # meat

If your resource initializer does not have context manager built in, and you want a quick-and-dirty solution (given your cleanup is a one-liner). Use my library (lang.py) that recreates onCleanup():

"""
@author: [2019-04-23] Hoi Wong 
"""
class onCleanup:
    '''make sure you 'capture' the lambdas by initializing an intermediate running variable
       e.g. lambda s=ser: s.close()
       lambda: ser.close() will NOT work as ser is not 'captured''''
    def __init__(self, functor):
        self.task = functor;
    def __del__(self):
        self.task()

Then you can use the old way without nesting try/except:

try:
    f = File()
except:
    print("Cannot open file")
else:
    cleanup_obj = onCleanup(lambda x = f: x.custom_cleanup())
    # run core code that uses resource f

Check with the provider of your resource initializer to see if context manager is already implemented. Use onCleanup() only when you don’t have this facility and you don’t want to build a whole context manager (even with decorators) for a one-liner cleanup.

 533 total views

Python startup management

The startup script is simply startup.m in whatever folder MATLAB start with.

Now how about Python? For plain Python (anything that you launch in command line, NOT Spyder though), you’ll need to ADD a new environment variable PYTHONSTARTUP to point to your startup script (same drill for Windows and Linux).

For Spyder, it’s Tools>Preferences>IPython console>Startup>”Run a file”:

but you don’t need that if you already have new environment variable PYTHONSTARTUP correctly setup.

 

 802 total views

MATLAB and Python paths

MATLAB’s path() is equal to Python’s sys.path().


To add paths in MATLAB, use the obviously named function addpath(). Supply the optional -end argument if you don’t want any potential shadowing (i.e. the folder to import has lower priority if there’s an existing function with the same name).

I generally avoid userpath() or the graphical tools because the results are sticky (persists between sessions). The best way is to exclusively manage your paths with startup.m so you always know what you are getting. If you want full certainty, you can start with restoredefaultpath() in MATLAB.


Python’s suggested these as equivalents of MATLAB’s addpath():

sys.path.insert(0, folder_to_add_to_path)
sys.path.append(folder_to_add_to_path)

but just like MATLAB’s addpath() which works with strings only (not cellstr), these Python options do not work correctly  with Python lists because the methods in sys.path are as primitive as doing [sys.path, new_stuff]:

  1. This means you’ll end up with list of lists if you supplied Python lists as inputs to the above
    (MATLAB will throw an exception if you try to feed it with cellstr instead of polluting your path space with garbage)
  2. This also means it doesn’t check for duplicates! It’ll keep stacking entries!

To address the first problem, we use sys.path.extend() instead. It’s like doing addpath(..., '-end') in MATLAB. If you want it to be inserted at the front (higher priority, shadows existing), you’ll need sys.path = list_of_new_paths + sys.path. For MATLAB, you can make a path string like DOS by using pathsep:

addpath(strjoin(cellstr_of_paths, pathsep)))

Note that  sys.path.extend() expect iterables so if you feed it a string, which Python will consider it a list of characters, you will get a bunch of one character paths inserted!

On the other hand, DO NOT TRY to get around it in Python with the same trick like MATLAB by doing sys.path.append( ';'.join(path_list)). Python recognize sys.path as a list, NOT a one long string like MATLAB/Windows path, despite insert() and append() accepts only strings!

Aargh!

The second problem (which does NOT happen in MATLAB) is slightly more work. You’ll need to subtract out the existing paths before you add to it so that you won’t drag your system down by casually adding paths as you see fit. One way to do it:

def keep_only_new_set_of_paths(p):
    return set(p)-set(sys.path)

You should organize your programs and libraries in a directory tree structure and use code to crawl the right branch into a path list! Don’t let the lack of built-in support to tempt you to organize files in a mess. Keep the visuals clean as mental gymnastics/overheads can seriously distract you from the real work such as thinking through the requirements and coming up with the right architecture and data structures. If you constantly need to jump a few hoops to do something, do it only once or twice using the proper way (aka, NOT copying-and-pasting boilerplate code), and reuse the infrastructure.

At my previous workplaces, they had dozens and dozens of MATLAB files including all laying flat in one folder. The first thing I did when I join a new team is showing everybody this idiom that recursively adds everything under the folder into MATLAB paths:

addpath(genpath())

Actually the built-in support for recursive directory search sucks for both MATLAB and Python.  Most often what we need is just a list of full paths for a path pattern that we search recursively, basically dir/w/s *. None of them has this right out of the box. They both make you go through the comprehensive data structure returned (let it be tuples from os.walk() in Python or dir() in MATLAB) and so some manipulations to get to this form.

genpath() itself is slow and ugly. It’s basically a recursive wrapper around dir() that cleans up garbage like '.' and '..'.  Instead of getting a newline character, a different row (as a char array) or a different cell (as cellstr), you get semi-colons (;) as pathsep in between. Nonetheless, I still use it because despite I have recursive path tools in my own libraries, I’ll need to load the library first in my startup file, which requires a recursive path tool like genpath(). This bootstraps me out of a chicken-and-egg problem without too much ugly syntax.


Most people will tell you to do a os.walk() and use listcomp to get it in the typical full path form, but I’m not settling for distracting syntax like this. People in the community suggested using glob for a relatively simple alternative to genpath()

Here’s a cleaner way:

def list_subfolders_recursively(p):
    p = p + '/**/' 
    return glob.glob(p, recursive=True);

It’s also worth noting that Python follows Linux’s file search pattern where directory terminates with a filesep (/) while MATLAB’s dir() command follows the OS, which in Windows, it’s *..

Both MATLAB and Python uses ** to mean regardless of levels, but you’ll have to turn on the recursive=True in glob manually. ** is already implied to be recursive in MATLAB’s dir() command.


Considering there’s quite a bit of plumbing associated with weak set of sys.path methods provided in Python, I created a qpath.py next to my startup.py:

''' This is the quick and dirty version to bootstrap startup.py
Should use files.py that issue direct OS calls for speed'''

import sys
import glob

def list_subfolders_recursively(p):
    p = p + '/**/' 
    return glob.glob(p, recursive=True);

def keep_only_new_set_of_paths(p):
    return set(p)-set(sys.path)

def set_of_new_subfolders_recursively(p):
    return keep_only_new_set_of_paths( list_subfolders_recursively(p) )

def add_paths_recursively_bottom(p):
    sys.path.extend(set_of_new_subfolders_recursively(p));

def add_paths_recursively_top(p):
    # operator+() does not take sets
    sys.path = list(set_of_new_subfolders_recursively(p)) + sys.path;

In order to be able to import my qpath module at startup.py before it adds the path, I’ll have put qpath.py in the same folder as startup.py, and request startup.py to add the folder where it lives to the system path (because your current Python working folder might be different from PYTHONSTARTUP) so it recognizes qpath.py.

This is the same technique I came up with for managing localized dependencies in MATLAB: I put the dependencies under the calling function’s folder, and use the path of the .m file for the function as the anchor-path to add paths inside the function. In MATLAB, it’s done this way:

function varargout = f(varargin)
  anchor_path = fileparts( mfilename('fullpath') );
  addpath( genpath(fullfile(anchor_path, 'dependencies')) );
  % Body code goes here

Analogously,

  • Python has __file__ variable (like the good old preprocessor days in C) in place of mfilename().
  • MATLAB’s  mfilename('fullpath') always gives the absolute path, but Python’s  __file__ is absolute if it’s is not in sys.path yet, and relative if it’s already in it.
  • So to ensure absolute path in Python, apply os.path.realpath(__file__). Actually this is a difficult feature to implement in MATLAB. It’s solved by a MATLAB FEX entry called GetFullPath().
  • Python os.path.dirname is the direct equivalent of fileparts() if you just take the first argument.

and in my startup.py (must be in the same folder as pathtools.py):

import os
import sys

sys.path.append(os.path.dirname(os.path.realpath(__file__)))

import pathtool

user_library_path = 'D:/Python/Libraries';
pathtool.add_paths_recursively_bottom(user_library_path)

This way I can make sure all the paths are deterministic and none of the depends on where I start Python.


Now I feel like Python is as mature as Octave. It’s usable, but it’s missing a lot of thoughtful features compared to MATLAB. Python’s entire ecosystem like at least 10 years behind MATLAB in terms of user friendliness. However, Python made it up with some pretty advanced language features that MATLAB doesn’t have, but nonetheless, we are still stuck with quite a bit of boilerplate code in Python, which decreases the expressiveness of the language (I’m a proponent of self-documenting code: variable and function names and their organization should be carefully designed to tell the story; comments are reserved for non-obvious tricks)

 1,604 total views

Getting pyinstaller 3.4 to work with Python 3.7

Python is an excellent language, but given that it’s free, it also comes with a lot of conspicuous loose-ends that you will not expect in commercially supported platforms like MATLAB.

Don’t expect everything to work right out of the box in Python. Everything is like 98% there, with the last 2% frustrate the heck out of you when you are rushing to get from point A to point B and you have to iron out a few dozen kinks before you can really start working.

When I tried use pyinstaller (v3.4) to compile my Python (v3.7) program into an executable, I ended up having to jump through a bunch of hoops:

  • pip install pyinstaller gives:
    ModuleNotFoundError: No module named 'cffi'
  • Then I looked up and installed cffi
    pip install cffi
  • After the dependency was addressed manually (it shouldn’t )  pip install pyinstaller worked
  • Then I tried to compile my first Python executable with pyinstaller, and I got this exception:
    File "C:\Python37\lib\site-packages\win32ctypes\core\cffi\_advapi32.py", line 198
    
            ^
        SyntaxError: invalid syntax
  • I searched the exact string and learned that pyinstaller (v3.4) is not ready for Python 3.7 yet! How come pip installer didn’t check for it? I opened up the offending file and looked for line 198 and saw this:
    c_creds.CredentialBlobSize = \
    
        ffi.sizeof(blob_data) - ffi.sizeof('wchar_t')

    It’s a freaking line continuation character \ (actually the extraneous CR before CRLF) that rooster-blocked it.

  • I just deleted the line continuation and merged the two lines, and saved _advapi32.py, then I was able to compile my Python v3.7 code (using pyinstaller 3.4) with no issues.

This is not something you’ll experience as a MATLAB user. The same company, TMW, wrote the MATLAB compiler as well as the rest. The toolbox/packages are released together in one piece so breaking changes that causes failure for the most obvious use case are caught before they get out of the door.

Another example of breaking changes that I ran into: ipdb does not allow you to move cursor backward.

Again, this is the cost associated with free software and access to the latest updates and new features without waiting for April/October (it’s the MATLAB regular release cycle). If hassle and the extra engineering time far exceed licensing MATLAB licensing costs, MATLAB is a better choice, especially if software is just a chore to get your company from point A to point B, and you are willing to pay big bucks to get there quickly and reliably.

Even with free software on the table, your platform choice is always determined by:

  • how much your time is worth wrestling problems
  • how much flexibility do you need (for customizing to your needs)
  • how much you are willing to pay for the licenses and support

In any case, the community did good work. Please consider sponsoring PyInstaller and PSF if you profit immensely from their work.

 1,410 total views