Useful Python tricks

This article is not for Pythonisms (like ContextManager), etc, the way things should normally be done in Python, but the more non-obvious way to solve problems or new features that are available specifically in Python.

Use / and * argument as separator for different forms of parameter entries!

There is special syntax to separate positional-only, positional/keyword, and keyword-only parameters.

Make a variant of existing class/object

This ninja technique is useful when you want to keep the object mostly the way it is but add/override a few things you think it didn’t do right without inheriting or use composition (hide a copy of object as a member) and write a proxy mirror for every member of it.

In C++, this situation is often used when you want to modify a concrete class that doesn’t have a virtual destructor, most notoriously STL which you are not supposed to inherit from (or else the client might pass a pointer to the parent/base so the child object’s destructors are not called as there are no vtables to keep track of which method to dispatch). In C++, this is often the few use cases that calls for private inheritance.

In Python, because everything is a recursive dict that specially named (magic) functions are recognized, there is a __getattr__ method that gets called whenever a member is accessed, which is the case when the member is called (in Python you simply get the functor as an attribute, aka value in the key-value pairs, in the dictionary and add brackets to call it). This means you can re-route what attributes (members) are returned simply by overloading this method!

If you can overload __getattr__, it also mean you can redefine the member interface of your entire class! So a strategy to make a class have the same exact interface as another class is to hide a copy/reference to the underlying class and re-route the __getattr__ to the underlying object’s __getattr__! Here’s the gist of it:

class M:
    def __init__(self, underlying_class):
          self.__obj = underlying_class  
    def __getattr__(self, attr):
        return getattr(self.__obj, attr)

This can be improved a little bit. Just pick a member name that won’t clash with the underlying object’s attributes/members and simply return the ‘hidden’ object when specifically requested, not self.__obj.__obj.

class M:
    def __init__(self, underlying_class):
          self.__obj = underlying_class  
    def __getattr__(self, attr):
        # Prevent self-calling        
        if attr != '__obj':
            return getattr(self.__obj, attr)
        else:
            return self.__obj

This is a very powerful luxury that makes Python so lovable if you are not here for industrial strength programming. MATLAB’s classes are hard-wired to your class definition .m file, not something you can update on the fly as you please. Any attempts to do so (aka breaking the safeguards) are Undocumented MATLAB territory where you mess with the Java under the hood and change the metadata property to fool the objects to do what you want.

Loading

Ways to clean up after yourself: C (no builtin exception handling)

[This is probably common knowledge that are repeated in many different places, but I want to arrange it in the perspective that helps my other article to explain how this influence the idea of ContextManager in Python and why ContextManager is still a clumsy way and there is a much neater way to do tackle this classic cleanup problem]

In C, there’s no built-in exception handling. Yet the goal for cleanups is for all manually checked and handled conditions (‘exceptions’) to land exactly in the same graveyard where the resources stands a chance to be released before the program ends. This screams goto (or longjump which is a non-local goto that can march outside the current function) and indeed it’s the only legit use of the goto statement I know that doesn’t make the code more error prone and confusing by littering the end of all your loops with if(error){break;}.

With this feedback approach, all hell breaks loose if you later add another layer and forget to add this. The mistake will break the feedback chain and the code continues to run in the layer after the end of the the while loop you forgot to place the check in, which is an insidious bug if the unwanted execution are benign under most situations.

The break clause will also become meaningless (and compiler invalid) if you convert your loops to non-loops or when you work in the top layer which is likely not inside a loop (even if you are in a bare metal embedded system with a while(0) loop, you don’t want to break that either).

break is a black-list approach which denies the rest of the code in the loop from running when the first error struck. Without break, you can do the reverse (the white-list approach) and put all code after the first check in if(!error) check blocks to authorize their execution instead

One hack to use break statement at the top-layer (no-loop) is to wrap the top layer with a do{BLOCK}while(false); loop which runs once, but the intention is not intuitive from the code so I wouldn’t do this to other programmers who don’t know the idiom without making a TRY-CATCH-FINALLY macro.

// Messy approach without do-while loop wrapper hack in top layer

// Convention: error=0 (false) means success
// The error code matches the check# it fails in this example
int error = 0;   
if( int* f = grab_resource_and_spit_zero_if_fail() )
{
  // This is hell of messy if you are not in a loop
  // that can take advantage of break-statement
  //
  // If you don't use breaks (which require it to be in a loop), 
  // you have to explicitly surround all code in if(!error) blocks
  ...
  // Approach 1: Nesting 
  // Upside:   Visually draws out what the logic relation is
  //           when you're doing checks all over the place
  //           It's hard to get it conceptually wrong
  //           (i.e. can debug blindly with mere semantics)
  // Downside: more checks means more nests 
  //           you either have excessive indentation
  //           or have fun tracking brackets
  if( !is_fail_1() )
  {
    ...
    if( !is_fail_2() )
    {
      ...
      if( !is_fail_3() )
      {
        (you get the idea)
        ...
      } else { error=3; }
    } else { error=2; }
  } else { error=1; }

  // Approach 2: Linear approach
  // Idea:       Surround everything under if(!error) check.
  //             Error code will stick to the first error as
  //             any non-zero error will short-circuit the 
  //             checks after it so the original error code stays
  // Upside:     No nesting. Easy to follow to recipe consistently
  // WARNING:    Thou shalt not be tempted to modify error code 
  //             in if(!error) blocks!
  // Downside:   If you violate the cardinal rule above, unintended
  //             chunks of code in if(!error) blocks might run and
  //             it's hard to debug/discover
  if(!error)
  {
    ...
  }
  // this idiom is a branchless way to do 
  // if(!error){ if(is_fail_13){error=13;} }
  // The error==0 should be placed in front to
  // take advantage of the short-circuit evaluation
  // to avoid actually running the check if there's
  // pre-existing error
  error = (error>0)*error + (error==0 && !is_fail_13())*13
  if(!error)
  {
    ...
  }
  error = (error>0)*error + (error==0 && !is_fail_14())*14
  if(!error)
  {
    ...
  }
  error = (error>0)*error + (error==0 && !is_fail_15())*15
  if(!error)
  {
    ...
  }
  // Now you are in the first loop so you are allowed to use break
  for(...)
  {
     ...
     (is_fails 16 .. 41)
     ...
     if( is_fail_42() )
     {
        error = 42;
        break;
     }
     ..
     while(...)
     {
       ... 
       (deep down in the nest)
       ...
       (is_fails 43 .. 101)
       ...
       if( is_fail_102() )
       {
          error = 102;
          break;
       }
       ...
     }
     if(error) { break; }
    ...
  }
  if(error) { break; }
  ...
  clean_the_f_up(f);
} 

// goto approach
if( int* f = grab_resource_and_spit_zero_if_fail() )
{
  ...
  (is_fails #1 .. 18)
  ...
  for(...)
  {
     ...
     (is_fails 19 .. 41)
     ...
     if( is_fail_41() )
     {
       goto graveyard;
     }
     ...
     while(...)
     {
       ... 
       (deep down in the nest)
       ...
       (is_fails 43 .. 101)
       ...
       if( is_fail_102() )
       {
          goto graveyard;
       }
       ...
     }
    ...
  }
  ...
  graveyard:
    clean_the_f_up(f);
} 

By jumping to the graveyard, we don’t need to litter the code with a long chain of error message/signal feedback and/or guard all chunks of code with if(!error) blocks, which is messy because it’s basically re-inventing a lightweight custom exception handling infrastructure that propagates the fault back to the top and give the intermediate layers a chance to intercept it.

As long as you are not using the goto approach to do complicated maneuvers and keep it simple: all faults go to the same bucket, no ifs-and-buts or detours (i.e. no code elsewhere/in-between can intercept the flow), it isn’t spaghetii code: there are no complicated code flow graphs, just every branch pointing to the same destination in one step. You don’t need to feel guilty about using the goto approach if your error handling flow is like this:

Loading

Rationale Behind C++ Commandments (5) – OOP design

The idea of bundling code and program into a layout (classes) and injecting it with different data (objects) leads to a ‘new’ way (newer than C) of organizing our programs through the worldview of objects.

Every unit is seen as

  • a state: all member variables
  • possible actions: methods = member functions.

that is ready to interact with other objects.


Encapsulation (through access control)

The first improvement upon OOP is privacy (data encapsulation). You can have finer controls of what to share and with who. In C++, your options are:

  • public: everybody
  • private: only within yourself (internal use)
  • protected: only shared with descendants (inheritance discussed below)

Granting certain class as friend (anywhere in the class declaration with friend class F) exposes the non-public sections specifically to the friend F. This is often a ‘loophole’ to access control that finds few legitimate uses other than testing.

friend functions are traditionally used in binary (2-input) operator overloading, but the modern wisdom is to screw it and just leave it out there as free functions!

protected has very few good uses other than preventing heap delete through base pointer non-polymorphically (child destructor not called: BAD) by making the base destructor non-public (i.e. meaning it’d be impossible to have base objects on stack) while letting the child chain the parent’s destructor (child can’t access it if it’s marked as private).

protected member variables are almost always a bad idea.


Inheritance

The second improvement is to allow classes to build on top of existing ones. What gets interesting (and difficult) is when the child ‘improve’ on the parent by either by replacing what they have (member variables) and what they do (methods) with their own.

Static data members inherit REFERENCES to the parent!

Inheritance AT LEAST always inherits an interface (can optionally inherit implementation).

Base implementation MUST NOT be inheritedpure virtual methods
Base implementation inherited by defaultvirtual
Base implementation MUST be inheritednon-virtual (and not shadow it)

Shadowing

Whenever the member (function or variable) name is used in any form (even with different argument types or signatures), the parent member with the same name will be hidden. The behavior is called shadowing, and it applies unless you’ve overridden ALL versions (signatures) of virutal parent methods which shares the same function name mentioned in child.

  • Any non-overriden method with the same name as the parent appearing in the child will shadow all parent methods with the same name regardless of whether they are declared virtual and overriden at child.
  • You can unhide parent methods with the same name (but different signature) by using Parent::f(..) declared at the child class.
  • Shadowing implies there’s always one parent version and one child version stored separately under all conditions {static or non-static}x{function or variable}
  • Static members don’t really ‘shadow’ because there’s only one global storage for each (parent and child) if you declare the same variable name again in the child. There’s nothing to hide because you cannot cast or slice a namespace! With static members, you have to be explicit about which class you are calling from with SRO like Parent::var or Child::var so there’s no potential for ambiguities.

Overriding

Just like C, C++ uses static binding that takes the programmer’s word for it for their declared types, especially through handles. Overriding is a concept only needed when you plan to upcast your objects (child accessed through pointer/reference) to handle a broader class of objects but intend to the underlying object’s own version (usually child) of the methods (especially destructors) called by default.

We do this by declaring the parent method virtual and implement the child versions (must be of the same function signature). Overriding only make sense for non-static methods because

  • data members cannot be overridden (it’d confusing if it’s possible. We down-delegate functions/behavior but not the data/state). It’s better off hiding data members behind getters/setters to declare the intention.
  • static members and methods behaves like static variable/functions (living in .data or .bss) using namespaces, so we can only refer to them with SRO by the class names like Parent::f() and Child::a, not a class type like Parent p; p.f() and Child c; c.a. There’s no object c for you to upcast to Parent so there’s place for polymorphic behavior.

Overriding involves leaving clues in objects so the upcasted references can figure out the correct methods of the underlying objects to call. In C++ it’s done with having a vtable (pointers to overridable methods, often stored in .rodata with string literals) for each class in the hierarchy and each object contains a pointer to the vtable that matches its underlying class.

[38] virtual only applies to methods’ signatures (function name and the data types in the argument list). vtable do not keep track of argument’s default values (if assigned) for efficiency (it’ll always read the static upcast, aka parent methods’ default values).


Classes (after considering inheritance)

Design relationships

  • class behaves through public methods
  • Inheritance at least always inherits an interface
  • IS-A relationship is done with public-inheritance
  • … (incomplete, will update later)

Loading

Rationale Behind C++ Commandments (4) – Method Signature System

Function signature system, which allows users to use the same function name in different functions as long as they differ in the combination of

  • input arguments types
  • const modifiers counts as a different input argument type
  • object const-ness (whether it’s const-method or not) – this only make sense with classes

and C++ will figure out what to call by matching the call with the available combinations (signatures).

C does not allow the same function name to be used in different places, so under the hood, it’s done through name mangling (generating a unique ‘under-the-hood’ function name based on the signature). This mechanism has a lot of implications that a professional programmer should observe:

  • since C does not mangle its names in the object code, they’ll need to be wrapped around with extern “C” block in a C++ program so C++ won’t pervert (mangle) their function names with input arguments.
  • [24] parameter defaulting might be ambiguous with another function that does not have the said parameter (the compiler will cry about it)
  • [26] access controls/levels must play no part in resolving signatures because access level must not change the meaning of a program!

C++ resolve function overloading using signatures within its local namespace. Function overloading works for both

  • free functions (free functions are at the root namespace), as well as
  • classes (the name of the class itself is the namespace)

Loading

Rationale Behind C++ Commandments (3) – Classes came from emulating POD data types through struct and namespaces

In structured programming (like C and C++), the building abstractions is program (functions) and data (variables).

Under the hood, especially in von-Neumann architecture’s perspective, functions and variables are both just data (a stream of numbers) that can be moved and manipulated the same way just like data. It’s all up to how the program designer and the hardware choose to give meaning to the bit stream.


Namespaces

In C, we can only scope our variables 3 ways: global, static (stays within same file/translation unit) and local. Sharing variables across functions in different translation units can only be done through

  • globals (pollutes namespace and it’s difficult to keep track of who is doing what to the variables and the state at any time)
  • passing (the more solid way that gives tighter control and clearer data flow, but managing how to pass many variables in many places is messy, even with struct syntax)

Bundling program with data gives a new way to tightly control the scope of variables: you can specify a group functions allowed to share the same set of variables in the bundle WITHOUT PASSING arguments.

The toolchain modified to recognize the user-defined scope boundaries which bundles program and data into packages, thus reducing root namespace pollution. The is implemented as namespace keyword in C++

Organizing with namespaces is basically justifying the mentality of using globals (in place of passing variables around intended functions) except it’s in a more controlled manner to keep the damages at bay. The same nasty things with gloabls can still appear if we didn’t design the namespace boundaries tightly so certain functions have access to variables that’s not intended for it.

Therefore, namespaces works nearly identical to a super-simple purely static class (see below) except you lose inheritance and access modifiers in classes in exchange for allowing anonymous namespaces.

Basically namespaces + structs + inheritance + encapsulation (access modifiers) = classes


Classes

Classes extends the idea of namespaces by allowing objects (each assigned their own storage space for the variables following the same variable layout) to be instantiated, so they behave like POD (Plain Old Data) in C. We should observe that when overloading operators

  • [15] allow (a=b)=c chaining by returning *this for operator=
  • [21] disallow rvalue assignment (a+b)=c by returning const object

In the most primitive form (no dynamic binding and types, aka virtuals and RTTI), function (method) info is not stored within instantiated objects as the compiler will sort out what classes/namespace they belong to. So it screams struct in C!

C struct is what makes (instantiates) objects from classes!

Note that C structs do not allow ‘static fields’ because static members is solely a construct of namespaces idea in C++! C++ has chosen to expand structs to be synonymous to classes that defaults to private access (if not specified) so code written as C structs behaves as expected in C++.

Loading