Category: Programming

Python packages, modules and imports

MATLAB, Python

Python’s import structure is freaking confusing. Learning by examples (i.e. imitating example code) does not help understanding the logic of it, and there are a lot of possible invalid combinations that are dead ends. You need to understand the concepts below to use it confidently!

Just like C++ quirks, very often there’s valid reasoning behind this confusing Python design choice and it’s not immediately obvious. Each language cater certain set of use cases at the expense of making other scenarios miserable. That’s why there’s no best universal language for all projects. Know the trade-offs of the languages so you can pick the right tool for the job.

MATLAB’s one file per function/script design

MATLAB made the choice of having one file describe one exposed object/function/class/script so it maps directly into the mental model of file systems. This is good for both user’s sanity and have behavioral advantages for MATLAB’s interpreter

  1. Users can reason the same same way as they do with files, which is less mental gymnastics
  2. Users can keep track of what’s available to them simply by browsing the directory tree and filenames because file names are function names, which should be sensibly chosen.
  3. Just like users, MATLAB also leverage the file system for indexing available functions and defer loading the contents to the memory until it’s called at runtime, which means changes are reflected automatically.

Package/modules namespace models in MATLAB vs Python

MATLAB traditionally dumps all free functions (.m files) available in its search paths into the root workspace. Users are responsible for not picking colliding names. Classes, namespaces and packages are after-thoughts in MATLAB while the OOP dogma is the central theme of Python, so obviously such practices are frowned upon.

RANT: OOP is basically a worldview formed by adding artificial man-made constructs (meanings such as agents, hierarchy, relationships) to the idea of bundling code (programs) and data (variables) in isolated packages controlled (scoped) by namespaces (which is just the lexer in your compiler enforcing man-made rules). The idea of code and data being the same thing came from Von Neumann Architecture: your hard drive or RAM doesn’t care what the bits stands for; it’s up to your processor and OS to exercise self-restraint. People are often tempted to follow rules too rigidly or not to take them seriously when what really matters is understanding where the rules came from, why they are useful in certain contexts and where they do not apply.

Packages namespaces are pretty much the skeleton of classes so the structure and syntax is the same for both. From my memory, it was at around 2015 that MATLAB started actively encouraging users (and their own internal development) to move away from the flat root workspace model and use packages to tuck away function names that are not immediately relevant to their interests and summon them through import syntax as needed. This practice is mandatory (enforced) in Python!

However are a few subtle differences between the two in terms of the package/module systems:

  • MATLAB does not have from statement because import do not have the option to expose the (nested tree of) package name to the workspace. It always dumps the leaf-node to the current workspace, the same way as from ... import syntax is used in Python.
  • MATLAB does not have an optional as statement for you to give an alternative name to the package you just imported. In my opinion, Python has to provide the as statement as an option to shorten package/module names because it was too aggressively tucking away commonly used packages (such as numpy) that forcing people to spell the informative names in full is going to be an outcry.
  • Unlike free functions (.m files), MATLAB classes are cached once the object is instantiated until clear classes or the like that gets rid of all instances in the workspace. Python’s module has the same behavior, which you need to unload with del (which is like MATLAB’s clear).
  • Python’s modules are not classes, though most of the time they behave like MATLAB’s static classes. Because the lack of instantiated instances, you can reload Python modules with importlib.reload(). On the other hand, since MATLAB packages merely manages when the .m files can get into the current scope (with import command), the file system still indexes the available function list. Changes in .m file functions reflects immediately on the next call in MATLAB, yet Python has to reload the module to update the function names index because the only way to look at what functions are available is revisiting the contents of an updated .py file!
  • MATLAB abstracts folder names (that starts with + symbol) as packages and functions as .m files while Python abstracts the .py file as a module (like MATLAB’s package) and the objects are the contents inside it. Therefore Python packages is analogous to the outer level of a double-packed (nested) MATLAB package. I’ll explain this in detail in the next sections.

Files AND directories are treated the same way in module hierarchy!

This comes with a few implications

  • if you name your project /myproj/myproj.py with a function def myproj(), which is a very usual thing most MATLAB users would do, your module is called myproj.myproj and if you just import myproj, you will call your function as myproj.myproj.myproj()!
  • you can confuse Python module loader if you have a subfolder named the same as a .py file at the same level. The subfolder will prevail and the .py file with the same name is shadowed!

The reason is that Python allows users to mix scripts, functions, classes in the same file and they classes or functions do not need to match the filenames in order for Python to find it, therefore the filename itself serves as the label for the collection (module) of functions, classes and other (script) objects inside! The directory is a collection of these files which itself is a collection, so it’s a two level nest because a directory containing a .py file is a collection of collection!

On the other hand, in MATLAB, it’s one .m file per (publicly exposed) function, classes or scripts, so the system registers and calls them by the filename, not really by how you named it inside. If you have a typo in your function name that doesn’t match your filename, your filename will prevail if there’s only one function there. Helper functions not matching the filename will not be exposed and it will have a static/file-local scope.

Packages in MATLAB are done in folders that starts with a + symbol. Packages by default are not exposed to global namespaces in your MATLAB’s paths. They work like Python’s module so you also get them into your current workspace with import. This means it’s not possible to define a module in a file like Python. Each filename exclusively represent one accessible function or classes in the package (no script variables though).

So in other words, there are no such thing called modules in MATLAB because the concept is called package. Python separated the two concepts because .py file allowing a mixture of scripts, classes and loose functions formed a logical unit with the same structure as packages itself, so they need another name called module to separate folder-based collection (logical unit) and file-based collections (logical unit).

This is very counterintuitive at the surface (because it defeats the point of directories) if you don’t know Python allowing user to mix scripts, functions and classes in a file meant the file itself is a module/collection of executable contents.

from (package/module) import (package/module or objectS) <as (namespace)>

This syntax is super confusing, especially before we understand that

  1. packages has to be folders (folder form of modules)
  2. modules can be .py files as well as packages
  3. packages/modules are technically objects

The hierarchy for the from import as syntax looks like this:

package_folder > file.py > (obj1, obj2, ... )

This has the following implications:

  • from strips the specified namespace so import dumps the node contents to root workspace
  • import without from exposes the entire hierarchy to the root workspace.
  • functions, classes and variables in the scripts are ALL OBJECTS.
  • if you do import mymodule, a function f in mymodule.py can only be accessed through mymodule.f(), if you want to just call f() at the workspace, do from mymodule import f

These properties also shapes the rules for where wildcards are used in the statement:

  • from cannot have wildcards because they are either a folder (package) or a file (module)
  • import is the only place that can have wildcards * because it is only possible to load multiple objects from one .py file.
  • import * cannot be used without from statement because you need to at some point load a .py file
  • it’s a dead end to do from package import * beacuse it’s trying to load the files to the root workspace which they are uncallalble.
  • it also does not make sense (nor possible) to follow import * with as statement because there is no mechanism to map multiple objects into one object name

So the bottom line is that your from import as statement has to somehow load a .py file in order to be valid. You can only choose between these two usage:

  • load the .py file with from statement and pick the objects at import, or
  • skip the from statement and import the .py file, not getting to choose the objects inside it.

as statement can only work if you have only one item specified in import, whether it’s the .py file or the objects inside it. Also, if you understand the rationales above, you’ll see that these two are equivalent:

from package_A import module_file_B as namespace_C
import package_A.module_file_B as namespace_C

because with as statement, whatever node you have selected is accessed through the output namespace you have specified, so whether you choose to strip the path name structure in the extracted output (i.e. use from statement) is irrelevant since you are not using the package and module names in the root namespace anymore.

The behavior of from import as is very similar to the choices you have to make extracting a zip file with nested folder structures, except that you have to make a mental substitution that a .py file is analogous to a subfolder while the objects described in the .py file is analogous to files in the said subfolder. Aargh!

 80 total views

Windows 10 Python Smart Aleck

Python, Windows

Windows 10 comes with a default alias that if you type python anywhere in terminal, powershell, run, etc, It will run a stub that points you to getting it in Windows Store. WTF man! I hate these stubs that are nothing but advertising! People will know there’s Python available in the store if Python Software Foundation’s website announces it. There’s no need to hijack the namespace with a useless stub!

After I install Spyder 5.3.0, it started with a Windows console instead of a Python Interpreter console, so when I typed Python (Spyder 5.3.0 came with Python 3.8.10 in its subfolder), this damn App store stub came up:

When I tried to force a .exe exceution in Powershell, I saw this:

So there’s a way to disable this bugger off!

It’s not the first time Spyder not working as intended out of the box, but Microsoft’s overzealous promotion of their ‘good ideas’ causes grief and agony to people who simply want things done.

It’s

 69 total views

Powershell notes (for MATLAB/python users)

MATLAB, Powershell, Python, Windows

Data Type characteristics

PowershellMATLABPython
Nearly everything is a/anObjectMatrix (APL-philosophy)Object (which are dictionaries)
Assignment behavior*Reassigned referenceCopy-on-writeReassigned reference
Monads (wrapper for heterogenous data )Array/CollectionsCellsLists

* Shallow assignment (transferring reference only) means the LHS does not have its own copy, so modifying the new reference will modify the underlying data on the RHS.

Syntax / Usage

PowershellMATLABPython
Method chainingYesMight misbehaveYes
List Comprehension–No. Map first then filterYes
Named input argumentsNative
f -a 1 -b 22		Name-Value pairs parsed insideNative
f(a=13, b=22)
Implicit NON-NULL return valueOptional––
Binary map operation–Native matrix ops
*fun() does n-ary		Use numpy
list( map(operator.add, L1, L2) )
Check Type$g -is [type]is*() or isa()isinstance(val, type)
Unpacking (flattening)
monads in monads		Default
Use unary , to avoid		No
Use [{:}] to perform		No
Use *, list comp, or
list(itertools.chain(*ls))
Conditional/statement block inside container creationYes–?
View Object Info with Data| Format-List -Property *
or
Format-List -InputObject		properties()
methods()
get()
List members (method and properties)’s prototypes| Get-Member

Powershell specific

  • The UNCAPTURED output value in the last line of the block is the return value! Unary side effect statements such as $x++ do not have output value. Watch out for statements that looks like it’s going nowhere at the end of the code as these are not nop/bugs, but return value. This has the same stench as fall-throughs.
  • foreach() follows the last uncaptured output value return rule above doing a 1-to-1 map from the input collection to output collection (you can assign output to foreach() as it’s also seen as a function)
  • Powershell suck at binary operations between two arrays. Just an elementwise A+B you’d be thinking in terms of loops and worry about dimensions.
  • You can put if and loop blocks inside collections list construction, like this:
@( 3, if(cond1){...; $v1}  do{...; $v2}while(cond2) )

MATLAB specific

  • When used with classes and custom matrices/arrays, chaining fields/properties/methods by indices often do not work, when they do, they often give out only the first element instead of the entire array (IIRC, there are operator methods that needs to be coordinated in the classes involved to make sure they chain correctly). In short, just don’t chain unless in very simple, scalar cases. Always output it to a variable a access the leaf.

Range & Indexing

PowershellMATLABPython
Logical Indexing–YesNo. Use list comprehension/Numpy
Negative (cyclic) IndexingYes–Yes
‘end‘ of array keyword–YesNo. Skip stop in slice instead
Step (skip every n items)–YesYes. Both range or slice
Detect descending rangeYes––
Automatic extend array–Yes–
Reading array out of boundsDo nothingErrorError

Negative (cyclic) indexing along with automatic descending range, along with the lack of ‘end’ keyword is a huge pain in the rear when you want to scan from left to right like A[5:end].

Instead, you’ll have to do $A[4..($A.length-1)] because the range 4..-1 inside A[4..-1] is unrolled as 4,3,2,1,0,-1 (thus scanning from right to left and wraps around) without first consulting with the array A like the end keyword in MATLAB does so it can substitute the ends of the range with the array information before it unrolls.

I am willing to bet that this behavior does not have a sound basis other than people thinking negative indices and descending ranges alone are two good ideas without realizing that nearly nobody freaking wants to scan from right to left and wrap around!

I had the same gripes about negative indices in Python not carefully coordinating with other combinations in common use cases which cases unintuitive behavior.

Range indexing syntax

# Powershell
1..10 # No step/skip for range creation
A[1..10]  # No special treatment in array such as figuring out the 'end'

% MATLAB
A[start:(step):stop]

# Python
A[range(start,stop,step)]
# Slicing (it's not range)
A[(start):(stop):(step)] # Can skip everything 
# In Python, A=X merely reassign the label A as the alias for X.
# Modifying the reassigned A through A=X will modify underlying contents of X
# To deep-copy contents without .Clone(), assign the full slice
A[:] = X

Hasthtable / Dictionaries

% MATLAB: Use dynamic fields in struct or containers.Map()
# Python: dictionaries such as {a:1, b='x'}
# Powershell: @{a=1, b='x'}

Structs

Powershell does not have direct struct or dynamic field name struct. Instead if your object is uniform (you expect the fields not to change much), use [PSCustomObject]@{}. You can also just use simple hashtable @{}, but for some reason it doesn’t work the way I expected when put into arrays when I try to reference it by array index.

Array rules surprises

  • Array comparisons are filtering operation (not boolean array output like MATLAB). (0..9) -ge 5 gives 5 to 9, not a list of False … False, True … True. To get a boolean array, use this shortcut:
(0..9) | % {$_ -ge 5}

Map-filter combo syntax is | ? instead of Map syntax | %

  • Monad (Cells in MATLAB) are unpacked and stacked by default (in MATLAB, I had to write a lot of routines to unpack and stack cells of cells). To keep cells packed (in MATLAB lingo, it’s like ‘UniformOutput’, false in cellfun), add a comma unary operator in front of the operation that are expected to be unpacked like this:
.$_.Split('_')

Set Operations

This is one of the WTF moments of Powershell as a programming language. Convenient set operations is essential for most of the routine boring stuff that involves relational data. A lot of Powershell’s intended audience works in database like environment (like IT managers dealing with Active Directory), they have Group-Object for typical data analysis tasks, yet they make life miserable just to do basic set operations like intersection and differencing!

Powershell has a Compare-Object, but this is as unnatural and annoying to use as users are effectively rebuilding all 4 basic set-ops (intersection, union, set-diff, xor) based on any two! Not to mention you have to sift through table to get to the piece you wanted!

Basically Compare-Object out of the box

  • is a set-diff showing both directions (A\B and also B\A) at the same time. If you throw away the direction info, it’s xor.
  • if you want intersection, you’ll need to add -IncludeEqual -ExcludeDifferent
  • (WTF!) If you just specify -ExcludeDifferent, by definition there’s no output because by default Compare-Object shows you ONLY the two set-diffs and you are telling it to not show any diffs!
  • Union is specifying -IncludeEqual only. But it’d rather stack both then do a | Sort-Object - Unique

Some people might suggest doing | ? {$_ -eq $B} for intersection (or is-member). This is generally a bad idea if you have a lot of data because it’s in the O(n*n) runtime algorithm (loop-within-loop) while any properly done intersection algorithm will just sort then scan the adjacent item to check for duplicates, which gives O(n log(n)) time (typical sorting algorithm takes up most of the time).

If you noticed, it’s set operations within the outputs of Compare-Objects with the Venn diagram of -IncludeEqual -ExcludeDifferent switches! It’s doable, but totally unnecessary mindfuck that should not be repeated frequently.

In MATLAB land, I made my own overloading operators that do set operation over cellstr(), categorical and tabular objects (I went into their code and added the features and talked to TMW so they added the features later), sometimes getting into their sort and indexing logic as necessary. This shows how badly do I need set operations to come naturally.

One might not deal with it too much in low level languages like C++ (STL set doesn’t get used as much compared to other containers), but for a language made to get a lot of common things done (i.e. the language designer kind of reads the users mind), I’m surprised that the Powershell team overlooked the set operations!

Sets are very powerful abstractions that should not be made less descriptive (hard to read) by dancing around it with equivalent operations with some programming gymnastics! If these basic stuff are not built in, we are going to see a lot of people taking ugly shortcuts to avoid coding up these bread and butter functions and put it in libraries (or downloading 3rd-party libraries)!

Powershell surprises

  • Typical symbolic comparison operators do not work because ‘>’ can be misinterpreted as redirection in command prompts. Use switches like -gt (greater than) instead.
  • Redirection’s default text output uses UTF16-LE encoding (2 bytes per character). Programs assuming ASCII (1 byte per character) might not behave as intended (e.g. if you use copy command merge an ASCII/UTF8 file with UTF16-LE, you might end up with spaces in the sections that are formatted with UTF16-LE)
  • Cannot extract string matches from regex without executing a -match which returns boolean unless we use the the $matches$ spilled into variable space. Consider [regex]::Match($Text, $Pattern).Groups[1].Value
  • Methods are called with parenthesis yet functions are not called with parenthesis, just like cmd-lets! Trying to call a function with multiple input arguments with parenthesis like f(3,5) will be interpreted as calling f with ONE ARGUMENT containing an ARRAY of 3 and 5!
  • Write-Host takes everything after it literally (white spaces included, almost like echo command), with the exception of plugging in $variables! If you want anything interpreted, such as concatenation, you need to put the bracket around the whole statement!

Libraries and Modules

  • Reload module using Import-Module $moduleName -Force

 68 total views

Regex Notes

Regex

Concepts

Mechanics

  • . any character
  • \ escapes special characters
  • characters (\d digits,\w word (i.e. letter/digit/underscore), \s whitespace).
  • [] character classes (define rules over what characters are accepted, unlike the . wildcard)
    [3-7] hypen inside [] bracket can specify ranges to mean things such as `[3,4,5,6,7]`
    [^ ...] is the mirror of it to exclude the mentioned characters
  • | choices (think of it as OR)
  • Complement (i.e. everything but) version are capitalized, such as \D is everything not a \d
  • whitespaces (\n newline, \t tab,

Modifiers

  • repetition quantifiers (? 0~1 times, + at least once, * any times, {match how many times})
  • (? ...) inline modifiers alters behaviors such as how newlines, case sensitivity, whether (...) captures or just groups, and comments within patterns are handled

Positioning rules

  • anchors (^ begins with, $ ends with)
  • \b word boundary

Output behavior

  • (...) capturing group, (?: ...) non-capturing group
  • \(index) content of previous matched groups/chunks referred to by indices.
    This feature generates derived new content instead of just extracting
  • (?( = | <= | ! | <! ) ...assertions...) lookarounds skips the contents mentioned in ...assertion... before/after the pattern so you can toss out the matched assertion from your capture results.

(?s) Also match newline characters (‘single-line’ or DOTALL mode)

Starting with (?s) flag (also called inline modifiers) expands the . (dot) single character pattern to ALSO match multiple lines (not by default).

Useful for extracting the contents of HTML blocks blindly and post-process it elsewhere

(?m) Pattern starts over as a new string for each line (‘multi-line’ mode)

Starting with (?m) flag tells anchors ^ (begin with) and $ (end with) to

Assertions: use lookarounds to skip (not capture) patterns
(?( = | <= | ! | <! ) assertion pattern)

  • < is lookbehind, no prefix-character is lookahead.
    -ahead/-behind refers to WHERE the you want TO CAPTURE relative to the assertion pattern,
    NOT what you want to assert (match and throw) away (inside the (? ...) )
  • = (positive) asserts the pattern inside the lookaround bracket,
    ! (negative) asserts the pattern inside the lookaround bracket MUST BE FALSE.

Assertions are very useful for getting to the meat you really want to capture rather than sifting through patterns introduced solely for making assertions that you intended to throw away

Extract HTML block

(?ms)(?<= starting tag pattern) body pattern (?= terminating tag pattern)

 61 total views

Cinnamon Desktop UI design WTFs (1)

GUI, Linux, Windows

Out of the box, Cinnamon decides to group the taskbar buttons like later Windows did. It’s often a huge annoyance to people who hates context switching in our head (I like huge workspaces that I can see everything at once so I don’t overlook clues from the relationship between things I’m working on. This is how I find difficult twists in research problems that other people give up solving).

In Windows, you right clock the taskbar, get to settings and there’s a pulldown menu for you to decide whether and how the buttons are grouped. Easy. But Cinnamon still have the Linux smell: organize things that are logical to programmers but not users (Tektronix, DD-WRT, etc. does that too), then surprise users with poorly thought out default behavior.

This time it’s a can of worms that requires some web searching to find people with the same exact specific problem (it’s a sign of poor UI design if the users cannot guess from the UI how to do what they want).

  1. Needing to change whether buttons are grouped is common. It should not take a lot of steps to change the behavior, preferably a right click context menu
  1. I would have thought it’s under Panel Settings, but hell no, things has to be organized the way the code was designed (sarcasm). It turns out that the windows button grouping is handled by an Applet called “Grouped Window List”
  1. Some user suggested removing the applet altogether (turns out it’s wrong and unnecessary as turning it off will disable the taskbar altogether and there’s an option to disable it within the applet’s setting: the applet itself is the windows list, not just the grouping feature), but fuck by default the applet was not activated the settings button is dead. I have to go to the bottom navigation bar in the window and hit the ‘+’ sign to get to the settings so now there’s a check mark next to it and the setting (gear) button is now activated.

    They also did not dim the settings button (two gears) when the ‘grouped window list’ is not activated (bug?), which made me think I can configure an Applet that’s not in use. Not to mention the previous settings got cleared (reset) if I disable the Applet and re-enable immediately afterwards (bug?)!
  1. Now I can finally get to turn this shit off
Chris Rock’s #HNTGYAKBTP Step #4 (1. OTL, 2. UCS, 3. SI, 4. TTSO, 5. BP, 6. STFU, 7. GAWF, 8. DRWAMW)
  1. This is where I think the UI design’s really fucked up. After you activate/deactivate the “Grouped windows list” applet, the buttons aligned right instead of left (default)! WTF!?! Do not do shit to surprise users! There’s absolutely no freaking logical reason why the taskbar button alignment should change the default (or the current state) for any reason!
  2. To fix this, you have to so something similar to unlocking the taskbar in Microsoft Windows to move the task button bar. It’s easy in MS Windows as you just right click context menu on the taskbar to unlock and just drag the starting separator (the || bar on the leftmost where the taskbar starts) to specific position you wanted. In Linux/Cinnamon, you have to enter the ‘Panel Edit Mode’ to unlock the taskbar so you can drag things around:
  1. I was confused while dragging the task button bar because there’s no clear position markers of where the task button starts and where it can ‘snap to grid’. It’s easy to drop it to the center to align center, but to align left, you have to watch for the buttons you want to insert before to move around to tell if it was a valid place to drop your new taskbar position What a pain in the butt!

This UI design suck, and I can totally understand why they would do something like this because of my programming background. It’s very logical for the programmer to modularize it as one applet, but first of all, generic suffixes like -let and -get does not help users get what the name means: it’s geeks’ way to name abstract concepts without getting the essence of the use case.

In MS Windows, the ‘Applets’ are organized roughly the same as ‘Toolbar’, except Windows is slightly more specialized that they have a ‘Toolbar’, ‘Start Menu’ and ‘Systray’ as distinct concepts instead of abstracting them into a higher level object as in ‘Applets’.

The biggest gripe I have about Cinnamon’s design choices is that detailed position adjustment needs to be easily accessible it’s likely that user preferences may vary a lot.

  • By not having a separate Toolbar concept, they forgot to add direct ‘unlock grouped windows list (aka tasklist toolbar in MS Windows)’ option (context menu item). You have to click through ‘Preference > Configure’ to get to get to configure the ‘Grouped window list’
  • Since the ‘Grouped window list’ is a (container) ‘bar’ within a bigger’ bar’ (Panel), the position of the window taskbar is logically organized under the platform (the bigger bar, hence the Panel), therefore the unlock window taskbar setting belongs to Panel, not Applet. This makes sense to programmers who knows that the feature is conceptually organized as container objects, but this is hell of confusing for users if they have to reason through this when they are trying to do one of the most common things!
  • Unlike MS Windows, you cannot use the task buttons while you are in Panel edit mode. Panel edit mode (you enter a special mode where you drag objects into positions you like, but cannot actually use them, then freeze it after you leave the mode) is the same concept used in Interactive Broker’s Trader Workstation (TWS), which is a pain in the ass but I understand the massive work saved for the people who designs the code/UI. Of course it comes at the expense of user frustration.
  • The solution article was written in 2018 and I’m surprised I still need that in 2022!

 74 total views

GUI Paradigms (1): Redux (Flutter/React) translated to MATLAB

Concepts, Dart, GUI, MATLAB, Programming Languages

For GUI development, we often start with controls (or widgets) that user interact with and it’ll emit/run the callback function we registered for the event when the event happens.

Most of the time we just want to read the state/properties of certain controls, process them, and update other controls to show the result. Model-View-Controller (MVC) puts strict boundaries between interaction, data processing and display.

The most common schematic for MVC is a circle showing the cycle of Controller->Model->View, but in practice, it’s the controller that’s the brains. The view can simultaneously accept user interactions, such as a editable text box or a list. The model usually don’t directly update the view directly on its own like the idealized diagram.

MVC with User Action
From https://www.educative.io/blog/mvc-tutorial

With MVC, basically we are concentrating the control’s callbacks to the controller object instead of just letting each control’s callback interact with the data store (model) and view in an unstructured way.

When learning Flutter, I was exposed to the Redux pattern (which came from React). Because the tutorials was designed around the language features of Dart, the documentation kind of obscured the essence of the idea (why do we want to do this) as it dwelt on the framework (structure can be refactored into a package). The docs talked a lot about boundaries but wasn’t clearly why they have to be meticulously observed, which I’ll get to later.

The core inspiration in Redux/BLoC is taking advantage of the concept of ‘listening to a data object for changes’ (instead of UI controls/widget events)!

Instead of having the UI control’s callback directly change other UI control’s state (e.g. for display), we design a state vector/dictionary/struct/class that holds contents (state variables) that we care. It doesn’t have to map 1-1 to input events or 1-1 to output display controls.

When an user interaction (input) event emitted a callback, the control’s callback do whatever it needs to produce the value(s) for the relevant state variable(s) and change the state vector. The changed state vector will trigger the listener that scans for what needs to be updated to reflect the new state and change the states of the appropriate view UI controls.

This way the input UI controls’ callbacks do not have to micromanage what output UI controls to update, so it can focus on the business logic that generates the content pool that will be picked up by the view UI controls to display the results. In Redux, you are free to design your state variables to match more closely to the input data from UI controls or output/view controls’ state. I personally prefer a state vector design that is closer to the output view than input controls.

The intuition above is not the complete/exact Redux, especially with Dart/Flutter/React. We also have to to keep the state in ONE place and make the order of state changes (thus behavior) predictable!

  • Actions and reducers are separate. Every input control fires a event (action signal) and we’ll wait until the reducers (registered to the actions) to pick it up during dispatch() instead of jumping on it. This way there’s only ONE place that can change states. Leave all the side effects in the control callback where you generate the action. No side effects (like changing other controls) allowed in reducers!
  • Reducers do not update the state in place (it’s read only). Always generate a new state vector to replace the old one (for performance, we’ll replace the state vector if we verified the contents actually changed). This will make timing predictable (you are stepping through state changes one by one)

In Javascript, there isn’t really a listener actively listening state variable changes. Dispatch (which will be called every time the user interacts using control) just runs through all the listeners registered at the very end after it has dispatched all the reducers. In MATLAB, you can optionally set the state vector to be Observable and attach the change listener callback instead of explicitly calling it within dispatch.

https://gist.github.com/gaearon/ffd88b0e4f00b22c3159

Here is an example of a MATLAB class that captures the spirit of Redux. I added a 2 second delay to emulate long operations and used enableDisableFig() to avoid dealing with queuing user interactions while it’s going through a long operation.

classdef ReduxStoreDemo < handle

    % Should be made private later
    properties (SetAccess = private, SetObservable)
        state % {count}
    end
        
    methods (Static)
        % Made static so reducer cannot call dispatch and indirectly do
        % side effect or create loops
        function state = reducer(state, action)        
            % Can use str2fun(action) here or use a function map
            switch action
                case 'increment'
                    fprintf('Wait 2 secs before incrementing\n');
                    pause(2)
                    state.count = state.count + 1;
                    fprintf('Incremented\n');
            end                
        end        
    end
    
    % We keep all the side-effect generating operations (such as
    % temporarily changing states in the GUI) in dispatch() so
    % there's only ONE PLACE where state can change
    methods
        function dispatch(obj, action, src, evt)
            % Disable all figures during an interaction
            figures = findobj(groot, 'type', 'figure');
            old_fig_states = arrayfun(@(f) enableDisableFig(f, 'off'), figures);      
            src.String = 'Wait ...';
            
                new_state = ReduxStoreDemo.reducer(obj.state, action);
                
                % Don't waste cycles updating nops 
                if( ~isequal(new_state, obj.state) )
                    % MATLAB already have listeners attached.
                    % So no need to scan listeners like React Redux            
                    obj.state = new_state;
                end                
                
            % Re-enable figure obj.controls after it's done
            arrayfun(@(f, os) enableDisableFig(f, os), figures, old_fig_states);                        
            src.String = 'Increment';
        end
    end
    
    methods
        function obj = ReduxStoreDemo()
            figure();            
            obj.state.count = 0;                        
            
            h_1x  = uicontrol('style', 'text', 'String', '1x Box', ...
                              'Units', 'Normalized', ...
                              'Position', [0.1 0.3, 0.2, 0.1], ...
                              'HorizontalAlignment', 'left');                          
            addlistener(obj, 'state', ...
                        'PostSet', @(varargin) obj.update_count_1x( h_1x , varargin{:})); 
                        
            uicontrol('style', 'pushbutton', 'String', 'Increment', ...
                      'Units', 'Normalized', ...
                      'Position', [0.1 0.1, 0.15, 0.1], ...
                      'Callback', @(varargin) obj.dispatch('increment', varargin{:}));                             
            
            % Force trigger the listeners to reflect the initial state
            obj.state = obj.state;
        end
    end
    
    %% These are 'renders' registered when the uiobj.controls are created
    % Should stick to reading off the state. Do not call dispatch here
    % (just leave it for the next action to pick up the consequentials)
    methods
        % The (src, event) is useless for listeners because it's not the 
        % uicontrol handle but the state property's metainfo (access modifiers, etc)
        function update_count_1x(obj, hObj, varargin)            
            hObj.String = num2str(obj.state.count);
        end        
    end
    
end

 182 total views

Rationale Behind C++ Commandments (5) – OOP design

C, C++, RBCC Series

The idea of bundling code and program into a layout (classes) and injecting it with different data (objects) leads to a ‘new’ way (newer than C) of organizing our programs through the worldview of objects.

Every unit is seen as

  • a state: all member variables
  • possible actions: methods = member functions.

that is ready to interact with other objects.

Encapsulation (through access control)

The first improvement upon OOP is privacy (data encapsulation). You can have finer controls of what to share and with who. In C++, your options are:

  • public: everybody
  • private: only within yourself (internal use)
  • protected: only shared with descendants (inheritance discussed below)

Granting certain class as friend (anywhere in the class declaration with friend class F) exposes the non-public sections specifically to the friend F. This is often a ‘loophole’ to access control that finds few legitimate uses other than testing.

friend functions are traditionally used in binary (2-input) operator overloading, but the modern wisdom is to screw it and just leave it out there as free functions!

protected has very few good uses other than preventing heap delete through base pointer non-polymorphically (child destructor not called: BAD) by making the base destructor non-public (i.e. meaning it’d be impossible to have base objects on stack) while letting the child chain the parent’s destructor (child can’t access it if it’s marked as private).

protected member variables are almost always a bad idea.

Inheritance

The second improvement is to allow classes to build on top of existing ones. What gets interesting (and difficult) is when the child ‘improve’ on the parent by either by replacing what they have (member variables) and what they do (methods) with their own.

Static data members inherit REFERENCES to the parent!

Inheritance AT LEAST always inherits an interface (can optionally inherit implementation).

Base implementation MUST NOT be inheritedpure virtual methods
Base implementation inherited by defaultvirtual
Base implementation MUST be inheritednon-virtual (and not shadow it)

Shadowing

Whenever the member (function or variable) name is used in any form (even with different argument types or signatures), the parent member with the same name will be hidden. The behavior is called shadowing, and it applies unless you’ve overridden ALL versions (signatures) of virutal parent methods which shares the same function name mentioned in child.

  • Any non-overriden method with the same name as the parent appearing in the child will shadow all parent methods with the same name regardless of whether they are declared virtual and overriden at child.
  • You can unhide parent methods with the same name (but different signature) by using Parent::f(..) declared at the child class.
  • Shadowing implies there’s always one parent version and one child version stored separately under all conditions {static or non-static}x{function or variable}
  • Static members don’t really ‘shadow’ because there’s only one global storage for each (parent and child) if you declare the same variable name again in the child. There’s nothing to hide because you cannot cast or slice a namespace! With static members, you have to be explicit about which class you are calling from with SRO like Parent::var or Child::var so there’s no potential for ambiguities.

Overriding

Just like C, C++ uses static binding that takes the programmer’s word for it for their declared types, especially through handles. Overriding is a concept only needed when you plan to upcast your objects (child accessed through pointer/reference) to handle a broader class of objects but intend to the underlying object’s own version (usually child) of the methods (especially destructors) called by default.

We do this by declaring the parent method virtual and implement the child versions (must be of the same function signature). Overriding only make sense for non-static methods because

  • data members cannot be overridden (it’d confusing if it’s possible. We down-delegate functions/behavior but not the data/state). It’s better off hiding data members behind getters/setters to declare the intention.
  • static members and methods behaves like static variable/functions (living in .data or .bss) using namespaces, so we can only refer to them with SRO by the class names like Parent::f() and Child::a, not a class type like Parent p; p.f() and Child c; c.a. There’s no object c for you to upcast to Parent so there’s place for polymorphic behavior.

Overriding involves leaving clues in objects so the upcasted references can figure out the correct methods of the underlying objects to call. In C++ it’s done with having a vtable (pointers to overridable methods, often stored in .rodata with string literals) for each class in the hierarchy and each object contains a pointer to the vtable that matches its underlying class.

[38] virtual only applies to methods’ signatures (function name and the data types in the argument list). vtable do not keep track of argument’s default values (if assigned) for efficiency (it’ll always read the static upcast, aka parent methods’ default values).

Classes (after considering inheritance)

Design relationships

  • class behaves through public methods
  • Inheritance at least always inherits an interface
  • IS-A relationship is done with public-inheritance
  • … (incomplete, will update later)

 86 total views

Rationale Behind C++ Commandments (4) – Method Signature System

C++, RBCC Series

Function signature system, which allows users to use the same function name in different functions as long as they differ in the combination of

  • input arguments types
  • const modifiers counts as a different input argument type
  • object const-ness (whether it’s const-method or not) – this only make sense with classes

and C++ will figure out what to call by matching the call with the available combinations (signatures).

C does not allow the same function name to be used in different places, so under the hood, it’s done through name mangling (generating a unique ‘under-the-hood’ function name based on the signature). This mechanism has a lot of implications that a professional programmer should observe:

  • since C does not mangle its names in the object code, they’ll need to be wrapped around with extern “C” block in a C++ program so C++ won’t pervert (mangle) their function names with input arguments.
  • [24] parameter defaulting might be ambiguous with another function that does not have the said parameter (the compiler will cry about it)
  • [26] access controls/levels must play no part in resolving signatures because access level must not change the meaning of a program!

C++ resolve function overloading using signatures within its local namespace. Function overloading works for both

  • free functions (free functions are at the root namespace), as well as
  • classes (the name of the class itself is the namespace)

 87 total views

Rationale Behind C++ Commandments (3) – Classes came from emulating POD data types through struct and namespaces

C, C++, RBCC Series

In structured programming (like C and C++), the building abstractions is program (functions) and data (variables).

Under the hood, especially in von-Neumann architecture’s perspective, functions and variables are both just data (a stream of numbers) that can be moved and manipulated the same way just like data. It’s all up to how the program designer and the hardware choose to give meaning to the bit stream.

Namespaces

In C, we can only scope our variables 3 ways: global, static (stays within same file/translation unit) and local. Sharing variables across functions in different translation units can only be done through

  • globals (pollutes namespace and it’s difficult to keep track of who is doing what to the variables and the state at any time)
  • passing (the more solid way that gives tighter control and clearer data flow, but managing how to pass many variables in many places is messy, even with struct syntax)

Bundling program with data gives a new way to tightly control the scope of variables: you can specify a group functions allowed to share the same set of variables in the bundle WITHOUT PASSING arguments.

The toolchain modified to recognize the user-defined scope boundaries which bundles program and data into packages, thus reducing root namespace pollution. The is implemented as namespace keyword in C++

Organizing with namespaces is basically justifying the mentality of using globals (in place of passing variables around intended functions) except it’s in a more controlled manner to keep the damages at bay. The same nasty things with gloabls can still appear if we didn’t design the namespace boundaries tightly so certain functions have access to variables that’s not intended for it.

Therefore, namespaces works nearly identical to a super-simple purely static class (see below) except you lose inheritance and access modifiers in classes in exchange for allowing anonymous namespaces.

Basically namespaces + structs + inheritance + encapsulation (access modifiers) = classes

Classes

Classes extends the idea of namespaces by allowing objects (each assigned their own storage space for the variables following the same variable layout) to be instantiated, so they behave like POD (Plain Old Data) in C. We should observe that when overloading operators

  • [15] allow (a=b)=c chaining by returning *this for operator=
  • [21] disallow rvalue assignment (a+b)=c by returning const object

In the most primitive form (no dynamic binding and types, aka virtuals and RTTI), function (method) info is not stored within instantiated objects as the compiler will sort out what classes/namespace they belong to. So it screams struct in C!

C struct is what makes (instantiates) objects from classes!

Note that C structs do not allow ‘static fields’ because static members is solely a construct of namespaces idea in C++! C++ has chosen to expand structs to be synonymous to classes that defaults to private access (if not specified) so code written as C structs behaves as expected in C++.

 85 total views

‘Static classes’ are unlike instantiable (object-bearing) classes in many ways

C++

Technically there’s no static class in C, but a class with all members and functions declared static.

Static classes are like namespaces in many ways. Because no object is constructed (it’s just holding a bunch of variables and functions in the free space), a lot of features and syntax with regular classes do not make sense with static classes.

Because no objects are instantiated

  • No constructors or destructors (no objects to make/destroy)
  • No operator overloading (you need an instantiation to pass arguments to operator methods)
  • No overriding because there are no objects for you to upcast
    (nor there’s an object to store the vtable from the virtual keyword)!

Static members and methods are treated as free objects scoped by namespaces

  • Like C, static members variables live in .bss (not explicitly initialized ones will be zero-initialized) or .data (initialized) sections, not on stack/heap!
    Exception: static const int is internally seen as enum, which the compiler uses it to plug values in the code instead of allocating space for it.
  • Therefore the syntax is pretty much like free static/global variables
  • No constructor to build member variables within the class definition, so they must be defined OUTSIDE the class definition at the top level (just like static/globals), with a SRO (scope resolution operator).
  • Static methods acts like (and function overloads the same way as) free functions.
    That’s why we often use static methods for helpers.

Namespaces has no access modifiers (public/protected/private/friend), but in return only namespaces can be unnamed/anonymous (which behaves as private)!

Namespaces cannot be inherited, but static classes can!

  • Inherited members ARE REFERENCES to the parent!
    There’s no extra copies of underlying data if that member is successfully inherited (not shadowed)!
  • Members (function or variables) can only be shadowed in the child (never overridden since it’s not an object), which creates a NEW stack variable and hid the reference to the parent member

Static class’s inheritance behavior is the same across static classes object-bearing classes! It’s actually more explicit with static members as you’ll need two declarations outside the classes if you shadow.

I am pointing this out to show that inheriting static classes IS NOT cloning namespaces! Static classes behaves as if it’s just ONE CHILD object created on the .bss/.data section (the section for static variables).

This means unlike object-bearing classes, the static class Parent cannot exist on its own if its children are defined!

C++ rules are almost always sensible and coherent; but when combined, sometimes the implications could be surprising on the first sight! When we try to extrapolate expected behaviors in C++, very often we have to think not in terms of the convenient syntax, but the implications of its ground rules (a lot of them stems from C)!

 91 total views