Daily Archives: February 19, 2025

Pandas DataFrame in Python (1): Disadvantage of using attributes to access columns

Posted on by
Reply

There are two ways to access columns in DataFrame. The preferred way is by square brackets (indexing into it like a dictionary), while it’s tempting to use the neater dot notation (treating columns like an attribute), my recommendation is don’t!

Python has dictionaries that handles arbitary labels well while it doesn’t have dynamic field names like MATLAB do. This puts DataFrame at a disadvantage developing dot notation syntax while the dictionary syntax opens up a lot of possibilities that are worth giving up dot notation for. The nature of the language design makes the dot notation very half-baked in Python and it’s better to avoid it altogether

Reason 1: Cannot create new columns with dot notation

UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access

Reason 2: Only column names that doesn’t happen to be valid Python attribute names AND DataFrame do not have any method with the same name can be accessed through dot notation.

Take an example of dataframe constructed from device info dictionaries created by the package pyft4222. I added a column called 'test me' to a table converted from the dictionary of device info. The tabe T looks like this:

I tried dir() on the table and noticed:

  • The column name "test me" did not appear anywhere, not even mangled. It has a space in between so it’s not a valid attribute or variable name, so this column is effectively hidden from the dot notation
  • flags is an internal attribute of DataFrame and it was not overriden by the data column flags when called by the dot notation. This means the flags column was also hidden to the dot notation as there were no mangled name for it either

Loading

Dictionary of equivalent/analogous concepts in programming languages

Posted on by
Reply
CommonCC++MATLABPython
Variable arguments<stdarg.h>
T f(...)
Packed in va_arg		Very BAD!

Cannot overload
when signatures are uncertain.		varargin
varargout

Both packed as cells.

MATLAB does not have named arguments		*args (simple, stored as tuples)

**kwargs (specify input by keyword, stored as a dictionary)
Referencing
N/A		operator[](_) is for references
subsindex
subsassgn

[_] is for concat
{_} is for (un)pack		__getitem__()
__setitem__()
Logical
Indexing		N/AN/ANative (first-class)List comprehension
(Raw Python)

Boolean Indexing native with numpy/pandas
Default
values		N/ASupportedNot supported.
Manage with inputParser()		Non-intuitive static data behavior. Stick to None or immutables.
Major
Dimension		RowRowColumnRow (Native/Numpy)
Column for Pandas
ConstnessconstconstOnly in classesN/A (Consenting adults)
Variable
Aliasing		PointersReferencesNO!
Copy-on-write		References
= assignmentCopy one
element		Values: Copy
References: Bind		New Copy
Copy-on-write		NO VALUES
Bind references only
(could be to unnamed objects)
Chained
Operations		N/A

Assignment’s value is assigned value		Difficult to get it rightDifficult to get it right. MATLAB had some chaining bugs with dataset() as well.Chains correctly natively

Data Types

CommonCC++MATLABPython
SetsN/Astd::setOnly set operations, not set data type{ , , ...}
Dictionariesstd::unordered_map– Dynamic fieldnames
(qualified varnames as keys)
– containers.Map() or dictionary() since R2022b		Dictionaries
{key:value}
(Native)
Heterogeneous containerscellslists (mutable)
tuples (immutable)
Structured
Heterogeneous		table()
dataset() [Old]		Pandas Dataframe
Array,
Matrices &
Tensors		Native [ , ; , ]Numpy/PyTorch
Recordsstructclass
(members)		dynamic field (structs)
properties (class)

getfield()/setfield()		No structs
(use dicts)

attribute (class)
Native sets operations in Python are not stable and there’s no option to use stable algorithm like MATLAB does. Consider installing orderly-set package.

Editor Syntax

CommonCC++MATLABPython
Commenting/* ... */

// (only for newer C)		// (single line)

/* ... */ (block)		% (single line)

(Block):
%{
...
%} 		# (single line)

""" or '''
is docstring which might be undersirably picked up
Reliable multi-line
commenting
(IDE)		Ctrl+(Shift)+R(Windows), / (Mac or Linux)[Spyder]:
Ctrl+1(toggle), 4(comment), 5(uncomment)
Code cell
(IDE)		%%[Spyder]:
# %%
Line
Continuation		\\...\

Object Oriented Programming Constructs

CommonC++MATLABPython
Getters
Setters		No native syntax.

Name mangle (prefix or suffix) yourself to manage		Define methods:
get.x
set.x		Getter:
@property
def x(self): ...

Setter:
@x.setter
def x(self, value): ...
DeletersMembers can’t be
changed on the fly		Members can’t be
changed on the fly		Deleter (removing attributes
dynamically by del)
Overloading
(Dispatch function by signature)		OverloadingOverload only by
first argument		@overload (Static type)
@singledispath
@multipledispatch
Initializing class variablesInitializer Lists
Constructor		ConstructorConstructor
ConstructorClassName()ClassName()__init__()
Destructor~ClassName()delete()__del__()

Functional Programming Constructs

CommonC++MATLABPython
Function as
variable		Functors
(Function Objects)
operator()		Function HandleCallables
(Function Objects)
__call__()
Lambda
Syntax		Lambda
(C++11)		Anonymous FunctionLambda
Closure
(Early binding): an
instance of lambda		Capture [] only as necessary.

Early binding [=] is capture all.		Early binding ONLY.

Takes snapshot of workspace values involved when instantiated (anonymous function object is created)		Late binding* by default.

Can capture Po through default values
lambda x,P=Po: x+P
(We’re relying users to not enter the captured/optional input argument)
Concepts of Early/Late Binding also apply to non-lambda functions. It’s about when to access (usually read) the ‘global’ or broader scope (such as during nested functions) variables that gets recruited as a non-input variable that’s local to the function itself.

The more proper way (without creating an extra optional argument that’s not supposed to be used, aka defaults overridden) to convert late binding to early binding (by capturing variables) is called partial application, where you freeze the parameters (to be captured) by making them inputs to an outer layer function and return a function object (could be lambda) that uses these parameters.

Currying is partial application one parameter at a time, which is tedious way to stay faithful to pure functional programming.

Loading