Python 3 Scientific Installation To-do List

Although I am a big fan of MATLAB, it’s time for me to really try out Python so I can fairly compare the pros and cons of both languages.

The first tiny hurdle for Python is its scattered installation process for Windows. I thought Python(x,y) will give me everything in one place, but turns out the Spyder is stuck in Python 2.7. To install Python 3.7, I’ll need to do it from scratch. Here are the steps:

  1. Download official Python 3 (https://www.python.org/downloads/windows/). You will need that for the “pip” package manager
  2. In command prompt anywhere,
    pip install scipy
    It might complain that {python37}/scripts is not in PATH variable, but I checked and the folder is already in PATH. Can safely ignore it.
  3. I took the advice to upgrade PIP. Can run it anywhere in command prompt
    python -m pip install –upgrade pip
  4. Now I’ll need Spyder3, a MATLAB-like IDE. Qt5 is one of the pre-req:
    pip3 install PyQt5
  5. And finally Spyder3
    pip install Sypder
  6. PyVISA is the analog of “Instrument Control Toolbox” in MATLAB.
    pip install pyvisa
  7. Turns out that only NumPy and IPython is installed with SciPy, not the entire ecosystem.
    pip3 install pandas
    pip3 install matplotlib
    If you know the power of dataset/table objects in MATLAB like I do, you’ll jump for dataframes in panadas.
  8. SymPy, the analog of MATLAB’s symoblic math toolbox, needs to be installed separately
    pip3 install sympy
  9. IPython gives the ‘notebook’ feel in Mathematica, MathCAD and Maple, where the returned results are directly pasted in the same area where your command/syntax is. I rarely cared for it because I usually want the max visual real estate for my plots.

pip does not install icons in your start menu. So I’ll need to manually create a shortcut
“{Python37}/Scripts/spyder3.exe”

.py files are not associated with Spyder3 (normally it’ll just directly run the python script with python3). I usually manually change the association in Windows to Sypder3.

11 total views, 1 views today

What is calibration? Hint: It is not adjustment!

Very often people doing R&D ask me if they need to have their oscilloscope calibrated. And for most of the time, my answer is no unless they need to have the NIST traceability or the calibration sticker to keep the regulatory bodies happy. They often thought it’s adjusting the calibration coefficients (or knobs) to make the unit more accurate. This is COMPLETELY WRONG.

In EEVBlog, they showed a video interview with Agilent Metrologist explaining what calibration actually does: it gives you the sample data points against trusted references about how your test instruments’ references has drifted between calibrations. Actually it’s preferable to not adjust the instruments if it’s already within specs.

 

157 total views, 1 views today

TDS 500~800 series Monochrome CRT Driver Repair

The most common mode of CRT display failure in the TDS 500~800 series (Monochrome models) is the flyback transformer. The symptom is that after leaving the screen on for a couple of hours, the screen started stretching vertically until it disappears.

It also happens the failure only happens to a batch of CRT boards. The batch I’ve seen looks new (with modern markings that tells you the purpose of the trimpots) and lightly used, so I’m sure it’s infant mortality. Here’s the broken CRT driver board to be repaired:

There’s practically nowhere you can find this obsolete replacement because it’s not a common configuration. I sourced a batch from China that claimed the part number, and I spent whole day cursing the vendor when the flyback transformer arrived. Here’s what I saw:

The one on the right was the broken original flyback transformer, and two on the left were my new orders. Not only that the shapes are completely different, the number of pins doesn’t even match. WTF?!!!

The seller told me that it works. Forget about how to fit that in the board for a moment. How the f*** am I supposed to know which pins goes to which spot? Not to mention there are 11 nibs  when the original only has 8+1 (actually 7+1, pin#8 is not used). I said nibs instead of pins because not all of them are populated with a pin, and the pins that are missing were not even consistent across the transformers in the batch.

I cannot even guess with a multimeter because it’s not a simple, uniform transformer. Even if I know which ones are connected, I could have ruined the whole thing by having the wrong number of coil turns/inductances because I switched a pair or two!

I had to push the seller really hard for him to dig up the actually mapping and draw me the pinouts on the pictures I’ve sent him (I’m sure the whole batch will be trash if I could not communicate with them in Chinese). The ‘product’ must have been designed and the manufacturing line ran by a bunch of village idiots. Nothing is right about it other than the windings inside are electrically usable (can’t even say compatible because I need to hack it really hard to get the correct display). Here’s the pinout:

Here’s another transformer that doesn’t have the ground pin (unnumbered), turns out the transformer works without it:

The space inside the oscilloscope case is pretty tight, and I managed to find one orientation that lines up with the case nicely, but it’s ugly as hell:

I held it down with hot-glue, caulk to stabilize it. A rubber band was put over it so that if the glue fails, the transformer won’t roll inside the compartment causing mayhem (later units I used cable ties since rubber band might deteriorate with heat. You get the idea.):

I have a few of these transformers available for $80/pc. Unless you have the time on your hands to figure out the details, the learning curve is so steep that I wouldn’t recommend DIY.

I hate to see perfectly good units going to the landfill for stupid reasons. Although it’s less than what my time is worth, for $350, I can provide all the necessary part and do the CRT driver surgery if you anybody wants to revive a unit.

If time is pressing and you don’t want to ship the unit back and forth, I have ready-made CRT driver boards for $300 (you’ll need to trade in the bad CRT driver board). Of course, you have to do the CRT screen adjustments yourself since each tube is different.

247 total views, 2 views today

MATLAB Techniques: variadic (variable input and output) arguments

I’ve seen a lot of ugly implementations from people trying to deal with variable number of input and output arguments. The most horrendous one I’ve seen so far came from MIT’s Physionet’s WFDB MATLAB Toolbox. Here’s a snippet showing how wfdbdesc.m handles variable input and output arguments:

function varargout=wfdbdesc(varargin)
% [siginfo,Fs,sigClass]=wfdbdesc(recordName)
...
%Set default pararamter values
inputs={'recordName'};
outputs={'siginfo','Fs','sigClass'};

for n=1:nargin
    if(~isempty(varargin{n}))
        eval([inputs{n} '=varargin{n};'])
    end
end
...
if(nargout>2)
    %Get signal class
    sigClass=getSignalClass(siginfo,config);
end
for n=1:nargout
    eval(['varargout{n}=' outputs{n} ';'])
end

 

The code itself reeks a very ‘smart’ beginner who didn’t RTFM. The code is so smart (shows some serious thoughts):

  • Knows to use nargout to control varargout to avoid the side effects when no output is requested
  • [Cargo cult practice]: (unnecessarily) track your variable names
  • [Cargo cult practice]: using varargin so it can be symmetric to varargout (also handled similarly). varargout might have a benefit mentioned above, but there is absolutely no benefit to use varargin over direct variable names when you are not forwarding or use inputParser().
  • [Cargo cult practice]: tries to be efficient to skip processing empty inputs. Judicially non-symmetric this time (not done to output variables)!

but yet so dumb (hell of unwise, absolutely no good reason for almost every ‘thoughtful’ act put in) at the same time. Definitely MIT: Make it Tough!

This code pattern is so wrong in many levels:

  • Unnecessarily obscuring the names by using varargin/varargout
  • Managing a list of variable names manually.
  • Loop through each item of varargin and varargout cells unnecessarily
  • Use eval() just to do simple cell assignments! Makes me cringe!

Actually, eval() is not even needed to achieve all the remaining evils above. Could have used S_in = cell2struct(varargin) and varargout=struct2cell(S_out) instead if one really wants to control the list of variable names manually!


The hurtful sins above came from not knowing a few common cell packing/unpacking idioms when dealing with varargin and varargout, which are cells by definition. Here are the few common use cases:

  1. Passing variable arguments to another function (called perfect forwarding in C++): remember C{:} unpacks to comma separated lists!
    function caller(varargin)
       callee(varargin{:});
  2. Limiting the number of outputs to what is actually requested: remember [C{:}] on left hand side (assignment) means the outputs are distributed as components of C that would have been unpacked as comma separated lists, i.e. [C{:}] = f(); means [C{1}, C{2}, C{3}, ...] = f();
    function varargout = f()
    // This will output no arguments when not requested,
    // avoiding echoing in command prompt when the call is not terminated by a semicolon
        [varargout{1:nargout}] = eig(rand(3));
  3. You can directly modify varargin and varargout by cells without de-referencing them with braces!
    function varargout = f(varargin)
    // This one is effectively deal()
        varargout = varargin(1:nargout);
    end
    
    function varargout = f(C)
    // This one unpacks each cell's content to each output arguments
        varargout = C(1:nargout);
    end

One good example combining all of the above is to achieve the no-output argument example in #2 yet neatly return the variables in the workspace directly by name.

function [a, b] = f()
// Original way to code: will return a = 4 when "f()" is called without a semicolon
    a = 4;
    b = 2;
end

function varargout = f()
// New way: will not return anything even when "f()" is called without a semicolon
    a = 4;
    b = 2;
    varargout = manage_return_arguments(nargout, a, b);
end

function C = manage_return_arguments(nargs, varargin)
    C = varargin(1:nargs);
end

I could have skipped nargs in manage_return_arguments() and use evalin(), but this will make the code nastily non-transparent. As a bonus, nargs can be fed with min(nargout, 3) instead of nargout for extra flexibility.


With the technique above, wfdbdesc.m can be simply rewritten as:

function varargout = wfdbdesc(recordName)
% varargout: siginfo, Fs, sigClass
...
varargout = manage_return_arguments(nargout, siginfo, Fs, sigClass);

Unless you are forwarding variable arguments (with technique#1 mentioned above), input arguments can be (and should be) named explicitly. Using varargin would not help you avoid padding the unused input arguments anyway, so there is absolutely no good reason to manage input variables with a flexible list. MATLAB already knows to skip unused arguments at the end as long as the code doesn’t need it. Use exist('someVariable', 'var') instead.

 

 

225 total views, 3 views today

MATLAB Practices: Code and variable transparency. eval() is one letter away from evil() Transparency means that all references to variables must be visible in the text of the code

The general wisdom about eval() is: do not use it! At least not until you are really out of reasonable options after consulting more than 3 experts on the newsgroups, forums and support@mathworks.com (if your SMS is current)!

Abusing eval() turns it into evil()!

The elves running inside MATLAB needs to be able to track your variables to reason through your code because:

  • it helps your code run much faster (eval() cannot be pre-compiled)
  • able to use parallel computing toolbox (it has to know absolutely for sure about any shared writes)
  • mlint can warn you about potentially pitfalls through code smell.
  • it keeps you sane while debugging!

This is called ‘transparency’: MATLAB has to see what you are doing every step of the way. According to MATLAB’s parallel computing toolbox’s documentation,

Transparency means that all reference to variables must be visible in the text of the code

which I used as a subtitle of this post.


The 3 major built-in functions that breaks transparency are:

  1. eval(), evalc(): arbitrary code execution resulting in read and write access to its caller workspace.
  2. assignin(): it poofs variables in its caller’s workspace!
  3. evalin(): it breaks open the stack and read the variables in its caller’s workspace!

They should have been replaced by skillful use of dynamic field names, advanced uses of left assignment techniques, and freely passing variables as input arguments (remember MATLAB uses copy-on-write: nothing is copied if you just read it).


 

There are other frequently used native MATLAB functions (under certain usages) that breaks transparency:

  1. load(): poof variables from the file to the workspace. The best practice is to load the file as a struct, like S=load('file.mat'); , which is fully transparent. Organizing variables into structs actually reduces mental clutter (namespace pollution)!
  2. save(), who(), whos(): basically anything that takes variable names as input and act on the requested variable violates transparency because it’s has the effect of evalin(). I guess the save() function chose to use variable names instead of the actual variables as input because copy-on-write wasn’t available in early days of MATLAB. A example workaround would be:
    function save_transparent(filename, varargin)
        VN = arrayfun(@inputname, (2:nargin)', 'UniformOutput', false);     
        if( any( cellfun(@isempty, VN) ) )
            error('All variables to save must be named. Cannot be temporaries');
        end
        
        S = cell2struct(varargin(:), VN, 1);
        save(filename, '-struct', 'S');
    end
    
    function save_struct_transparent(filename, S)
        save(filename, '-struct', 'S');
    end

The good practices to avoid non-transparent load/save also reduces namespace pollution. For example, inputname() peeks into the caller to see what the variable names are, which should not be used lightly. The example above is one of the few uses that I consider justified. I’ve seen novice abusing inputname() because they were not comfortable with cells and structs yet, making it a total mindfuck to reason through.

258 total views, 2 views today

Experimental Worldview Framework Desires x Problems x Mechanisms x Device

I am experimenting with a framework to summarize how I observe things that are going on around me, analyzing situations and coming up with solution approaches. Currently this is what I have:

{Desires} × {Problems} × {Mechanisms} × {Devices}

Everything I see can be analyzed as a result of the cross-product (a fancy word for combinations of contents) between these 4 broad categories. To make it easier to remember, they can be factored into 2 major categories:

{Questions} × {Answers}

Where obviously

  • [Questions] Desires (objectives) lead to problems (practicalities) to solve
  • [Answers] Mechanisms (abstract concepts) hosted by a device (implementation) to address questions

Why the cross product? By tabulating everything I learned or know in 4 columns (categories), I can always select a few of them (subset) and notice a lot of recurring themes (questions) and common solution approaches (answers). This corresponds to an old saying “there’s nothing new under the sun”.

Then what about innovations? Are we constantly creating something new? Yes, we still are, but if you look closely, there are very few ideas that are fundamentally new that cannot be synthesized by combining the old ones (sometimes recursively).

Let me use the framework itself as an example on how to apply this framework (yes, it’s recursive):

  • Desires: predict and understand many phenomenon
  • Problems: mental capacity is limited
  • Mechanisms: this framework (breaking observations into 4 categories)
  • Devices: tabulation (cross-products, order reduction)

Feedback as an example:

  • Desires: have good outcomes (or meet set objectives)
  • Problems: not there yet
  • Mechanism: take advantage of past data for future outputs (through correction)
  • Devices: feedback path (e.g. regulator or control systems.)

Feedforward as an example that shares a lot of properties as feedback:

  • Desires: have good outcomes (or meet set objectives)
  • Problems: not there yet
  • Mechanism: take advantage of past data for future outputs (through prediction)
  • Devices: predictor (e.g. algorithm or formula)

Abstraction as an example:

  • Desires: understand complexities (e.g. large code base)
  • Problems: limited mental capacity (programmers are humans after all)
  • Mechanism: abstraction (generic view grouping similar ideas together)
  • Devices: black-boxes (e.g. functions, classes)

Trade as an example:

  • Desires: improves utility (utility = happiness in economics lingo)
  • Problems: one cannot do everything on its own (limited capacity)
  • Mechanism: exchange competitive advantages
  • Devices: market (goods and services)

Business as an example:

  • Desires: improves utility (through trade)
  • Problems: need to offer something for trade
  • Mechanism: create value
  • Devices: operations

Money (and Markets) as two examples:

  • Desires: facilitate trade
  • Problems: difficult valuation and transfer through barter, decentralized
  • Mechanism: a common medium
  • Devices: money (currencies), markets (platform for trade)

Law as an example:

  • Desires: make the pie bigger by working cooperatively
  • Problems: every individual tries to maximize their own interest but mentally too limited to consider working together to grow the pie (pareto efficient solutions)
  • Mechanism: set rules and boundaries (I personally think it’s a sloppy patch fix that is way overused and way abused) and get everybody to buy it
  • Devices: law and enforcement

Religion as an example

  • Desires: coexist peacefully
  • Problems: irreconcilable differences
  • Mechanism: blind unverified trust (faith)
  • Devices: Deities and religion

Just with the examples above, many desires can be consolidated along the lines of making ourselves better off, and many problems can be consolidated along the lines of we’re too stupid. Of course it’s not everything, but it shows the power of tabulating into 4 categories and consolidating the parts into few unique themes.

I chose to abstract the framework into 4 broad categories instead of 2 because two are too simplified to be useful: since the framework is a way to organize (compactify) observations into manageable number of unique items, there will be too many distinct entries if I have only two categories. Nonetheless, I would refrain from having more than 7 categories because most humans cannot reason effectively with that many levels of nested for-loops (that’s how cross products boils into).

I also separated desires from problems because I realized that way too often people (manager, clients, customers, government, etc.) ask the wrong question (because they narrowed it to the wrong problem) that leads to a lot of wasted work and frequent direction changes. People are too often asked to solve hard problems that turns out to be the wrong ones for fulfilling the motivating desires. Very few learn to trace it back to the source (desires) and find the correct problem to address, which often have easy solutions that’s more valuable to the requester than what was originally asked. This often leads to unhappy outcomes for everybody that’s avoidable. An emphasis on desires is one of my frequently used approaches to prevent these kind of mishaps.

196 total views, 2 views today