I suspect TMW (The MathWorks, maker of MATLAB) hasn’t really thought about dead levels when a categorical object\u00a0(I mean nominal() and ordinal() as well since they are wrapper child class of categorical()) has elements removed so that some levels doesn’t map to any elements anymore.<\/p>\n
For performance reasons, it makes sense to keep the dead levels in\u00a0because the user can repetitively add and remove the same last level by deleting and adding the same element, causing unnecessary work each time. Naturally, there’s a getlevels()\/getlabels()\/categories() method\u00a0in nominal(),ordinal()\/categorical() class\u00a0so you know what raw<\/span> levels are\u00a0available. Turns out it’s a horrible idea to expose the raw\u00a0levels when dead levels are allowed!<\/p>\n Unless you are dealing with the internals of categorical objects, there’s very little reason why one would\u00a0care or want to know about the dead\u00a0levels (it’s just a cache for performance). It’s the active levels that are currently mapped to some elements that matters when user make such queries, which is\u00a0handled correctly by unique().<\/p>\n If there are no dead levels, getlevels() is equivalent to unique(), while categorical() and getlabels() are equivalent to unique(cellstr()), but I’m very likely to run into dead levels\u00a0because I delete rows of data when I filter by certain criterion.<\/p>\n My first take on it would be to hide\u00a0getlevels()\/getlabels()\/categories() from users. But over the years, I’ve grown from a conservative software point of view to accepting more liberal approach, especially after exposure to functional programming ideas. That means I’d rather have a way to know what’s going on inside (keep those functions there), but I’d like to be warned that it’s an evil feature that shouldn’t be used lightly.<\/p>\n Yes,\u00a0I’m\u00a0dissing the use of\u00a0getlevels()\/getlabels()\/categories() like the infamous eval(). Once in a long while, it might be a legitimate neat approach. But for 99% of the time, it’s a strictly worse solution that causes a lot of damages. It’s way more unlikely that\u00a0getlevels()\/getlabels()\/categories() will yield what you really mean with dead levels than multiple inheritance in C++ being the right approach on the first try.<\/p>\n If I use unique() all the time, why would I even bother to talk about getlevels()\/getlabels()\/categories() since I never used them? It’s because\u00a0TMW didn’t warn users about the dangers\u00a0in their documentation. These methods looks legit and innocent, but it’s a usage trap like returning stack pointers in C\/C++ (you can technically do it, but\u00a0with almost 100% certainty, you are telling the computer to do something you don’t mean to, in short: wrong).<\/p>\n I have two encounters that other people using\u00a0the raw categorical levels that harmed me:<\/p>\n Apparently the author of the factory-shipped code forgot that there’s a reason why the categorical\/unique() has the same\u00a0function name as double\/unique() and cellstr\/unique():\u00a0the point of overloading is to have the same function name for\u00a0the same intention<\/span>! The intention<\/span> of unique() should be uniformly applied across all the data types applicable. Think twice before relying on language support for type info (like type traits in C++) to switch code when you can use function overloading (MATLAB differentiates by the type of the first argument, C++ looks at the whole signature). A good architecture should lead you to the correct code logic without the need of overriding good practices.<\/p>\n Rants aside, grpstats() will work as intended if those lines in grp2idx() are changed to:<\/p>\n A higher level fix would be applying grp2idx() to the grouping variable before it was fed into grpstats():<\/p>\n The rationale is that\u00a0the underlying\u00a0contents\u00a0doesn’t matter for grouping variables as long as each of them\u00a0uniquely stand for\u00a0the\u00a0group they\u00a0represent! In other words,\u00a0categorical() objects are seen as nothing but a bunch of integers, which can be obtained by casting it to double():<\/p>\n This is what grp2idx() calls under the hood anyway when it sees a categorical. The grp2idx() called from\u00a0grpstats() will see a bunch of integers, which will correctly apply unique() to them, thus removing all dead levels.<\/p>\n Of course, use grp2idx() instead of double() because it works across all data types that applies. Why future-constrain yourself when a more generic implementation is already available?<\/p>\n The sin committed by grpstats() over nominal() is that the variables in glevels and gnames shouldn’t get involved in the first place because they don’t matter and shouldn’t even show up in the outputs. This is what’s fundamentally wrong about it:<\/p>\n We can either blame the for-loop for not skipping dead levels, or blame mgrp2idx (a wrapper of grp2idx) for spitting out the dead levels. It doesn’t really matter which way it is. The most important thing is that\u00a0dead levels were\u00a0let loose, and nobody in the developer-user chain understand the implications\u00a0enough to stop the problem from propagating to the final output.<\/li>\n<\/ol>\n To summarize, the raw<\/span> levels in categorical objects is a dirty cache including junk you do not want 99.99% of the time. Use unique() to get the meaningful unique levels instead.<\/p>\n\n
gidx = double(s);\n...\ngnames = getlabels(s)';\nglevels = getlevels(s)';<\/pre>\n
gidx = double(s);\n...\nglevels = unique(s(:));\ngnames = cellstr(glevels);\n<\/pre>\n
grpstats(X, grp2idx(g), ...)<\/pre>\n
gidx = double(s);\ngrpstats(X, gidx, ...)<\/pre>\n
[group,...,ngroups] = mgrp2idx(group,rows);\n...\n\/\/ This code assumes there are no gaps in group levels (gnum), which is not always true.\nfor gnum = 1:ngroups\n groups{gnum} = find(group==gnum);\nend<\/pre>\n