Solution to DISTINCT Case Problem

First off, take a look at my post from yesterday.  I was given to understand that DISTINCT is not case-sensitive.  The issue at hand actually has less to do with DISTINCT itself, and more to do with the character set and collation that are used when comparing strings.  The best way to describe this comes from the MySQL documentation:

Suppose that we have an alphabet with four letters: A, B, a, b. We give each letter a number: A = 0, B = 1, a = 2, b = 3. The letter A is a symbol, the number 0 is the encoding for A, and the combination of all four letters and their encodings is a character set.
Suppose that we want to compare two string values, A and B. The simplest way to do this is to look at the encodings: 0 for A and 1 for B. Because 0 is less than 1, we say A is less than B. What we’ve just done is apply a collation to our character set. The collation is a set of rules (only one rule in this case): “compare the encodings.” We call this simplest of all possible collations a binary collation.
But what if we want to say that the lowercase and uppercase letters are equivalent? Then we would have at least two rules: (1) treat the lowercase letters a and b as equivalent to A and B; (2) then compare the encodings. We call this a case-insensitive collation. It is a little more complex than a binary collation.
In real life, most character sets have many characters: not just A and B but whole alphabets, sometimes multiple alphabets or eastern writing systems with thousands of characters, along with many special symbols and punctuation marks. Also in real life, most collations have many rules, not just for whether to distinguish lettercase, but also for whether to distinguish accents (an “accent” is a mark attached to a character as in German Ö), and for multiple-character mappings (such as the rule that Ö = OE in one of the two German collations).

So it’s more about how the table is configured to sort strings.  This is the collation.  In my case, the table is set to have a collation of “utf8_unicode_ci”.  When I grab all of the unique passwords, then, I can use a utf8 binary collation:

select distinct password collate utf8_bin from table;

This will maintain all of the upper-case and lower-case permutations of a given password.  By doing this, I can generate all of the hashes of the absolutely unique passwords known to have been used by someone from the data set that I have.

So the next time you use DISTINCT in a query, you may want to make sure you’re getting what you think you are.