Bias in the Machine

Ahren Lehnert, Senior Manager at Knowledge organisation solutions provider Synaptica has recently published an article that examines the effects of bias in the content cycle.  To kick off his study and set the scene, he revisits an incident that occurred over three years ago when Google’s facial recognition technology incorrectly identified two African American subjects as ‘Gorillas’.  At the time, another article by my co-contributor Ralph Windsor also examined the perils and politics of automated metadata.

“What happened? Is Google designing racist software? Is automatic image categorization horribly inaccurate by nature? Did a software engineering team not do its job?”  [Read More]

Interestingly, instead of fixing the issue in the software, there are numerous reports that Google have simply removed gorillas (and other primates) from the service’s image labelling technology.  The fact that such a workaround has been deployed might not only highlight the weight of the risk and subsequent embarrassment should it happen again, but also the inherent complexity of overcoming some of the most basic visual cognitive hurdles required to make image recognition as good as its human counterparts.

Ahren’s article adopts a forensics role and sets up a number of initial possibilities as to why such mistakes might happen within an AI-assisted tagging system.  At a base level, this may be due to a lack, or over-abundance of, training material:

“In this case, there may have been insufficient examples of people with a variety of skin tones training the “people” category. Likewise, there may have been an overabundance of ape and monkey sample images. The final result is a machine learning algorithm which can more easily distinguish a light skin-toned individual as a person while not being able to distinguish between the human-like faces of apes and monkeys and the darker skin tones of human faces.”  [Read More]

In turn, the article points the finger at both the human and technological modus operandi throughout the content lifecycle, with a specific focus on bias.  Let’s take a look at Wikipedia’s definition of bias:

“Bias is disproportionate weight in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair. Biases can be learned implicitly within cultural contexts.”

Ahren breaks this bias down into five areas:

• Bias in the machine
• Bias in the content cycle
• Bias in the taxonomy
• Bias in the text analytics

The first bias has already been covered and might be traced to an unevenly distributed amount of training material (an abundance of Caucasian faces, not enough faces with darker skin tones, too many primates, no albino apes etc.).  If we only train the machine with pictures of apples, how can we possibly expect it to recognise an orange?  Of course, such bias is not limited to images and any type of system can suffer from this lack of diversity at the input stage and thus skew the output.

Bias in the content cycle addresses the generation of the content itself – a task that is usually undertaken by humans.  Shortcomings in this area tend to be more prevalent.

“Content is usually generated by people, and people have biases. The language in the document particular to the writer, the thoughts expressed in the content, or the amount of content generated about one subject and not another are all biases feeding into the content cycle. The content may be heavily skewed toward one area and minimal in another, presenting an inaccurate and slanted view of the subject matter. Intentionally or not, content may also carry political baggage, expressing a viewpoint shared by a minority in the organization or a viewpoint supported by inaccurate assumptions. The content may also be outdated, expressing concepts in terms which are no longer acceptable or accurate. If you are modeling the current world on old information, there is bound to be a disconnect.”  [Read More]

Next up is bias in the taxonomy.  This is where the chicken-or-egg type paradox often occurs as taxonomies are generally built based on the content, yet if there isn’t an obvious container for the content in the taxonomy, it might be dismissed or incorrectly categorised.  For example, in our fruit taxonomy, what happens when the first orange comes along if there’s no slot for it?  As Ahren mentions, “bias in, bias out”.  He also addresses the issue of bias in the taxonomy designer:

“Even if the content of the taxonomy maintains a layer of separation from the content, the taxonomy builder may have biases about information organization based on his or her role as a central point for metadata creation. His or her perception about the way information should be organized and the concepts used for that organization may be biased by the information organization principles themselves.” [Read More]

Bias in the text analysis is the next suspect in the line-up and covers the pitfalls of misrepresentation due to shortcomings in language processing, applying unbalanced weightings, or arbitrarily relying on frequency to gauge the relevance of the content.  This is also where bias can literally become a case of lost in translation.

“The frequency or importance of concepts can be further diluted when captured and made part of a taxonomy. If a concept appears 1000 times in 10 documents and another term appears 100 times in those same 10 documents, they can both become a single instance of a concept in the taxonomy. A leveling effect like this, while creating potential equality of concepts in the taxonomy, doesn’t translate the importance back to the content when applied as metadata.” [Read More]

The article concludes by stating that, although examples of biased software can be shrugged off as teething trouble, the real-world consequences when such algorithms are used in crime prevention, high-risk investment or autonomous vehicles – as they invariably will – might be considerably more damaging.

Share this Article: