Millions of black people affected by racial bias in health-care algorithms

This is the study on which the article is based:
Dissecting racial bias in an algorithm used to manage the health of populations

 

The point is that this is often not coding but rather it is due to machine learning where the bias is a result of the choice of training samples and can often be implicit.

With Bias in ML algorithms sometimes people pull out two concerns

1) The data set selected just doesn't have samples from particular minority groups. So there are a number of examples of things like facial recognition systems or even automated taps that work less well with black people because the developers didn't include data from black people in the training set.

2) Inferred Bias. This is where there is potential correlation which can also represent societal biases in datasets which is then learned into the algorithm and features that correlate become used to predict. An example given here is around crime prediction where again race becomes a 'predicting factor' where it may really relate to being disadvantaged but the easy feature for a classifier to pick up on is race.

There are many more issues with the use of ML and biases and there is an active research field around the ethics of AI and how things like explanation become important tools that should be used prior to deploying a model.

A classic example was where a classifier was trained to recognize cats and dogs but all the cats pictures included grass and all the dog pictures didn't. So what it learned was to recognize small furry animals on grass or other surfaces. This was due to bad training (and test) data. It reflects the dangers of ML and the need to be very careful about inbuilt biases.

Maybe the issue is that ML really isn't very good and can pick up on the wrong features and will only generalize from the examples that it has been given. Image recognition systems will, for example, often pick up on texture rather than shape so if you replace the fur on a cat picture with a texture like rhino skin then an ML algorithm will often see the cat as a rhino even though the shape is very different.

That is without going into the attack space where someone may be trying to force a bad decision. My new avatar is an adversarial patch so if you put a small image of that next to a object and get a ML (or one particular algorithm) to classify the object it has a high chance of classifying it as an otter.

 

The issue from an algorithmic perspective (or really an machine learning one) is that if you train on data which include these everyday biases then the algorithm picks up on them. For example in algorithms that sift for good CVs if they are trained on data of what a good CV is based on previous job selections then it will pick up on all the biases of past selections (i.e. it builds in sex, social biases etc)


A good short blog here:
https://www.clsa.com/idea/ethical-ai/

 

Its an example I have heard others giving. I've not gone into detail the point for me is around implicit biases in data sets where an algorithm extracts information that it uses to predict which demonstrates a bias which reflects a correlation in the data (perhaps due to bias) but one that shouldn't be predictive. @Wilhelmina Jenkins point about black people being less likely to turn to the health system and therefore treated as not as sick is another really good example.


It is useful to look at it from an ME perspective but it is also useful to understand the general points that are being made in an area like this (bias in AI/ML algorithms) as it is a very active research area. Then it becomes interesting to put the general points back into the ME world.

For example, I worry that if ML algorithms were to be applied to ME diagnosis the data may reflect something about the doctor that someone visits (and hence the diagnosis ME, CFS, MUS, BDS,....) rather than the symptoms and accuracy of the diagnosis. This in turn could pull out strange correlations are predictions such as wealth (to see private doctors) or where someone lives. Rather than picking up on actual symptoms.

Another concern which I have read is that if algorithms reflect current practices (say with automated diagnosis and treatment recommendations) it could become very hard to update medical knowledge and it could mean treatment strategies become very static. So for example an AI system could learn that on diagnosis doctors recommend CBT/GET for ME patients. If this process become automated it becomes entrenched and self-reinforcing and very hard to change as new knowledge should influence practice.