It is necessary to open the "black box" of artificial intelligence until it's too late

For several years in the 1980s, the entrants to St. George's Hospital School of Medicine in London were selected using a high-tech method. A computer program, one of the first of its kind, looked through the summary, choosing from among all applications about 2000 candidates per year. The program analyzed the records of receipts, examining the characteristics of successful applicants, and adjusted until its decisions did not coincide with the opinion of the admissions committee.

However, the program has learned to find more than good grades and signs of academic achievement. Four years after the program, two doctors in the hospital found that the program, as a rule, rejected women applicants and persons with non-European names, regardless of their academic merits. Doctors found that about 60 applicants were simply denied an interview every year because of their gender or race. The program included gender and racial prejudices in the data used for its training - in fact, it was learned that doctors and foreigners are not the best candidates for a doctor.

Thirty years later, we encountered a similar problem, but programs with internal prejudices are now more widespread and make decisions with even higher rates. Artificial intelligence algorithms based on machine learning are used everywhere, from government agencies to the healthcare sector, making decisions and making predictions based on historical data. Studying the patterns in the data, they also absorb and prejudice in them. Google, for example, shows more advertising for low-paid jobs for women than for men; one-day delivery of Amazon passes black neighborhoods, and digital cameras can hardly recognize faces that are not white.

It is difficult to understand whether the algorithm is biased or fair, and so even computer experts believe. One of the reasons is that the details of creating an algorithm are often considered patented information, so they are carefully guarded by the owners. In more complex cases, the algorithms are so complex that even the creators do not know exactly how they work. This is the problem of the so-called "black box" of AI - our inability to see the inside of the algorithm and understand how it comes to solution. If we leave it locked, our society can seriously suffer: the digital environment realizes historical discrimination, which we have struggled with for many years, from slavery and serfdom to discrimination against women.

These concerns, voiced in small communities of computer scientists before, are now gaining a serious turn. Over the past two years, a lot of publications on the transparency of artificial intelligence have appeared in this area. Along with this awareness, a sense of responsibility is growing. "Could there be any things that we should not build?" Asks Keith Crawford, a Microsoft researcher and co-founder of the AI ​​Now Insitute in New York.

"Machine learning has finally come to the fore. Now we are trying to use it for hundreds of different tasks in the real world, "says Rich Caruana, a senior research fellow at Microsoft. "It is possible that people will be able to deploy malicious algorithms that will significantly affect society in the long term. Now, it seems, suddenly everyone understood that this is an important chapter in our field. "

Unauthorized Algorithm

We have been using algorithms for a long time, but the black box problem has no precedents. The first algorithms were simple and transparent. Many of them we still use - for example, to assess creditworthiness. With each new use, regulation comes into play.

"People used algorithms to assess credit worthiness for decades, but in these areas there were pretty strong settlements that grew in parallel with the use of predictive algorithms," says Caruana. The rules of regulation ensure that the forecasting algorithms give an explanation for each score: you were denied because you have a large loan or too low income.

In other areas, such as the legal system and advertising, there are no rules prohibiting the use of deliberately unspoken algorithms. You may not know why you were refused a loan or taken to work, because no one forces the algorithm owner to explain how it works. "But we know that because algorithms are trained on real-world data, they must be biased - because the real world is being foretold," says Caruana.

Consider, for example, language - one of the most obvious sources of bias. When algorithms are trained on written text, they form some associations between words that appear together more often. For example, they learn that "for a man to be a computer programmer is the same as for a woman to be a housewife." When this algorithm is instructed to find a suitable resume for the programmer, most likely, he will choose among the male candidates.

Such problems are quite easy to correct, but many companies will not simply do it. Instead, they will hide such inconsistencies behind the shield of protected information. Without access to the details of the algorithm, experts in many cases will not be able to determine whether there is a prejudice or not.

Since these algorithms are secret and remain outside the jurisdiction of regulatory bodies, it is almost impossible for citizens to sue the creators of algorithms. In 2016, the Supreme Court of Wisconsin rejected a person's request to consider the internal work of COMPAS. The man, Eric Loomis, was sentenced to six years in prison, in part because COMPAS considered him "high-risk." Loomis says that his right to proper procedure was violated by the judge's dependence on the non-transparent algorithm. The final application for consideration of the case in the US Supreme Court failed in June 2017.

But secret companies will not enjoy their freedom for an unlimited time. By March, the EU will adopt laws that will require companies to explain to interested customers how their algorithms work and how they make decisions. The US does not have such legislation in development.

Forensics black box

Regardless of whether regulators are involved in all this, the cultural shift in how algorithms are developed and deployed can reduce the prevalence of biased algorithms. As more and more companies and programmers commit to making their algorithms transparent and understandable, some hope that companies that do not do this will lose a good reputation in the public eye.

The growth of computing power allowed creating algorithms that are both accurate and explainable - developers could not overcome this technical task historically. Recent research shows that it is possible to create explainable models that predict the recurrence of criminal subjects as accurately as a black box of criminalists like COMPAS.

"Everything is ready - we know how to create models without black boxes," says Cynthia Rudin, assistant professor of computer science and electrical engineering at Duke University. "But it's not so easy to draw people's attention to this work. If government agencies stop paying for black box models, this would help. If the judges refuse to use the black box model for sentencing, this will also help. "

Others are working to come up with ways to test the validity of algorithms by creating a system of checks and balances before the algorithm is released into the world, just as every new drug passes the test.

"Now the models are being made and deployed too quickly. No proper checks are made before the algorithm is released into the world, "says Sara Tan of Cornell University.

Ideally, developers should sweep out known biases - for example, by sex, age and race - and run internal simulations to test their algorithms for other problems.

In the meantime, before reaching the point where all algorithms will be thoroughly tested before release, it is already possible to determine which ones will suffer from bias.

In their latest work, Tang, Caruana and their colleagues described a new way to understand what could happen under the hood of the black box algorithms. Scientists have created a model that simulates the black box algorithm, learning to assess the risk of recidivism according to COMPAS. They also created another model that was trained according to the real world, which shows whether the predicted recidivism really happened. Comparison of the two models allowed scientists to estimate the accuracy of the predicted score without analyzing the algorithm. Differences in the results of the two models can show which variables, such as race or age, can be more important in a particular model. Their results showed that COMPAS prejudiced against black people.

Correctly constructed algorithms can eliminate long-established prejudices in the field of criminal justice, police activities and many other spheres of society.

The article is based on materials https://hi-news.ru/computers/nuzhno-vskryt-chernyj-yashhik-iskusstvennogo-intellekta-poka-ne-stanet-slishkom-pozdno.html.