Annual Index Report Measures the State of AI

Artificial intelligence has dominated tech conversations lately, from data protections to advances that bring the technology into both the home and the workplace. But with so much happening, the state of AI has been difficult at times to fully grasp. What is progress, and what is merely hype? Enter the AI Index 2018 Annual Report, authored by authorities from a variety of institutions including MIT, Harvard, and OpenAI. They examined data across all aspects of AI to measure its ongoing boom. In their words, the index report is part of a mission to ground the conversation about AI in data. The report, which aims to be a comprehensive resource for policymakers, journalists, researchers and executives about a complex field looked at everything from AI research and policy to diversity in the field.

Several things about the state of AI stand out in the report. For starters, from its beginnings in academic research, AI is increasingly being commercialized, and the number of artificial intelligence-related startups is growing significantly each year. That investment will drive development, but it also points to the hype surrounding AI--which doesn’t always pan out into real progress or development.

The index showed that AI is still centered in China and the United States, though European development is also significant. For example, 83 percent of the AI papers on Scopus from 2017 originate outside the United States, and 28 percent of those originate in Europe. In 2014, South Korea and Japan were the second- and third-largest producers of AI patents after the United States. And educational interest in artificial intelligence is increasing worldwide, as well. For example, at Tsinghua in China, the combined artificial intelligence and machine learning course enrollment was 16 times larger in 2017 than in 2010.

“AI is a more global field today than perhaps ever before, with a significant uptick [in the last several years] in the extent to which governments, companies and funders across the world are paying attention and participating,” said Terah Lyons, executive director of the Partnership on AI.

The field is also still heavily male dominated, with sectors like speech recognition and natural language processing having nearly three male job applicants for each female applicant, and educators at major institutions including University College London and and Stanford remaining overwhelmingly male.

“Diversity remains a central challenge of the field,” said Lyons, who was on the index’s steering committee. “Much work remains to be done to make the AI community more representative.” The work of a few organizations in this area was recognized in the index, she said, but there are several others working to make AI more diverse by gender and racial identity.

The index's findings on the state of AI were echoed by the recent World Economic Forum report on the global economic gender gap, Chessa Eskandanian-Yee, CEO of LeaderEQ, pointed out. The WEF report found that AI’s gender gap is three times larger than that in other industries, with 78 percent of the professionals with AI skills being male.

Addressing that gap is about not just hiring but also about product development, Eskandanian-Yee said, because machines depend on the data supplied by their inventor.

“Being a minority woman in a business that addresses the importance of emotional intelligence in the workplace and particularly with engineers in the tech space, it was imperative that our product communicated the assessment and data in a non-binary format, she said. “From the wording of our questions, to the way our data is collected, the algorithm is non-biased to gender and race.”

But the index also found that despite AI’s challenges, the technology is improving--quickly.

For example, the accuracy of object recognition in images, measured against human performance, went from about 72 percent in 2010 to about 97 percent--better than human performance--in 2017, the most recent year measured. The accuracy of machine translations of news articles has also improved quickly, as measured by scores assigned by human judges.

The interest in topics like neural networks and machine learning is driving AI’s continuous advancement, said Briana Brownell, founder and CEO of PureStrategy.ai.

“It's done wonders for both industry and academia,” Brownell said. “Performance compared to benchmarks is getting better and better--so much more is now possible.”