Some researchers welcomed the move, suggesting it signals a growing awareness of the importance of this area of inquiry. Others raised concerns about Amazon’s participation, pointing to potential conflicts of interest for researchers who would scrutinize artificial intelligence technologies at both the company and its competitors.
One researcher likened the arrangement to the fox guarding the hen house.
A grant solicitation released last week is one of the first from the foundation — the foremost federal funder of computer-science research at universities — specifically focused on fairness in AI, though the agency has long supported research in areas such as big-data analysis, computer vision and machine learning that underpin the systems lumped under the catchall term AI.
The goal of the Amazon-National Science Foundation joint funding is to support research into “transparency, explainability, accountability, potential adverse biases and effects, mitigation strategies, validation of fairness, and considerations of inclusivity” in AI systems, enabling “broadened acceptance” of the systems. The program would parcel out up to $7.6 million — half from Amazon — over three years to as many as nine selected projects.
Even that relatively small sum is a meaningful infusion into a crucial, complex aspect of AI research that has been understudied, as evidenced by a growing catalog of instances in which AI systems have produced racially biased recidivism predictions, misidentified people of color, generated gender-biased language and downgraded resumes from female job applicants for technical roles.
“It’s really quite huge,” said Bill Howe, an associate professor in University of Washington’s Information School, referring both to the funding and Amazon’s participation. He added: “This is the first that’s expressly focused on fairness for AI. So it’s something that’s been a long time coming and it’s good to see it out there.”
An Amazon executive said in a blog post that the company and the National Science Foundation have each committed to $10 million in funding through 2021, and plan additional calls for proposals.
Other researchers say Amazon’s participation — as a corporation that fields and profits from the AI technologies in question — is problematic, raising potential conflicts of interest and giving the company insight into the work of a research community that may be scrutinizing its products, and those of its competitors, on the very topics that the grant would fund.
“It puts researchers in an odd predicament,” said Nicholas Weber, an assistant professor, also at the UW’s Information School, who studies research funding and transparency, as well as national research policy. “These corporate co-sponsors are more or less piggybacking on the thorough and unique system of peer review that NSF organizes.”
The commingling of funding from Amazon and the foundation could cause researchers who don’t want to accept money from the company to avoid a conflict of interest not to apply to a federal funding program that is otherwise uniquely tailored to their area of focus, he said.
Sarah T. Roberts, an assistant professor in the U.C.L.A. department of information studies, said on Twitter last week: “I’d be the first to love a cut of this funding to support research and researchers but … ever feel like there are a lot of foxes guarding the hen house?”
The varied reactions to Amazon’s co-sponsorship of this federal research program point to a tension between the technology giants that develop and deploy AI, and their erstwhile collaborators in academia.
There is a long history of industry-academic collaboration, particularly in the technology sector, with universities providing basic research and trained professionals, and their industrial partners providing guidance, funding and employment opportunities. The funding from the National Science Foundation is an increasingly important conduit.
But there are examples of that relationship becoming adversarial as academic researchers identify and publicize the shortcomings and negative repercussions of AI systems.
A study by Deborah Raji of the University of Toronto and Joy Buolamwini of MIT Media Lab found Amazon’s facial-recognition technology, Rekognition, was worse at identifying gender when analyzing female and darker-skinned faces than competing technologies from IBM and Microsoft. Amazon pushed back on the researchers’ methodology and findings and said a more recent version of the technology was not tested.
“Their claims of being bias free are based on internal evaluations,” Buolamwini wrote in a response earlier this year. “This is why we did an external evaluation to provide an outside perspective.”
The Amazon-NSF grant program, like other corporate partnerships at the National Science Foundation, is structured to maintain research independence and was evaluated to ensure it gives Amazon no competitive advantage, said spokesman Robert Margetta.
“There’s a mutual understanding between NSF and Amazon, as with all public private partnerships that we enter in to, that the purpose of this is to advance fundamental research,” he said.
Corporate research partnerships are not new at the foundation. Margetta said the computer-science directorate within the agency has run corporate partnerships for at least a decade, with an increased emphasis in recent years.
Meanwhile, the foundation’s budget is shrinking. The agency’s budget — mostly allocated to funding basic research — is nearly $8.1 billion in the current federal fiscal year. Its proposed 2020 budget cuts funding by more than a $1 billion. (National Science Foundation-funded projects at the UW amounted to $116 million last fiscal year.)
Miriam Vogel, executive director of EqualAI, one of a growing number of nonprofits and research groups focused on issues of racial and gender bias in AI, said it’s encouraging to see Amazon and the foundation take this step. Vogel previously worked at the Department of Justice, leading a program to train federal law-enforcement agents to recognize implicit or unconscious bias.
Many of the biased outcomes produced by AI systems today reflect biases in the data sets that train them. Those data sets, such as arrest records or hiring patterns, may contain racial or gender biases.
Vogel said researchers and the industry need to develop tools that can identify those latent biases before they are perpetuated and amplified by the AI systems.
“This is not an insignificant problem,” she said. “This is something that they need to be aware of.”
Amazon Vice President Prem Natarajan, who heads natural language understanding work in the company’s Alexa AI group, said in a blog post that “the fairness of the machine-learning systems we build to support our businesses is critical to establishing and maintaining our customers’ trust.”
He said Amazon believes it “must work closely with academic researchers” on ways “to ensure that biases in data don’t get embedded in the systems we create.”
The AI fairness grant solicitation states explicitly that Amazon will not be part of the selection process, and that the company does not seek information from researchers that would not otherwise be widely shared.
But Weber at the UW said a closer read of the solicitation reveals that Amazon will benefit in other ways.
The company will get insight into the researchers and institutions that apply, and access to those who win awards.
Winners will have the option to consult with Amazon researchers, and their students may be offered internship opportunities with the company. They may also have access to Amazon technology and internal research.
Grant winners will enter into an agreement with Amazon, such as a contract or gift agreement, for the company’s portion of their funding.
As with other similar public-private partnerships, Amazon would have the same rights to intellectual property developed through the funding as the federal government. “That includes a nonexclusive, royalty-free right to the technology,” Margetta said. “The research community is going to know that from the moment they apply. That transparency is really important to us.”
A senior representative of each funded research team will be required to attend two annual meetings with other researchers, one of which is to be held in Seattle. “The one in Seattle will be held at Amazon,” Margetta said.
Also, the solicitation points researchers to an “additional and separate funding opportunity”: Amazon’s own research awards program.
Weber said the call-out of a separate corporate research awards program in the National Science Foundation solicitation is “completely bizarre and novel.” Margetta said that was indeed new in the Amazon solicitation, but is in keeping with the agency’s goal “to stimulate research and connect researchers with all the funding sources we can.”
©2019 The Seattle Times. Distributed by Tribune Content Agency, LLC.
